Alternatives and similar repositories for deduplicate-text-datasets

Users that are interested in deduplicate-text-datasets are comparing it to the libraries listed below

• ChenghaoMou / text-dedup
  All-in-one text de-duplication
  ☆706Updated 3 weeks ago
• facebookresearch / cc_net
  Tools to download and cleanup Common Crawl data
  ☆1,024Updated 2 years ago
• google-research / FLAN
  ☆1,535Updated 2 weeks ago
• facebookresearch / contriever
  Contriever: Unsupervised Dense Information Retrieval with Contrastive Learning
  ☆751Updated 2 years ago
• allenai / natural-instructions
  Expanding natural instructions
  ☆1,012Updated last year
• Muennighoff / sgpt
  SGPT: GPT Sentence Embeddings for Semantic Search
  ☆870Updated last year
• beir-cellar / beir
  A Heterogeneous Benchmark for Information Retrieval. Easy to use, evaluate your models across 15+ diverse IR datasets.
  ☆1,921Updated 2 months ago
• castorini / pyserini
  Pyserini is a Python toolkit for reproducible information retrieval research with sparse and dense representations.
  ☆1,917Updated last week
• google-research / prompt-tuning
  Original Implementation of Prompt Tuning from Lester, et al, 2021
  ☆693Updated 5 months ago
• bigscience-workshop / bigscience
  Central place for the engineering/scaling WG: documentation, SLURM scripts and logs, compute environment and data.
  ☆1,006Updated last year
• facebookresearch / DPR
  Dense Passage Retriever - is a set of tools and models for open domain Q&A task.
  ☆1,828Updated 2 years ago
• facebookresearch / atlas
  Code repository for supporting the paper "Atlas Few-shot Learning with Retrieval Augmented Language Models",(https//arxiv.org/abs/2208.03…
  ☆547Updated last year
• EleutherAI / the-pile
  ☆1,593Updated 2 years ago
• allenai / RL4LMs
  A modular RL library to fine-tune language models to human preferences
  ☆2,340Updated last year
• bigscience-workshop / Megatron-DeepSpeed
  Ongoing research training transformer language models at scale, including: BERT & GPT-2
  ☆1,411Updated last year
• bigscience-workshop / data-preparation
  Code used for sourcing and cleaning the BigScience ROOTS corpus
  ☆314Updated 2 years ago
• anthropics / hh-rlhf
  Human preference data for "Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback"
  ☆1,775Updated 2 months ago
• allenai / dolma
  Data and tools for generating and inspecting OLMo pre-training data.
  ☆1,296Updated last week
• bigscience-workshop / t-zero
  Reproduce results and replicate training fo T0 (Multitask Prompted Training Enables Zero-Shot Task Generalization)
  ☆462Updated 2 years ago
• huggingface / transformers-bloom-inference
  Fast Inference Solutions for BLOOM
  ☆564Updated 10 months ago
• bigscience-workshop / xmtf
  Crosslingual Generalization through Multitask Finetuning
  ☆538Updated 11 months ago
• dorianbrown / rank_bm25
  A Collection of BM25 Algorithms in Python
  ☆1,223Updated 10 months ago
• lucidrains / RETRO-pytorch
  Implementation of RETRO, Deepmind's Retrieval based Attention net, in Pytorch
  ☆871Updated last year
• yaodongC / awesome-instruction-dataset
  A collection of open-source dataset to train instruction-following LLMs (ChatGPT,LLaMA,Alpaca)
  ☆1,128Updated last year
• SinclairCoder / Instruction-Tuning-Papers
  Reading list of Instruction-tuning. A trend starts from Natrural-Instruction (ACL 2022), FLAN (ICLR 2022) and T0 (ICLR 2022).
  ☆769Updated 2 years ago
• google-research / bleurt
  BLEURT is a metric for Natural Language Generation based on transfer learning.
  ☆751Updated 2 years ago
• sylinrl / TruthfulQA
  TruthfulQA: Measuring How Models Imitate Human Falsehoods
  ☆792Updated 7 months ago
• microsoft / DeBERTa
  The implementation of DeBERTa
  ☆2,130Updated last year
• ContextualAI / gritlm
  Generative Representational Instruction Tuning
  ☆664Updated last month
• declare-lab / instruct-eval
  This repository contains code to quantitatively evaluate instruction-tuned models such as Alpaca and Flan-T5 on held-out tasks.
  ☆547Updated last year