Alternatives and similar repositories for deduplicate-text-datasets

Users that are interested in deduplicate-text-datasets are comparing it to the libraries listed below

• ChenghaoMou / text-dedup
  All-in-one text de-duplication
  ☆726Updated 2 months ago
• facebookresearch / cc_net
  Tools to download and cleanup Common Crawl data
  ☆1,030Updated 2 years ago
• google-research / FLAN
  ☆1,550Updated last week
• facebookresearch / contriever
  Contriever: Unsupervised Dense Information Retrieval with Contrastive Learning
  ☆760Updated 2 years ago
• allenai / natural-instructions
  Expanding natural instructions
  ☆1,022Updated last year
• Muennighoff / sgpt
  SGPT: GPT Sentence Embeddings for Semantic Search
  ☆873Updated last year
• castorini / pyserini
  Pyserini is a Python toolkit for reproducible information retrieval research with sparse and dense representations.
  ☆1,974Updated this week
• beir-cellar / beir
  A Heterogeneous Benchmark for Information Retrieval. Easy to use, evaluate your models across 15+ diverse IR datasets.
  ☆1,985Updated 3 weeks ago
• facebookresearch / DPR
  Dense Passage Retriever - is a set of tools and models for open domain Q&A task.
  ☆1,840Updated 2 years ago
• google-research / prompt-tuning
  Original Implementation of Prompt Tuning from Lester, et al, 2021
  ☆696Updated 8 months ago
• bigscience-workshop / bigscience
  Central place for the engineering/scaling WG: documentation, SLURM scripts and logs, compute environment and data.
  ☆1,006Updated last year
• facebookresearch / atlas
  Code repository for supporting the paper "Atlas Few-shot Learning with Retrieval Augmented Language Models",(https//arxiv.org/abs/2208.03…
  ☆549Updated last year
• google-research / bleurt
  BLEURT is a metric for Natural Language Generation based on transfer learning.
  ☆768Updated 2 years ago
• texttron / tevatron
  Tevatron - Unified Document Retrieval Toolkit across Scale, Language, and Modality. Demo in SIGIR 2023, SIGIR 2025.
  ☆702Updated last month
• bigscience-workshop / xmtf
  Crosslingual Generalization through Multitask Finetuning
  ☆537Updated last year
• ContextualAI / gritlm
  Generative Representational Instruction Tuning
  ☆677Updated 4 months ago
• allenai / RL4LMs
  A modular RL library to fine-tune language models to human preferences
  ☆2,363Updated last year
• anthropics / hh-rlhf
  Human preference data for "Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback"
  ☆1,792Updated 4 months ago
• bigscience-workshop / data-preparation
  Code used for sourcing and cleaning the BigScience ROOTS corpus
  ☆316Updated 2 years ago
• lucidrains / RETRO-pytorch
  Implementation of RETRO, Deepmind's Retrieval based Attention net, in Pytorch
  ☆875Updated 2 years ago
• dorianbrown / rank_bm25
  A Collection of BM25 Algorithms in Python
  ☆1,260Updated last year
• SinclairCoder / Instruction-Tuning-Papers
  Reading list of Instruction-tuning. A trend starts from Natrural-Instruction (ACL 2022), FLAN (ICLR 2022) and T0 (ICLR 2022).
  ☆769Updated 2 years ago
• SeanLee97 / AnglE
  Train and Infer Powerful Sentence Embeddings with AnglE | 🔥 SOTA on STS and MTEB Leaderboard
  ☆558Updated 3 weeks ago
• allenai / dolma
  Data and tools for generating and inspecting OLMo pre-training data.
  ☆1,340Updated this week
• huggingface / transformers-bloom-inference
  Fast Inference Solutions for BLOOM
  ☆565Updated last year
• bigscience-workshop / Megatron-DeepSpeed
  Ongoing research training transformer language models at scale, including: BERT & GPT-2
  ☆1,425Updated last year
• facebookresearch / FiD
  Fusion-in-Decoder
  ☆587Updated 2 years ago
• EleutherAI / the-pile
  ☆1,610Updated 2 years ago
• Shark-NLP / OpenICL
  OpenICL is an open-source framework to facilitate research, development, and prototyping of in-context learning.
  ☆576Updated 2 years ago
• bigscience-workshop / promptsource
  Toolkit for creating, sharing and using natural language prompts.
  ☆2,960Updated 2 years ago