Alternatives and similar repositories for deduplicate-text-datasets

Users that are interested in deduplicate-text-datasets are comparing it to the libraries listed below

• ChenghaoMou / text-dedup
  All-in-one text de-duplication
  ☆727Updated 2 months ago
• facebookresearch / cc_net
  Tools to download and cleanup Common Crawl data
  ☆1,030Updated 2 years ago
• google-research / FLAN
  ☆1,551Updated 2 weeks ago
• facebookresearch / contriever
  Contriever: Unsupervised Dense Information Retrieval with Contrastive Learning
  ☆760Updated 2 years ago
• beir-cellar / beir
  A Heterogeneous Benchmark for Information Retrieval. Easy to use, evaluate your models across 15+ diverse IR datasets.
  ☆1,996Updated last month
• castorini / pyserini
  Pyserini is a Python toolkit for reproducible information retrieval research with sparse and dense representations.
  ☆1,978Updated this week
• Muennighoff / sgpt
  SGPT: GPT Sentence Embeddings for Semantic Search
  ☆873Updated last year
• facebookresearch / DPR
  Dense Passage Retriever - is a set of tools and models for open domain Q&A task.
  ☆1,843Updated 2 years ago
• allenai / natural-instructions
  Expanding natural instructions
  ☆1,023Updated last year
• facebookresearch / atlas
  Code repository for supporting the paper "Atlas Few-shot Learning with Retrieval Augmented Language Models",(https//arxiv.org/abs/2208.03…
  ☆551Updated last year
• bigscience-workshop / bigscience
  Central place for the engineering/scaling WG: documentation, SLURM scripts and logs, compute environment and data.
  ☆1,006Updated last year
• google-research / prompt-tuning
  Original Implementation of Prompt Tuning from Lester, et al, 2021
  ☆696Updated 8 months ago
• texttron / tevatron
  Tevatron - Unified Document Retrieval Toolkit across Scale, Language, and Modality. Demo in SIGIR 2023, SIGIR 2025.
  ☆704Updated last month
• dorianbrown / rank_bm25
  A Collection of BM25 Algorithms in Python
  ☆1,265Updated last year
• allenai / dolma
  Data and tools for generating and inspecting OLMo pre-training data.
  ☆1,343Updated last week
• bigscience-workshop / Megatron-DeepSpeed
  Ongoing research training transformer language models at scale, including: BERT & GPT-2
  ☆1,427Updated last year
• huggingface / evaluate
  🤗 Evaluate: A library for easily evaluating machine learning models and datasets.
  ☆2,359Updated this week
• bigscience-workshop / data-preparation
  Code used for sourcing and cleaning the BigScience ROOTS corpus
  ☆316Updated 2 years ago
• bigscience-workshop / t-zero
  Reproduce results and replicate training fo T0 (Multitask Prompted Training Enables Zero-Shot Task Generalization)
  ☆462Updated 3 years ago
• lucidrains / RETRO-pytorch
  Implementation of RETRO, Deepmind's Retrieval based Attention net, in Pytorch
  ☆875Updated 2 years ago
• SinclairCoder / Instruction-Tuning-Papers
  Reading list of Instruction-tuning. A trend starts from Natrural-Instruction (ACL 2022), FLAN (ICLR 2022) and T0 (ICLR 2022).
  ☆768Updated 2 years ago
• SeanLee97 / AnglE
  Train and Infer Powerful Sentence Embeddings with AnglE | 🔥 SOTA on STS and MTEB Leaderboard
  ☆559Updated 3 weeks ago
• google-research / bleurt
  BLEURT is a metric for Natural Language Generation based on transfer learning.
  ☆769Updated 2 years ago
• naver / splade
  SPLADE: sparse neural search (SIGIR21, SIGIR22)
  ☆943Updated last year
• facebookresearch / FiD
  Fusion-in-Decoder
  ☆587Updated 2 years ago
• huggingface / transformers-bloom-inference
  Fast Inference Solutions for BLOOM
  ☆564Updated last year
• ContextualAI / gritlm
  Generative Representational Instruction Tuning
  ☆678Updated 4 months ago
• allenai / RL4LMs
  A modular RL library to fine-tune language models to human preferences
  ☆2,366Updated last year
• Shark-NLP / OpenICL
  OpenICL is an open-source framework to facilitate research, development, and prototyping of in-context learning.
  ☆578Updated 2 years ago
• microsoft / DeBERTa
  The implementation of DeBERTa
  ☆2,164Updated 2 years ago