Alternatives and similar repositories for deduplicate-text-datasets

Users that are interested in deduplicate-text-datasets are comparing it to the libraries listed below

• ChenghaoMou / text-dedup
  All-in-one text de-duplication
  ☆697Updated last month
• facebookresearch / cc_net
  Tools to download and cleanup Common Crawl data
  ☆1,017Updated 2 years ago
• facebookresearch / DPR
  Dense Passage Retriever - is a set of tools and models for open domain Q&A task.
  ☆1,816Updated 2 years ago
• facebookresearch / contriever
  Contriever: Unsupervised Dense Information Retrieval with Contrastive Learning
  ☆744Updated 2 years ago
• google-research / FLAN
  ☆1,528Updated 2 weeks ago
• Muennighoff / sgpt
  SGPT: GPT Sentence Embeddings for Semantic Search
  ☆868Updated last year
• beir-cellar / beir
  A Heterogeneous Benchmark for Information Retrieval. Easy to use, evaluate your models across 15+ diverse IR datasets.
  ☆1,865Updated last month
• castorini / pyserini
  Pyserini is a Python toolkit for reproducible information retrieval research with sparse and dense representations.
  ☆1,885Updated this week
• bigscience-workshop / bigscience
  Central place for the engineering/scaling WG: documentation, SLURM scripts and logs, compute environment and data.
  ☆1,000Updated 11 months ago
• facebookresearch / atlas
  Code repository for supporting the paper "Atlas Few-shot Learning with Retrieval Augmented Language Models",(https//arxiv.org/abs/2208.03…
  ☆540Updated last year
• lucidrains / RETRO-pytorch
  Implementation of RETRO, Deepmind's Retrieval based Attention net, in Pytorch
  ☆868Updated last year
• google-research / prompt-tuning
  Original Implementation of Prompt Tuning from Lester, et al, 2021
  ☆686Updated 4 months ago
• bigscience-workshop / data-preparation
  Code used for sourcing and cleaning the BigScience ROOTS corpus
  ☆313Updated 2 years ago
• allenai / natural-instructions
  Expanding natural instructions
  ☆1,007Updated last year
• huggingface / transformers-bloom-inference
  Fast Inference Solutions for BLOOM
  ☆564Updated 9 months ago
• allenai / tango
  Organize your experiments into discrete steps that can be cached and reused throughout the lifetime of your research project.
  ☆562Updated last year
• bigscience-workshop / Megatron-DeepSpeed
  Ongoing research training transformer language models at scale, including: BERT & GPT-2
  ☆1,398Updated last year
• google-research / bleurt
  BLEURT is a metric for Natural Language Generation based on transfer learning.
  ☆739Updated last year
• bigscience-workshop / t-zero
  Reproduce results and replicate training fo T0 (Multitask Prompted Training Enables Zero-Shot Task Generalization)
  ☆463Updated 2 years ago
• facebookresearch / FiD
  Fusion-in-Decoder
  ☆572Updated last year
• SinclairCoder / Instruction-Tuning-Papers
  Reading list of Instruction-tuning. A trend starts from Natrural-Instruction (ACL 2022), FLAN (ICLR 2022) and T0 (ICLR 2022).
  ☆769Updated last year
• PiotrNawrot / nanoT5
  Fast & Simple repository for pre-training and fine-tuning T5-style models
  ☆1,005Updated 10 months ago
• Tiiiger / bert_score
  BERT score for text generation
  ☆1,760Updated 11 months ago
• allenai / RL4LMs
  A modular RL library to fine-tune language models to human preferences
  ☆2,325Updated last year
• princeton-nlp / DensePhrases
  [ACL 2021] Learning Dense Representations of Phrases at Scale; EMNLP'2021: Phrase Retrieval Learns Passage Retrieval, Too https://arxiv.o…
  ☆604Updated 3 years ago
• dorianbrown / rank_bm25
  A Collection of BM25 Algorithms in Python
  ☆1,200Updated 9 months ago
• google / seqio
  Task-based datasets, preprocessing, and evaluation for sequence models.
  ☆583Updated 2 months ago
• ict-bigdatalab / awesome-pretrained-models-for-information-retrieval
  A curated list of awesome papers related to pre-trained models for information retrieval (a.k.a., pretraining for IR).
  ☆670Updated last year
• huggingface / cosmopedia
  ☆522Updated 7 months ago
• naver / splade
  SPLADE: sparse neural search (SIGIR21, SIGIR22)
  ☆866Updated last year