Alternatives and similar repositories for pile_dedupe:

Users that are interested in pile_dedupe are comparing it to the libraries listed below

• gmftbyGMFTBY / Rep-Dropout
  [NeurIPS 2023] Repetition In Repetition Out: Towards Understanding Neural Text Degeneration from the Data Perspective
  ☆30Updated last year
• ndaheim / faithful-dialogue
  ☆23Updated last year
• swj0419 / in-context-pretraining
  ☆47Updated 9 months ago
• yuzhaouoe / pretraining-data-packing
  [ACL'24 Oral] Analysing The Impact of Sequence Composition on Language Model Pre-Training
  ☆18Updated 5 months ago
• Muennighoff / FLAN
  Provides a minimal implementation to extract FLAN datasets for further processing
  ☆11Updated last year
• ChiyuSONG / dynamics-of-instruction-tuning
  ☆16Updated 10 months ago
• nkandpa2 / long_tail_knowledge
  Repo for the paper "Large Language Models Struggle to Learn Long-Tail Knowledge"
  ☆74Updated last year
• allenai / data-efficient-finetuning
  Code for paper 'Data-Efficient FineTuning'
  ☆29Updated last year
• kernelmachine / demix
  DEMix Layers for Modular Language Modeling
  ☆53Updated 3 years ago
• KwanWaiChung / M4LE
  Code for M4LE: A Multi-Ability Multi-Range Multi-Task Multi-Domain Long-Context Evaluation Benchmark for Large Language Models
  ☆22Updated 5 months ago
• krystalan / chatgpt_as_nlg_evaluator
  Technical Report: Is ChatGPT a Good NLG Evaluator? A Preliminary Study
  ☆43Updated last year
• jzbjyb / lm-calibration
  ☆33Updated 3 years ago
• yxuansu / Contrastive_Search_versus_Contrastive_Decoding
  An Empirical Study On Contrastive Search And Contrastive Decoding For Open-ended Text Generation
  ☆26Updated 7 months ago
• swj0419 / kNN_prompt
  TBC
  ☆26Updated 2 years ago
• oriram / spider
  ☆55Updated 2 years ago
• salesforce / factualNLG
  Code for the arXiv paper: "LLMs as Factual Reasoners: Insights from Existing Benchmarks and Beyond"
  ☆59Updated 9 months ago
• jzbjyb / ReAtt
  Retrieval as Attention
  ☆83Updated 2 years ago
• wzhouad / context-faithful-llm
  Code and data for paper "Context-faithful Prompting for Large Language Models".
  ☆39Updated last year
• frankxu2004 / knnlm-why
  Repo for ICML23 "Why do Nearest Neighbor Language Models Work?"
  ☆56Updated 2 years ago
• OpenLMLab / LongWanjuan
  Towards Systematic Measurement for Long Text Quality
  ☆31Updated 4 months ago
• RUCAIBox / ELMER
  This repository is the official implementation of our EMNLP 2022 paper ELMER: A Non-Autoregressive Pre-trained Language Model for Efficie…
  ☆27Updated 2 years ago
• gmftbyGMFTBY / MomentumDecoding
  Momentum Decoding: Open-ended Text Generation as Graph Exploration
  ☆19Updated last year
• cooelf / CompassMTL
  Task Compass: Scaling Multi-task Pre-training with Task Prefix (EMNLP 2022: Findings) (stay tuned & more will be updated)
  ☆22Updated 2 years ago
• PrasannS / rlhf-length-biases
  ☆27Updated 10 months ago
• microsoft / SEED-Encoder
  ☆43Updated 3 years ago
• princeton-nlp / LM-Kernel-FT
  A Kernel-Based View of Language Model Fine-Tuning https://arxiv.org/abs/2210.05643
  ☆74Updated last year
• INK-USC / FiD-ICL
  "FiD-ICL: A Fusion-in-Decoder Approach for Efficient In-Context Learning" (ACL 2023)
  ☆13Updated last year
• allenai / Lila
  A unified benchmark for math reasoning
  ☆87Updated last year
• sail-sg / symbolic-instruction-tuning
  The official repository for the paper "From Zero to Hero: Examining the Power of Symbolic Tasks in Instruction Tuning".
  ☆63Updated last year
• microsoft / Efficient-Large-LM-Trainer
  ☆36Updated 5 months ago