Alternatives and similar repositories for data-preparation:

Users that are interested in data-preparation are comparing it to the libraries listed below

• sangmichaelxie / doremi
  Pytorch implementation of DoReMi, a method for optimizing the data mixture weights in language modeling datasets
  ☆313Updated last year
• p-lambda / dsir
  DSIR large-scale data selection framework for language model training
  ☆242Updated 9 months ago
• HuangLK / transpeeder
  train llama on a single A100 80G node using 🤗 transformers and 🚀 Deepspeed Pipeline Parallelism
  ☆212Updated last year
• l294265421 / alpaca-rlhf
  Finetuning LLaMA with RLHF (Reinforcement Learning with Human Feedback) based on DeepSpeed Chat
  ☆109Updated last year
• staoxiao / RetroMAE
  Codebase for RetroMAE and beyond.
  ☆246Updated 7 months ago
• shjwudp / c4-dataset-script
  Inspired by google c4, here is a series of colossal clean data cleaning scripts focused on CommonCrawl data processing. Including Chinese…
  ☆119Updated last year
• raunak-agarwal / instruction-datasets
  All available datasets for Instruction Tuning of Large Language Models
  ☆240Updated last year
• OFA-Sys / InsTag
  InsTag: A Tool for Data Analysis in LLM Supervised Fine-tuning
  ☆236Updated last year
• ChenghaoMou / text-dedup
  All-in-one text de-duplication
  ☆648Updated 7 months ago
• ZigeW / data_management_LLM
  Collection of training data management explorations for large language models
  ☆302Updated 5 months ago
• ACL2023-Retrieval-LM / ACL2023-Retrieval-LM.github.io
  https://acl2023-retrieval-lm.github.io/
  ☆154Updated last year
• hkust-nlp / deita
  Deita: Data-Efficient Instruction Tuning for Alignment [ICLR2024]
  ☆524Updated last month
• OpenLMLab / LEval
  [ACL'24 Outstanding] Data and code for L-Eval, a comprehensive long context language models evaluation benchmark
  ☆368Updated 6 months ago
• AI21Labs / Parallel-Context-Windows
  ☆104Updated last year
• neelsjain / NEFTune
  Official repository of NEFTune: Noisy Embeddings Improves Instruction Finetuning
  ☆388Updated 8 months ago
• CASIA-LM / ChineseWebText
  ☆161Updated last year
• ProjectD-AI / LLaMA-Megatron-DeepSpeed
  Ongoing research training transformer language models at scale, including: BERT & GPT-2
  ☆69Updated last year
• allenai / wimbd
  What's In My Big Data (WIMBD) - a toolkit for analyzing large text datasets
  ☆200Updated 2 months ago
• Spico197 / Humback
  🐋 An unofficial implementation of Self-Alignment with Instruction Backtranslation.
  ☆136Updated 6 months ago
• Strivin0311 / long-llms-learning
  A repository sharing the literatures about long-context large language models, including the methodologies and the evaluation benchmarks
  ☆255Updated 5 months ago
• princeton-nlp / LESS
  [ICML 2024] LESS: Selecting Influential Data for Targeted Instruction Tuning
  ☆400Updated 2 months ago
• OpenBMB / UltraEval
  [ACL 2024 Demo] Official GitHub repo for UltraEval: An open source framework for evaluating foundation models.
  ☆230Updated 2 months ago
• princeton-nlp / CEPE
  [ACL 2024] Long-Context Language Modeling with Parallel Encodings
  ☆154Updated 7 months ago
• icip-cas / ChatAlpaca
  A Multi-Turn Dialogue Corpus based on Alpaca Instructions
  ☆164Updated last year
• huggingface / transformers-bloom-inference
  Fast Inference Solutions for BLOOM
  ☆563Updated 3 months ago
• FlagOpen / FlagInstruct
  ☆173Updated last year
• QwenLM / AutoIF
  ☆247Updated 5 months ago
• FreedomIntelligence / InstructionZoo
  ☆276Updated 8 months ago
• AI21Labs / in-context-ralm
  ☆268Updated last year
• OpenBMB / InfiniteBench
  Codes for the paper "∞Bench: Extending Long Context Evaluation Beyond 100K Tokens": https://arxiv.org/abs/2402.13718
  ☆303Updated 3 months ago