allenai / duplodocusLinks
Tooling for exact and MinHash deduplication of large-scale text datasets
☆68Updated last week
Alternatives and similar repositories for duplodocus
Users that are interested in duplodocus are comparing it to the libraries listed below
Sorting:
- Train, tune, and infer Bamba model☆137Updated 8 months ago
- ☆41Updated last year
- ☆82Updated 2 months ago
- Verifiers for LLM Reinforcement Learning☆80Updated 9 months ago
- The source code of our work "Prepacking: A Simple Method for Fast Prefilling and Increased Throughput in Large Language Models" [AISTATS …☆60Updated last year
- DPO, but faster 🚀☆47Updated last year
- Supercharge huggingface transformers with model parallelism.☆78Updated 6 months ago
- A collection of reproducible inference engine benchmarks☆38Updated 9 months ago
- EvaByte: Efficient Byte-level Language Models at Scale☆115Updated 9 months ago
- ☆48Updated last year
- MatFormer repo☆70Updated last year
- Lightweight toolkit package to train and fine-tune 1.58bit Language models☆112Updated 8 months ago
- ☆56Updated last year
- Aioli: A unified optimization framework for language model data mixing☆32Updated last year
- ☆96Updated 2 weeks ago
- [TMLR 2026] When Attention Collapses: How Degenerate Layers in LLMs Enable Smaller, Stronger Models☆122Updated last year
- This is a new metric that can be used to evaluate faithfulness of text generated by LLMs. The work behind this repository can be found he…☆31Updated 2 years ago
- vLLM adapter for a TGIS-compatible gRPC server.☆51Updated this week
- Data mapping framework for rust stuff☆44Updated this week
- ☆52Updated last year
- ☆50Updated 3 months ago
- Simple & Scalable Pretraining for Neural Architecture Research☆308Updated 2 months ago
- ☆102Updated last month
- Layer-Condensed KV cache w/ 10 times larger batch size, fewer params and less computation. Dramatic speed up with better task performance…☆157Updated 10 months ago
- Experiments on speculative sampling with Llama models☆128Updated 2 years ago
- Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignment☆61Updated last year
- A collection of lightweight interpretability scripts to understand how LLMs think☆89Updated 2 weeks ago
- Efficient non-uniform quantization with GPTQ for GGUF☆58Updated 4 months ago
- Simple and efficient DeepSeek V3 SFT using pipeline parallel and expert parallel, with both FP8 and BF16 trainings☆114Updated 6 months ago
- Simple high-throughput inference library☆155Updated 8 months ago