Code for collecting, processing, and preparing datasets for the Common Pile
☆261Feb 11, 2026Updated 6 months ago
Alternatives and similar repositories for common-pile
Users that are interested in common-pile are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- ☆17Nov 26, 2024Updated last year
- Selected code and data for The Online Books Page and related applications☆12Aug 1, 2026Updated 2 weeks ago
- A large (>5k) collection of search questions asked about Coronavirus 🦠☆14Mar 21, 2020Updated 6 years ago
- ☆26Aug 19, 2025Updated 11 months ago
- Efficiently computing & storing token n-grams from large corpora☆28Jun 15, 2026Updated 2 months ago
- Open source password manager - Proton Pass • AdSecurely store, share, and autofill your credentials with Proton Pass, the end-to-end encrypted password manager trusted by millions.
- ☆35Dec 2, 2024Updated last year
- ☆39Apr 17, 2024Updated 2 years ago
- [ACL 2025] 🔍 Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignment☆11Apr 6, 2025Updated last year
- TokEval: intrinsic quality metrics for tokenizers across natural language, code, and math☆47Aug 11, 2026Updated last week
- ☆79Dec 7, 2023Updated 2 years ago
- Pile Deduplication Code☆18May 15, 2023Updated 3 years ago
- Code for boomerang distillation enables zero-shot model size interpolation.☆22Jul 10, 2026Updated last month
- Luth is a state-of-the-art series of fine-tuned LLMs for French☆47Oct 12, 2025Updated 10 months ago
- The Institutional Data Initiative's pipeline for analyzing, refining, and publishing the Institutional Books 1.0 collection.☆56Nov 21, 2025Updated 8 months ago
- 1-Click AI Models by DigitalOcean Gradient • AdDeploy popular AI models on DigitalOcean Gradient GPU virtual machines with just a single click. Zero configuration with optimized deployments.
- DPO, but faster 🚀☆51Dec 6, 2024Updated last year
- QAmeleon introduces synthetic multilingual QA data using PaLM, a 540B large language model. This dataset was generated by prompt tuning P…☆34Aug 15, 2023Updated 3 years ago
- data related codebase for polyglot project☆19Mar 30, 2023Updated 3 years ago
- Official code for the paper CipherDAug: Ciphertext based Data Augmentation for Neural Machine Translation published at ACL 2022 main conf…☆12Apr 6, 2023Updated 3 years ago
- Using experimental methods to merge large language models☆11Jul 11, 2026Updated last month
- PathPiece tokenizer☆14Nov 10, 2024Updated last year
- ☆33Jul 8, 2024Updated 2 years ago
- Utilities for PyTorch distributed☆26Feb 27, 2025Updated last year
- ☆16Feb 10, 2026Updated 6 months ago
- Managed hosting for WordPress and PHP on Cloudways • AdManaged hosting for WordPress, Magento, Laravel, or PHP apps, on multiple cloud providers. Deploy in minutes on Cloudways by DigitalOcean.
- ☆17Aug 23, 2025Updated 11 months ago
- Highly concurrent and fast content processing for Mighty Inference Server☆10Feb 6, 2023Updated 3 years ago
- ☆257Oct 27, 2025Updated 9 months ago
- A lightweight, user-friendly data-plane for LLM training.☆40Sep 10, 2025Updated 11 months ago
- ☆310Apr 23, 2025Updated last year
- A low-level virtual machine for mobile code☆12Sep 23, 2017Updated 8 years ago
- Gemini API for OCR☆15Nov 17, 2025Updated 9 months ago
- Data and tools for generating and inspecting OLMo pre-training data.☆1,535Nov 5, 2025Updated 9 months ago
- Tooling for exact and MinHash deduplication of large-scale text datasets☆94Mar 24, 2026Updated 4 months ago
- Managed hosting for WordPress and PHP on Cloudways • AdManaged hosting for WordPress, Magento, Laravel, or PHP apps, on multiple cloud providers. Deploy in minutes on Cloudways by DigitalOcean.
- Download, parse, and filter data from Court Listener, part of the FreeLaw projects. Data-ready for The-Pile.☆16Jun 3, 2023Updated 3 years ago
- Repository of data on web domains.☆19May 24, 2023Updated 3 years ago
- SKT A.X LLM 3.1☆13Jul 24, 2025Updated last year
- This project studies the performance and robustness of language models and task-adaptation methods.☆154May 18, 2024Updated 2 years ago
- A pipeline for LLM knowledge distillation☆117May 7, 2026Updated 3 months ago
- Adversarial Tokenization☆39Nov 21, 2025Updated 8 months ago
- The pipeline for the OSCAR corpus☆178Nov 9, 2025Updated 9 months ago