Code for collecting, processing, and preparing datasets for the Common Pile
☆261Feb 11, 2026Updated 8 months ago
Alternatives and similar repositories for common-pile
Users that are interested in common-pile are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- ☆17Nov 26, 2024Updated last year
- [ICML'25] "Rethinking Addressing in Language Models via Contextualized Equivariant Positional Encoding" by Jiajun Zhu, Peihao Wang, Ruisi…☆15Jun 6, 2025Updated last year
- Selected code and data for The Online Books Page and related applications☆12Oct 1, 2026Updated 2 weeks ago
- A large (>5k) collection of search questions asked about Coronavirus 🦠☆14Mar 21, 2020Updated 6 years ago
- Code for the paper "BPE stays on SCRIPT", "Which Pieces Does Unigram Tokenization Really Need?" and MinGram☆22Sep 22, 2026Updated 3 weeks ago
- Managed hosting for WordPress and PHP on Cloudways • AdManaged hosting for WordPress, Magento, Laravel, or PHP apps, on multiple cloud providers. Deploy in minutes on Cloudways by DigitalOcean.
- ☆26Aug 19, 2025Updated last year
- DKPro C4CorpusTools is a collection of tools for processing CommonCrawl corpus, including Creative Commons license detection, boilerplate…☆54Jun 12, 2020Updated 6 years ago
- Efficiently computing & storing token n-grams from large corpora☆28Jun 15, 2026Updated 4 months ago
- ☆39Apr 17, 2024Updated 2 years ago
- TokEval: intrinsic quality metrics for tokenizers across natural language, code, and math☆56Aug 31, 2026Updated last month
- OLMost every training recipe you need to perform data interventions with the OLMo family of models.☆76Jul 21, 2026Updated 2 months ago
- Pile Deduplication Code☆18May 15, 2023Updated 3 years ago
- [ACL 2025] 🔍 Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignment☆11Apr 6, 2025Updated last year
- ☆78Dec 7, 2023Updated 2 years ago
- Deploy to Railway using AI coding agents - Free Credits Offer • AdUse Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
- Luth is a state-of-the-art series of fine-tuned LLMs for French☆47Oct 12, 2025Updated last year
- Code for boomerang distillation enables zero-shot model size interpolation.☆23Jul 10, 2026Updated 3 months ago
- A polite and user-friendly downloader for Common Crawl data☆97Aug 5, 2026Updated 2 months ago
- QAmeleon introduces synthetic multilingual QA data using PaLM, a 540B large language model. This dataset was generated by prompt tuning P…☆34Aug 15, 2023Updated 3 years ago
- Dataset: BuzzFeed News “Trending” Strip, 2018–2023☆18May 24, 2023Updated 3 years ago
- Official code for the paper CipherDAug: Ciphertext based Data Augmentation for Neural Machine Translation published at ACL 2022 main conf…☆12Apr 6, 2023Updated 3 years ago
- Collection of academic works in natural language processing, computational linguistics, and computational cognitive science that study th…☆24Mar 20, 2024Updated 2 years ago
- PathPiece tokenizer☆14Nov 10, 2024Updated last year
- Utilities for PyTorch distributed☆26Feb 27, 2025Updated last year
- Virtual machines for every use case on DigitalOcean • AdGet dependable uptime with 99.99% SLA, simple security tools, and predictable monthly pricing with DigitalOcean's virtual machines, called Droplets.
- Replication code for "With Little Power Comes Great Responsibility"☆39Oct 15, 2020Updated 6 years ago
- ☆17Feb 10, 2026Updated 8 months ago
- ☆17Aug 23, 2025Updated last year
- MapInsight Add on for Freeplane☆14Feb 18, 2023Updated 3 years ago
- ☆16Aug 5, 2025Updated last year
- Code for NeurIPS 2024 Spotlight: "Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations"☆94Oct 30, 2024Updated last year
- Highly concurrent and fast content processing for Mighty Inference Server☆10Feb 6, 2023Updated 3 years ago
- ☆265Oct 27, 2025Updated 11 months ago
- ☆311Apr 23, 2025Updated last year
- Serverless GPU API endpoints on Runpod - Get Bonus Credits • AdSkip the infrastructure headaches. Auto-scaling, pay-as-you-go, no-ops approach lets you focus on innovating your application.
- Gemini API for OCR☆15Nov 17, 2025Updated 11 months ago
- A lightweight, user-friendly data-plane for LLM training.☆43Sep 10, 2025Updated last year
- Data and tools for generating and inspecting OLMo pre-training data.☆1,559Aug 24, 2026Updated last month
- The Foundation Model Transparency Index☆89Dec 9, 2025Updated 10 months ago
- Tooling for exact and MinHash deduplication of large-scale text datasets☆98Mar 24, 2026Updated 6 months ago
- Repository of data on web domains.☆19May 24, 2023Updated 3 years ago
- This project studies the performance and robustness of language models and task-adaptation methods.☆155May 18, 2024Updated 2 years ago