Code for collecting, processing, and preparing datasets for the Common Pile
☆258Feb 11, 2026Updated 5 months ago
Alternatives and similar repositories for common-pile
Users that are interested in common-pile are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- ☆17Nov 26, 2024Updated last year
- [ICML'25] "Rethinking Addressing in Language Models via Contextualized Equivariant Positional Encoding" by Jiajun Zhu, Peihao Wang, Ruisi…☆15Jun 6, 2025Updated last year
- Few-shot Learning with Auxiliary Data☆31Dec 8, 2023Updated 2 years ago
- Selected code and data for The Online Books Page and related applications☆12Jul 1, 2026Updated 3 weeks ago
- A large (>5k) collection of search questions asked about Coronavirus 🦠☆14Mar 21, 2020Updated 6 years ago
- 1-Click AI Models by DigitalOcean Gradient • AdDeploy popular AI models on DigitalOcean Gradient GPU virtual machines with just a single click. Zero configuration with optimized deployments.
- Code for the paper "BPE stays on SCRIPT", "Which Pieces Does Unigram Tokenization Really Need?" and MinGram☆18Updated this week
- DKPro C4CorpusTools is a collection of tools for processing CommonCrawl corpus, including Creative Commons license detection, boilerplate…☆53Jun 12, 2020Updated 6 years ago
- ☆35Dec 2, 2024Updated last year
- ☆39Apr 17, 2024Updated 2 years ago
- [ACL 2025] 🔍 Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignment☆11Apr 6, 2025Updated last year
- OLMost every training recipe you need to perform data interventions with the OLMo family of models.☆72Jul 21, 2026Updated last week
- TokEval: intrinsic quality metrics for tokenizers across natural language, code, and math☆46Jul 4, 2026Updated 3 weeks ago
- ☆79Dec 7, 2023Updated 2 years ago
- Pile Deduplication Code☆18May 15, 2023Updated 3 years ago
- Deploy to Railway using AI coding agents - Free Credits Offer • AdUse Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
- Code for boomerang distillation enables zero-shot model size interpolation.☆22Jul 10, 2026Updated 2 weeks ago
- The Institutional Data Initiative's pipeline for analyzing, refining, and publishing the Institutional Books 1.0 collection.☆54Nov 21, 2025Updated 8 months ago
- A polite and user-friendly downloader for Common Crawl data☆87Jul 13, 2026Updated 2 weeks ago
- DPO, but faster 🚀☆52Dec 6, 2024Updated last year
- QAmeleon introduces synthetic multilingual QA data using PaLM, a 540B large language model. This dataset was generated by prompt tuning P…☆34Aug 15, 2023Updated 2 years ago
- data related codebase for polyglot project☆19Mar 30, 2023Updated 3 years ago
- Intersectional bias in hate speech and abusive language datasets☆15Jan 25, 2024Updated 2 years ago
- Official code for the paper CipherDAug: Ciphertext based Data Augmentation for Neural Machine Translation published at ACL 2022 main conf…☆12Apr 6, 2023Updated 3 years ago
- Collection of academic works in natural language processing, computational linguistics, and computational cognitive science that study th…☆22Mar 20, 2024Updated 2 years ago
- Managed hosting for WordPress and PHP on Cloudways • AdManaged hosting for WordPress, Magento, Laravel, or PHP apps, on multiple cloud providers. Deploy in minutes on Cloudways by DigitalOcean.
- PathPiece tokenizer☆14Nov 10, 2024Updated last year
- ☆33Jul 8, 2024Updated 2 years ago
- Utilities for PyTorch distributed☆26Feb 27, 2025Updated last year
- Replication code for "With Little Power Comes Great Responsibility"☆39Oct 15, 2020Updated 5 years ago
- ☆16Feb 10, 2026Updated 5 months ago
- ☆16Aug 23, 2025Updated 11 months ago
- Code for NeurIPS 2024 Spotlight: "Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations"☆93Oct 30, 2024Updated last year
- Whiskey AI lets you create autonomous AI agents without code. Connect APIs, automate Solana actions, launch tokens, run workflows, scrape…☆73Dec 8, 2025Updated 7 months ago
- Highly concurrent and fast content processing for Mighty Inference Server☆10Feb 6, 2023Updated 3 years ago
- Virtual machines for every use case on DigitalOcean • AdGet dependable uptime with 99.99% SLA, simple security tools, and predictable monthly pricing with DigitalOcean's virtual machines, called Droplets.
- ☆256Oct 27, 2025Updated 9 months ago
- A lightweight, user-friendly data-plane for LLM training.☆40Sep 10, 2025Updated 10 months ago
- ☆310Apr 23, 2025Updated last year
- ☆16Updated this week
- Data and tools for generating and inspecting OLMo pre-training data.☆1,528Nov 5, 2025Updated 8 months ago
- API client for Aleph, supports bulk entity and document upload.☆30Mar 5, 2026Updated 4 months ago
- The Foundation Model Transparency Index☆89Dec 9, 2025Updated 7 months ago