timinar / BabyLlamaLinks
Training code for Baby-Llama, our submission to the strict-small track of the BabyLM challenge.
☆80Updated last year
Alternatives and similar repositories for BabyLlama
Users that are interested in BabyLlama are comparing it to the libraries listed below
Sorting:
- [ICLR 2024] CLEX: Continuous Length Extrapolation for Large Language Models☆78Updated last year
- [ACL 2024] Long-Context Language Modeling with Parallel Encodings☆153Updated 11 months ago
- Official PyTorch implementation of DistiLLM: Towards Streamlined Distillation for Large Language Models (ICML 2024)☆217Updated 2 months ago
- [ICML'24] The official implementation of “Rethinking Optimization and Architecture for Tiny Language Models”☆121Updated 4 months ago
- Unofficial implementation for the paper "Mixture-of-Depths: Dynamically allocating compute in transformer-based language models"☆161Updated 11 months ago
- Implementation of Speculative Sampling as described in "Accelerating Large Language Model Decoding with Speculative Sampling" by Deepmind☆96Updated last year
- Code for paper "Patch-Level Training for Large Language Models"☆86Updated 6 months ago
- Parameter-Efficient Sparsity Crafting From Dense to Mixture-of-Experts for Instruction Tuning on General Tasks☆143Updated 8 months ago
- ☆125Updated last year
- Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models☆132Updated 11 months ago
- Positional Skip-wise Training for Efficient Context Window Extension of LLMs to Extremely Length (ICLR 2024)☆203Updated last year
- ☆79Updated 4 months ago
- Layer-Condensed KV cache w/ 10 times larger batch size, fewer params and less computation. Dramatic speed up with better task performance…☆149Updated 2 months ago
- ☆129Updated 3 months ago
- ☆259Updated last year
- Code and Data for "Long-context LLMs Struggle with Long In-context Learning" [TMLR2025]☆106Updated 3 months ago
- Reformatted Alignment☆114Updated 8 months ago
- Block Transformer: Global-to-Local Language Modeling for Fast Inference (NeurIPS 2024)☆156Updated last month
- ☆37Updated 9 months ago
- Easy control for Key-Value Constrained Generative LLM Inference(https://arxiv.org/abs/2402.06262)☆63Updated last year
- [ICLR 2025] 🧬 RegMix: Data Mixture as Regression for Language Model Pre-training (Spotlight)☆141Updated 3 months ago
- ☆222Updated 11 months ago
- [ICML 2024] Selecting High-Quality Data for Training Language Models☆175Updated 11 months ago
- A prototype repo for hybrid training of pipeline parallel and distributed data parallel with comments on core code snippets. Feel free to…☆55Updated last year
- Official implementation of "DoRA: Weight-Decomposed Low-Rank Adaptation"☆123Updated last year
- [EMNLP 2024] LongAlign: A Recipe for Long Context Alignment of LLMs☆250Updated 5 months ago
- Unofficial implementation of AlpaGasus☆91Updated last year
- A Comprehensive Survey on Long Context Language Modeling☆147Updated this week
- [ACL 2024] LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancement☆183Updated last year
- Unofficial implementations of block/layer-wise pruning methods for LLMs.☆69Updated last year