timinar / BabyLlama
Training code for Baby-Llama, our submission to the strict-small track of the BabyLM challenge.
☆67Updated last year
Related projects ⓘ
Alternatives and complementary repositories for BabyLlama
- Official PyTorch implementation of DistiLLM: Towards Streamlined Distillation for Large Language Models (ICML 2024)☆134Updated last month
- ☆121Updated 9 months ago
- Unofficial implementation for the paper "Mixture-of-Depths: Dynamically allocating compute in transformer-based language models"☆134Updated 4 months ago
- Implementation of Speculative Sampling as described in "Accelerating Large Language Model Decoding with Speculative Sampling" by Deepmind☆79Updated 8 months ago
- Parameter-Efficient Sparsity Crafting From Dense to Mixture-of-Experts for Instruction Tuning on General Tasks☆129Updated last month
- Layer-Condensed KV cache w/ 10 times larger batch size, fewer params and less computation. Dramatic speed up with better task performance…☆137Updated this week
- ☆197Updated 5 months ago
- EE-LLM is a framework for large-scale training and inference of early-exit (EE) large language models (LLMs).☆47Updated 4 months ago
- [ICLR 2024] CLEX: Continuous Length Extrapolation for Large Language Models☆72Updated 8 months ago
- Positional Skip-wise Training for Efficient Context Window Extension of LLMs to Extremely Length (ICLR 2024)☆200Updated 5 months ago
- ☆62Updated last month
- ☆95Updated last month
- Repo for Rho-1: Token-level Data Selection & Selective Pretraining of LLMs.☆302Updated 6 months ago
- [ICLR 2024 Spotlight] Code for the paper "Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy"☆64Updated 5 months ago
- Code and Data for "Long-context LLMs Struggle with Long In-context Learning"☆91Updated 4 months ago
- Official implementation of "DoRA: Weight-Decomposed Low-Rank Adaptation"☆123Updated 6 months ago
- Explorations into some recent techniques surrounding speculative decoding☆210Updated last year
- [ICML'24] The official implementation of “Rethinking Optimization and Architecture for Tiny Language Models”☆116Updated 4 months ago
- Astraios: Parameter-Efficient Instruction Tuning Code Language Models☆57Updated 7 months ago
- ☆245Updated last year
- Unofficial implementation of AlpaGasus☆84Updated last year
- [ICML'24] Data and code for our paper "Training-Free Long-Context Scaling of Large Language Models"☆352Updated 3 weeks ago
- [ACL 2024] Long-Context Language Modeling with Parallel Encodings☆143Updated 4 months ago
- ☆183Updated 6 months ago
- Ouroboros: Speculative Decoding with Large Model Enhanced Drafting (EMNLP 2024 main)☆76Updated last month
- Low-bit optimizers for PyTorch☆118Updated last year
- Expert Specialized Fine-Tuning☆144Updated last month
- Unofficial implementations of block/layer-wise pruning methods for LLMs.☆49Updated 6 months ago
- REST: Retrieval-Based Speculative Decoding, NAACL 2024☆175Updated last month
- ☆62Updated last month