thepowerfuldeez / OLMo
My fork os allen AI's OLMo for educational purposes.
☆27Updated 5 months ago
Related projects: ⓘ
- Official repository for the paper "SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention"☆87Updated 8 months ago
- Parameter-Efficient Sparsity Crafting From Dense to Mixture-of-Experts for Instruction Tuning on General Tasks☆123Updated 6 months ago
- ☆42Updated 3 weeks ago
- Set of scripts to finetune LLMs☆36Updated 5 months ago
- A single repo with all scripts and utils to train / fine-tune the Mamba model with or without FIM☆46Updated 5 months ago
- A pipeline for LLM knowledge distillation☆68Updated last month
- Official Repository of The Mamba in the Llama: Distilling and Accelerating Hybrid Models☆130Updated this week
- This is the official repository for Inheritune.☆89Updated 4 months ago
- ☆50Updated last month
- A repository for research on medium sized language models.☆71Updated 3 months ago
- Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignment☆39Updated 2 weeks ago
- A public implementation of the ReLoRA pretraining method, built on Lightning-AI's Pytorch Lightning suite.☆33Updated 6 months ago
- The official repo for "LLoCo: Learning Long Contexts Offline"☆104Updated 3 months ago
- Collection of autoregressive model implementation☆62Updated 2 weeks ago
- ☆77Updated 3 weeks ago
- Block Transformer: Global-to-Local Language Modeling for Fast Inference (Official Code)☆118Updated 2 weeks ago
- Implementation of the paper: "Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention" from Google in pyTO…☆48Updated last week
- From GaLore to WeLore: How Low-Rank Weights Non-uniformly Emerge from Low-Rank Gradients. Ajay Jaiswal, Lu Yin, Zhenyu Zhang, Shiwei Liu,…☆43Updated 2 months ago
- Co-LLM: Learning to Decode Collaboratively with Multiple Language Models☆89Updated 4 months ago
- FuseAI Project☆75Updated 3 weeks ago
- Data preparation code for CrystalCoder 7B LLM☆42Updated 4 months ago
- Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients.☆158Updated 2 months ago
- Positional Skip-wise Training for Efficient Context Window Extension of LLMs to Extremely Length (ICLR 2024)☆195Updated 3 months ago
- A toolkit for fine-tuning, inferencing, and evaluating GreenBitAI's LLMs.☆68Updated 2 months ago
- Cascade Speculative Drafting☆23Updated 5 months ago
- Small and Efficient Mathematical Reasoning LLMs☆69Updated 7 months ago
- ☆75Updated 3 weeks ago
- ☆85Updated 7 months ago
- EvolKit is an innovative framework designed to automatically enhance the complexity of instructions used for fine-tuning Large Language M…☆118Updated last week
- Repo hosting codes and materials related to speeding LLMs' inference using token merging.☆28Updated 4 months ago