PrimeIntellect-ai / OpenDilocoLinks
OpenDiLoCo: An Open-Source Framework for Globally Distributed Low-Communication Training
☆508Updated 5 months ago
Alternatives and similar repositories for OpenDiloco
Users that are interested in OpenDiloco are comparing it to the libraries listed below
Sorting:
- prime is a framework for efficient, globally distributed training of AI models over the internet.☆770Updated last month
- Distributed Training Over-The-Internet☆940Updated last month
- prime-rl is a codebase for decentralized async RL training at scale☆347Updated this week
- Memory layers use a trainable key-value lookup mechanism to add extra parameters to a model without increasing FLOPs. Conceptually, spars…☆339Updated 6 months ago
- VPTQ, A Flexible and Extreme low-bit quantization algorithm☆646Updated 2 months ago
- Muon is Scalable for LLM Training☆1,081Updated 3 months ago
- Efficient LLM Inference over Long Sequences☆378Updated 3 weeks ago
- Minimalistic large language model 3D-parallelism training☆1,942Updated this week
- Scalable toolkit for efficient model reinforcement☆448Updated this week
- KernelBench: Can LLMs Write GPU Kernels? - Benchmark with Torch -> CUDA problems☆425Updated 3 weeks ago
- Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients.☆198Updated 11 months ago
- Transformers-compatible library for applying various compression algorithms to LLMs for optimized deployment with vLLM☆1,518Updated this week
- Pretraining code for a large-scale depth-recurrent language model☆783Updated 2 weeks ago
- DeMo: Decoupled Momentum Optimization☆188Updated 6 months ago
- Official codebase for "SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution"☆551Updated 3 months ago
- Official implementation of Half-Quadratic Quantization (HQQ)☆837Updated last week
- ☆472Updated 2 months ago
- Mirage: Automatically Generating Fast GPU Kernels without Programming in Triton/CUDA☆1,322Updated this week
- OLMoE: Open Mixture-of-Experts Language Models☆788Updated 3 months ago
- PyTorch implementation of Infini-Transformer from "Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention…☆289Updated last year
- A project to improve skills of large language models☆429Updated this week
- LLM KV cache compression made easy☆520Updated last week
- Atropos is a Language Model Reinforcement Learning Environments framework for collecting and evaluating LLM trajectories through diverse …☆514Updated this week
- Single File, Single GPU, From Scratch, Efficient, Full Parameter Tuning library for "RL for LLMs"☆476Updated last month
- 🐳 Efficient Triton implementations for "Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention"☆702Updated 3 months ago
- Code to train and evaluate Neural Attention Memory Models to obtain universally-applicable memory systems for transformers.☆311Updated 8 months ago
- [ICML 2024] CLLMs: Consistency Large Language Models☆395Updated 7 months ago
- Beyond Language Models: Byte Models are Digital World Simulators☆322Updated last year
- A throughput-oriented high-performance serving framework for LLMs☆832Updated 3 weeks ago
- noise_step: Training in 1.58b With No Gradient Memory☆219Updated 6 months ago