PrimeIntellect-ai / OpenDilocoLinks
OpenDiLoCo: An Open-Source Framework for Globally Distributed Low-Communication Training
☆550Updated 10 months ago
Alternatives and similar repositories for OpenDiloco
Users that are interested in OpenDiloco are comparing it to the libraries listed below
Sorting:
- prime is a framework for efficient, globally distributed training of AI models over the internet.☆850Updated 2 weeks ago
- Distributed Training Over-The-Internet☆966Updated last month
- Async RL Training at Scale☆867Updated this week
- Memory layers use a trainable key-value lookup mechanism to add extra parameters to a model without increasing FLOPs. Conceptually, spars…☆360Updated 11 months ago
- VPTQ, A Flexible and Extreme low-bit quantization algorithm☆668Updated 7 months ago
- ☆581Updated last year
- Efficient LLM Inference over Long Sequences☆392Updated 5 months ago
- scalable and robust tree-based speculative decoding algorithm☆363Updated 10 months ago
- Checkpoint-engine is a simple middleware to update model weights in LLM inference engines☆851Updated last week
- ☆219Updated 10 months ago
- ☆917Updated last month
- Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients.☆202Updated last year
- Long context evaluation for large language models☆224Updated 9 months ago
- LLM KV cache compression made easy☆701Updated this week
- Fault tolerance for PyTorch (HSDP, LocalSGD, DiLoCo, Streaming DiLoCo)☆454Updated 3 weeks ago
- Official implementation of Half-Quadratic Quantization (HQQ)☆894Updated last month
- [ICML 2024] CLLMs: Consistency Large Language Models☆407Updated last year
- Atropos is a Language Model Reinforcement Learning Environments framework for collecting and evaluating LLM trajectories through diverse …☆760Updated this week
- Advanced quantization toolkit for LLMs and VLMs. Native support for WOQ, MXFP4, NVFP4, GGUF, Adaptive Bits and seamless integration with …☆735Updated this week
- noise_step: Training in 1.58b With No Gradient Memory☆221Updated 11 months ago
- Repo for "LoLCATs: On Low-Rank Linearizing of Large Language Models"☆249Updated 10 months ago
- OLMoE: Open Mixture-of-Experts Language Models☆919Updated 2 months ago
- PyTorch-native post-training at scale☆549Updated last week
- Pretraining and inference code for a large-scale depth-recurrent language model☆850Updated last month
- Code to train and evaluate Neural Attention Memory Models to obtain universally-applicable memory systems for transformers.☆329Updated last year
- 🚀 Efficiently (pre)training foundation models with native PyTorch features, including FSDP for training and SDPA implementation of Flash…☆271Updated last week
- Muon is Scalable for LLM Training☆1,372Updated 4 months ago
- ☆570Updated last year
- DFloat11: Lossless LLM Compression for Efficient GPU Inference☆569Updated last week
- A high-throughput and memory-efficient inference and serving engine for LLMs☆267Updated last year