PrimeIntellect-ai / OpenDilocoLinks
OpenDiLoCo: An Open-Source Framework for Globally Distributed Low-Communication Training
☆554Updated 11 months ago
Alternatives and similar repositories for OpenDiloco
Users that are interested in OpenDiloco are comparing it to the libraries listed below
Sorting:
- prime is a framework for efficient, globally distributed training of AI models over the internet.☆849Updated last month
- Distributed Training Over-The-Internet☆973Updated 2 months ago
- Async RL Training at Scale☆950Updated this week
- Memory layers use a trainable key-value lookup mechanism to add extra parameters to a model without increasing FLOPs. Conceptually, spars…☆362Updated last year
- Checkpoint-engine is a simple middleware to update model weights in LLM inference engines☆871Updated this week
- OLMoE: Open Mixture-of-Experts Language Models☆930Updated 3 months ago
- VPTQ, A Flexible and Extreme low-bit quantization algorithm☆670Updated 8 months ago
- Efficient LLM Inference over Long Sequences☆394Updated 6 months ago
- ☆941Updated last month
- LLM KV cache compression made easy☆729Updated last week
- Pretraining and inference code for a large-scale depth-recurrent language model☆856Updated 2 months ago
- scalable and robust tree-based speculative decoding algorithm☆366Updated 10 months ago
- Fault tolerance for PyTorch (HSDP, LocalSGD, DiLoCo, Streaming DiLoCo)☆459Updated 2 weeks ago
- Atropos is a Language Model Reinforcement Learning Environments framework for collecting and evaluating LLM trajectories through diverse …☆780Updated this week
- [ICML 2024] CLLMs: Consistency Large Language Models☆409Updated last year
- Reference implementation of Megalodon 7B model☆527Updated 7 months ago
- Repo for "LoLCATs: On Low-Rank Linearizing of Large Language Models"☆249Updated 10 months ago
- PyTorch building blocks for the OLMo ecosystem☆612Updated this week
- ☆583Updated last year
- Muon is Scalable for LLM Training☆1,387Updated 4 months ago
- ☆219Updated 11 months ago
- Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients.☆202Updated last year
- Long context evaluation for large language models☆224Updated 9 months ago
- ☆610Updated last week
- ☆461Updated last month
- Memory optimization and training recipes to extrapolate language models' context length to 1 million tokens, with minimal hardware.☆751Updated last year
- [NeurIPS'24 Spotlight, ICLR'25, ICML'25] To speed up Long-context LLMs' inference, approximate and dynamic sparse calculate the attention…☆1,169Updated 2 months ago
- 🚀 Efficiently (pre)training foundation models with native PyTorch features, including FSDP for training and SDPA implementation of Flash…☆275Updated last month
- Advanced quantization toolkit for LLMs and VLMs. Support for WOQ, MXFP4, NVFP4, GGUF, Adaptive Schemes and seamless integration with Tra…☆775Updated this week
- Single File, Single GPU, From Scratch, Efficient, Full Parameter Tuning library for "RL for LLMs"☆569Updated 2 months ago