PrimeIntellect-ai / OpenDilocoLinks
OpenDiLoCo: An Open-Source Framework for Globally Distributed Low-Communication Training
☆528Updated 7 months ago
Alternatives and similar repositories for OpenDiloco
Users that are interested in OpenDiloco are comparing it to the libraries listed below
Sorting:
- prime is a framework for efficient, globally distributed training of AI models over the internet.☆815Updated 3 months ago
- Distributed Training Over-The-Internet☆955Updated 3 months ago
- Decentralized RL Training at Scale☆472Updated this week
- Memory layers use a trainable key-value lookup mechanism to add extra parameters to a model without increasing FLOPs. Conceptually, spars…☆346Updated 8 months ago
- VPTQ, A Flexible and Extreme low-bit quantization algorithm☆652Updated 4 months ago
- Efficient LLM Inference over Long Sequences☆390Updated 2 months ago
- ☆561Updated last year
- Fault tolerance for PyTorch (HSDP, LocalSGD, DiLoCo, Streaming DiLoCo)☆389Updated 2 weeks ago
- LLM KV cache compression made easy☆596Updated this week
- [ICML 2024] CLLMs: Consistency Large Language Models☆401Updated 9 months ago
- ☆217Updated 7 months ago
- Beyond Language Models: Byte Models are Digital World Simulators☆328Updated last year
- Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients.☆200Updated last year
- scalable and robust tree-based speculative decoding algorithm☆355Updated 7 months ago
- Pretraining and inference code for a large-scale depth-recurrent language model☆822Updated last month
- Code for "LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding", ACL 2024☆331Updated 3 months ago
- DFloat11: Lossless LLM Compression for Efficient GPU Inference☆524Updated last week
- Muon is Scalable for LLM Training☆1,289Updated 3 weeks ago
- GRadient-INformed MoE☆264Updated 11 months ago
- Atropos is a Language Model Reinforcement Learning Environments framework for collecting and evaluating LLM trajectories through diverse …☆651Updated this week
- PCCL (Prime Collective Communications Library) implements fault tolerant collective communications over IP☆111Updated this week
- OLMoE: Open Mixture-of-Experts Language Models☆850Updated 5 months ago
- Code to train and evaluate Neural Attention Memory Models to obtain universally-applicable memory systems for transformers.☆319Updated 10 months ago
- An efficent implementation of the method proposed in "The Era of 1-bit LLMs"☆155Updated 10 months ago
- Implementation of 💍 Ring Attention, from Liu et al. at Berkeley AI, in Pytorch☆536Updated 3 months ago
- [ICLR 2025] Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling☆907Updated 4 months ago
- Modded vLLM to run pipeline parallelism over public networks☆38Updated 3 months ago
- Advanced Quantization Algorithm for LLMs and VLMs, with support for CPU, Intel GPU, CUDA and HPU. Seamlessly integrated with Torchao, Tra…☆607Updated this week
- Long context evaluation for large language models☆220Updated 5 months ago
- Reference implementation of Megalodon 7B model☆524Updated 3 months ago