PrimeIntellect-ai / OpenDiloco
OpenDiLoCo: An Open-Source Framework for Globally Distributed Low-Communication Training
☆487Updated 3 months ago
Alternatives and similar repositories for OpenDiloco:
Users that are interested in OpenDiloco are comparing it to the libraries listed below
- prime is a framework for efficient, globally distributed training of AI models over the internet.☆708Updated this week
- Distributed Training Over-The-Internet☆902Updated 4 months ago
- Memory layers use a trainable key-value lookup mechanism to add extra parameters to a model without increasing FLOPs. Conceptually, spars…☆320Updated 4 months ago
- ☆529Updated 8 months ago
- VPTQ, A Flexible and Extreme low-bit quantization algorithm☆628Updated 3 weeks ago
- Efficient LLM Inference over Long Sequences☆370Updated this week
- Muon is Scalable for LLM Training☆1,029Updated 3 weeks ago
- Minimalistic 4D-parallelism distributed training framework for education purpose☆991Updated last month
- Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients.☆198Updated 9 months ago
- LLM KV cache compression made easy☆463Updated last week
- Transformers-compatible library for applying various compression algorithms to LLMs for optimized deployment with vLLM☆1,251Updated this week
- Minimalistic large language model 3D-parallelism training☆1,808Updated this week
- Pretraining code for a large-scale depth-recurrent language model☆745Updated last week
- scalable and robust tree-based speculative decoding algorithm☆344Updated 2 months ago
- Memory optimization and training recipes to extrapolate language models' context length to 1 million tokens, with minimal hardware.☆719Updated 6 months ago
- [ICML 2024] CLLMs: Consistency Large Language Models☆390Updated 5 months ago
- An efficent implementation of the method proposed in "The Era of 1-bit LLMs"☆154Updated 6 months ago
- An Open Source Toolkit For LLM Distillation☆579Updated 3 months ago
- OLMoE: Open Mixture-of-Experts Language Models☆723Updated last month
- A high-throughput and memory-efficient inference and serving engine for LLMs☆262Updated 6 months ago
- ☆208Updated 3 months ago
- Code for the paper "Rethinking Benchmark and Contamination for Language Models with Rephrased Samples"☆299Updated last year
- Repo for "LoLCATs: On Low-Rank Linearizing of Large Language Models"☆231Updated 2 months ago
- PyTorch per step fault tolerance (actively under development)☆284Updated this week
- Long context evaluation for large language models☆207Updated last month
- DeMo: Decoupled Momentum Optimization☆186Updated 4 months ago
- Serverless LLM Serving for Everyone.☆460Updated this week
- Fast, Flexible and Portable Structured Generation☆888Updated 2 weeks ago
- Code to train and evaluate Neural Attention Memory Models to obtain universally-applicable memory systems for transformers.☆304Updated 6 months ago
- A throughput-oriented high-performance serving framework for LLMs☆797Updated 7 months ago