PrimeIntellect-ai / diloco_simpleLinks
torch implementation of diloco
☆22Updated last year
Alternatives and similar repositories for diloco_simple
Users that are interested in diloco_simple are comparing it to the libraries listed below
Sorting:
- Simple and efficient DeepSeek V3 SFT using pipeline parallel and expert parallel, with both FP8 and BF16 trainings☆114Updated 6 months ago
- Cold Compress is a hackable, lightweight, and open-source toolkit for creating and benchmarking cache compression methods built on top of…☆148Updated last year
- A 7B parameter model for mathematical reasoning☆42Updated 11 months ago
- Repo for "LoLCATs: On Low-Rank Linearizing of Large Language Models"☆252Updated last year
- A MAD laboratory to improve AI architecture designs 🧪☆137Updated last year
- Official code for "SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficient"☆149Updated 2 years ago
- ☆47Updated 2 years ago
- Simple and efficient pytorch-native transformer training and inference (batched)☆79Updated last year
- nanoGPT-like codebase for LLM training☆113Updated 3 months ago
- Tree Attention: Topology-aware Decoding for Long-Context Attention on GPU clusters☆131Updated last year
- A place to store reusable transformer components of my own creation or found on the interwebs☆72Updated 3 weeks ago
- ☆219Updated last year
- MoE training for Me and You and maybe other people☆335Updated last month
- PCCL (Prime Collective Communications Library) implements fault tolerant collective communications over IP☆141Updated 4 months ago
- ☆54Updated last year
- Code for exploring Based models from "Simple linear attention language models balance the recall-throughput tradeoff"☆247Updated 8 months ago
- Understand and test language model architectures on synthetic tasks.☆252Updated 3 weeks ago
- Muon fsdp 2☆52Updated 5 months ago
- ☆115Updated last year
- train with kittens!☆63Updated last year
- Layer-Condensed KV cache w/ 10 times larger batch size, fewer params and less computation. Dramatic speed up with better task performance…☆157Updated 10 months ago
- ☆83Updated 2 years ago
- Triton-based implementation of Sparse Mixture of Experts.☆263Updated 4 months ago
- ☆22Updated last year
- some common Huggingface transformers in maximal update parametrization (µP)☆87Updated 3 years ago
- DeMo: Decoupled Momentum Optimization☆198Updated last year
- Simple high-throughput inference library☆155Updated 8 months ago
- Ship correct and fast LLM kernels to PyTorch☆140Updated 3 weeks ago
- ☆53Updated last year
- ☆51Updated last year