Infini-AI-Lab / UMbreLLaLinks
LLM Inference on consumer devices
☆128Updated 9 months ago
Alternatives and similar repositories for UMbreLLa
Users that are interested in UMbreLLa are comparing it to the libraries listed below
Sorting:
- ☆159Updated 5 months ago
- ☆66Updated 6 months ago
- KV cache compression for high-throughput LLM inference☆148Updated 10 months ago
- ☆113Updated last month
- Code for data-aware compression of DeepSeek models☆66Updated last week
- Samples of good AI generated CUDA kernels☆94Updated 6 months ago
- Sparse Inferencing for transformer based LLMs☆215Updated 4 months ago
- Official PyTorch implementation for Hogwild! Inference: Parallel LLM Generation with a Concurrent Attention Cache☆136Updated 4 months ago
- Training-free Post-training Efficient Sub-quadratic Complexity Attention. Implemented with OpenAI Triton.☆148Updated last month
- scalable and robust tree-based speculative decoding algorithm☆365Updated 10 months ago
- ☆63Updated 7 months ago
- REAP: Router-weighted Expert Activation Pruning for SMoE compression☆145Updated last week
- [ACL 2025 Main] EfficientQAT: Efficient Quantization-Aware Training for Large Language Models☆317Updated 3 weeks ago
- Work in progress.☆75Updated 3 weeks ago
- RWKV-7: Surpassing GPT☆101Updated last year
- ☆219Updated 10 months ago
- [ICLR'25] Fast Inference of MoE Models with CPU-GPU Orchestration☆249Updated last year
- QuIP quantization☆61Updated last year
- Tree Attention: Topology-aware Decoding for Long-Context Attention on GPU clusters☆130Updated last year
- Automated Identification of Redundant Layer Blocks for Pruning in Large Language Models☆257Updated last year
- Efficient LLM Inference over Long Sequences☆393Updated 5 months ago
- ☆81Updated last year
- A safetensors extension to efficiently store sparse quantized tensors on disk☆220Updated this week
- Repo for "LoLCATs: On Low-Rank Linearizing of Large Language Models"☆249Updated 10 months ago
- Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients.☆202Updated last year
- ☆64Updated last year
- An efficent implementation of the method proposed in "The Era of 1-bit LLMs"☆155Updated last year
- 3x Faster Inference; Unofficial implementation of EAGLE Speculative Decoding☆80Updated 5 months ago
- PyTorch implementation of models from the Zamba2 series.☆186Updated 10 months ago
- ☆460Updated 3 weeks ago