Infini-AI-Lab / UMbreLLaLinks
LLM Inference on consumer devices
☆129Updated 10 months ago
Alternatives and similar repositories for UMbreLLa
Users that are interested in UMbreLLa are comparing it to the libraries listed below
Sorting:
- ☆163Updated 7 months ago
- Official PyTorch implementation for Hogwild! Inference: Parallel LLM Generation with a Concurrent Attention Cache☆140Updated 5 months ago
- KV cache compression for high-throughput LLM inference☆151Updated last year
- ☆71Updated 7 months ago
- Samples of good AI generated CUDA kernels☆99Updated 8 months ago
- Sparse Inferencing for transformer based LLMs☆218Updated 5 months ago
- scalable and robust tree-based speculative decoding algorithm☆366Updated last year
- Automated Identification of Redundant Layer Blocks for Pruning in Large Language Models☆260Updated last year
- ☆64Updated 8 months ago
- Block Diffusion for Ultra-Fast Speculative Decoding☆459Updated this week
- Code for data-aware compression of DeepSeek models☆69Updated last month
- Training-free Post-training Efficient Sub-quadratic Complexity Attention. Implemented with OpenAI Triton.☆148Updated 3 months ago
- [ICLR'25] Fast Inference of MoE Models with CPU-GPU Orchestration☆258Updated last year
- [NeurIPS'25 Oral] Query-agnostic KV cache eviction: 3–4× reduction in memory and 2× decrease in latency (Qwen3/2.5, Gemma3, LLaMA3)☆196Updated 2 weeks ago
- REAP: Router-weighted Expert Activation Pruning for SMoE compression☆222Updated last month
- Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients.☆201Updated last year
- ☆118Updated 3 weeks ago
- Code for paper: "QuIP: 2-Bit Quantization of Large Language Models With Guarantees" adapted for Llama models☆41Updated 2 years ago
- Lightweight toolkit package to train and fine-tune 1.58bit Language models☆110Updated 8 months ago
- ArcticInference: vLLM plugin for high-throughput, low-latency inference☆384Updated this week
- [ACL 2025 Main] EfficientQAT: Efficient Quantization-Aware Training for Large Language Models☆326Updated 2 months ago
- Accelerating your LLM training to full speed! Made with ❤️ by ServiceNow Research☆282Updated last week
- Easy, Fast, and Scalable Multimodal AI☆106Updated this week
- 1.58-bit LLaMa model☆82Updated last year
- [ICLR2025] Breaking Throughput-Latency Trade-off for Long Sequences with Speculative Decoding☆137Updated last year
- 3x Faster Inference; Unofficial implementation of EAGLE Speculative Decoding☆83Updated 7 months ago
- An early research stage expert-parallel load balancer for MoE models based on linear programming.☆495Updated 2 months ago
- Tree Attention: Topology-aware Decoding for Long-Context Attention on GPU clusters☆131Updated last year
- QuIP quantization☆61Updated last year
- Reverse Engineering Gemma 3n: Google's New Edge-Optimized Language Model☆262Updated 8 months ago