Infini-AI-Lab / UMbreLLaLinks
LLM Inference on consumer devices
☆119Updated 3 months ago
Alternatives and similar repositories for UMbreLLa
Users that are interested in UMbreLLa are comparing it to the libraries listed below
Sorting:
- ☆137Updated this week
- Official PyTorch implementation for Hogwild! Inference: Parallel LLM Generation with a Concurrent Attention Cache☆109Updated 2 months ago
- ☆68Updated this week
- Samples of good AI generated CUDA kernels☆83Updated 3 weeks ago
- KV cache compression for high-throughput LLM inference☆131Updated 4 months ago
- ☆38Updated this week
- [ICLR'25] Fast Inference of MoE Models with CPU-GPU Orchestration☆213Updated 7 months ago
- ☆55Updated 7 months ago
- Sparse Inferencing for transformer based LLMs☆183Updated this week
- Work in progress.☆69Updated 2 weeks ago
- Code for data-aware compression of DeepSeek models☆35Updated 2 weeks ago
- An efficent implementation of the method proposed in "The Era of 1-bit LLMs"☆154Updated 8 months ago
- ☆159Updated this week
- A safetensors extension to efficiently store sparse quantized tensors on disk☆129Updated this week
- ☆50Updated last month
- Query-agnostic KV cache eviction: 3–4× reduction in memory and 2× decrease in latency (Qwen3/2.5, Gemma3, LLaMA3)☆85Updated 2 weeks ago
- Training-free Post-training Efficient Sub-quadratic Complexity Attention. Implemented with OpenAI Triton.☆136Updated last week
- Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients.☆198Updated 11 months ago
- Tree Attention: Topology-aware Decoding for Long-Context Attention on GPU clusters☆126Updated 6 months ago
- [ACL 2025 Main] EfficientQAT: Efficient Quantization-Aware Training for Large Language Models☆273Updated last month
- The evaluation framework for training-free sparse attention in LLMs☆69Updated last week
- A collection of tricks and tools to speed up transformer models☆167Updated 3 weeks ago
- RWKV-7: Surpassing GPT☆91Updated 7 months ago
- scalable and robust tree-based speculative decoding algorithm☆348Updated 4 months ago
- InferX is a Inference Function as a Service Platform☆111Updated last week
- 1.58-bit LLaMa model☆81Updated last year
- [ICML 2024] KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache☆303Updated 5 months ago
- QuIP quantization☆54Updated last year
- Fast low-bit matmul kernels in Triton☆322Updated last week
- Official repository for the paper "NeuZip: Memory-Efficient Training and Inference with Dynamic Compression of Neural Networks". This rep…☆58Updated 7 months ago