Infini-AI-Lab / UMbreLLaLinks
LLM Inference on consumer devices
☆115Updated 2 months ago
Alternatives and similar repositories for UMbreLLa
Users that are interested in UMbreLLa are comparing it to the libraries listed below
Sorting:
- ☆130Updated 2 months ago
- Samples of good AI generated CUDA kernels☆65Updated this week
- ☆55Updated 7 months ago
- Code for data-aware compression of DeepSeek models☆31Updated last month
- KV cache compression for high-throughput LLM inference☆129Updated 4 months ago
- Official PyTorch implementation for Hogwild! Inference: Parallel LLM Generation with a Concurrent Attention Cache☆105Updated last month
- ☆47Updated 3 weeks ago
- [ICLR'25] Fast Inference of MoE Models with CPU-GPU Orchestration☆211Updated 6 months ago
- DFloat11: Lossless LLM Compression for Efficient GPU Inference☆405Updated last week
- Training-free Post-training Efficient Sub-quadratic Complexity Attention. Implemented with OpenAI Triton.☆133Updated this week
- Official repository for the paper "NeuZip: Memory-Efficient Training and Inference with Dynamic Compression of Neural Networks". This rep…☆58Updated 7 months ago
- Work in progress.☆67Updated this week
- Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients.☆199Updated 10 months ago
- PyTorch implementation of models from the Zamba2 series.☆182Updated 4 months ago
- A safetensors extension to efficiently store sparse quantized tensors on disk☆117Updated this week
- Simple extension on vLLM to help you speed up reasoning model without training.☆152Updated this week
- ☆128Updated last month
- ☆46Updated last week
- [ICLR2025] Breaking Throughput-Latency Trade-off for Long Sequences with Speculative Decoding☆116Updated 6 months ago
- QuIP quantization☆52Updated last year
- Fast Matrix Multiplications for Lookup Table-Quantized LLMs☆369Updated last month
- ☆99Updated this week
- InferX is a Inference Function as a Service Platform☆105Updated this week
- [ACL 2025 Main] EfficientQAT: Efficient Quantization-Aware Training for Large Language Models☆272Updated 2 weeks ago
- [ICML 2024] KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache☆303Updated 4 months ago
- Fast low-bit matmul kernels in Triton☆311Updated this week
- 1.58-bit LLaMa model☆81Updated last year
- Tree Attention: Topology-aware Decoding for Long-Context Attention on GPU clusters☆126Updated 6 months ago
- scalable and robust tree-based speculative decoding algorithm☆345Updated 4 months ago
- Automated Identification of Redundant Layer Blocks for Pruning in Large Language Models☆236Updated last year