wdlctc / headinfer
☆44Updated last week
Alternatives and similar repositories for headinfer:
Users that are interested in headinfer are comparing it to the libraries listed below
- ☆56Updated last week
- A collection of tricks and tools to speed up transformer models☆155Updated 3 weeks ago
- ☆53Updated last month
- A toolkit for fine-tuning, inferencing, and evaluating GreenBitAI's LLMs.☆82Updated last month
- Cascade Speculative Drafting☆29Updated last year
- Code for data-aware compression of DeepSeek models☆20Updated 2 weeks ago
- Tree Attention: Topology-aware Decoding for Long-Context Attention on GPU clusters☆126Updated 4 months ago
- [ICLR2025] Breaking Throughput-Latency Trade-off for Long Sequences with Speculative Decoding☆115Updated 4 months ago
- ☆126Updated last month
- Layer-Condensed KV cache w/ 10 times larger batch size, fewer params and less computation. Dramatic speed up with better task performance…☆148Updated 2 weeks ago
- RWKV is an RNN with transformer-level LLM performance. It can be directly trained like a GPT (parallelizable). So it's combining the best…☆44Updated last month
- ☆53Updated 10 months ago
- Work in progress.☆56Updated 2 weeks ago
- QuIP quantization☆51Updated last year
- FBI-LLM: Scaling Up Fully Binarized LLMs from Scratch via Autoregressive Distillation☆48Updated 9 months ago
- ☆43Updated 2 months ago
- Simple extension on vLLM to help you speed up reasoning model without training.☆146Updated this week
- Chain of Experts (CoE) enables communication between experts within Mixture-of-Experts (MoE) models☆158Updated last month
- ☆67Updated this week
- RWKV-7: Surpassing GPT☆83Updated 5 months ago
- Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models☆131Updated 10 months ago
- KV cache compression for high-throughput LLM inference☆126Updated 2 months ago
- The source code of our work "Prepacking: A Simple Method for Fast Prefilling and Increased Throughput in Large Language Models" [AISTATS …☆59Updated 6 months ago
- Repo hosting codes and materials related to speeding LLMs' inference using token merging.☆36Updated 11 months ago
- A repository for research on medium sized language models.☆76Updated 11 months ago
- From GaLore to WeLore: How Low-Rank Weights Non-uniformly Emerge from Low-Rank Gradients. Ajay Jaiswal, Lu Yin, Zhenyu Zhang, Shiwei Liu,…☆44Updated this week
- EfficientQAT: Efficient Quantization-Aware Training for Large Language Models☆263Updated 6 months ago
- Boosting 4-bit inference kernels with 2:4 Sparsity☆72Updated 7 months ago
- Reward-guided Speculative Decoding (RSD) for efficiency and effectiveness.☆25Updated last month
- PB-LLM: Partially Binarized Large Language Models☆151Updated last year