infinigence / SpecEELinks
Repo for SpecEE: Accelerating Large Language Model Inference with Speculative Early Exiting (ISCA25)
☆48Updated 4 months ago
Alternatives and similar repositories for SpecEE
Users that are interested in SpecEE are comparing it to the libraries listed below
Sorting:
- High performance Transformer implementation in C++.☆129Updated 7 months ago
- Summary of some awesome work for optimizing LLM inference☆103Updated 3 months ago
- A lightweight design for computation-communication overlap.☆160Updated last week
- LLM Inference analyzer for different hardware platforms☆87Updated last month
- A prefill & decode disaggregated LLM serving framework with shared GPU memory and fine-grained compute isolation.☆105Updated 3 months ago
- ☆116Updated 9 months ago
- ☆23Updated last year
- InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management (OSDI'24)☆152Updated last year
- llm theoretical performance analysis tools and support params, flops, memory and latency analysis.☆104Updated last month
- Curated collection of papers in MoE model inference☆250Updated last month
- A GPU-optimized system for efficient long-context LLMs decoding with low-bit KV cache.☆58Updated this week
- Since the emergence of chatGPT in 2022, the acceleration of Large Language Model has become increasingly important. Here is a list of pap…☆266Updated 5 months ago
- Examples of CUDA implementations by Cutlass CuTe☆225Updated 2 months ago
- Analyze the inference of Large Language Models (LLMs). Analyze aspects like computation, storage, transmission, and hardware roofline mod…☆543Updated 11 months ago
- ☆117Updated last month
- [DAC'25] Official implement of "HybriMoE: Hybrid CPU-GPU Scheduling and Cache Management for Efficient MoE Inference"☆69Updated 2 months ago
- ☆150Updated last year
- [ICML 2025] Official PyTorch implementation of "FlatQuant: Flatness Matters for LLM Quantization"☆156Updated last month
- Summary of the Specs of Commonly Used GPUs for Training and Inference of LLM☆63Updated 3 weeks ago
- This repository is established to store personal notes and annotated papers during daily research.☆148Updated 3 weeks ago
- [ICLR2025]: OSTQuant: Refining Large Language Model Quantization with Orthogonal and Scaling Transformations for Better Distribution Fitt…☆72Updated 4 months ago
- ☆23Updated 5 months ago
- DeepSeek-V3/R1 inference performance simulator☆165Updated 5 months ago
- ☆146Updated 5 months ago
- LLM serving cluster simulator☆108Updated last year
- ☆131Updated 8 months ago
- ☆69Updated last year
- ☆150Updated 7 months ago
- Code Repository of Evaluating Quantized Large Language Models☆130Updated 11 months ago
- ☆101Updated 3 months ago