chen-ace / LLM-Prefill-Decode-BenchmarkLinks
通过实验对比LLM推理中Prefill和Decoding阶段的吞吐量差异，揭示性能瓶颈，解释PD分离优化技术的原理。包含CUDA和Apple MPS (M系列芯片) 的测试脚本。
☆19Updated 8 months ago
Alternatives and similar repositories for LLM-Prefill-Decode-Benchmark
Users that are interested in LLM-Prefill-Decode-Benchmark are comparing it to the libraries listed below
Sorting:
- ☆155Updated 11 months ago
- Compare different hardware platforms via the Roofline Model for LLM inference tasks.☆120Updated last year
- LLM training technologies developed by kwai☆70Updated 2 weeks ago
- ATC23 AE☆46Updated 2 years ago
- [ICLR 2025] PEARL: Parallel Speculative Decoding with Adaptive Draft Length☆147Updated last month
- Bridge Megatron-Core to Hugging Face/Reinforcement Learning☆191Updated last week
- ☆83Updated last year
- Pipeline-Parallel Lecture: Simplest Dualpipe Implementation.☆31Updated 4 months ago
- A high-performance distributed deep learning system targeting large-scale and automated distributed training. If you have any interests, …☆124Updated 2 years ago
- DeepSeek-V3/R1 inference performance simulator☆176Updated 10 months ago
- A tiny yet powerful LLM inference system tailored for researching purpose. vLLM-equivalent performance with only 2k lines of code (2% of …☆313Updated 8 months ago
- Implement some method of LLM KV Cache Sparsity☆41Updated last year
- [NeurIPS 2024] Efficient LLM Scheduling by Learning to Rank☆69Updated last year
- Fast and memory-efficient exact attention☆111Updated last week
- ☆150Updated last year
- A prefill & decode disaggregated LLM serving framework with shared GPU memory and fine-grained compute isolation.☆123Updated last month
- ☆47Updated last year
- ☆82Updated 8 months ago
- llm theoretical performance analysis tools and support params, flops, memory and latency analysis.☆115Updated 6 months ago
- Materials for learning SGLang☆738Updated last month
- Efficient, Flexible, and Highly Fault-Tolerant Model Service Management Based on SGLang☆61Updated last year
- FlagCX is a scalable and adaptive cross-chip communication library.☆172Updated this week
- SGLang kernel library for NPU☆96Updated this week
- Summary of the Specs of Commonly Used GPUs for Training and Inference of LLM☆75Updated 5 months ago
- Latency and Memory Analysis of Transformer Models for Training and Inference☆478Updated 9 months ago
- A flexible and efficient training framework for large-scale alignment tasks☆447Updated 3 months ago
- FlagScale is a large model toolkit based on open-sourced projects.☆474Updated this week
- AI Accelerator Benchmark focuses on evaluating AI Accelerators from a practical production perspective, including the ease of use and ver…☆298Updated 3 weeks ago
- ☆85Updated 9 months ago
- Stateful LLM Serving☆95Updated 10 months ago