ScalingIntelligence / hydragen
Hydragen: High-Throughput LLM Inference with Shared Prefixes
☆24Updated 6 months ago
Related projects ⓘ
Alternatives and complementary repositories for hydragen
- Code for Palu: Compressing KV-Cache with Low-Rank Projection☆57Updated this week
- ☆46Updated 5 months ago
- LLM Serving Performance Evaluation Harness☆56Updated 2 months ago
- PyTorch library for cost-effective, fast and easy serving of MoE models.☆103Updated 3 months ago
- Odysseus: Playground of LLM Sequence Parallelism☆57Updated 5 months ago
- PyTorch bindings for CUTLASS grouped GEMM.☆53Updated 3 weeks ago
- ☆38Updated 4 months ago
- Breaking Throughput-Latency Trade-off for Long Sequences with Speculative Decoding☆79Updated this week
- GPTQ inference TVM kernel☆36Updated 6 months ago
- Materials for learning SGLang☆105Updated this week
- NAACL '24 (Best Demo Paper RunnerUp) / MlSys @ NeurIPS '23 - RedCoast: A Lightweight Tool to Automate Distributed Training and Inference☆61Updated last month
- Boosting 4-bit inference kernels with 2:4 Sparsity☆51Updated 2 months ago
- Quantized Attention on GPU☆30Updated 2 weeks ago
- GEAR: An Efficient KV Cache Compression Recipefor Near-Lossless Generative Inference of LLM☆147Updated 4 months ago
- A minimal implementation of vllm.☆30Updated 3 months ago
- ☆55Updated 5 months ago
- TidalDecode: A Fast and Accurate LLM Decoding with Position Persistent Sparse Attention☆21Updated last month
- ☆51Updated last month
- [ICML 2024] Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference☆202Updated 2 weeks ago
- Implementation of Kangaroo: Lossless Self-Speculative Decoding via Double Early Exiting☆44Updated 4 months ago
- [OSDI'24] Serving LLM-based Applications Efficiently with Semantic Variable☆114Updated 2 months ago
- PyTorch implementation of paper "Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline".☆74Updated last year
- ☆45Updated 2 weeks ago
- Efficient, Flexible and Portable Structured Generation☆53Updated this week
- 16-fold memory access reduction with nearly no loss☆59Updated last week
- PipeRAG: Fast Retrieval-Augmented Generation via Algorithm-System Co-design (KDD 2025)☆11Updated 5 months ago
- ☆42Updated 6 months ago
- MagicPIG: LSH Sampling for Efficient LLM Generation☆59Updated 3 weeks ago
- Transformers components but in Triton☆27Updated this week
- [ACL 2024] RelayAttention for Efficient Large Language Model Serving with Long System Prompts☆34Updated 8 months ago