AutonomicPerfectionist / PipeInfer
PipeInfer: Accelerating LLM Inference using Asynchronous Pipelined Speculation
☆28Updated 4 months ago
Alternatives and similar repositories for PipeInfer:
Users that are interested in PipeInfer are comparing it to the libraries listed below
- ☆45Updated 9 months ago
- [ICLR 2025] TidalDecode: A Fast and Accurate LLM Decoding with Position Persistent Sparse Attention☆31Updated this week
- ☆55Updated 9 months ago
- ☆68Updated 2 months ago
- Official Repo for "LLM-PQ: Serving LLM on Heterogeneous Clusters with Phase-Aware Partition and Adaptive Quantization"☆29Updated last year
- LLM Serving Performance Evaluation Harness☆73Updated last month
- [ICLR2025] Breaking Throughput-Latency Trade-off for Long Sequences with Speculative Decoding☆111Updated 4 months ago
- ☆49Updated 2 weeks ago
- ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training Multiplication-Less Reparameterization☆105Updated 5 months ago
- ☆29Updated last year
- [NeurIPS 2024] Efficient LLM Scheduling by Learning to Rank☆43Updated 5 months ago
- Compression for Foundation Models☆30Updated last week
- ☆26Updated 2 weeks ago
- FlexFlow Serve: Low-Latency, High-Performance LLM Serving☆34Updated this week
- Stateful LLM Serving☆50Updated 3 weeks ago
- A minimal implementation of vllm.☆37Updated 8 months ago
- [ICML 2024 Oral] Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMs☆100Updated 3 months ago
- 16-fold memory access reduction with nearly no loss☆86Updated last week
- [ICLR 2025] Palu: Compressing KV-Cache with Low-Rank Projection☆94Updated last month
- [ICLR'25] Fast Inference of MoE Models with CPU-GPU Orchestration☆202Updated 4 months ago
- ☆53Updated last year
- [OSDI'24] Serving LLM-based Applications Efficiently with Semantic Variable☆151Updated 6 months ago
- PyTorch library for cost-effective, fast and easy serving of MoE models.☆161Updated last week
- Boosting 4-bit inference kernels with 2:4 Sparsity☆72Updated 7 months ago
- ☆24Updated 4 months ago
- Activation-aware Singular Value Decomposition for Compressing Large Language Models☆60Updated 5 months ago
- GPU operators for sparse tensor operations☆31Updated last year
- Decoding Attention is specially optimized for MHA, MQA, GQA and MLA using CUDA core for the decoding stage of LLM inference.☆35Updated 3 weeks ago
- ☆113Updated last week
- Explore Inter-layer Expert Affinity in MoE Model Inference☆7Updated 10 months ago