Alternatives and similar repositories for Awesome-LLM-Inference-Engine

Users that are interested in Awesome-LLM-Inference-Engine are comparing it to the libraries listed below

• tyler-griggs / melange-release
  ☆45Updated 10 months ago
• project-etalon / etalon
  LLM Serving Performance Evaluation Harness
  ☆78Updated 2 months ago
• AutonomicPerfectionist / PipeInfer
  PipeInfer: Accelerating LLM Inference using Asynchronous Pipelined Speculation
  ☆28Updated 6 months ago
• tile-ai / tvm
  Open deep learning compiler stack for cpu, gpu and specialized accelerators
  ☆18Updated last week
• Infini-AI-Lab / MagicDec
  [ICLR2025] Breaking Throughput-Latency Trade-off for Long Sequences with Speculative Decoding
  ☆116Updated 5 months ago
• eth-easl / deltazip
  Compression for Foundation Models
  ☆31Updated last month
• DD-DuDa / BitDecoding
  A GPU-optimized system for efficient long-context LLMs decoding with low-bit KV cache.
  ☆34Updated 3 weeks ago
• flashinfer-ai / cutlass-viz
  ☆58Updated 3 weeks ago
• flexflow / flexflow-serve
  FlexFlow Serve: Low-Latency, High-Performance LLM Serving
  ☆39Updated last week
• hao-ai-lab / MuxServe
  ☆60Updated 11 months ago
• ademeure / DeeperGEMM
  DeeperGEMM: crazy optimized version
  ☆69Updated last week
• microsoft / AttentionEngine
  ☆70Updated last week
• IntelLabs / Hardware-Aware-Automated-Machine-Learning
  ☆54Updated 2 weeks ago
• Bruce-Lee-LY / decoding_attention
  Decoding Attention is specially optimized for MHA, MQA, GQA and MLA using CUDA core for the decoding stage of LLM inference.
  ☆36Updated last month
• amazon-science / piperag
  PipeRAG: Fast Retrieval-Augmented Generation via Algorithm-System Co-design (KDD 2025)
  ☆20Updated 11 months ago
• li-plus / flash-preference
  Accelerate LLM preference tuning via prefix sharing with a single line of code
  ☆41Updated 2 weeks ago
• MDK8888 / vllmini
  A minimal implementation of vllm.
  ☆40Updated 9 months ago
• NEO-MLSys25 / NEO
  NEO is a LLM inference engine built to save the GPU memory crisis by CPU offloading
  ☆27Updated 2 months ago
• microsoft / FractalTensor
  FractalTensor is a programming framework that introduces a novel approach to organizing data in deep neural networks (DNNs) as a list of …
  ☆26Updated 4 months ago
• INT-FlashAttention2024 / INT-FlashAttention
  ☆70Updated 3 months ago
• cassiewilliam / cuda_op_benchmark
  方便扩展的Cuda算子理解和优化框架，仅用在学习使用
  ☆15Updated 11 months ago
• DerrickYLJ / TidalDecode
  [ICLR 2025] TidalDecode: A Fast and Accurate LLM Decoding with Position Persistent Sparse Attention
  ☆36Updated 3 weeks ago
• argonne-lcf / LLM-Inference-Bench
  LLM-Inference-Bench
  ☆40Updated 4 months ago
• mlc-ai / mlc-python
  ☆32Updated this week
• hetailang / SqueezeAttention
  ☆37Updated 7 months ago
• microsoft / ParrotServe
  [OSDI'24] Serving LLM-based Applications Efficiently with Semantic Variable
  ☆155Updated 7 months ago
• LMCache / lmcache-vllm
  The driver for LMCache core to run in vLLM
  ☆40Updated 3 months ago
• opengear-project / GEAR
  GEAR: An Efficient KV Cache Compression Recipefor Near-Lossless Generative Inference of LLM
  ☆161Updated 10 months ago
• merrymercy / Awesome-Efficient-LLM
  A curated list for Efficient Large Language Models
  ☆11Updated last year
• yale-sys / prompt-cache
  Modular and structured prompt caching for low-latency LLM inference
  ☆92Updated 6 months ago