SJTU-IPADS / Bamboo

Related projects ⓘ

Alternatives and complementary repositories for Bamboo

• efeslab / fiddler
  Fast Inference of MoE Models with CPU-GPU Orchestration
  ☆170Updated last week
• TorchMoE / MoE-Infinity
  PyTorch library for cost-effective, fast and easy serving of MoE models.
  ☆101Updated 2 months ago
• sgl-project / sgl-learning-materials
  Materials for learning SGLang
  ☆75Updated this week
• Infini-AI-Lab / MagicDec
  Breaking Throughput-Latency Trade-off for Long Sequences with Speculative Decoding
  ☆70Updated this week
• HanGuo97 / flute
  Fast Matrix Multiplications for Lookup Table-Quantized LLMs
  ☆183Updated last month
• efeslab / Atom
  [MLSys'24] Atom: Low-bit Quantization for Efficient and Accurate LLM Serving
  ☆277Updated 4 months ago
• LMCache / LMCache
  Ultra-Fast and Cheaper Long-Context LLM Inference
  ☆194Updated this week
• microsoft / ParrotServe
  [OSDI'24] Serving LLM-based Applications Efficiently with Semantic Variable
  ☆109Updated last month
• tyler-griggs / melange-release
  ☆36Updated 4 months ago
• yale-sys / prompt-cache
  Modular and structured prompt caching for low-latency LLM inference
  ☆65Updated 5 months ago
• jy-yuan / KIVI
  KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache
  ☆240Updated 3 weeks ago
• microsoft / vidur
  A large-scale simulation framework for LLM inference
  ☆271Updated last month
• opengear-project / GEAR
  GEAR: An Efficient KV Cache Compression Recipefor Near-Lossless Generative Inference of LLM
  ☆146Updated 3 months ago
• FasterDecoding / SnapKV
  ☆183Updated 6 months ago
• Infini-AI-Lab / Sequoia
  scalable and robust tree-based speculative decoding algorithm
  ☆313Updated 2 months ago
• mobiusml / gemlite
  Simple and fast low-bit matmul kernels in CUDA / Triton
  ☆133Updated this week
• WukLab / preble
  Stateful LLM Serving
  ☆37Updated 3 months ago
• FasterDecoding / TEAL
  ☆95Updated last month
• YaoJiayi / CacheBlend
  ☆51Updated last month
• microsoft / vattention
  Dynamic Memory Management for Serving LLMs without PagedAttention
  ☆222Updated this week
• MDK8888 / vllmini
  A minimal implementation of vllm.
  ☆30Updated 3 months ago
• IST-DASLab / Sparse-Marlin
  Boosting 4-bit inference kernels with 2:4 Sparsity
  ☆51Updated 2 months ago
• whyNLP / LCKV
  Layer-Condensed KV cache w/ 10 times larger batch size, fewer params and less computation. Dramatic speed up with better task performance…
  ☆137Updated this week
• interestingLSY / swiftLLM
  A tiny yet powerful LLM inference system tailored for researching purpose. vLLM-equivalent performance with only 2k lines of code (2% of …
  ☆98Updated 4 months ago
• apple / ml-recurrent-drafter
  ☆96Updated last month
• SqueezeAILab / KVQuant
  [NeurIPS 2024] KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization
  ☆302Updated 2 months ago
• mit-han-lab / Quest
  [ICML 2024] Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference
  ☆195Updated last week
• UChi-JCL / CacheGen
  ☆62Updated last month
• SymbioticLab / Oobleck
  A resilient distributed training framework
  ☆85Updated 6 months ago
• mlc-ai / llm-perf-bench
  ☆114Updated 6 months ago