Infini-AI-Lab / TriForce

Related projects ⓘ

Alternatives and complementary repositories for TriForce

• bytedance / ShadowKV
  ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference
  ☆124Updated 3 weeks ago
• AIoT-MLSys-Lab / SVD-LLM
  Official Code for "SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression"
  ☆95Updated last month
• Qcompiler / MIXQ
  MIXQ: Taming Dynamic Outliers in Mixed-Precision Quantization by Online Prediction
  ☆81Updated 3 weeks ago
• bytedance / ABQ-LLM
  An acceleration library that supports arbitrary bit-width combinatorial quantization operations
  ☆226Updated last month
• mit-han-lab / Quest
  [ICML 2024] Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference
  ☆202Updated 2 weeks ago
• FasterDecoding / SnapKV
  ☆188Updated 6 months ago
• jy-yuan / KIVI
  [ICML 2024] KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache
  ☆241Updated last month
• opengear-project / GEAR
  GEAR: An Efficient KV Cache Compression Recipefor Near-Lossless Generative Inference of LLM
  ☆147Updated 4 months ago
• Zefan-Cai / KVCache-Factory
  Unified KV Cache Compression Methods for LLMs
  ☆728Updated this week
• Infini-AI-Lab / MagicDec
  Breaking Throughput-Latency Trade-off for Long Sequences with Speculative Decoding
  ☆78Updated this week
• efeslab / Atom
  [MLSys'24] Atom: Low-bit Quantization for Efficient and Accurate LLM Serving
  ☆278Updated 4 months ago
• hemingkx / Spec-Bench
  Spec-Bench: A Comprehensive Benchmark and Unified Evaluation Platform for Speculative Decoding (ACL 2024 Findings)
  ☆188Updated 3 weeks ago
• thunlp / Ouroboros
  Ouroboros: Speculative Decoding with Large Model Enhanced Drafting (EMNLP 2024 main)
  ☆76Updated last month
• October2001 / Awesome-KV-Cache-Compression
  📰 Must-read papers on KV Cache Compression (constantly updating 🤗).
  ☆136Updated this week
• RulinShao / LightSeq
  Official repository for LightSeq: Sequence Level Parallelism for Distributed Training of Long Context Transformers
  ☆195Updated 3 months ago
• dilab-zju / self-speculative-decoding
  Code associated with the paper **Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding**
  ☆138Updated 5 months ago
• FasterDecoding / REST
  REST: Retrieval-Based Speculative Decoding, NAACL 2024
  ☆176Updated last month
• Zefan-Cai / Awesome-LLM-KV-Cache
  Awesome-LLM-KV-Cache: A curated list of 📙Awesome LLM KV Cache Papers with Codes.
  ☆106Updated last week
• FFY0 / AdaKV
  The Official Implementation of Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference
  ☆38Updated this week
• FasterDecoding / TEAL
  ☆96Updated last month
• mit-han-lab / Block-Sparse-Attention
  A sparse attention kernel supporting mix sparse patterns
  ☆55Updated last month
• astramind-ai / Mixture-of-depths
  Unofficial implementation for the paper "Mixture-of-Depths: Dynamically allocating compute in transformer-based language models"
  ☆134Updated 5 months ago
• SqueezeAILab / KVQuant
  [NeurIPS 2024] KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization
  ☆305Updated 3 months ago
• Ledzy / BAdam
  [NeurIPS 2024] BAdam: A Memory Efficient Full Parameter Optimization Method for Large Language Models
  ☆214Updated last month
• Equationliu / Kangaroo
  Implementation of Kangaroo: Lossless Self-Speculative Decoding via Double Early Exiting
  ☆44Updated 4 months ago
• IsaacRe / vllm-kvcompress
  KV cache compression for high-throughput LLM inference
  ☆87Updated this week
• FMInference / H2O
  [NeurIPS'23] H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models.
  ☆391Updated 3 months ago
• andy-yang-1 / DoubleSparse
  16-fold memory access reduction with nearly no loss
  ☆59Updated last week
• shadowpa0327 / Palu
  Code for Palu: Compressing KV-Cache with Low-Rank Projection
  ☆57Updated this week
• interestingLSY / swiftLLM
  A tiny yet powerful LLM inference system tailored for researching purpose. vLLM-equivalent performance with only 2k lines of code (2% of …
  ☆104Updated 4 months ago