Alternatives and similar repositories for TriForce:

Users that are interested in TriForce are comparing it to the libraries listed below

• bytedance / ShadowKV
  ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference
  ☆145Updated 2 months ago
• AIoT-MLSys-Lab / SVD-LLM
  Official Code for "SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression"
  ☆112Updated 3 months ago
• Qcompiler / MIXQ
  MIXQ: Taming Dynamic Outliers in Mixed-Precision Quantization by Online Prediction
  ☆76Updated 2 months ago
• bytedance / ABQ-LLM
  An acceleration library that supports arbitrary bit-width combinatorial quantization operations
  ☆212Updated 3 months ago
• mit-han-lab / Quest
  [ICML 2024] Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference
  ☆234Updated last month
• jy-yuan / KIVI
  [ICML 2024] KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache
  ☆264Updated this week
• FasterDecoding / SnapKV
  ☆212Updated 8 months ago
• Zefan-Cai / KVCache-Factory
  Unified KV Cache Compression Methods for Auto-Regressive Models
  ☆852Updated 2 weeks ago
• Infini-AI-Lab / MagicDec
  Breaking Throughput-Latency Trade-off for Long Sequences with Speculative Decoding
  ☆107Updated last month
• Equationliu / Kangaroo
  [NeurIPS 2024] The official implementation of "Kangaroo: Lossless Self-Speculative Decoding for Accelerating LLMs via Double Early Exitin…
  ☆47Updated 6 months ago
• Qcompiler / vllm-mixed-precision
  Support mixed-precsion inference with vllm
  ☆79Updated last week
• efeslab / Atom
  [MLSys'24] Atom: Low-bit Quantization for Efficient and Accurate LLM Serving
  ☆291Updated 6 months ago
• FasterDecoding / REST
  REST: Retrieval-Based Speculative Decoding, NAACL 2024
  ☆186Updated last month
• FasterDecoding / TEAL
  ☆107Updated 3 months ago
• thunlp / Ouroboros
  Ouroboros: Speculative Decoding with Large Model Enhanced Drafting (EMNLP 2024 main)
  ☆85Updated 3 months ago
• opengear-project / GEAR
  GEAR: An Efficient KV Cache Compression Recipefor Near-Lossless Generative Inference of LLM
  ☆152Updated 6 months ago
• RulinShao / LightSeq
  Official repository for LightSeq: Sequence Level Parallelism for Distributed Training of Long Context Transformers
  ☆204Updated 5 months ago
• hemingkx / Spec-Bench
  Spec-Bench: A Comprehensive Benchmark and Unified Evaluation Platform for Speculative Decoding (ACL 2024 Findings)
  ☆214Updated 2 months ago
• FFY0 / AdaKV
  The Official Implementation of Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference
  ☆54Updated last month
• smart-lty / ParallelSpeculativeDecoding
  The official code for paper "parallel speculative decoding with adaptive draft length."
  ☆32Updated 4 months ago
• IsaacRe / vllm-kvcompress
  KV cache compression for high-throughput LLM inference
  ☆104Updated last month
• astramind-ai / Mixture-of-depths
  Unofficial implementation for the paper "Mixture-of-Depths: Dynamically allocating compute in transformer-based language models"
  ☆145Updated 7 months ago
• Infini-AI-Lab / MagicPIG
  MagicPIG: LSH Sampling for Efficient LLM Generation
  ☆179Updated last month
• ChenMnZ / PrefixQuant
  An algorithm for static activation quantization of LLMs
  ☆107Updated last week
• October2001 / Awesome-KV-Cache-Compression
  📰 Must-read papers on KV Cache Compression (constantly updating 🤗).
  ☆260Updated this week
• SqueezeAILab / KVQuant
  [NeurIPS 2024] KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization
  ☆325Updated 5 months ago
• andy-yang-1 / DoubleSparse
  16-fold memory access reduction with nearly no loss
  ☆63Updated 2 months ago
• zhengzangw / Sequence-Scheduling
  PyTorch implementation of paper "Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline".
  ☆81Updated last year
• FMInference / H2O
  [NeurIPS'23] H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models.
  ☆416Updated 5 months ago
• thu-nics / MoA
  The official implementation of the paper <MoA: Mixture of Sparse Attention for Automatic Large Language Model Compression>
  ☆110Updated last month