Infini-AI-Lab / SequoiaLinks
scalable and robust tree-based speculative decoding algorithm
☆348Updated 4 months ago
Alternatives and similar repositories for Sequoia
Users that are interested in Sequoia are comparing it to the libraries listed below
Sorting:
- [NeurIPS 2024] KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization☆359Updated 10 months ago
- ☆542Updated 10 months ago
- [ICML 2024] CLLMs: Consistency Large Language Models☆394Updated 7 months ago
- For releasing code related to compression methods for transformers, accompanying our publications☆431Updated 5 months ago
- Code for paper: "QuIP: 2-Bit Quantization of Large Language Models With Guarantees"☆370Updated last year
- Explorations into some recent techniques surrounding speculative decoding☆269Updated 6 months ago
- Code for the paper "QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Models".☆277Updated last year
- [ICLR'25] Fast Inference of MoE Models with CPU-GPU Orchestration☆213Updated 7 months ago
- ☆198Updated 6 months ago
- ☆541Updated 7 months ago
- KV cache compression for high-throughput LLM inference☆131Updated 4 months ago
- [MLSys'24] Atom: Low-bit Quantization for Efficient and Accurate LLM Serving☆310Updated 11 months ago
- [ICML 2024] KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache☆303Updated 5 months ago
- REST: Retrieval-Based Speculative Decoding, NAACL 2024☆204Updated 6 months ago
- Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients.☆198Updated 11 months ago
- ☆256Updated last year
- LLM KV cache compression made easy☆520Updated this week
- 🚀 Efficiently (pre)training foundation models with native PyTorch features, including FSDP for training and SDPA implementation of Flash…☆253Updated this week
- Repo for "LoLCATs: On Low-Rank Linearizing of Large Language Models"☆239Updated 4 months ago
- Implementation of Speculative Sampling as described in "Accelerating Large Language Model Decoding with Speculative Sampling" by Deepmind☆98Updated last year
- Repository for the QUIK project, enabling the use of 4bit kernels for generative inference - EMNLP 2024☆180Updated last year
- Cold Compress is a hackable, lightweight, and open-source toolkit for creating and benchmarking cache compression methods built on top of…☆137Updated 10 months ago
- 3x Faster Inference; Unofficial implementation of EAGLE Speculative Decoding☆66Updated this week
- Official repository for DistFlashAttn: Distributed Memory-efficient Attention for Long-context LLMs Training☆210Updated 10 months ago
- Code for Neurips24 paper: QuaRot, an end-to-end 4-bit inference of large language models.☆395Updated 6 months ago
- Experiments on speculative sampling with Llama models☆128Updated 2 years ago
- An efficient GPU support for LLM inference with x-bit quantization (e.g. FP6,FP5).☆252Updated 7 months ago
- GPTQ inference Triton kernel☆302Updated 2 years ago
- ☆130Updated 4 months ago
- A safetensors extension to efficiently store sparse quantized tensors on disk☆126Updated this week