Qcompiler / MIXQ
MIXQ: Taming Dynamic Outliers in Mixed-Precision Quantization by Online Prediction
☆87Updated 5 months ago
Alternatives and similar repositories for MIXQ:
Users that are interested in MIXQ are comparing it to the libraries listed below
- Support mixed-precsion inference with vllm☆83Updated 3 months ago
- ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference☆158Updated 5 months ago
- Mixed precision inference by Tensorrt-LLM☆79Updated 5 months ago
- An acceleration library that supports arbitrary bit-width combinatorial quantization operations☆221Updated 6 months ago
- [COLM 2024] TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding☆247Updated 7 months ago
- [ICLR 2025🔥] SVD-LLM & [NAACL 2025🔥] SVD-LLM V2☆197Updated last month
- ☆106Updated 4 years ago
- Build CUDA Neural Network From Scratch☆17Updated 7 months ago
- official implementation of paper SDP4Bit: Toward 4-bit Communication Quantization in Sharded Data Parallelism for LLM Training☆35Updated 4 months ago
- The framework to prune LLMs to any size and any config.☆91Updated last year
- [ACL 2024] A novel QAT with Self-Distillation framework to enhance ultra low-bit LLMs.☆109Updated 11 months ago
- ☆92Updated 7 months ago
- Official Implementation of "Accel-GNN: High-Performance GPU Accelerator Design for Graph Neural Networks"☆49Updated 3 weeks ago
- Higher performance OpenAI LLM service than vLLM serve: A pure C++ high-performance OpenAI LLM service implemented with GPRS+TensorRT-LLM+…☆130Updated this week
- SQuant [ICLR22]☆131Updated 2 years ago
- The Official Implementation of Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference☆72Updated 2 months ago
- Official implementation of "MaxK-GNN: Extremely Fast GPU Kernel Design for Accelerating Graph Neural Networks Training"☆37Updated last year
- This is the official code repository of MoTCoder: Elevating Large Language Models with Modular of Thought for Challenging Programming Tas…☆79Updated 3 weeks ago
- A GPU-optimized system for efficient long-context LLMs decoding with low-bit KV cache.☆32Updated 3 weeks ago
- APOLLO: SGD-like Memory, AdamW-level Performance☆202Updated this week
- Official implementation of ICML 2024 paper "ExCP: Extreme LLM Checkpoint Compression via Weight-Momentum Joint Shrinking".☆47Updated 9 months ago
- [ECCV 2024] Efficient Inference of Vision Instruction-Following Models with Elastic Cache☆42Updated 8 months ago
- Official PyTorch implementation of "FlatQuant: Flatness Matters for LLM Quantization"☆117Updated 3 weeks ago
- Code Repository of Evaluating Quantized Large Language Models☆121Updated 7 months ago
- ☆68Updated 4 months ago
- An easy-to-use package for implementing SmoothQuant for LLMs☆96Updated last week
- Implement Flash Attention using Cute.☆74Updated 4 months ago
- QQQ is an innovative and hardware-optimized W4A8 quantization solution for LLMs.☆115Updated last week
- QAQ: Quality Adaptive Quantization for LLM KV Cache☆49Updated last year
- RobustFT: Robust Supervised Fine-tuning for Large Language Models under Noisy Response☆40Updated 3 months ago