bytedance / ABQ-LLM
An acceleration library that supports arbitrary bit-width combinatorial quantization operations
☆214Updated 4 months ago
Alternatives and similar repositories for ABQ-LLM:
Users that are interested in ABQ-LLM are comparing it to the libraries listed below
- Unified KV Cache Compression Methods for Auto-Regressive Models☆886Updated last month
- MIXQ: Taming Dynamic Outliers in Mixed-Precision Quantization by Online Prediction☆78Updated 3 months ago
- Support mixed-precsion inference with vllm☆80Updated last month
- ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference☆148Updated 3 months ago
- [COLM 2024] TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding☆239Updated 5 months ago
- A highly optimized LLM inference acceleration engine for Llama and its variants.☆855Updated this week
- ☆107Updated 4 years ago
- 【高性能OpenAI LLM服务】通过GPRS+TensorRT-LLM+Tokenizers.cpp实现纯C++版高性能OpenAI LLM服务，支持chat和function call模式，支持ai agent，支持分布式多卡推理，支持多模态，支持gradio聊天界面。☆95Updated last week
- SQuant [ICLR22]☆131Updated 2 years ago
- Mixed precision inference by Tensorrt-LLM☆76Updated 3 months ago
- 使用deepspeed从头开始训练一个LLM,经过pretrain和sft阶段,验证llm学习知识、理解语言、回答问题 的能力☆141Updated 7 months ago
- Official Code for "SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression"☆115Updated 3 weeks ago
- TVM Documentation in Chinese Simplified / TVM 中文文档☆805Updated this week
- The framework to prune LLMs to any size and any config.☆87Updated 11 months ago
- Official PyTorch implementation of FlatQuant: Flatness Matters for LLM Quantization☆102Updated 3 weeks ago
- APOLLO: SGD-like Memory, AdamW-level Performance☆150Updated last week
- JittorGeometric is a Jittor-based graph machine learning library.☆124Updated last month
- 【深度学习模型部署框架】支持tf/torch/trt/trtllm/vllm以及更多nn框架，支持dynamic batching、streaming模式，支持python/c++双语言，可限制，可拓展，高性能。帮助用户快速地将模型部署到线上，并通过http/rpc接口方式…☆155Updated last month
- Reverse Chain-of-Thought Problem Generation for Geometric Reasoning in Large Multimodal Models☆169Updated 3 months ago
- A toolkit enhances PyTorch with specialized functions for low-bit quantized neural networks.☆196Updated 7 months ago
- QQQ is an innovative and hardware-optimized W4A8 quantization solution for LLMs.☆101Updated this week
- An easy-to-use package for implementing SmoothQuant for LLMs☆92Updated 9 months ago
- Lyra: An Efficient and Speech-Centric Framework for Omni-Cognition☆276Updated last month
- MPLSandbox is an out-of-the-box multi-programming language sandbox designed to provide unified and comprehensive feedback from compiler a…☆172Updated 3 months ago
- [NeurIPS 2024] BAdam: A Memory Efficient Full Parameter Optimization Method for Large Language Models☆243Updated 2 months ago
- ☆81Updated 5 months ago
- [ACL 2024] A novel QAT with Self-Distillation framework to enhance ultra low-bit LLMs.☆100Updated 9 months ago
- FlagPerf is an open-source software platform for benchmarking AI chips.☆321Updated 2 weeks ago
- An algorithm for static activation quantization of LLMs☆115Updated 2 weeks ago