neuralmagic / compressed-tensors
A safetensors extension to efficiently store sparse quantized tensors on disk
☆64Updated this week
Alternatives and similar repositories for compressed-tensors:
Users that are interested in compressed-tensors are comparing it to the libraries listed below
- Boosting 4-bit inference kernels with 2:4 Sparsity☆64Updated 4 months ago
- Fast Matrix Multiplications for Lookup Table-Quantized LLMs☆219Updated last week
- Tritonbench is a collection of PyTorch custom operators with example inputs to measure their performance.☆75Updated this week
- Applied AI experiments and examples for PyTorch☆211Updated this week
- Fast low-bit matmul kernels in Triton☆187Updated last week
- Cataloging released Triton kernels.☆155Updated last week
- Collection of kernels written in Triton language☆90Updated 2 months ago
- ☆57Updated 7 months ago
- extensible collectives library in triton☆76Updated 3 months ago
- High-speed GEMV kernels, at most 2.7x speedup compared to pytorch baseline.☆93Updated 6 months ago
- ☆170Updated this week
- This repository contains the experimental PyTorch native float8 training UX☆219Updated 5 months ago
- Repository for Sparse Finetuning of LLMs via modified version of the MosaicML llmfoundry☆40Updated last year
- PyTorch bindings for CUTLASS grouped GEMM.☆58Updated 2 months ago
- ☆96Updated 4 months ago
- ☆64Updated 2 months ago
- An efficient GPU support for LLM inference with x-bit quantization (e.g. FP6,FP5).☆230Updated 2 months ago
- Breaking Throughput-Latency Trade-off for Long Sequences with Speculative Decoding☆107Updated last month
- ☆99Updated 3 weeks ago
- Odysseus: Playground of LLM Sequence Parallelism☆64Updated 7 months ago
- 🚀 Collection of components for development, training, tuning, and inference of foundation models leveraging PyTorch native components.☆182Updated this week
- ☆167Updated 3 months ago
- LLM KV cache compression made easy☆303Updated this week
- ring-attention experiments☆116Updated 3 months ago
- Triton-based implementation of Sparse Mixture of Experts.☆192Updated last month
- GEAR: An Efficient KV Cache Compression Recipefor Near-Lossless Generative Inference of LLM☆152Updated 6 months ago
- ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training Multiplication-Less Reparameterization☆100Updated 3 months ago
- Fast Hadamard transform in CUDA, with a PyTorch interface☆132Updated 7 months ago
- QUICK: Quantization-aware Interleaving and Conflict-free Kernel for efficient LLM inference☆114Updated 10 months ago
- [MLSys'24] Atom: Low-bit Quantization for Efficient and Accurate LLM Serving☆290Updated 6 months ago