neuralmagic / compressed-tensors

Related projects: ⓘ

• mobiusml / gemlite
  Simple and fast low-bit matmul kernels in CUDA
  ☆48Updated this week
• HanGuo97 / flute
  Fast Matrix Multiplications for Lookup Table-Quantized LLMs
  ☆156Updated this week
• IST-DASLab / SparseFinetuning
  Repository for Sparse Finetuning of LLMs via modified version of the MosaicML llmfoundry
  ☆36Updated 8 months ago
• AnswerDotAI / cold-compress
  Cold Compress is a hackable, lightweight, and open-source toolkit for creating and benchmarking cache compression methods built on top of…
  ☆73Updated last month
• IST-DASLab / Sparse-Marlin
  Boosting 4-bit inference kernels with 2:4 Sparsity
  ☆47Updated 2 weeks ago
• cuda-mode / triton-index
  Cataloging released Triton kernels.
  ☆111Updated 3 weeks ago
• shadowpa0327 / Palu
  Code for Palu: Compressing KV-Cache with Low-Rank Projection
  ☆39Updated this week
• Deep-Learning-Profiling-Tools / triton-viz
  ☆124Updated last week
• pytorch-labs / applied-ai
  Applied AI experiments and examples for PyTorch
  ☆123Updated last month
• siyan-zhao / prepacking
  The source code of our work "Prepacking: A Simple Method for Fast Prefilling and Increased Throughput in Large Language Models"
  ☆57Updated 5 months ago
• IST-DASLab / QIGen
  Repository for CPU Kernel Generation for LLM Inference
  ☆25Updated last year
• Zyphra / tree_attention
  Tree Attention: Topology-aware Decoding for Long-Context Attention on GPU clusters
  ☆94Updated 2 weeks ago
• srush / triton-autodiff
  Experiment of using Tangent to autodiff triton
  ☆66Updated 7 months ago
• apple / ml-recurrent-drafter
  ☆61Updated 3 weeks ago
• aredden / torch-cublas-hgemm
  PyTorch half precision gemm lib w/ fused optional bias + optional relu/gelu
  ☆25Updated 3 weeks ago
• mgmalek / efficient_cross_entropy
  ☆66Updated 3 months ago
• softmax1 / Flash-Attention-Softmax-N
  CUDA and Triton implementations of Flash Attention with SoftmaxN.
  ☆66Updated 3 months ago
• pytorch-labs / float8_experimental
  This repository contains the experimental PyTorch native float8 training UX
  ☆210Updated last month
• siboehm / ShallowSpeed
  Small scale distributed training of sequential deep learning models, built on Numpy and MPI.
  ☆88Updated 11 months ago
• cuda-mode / ring-attention
  ring-attention experiments
  ☆89Updated 5 months ago
• opengear-project / GEAR
  GEAR: An Efficient KV Cache Compression Recipefor Near-Lossless Generative Inference of LLM
  ☆134Updated 2 months ago
• Dao-AILab / fast-hadamard-transform
  Fast Hadamard transform in CUDA, with a PyTorch interface
  ☆87Updated 3 months ago
• zankner / Hydra
  ☆32Updated 7 months ago
• drisspg / transformer_nuggets
  A place to store reusable transformer components of my own creation or found on the interwebs
  ☆43Updated 3 weeks ago
• fw-ai / llama-cuda-graph-example
  Example of applying CUDA graphs to LLaMA-v2
  ☆10Updated last year
• foundation-model-stack / foundation-model-stack
  🚀 Collection of components for development, training, tuning, and inference of foundation models leveraging PyTorch native components.
  ☆144Updated this week
• GreenBitAI / green-bit-llm
  A toolkit for fine-tuning, inferencing, and evaluating GreenBitAI's LLMs.
  ☆68Updated 2 months ago
• wangsiping97 / FastGEMV
  High-speed GEMV kernels, at most 2.7x speedup compared to pytorch baseline.
  ☆81Updated 2 months ago
• efeslab / fiddler
  Fast Inference of MoE Models with CPU-GPU Orchestration
  ☆163Updated 3 months ago
• Infini-AI-Lab / MagicDec
  Breaking Throughput-Latency Trade-off for Long Sequences with Speculative Decoding
  ☆55Updated this week