Alternatives and similar repositories for fastgen

Users that are interested in fastgen are comparing it to the libraries listed below

• Michaelvll / llm-ie-benchmarks
  A collection of reproducible inference engine benchmarks
  ☆30Updated 3 weeks ago
• nyunAI / PruneGPT
  ☆53Updated 11 months ago
• Infini-AI-Lab / MagicDec
  [ICLR2025] Breaking Throughput-Latency Trade-off for Long Sequences with Speculative Decoding
  ☆116Updated 5 months ago
• UmerHA / triton_util
  Make triton easier
  ☆47Updated 11 months ago
• BlinkDL / modded-nanogpt-rwkv
  RWKV-7: Surpassing GPT
  ☆84Updated 6 months ago
• BlinkDL / fast.c
  Prepare for DeekSeek R1 inference: Benchmark CPU, DRAM, SSD, iGPU, GPU, ... with efficient code.
  ☆72Updated 3 months ago
• IsaacRe / vllm-kvcompress
  KV cache compression for high-throughput LLM inference
  ☆127Updated 3 months ago
• dust-tt / llama-ssp
  Experiments on speculative sampling with Llama models
  ☆126Updated last year
• Zyphra / tree_attention
  Tree Attention: Topology-aware Decoding for Long-Context Attention on GPU clusters
  ☆126Updated 5 months ago
• IST-DASLab / MoE-Quant
  Code for data-aware compression of DeepSeek models
  ☆24Updated last month
• IST-DASLab / Sparse-Marlin
  Boosting 4-bit inference kernels with 2:4 Sparsity
  ☆73Updated 8 months ago
• simon-mo / vLLM-Benchmark
  ☆25Updated 3 weeks ago
• ScalingIntelligence / hydragen
  Hydragen: High-Throughput LLM Inference with Shared Prefixes
  ☆36Updated last year
• fw-ai / benchmark
  Benchmark suite for LLMs from Fireworks.ai
  ☆73Updated this week
• snowflakedb / ArcticInference
  ☆59Updated this week
• astramind-ai / BitMat
  An efficent implementation of the method proposed in "The Era of 1-bit LLMs"
  ☆154Updated 7 months ago
• wdlctc / headinfer
  ☆46Updated this week
• zhuzilin / faster-nougat
  Implementation of nougat that focuses on processing pdf locally.
  ☆81Updated 4 months ago
• kyleliang919 / Super_Muon
  ☆55Updated last month
• eth-easl / deltazip
  Compression for Foundation Models
  ☆31Updated last month
• EmbeddedLLM / vllm
  vLLM: A high-throughput and memory-efficient inference and serving engine for LLMs
  ☆86Updated this week
• kroggen / mamba.c
  Inference of Mamba models in pure C
  ☆188Updated last year
• abacusai / gh200-llm
  Docker image NVIDIA GH200 machines - optimized for vllm serving and hf trainer finetuning
  ☆40Updated 2 months ago
• GreenBitAI / green-bit-llm
  A toolkit for fine-tuning, inferencing, and evaluating GreenBitAI's LLMs.
  ☆84Updated 2 months ago
• NolanoOrg / SpectraSuite
  ☆47Updated 10 months ago
• shreyansh26 / Attention-Mask-Patterns
  Using FlexAttention to compute attention with different masking patterns
  ☆43Updated 7 months ago
• LLM360 / crystalcoder-data-prep
  Data preparation code for CrystalCoder 7B LLM
  ☆44Updated last year
• LMCache / lmcache-vllm
  The driver for LMCache core to run in vLLM
  ☆40Updated 3 months ago
• opendatahub-io / vllm-tgis-adapter
  vLLM adapter for a TGIS-compatible gRPC server.
  ☆27Updated this week
• IST-DASLab / QIGen
  Repository for CPU Kernel Generation for LLM Inference
  ☆26Updated last year