facebookresearch / fastgenLinks
Simple high-throughput inference library
☆119Updated last month
Alternatives and similar repositories for fastgen
Users that are interested in fastgen are comparing it to the libraries listed below
Sorting:
- Lightweight toolkit package to train and fine-tune 1.58bit Language models☆80Updated last month
- RWKV-7: Surpassing GPT☆92Updated 7 months ago
- Tree Attention: Topology-aware Decoding for Long-Context Attention on GPU clusters☆126Updated 6 months ago
- an open source reproduction of NVIDIA's nGPT (Normalized Transformer with Representation Learning on the Hypersphere)☆101Updated 3 months ago
- Samples of good AI generated CUDA kernels☆83Updated 3 weeks ago
- Load compute kernels from the Hub☆191Updated last week
- 👷 Build compute kernels☆68Updated this week
- NanoGPT-speedrunning for the poor T4 enjoyers☆66Updated 2 months ago
- A safetensors extension to efficiently store sparse quantized tensors on disk☆129Updated this week
- ☆109Updated last year
- Query-agnostic KV cache eviction: 3–4× reduction in memory and 2× decrease in latency (Qwen3/2.5, Gemma3, LLaMA3)☆86Updated 2 weeks ago
- ☆68Updated this week
- Repo for "LoLCATs: On Low-Rank Linearizing of Large Language Models"☆239Updated 4 months ago
- research impl of Native Sparse Attention (2502.11089)☆54Updated 4 months ago
- QuIP quantization☆54Updated last year
- An easy-to-understand framework for LLM samplers that rewind and revise generated tokens☆140Updated 4 months ago
- ☆18Updated last year
- The source code of our work "Prepacking: A Simple Method for Fast Prefilling and Increased Throughput in Large Language Models" [AISTATS …☆59Updated 8 months ago
- Training-free Post-training Efficient Sub-quadratic Complexity Attention. Implemented with OpenAI Triton.☆137Updated this week
- Block Transformer: Global-to-Local Language Modeling for Fast Inference (NeurIPS 2024)☆158Updated 2 months ago
- ☆48Updated 11 months ago
- ☆51Updated 7 months ago
- ☆49Updated last year
- Experiments on speculative sampling with Llama models☆128Updated 2 years ago
- Token Omission Via Attention☆128Updated 8 months ago
- ☆56Updated 3 months ago
- PCCL (Prime Collective Communications Library) implements fault tolerant collective communications over IP☆95Updated last month
- [ICLR2025] Breaking Throughput-Latency Trade-off for Long Sequences with Speculative Decoding☆116Updated 6 months ago
- An efficent implementation of the method proposed in "The Era of 1-bit LLMs"☆154Updated 8 months ago
- Testing LLM reasoning abilities with family relationship quizzes.☆62Updated 4 months ago