facebookresearch / fastgenLinks
Simple high-throughput inference library
☆149Updated 6 months ago
Alternatives and similar repositories for fastgen
Users that are interested in fastgen are comparing it to the libraries listed below
Sorting:
- Samples of good AI generated CUDA kernels☆91Updated 5 months ago
- RWKV-7: Surpassing GPT☆100Updated 11 months ago
- Lightweight toolkit package to train and fine-tune 1.58bit Language models☆98Updated 5 months ago
- Tree Attention: Topology-aware Decoding for Long-Context Attention on GPU clusters☆130Updated 11 months ago
- Make triton easier☆48Updated last year
- Storing long contexts in tiny caches with self-study☆213Updated 3 weeks ago
- Inference of Mamba models in pure C☆192Updated last year
- A collection of reproducible inference engine benchmarks☆37Updated 6 months ago
- ☆39Updated last year
- PyTorch implementation of models from the Zamba2 series.☆185Updated 9 months ago
- Simple and efficient DeepSeek V3 SFT using pipeline parallel and expert parallel, with both FP8 and BF16 trainings☆91Updated 3 months ago
- Efficient non-uniform quantization with GPTQ for GGUF☆53Updated last month
- train with kittens!☆63Updated last year
- Optimizing Causal LMs through GRPO with weighted reward functions and automated hyperparameter tuning using Optuna☆58Updated 3 weeks ago
- An efficent implementation of the method proposed in "The Era of 1-bit LLMs"☆154Updated last year
- Token Omission Via Attention☆127Updated last year
- Repo for "LoLCATs: On Low-Rank Linearizing of Large Language Models"☆249Updated 9 months ago
- 👷 Build compute kernels☆171Updated this week
- Training-free Post-training Efficient Sub-quadratic Complexity Attention. Implemented with OpenAI Triton.☆148Updated 2 weeks ago
- ☆52Updated last year
- ☆50Updated last year
- an open source reproduction of NVIDIA's nGPT (Normalized Transformer with Representation Learning on the Hypersphere)☆107Updated 8 months ago
- NanoGPT-speedrunning for the poor T4 enjoyers☆72Updated 6 months ago
- vLLM adapter for a TGIS-compatible gRPC server.☆44Updated this week
- Google TPU optimizations for transformers models☆122Updated 9 months ago
- The source code of our work "Prepacking: A Simple Method for Fast Prefilling and Increased Throughput in Large Language Models" [AISTATS …☆60Updated last year
- Experiments on speculative sampling with Llama models☆126Updated 2 years ago
- DeMo: Decoupled Momentum Optimization☆197Updated 11 months ago
- ☆218Updated 9 months ago
- Prepare for DeekSeek R1 inference: Benchmark CPU, DRAM, SSD, iGPU, GPU, ... with efficient code.☆73Updated 9 months ago