microsoft / glinthawkLinks
An LLM inference engine, written in C++
☆15Updated last week
Alternatives and similar repositories for glinthawk
Users that are interested in glinthawk are comparing it to the libraries listed below
Sorting:
- FractalTensor is a programming framework that introduces a novel approach to organizing data in deep neural networks (DNNs) as a list of …☆26Updated 6 months ago
- PipeInfer: Accelerating LLM Inference using Asynchronous Pipelined Speculation☆30Updated 7 months ago
- Efficient Compute-Communication Overlap for Distributed LLM Inference☆13Updated this week
- ☆47Updated last year
- ☆32Updated last year
- Compression for Foundation Models☆32Updated 3 months ago
- Beyond KV Caching: Shared Attention for Efficient LLMs☆19Updated 11 months ago
- A lightweight, user-friendly data-plane for LLM training.☆19Updated 2 months ago
- A resilient distributed training framework☆95Updated last year
- A minimal implementation of vllm.☆44Updated 11 months ago
- [ICLR2025] Breaking Throughput-Latency Trade-off for Long Sequences with Speculative Decoding☆116Updated 6 months ago
- ☆62Updated last year
- Explore training for quantized models☆18Updated last week
- Modular and structured prompt caching for low-latency LLM inference☆96Updated 7 months ago
- ☆35Updated last month
- FlexFlow Serve: Low-Latency, High-Performance LLM Serving☆42Updated last month
- [OSDI'24] Serving LLM-based Applications Efficiently with Semantic Variable☆163Updated 9 months ago
- NEO is a LLM inference engine built to save the GPU memory crisis by CPU offloading☆39Updated last week
- Aims to implement dual-port and multi-qp solutions in deepEP ibrc transport☆50Updated last month
- [ICLR 2025] TidalDecode: A Fast and Accurate LLM Decoding with Position Persistent Sparse Attention☆39Updated 2 months ago
- LLM Serving Performance Evaluation Harness☆78Updated 4 months ago
- ☆109Updated 8 months ago
- Open deep learning compiler stack for cpu, gpu and specialized accelerators☆19Updated this week
- Personal solutions to the Triton Puzzles☆19Updated 11 months ago
- ☆48Updated 11 months ago
- PipeRAG: Fast Retrieval-Augmented Generation via Algorithm-System Co-design (KDD 2025)☆21Updated last year
- 方便扩展的Cuda算子理解和优化框架，仅用在学习使用☆15Updated last year
- My CUDA solution to the 1BRC☆10Updated last year
- A model serving framework for various research and production scenarios. Seamlessly built upon the PyTorch and HuggingFace ecosystem.☆23Updated 8 months ago
- Implementation from scratch in C of the Multi-head latent attention used in the Deepseek-v3 technical paper.☆18Updated 5 months ago