microsoft / glinthawk
An LLM inference engine, written in C++
☆14Updated 4 months ago
Alternatives and similar repositories for glinthawk
Users that are interested in glinthawk are comparing it to the libraries listed below
Sorting:
- PipeInfer: Accelerating LLM Inference using Asynchronous Pipelined Speculation☆28Updated 6 months ago
- How much energy do GenAI models consume?☆42Updated this week
- ☆32Updated last week
- A resilient distributed training framework☆95Updated last year
- A minimal implementation of vllm.☆40Updated 9 months ago
- Open deep learning compiler stack for cpu, gpu and specialized accelerators☆18Updated last week
- FlexFlow Serve: Low-Latency, High-Performance LLM Serving☆39Updated last week
- FractalTensor is a programming framework that introduces a novel approach to organizing data in deep neural networks (DNNs) as a list of …☆26Updated 4 months ago
- [OSDI'24] Serving LLM-based Applications Efficiently with Semantic Variable☆155Updated 7 months ago
- NEO is a LLM inference engine built to save the GPU memory crisis by CPU offloading☆27Updated 2 months ago
- Compression for Foundation Models☆31Updated last month
- A curated list for Efficient Large Language Models☆11Updated last year
- ☆25Updated 3 months ago
- [ICLR2025] Breaking Throughput-Latency Trade-off for Long Sequences with Speculative Decoding☆116Updated 5 months ago
- [ICLR 2025] TidalDecode: A Fast and Accurate LLM Decoding with Position Persistent Sparse Attention☆36Updated 3 weeks ago
- ☆32Updated this week
- ☆45Updated 10 months ago
- [NeurIPS 2024] Efficient LLM Scheduling by Learning to Rank☆48Updated 6 months ago
- ☆60Updated 11 months ago
- Accelerate LLM preference tuning via prefix sharing with a single line of code☆41Updated 2 weeks ago
- PyTorch implementation of paper "Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline".☆85Updated last year
- ☆12Updated 11 months ago
- Official Implementation of "CheckEmbed: Effective Verification of LLM Solutions to Open-Ended Tasks"☆19Updated last week
- Explore training for quantized models☆18Updated 4 months ago
- Cavs: An Efficient Runtime System for Dynamic Neural Networks☆14Updated 4 years ago
- Official Implementation of SAM-Decoding: Speculative Decoding via Suffix Automaton☆26Updated 3 months ago
- Artifact for "Apparate: Rethinking Early Exits to Tame Latency-Throughput Tensions in ML Serving" [SOSP '24]☆24Updated 5 months ago
- TritonBench: Benchmarking Large Language Model Capabilities for Generating Triton Operators☆38Updated 2 months ago
- An Attention Superoptimizer☆21Updated 3 months ago
- 方便扩展的Cuda算子理解和优化框架，仅用在学习使用☆15Updated 11 months ago