Layered prefill changes the scheduling axis from tokens to layers and removes redundant MoE weight reloads while keeping decode stall free. The result is lower TTFT, lower end-to-end latency, and lower energy per token without hurting TBT stability.
☆18Mar 9, 2026Updated 4 months ago
Alternatives and similar repositories for layered-prefill
Users that are interested in layered-prefill are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- ☆55Oct 14, 2025Updated 9 months ago
- C++17 implementation of einops for libtorch - clear and reliable tensor manipulations with einstein-like notation☆12Oct 16, 2023Updated 2 years ago
- [PACT'24] GraNNDis. A fast and unified distributed graph neural network (GNN) training framework for both full-batch (full-graph) and min…☆10Aug 13, 2024Updated last year
- vLLM plugin for RBLN NPU☆56Updated this week
- A collection of GPU experiments and benchmarks for my personal understanding and research.☆34Updated this week
- Bare Metal GPUs on DigitalOcean Gradient AI • AdPurpose-built for serious AI teams training foundational models, running large-scale inference, and pushing the boundaries of what's possible.
- ☆11Aug 23, 2023Updated 2 years ago
- PyTorchSim is a Comprehensive, Fast, and Accurate NPU Simulation Framework☆131Updated this week
- ☆116Jul 4, 2024Updated 2 years ago
- It's All In the Teacher: Zero-Shot Quantization Brought Closer to the Teacher [CVPR 2022 Oral]☆29Sep 15, 2022Updated 3 years ago
- Proteus: A High-Throughput Inference-Serving System with Accuracy Scaling☆13Mar 7, 2024Updated 2 years ago
- ☆13Jun 14, 2026Updated last month
- A Full-System Simulator for CXL-Based SSD Memory System☆44Dec 24, 2024Updated last year
- Sample Codes using NVSHMEM on Multi-GPU☆30Jan 22, 2023Updated 3 years ago
- Benchmarking OpenBLAS on the Apple M1☆17Dec 31, 2020Updated 5 years ago
- Deploy on Railway without the complexity - Free Credits Offer • AdConnect your repo and Railway handles the rest with instant previews. Quickly provision container image services, databases, and storage volumes.
- [CVPR 2023] "TrojViT: Trojan Insertion in Vision Transformers" by Mengxin Zheng, Qian Lou, Lei Jiang☆15Jan 5, 2024Updated 2 years ago
- Panopticon is a complete in-DRAM RowHammer mitigation. This code simulates an implementation of Panopticon in DDR5.☆14Jun 2, 2023Updated 3 years ago
- A retargetable and extensible synthesis-based compiler for modern hardware architectures☆20Nov 20, 2025Updated 7 months ago
- A backend agnostic modular FHE library over the Torus using bivariate polynomial representation☆71Updated this week
- ☆17Apr 9, 2025Updated last year
- matmul using AMX instructions☆24May 7, 2024Updated 2 years ago
- Secure Inference Resilient Against Malicious Clients☆14May 3, 2022Updated 4 years ago
- ⚡ A seamless integration of HuggingFace Transformers & Diffusers with RBLN SDK for efficient inference on RBLN NPUs.☆19Updated this week
- A Cycle-level simulator for M2NDP☆40Aug 14, 2025Updated 11 months ago
- Deploy to Railway using AI coding agents - Free Credits Offer • AdUse Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
- [ICLR 2022] "PipeGCN: Efficient Full-Graph Training of Graph Convolutional Networks with Pipelined Feature Communication" by Cheng Wan, Y…☆34Mar 15, 2023Updated 3 years ago
- ☆28Nov 29, 2024Updated last year
- Preview Code for Continuum Paper☆89Updated this week
- Ok-Topk is a scheme for distributed training with sparse gradients. Ok-Topk integrates a novel sparse allreduce algorithm (less than 6k c…☆27Dec 10, 2022Updated 3 years ago
- ☆22Feb 26, 2023Updated 3 years ago
- SwarmIO is an SSD emulation framework for next-generation GPU-centric storage systems research☆52May 24, 2026Updated last month
- Official Implementation of the CVPR'23 paper 'Regularization of polynomial networks for image recognition'.☆10Jun 8, 2023Updated 3 years ago
- Multi-layer agent orchestration. PM plans, specialists execute.☆16May 24, 2026Updated last month
- Prompt format and padding guide for Llama 2☆12Sep 18, 2023Updated 2 years ago
- Deploy on Railway without the complexity - Free Credits Offer • AdConnect your repo and Railway handles the rest with instant previews. Quickly provision container image services, databases, and storage volumes.
- [ACL 2025] Squeezed Attention: Accelerating Long Prompt LLM Inference☆58Nov 20, 2024Updated last year
- [ISCA'25] LIA: A Single-GPU LLM Inference Acceleration with Cooperative AMX-Enabled CPU-GPU Computation and CXL Offloading☆13Jun 28, 2025Updated last year
- ☆10Jun 24, 2021Updated 5 years ago
- A library for lattice-based homomorphic encryption in Go☆14May 15, 2022Updated 4 years ago
- ☆32Jan 16, 2025Updated last year
- Accelerating CNN's convolution operation on GPUs by using memory-efficient data access patterns.☆14Dec 8, 2017Updated 8 years ago
- Pie: Programmable LLM Serving☆183Updated this week