ModelEngine-Group / unified-cache-managementLinks
Persist and reuse KV Cache to speedup your LLM.
☆158Updated this week
Alternatives and similar repositories for unified-cache-management
Users that are interested in unified-cache-management are comparing it to the libraries listed below
Sorting:
- Efficient and easy multi-instance LLM serving☆517Updated 3 months ago
- GLake: optimizing GPU memory management and IO transmission.☆491Updated 8 months ago
- KV cache store for distributed LLM inference☆371Updated last month
- Disaggregated serving system for Large Language Models (LLMs).☆747Updated 8 months ago
- A prefill & decode disaggregated LLM serving framework with shared GPU memory and fine-grained compute isolation.☆118Updated 6 months ago
- DeepSeek-V3/R1 inference performance simulator☆169Updated 8 months ago
- ☆73Updated last year
- Research prototype of PRISM — a cost-efficient multi-LLM serving system with flexible time- and space-based GPU sharing.☆49Updated 3 months ago
- Virtualized Elastic KV Cache for Dynamic GPU Sharing and Beyond☆701Updated 2 weeks ago
- Offline optimization of your disaggregated Dynamo graph☆121Updated this week
- ☆515Updated 3 weeks ago
- NVIDIA NCCL Tests for Distributed Training☆129Updated this week
- SpotServe: Serving Generative Large Language Models on Preemptible Instances☆133Updated last year
- OME is a Kubernetes operator for enterprise-grade management and serving of Large Language Models (LLMs)☆324Updated last week
- High performance Transformer implementation in C++.☆143Updated 10 months ago
- Hooked CUDA-related dynamic libraries by using automated code generation tools.☆172Updated 2 years ago
- FlagScale is a large model toolkit based on open-sourced projects.☆421Updated last week
- This repository organizes materials, recordings, and schedules related to AI-infra learning meetings.☆262Updated last week
- ☆75Updated last year
- AI Accelerator Benchmark focuses on evaluating AI Accelerators from a practical production perspective, including the ease of use and ver…☆282Updated 3 months ago
- A workload for deploying LLM inference services on Kubernetes☆123Updated last week
- ☆127Updated this week
- ☆328Updated last month
- Analyze the inference of Large Language Models (LLMs). Analyze aspects like computation, storage, transmission, and hardware roofline mod…☆594Updated last year
- TePDist (TEnsor Program DISTributed) is an HLO-level automatic distributed system for DL models.☆98Updated 2 years ago
- NVIDIA Inference Xfer Library (NIXL)☆753Updated this week
- Materials for learning SGLang☆682Updated last week
- A low-latency & high-throughput serving engine for LLMs☆454Updated last month
- Stateful LLM Serving☆89Updated 9 months ago
- Since the emergence of chatGPT in 2022, the acceleration of Large Language Model has become increasingly important. Here is a list of pap…☆282Updated 9 months ago