jd-opensource / xllmLinks
A high-performance inference engine for LLMs, optimized for diverse AI accelerators.
☆199Updated this week
Alternatives and similar repositories for xllm
Users that are interested in xllm are comparing it to the libraries listed below
Sorting:
- KV cache store for distributed LLM inference☆321Updated 3 months ago
- High performance Transformer implementation in C++.☆129Updated 7 months ago
- Materials for learning SGLang☆554Updated this week
- Perplexity GPU Kernels☆451Updated last month
- A prefill & decode disaggregated LLM serving framework with shared GPU memory and fine-grained compute isolation.☆105Updated 3 months ago
- A tiny yet powerful LLM inference system tailored for researching purpose. vLLM-equivalent performance with only 2k lines of code (2% of …☆249Updated 2 months ago
- Train speculative decoding models effortlessly and port them smoothly to SGLang serving.☆344Updated last week
- Efficient and easy multi-instance LLM serving☆475Updated 2 weeks ago
- FlexFlow Serve: Low-Latency, High-Performance LLM Serving☆56Updated this week
- ☆288Updated last week
- ☆124Updated 2 months ago
- A fast communication-overlapping library for tensor/expert parallelism on GPUs.☆1,091Updated last week
- GLake: optimizing GPU memory management and IO transmission.☆478Updated 5 months ago
- Disaggregated serving system for Large Language Models (LLMs).☆683Updated 5 months ago
- DeepSeek-V3/R1 inference performance simulator☆165Updated 5 months ago
- A large-scale simulation framework for LLM inference☆428Updated last month
- A lightweight design for computation-communication overlap.☆161Updated this week
- ☆132Updated last month
- Dynamic Memory Management for Serving LLMs without PagedAttention☆410Updated 3 months ago
- Serverless LLM Serving for Everyone.☆531Updated last week
- Since the emergence of chatGPT in 2022, the acceleration of Large Language Model has become increasingly important. Here is a list of pap…☆266Updated 6 months ago
- Repo for SpecEE: Accelerating Large Language Model Inference with Speculative Early Exiting (ISCA25)☆48Updated 4 months ago
- kvcached: Elastic KV cache for dynamic GPU sharing and efficient multi-LLM inference.☆75Updated this week
- Analyze the inference of Large Language Models (LLMs). Analyze aspects like computation, storage, transmission, and hardware roofline mod…☆546Updated 11 months ago
- PyTorch distributed training acceleration framework☆52Updated 3 weeks ago
- RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.☆844Updated last month
- ☆85Updated this week
- Ultra and Unified CCL☆522Updated this week
- A throughput-oriented high-performance serving framework for LLMs☆881Updated 3 weeks ago
- ☆489Updated 3 weeks ago