vllm-project / vllm-nccl
Manages vllm-nccl dependency
☆17Updated 5 months ago
Related projects ⓘ
Alternatives and complementary repositories for vllm-nccl
- Efficient, Flexible, and Highly Fault-Tolerant Model Service Management Based on SGLang☆22Updated this week
- ☆41Updated 6 months ago
- Elixir: Train a Large Language Model on a Small GPU Cluster☆13Updated last year
- [ACL 2024] RelayAttention for Efficient Large Language Model Serving with Long System Prompts☆34Updated 8 months ago
- Ouroboros: Speculative Decoding with Large Model Enhanced Drafting (EMNLP 2024 main)☆75Updated 3 weeks ago
- Odysseus: Playground of LLM Sequence Parallelism☆53Updated 4 months ago
- A MoE impl for PyTorch, [ATC'23] SmartMoE☆57Updated last year
- Distributed IO-aware Attention algorithm☆17Updated 2 months ago
- Decoding Attention is specially optimized for multi head attention (MHA) using CUDA core for the decoding stage of LLM inference.☆23Updated this week
- ☆13Updated last year
- Fast LLM Training CodeBase With dynamic strategy choosing [Deepspeed+Megatron+FlashAttention+CudaFusionKernel+Compiler];☆34Updated 10 months ago
- ☆63Updated 3 months ago
- Fast and memory-efficient exact attention☆28Updated 2 weeks ago
- Summary of system papers/frameworks/codes/tools on training or serving large model☆56Updated 10 months ago
- The official code for paper "parallel speculative decoding with adaptive draft length."☆23Updated 2 months ago
- A more efficient GLM implementation!☆55Updated last year
- Efficient Retrieval with Learned Similarities☆13Updated 3 months ago
- Transformers components but in Triton☆24Updated this week
- Boosting 4-bit inference kernels with 2:4 Sparsity☆51Updated 2 months ago
- The Official Implementation of Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference☆35Updated this week
- The official implementation of paper: SimLayerKV: A Simple Framework for Layer-Level KV Cache Reduction.☆35Updated 3 weeks ago
- Sequence-level 1F1B schedule for LLMs.☆17Updated 5 months ago
- TensorRT LLM Benchmark Configuration☆11Updated 3 months ago
- This is a personal reimplementation of Google's Infini-transformer, utilizing a small 2b model. The project includes both model and train…☆52Updated 6 months ago
- Leveraging passage embeddings for efficient listwise reranking with large language models.☆32Updated last month
- SELF-GUIDE: Better Task-Specific Instruction Following via Self-Synthetic Finetuning. COLM 2024 Accepted Paper☆25Updated 5 months ago
- ☆74Updated 10 months ago
- Official implementation of ICML 2024 paper "ExCP: Extreme LLM Checkpoint Compression via Weight-Momentum Joint Shrinking".☆40Updated 3 months ago
- Compare different hardware platforms via the Roofline Model for LLM inference tasks.☆74Updated 7 months ago
- GPTQ inference TVM kernel☆35Updated 6 months ago