linxihui / dkernel
☆18Updated last week
Alternatives and similar repositories for dkernel:
Users that are interested in dkernel are comparing it to the libraries listed below
- Vocabulary Parallelism☆17Updated 3 months ago
- Repository for Sparse Finetuning of LLMs via modified version of the MosaicML llmfoundry☆40Updated last year
- The official implementation of paper: SimLayerKV: A Simple Framework for Layer-Level KV Cache Reduction.☆42Updated 4 months ago
- Boosting 4-bit inference kernels with 2:4 Sparsity☆64Updated 5 months ago
- An innovative method expediting LLMs via streamlined semi-autoregressive generation and draft verification.☆24Updated last year
- Odysseus: Playground of LLM Sequence Parallelism☆64Updated 8 months ago
- Ouroboros: Speculative Decoding with Large Model Enhanced Drafting (EMNLP 2024 main)☆84Updated 4 months ago
- Official implementation of the ICML 2024 paper RoSA (Robust Adaptation)☆38Updated last year
- ☆22Updated last year
- 🔥 A minimal training framework for scaling FLA models☆59Updated this week
- PyTorch bindings for CUTLASS grouped GEMM.☆64Updated 3 months ago
- Transformers components but in Triton☆31Updated 3 months ago
- [ICLR2025] Breaking Throughput-Latency Trade-off for Long Sequences with Speculative Decoding☆107Updated 2 months ago
- Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decoding (EMNLP 2023 Long)☆56Updated 4 months ago
- Simple and efficient pytorch-native transformer training and inference (batched)☆68Updated 10 months ago
- ☆111Updated this week
- The source code of our work "Prepacking: A Simple Method for Fast Prefilling and Increased Throughput in Large Language Models"☆59Updated 4 months ago
- The simplest implementation of recent Sparse Attention patterns for efficient LLM inference.☆57Updated 3 weeks ago
- [NeurIPS 2024] Fast Best-of-N Decoding via Speculative Rejection☆39Updated 3 months ago
- [ACL 2024] RelayAttention for Efficient Large Language Model Serving with Long System Prompts☆38Updated 11 months ago
- ☆61Updated 3 weeks ago
- Linear Attention Sequence Parallelism (LASP)☆77Updated 8 months ago
- Repository for CPU Kernel Generation for LLM Inference☆25Updated last year
- ☆49Updated 7 months ago
- Quantized Attention on GPU☆34Updated 2 months ago
- Implementation of the paper: "Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention" from Google in pyTO…☆53Updated last week
- Distributed IO-aware Attention algorithm☆18Updated 5 months ago
- SQUEEZED ATTENTION: Accelerating Long Prompt LLM Inference☆40Updated 3 months ago
- GEAR: An Efficient KV Cache Compression Recipefor Near-Lossless Generative Inference of LLM☆157Updated 7 months ago