mscheong01 / speculative_decoding.c
minimal C implementation of speculative decoding based on llama2.c
☆16Updated 2 months ago
Related projects: ⓘ
- Simple and fast low-bit matmul kernels in CUDA☆48Updated this week
- Breaking Throughput-Latency Trade-off for Long Sequences with Speculative Decoding☆55Updated this week
- Experiments with BitNet inference on CPU☆46Updated 5 months ago
- Repository for Sparse Finetuning of LLMs via modified version of the MosaicML llmfoundry☆36Updated 8 months ago
- My Implementation of Q-Sparse: All Large Language Models can be Fully Sparsely-Activated☆27Updated last month
- Official Implementation of SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blocks☆28Updated 2 months ago
- Repository for CPU Kernel Generation for LLM Inference☆25Updated last year
- Official implementation of the ICLR 2024 paper AffineQuant☆16Updated 5 months ago
- ☆17Updated 5 months ago
- ☆38Updated 9 months ago
- ☆27Updated 3 weeks ago
- Boosting 4-bit inference kernels with 2:4 Sparsity☆47Updated 2 weeks ago
- ☆75Updated this week
- ☆22Updated 8 months ago
- ☆42Updated 3 weeks ago
- Token Omission Via Attention☆118Updated 7 months ago
- PB-LLM: Partially Binarized Large Language Models☆143Updated 10 months ago
- Fast Matrix Multiplications for Lookup Table-Quantized LLMs☆156Updated this week
- Block Transformer: Global-to-Local Language Modeling for Fast Inference (Official Code)☆118Updated 2 weeks ago
- ☆26Updated this week
- Nsight Compute in Docker☆11Updated 9 months ago
- Attention in SRAM on Tenstorrent Grayskull☆22Updated 2 months ago
- RWKV, in easy to read code☆52Updated 6 months ago
- ring-attention experiments☆89Updated 5 months ago
- The source code of our work "Prepacking: A Simple Method for Fast Prefilling and Increased Throughput in Large Language Models"☆57Updated 5 months ago
- QuIP quantization☆41Updated 6 months ago
- ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training Multiplication-Less Reparameterization☆73Updated 3 weeks ago
- [ACL 2024] RelayAttention for Efficient Large Language Model Serving with Long System Prompts☆29Updated 6 months ago
- Code for Palu: Compressing KV-Cache with Low-Rank Projection☆39Updated this week
- This code repository contains the code used for my "Optimizing Memory Usage for Training LLMs and Vision Transformers in PyTorch" blog po…☆84Updated last year