taehokim20 / LLMemLinks
LLMem: GPU Memory Estimation for Fine-Tuning Pre-Trained LLMs
☆22Updated 3 months ago
Alternatives and similar repositories for LLMem
Users that are interested in LLMem are comparing it to the libraries listed below
Sorting:
- Ouroboros: Speculative Decoding with Large Model Enhanced Drafting (EMNLP 2024 main)☆109Updated 5 months ago
- Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models☆136Updated last year
- [ICLR2025] Breaking Throughput-Latency Trade-off for Long Sequences with Speculative Decoding☆125Updated 8 months ago
- Efficient, Flexible, and Highly Fault-Tolerant Model Service Management Based on SGLang☆56Updated 9 months ago
- Multi-Candidate Speculative Decoding☆36Updated last year
- Fast LLM Training CodeBase With dynamic strategy choosing [Deepspeed+Megatron+FlashAttention+CudaFusionKernel+Compiler];☆41Updated last year
- GEAR: An Efficient KV Cache Compression Recipefor Near-Lossless Generative Inference of LLM☆165Updated last year
- KV cache compression for high-throughput LLM inference☆136Updated 6 months ago
- Unofficial implementation for the paper "Mixture-of-Depths: Dynamically allocating compute in transformer-based language models"☆170Updated last year
- [NeurIPS 2024] The official implementation of "Kangaroo: Lossless Self-Speculative Decoding for Accelerating LLMs via Double Early Exitin…☆59Updated last year
- Boosting 4-bit inference kernels with 2:4 Sparsity☆81Updated 11 months ago
- Simple extension on vLLM to help you speed up reasoning model without training.☆181Updated 3 months ago
- ☆41Updated 4 months ago
- REST: Retrieval-Based Speculative Decoding, NAACL 2024☆207Updated 9 months ago
- [NeurIPS 2024] Fast Best-of-N Decoding via Speculative Rejection☆51Updated 10 months ago
- An innovative method expediting LLMs via streamlined semi-autoregressive generation and draft verification.☆26Updated 4 months ago
- A lightweight reinforcement learning framework that integrates seamlessly into your codebase, empowering developers to focus on algorithm…☆61Updated last week
- ☆77Updated 10 months ago
- Repository of LV-Eval Benchmark☆70Updated last year
- Implementation of NAACL 2024 Outstanding Paper "LM-Infinite: Simple On-the-Fly Length Generalization for Large Language Models"☆148Updated 5 months ago
- [ICLR 2025] PEARL: Parallel Speculative Decoding with Adaptive Draft Length☆108Updated 4 months ago
- Inferflow is an efficient and highly configurable inference engine for large language models (LLMs).☆247Updated last year
- The official implementation of paper: SimLayerKV: A Simple Framework for Layer-Level KV Cache Reduction.☆48Updated 10 months ago
- Summary of system papers/frameworks/codes/tools on training or serving large model☆57Updated last year
- Modular and structured prompt caching for low-latency LLM inference☆99Updated 9 months ago
- ☆20Updated 4 months ago
- ☆117Updated 2 months ago
- Due to the huge vocaburary size (151,936) of Qwen models, the Embedding and LM Head weights are excessively heavy. Therefore, this projec…☆26Updated last year
- Layer-Condensed KV cache w/ 10 times larger batch size, fewer params and less computation. Dramatic speed up with better task performance…☆153Updated 4 months ago
- Implementation for FP8/INT8 Rollout for RL training without performence drop.☆184Updated this week