qhliu26 / Dive-into-Big-Model-Training
📑 Dive into Big Model Training
☆111Updated 2 years ago
Alternatives and similar repositories for Dive-into-Big-Model-Training:
Users that are interested in Dive-into-Big-Model-Training are comparing it to the libraries listed below
- Official repository for DistFlashAttn: Distributed Memory-efficient Attention for Long-context LLMs Training☆209Updated 8 months ago
- ☆82Updated 3 years ago
- Triton-based implementation of Sparse Mixture of Experts.☆210Updated 4 months ago
- PyTorch bindings for CUTLASS grouped GEMM.☆120Updated 3 months ago
- A collection of memory efficient attention operators implemented in the Triton language.☆262Updated 10 months ago
- Odysseus: Playground of LLM Sequence Parallelism☆68Updated 10 months ago
- ☆103Updated 8 months ago
- [ICLR 2025] PEARL: Parallel Speculative Decoding with Adaptive Draft Length☆77Updated last week
- Latency and Memory Analysis of Transformer Models for Training and Inference☆403Updated last week
- ☆117Updated last year
- Zero Bubble Pipeline Parallelism☆383Updated 2 weeks ago
- A high-performance distributed deep learning system targeting large-scale and automated distributed training. If you have any interests, …☆110Updated last year
- Code associated with the paper **Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding**☆183Updated 2 months ago
- ☆147Updated last year
- A Python library transfers PyTorch tensors between CPU and NVMe☆113Updated 4 months ago
- REST: Retrieval-Based Speculative Decoding, NAACL 2024☆199Updated 4 months ago
- 🚀 Collection of components for development, training, tuning, and inference of foundation models leveraging PyTorch native components.☆192Updated this week
- USP: Unified (a.k.a. Hybrid, 2D) Sequence Parallel Attention for Long Context Transformers Model Training and Inference☆477Updated this week
- PyTorch bindings for CUTLASS grouped GEMM.☆81Updated 5 months ago
- Scalable PaLM implementation of PyTorch☆190Updated 2 years ago
- Sequence-level 1F1B schedule for LLMs.☆17Updated 10 months ago
- ☆313Updated last year
- [MLSys'24] Atom: Low-bit Quantization for Efficient and Accurate LLM Serving☆305Updated 9 months ago
- [ICLR 2025] COAT: Compressing Optimizer States and Activation for Memory-Efficient FP8 Training☆184Updated last week
- ☆82Updated last month
- [NeurIPS'23] Speculative Decoding with Big Little Decoder☆90Updated last year
- A simple calculation for LLM MFU.☆36Updated last month
- ☆143Updated last year
- ☆78Updated last year
- Triton implementation of FlashAttention2 that adds Custom Masks.☆109Updated 8 months ago