Alternatives and similar repositories for Dive-into-Big-Model-Training:

Users that are interested in Dive-into-Big-Model-Training are comparing it to the libraries listed below

• fanshiqing / grouped_gemm
  PyTorch bindings for CUTLASS grouped GEMM.
  ☆111Updated 2 months ago
• RulinShao / LightSeq
  Official repository for LightSeq: Sequence Level Parallelism for Distributed Training of Long Context Transformers
  ☆207Updated 7 months ago
• thu-pacman / FasterMoE
  ☆81Updated 2 years ago
• sail-sg / zero-bubble-pipeline-parallelism
  Zero Bubble Pipeline Parallelism
  ☆375Updated 3 weeks ago
• anyscale / llm-continuous-batching-benchmarks
  ☆116Updated last year
• stanford-futuredata / stk
  ☆102Updated 7 months ago
• feifeibear / Odysseus-Transformer
  Odysseus: Playground of LLM Sequence Parallelism
  ☆68Updated 9 months ago
• FlagOpen / FlagAttention
  A collection of memory efficient attention operators implemented in the Triton language.
  ☆257Updated 9 months ago
• shawntan / scattermoe
  Triton-based implementation of Sparse Mixture of Experts.
  ☆209Updated 4 months ago
• hpcaitech / TensorNVMe
  A Python library transfers PyTorch tensors between CPU and NVMe
  ☆111Updated 4 months ago
• thunlp / Ouroboros
  Ouroboros: Speculative Decoding with Large Model Enhanced Drafting (EMNLP 2024 main)
  ☆90Updated last week
• haochengxi / Train_Transformers_with_INT4
  ☆145Updated last year
• FasterDecoding / REST
  REST: Retrieval-Based Speculative Decoding, NAACL 2024
  ☆198Updated 3 months ago
• cli99 / llm-analysis
  Latency and Memory Analysis of Transformer Models for Training and Inference
  ☆401Updated 3 weeks ago
• pytorch-labs / applied-ai
  Applied AI experiments and examples for PyTorch
  ☆251Updated last week
• feifeibear / long-context-attention
  USP: Unified (a.k.a. Hybrid, 2D) Sequence Parallel Attention for Long Context Transformers Model Training and Inference
  ☆458Updated last week
• foundation-model-stack / fms-fsdp
  🚀 Efficiently (pre)training foundation models with native PyTorch features, including FSDP for training and SDPA implementation of Flash…
  ☆234Updated this week
• smart-lty / ParallelSpeculativeDecoding
  [ICLR 2025] PEARL: Parallel Speculative Decoding with Adaptive Draft Length
  ☆67Updated 2 weeks ago
• kssteven418 / BigLittleDecoder
  [NeurIPS'23] Speculative Decoding with Big Little Decoder
  ☆90Updated last year
• nbasyl / LLM-FP4
  The official implementation of the EMNLP 2023 paper LLM-FP4
  ☆192Updated last year
• tgale96 / grouped_gemm
  PyTorch bindings for CUTLASS grouped GEMM.
  ☆77Updated 5 months ago
• openpsi-project / ReaLHF
  Super-Efficient RLHF Training of LLMs with Parameter Reallocation
  ☆255Updated 2 months ago
• dilab-zju / self-speculative-decoding
  Code associated with the paper **Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding**
  ☆176Updated last month
• zhuzilin / ring-flash-attention
  Ring attention implementation with flash attention
  ☆717Updated last month
• foundation-model-stack / foundation-model-stack
  🚀 Collection of components for development, training, tuning, and inference of foundation models leveraging PyTorch native components.
  ☆189Updated this week
• mit-han-lab / Quest
  [ICML 2024] Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference
  ☆265Updated 4 months ago
• kssteven418 / LTP
  [KDD'22] Learned Token Pruning for Transformers
  ☆96Updated 2 years ago
• Ascend / AscendSpeed
  ☆78Updated last year
• lucidrains / speculative-decoding
  Explorations into some recent techniques surrounding speculative decoding
  ☆250Updated 3 months ago
• facebookresearch / LLM-QAT
  Code repo for the paper "LLM-QAT Data-Free Quantization Aware Training for Large Language Models"
  ☆278Updated 3 weeks ago