Alternatives and similar repositories for Tutel

Users that are interested in Tutel are comparing it to the libraries listed below

• Azure / MS-AMP
  Microsoft Automatic Mixed Precision Library
  ☆619Updated 11 months ago
• haoliuhl / ringattention
  Large Context Attention
  ☆736Updated 7 months ago
• zhuzilin / ring-flash-attention
  Ring attention implementation with flash attention
  ☆866Updated this week
• pytorch / PiPPy
  Pipeline Parallelism for PyTorch
  ☆779Updated last year
• laekov / fastmoe
  A fast MoE impl for PyTorch
  ☆1,782Updated 7 months ago
• codecaution / Awesome-Mixture-of-Experts-Papers
  A curated reading list of research in Mixture-of-Experts(MoE).
  ☆642Updated 10 months ago
• sail-sg / zero-bubble-pipeline-parallelism
  Zero Bubble Pipeline Parallelism
  ☆426Updated 4 months ago
• lucidrains / mixture-of-experts
  A Pytorch implementation of Sparsely-Gated Mixture of Experts, for massively increasing the parameter count of language models
  ☆808Updated 2 years ago
• google-research / vmoe
  ☆671Updated last month
• mit-han-lab / smoothquant
  [ICML 2023] SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
  ☆1,496Updated last year
• feifeibear / long-context-attention
  USP: Unified (a.k.a. Hybrid, 2D) Sequence Parallel Attention for Long Context Transformers Model Training and Inference
  ☆562Updated last week
• lucidrains / ring-attention-pytorch
  Implementation of 💍 Ring Attention, from Liu et al. at Berkeley AI, in Pytorch
  ☆537Updated 4 months ago
• meta-pytorch / attention-gym
  Helpful tools and examples for working with flex-attention
  ☆970Updated this week
• hao-ai-lab / LookaheadDecoding
  [ICML 2024] Break the Sequential Dependency of LLM Inference Using Lookahead Decoding
  ☆1,276Updated 6 months ago
• bigscience-workshop / Megatron-DeepSpeed
  Ongoing research training transformer language models at scale, including: BERT & GPT-2
  ☆1,415Updated last year
• FMInference / H2O
  [NeurIPS'23] H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models.
  ☆473Updated last year
• FMInference / DejaVu
  ☆335Updated last year
• deepspeedai / Megatron-DeepSpeed
  Ongoing research training transformer language models at scale, including: BERT & GPT-2
  ☆2,159Updated last month
• cli99 / llm-analysis
  Latency and Memory Analysis of Transformer Models for Training and Inference
  ☆454Updated 4 months ago
• volcengine / veScale
  A PyTorch Native LLM Training Framework
  ☆863Updated 2 months ago
• RulinShao / LightSeq
  Official repository for DistFlashAttn: Distributed Memory-efficient Attention for Long-context LLMs Training
  ☆216Updated last year
• alibaba / Megatron-LLaMA
  Best practice for training LLaMA models in Megatron-LM
  ☆661Updated last year
• BlackSamorez / tensor_parallel
  Automatically split your PyTorch models on multiple GPUs for training & inference
  ☆657Updated last year
• feifeibear / LLMSpeculativeSampling
  Fast inference from large lauguage models via speculative decoding
  ☆815Updated last year
• fla-org / native-sparse-attention
  🐳 Efficient Triton implementations for "Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention"
  ☆847Updated 5 months ago
• NVIDIA / NeMo-Aligner
  Scalable toolkit for efficient model alignment
  ☆838Updated last month
• facebookresearch / LLM-QAT
  Code repo for the paper "LLM-QAT Data-Free Quantization Aware Training for Large Language Models"
  ☆312Updated 6 months ago
• XueFuzhao / awesome-mixture-of-experts
  A collection of AWESOME things about mixture-of-experts
  ☆1,202Updated 9 months ago
• fanshiqing / grouped_gemm
  PyTorch bindings for CUTLASS grouped GEMM.
  ☆145Updated 2 weeks ago
• facebookresearch / HolisticTraceAnalysis
  A library to analyze PyTorch traces.
  ☆409Updated this week