Alternatives and similar repositories for An-Empirical-Model-of-Large-Batch-Training

Users that are interested in An-Empirical-Model-of-Large-Batch-Training are comparing it to the libraries listed below

• berlino / seq_icl
  ☆53Updated last year
• PiotrNawrot / sparse-frontier
  The evaluation framework for training-free sparse attention in LLMs
  ☆102Updated 2 weeks ago
• RobertCsordas / moeut
  ☆86Updated last year
• maximzubkov / fft-scan
  Efficient PScan implementation in PyTorch
  ☆16Updated last year
• mnoukhov / async_rlhf
  Code and Configs for Asynchronous RLHF: Faster and More Efficient RL for Language Models
  ☆64Updated 6 months ago
• howard-hou / RWKV-X
  RWKV-X is a Linear Complexity Hybrid Language Model based on the RWKV architecture, integrating Sparse Attention to improve the model's l…
  ☆51Updated 3 months ago
• proger / hippogriff
  Griffin MQA + Hawk Linear RNN Hybrid
  ☆89Updated last year
• BlinkDL / LinearAttentionArena
  Here we will test various linear attention designs.
  ☆61Updated last year
• JonasGeiping / linear_cross_entropy_loss
  A fusion of a linear layer and a cross entropy loss, written for pytorch in triton.
  ☆69Updated last year
• test-time-training / ttt-tk
  ☆41Updated 2 weeks ago
• nikhilvyas / SOAP_MUON
  Combining SOAP and MUON
  ☆16Updated 8 months ago
• nanowell / Q-Sparse-LLM
  My Implementation of Q-Sparse: All Large Language Models can be Fully Sparsely-Activated
  ☆33Updated last year
• kyleliang919 / Online-Subspace-Descent
  [NeurIPS 2024] Low rank memory efficient optimizer without SVD
  ☆30Updated 4 months ago
• fal-ai-community / nano-mdm
  Tiny re-implementation of MDM in style of LLaDA and nano-gpt speedrun
  ☆56Updated 7 months ago
• CLAIRE-Labo / StructuredFFN
  The official code of "Building on Efficient Foundations: Effectively Training LLMs with Structured Feedforward Layers"
  ☆19Updated last year
• samsja / muon_fsdp_2
  Muon fsdp 2
  ☆44Updated 2 months ago
• NX-AI / mlstm_kernels
  Tiled Flash Linear Attention library for fast and efficient mLSTM Kernels.
  ☆72Updated 2 weeks ago
• sustcsonglin / mamba-triton
  ☆48Updated last year
• lucidrains / PEER-pytorch
  Pytorch implementation of the PEER block from the paper, Mixture of A Million Experts, by Xu Owen He at Deepmind
  ☆129Updated last year
• HazyResearch / prefix-linear-attention
  ☆56Updated last year
• erogol / BlaGPT
  Experimental playground for benchmarking language model (LM) architectures, layers, and tricks on smaller datasets. Designed for flexible…
  ☆84Updated 2 weeks ago
• Jaykef / Triton-nanoGPT
  Custom triton kernels for training Karpathy's nanoGPT.
  ☆19Updated last year
• OpenNLPLab / HGRN
  [NeurIPS 2023 spotlight] Official implementation of HGRN in our NeurIPS 2023 paper - Hierarchically Gated Recurrent Neural Network for Se…
  ☆65Updated last year
• jzhang38 / LongMamba
  Some preliminary explorations of Mamba's context scaling.
  ☆216Updated last year
• LIONS-EPFL / scion
  ☆45Updated last week
• TianjinYellow / SPAM-Optimizer
  ☆34Updated 7 months ago
• jopetty / word-problem
  Experiments on the impact of depth in transformers and SSMs.
  ☆36Updated last week
• amazon-science / mxfp4-llm
  Official implementation for Training LLMs with MXFP4
  ☆101Updated 6 months ago
• ethansmith2000 / TransformerExperiments
  ☆19Updated 5 months ago
• itsnamgyu / block-transformer
  Block Transformer: Global-to-Local Language Modeling for Fast Inference (NeurIPS 2024)
  ☆162Updated 6 months ago