ZixuanJiang / pre-rmsnorm-transformer

Related projects: ⓘ

• graphcore-research / out-of-the-box-fp8-training
  Demo of the unit_scaling library, showing how a model can be easily adapted to train in FP8.
  ☆34Updated 2 months ago
• mobiusml / gemlite
  Simple and fast low-bit matmul kernels in CUDA
  ☆48Updated this week
• facebookresearch / projUNN
  Fast training of unitary deep network layers from low-rank updates
  ☆28Updated last year
• srush / triton-autodiff
  Experiment of using Tangent to autodiff triton
  ☆66Updated 7 months ago
• alexzhang13 / Triton-Puzzles-Solutions
  Personal solutions to the Triton Puzzles
  ☆11Updated 2 months ago
• CerebrasResearch / Sparse-IFT
  Official repository of Sparse ISO-FLOP Transformations for Maximizing Training Efficiency
  ☆23Updated last month
• HazyResearch / fly
  ☆187Updated last year
• stas00 / ml-ways
  ML/DL Math and Method notes
  ☆56Updated 9 months ago
• DS3Lab / CocktailSGD
  ☆24Updated last year
• spcl / sten
  Sparsity support for PyTorch
  ☆31Updated 11 months ago
• shikaiqiu / compute-better-spent
  ☆42Updated 3 months ago
• google-deepmind / tf2jax
  ☆104Updated last week
• sholtodouglas / scalingExperiments
  ☆56Updated 2 years ago
• HazyResearch / butterfly
  Butterfly matrix multiplication in PyTorch
  ☆160Updated 11 months ago
• facebookresearch / LAWT
  Code for papers Linear Algebra with Transformers (TMLR) and What is my Math Transformer Doing? (AI for Maths Workshop, Neurips 2022)
  ☆61Updated last month
• google-research / jaxpruner
  ☆202Updated 4 months ago
• IST-DASLab / MicroAdam
  This repository contains code for the MicroAdam paper.
  ☆9Updated 2 months ago
• athms / mad-lab
  A MAD laboratory to improve AI architecture designs 🧪
  ☆84Updated 4 months ago
• lucaslingle / mu_transformer
  Transformer with Mu-Parameterization, implemented in Jax/Flax. Supports FSDP on TPU pods.
  ☆29Updated 3 weeks ago
• Dao-AILab / fast-hadamard-transform
  Fast Hadamard transform in CUDA, with a PyTorch interface
  ☆87Updated 3 months ago
• epfml / dynamic-sparse-flash-attention
  ☆129Updated last year
• lucidrains / taylor-series-linear-attention
  Explorations into the recently proposed Taylor Series Linear Attention
  ☆85Updated last month
• facebookresearch / MODel_opt
  Memory Optimizations for Deep Learning (ICML 2023)
  ☆58Updated 6 months ago
• insuhan / hyper-attn
  ☆65Updated 9 months ago
• HazyResearch / nanoGPT-TK
  The simplest, fastest repository for training/finetuning medium-sized GPTs. Now, with kittens!
  ☆45Updated last month
• lucidrains / mixture-of-attention
  Some personal experiments around routing tokens to different autoregressive attention, akin to mixture-of-experts
  ☆101Updated last year
• Zyphra / tree_attention
  Tree Attention: Topology-aware Decoding for Long-Context Attention on GPU clusters
  ☆94Updated 2 weeks ago
• young-geng / scalax
  A simple library for scaling up JAX programs
  ☆116Updated last month
• AllanYangZhou / universal_neural_functional
  ☆46Updated 7 months ago
• wdlctc / mini-s
  ☆26Updated this week