Alternatives and similar repositories for transformer-tricks:

Users that are interested in transformer-tricks are comparing it to the libraries listed below

• fla-org / flame
  🔥 A minimal training framework for scaling FLA models
  ☆94Updated last week
• NVIDIA / Star-Attention
  Efficient LLM Inference over Long Sequences
  ☆366Updated last month
• JT-Ushio / MHA2MLA
  Towards Economical Inference: Enabling DeepSeek's Multi-Head Latent Attention in Any Transformer-based LLMs
  ☆158Updated this week
• sustcsonglin / linear-attention-and-beyond-slides
  ☆67Updated last month
• Zyphra / tree_attention
  Tree Attention: Topology-aware Decoding for Long-Context Attention on GPU clusters
  ☆125Updated 4 months ago
• XunhaoLai / native-sparse-attention-triton
  Efficient triton implementation of Native Sparse Attention.
  ☆134Updated last week
• NVIDIA / ngpt
  Normalized Transformer (nGPT)
  ☆166Updated 4 months ago
• huggingface / kernels
  Load compute kernels from the Hub
  ☆113Updated this week
• OpenNLPLab / lightning-attention
  Lightning Attention-2: A Free Lunch for Handling Unlimited Sequence Lengths in Large Language Models
  ☆277Updated last month
• NVlabs / GatedDeltaNet
  [ICLR 2025] Official PyTorch Implementation of Gated Delta Networks: Improving Mamba2 with Delta Rule
  ☆147Updated 3 weeks ago
• joey00072 / nanoGRPO
  nanoGRPO is a lightweight implementation of Group Relative Policy Optimization (GRPO)
  ☆96Updated last week
• xiayuqing0622 / flex_head_fa
  Fast and memory-efficient exact attention
  ☆67Updated last month
• itsnamgyu / block-transformer
  Block Transformer: Global-to-Local Language Modeling for Fast Inference (NeurIPS 2024)
  ☆151Updated 3 months ago
• PiotrNawrot / nano-sparse-attention
  The simplest implementation of recent Sparse Attention patterns for efficient LLM inference.
  ☆59Updated 2 months ago
• facebookresearch / memory
  Memory layers use a trainable key-value lookup mechanism to add extra parameters to a model without increasing FLOPs. Conceptually, spars…
  ☆313Updated 3 months ago
• mit-han-lab / x-attention
  XAttention: Block Sparse Attention with Antidiagonal Scoring
  ☆134Updated last week
• wdlctc / headinfer
  ☆41Updated last week
• kyleliang919 / C-Optim
  When it comes to optimizers, it's always better to be safe than sorry
  ☆216Updated last week
• Zyphra / Zamba2
  PyTorch implementation of models from the Zamba2 series.
  ☆178Updated 2 months ago
• kyleliang919 / Super_Muon
  ☆50Updated 2 weeks ago
• ZihanWang314 / CoE
  Chain of Experts (CoE) enables communication between experts within Mixture-of-Experts (MoE) models
  ☆154Updated 3 weeks ago
• opengear-project / GEAR
  GEAR: An Efficient KV Cache Compression Recipefor Near-Lossless Generative Inference of LLM
  ☆158Updated 8 months ago
• dvruette / gidd
  Code accompanying the paper "Generalized Interpolating Discrete Diffusion"
  ☆70Updated 3 weeks ago
• HazyResearch / lolcats
  Repo for "LoLCATs: On Low-Rank Linearizing of Large Language Models"
  ☆229Updated 2 months ago
• thu-ml / low-bit-optimizers
  Low-bit optimizers for PyTorch
  ☆126Updated last year
• VITA-Group / Q-GaLore
  Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients.
  ☆195Updated 8 months ago
• BlinkDL / modded-nanogpt-rwkv
  RWKV-7: Surpassing GPT
  ☆82Updated 4 months ago
• HanGuo97 / flute
  Fast Matrix Multiplications for Lookup Table-Quantized LLMs
  ☆236Updated last month
• SkyworkAI / Skywork-MoE
  Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models
  ☆130Updated 9 months ago
• lucidrains / nGPT-pytorch
  Quick implementation of nGPT, learning entirely on the hypersphere, from NvidiaAI
  ☆277Updated 3 weeks ago