Alternatives and similar repositories for DifferentialTransformer

Users that are interested in DifferentialTransformer are comparing it to the libraries listed below

• kyegomez / Griffin
  Implementation of Griffin from the paper: "Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models"
  ☆55Updated 2 months ago
• NX-AI / mlstm_kernels
  Tiled Flash Linear Attention library for fast and efficient mLSTM Kernels.
  ☆57Updated last month
• knotgrass / attention
  several types of attention modules written in PyTorch for learning purposes
  ☆52Updated 8 months ago
• lucidrains / agent-attention-pytorch
  Implementation of Agent Attention in Pytorch
  ☆90Updated 11 months ago
• lucidrains / adam-atan2-pytorch
  Implementation of the proposed Adam-atan2 from Google Deepmind in Pytorch
  ☆108Updated 6 months ago
• lucidrains / kalman-filtering-attention
  Implementation of the Kalman Filtering Attention proposed in "Kalman Filtering Attention for User Behavior Modeling in CTR Prediction"
  ☆58Updated last year
• SeunghyunSEO / optimized_hf_llama_class_for_training
  ☆47Updated 9 months ago
• lucidrains / light-recurrent-unit-pytorch
  Implementation of a Light Recurrent Unit in Pytorch
  ☆48Updated 8 months ago
• lucidrains / infini-transformer-pytorch
  Implementation of Infini-Transformer in Pytorch
  ☆111Updated 5 months ago
• ambisinister / mla-experiments
  Experiments on Multi-Head Latent Attention
  ☆92Updated 10 months ago
• lucidrains / pytorch-custom-utils
  Just some miscellaneous utility functions / decorators / modules related to Pytorch and Accelerate to help speed up implementation of new…
  ☆122Updated 10 months ago
• lucidrains / hyper-connections
  Attempt to make multiple residual streams from Bytedance's Hyper-Connections paper accessible to the public
  ☆84Updated last week
• google-deepmind / randomized_positional_encodings
  Randomized Positional Encodings Boost Length Generalization of Transformers
  ☆81Updated last year
• YihongDong / FANformer
  ☆29Updated 3 weeks ago
• lucidrains / grokfast-pytorch
  Explorations into the proposal from the paper "Grokfast, Accelerated Grokking by Amplifying Slow Gradients"
  ☆101Updated 6 months ago
• piotrpiekos / MoSA
  User-friendly implementation of the Mixture-of-Sparse-Attention (MoSA). MoSA selects distinct tokens for each head with expert choice rou…
  ☆21Updated last month
• andrewgcodes / xlstm
  my attempts at implementing various bits of Sepp Hochreiter's new xLSTM architecture
  ☆130Updated last year
• OpenNLPLab / HGRN
  [NeurIPS 2023 spotlight] Official implementation of HGRN in our NeurIPS 2023 paper - Hierarchically Gated Recurrent Neural Network for Se…
  ☆65Updated last year
• myscience / x-lstm
  Pytorch implementation of the xLSTM model by Beck et al. (2024)
  ☆167Updated 10 months ago
• kyegomez / DifferentialTransformer
  An open source community implementation of the model from "DIFFERENTIAL TRANSFORMER" paper by Microsoft.
  ☆30Updated this week
• lucidrains / gateloop-transformer
  Implementation of GateLoop Transformer in Pytorch and Jax
  ☆89Updated last year
• AI-Guru / helibrunna
  A HuggingFace compatible Small Language Model trainer.
  ☆75Updated 4 months ago
• nanowell / Differential-Transformer-PyTorch
  PyTorch implementation of the Differential-Transformer architecture for sequence modeling, specifically tailored as a decoder-only model …
  ☆70Updated 7 months ago
• proger / hippogriff
  Griffin MQA + Hawk Linear RNN Hybrid
  ☆87Updated last year
• myscience / mamba
  Pytorch (Lightning) implementation of the Mamba model
  ☆29Updated 2 months ago
• Doraemonzzz / hgru-pytorch
  ☆27Updated 11 months ago
• expz / annotated-hyena
  An annotated implementation of the Hyena Hierarchy paper
  ☆33Updated 2 years ago
• epfml / DenseFormer
  ☆81Updated last year
• srush / mamba-primer
  ☆37Updated last year
• lucidrains / mixture-of-attention
  Some personal experiments around routing tokens to different autoregressive attention, akin to mixture-of-experts
  ☆119Updated 8 months ago