tech-srl / layer_norm_expressivity_role

Related projects ⓘ

Alternatives and complementary repositories for layer_norm_expressivity_role

• RobertCsordas / linear_layer_as_attention
  The official repository for our paper "The Dual Form of Neural Networks Revisited: Connecting Test Time Predictions to Training Patterns …
  ☆16Updated last year
• gregorbachmann / scaling_mlps
  ☆51Updated 5 months ago
• ahennequ / pytorch-custom-mma
  ☆29Updated 2 years ago
• srush / mamba-primer
  ☆35Updated 7 months ago
• VITA-Group / Random-MoE-as-Dropout
  [ICLR 2023] "Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers" by Tianlong Chen*, Zhenyu Zhang*, Ajay Jaiswal…
  ☆44Updated last year
• HazyResearch / prefix-linear-attention
  ☆45Updated 4 months ago
• Optimization-AI / fast_clip
  ☆19Updated last month
• epfml / schedules-and-scaling
  ☆53Updated 3 weeks ago
• dangxingyu / rnn-icrag
  Official repository of paper "RNNs Are Not Transformers (Yet): The Key Bottleneck on In-context Retrieval"
  ☆24Updated 7 months ago
• lucidrains / einops-exts
  Implementation of some personal helper functions for Einops, my most favorite tensor manipulation library ❤️
  ☆52Updated last year
• lucidrains / pause-transformer
  Yet another random morning idea to be quickly tried and architecture shared if it works; to allow the transformer to pause for any amount…
  ☆49Updated last year
• bdusell / stack-attention
  Code for the paper "Stack Attention: Improving the Ability of Transformers to Model Hierarchical Patterns"
  ☆16Updated 8 months ago
• lucidrains / coordinate-descent-attention
  Implementation of an Attention layer where each head can attend to more than just one token, using coordinate descent to pick topk
  ☆46Updated last year
• sustcsonglin / gated_linear_attention_layer
  ☆31Updated 10 months ago
• lucidrains / token-shift-gpt
  Implementation of Token Shift GPT - An autoregressive model that solely relies on shifting the sequence space for mixing
  ☆47Updated 2 years ago
• yikangshen / megablocks
  ☆18Updated 5 months ago
• LAION-AI / Conditional-Pretraining-of-Large-Language-Models
  ☆36Updated last year
• gregorbachmann / Next-Token-Failures
  ☆69Updated 8 months ago
• sjelassi / transformers_ssm_copy
  ☆24Updated 8 months ago
• juvi21 / CoPE-cuda
  Contextual Position Encoding but with some custom CUDA Kernels https://arxiv.org/abs/2405.18719
  ☆20Updated 5 months ago
• VITA-Group / Data-Efficient-Scaling
  [ICML 2023] "Data Efficient Neural Scaling Law via Model Reusing" by Peihao Wang, Rameswar Panda, Zhangyang Wang
  ☆13Updated 10 months ago
• shreyansh26 / Attention-Mask-Patterns
  Using FlexAttention to compute attention with different masking patterns
  ☆40Updated 2 months ago
• BlinkDL / LinearAttentionArena
  Here we will test various linear attention designs.
  ☆56Updated 6 months ago
• lucidrains / kalman-filtering-attention
  Implementation of the Kalman Filtering Attention proposed in "Kalman Filtering Attention for User Behavior Modeling in CTR Prediction"
  ☆57Updated last year
• shoaibahmed / metadata_archaeology
  Official code for the paper: "Metadata Archaeology"
  ☆18Updated last year
• lucidrains / tableformer-pytorch
  Implementation of TableFormer, Robust Transformer Modeling for Table-Text Encoding, in Pytorch
  ☆36Updated 2 years ago
• EleutherAI / rnngineering
  Engineering the state of RNN language models (Mamba, RWKV, etc.)
  ☆32Updated 5 months ago
• yale-nlp / refdpo
  ☆15Updated 4 months ago
• Doraemonzzz / hgru2-pytorch
  ☆21Updated last month