google-deepmind / scaling_laws_for_routing

Related projects: ⓘ

• sustcsonglin / mamba-triton
  ☆42Updated 7 months ago
• felixzli / synthetic_pretraining
  ☆40Updated 2 years ago
• abhishekpanigrahi1996 / transformer_in_transformer
  ☆44Updated 11 months ago
• JeanKaddour / NoTrainNoGain
  Revisiting Efficient Training Algorithms For Transformer-based Language Models (NeurIPS 2023)
  ☆77Updated last year
• princeton-nlp / LM-Kernel-FT
  A Kernel-Based View of Language Model Fine-Tuning https://arxiv.org/abs/2210.05643
  ☆68Updated last year
• shreyansh26 / Attention-Mask-Patterns
  Using FlexAttention to compute attention with different masking patterns
  ☆28Updated last week
• gregorbachmann / Next-Token-Failures
  ☆61Updated 6 months ago
• kamanphoebe / Look-into-MoEs
  A Closer Look into Mixture-of-Experts in Large Language Models
  ☆38Updated last month
• microsoft / EfficientLongSequenceModeling
  ☆48Updated last year
• schwartz-lab-NLP / papa
  Code for the PAPA paper
  ☆27Updated last year
• HazyResearch / prefix-linear-attention
  ☆41Updated 2 months ago
• Noahs-ARK / RFA
  ☆32Updated 3 years ago
• microsoft / SparseMixer
  Sparse Backpropagation for Mixture-of-Expert Training
  ☆17Updated 2 months ago
• frankxu2004 / knnlm-why
  Repo for ICML23 "Why do Nearest Neighbor Language Models Work?"
  ☆56Updated last year
• srush / mamba-primer
  ☆35Updated 5 months ago
• VITA-Group / Random-MoE-as-Dropout
  [ICLR 2023] "Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers" by Tianlong Chen*, Zhenyu Zhang*, Ajay Jaiswal…
  ☆42Updated last year
• ahennequ / pytorch-custom-mma
  ☆29Updated last year
• Edward-Sun / gpt-accelera
  Simple and efficient pytorch-native transformer training and inference (batched)
  ☆53Updated 5 months ago
• google-deepmind / emergent_in_context_learning
  ☆75Updated last month
• microsoft / AutoMoE
  AutoMoE: Neural Architecture Search for Efficient Sparsely Activated Transformers
  ☆40Updated last year
• facebookresearch / RAM
  A framework to study AI models in Reasoning, Alignment, and use of Memory (RAM).
  ☆15Updated last week
• aniquetahir / JORA
  JORA: JAX Tensor-Parallel LoRA Library (ACL 2024)
  ☆28Updated 4 months ago
• dangxingyu / rnn-icrag
  Official repository of paper "RNNs Are Not Transformers (Yet): The Key Bottleneck on In-context Retrieval"
  ☆24Updated 5 months ago
• tech-srl / layer_norm_expressivity_role
  Code for the paper "On the Expressivity Role of LayerNorm in Transformers' Attention" (Findings of ACL'2023)
  ☆43Updated last year
• microsoft / Stochastic-Mixture-of-Experts
  This package implements THOR: Transformer with Stochastic Experts.
  ☆60Updated 2 years ago
• bergen / EdgeTransformer
  ☆21Updated 2 years ago
• tanyuqian / redco
  NAACL '24 (Best Demo Paper RunnerUp) / MlSys @ NeurIPS '23 - RedCoast: A Lightweight Tool to Automate Distributed Training and Inference
  ☆58Updated this week
• princeton-nlp / WhatICLLearns
  [ACL 2023 Findings] What In-Context Learning “Learns” In-Context: Disentangling Task Recognition and Task Learning
  ☆21Updated last year
• srush / mamba-scans
  Blog post
  ☆16Updated 7 months ago
• Shark-NLP / CAB
  ☆28Updated last year