lucidrains / st-moe-pytorch

Related projects ⓘ

Alternatives and complementary repositories for st-moe-pytorch

• lucidrains / soft-moe-pytorch
  Implementation of Soft MoE, proposed by Brain's Vision team, in Pytorch
  ☆246Updated 6 months ago
• lucidrains / ring-attention-pytorch
  Implementation of 💍 Ring Attention, from Liu et al. at Berkeley AI, in Pytorch
  ☆476Updated 3 weeks ago
• lucidrains / mixture-of-experts
  A Pytorch implementation of Sparsely-Gated Mixture of Experts, for massively increasing the parameter count of language models
  ☆642Updated last year
• HazyResearch / zoology
  Understand and test language model architectures on synthetic tasks.
  ☆162Updated 6 months ago
• lucidrains / CoLT5-attention
  Implementation of the conditionally routed attention in the CoLT5 architecture, in Pytorch
  ☆225Updated 2 months ago
• pytorch-labs / attention-gym
  Helpful tools and examples for working with flex-attention
  ☆469Updated 3 weeks ago
• lucidrains / recurrent-memory-transformer-pytorch
  Implementation of Recurrent Memory Transformer, Neurips 2022 paper, in Pytorch
  ☆394Updated this week
• haoliuhl / ringattention
  Transformers with Arbitrarily Large Context
  ☆641Updated 3 months ago
• epfml / dynamic-sparse-flash-attention
  ☆132Updated last year
• jzhang38 / LongMamba
  Some preliminary explorations of Mamba's context scaling.
  ☆191Updated 9 months ago
• HazyResearch / based
  Code for exploring Based models from "Simple linear attention language models balance the recall-throughput tradeoff"
  ☆214Updated 3 months ago
• huggingface / datablations
  Scaling Data-Constrained Language Models
  ☆321Updated last month
• llm-random / llm-random
  ☆175Updated this week
• lucidrains / simple-hierarchical-transformer
  Experiments around a simple idea for inducing multiple hierarchical predictive model within a GPT
  ☆205Updated 3 months ago
• google-research / vmoe
  ☆574Updated last week
• lucidrains / infini-transformer-pytorch
  Implementation of Infini-Transformer in Pytorch
  ☆104Updated last month
• facebookresearch / mega
  Sequence modeling with Mega.
  ☆298Updated last year
• for-ai / parameter-efficient-moe
  ☆247Updated last year
• HazyResearch / m2
  Repo for "Monarch Mixer: A Simple Sub-Quadratic GEMM-Based Architecture"
  ☆537Updated 6 months ago
• yuhuixu1993 / qa-lora
  Official PyTorch implementation of QA-LoRA
  ☆117Updated 8 months ago
• lucidrains / rotary-embedding-torch
  Implementation of Rotary Embeddings, from the Roformer paper, in Pytorch
  ☆571Updated last week
• zyushun / Adam-mini
  Code for Adam-mini: Use Fewer Learning Rates To Gain More https://arxiv.org/abs/2406.16793
  ☆328Updated 3 weeks ago
• booydar / LM-RMT
  Recurrent Memory Transformer
  ☆149Updated last year
• microsoft / Tutel
  Tutel MoE: An Optimized Mixture-of-Experts Implementation
  ☆735Updated this week
• lucidrains / CALM-pytorch
  Implementation of CALM from the paper "LLM Augmented LLMs: Expanding Capabilities through Composition", out of Google Deepmind
  ☆169Updated 2 months ago
• ofirpress / attention_with_linear_biases
  Code for the ALiBi method for transformer language models (ICLR 2022)
  ☆507Updated last year
• srush / annotated-mamba
  Annotated version of the Mamba paper
  ☆457Updated 8 months ago
• fkodom / grouped-query-attention-pytorch
  (Unofficial) PyTorch implementation of grouped-query attention (GQA) from "GQA: Training Generalized Multi-Query Transformer Models from …
  ☆133Updated 6 months ago
• HazyResearch / flash-fft-conv
  FlashFFTConv: Efficient Convolutions for Long Sequences with Tensor Cores
  ☆281Updated last month
• mlfoundations / open_lm
  A repository for research on medium sized language models.
  ☆479Updated this week