OpenNLPLab / Transnormer

Related projects: ⓘ

• OpenNLPLab / Tnn
  [ICLR 2023] Official implementation of Transnormer in our ICLR 2023 paper - Toeplitz Neural Network for Sequence Modeling
  ☆70Updated 4 months ago
• berlino / gated_linear_attention
  ☆94Updated 6 months ago
• WailordHe / DenseSSM
  A repository for DenseSSMs
  ☆86Updated 5 months ago
• HKUNLP / efficient-attention
  [EVA ICLR'23; LARA ICML'22] Efficient attention mechanisms via control variates, random features, and importance sampling
  ☆78Updated last year
• VITA-Group / Random-MoE-as-Dropout
  [ICLR 2023] "Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers" by Tianlong Chen*, Zhenyu Zhang*, Ajay Jaiswal…
  ☆42Updated last year
• bojone / FSQ
  Keras implement of Finite Scalar Quantization
  ☆58Updated 10 months ago
• YuchuanTian / DiJiang
  [ICML'24 Oral] The official code of "DiJiang: Efficient Large Language Models through Compact Kernelization", a novel DCT-based linear at…
  ☆95Updated 3 months ago
• BaohaoLiao / mefts
  [NeurIPS2023] Make Your Pre-trained Model Reversible: From Parameter to Memory Efficient Fine-Tuning
  ☆28Updated last year
• OpenNLPLab / HGRN
  [NeurIPS 2023 spotlight] Official implementation of HGRN in our NeurIPS 2023 paper - Hierarchically Gated Recurrent Neural Network for Se…
  ☆60Updated 4 months ago
• HazyResearch / prefix-linear-attention
  ☆41Updated 2 months ago
• DRSY / EMO
  [ICLR 2024]EMO: Earth Mover Distance Optimization for Auto-Regressive Language Modeling(https://arxiv.org/abs/2310.04691)
  ☆111Updated 6 months ago
• ag1988 / top_k_attention
  The accompanying code for "Memory-efficient Transformers via Top-k Attention" (Ankit Gupta, Guy Dar, Shaya Goodman, David Ciprut, Jonatha…
  ☆58Updated 3 years ago
• bwconrad / soft-moe
  PyTorch implementation of "From Sparse to Soft Mixtures of Experts"
  ☆38Updated last year
• BlinkDL / LinearAttentionArena
  Here we will test various linear attention designs.
  ☆55Updated 4 months ago
• bojone / tiger
  A Tight-fisted Optimizer
  ☆46Updated last year
• OpenNLPLab / lightning-attention
  Lightning Attention-2: A Free Lunch for Handling Unlimited Sequence Lengths in Large Language Models
  ☆182Updated 4 months ago
• yegcjs / DiffusionLLM
  Code for paper "Diffusion Language Models Can Perform Many Tasks with Scaling and Instruction-Finetuning"
  ☆59Updated 7 months ago
• nengwp / Lion-vs-Adam
  Lion and Adam optimization comparison
  ☆56Updated last year
• OpenNLPLab / LASP
  Linear Attention Sequence Parallelism (LASP)
  ☆64Updated 3 months ago
• Shwai-He / MEO
  The source code of "Merging Experts into One: Improving Computational Efficiency of Mixture of Experts (EMNLP 2023)":
  ☆31Updated 5 months ago
• mrflogs / LoRA-Pro
  Official code for our paper, "LoRA-Pro: Are Low-Rank Adapters Properly Optimized? "
  ☆49Updated last month
• wutaiqiang / MoSLoRA
  ☆54Updated 2 months ago
• yikangshen / MoA
  Mixture of Attention Heads
  ☆36Updated last year
• fxmeng / mixtral_spliter
  Converting Mixtral-8x7B to Mixtral-[1~7]x7B
  ☆20Updated 6 months ago
• YuchuanTian / RethinkTinyLM
  [ICML'24] The official implementation of “Rethinking Optimization and Architecture for Tiny Language Models”
  ☆114Updated 2 months ago
• xiangyu-mm / EasyGen
  The official code for paper "EasyGen: Easing Multimodal Generation with a Bidirectional Conditional Diffusion Model and LLMs"
  ☆71Updated 7 months ago
• RWKV / RWKV-LM
  RWKV is an RNN with transformer-level LLM performance. It can be directly trained like a GPT (parallelizable). So it's combining the best…
  ☆19Updated 5 months ago
• GATECH-EIC / Linearized-LLM
  [ICML 2024] When Linear Attention Meets Autoregressive Decoding: Towards More Effective and Efficient Linearized Large Language Models
  ☆22Updated 3 months ago
• test-time-training / ttt-lm-kernels
  Inference Speed Benchmark for Learning to (Learn at Test Time): RNNs with Expressive Hidden States
  ☆33Updated 2 months ago
• OpenNLPLab / HGRN2
  HGRN2: Gated Linear RNNs with State Expansion
  ☆46Updated 3 weeks ago