Alternatives and similar repositories for AttentionIsOFFByOne

Users that are interested in AttentionIsOFFByOne are comparing it to the libraries listed below

• OpenNLPLab / TransnormerLLM
  Official implementation of TransNormerLLM: A Faster and Better LLM
  ☆243Updated last year
• Arnav0400 / ViT-Slim
  Official code for our CVPR'22 paper “Vision Transformer Slimming: Multi-Dimension Searching in Continuous Optimization Space”
  ☆249Updated last year
• bojone / rerope
  Rectified Rotary Position Embeddings
  ☆370Updated last year
• OpenNLPLab / lightning-attention
  Lightning Attention-2: A Free Lunch for Handling Unlimited Sequence Lengths in Large Language Models
  ☆296Updated 3 months ago
• transformer-vq / transformer_vq
  ☆191Updated last year
• haochengxi / Train_Transformers_with_INT4
  ☆151Updated last year
• facebookresearch / LLM-QAT
  Code repo for the paper "LLM-QAT Data-Free Quantization Aware Training for Large Language Models"
  ☆287Updated 3 months ago
• nengwp / Lion-vs-Adam
  Lion and Adam optimization comparison
  ☆61Updated 2 years ago
• QingruZhang / AdaLoRA
  AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning (ICLR 2023).
  ☆327Updated 2 years ago
• fkodom / grouped-query-attention-pytorch
  (Unofficial) PyTorch implementation of grouped-query attention (GQA) from "GQA: Training Generalized Multi-Query Transformer Models from …
  ☆166Updated last year
• YuchuanTian / DiJiang
  [ICML'24 Oral] The official code of "DiJiang: Efficient Large Language Models through Compact Kernelization", a novel DCT-based linear at…
  ☆101Updated 11 months ago
• lucidrains / memory-efficient-attention-pytorch
  Implementation of a memory efficient multi-head attention as proposed in the paper, "Self-attention Does Not Need O(n²) Memory"
  ☆379Updated last year
• microsoft / Tutel
  Tutel MoE: Optimized Mixture-of-Experts Library, Support DeepSeek FP8/FP4
  ☆829Updated this week
• lucidrains / soft-moe-pytorch
  Implementation of Soft MoE, proposed by Brain's Vision team, in Pytorch
  ☆294Updated 2 months ago
• yxli2123 / LoftQ
  ☆222Updated 11 months ago
• YuchuanTian / RethinkTinyLM
  [ICML'24] The official implementation of “Rethinking Optimization and Architecture for Tiny Language Models”
  ☆121Updated 4 months ago
• google-research / vmoe
  ☆640Updated 2 weeks ago
• thu-ml / low-bit-optimizers
  Low-bit optimizers for PyTorch
  ☆128Updated last year
• Outsider565 / LoRA-GA
  ☆198Updated 7 months ago
• lucidrains / MEGABYTE-pytorch
  Implementation of MEGABYTE, Predicting Million-byte Sequences with Multiscale Transformers, in Pytorch
  ☆643Updated 5 months ago
• bojone / tiger
  A Tight-fisted Optimizer
  ☆48Updated 2 years ago
• ziplab / SN-Net
  [CVPR 2023 Highlight] This is the official implementation of "Stitchable Neural Networks".
  ☆247Updated 2 years ago
• kyegomez / NaViT
  My implementation of "Patch n’ Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution"
  ☆234Updated 2 months ago
• tsb0601 / EMP-SSL
  This repository contains the implementation for the paper "EMP-SSL: Towards Self-Supervised Learning in One Training Epoch."
  ☆228Updated last year
• kyegomez / FlashAttention20
  Get down and dirty with FlashAttention2.0 in pytorch, plug in and play no complex CUDA kernels
  ☆105Updated last year
• bzhangGo / rmsnorm
  Root Mean Square Layer Normalization
  ☆241Updated 2 years ago
• lucidrains / FLASH-pytorch
  Implementation of the Transformer variant proposed in "Transformer Quality in Linear Time"
  ☆363Updated last year
• OpenNLPLab / cosFormer
  [ICLR 2022] Official implementation of cosformer-attention in cosFormer: Rethinking Softmax in Attention
  ☆193Updated 2 years ago
• BlinkDL / RWKV-CUDA
  The CUDA version of the RWKV language model ( https://github.com/BlinkDL/RWKV-LM )
  ☆220Updated 5 months ago
• hahnyuan / RPTQ4LLM
  Reorder-based post-training quantization for large language model
  ☆190Updated 2 years ago