OpenNLPLab / lightning-attention

Related projects: ⓘ

• thu-ml / low-bit-optimizers
  Low-bit optimizers for PyTorch
  ☆109Updated 11 months ago
• OpenNLPLab / TransnormerLLM
  Official implementation of TransNormerLLM: A Faster and Better LLM
  ☆223Updated 7 months ago
• xijiu9 / Train_Transformers_with_INT4
  ☆130Updated last year
• SkyworkAI / Skywork-MoE
  Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models
  ☆121Updated 3 months ago
• transformer-vq / transformer_vq
  ☆170Updated 9 months ago
• YuchuanTian / RethinkTinyLM
  [ICML'24] The official implementation of “Rethinking Optimization and Architecture for Tiny Language Models”
  ☆114Updated 2 months ago
• YuchuanTian / DiJiang
  [ICML'24 Oral] The official code of "DiJiang: Efficient Large Language Models through Compact Kernelization", a novel DCT-based linear at…
  ☆95Updated 3 months ago
• kyegomez / FlashAttention20
  Get down and dirty with FlashAttention2.0 in pytorch, plug in and play no complex CUDA kernels
  ☆90Updated last year
• nbasyl / LLM-FP4
  The official implementation of the EMNLP 2023 paper LLM-FP4
  ☆156Updated 9 months ago
• fkodom / grouped-query-attention-pytorch
  (Unofficial) PyTorch implementation of grouped-query attention (GQA) from "GQA: Training Generalized Multi-Query Transformer Models from …
  ☆116Updated 4 months ago
• Outsider565 / LoRA-GA
  ☆125Updated this week
• berlino / gated_linear_attention
  ☆94Updated 6 months ago
• zms1999 / SmartMoE
  A MoE impl for PyTorch, [ATC'23] SmartMoE
  ☆56Updated last year
• Equationliu / Kangaroo
  Implementation of Kangaroo: Lossless Self-Speculative Decoding via Double Early Exiting
  ☆39Updated 2 months ago
• madsys-dev / deepseekv2-profile
  ☆60Updated last month
• FasterDecoding / TEAL
  ☆75Updated this week
• opengear-project / GEAR
  GEAR: An Efficient KV Cache Compression Recipefor Near-Lossless Generative Inference of LLM
  ☆134Updated 2 months ago
• FasterDecoding / SnapKV
  ☆164Updated 4 months ago
• FlagOpen / FlagAttention
  A collection of memory efficient attention operators implemented in the Triton language.
  ☆205Updated 3 months ago
• OpenNLPLab / Transnormer
  [EMNLP 2022] Official implementation of Transnormer in our EMNLP 2022 paper - The Devil in Linear Transformer
  ☆53Updated last year
• bojone / tiger
  A Tight-fisted Optimizer
  ☆46Updated last year
• shreyansh26 / FlashAttention-PyTorch
  Implementation of FlashAttention in PyTorch
  ☆95Updated last year
• yxli2123 / LoftQ
  ☆191Updated 3 months ago
• feifeibear / long-context-attention
  USP: Unified (a.k.a. Hybrid, 2D) Sequence Parallel Attention for Long Context Transformers Model Training and Inference
  ☆309Updated this week
• nbasyl / DoRA
  Official implementation of "DoRA: Weight-Decomposed Low-Rank Adaptation"
  ☆120Updated 4 months ago
• astramind-ai / Mixture-of-depths
  Unofficial implementation for the paper "Mixture-of-Depths: Dynamically allocating compute in transformer-based language models"
  ☆123Updated 2 months ago
• kyegomez / AttentionIsOFFByOne
  Implementation of "Attention Is Off By One" by Evan Miller
  ☆177Updated last year
• htqin / IR-QLoRA
  [ICML 2024 Oral] This project is the official implementation of our Accurate LoRA-Finetuning Quantization of LLMs via Information Retenti…
  ☆55Updated 5 months ago
• Chongjie-Si / Subspace-Tuning
  A generalized framework for subspace tuning methods in parameter efficient fine-tuning.
  ☆70Updated this week
• mit-han-lab / patch_conv
  Patch convolution to avoid large GPU memory usage of Conv2D
  ☆73Updated 3 months ago