Alternatives and similar repositories for Cross-Layer-Attention:

Users that are interested in Cross-Layer-Attention are comparing it to the libraries listed below

• fla-org / flash-bidirectional-linear-attention
  Triton implement of bi-directional (non-causal) linear attention
  ☆46Updated 2 months ago
• OpenSparseLLMs / Open-Pandora
  Open-Pandora: On-the-fly Control Video Generation
  ☆34Updated 4 months ago
• chuanyang-Zheng / DAPE
  The this is the official implementation of "DAPE: Data-Adaptive Positional Encoding for Length Extrapolation"
  ☆37Updated 6 months ago
• BBuf / flash-rwkv
  ☆30Updated 11 months ago
• OpenSparseLLMs / Linear-MoE
  ☆76Updated last week
• thunlp / SparsingLaw
  The open-source materials for paper "Sparsing Law: Towards Large Language Models with Greater Activation Sparsity".
  ☆20Updated 5 months ago
• OpenSparseLLMs / MoM
  ☆74Updated this week
• SprocketLab / sparse_matrix_fine_tuning
  Official repository for ICML 2024 paper "MoRe Fine-Tuning with 10x Fewer Parameters"
  ☆17Updated this week
• shaochenze / PatchTrain
  Code for paper "Patch-Level Training for Large Language Models"
  ☆82Updated 5 months ago
• MikaStars39 / StableMask
  PyTorch implementation of StableMask (ICML'24)
  ☆12Updated 9 months ago
• OpenSparseLLMs / Linearization
  ☆39Updated last month
• zhixuan-lin / forgetting-transformer
  [ICLR 2025] Official PyTorch implementation of "Forgetting Transformer: Softmax Attention with a Forget Gate"
  ☆95Updated 2 weeks ago
• yikangshen / MoA
  Mixture of Attention Heads
  ☆44Updated 2 years ago
• kamanphoebe / Look-into-MoEs
  [NAACL 2025] A Closer Look into Mixture-of-Experts in Large Language Models
  ☆51Updated 2 months ago
• savadikarc / gift
  GIFT: Generative Interpretable Fine-Tuning
  ☆20Updated 6 months ago
• sail-sg / LongSpec
  LongSpec: Long-Context Speculative Decoding with Efficient Drafting and Verification
  ☆50Updated last month
• sustcsonglin / linear-attention-and-beyond-slides
  ☆69Updated 2 months ago
• VITA-Group / Ms-PoE
  "Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encoding" Zhenyu Zhang, Runjin Chen, Shiw…
  ☆29Updated 11 months ago
• TIGER-AI-Lab / VISTA
  The code for "VISTA: Enhancing Long-Duration and High-Resolution Video Understanding by VIdeo SpatioTemporal Augmentation" [CVPR2025]
  ☆15Updated last month
• OpenNLPLab / HGRN2
  HGRN2: Gated Linear RNNs with State Expansion
  ☆54Updated 8 months ago
• pprp / Pruner-Zero
  Pruner-Zero: Evolving Symbolic Pruning Metric from scratch for LLMs
  ☆81Updated 5 months ago
• bojone / softtopk
  differentiable top-k operator
  ☆21Updated 3 months ago
• Infini-AI-Lab / S2FT
  ☆17Updated 3 months ago
• GATECH-EIC / Linearized-LLM
  [ICML 2024] When Linear Attention Meets Autoregressive Decoding: Towards More Effective and Efficient Linearized Large Language Models
  ☆30Updated 10 months ago
• BlinkDL / LinearAttentionArena
  Here we will test various linear attention designs.
  ☆60Updated last year
• sail-sg / scaling-with-vocab
  [NeurIPS-2024] 📈 Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies https://arxiv.org/abs/2407.13623
  ☆83Updated 7 months ago
• Shwai-He / MEO
  The source code of "Merging Experts into One: Improving Computational Efficiency of Mixture of Experts (EMNLP 2023)":
  ☆36Updated last year
• OpenGVLab / V2PE
  [ArXiv] V2PE: Improving Multimodal Long-Context Capability of Vision-Language Models with Variable Visual Position Encoding
  ☆44Updated 4 months ago
• yale-nlp / refdpo
  ☆16Updated 9 months ago
• pixeli99 / MixLN
  [ICLR 2025] Official Pytorch Implementation of "Mix-LN: Unleashing the Power of Deeper Layers by Combining Pre-LN and Post-LN" by Pengxia…
  ☆19Updated 4 months ago