fangyuan-ksgk / selective-attention-transformerLinks
Unofficial Implementation of Selective Attention Transformer
☆17Updated 9 months ago
Alternatives and similar repositories for selective-attention-transformer
Users that are interested in selective-attention-transformer are comparing it to the libraries listed below
Sorting:
- ☆34Updated 5 months ago
- Code for NeurIPS 2024 Spotlight: "Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations"☆81Updated 10 months ago
- ☆85Updated last year
- ☆34Updated 7 months ago
- ☆19Updated 5 months ago
- [ICLR 2025 & COLM 2025] Official PyTorch implementation of the Forgetting Transformer and Adaptive Computation Pruning☆124Updated 2 weeks ago
- One Initialization to Rule them All: Fine-tuning via Explained Variance Adaptation☆41Updated 10 months ago
- Official repository of "LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Merging"☆30Updated 9 months ago
- Remasking Discrete Diffusion Models with Inference-Time Scaling☆38Updated 5 months ago
- ☆20Updated last year
- Official PyTorch implementation and models for paper "Diffusion Beats Autoregressive in Data-Constrained Settings". We find diffusion mod…☆58Updated last week
- DeciMamba: Exploring the Length Extrapolation Potential of Mamba (ICLR 2025)☆29Updated 4 months ago
- SLTrain: a sparse plus low-rank approach for parameter and memory efficient pretraining (NeurIPS 2024)☆33Updated 9 months ago
- LongLLaDA: Unlocking Long Context Capabilities in Diffusion LLMs☆32Updated last month
- The official github repo for "Diffusion Language Models are Super Data Learners".☆103Updated 2 weeks ago
- [ICLR 2025] Official Pytorch Implementation of "Mix-LN: Unleashing the Power of Deeper Layers by Combining Pre-LN and Post-LN" by Pengxia…☆25Updated last month
- Official PyTorch Implementation for Vision-Language Models Create Cross-Modal Task Representations, ICML 2025☆30Updated 3 months ago
- Official code for the paper "Attention as a Hypernetwork"☆40Updated last year
- ☆31Updated 6 months ago
- ☆33Updated 10 months ago
- ☆23Updated last month
- [ICLR 2025] When Attention Sink Emerges in Language Models: An Empirical View (Spotlight)☆118Updated last month
- ☆33Updated last year
- HGRN2: Gated Linear RNNs with State Expansion☆54Updated last year
- [NeurIPS 2024] Low rank memory efficient optimizer without SVD☆30Updated last month
- [NAACL 2025] A Closer Look into Mixture-of-Experts in Large Language Models☆54Updated 6 months ago
- Official repository of paper "RNNs Are Not Transformers (Yet): The Key Bottleneck on In-context Retrieval"☆27Updated last year
- The this is the official implementation of "DAPE: Data-Adaptive Positional Encoding for Length Extrapolation"☆39Updated 10 months ago
- Code accompanying the paper "Massive Activations in Large Language Models"☆176Updated last year
- ☆33Updated last year