YihongDong / FANformer
☆24Updated last month
Alternatives and similar repositories for FANformer:
Users that are interested in FANformer are comparing it to the libraries listed below
- DeciMamba: Exploring the Length Extrapolation Potential of Mamba (ICLR 2025)☆26Updated 2 weeks ago
- ☆53Updated 9 months ago
- [NeurIPS 2023 spotlight] Official implementation of HGRN in our NeurIPS 2023 paper - Hierarchically Gated Recurrent Neural Network for Se…☆64Updated last year
- Implementation of Griffin from the paper: "Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models"☆52Updated 3 weeks ago
- ☆39Updated last month
- ☆76Updated last week
- ☆23Updated 7 months ago
- Repo for "Z1: Efficient Test-time Scaling with Code"☆55Updated 2 weeks ago
- Linear Attention Sequence Parallelism (LASP)☆82Updated 10 months ago
- Fourier Position Embedding: Enhancing Attention’s Periodic Extension for Length Generalization☆19Updated 3 months ago
- [ICLR 2023] "Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers" by Tianlong Chen*, Zhenyu Zhang*, Ajay Jaiswal…☆51Updated 2 years ago
- Implementation of the paper: "Mixture-of-Depths: Dynamically allocating compute in transformer-based language models"☆91Updated this week
- ☆78Updated 8 months ago
- ☆74Updated last week
- [NAACL 2025] A Closer Look into Mixture-of-Experts in Large Language Models☆51Updated 2 months ago
- [EMNLP 2022] Official implementation of Transnormer in our EMNLP 2022 paper - The Devil in Linear Transformer☆60Updated last year
- A large-scale RWKV v6, v7(World, ARWKV, PRWKV) inference. Capable of inference by combining multiple states(Pseudo MoE). Easy to deploy o…☆35Updated this week
- MambaFormer in-context learning experiments and implementation for https://arxiv.org/abs/2402.04248☆51Updated 10 months ago
- ☆23Updated 11 months ago
- ☆102Updated last year
- Efficient PScan implementation in PyTorch☆16Updated last year
- Here we will test various linear attention designs.☆60Updated last year
- [ICLR 2025] SuperCorrect: Advancing Small LLM Reasoning with Thought Template Distillation and Self-Correction☆68Updated last month
- continous batching and parallel acceleration for RWKV6☆24Updated 10 months ago
- Official Code Repository for the paper "Key-value memory in the brain"☆24Updated 2 months ago
- [NAACL 24 Oral] LoRETTA: Low-Rank Economic Tensor-Train Adaptation for Ultra-Low-Parameter Fine-Tuning of Large Language Models☆32Updated 3 months ago
- Official repository of paper "RNNs Are Not Transformers (Yet): The Key Bottleneck on In-context Retrieval"☆26Updated last year
- The official Pytorch implementation of the paper "Fourier Transformer: Fast Long Range Modeling by Removing Sequence Redundancy with FFT …☆37Updated last year
- ☆14Updated last year
- Tiled Flash Linear Attention library for fast and efficient mLSTM Kernels.☆55Updated 3 weeks ago