YihongDong / FANformer
☆19Updated last week
Alternatives and similar repositories for FANformer:
Users that are interested in FANformer are comparing it to the libraries listed below
- Implementation of Griffin from the paper: "Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models"☆52Updated 2 months ago
- Experiments on Multi-Head Latent Attention☆82Updated 7 months ago
- ☆72Updated last week
- [NeurIPS 2024] Fast Best-of-N Decoding via Speculative Rejection☆40Updated 5 months ago
- Research without Re-search: Maximal Update Parametrization Yields Accurate Loss Prediction across Scales☆32Updated last year
- ☆52Updated 8 months ago
- [NeurIPS 2023 spotlight] Official implementation of HGRN in our NeurIPS 2023 paper - Hierarchically Gated Recurrent Neural Network for Se…☆64Updated 11 months ago
- DeciMamba: Exploring the Length Extrapolation Potential of Mamba (ICLR 2025)☆25Updated 8 months ago
- A large-scale RWKV v6, v7(World, ARWKV, PRWKV) inference. Capable of inference by combining multiple states(Pseudo MoE). Easy to deploy o…☆33Updated last week
- ☆50Updated 5 months ago
- [ICML 2023] "Data Efficient Neural Scaling Law via Model Reusing" by Peihao Wang, Rameswar Panda, Zhangyang Wang☆14Updated last year
- ☆23Updated 6 months ago
- Exploration of automated dataset selection approaches at large scales.☆34Updated last month
- Official Code Repository for the paper "Key-value memory in the brain"☆24Updated last month
- The simplest implementation of recent Sparse Attention patterns for efficient LLM inference.☆59Updated 2 months ago
- Code for ICLR 2025 Paper "What is Wrong with Perplexity for Long-context Language Modeling?"☆47Updated last week
- [ICLR 2025] MiniPLM: Knowledge Distillation for Pre-Training Language Models☆34Updated 4 months ago
- ☆16Updated 3 months ago
- MambaFormer in-context learning experiments and implementation for https://arxiv.org/abs/2402.04248☆51Updated 9 months ago
- Official Implementation of "DeCoRe: Decoding by Contrasting Retrieval Heads to Mitigate Hallucination"☆22Updated 3 months ago
- Efficient PScan implementation in PyTorch☆16Updated last year
- Code for preprint "Metadata Conditioning Accelerates Language Model Pre-training (MeCo)"☆36Updated last week
- Code for ICML 2024 paper☆20Updated 2 weeks ago
- [ICLR2025] Codebase for "ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routing", built on Megatron-LM.☆66Updated 3 months ago
- continous batching and parallel acceleration for RWKV6☆24Updated 9 months ago
- [ICLR 2025] Official Code Release for Explaining Modern Gated-Linear RNNs via a Unified Implicit Attention Formulation☆41Updated last month
- From GaLore to WeLore: How Low-Rank Weights Non-uniformly Emerge from Low-Rank Gradients. Ajay Jaiswal, Lu Yin, Zhenyu Zhang, Shiwei Liu,…☆43Updated 8 months ago
- 🔥 A minimal training framework for scaling FLA models☆92Updated last week
- The official repository of the Omni-MATH benchmark.☆78Updated 3 months ago
- Linear Attention Sequence Parallelism (LASP)☆81Updated 9 months ago