prateekstark / retnet
☆14Updated last year
Related projects ⓘ
Alternatives and complementary repositories for retnet
- an implementation of paper"Retentive Network: A Successor to Transformer for Large Language Models" https://arxiv.org/pdf/2307.08621.pdf☆12Updated last year
- Implementation of Griffin from the paper: "Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models"☆49Updated this week
- PyTorch implementation of Retentive Network: A Successor to Transformer for Large Language Models☆16Updated last year
- HGRN2: Gated Linear RNNs with State Expansion☆49Updated 2 months ago
- [NeurIPS 2023 spotlight] Official implementation of HGRN in our NeurIPS 2023 paper - Hierarchically Gated Recurrent Neural Network for Se…☆61Updated 6 months ago
- My implementation of the original transformer model (Vaswani et al.). I've additionally included the playground.py file for visualizing o…☆41Updated 11 months ago
- ☆21Updated last month
- ☆52Updated this week
- Community Implementation of the paper: "Multi-Head Mixture-of-Experts" In PyTorch☆18Updated last week
- State Space Models☆62Updated 6 months ago
- A simple but robust PyTorch implementation of RetNet from "Retentive Network: A Successor to Transformer for Large Language Models" (http…☆100Updated 11 months ago
- ☆15Updated last year
- ☆41Updated 7 months ago
- A Triton Kernel for incorporating Bi-Directionality in Mamba2☆47Updated 2 months ago
- Minimal Mamba-2 implementation in PyTorch☆129Updated 4 months ago
- Transformer model based on Kolmogorov–Arnold Network(KAN), which is an alternative of Multi-Layer Perceptron(MLP)☆24Updated last month
- A repository for DenseSSMs☆88Updated 7 months ago
- ☆25Updated 4 months ago
- ☆44Updated 4 months ago
- PyTorch reimplementation of the Smooth ReLU activation function proposed in the paper "Real World Large Scale Recommendation Systems Repr…☆21Updated 2 years ago
- ☆14Updated 2 years ago
- This is a simple torch implementation of the high performance Multi-Query Attention☆15Updated last year
- ☆32Updated 5 months ago
- Implementation of MambaFormer in Pytorch ++ Zeta from the paper: "Can Mamba Learn How to Learn? A Comparative Study on In-Context Learnin…☆20Updated this week
- Implementation of xLSTM in Pytorch from the paper: "xLSTM: Extended Long Short-Term Memory"☆103Updated this week
- Pytorch (Lightning) implementation of the Mamba model☆14Updated 6 months ago
- Pytorch Implementation of the paper: "Learning to (Learn at Test Time): RNNs with Expressive Hidden States"☆23Updated last week
- Inference Speed Benchmark for Learning to (Learn at Test Time): RNNs with Expressive Hidden States☆38Updated 3 months ago
- Cuda implementation of Extended Long Short Term Memory (xLSTM) with C++ and PyTorch ports☆74Updated 5 months ago
- PyTorch implementation of Soft MoE by Google Brain in "From Sparse to Soft Mixtures of Experts" (https://arxiv.org/pdf/2308.00951.pdf)☆64Updated last year