DACUS1995 / pytorch-mmap-dataset
A custom pytorch Dataset extension that provides a faster iteration and better RAM usage
☆41Updated 7 months ago
Related projects ⓘ
Alternatives and complementary repositories for pytorch-mmap-dataset
- CUDA implementation of autoregressive linear attention, with all the latest research findings☆43Updated last year
- A practical implementation of GradNorm, Gradient Normalization for Adaptive Loss Balancing, in Pytorch☆77Updated 9 months ago
- ResiDual: Transformer with Dual Residual Connections, https://arxiv.org/abs/2304.14802☆87Updated last year
- [ICLR 2022] "Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practice" by Peihao Wang, Wen…☆75Updated 10 months ago
- 32 times longer context window than vanilla Transformers and up to 4 times longer than memory efficient Transformers.☆42Updated last year
- Implementation of Switch Transformers from the paper: "Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficien…☆53Updated this week
- [EMNLP 2022] Official implementation of Transnormer in our EMNLP 2022 paper - The Devil in Linear Transformer☆54Updated last year
- Skyformer: Remodel Self-Attention with Gaussian Kernel and Nystr\"om Method (NeurIPS 2021)☆59Updated 2 years ago
- A repository for DenseSSMs☆88Updated 6 months ago
- Explorations into the recently proposed Taylor Series Linear Attention☆89Updated 2 months ago
- ☆15Updated last year
- PyTorch implementation of "From Sparse to Soft Mixtures of Experts"☆44Updated last year
- The accompanying code for "Memory-efficient Transformers via Top-k Attention" (Ankit Gupta, Guy Dar, Shaya Goodman, David Ciprut, Jonatha…☆60Updated 3 years ago
- [EVA ICLR'23; LARA ICML'22] Efficient attention mechanisms via control variates, random features, and importance sampling☆79Updated last year
- Implementation of "compositional attention" from MILA, a multi-head attention variant that is reframed as a two-step attention process wi…☆50Updated 2 years ago
- State Space Models☆61Updated 6 months ago
- Linear Attention Sequence Parallelism (LASP)☆64Updated 5 months ago
- A simple implementation of [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://arxiv.org/abs/2312.00752)☆19Updated 9 months ago
- ☆29Updated 2 years ago
- ☆41Updated 11 months ago
- Pytorch Implementation of the sparse attention from the paper: "Generating Long Sequences with Sparse Transformers"☆59Updated this week
- Implementation of IceFormer: Accelerated Inference with Long-Sequence Transformers on CPUs (ICLR 2024).☆20Updated 4 months ago
- Inference Speed Benchmark for Learning to (Learn at Test Time): RNNs with Expressive Hidden States☆38Updated 3 months ago
- Official code for the paper "Attention as a Hypernetwork"☆23Updated 4 months ago
- ☆17Updated last year
- Learnable Fourier Features for Multi-Dimensional Spatial Positional Encoding☆42Updated last month
- Implementation of fused cosine similarity attention in the same style as Flash Attention☆207Updated last year
- [NeurIPS 2022 Spotlight] This is the official PyTorch implementation of "EcoFormer: Energy-Saving Attention with Linear Complexity"☆66Updated last year
- several types of attention modules written in PyTorch☆39Updated last month
- ☆15Updated last year