(Unofficial) PyTorch implementation of grouped-query attention (GQA) from "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints" (https://arxiv.org/pdf/2305.13245.pdf)
☆196May 9, 2024Updated 2 years ago
Alternatives and similar repositories for grouped-query-attention-pytorch
Users that are interested in grouped-query-attention-pytorch are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- The open source implementation of the multi grouped query attention by the paper "GQA: Training Generalized Multi-Query Transformer Model…☆17Dec 11, 2023Updated 2 years ago
- PyTorch implementation of BYOL: a fantastically simple method for self-supervised image representation learning with SOTA performance.☆11Mar 24, 2021Updated 5 years ago
- ☆20Oct 25, 2022Updated 4 years ago
- Ongoing Research Project for continaual pre-training LLM(dense mode)☆44Mar 3, 2025Updated last year
- Traceable TTS: Toward Watermark-Free TTS with Strong Traceability☆13Sep 4, 2025Updated last year
- Deploy on Railway without the complexity - Free Credits Offer • AdConnect your repo and Railway handles the rest with instant previews. Quickly provision container image services, databases, and storage volumes.
- A simple but robust PyTorch implementation of RetNet from "Retentive Network: A Successor to Transformer for Large Language Models" (http…☆105Nov 24, 2023Updated 2 years ago
- ☆20May 30, 2024Updated 2 years ago
- CUDA and Triton implementations of Flash Attention with SoftmaxN.☆75May 26, 2024Updated 2 years ago
- ☆22May 25, 2026Updated 5 months ago
- Triton-based implementation of Sparse Mixture of Experts.☆281Oct 3, 2025Updated last year
- This is the official code for the paper "Strip-MLP: Efficient Token Interaction for Vision MLP".☆13Feb 19, 2025Updated last year
- Fast and memory-efficient exact attention☆25,221Updated this week
- We introduce the LLAMA1 Test Set, a comprehensive open-domain world knowledge QA dataset for evaluating question-answering systems. We pr…☆23Mar 14, 2024Updated 2 years ago
- ☆10Dec 28, 2023Updated 2 years ago
- Deploy on Railway without the complexity - Free Credits Offer • AdConnect your repo and Railway handles the rest with instant previews. Quickly provision container image services, databases, and storage volumes.
- sigma-MoE layer☆21Jan 5, 2024Updated 2 years ago
- CoGenCast: A Coupled Autoregressive–Flow Generative Framework for Time Series Forecasting☆11May 31, 2026Updated 5 months ago
- Mixture of A Million Experts☆64Jul 30, 2024Updated 2 years ago
- ☆12Jul 17, 2025Updated last year
- Official implementation: "AudioSet-R: A Refined AudioSet with Multi-Stage LLM Label Reannotation"☆20Oct 9, 2025Updated last year
- KS-Transformer in PyTorch (ICASSP 2022)☆12Apr 2, 2023Updated 3 years ago
- ☆29May 4, 2024Updated 2 years ago
- PyTorch implementation of Soft MoE by Google Brain in "From Sparse to Soft Mixtures of Experts" (https://arxiv.org/pdf/2308.00951.pdf)☆83Oct 5, 2023Updated 3 years ago
- ☆24Sep 25, 2024Updated 2 years ago
- Open source password manager - Proton Pass • AdSecurely store, share, and autofill your credentials with Proton Pass, the end-to-end encrypted password manager trusted by millions.
- Time Series Representation Models☆13Jul 17, 2025Updated last year
- ☆23Oct 17, 2024Updated 2 years ago
- ☆11Nov 23, 2021Updated 4 years ago
- From scratch implementation of a sparse mixture of experts language model inspired by Andrej Karpathy's makemore :)☆820Oct 30, 2024Updated 2 years ago
- A lightweight design for computation-communication overlap.☆252Jan 20, 2026Updated 9 months ago
- ☆14Jan 22, 2025Updated last year
- This is a personal reimplementation of Google's Infini-transformer, utilizing a small 2b model. The project includes both model and train…☆58Apr 20, 2024Updated 2 years ago
- PyTorch implementation of Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation☆33Dec 29, 2021Updated 4 years ago
- Ring attention implementation with flash attention☆1,065Sep 10, 2025Updated last year
- Deploy on Railway without the complexity - Free Credits Offer • AdConnect your repo and Railway handles the rest with instant previews. Quickly provision container image services, databases, and storage volumes.
- ☆11Aug 9, 2025Updated last year
- An implementation of the CLEAN algorithm for extracting individual wave packets from a multi-component signal☆16Jun 15, 2018Updated 8 years ago
- ☆18Jul 22, 2026Updated 3 months ago
- Official implementation of our paper "Diving Deep into Regions: Exploiting Regional Information Transformer for Single Image Deraining."☆16Aug 24, 2024Updated 2 years ago
- ☆34Oct 30, 2023Updated 3 years ago
- A family of open-sourced Mixture-of-Experts (MoE) Large Language Models☆1,702Mar 8, 2024Updated 2 years ago
- ☆49Mar 31, 2024Updated 2 years ago