Flash Attention from scratch, tiled CUDA forward kernel, online softmax with running max and correction factor, recomputation trick in backward, O(N) memory, full forward and backward verified against PyTorch autograd to 1e-6.
☆19Mar 6, 2026Updated 8 months ago
Alternatives and similar repositories for FlashAttention-CuPy
Users that are interested in FlashAttention-CuPy are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- So, I trained a MoE based a 124M (8x12M) architecture I coded from ground up to build a small instruct model, going through the below-men…☆14Jul 18, 2025Updated last year
- Implemented and benchmarked KV cache compression methods for LLM inference in Triton. Optimized kernels for KIVI and TurboQuant with Flas…☆18Jun 19, 2026Updated 4 months ago
- Comparative study and experimentation on standard vs mHC vs attention residual (full and block)☆13Mar 17, 2026Updated 8 months ago
- a minimal paged attention implementation☆20Jan 30, 2026Updated 9 months ago
- ☆11Apr 20, 2020Updated 6 years ago
- Deploy to Railway using AI coding agents - Free Credits Offer • AdUse Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
- One RL training loop that runs CartPole on a laptop and disaggregated async RLVR on a GPU cluster. About 1,800 lines across 7 files. No R…☆13Oct 8, 2026Updated last month
- Custom memory allocator in C++ built from scratch using mmap. Allocates a 1MB memory pool upfront and carves blocks from it to keep all a…☆38Apr 29, 2026Updated 6 months ago
- PPO (Proximal Policy Optimization) agent for training on Atari environments using C++ with libtorch, integrated with the Arcade Learning …☆10Jun 14, 2026Updated 5 months ago
- JAX implementation of NeuroEvolution of Augmenting Topologies (Neat)☆11Jul 6, 2024Updated 2 years ago
- kaggle - RSNA STR Pulmonary Embolism Detection☆11Nov 22, 2020Updated 5 years ago
- 【GRSL 2021】ASEA for Change Detection☆11Sep 1, 2026Updated 2 months ago
- An unofficial implementation of MEME (Efficient Memory-based Exploration agent) from DeepMind☆11Apr 13, 2024Updated 2 years ago
- building various transformer model architectures and its modules from scratch.☆13Mar 14, 2025Updated last year
- Interactive version of the CuTe layout paper☆56Apr 14, 2026Updated 7 months ago
- Deploy to Railway using AI coding agents - Free Credits Offer • AdUse Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
- This is a simple automated license plate detector developed in C++ via OpenCV.☆11Sep 26, 2020Updated 6 years ago
- A GPT-2 inference engine written from scratch in CUDA and C++. Implements custom CUDA kernels for tiled matrix multiplication, LayerNorm,…☆44May 17, 2026Updated 6 months ago
- Code for paper "Understanding Generalization, Robustness, and Interpretability in Low-Capacity Neural Networks"☆13Jul 24, 2025Updated last year
- ☆10Mar 21, 2023Updated 3 years ago
- In-depth tutorials and examples on LLM training and inference infrastructure, such as, Pytorch, Fairscale, Nvidia AI Modules (cuDNN, tens…☆22May 19, 2025Updated last year
- ☆27Sep 9, 2024Updated 2 years ago
- A visual representation of Dijkstra's Algorithm using Libgdx.☆16Dec 24, 2021Updated 4 years ago
- Kaggle Notebooks☆12Updated this week
- Soft Actor-Critic implementation with SOTA model-free extension (REDQ) and SOTA model-based extension (MBPO).☆16Feb 21, 2021Updated 5 years ago
- Deploy on Railway without the complexity - Free Credits Offer • AdConnect your repo and Railway handles the rest with instant previews. Quickly provision container image services, databases, and storage volumes.
- 3rd Place solution☆12Nov 20, 2024Updated last year
- Raytracer written in pure Nix☆13Feb 12, 2024Updated 2 years ago
- Implementation and explorations into Blackbox Gradient Sensing (BGS), an evolutionary strategies approach proposed in a Google Deepmind p…☆20Apr 17, 2026Updated 7 months ago
- A neural network library written in jax☆14Feb 3, 2025Updated last year
- code for paper: BlindGuard: Safeguarding LLM-based Multi-Agent Systems under Unknown Attacks☆16Aug 24, 2026Updated 2 months ago
- Open-source Linux VM builder for nix on macOS using Virtualization.framework☆12Oct 19, 2026Updated last month
- 🚀 Collection of libraries used with fms-hf-tuning to accelerate fine-tuning and training of large models.☆14Oct 1, 2026Updated last month
- ☆12Sep 9, 2026Updated 2 months ago
- Mini RL Lab☆16Jun 17, 2024Updated 2 years ago
- Managed hosting for WordPress and PHP on Cloudways • AdManaged hosting for WordPress, Magento, Laravel, or PHP apps, on multiple cloud providers. Deploy in minutes on Cloudways by DigitalOcean.
- Official implementation of "NinA: Normalizing Flows in Action. Training VLA Models with Normalizing Flows"☆19Sep 22, 2025Updated last year
- This is the code repository of drone deep reinforcement learning☆12Mar 30, 2021Updated 5 years ago
- Centralized cooperative reinforcement learning☆13Jan 8, 2023Updated 3 years ago
- A test for RL application on f1tenth gym environment☆11Apr 10, 2023Updated 3 years ago
- This repository contains PyTorch implementations of deep reinforcement learning algorithms and environments for Robotics and Controls. T…☆19Mar 20, 2022Updated 4 years ago
- Scripts and config files for SLURM☆13Sep 12, 2025Updated last year
- A tool to simplify SVG files to use more basic features☆14Sep 16, 2026Updated 2 months ago
Are these results useful?