idiap / sigma-gpt
σ-GPT: A New Approach to Autoregressive Models
☆62Updated 7 months ago
Alternatives and similar repositories for sigma-gpt:
Users that are interested in sigma-gpt are comparing it to the libraries listed below
- ☆47Updated last year
- Focused on fast experimentation and simplicity☆70Updated 3 months ago
- The simplest, fastest repository for training/finetuning medium-sized GPTs.☆103Updated 4 months ago
- Explorations into the proposal from the paper "Grokfast, Accelerated Grokking by Amplifying Slow Gradients"☆98Updated 3 months ago
- A MAD laboratory to improve AI architecture designs 🧪☆108Updated 3 months ago
- ☆79Updated 11 months ago
- ☆53Updated last year
- an open source reproduction of NVIDIA's nGPT (Normalized Transformer with Representation Learning on the Hypersphere)☆91Updated 3 weeks ago
- ☆123Updated 3 weeks ago
- supporting pytorch FSDP for optimizers☆80Updated 3 months ago
- Minimal (400 LOC) implementation Maximum (multi-node, FSDP) GPT training☆123Updated 11 months ago
- Latent Diffusion Language Models☆68Updated last year
- ☆76Updated 8 months ago
- Pytorch implementation of the PEER block from the paper, Mixture of A Million Experts, by Xu Owen He at Deepmind☆123Updated 7 months ago
- ☆92Updated 2 months ago
- Scaling is a distributed training library and installable dependency designed to scale up neural networks, with a dedicated module for tr…☆58Updated 5 months ago
- Collection of autoregressive model implementation☆83Updated last month
- Normalized Transformer (nGPT)☆164Updated 4 months ago
- DeMo: Decoupled Momentum Optimization☆185Updated 4 months ago
- WIP☆93Updated 7 months ago
- ☆49Updated last year
- Code to reproduce "Transformers Can Do Arithmetic with the Right Embeddings", McLeish et al (NeurIPS 2024)☆187Updated 10 months ago
- ☆63Updated 6 months ago
- Latent Program Network (from the "Searching Latent Program Spaces" paper)☆76Updated 3 weeks ago
- ☆47Updated 4 months ago
- Explorations into the recently proposed Taylor Series Linear Attention☆96Updated 7 months ago
- LLMs represent numbers on a helix and manipulate that helix to do addition.☆21Updated last month
- Triton Implementation of HyperAttention Algorithm☆47Updated last year
- Implementation of the Llama architecture with RLHF + Q-learning☆163Updated 2 months ago
- ☆71Updated 7 months ago