idiap / sigma-gpt
σ-GPT: A New Approach to Autoregressive Models
☆62Updated 8 months ago
Alternatives and similar repositories for sigma-gpt:
Users that are interested in sigma-gpt are comparing it to the libraries listed below
- The simplest, fastest repository for training/finetuning medium-sized GPTs.☆105Updated 5 months ago
- Focused on fast experimentation and simplicity☆71Updated 4 months ago
- ☆48Updated last year
- supporting pytorch FSDP for optimizers☆80Updated 4 months ago
- ☆53Updated last year
- ☆94Updated 3 months ago
- Pytorch implementation of the PEER block from the paper, Mixture of A Million Experts, by Xu Owen He at Deepmind☆123Updated 8 months ago
- ☆143Updated 2 weeks ago
- an open source reproduction of NVIDIA's nGPT (Normalized Transformer with Representation Learning on the Hypersphere)☆96Updated last month
- ☆78Updated 9 months ago
- Explorations into the proposal from the paper "Grokfast, Accelerated Grokking by Amplifying Slow Gradients"☆98Updated 4 months ago
- Minimal (400 LOC) implementation Maximum (multi-node, FSDP) GPT training☆123Updated last year
- ☆79Updated last year
- ☆49Updated last year
- A MAD laboratory to improve AI architecture designs 🧪☆111Updated 4 months ago
- WIP☆93Updated 8 months ago
- Code to reproduce "Transformers Can Do Arithmetic with the Right Embeddings", McLeish et al (NeurIPS 2024)☆189Updated 10 months ago
- research impl of Native Sparse Attention (2502.11089)☆53Updated 2 months ago
- Fast, Modern, Memory Efficient, and Low Precision PyTorch Optimizers☆92Updated 9 months ago
- ☆63Updated 7 months ago
- Implementation of the Llama architecture with RLHF + Q-learning☆164Updated 2 months ago
- EvaByte: Efficient Byte-level Language Models at Scale☆88Updated this week
- ☆71Updated 7 months ago
- Latent Diffusion Language Models☆68Updated last year
- nanoGPT-like codebase for LLM training☆94Updated 3 weeks ago
- ☆61Updated last year
- Code accompanying the paper "Generalized Interpolating Discrete Diffusion"☆77Updated this week
- RWKV-7: Surpassing GPT☆83Updated 5 months ago
- Understand and test language model architectures on synthetic tasks.☆192Updated last month
- $100K or 100 Days: Trade-offs when Pre-Training with Academic Resources☆135Updated last month