wolfecameron / nanoMoELinks
An extension of the nanoGPT repository for training small MOE models.
☆164Updated 4 months ago
Alternatives and similar repositories for nanoMoE
Users that are interested in nanoMoE are comparing it to the libraries listed below
Sorting:
- ☆206Updated 5 months ago
- Normalized Transformer (nGPT)☆185Updated 8 months ago
- A repository to unravel the language of GPUs, making their kernel conversations easy to understand☆188Updated 2 months ago
- Memory layers use a trainable key-value lookup mechanism to add extra parameters to a model without increasing FLOPs. Conceptually, spars…☆344Updated 7 months ago
- minimal GRPO implementation from scratch☆94Updated 4 months ago
- Code for "LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding", ACL 2024☆323Updated 3 months ago
- PyTorch building blocks for the OLMo ecosystem☆269Updated this week
- 🚀 Efficiently (pre)training foundation models with native PyTorch features, including FSDP for training and SDPA implementation of Flash…☆258Updated last week
- ☆190Updated 7 months ago
- ☆184Updated 7 months ago
- A project to improve skills of large language models☆501Updated this week
- ArcticTraining is a framework designed to simplify and accelerate the post-training process for large language models (LLMs)☆190Updated this week
- Scalable toolkit for efficient model reinforcement☆558Updated this week
- Single File, Single GPU, From Scratch, Efficient, Full Parameter Tuning library for "RL for LLMs"☆512Updated 3 weeks ago
- Tina: Tiny Reasoning Models via LoRA☆274Updated 2 months ago
- LoRA and DoRA from Scratch Implementations☆207Updated last year
- Load compute kernels from the Hub☆220Updated this week
- Exploring Applications of GRPO☆245Updated 3 weeks ago
- ☆347Updated this week
- A family of compressed models obtained via pruning and knowledge distillation☆347Updated 8 months ago
- Repo for "LoLCATs: On Low-Rank Linearizing of Large Language Models"☆244Updated 6 months ago
- Simple & Scalable Pretraining for Neural Architecture Research☆277Updated last week
- The simplest, fastest repository for training/finetuning medium-sized GPTs.☆149Updated last month
- Understand and test language model architectures on synthetic tasks.☆221Updated 3 weeks ago
- Efficient LLM Inference over Long Sequences☆385Updated last month
- Best practices & guides on how to write distributed pytorch training code☆460Updated 5 months ago
- A simplified implementation for experimenting with RLVR on GSM8K, This repository provides a starting point for exploring reasoning.☆119Updated 5 months ago
- The official implementation of the paper "What Matters in Transformers? Not All Attention is Needed".☆174Updated 4 months ago
- nanoGRPO is a lightweight implementation of Group Relative Policy Optimization (GRPO)☆113Updated 2 months ago
- Decentralized RL Training at Scale☆400Updated this week