wolfecameron / nanoMoELinks
An extension of the nanoGPT repository for training small MOE models.
☆178Updated 5 months ago
Alternatives and similar repositories for nanoMoE
Users that are interested in nanoMoE are comparing it to the libraries listed below
Sorting:
- ☆211Updated 6 months ago
- FlexAttention based, minimal vllm-style inference engine for fast Gemma 2 inference.☆250Updated 2 weeks ago
- PyTorch building blocks for the OLMo ecosystem☆274Updated this week
- Memory layers use a trainable key-value lookup mechanism to add extra parameters to a model without increasing FLOPs. Conceptually, spars…☆344Updated 8 months ago
- A repository to unravel the language of GPUs, making their kernel conversations easy to understand☆190Updated 2 months ago
- Normalized Transformer (nGPT)☆186Updated 9 months ago
- Code for "LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding", ACL 2024☆327Updated 3 months ago
- Load compute kernels from the Hub☆244Updated this week
- minimal GRPO implementation from scratch☆96Updated 5 months ago
- ☆193Updated 8 months ago
- ArcticTraining is a framework designed to simplify and accelerate the post-training process for large language models (LLMs)☆200Updated last week
- Tina: Tiny Reasoning Models via LoRA☆275Updated last week
- 🚀 Efficiently (pre)training foundation models with native PyTorch features, including FSDP for training and SDPA implementation of Flash…☆260Updated last month
- Simple & Scalable Pretraining for Neural Architecture Research☆287Updated 2 weeks ago
- Repo for "LoLCATs: On Low-Rank Linearizing of Large Language Models"☆244Updated 6 months ago
- The simplest, fastest repository for training/finetuning medium-sized GPTs.☆152Updated last month
- The official implementation of the paper "What Matters in Transformers? Not All Attention is Needed".☆174Updated 4 months ago
- Scalable toolkit for efficient model reinforcement☆626Updated last week
- A simplified implementation for experimenting with RLVR on GSM8K, This repository provides a starting point for exploring reasoning.☆121Updated 6 months ago
- A project to improve skills of large language models☆529Updated this week
- code for training & evaluating Contextual Document Embedding models☆197Updated 3 months ago
- LoRA and DoRA from Scratch Implementations☆209Updated last year
- Decentralized RL Training at Scale☆441Updated this week
- Code for studying the super weight in LLM☆115Updated 8 months ago
- Single File, Single GPU, From Scratch, Efficient, Full Parameter Tuning library for "RL for LLMs"☆516Updated last month
- nanoGRPO is a lightweight implementation of Group Relative Policy Optimization (GRPO)☆116Updated 3 months ago
- A compact LLM pretrained in 9 days by using high quality data☆322Updated 4 months ago
- Efficient LLM Inference over Long Sequences☆389Updated last month
- rl from zero pretrain, can it be done? yes.☆250Updated last week
- ☆192Updated 7 months ago