MoonshotAI / MoonlightLinks
Muon is Scalable for LLM Training
☆1,052Updated 2 months ago
Alternatives and similar repositories for Moonlight
Users that are interested in Moonlight are comparing it to the libraries listed below
Sorting:
- 🐳 Efficient Triton implementations for "Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention"☆686Updated 2 months ago
- Official Repo for Open-Reasoner-Zero☆1,939Updated last month
- ☆773Updated last month
- Understanding R1-Zero-Like Training: A Critical Perspective☆956Updated last week
- MoBA: Mixture of Block Attention for Long-Context LLMs☆1,781Updated last month
- Muon optimizer: +>30% sample efficiency with <3% wallclock overhead☆661Updated this week
- Scalable RL solution for advanced reasoning of language models☆1,587Updated 2 months ago
- An Open-source RL System from ByteDance Seed and Tsinghua AIR☆1,284Updated 3 weeks ago
- OLMoE: Open Mixture-of-Experts Language Models☆764Updated 2 months ago
- Dream 7B, a large diffusion language model☆703Updated 3 weeks ago
- Large Reasoning Models☆804Updated 5 months ago
- MMaDA - Open-Sourced Multimodal Large Diffusion Language Models☆819Updated last week
- Ring attention implementation with flash attention☆771Updated last week
- An Open Large Reasoning Model for Real-World Solutions☆1,494Updated this week
- [NeurIPS'24 Spotlight, ICLR'25, ICML'25] To speed up Long-context LLMs' inference, approximate and dynamic sparse calculate the attention…☆1,035Updated last week
- Parallel Scaling Law for Language Model — Beyond Parameter and Inference Time Scaling☆345Updated 2 weeks ago
- Memory optimization and training recipes to extrapolate language models' context length to 1 million tokens, with minimal hardware.☆725Updated 8 months ago
- Official Implementation of EAGLE-1 (ICML'24), EAGLE-2 (EMNLP'24), and EAGLE-3.☆1,266Updated 2 weeks ago
- Fast, Flexible and Portable Structured Generation☆980Updated last week
- O1 Replication Journey☆1,990Updated 4 months ago
- A fork to add multimodal model training to open-r1☆1,281Updated 3 months ago
- ☆706Updated this week
- Kimi-VL: Mixture-of-Experts Vision-Language Model for Multimodal Reasoning, Long-Context Understanding, and Strong Agent Capabilities☆868Updated last month
- Official PyTorch implementation for "Large Language Diffusion Models"☆2,036Updated this week
- [ICLR 2025] DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads☆463Updated 3 months ago
- Memory layers use a trainable key-value lookup mechanism to add extra parameters to a model without increasing FLOPs. Conceptually, spars…☆333Updated 5 months ago
- RAGEN leverages reinforcement learning to train LLM reasoning agents in interactive, stochastic environments.☆1,908Updated this week
- Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models☆660Updated last month
- Implementation of the sparse attention pattern proposed by the Deepseek team in their "Native Sparse Attention" paper☆637Updated 2 weeks ago
- Recipes to scale inference-time compute of open models☆1,087Updated last week