So, I trained a MoE based a 124M (8x12M) architecture I coded from ground up to build a small instruct model, going through the below-mentioned stages from scratch. Trained on TiyStories dataset form HuggingFace consisting of 1M tokens for a total of 14000 steps
☆14Jul 18, 2025Updated last year
Alternatives and similar repositories for SmolMixtral
Users that are interested in SmolMixtral are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- So, I trained a Llama a 130M architecture I coded from ground up to build a small instruct model from scratch. Trained on FineWeb dataset…☆17Mar 26, 2025Updated last year
- Interactive evolutionary exploration of generative design spaces with large language models☆18Jun 26, 2024Updated 2 years ago
- A collection of my from-scratch implementation of various models deployed on HF Spaces☆14Nov 3, 2026Updated 2 weeks ago
- Flash Attention from scratch, tiled CUDA forward kernel, online softmax with running max and correction factor, recomputation trick in ba…☆19Mar 6, 2026Updated 8 months ago
- Demonstration of MoE distributed training using various techniques☆11Oct 31, 2024Updated 2 years ago
- AI Agents on DigitalOcean Gradient AI Platform • AdBuild production-ready AI agents using customizable tools or access multiple LLMs through a single endpoint. Create custom knowledge bases or connect external data.
- ☆16Mar 18, 2025Updated last year
- Science or Slop?: BENCHMARKING AND MITIGATING SCIENTIFIC SLOP IN AI-GENERATED PAPERS☆16Sep 28, 2026Updated last month
- Notes of ADRL course taught at IISC as part of MTech AI curriculum☆15Nov 30, 2024Updated last year
- A distilled DeepSeek-R1 variant built on Qwen2.5-32B, fine-tuned with curated data for enhanced performance and efficiency. <metadata> gp…☆14Mar 11, 2025Updated last year
- ☆11Apr 20, 2020Updated 6 years ago
- Книга "Графовые нейронные сети на Python"☆10Aug 31, 2024Updated 2 years ago
- ☆10Jun 8, 2024Updated 2 years ago
- Different post-training techniques for LLMs, including: SFT, DPO and Online RL☆10Sep 5, 2025Updated last year
- Rucio K8s tutorial☆12Jul 24, 2026Updated 3 months ago
- Deploy on Railway without the complexity - Free Credits Offer • AdConnect your repo and Railway handles the rest with instant previews. Quickly provision container image services, databases, and storage volumes.
- 使用trl、peft、transformers等库,实现对huggingface上模型的微调。☆10Mar 21, 2025Updated last year
- Phony News Classifier is a repository which contains analysis of a natural language processing application i.e fake news classifier with …☆10May 27, 2020Updated 6 years ago
- Indic-Bias is a comprehensive benchmark to evaluate the fairness of LLMs in Indian Contexts.☆18Aug 14, 2025Updated last year
- See https://github.com/cuda-mode/triton-index/ instead!☆11May 8, 2024Updated 2 years ago
- Custom triton kernels for training Karpathy's nanoGPT.☆19Oct 21, 2024Updated 2 years ago
- ☆35Oct 15, 2026Updated last month
- ☆19Updated this week
- ☆51May 27, 2025Updated last year
- RuCLIP-SB (Russian Contrastive Language–Image Pretraining SWIN-BERT) is a multimodal model for obtaining images and text similarities and…☆15Jan 25, 2022Updated 4 years ago
- Open source password manager - Proton Pass • AdSecurely store, share, and autofill your credentials with Proton Pass, the end-to-end encrypted password manager trusted by millions.
- Open source interpretability artefacts for R1.☆187Apr 21, 2025Updated last year
- ☆29Nov 10, 2026Updated last week
- aesthetic tensor visualiser☆28Apr 23, 2025Updated last year
- repo of paper implementations☆20Feb 25, 2025Updated last year
- A simple Pure Python/PyTorch performance daemon for training workloads☆15Jul 28, 2026Updated 3 months ago
- A curated and updated list of relevant articles and repositories on Reinforcement Learning from AI Feedback (RLAIF)☆12Jan 24, 2024Updated 2 years ago
- KiMONo is a network inference tool for multi-omics datasets. The multi-omics network is calculated by sparse-group-LASSO regression and c…☆11Nov 21, 2022Updated 3 years ago
- Udacity Flying Cars Engineering Nanodegree☆11Aug 21, 2021Updated 5 years ago
- Whalegrad 🐳 is a lightweight deep learning library written in C.☆10Jan 5, 2025Updated last year
- Wordpress hosting with auto-scaling - Free Trial Offer • AdFully Managed hosting for WordPress and WooCommerce businesses that need reliable, auto-scalable performance. Cloudways SafeUpdates now available.
- Our solution to ML Talent Match hackathon☆10Mar 22, 2024Updated 2 years ago
- PyTorch implementation of GRPO.☆16Apr 21, 2025Updated last year
- 2023 Capstone Design☆12Nov 2, 2023Updated 3 years ago
- In-depth tutorials and examples on LLM training and inference infrastructure, such as, Pytorch, Fairscale, Nvidia AI Modules (cuDNN, tens…☆22May 19, 2025Updated last year
- RL a search agent on SEC filings with Tinker (Dr. GRPO, gpt-oss-20b)☆24Sep 17, 2026Updated 2 months ago
- Image captioning using CNN and RNN☆11Mar 24, 2025Updated last year
- Research on DeepSeek Sparse Attention☆41Oct 8, 2025Updated last year
Are these results useful?