Sharing both practical insights and theoretical knowledge about LLM evaluation that we gathered while managing the Open LLM Leaderboard and designing lighteval!
☆2,166Dec 3, 2025Updated 11 months ago
Alternatives and similar repositories for evaluation-guidebook
Users that are interested in evaluation-guidebook are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends☆2,587Updated this week
- 🤗 Benchmark Large Language Models Reliably On Your Data☆464Oct 14, 2026Updated last month
- Distilabel is a framework for synthetic data and AI feedback for engineers who need fast, reliable and scalable pipelines based on verifi…☆3,467Updated this week
- Curated list of datasets and tools for post-training.☆4,838Apr 29, 2026Updated 6 months ago
- Everything about the SmolLM and SmolVLM family of models☆3,960Oct 6, 2026Updated last month
- Simple, predictable pricing with DigitalOcean hosting • AdAlways know what you'll pay with monthly caps and flat pricing. Enterprise-grade infrastructure trusted by 600k+ customers.
- A framework for few-shot evaluation of language models.☆14,515Nov 4, 2026Updated 2 weeks ago
- A course on aligning smol models.☆6,799Updated this week
- Robust recipes to align language models with human and AI preferences☆5,716Sep 23, 2026Updated last month
- Freeing data processing from scripting madness by providing a set of platform-agnostic customizable pipeline processing blocks.☆3,429Updated this week
- The LLM Evaluation Framework☆19,585Updated this week
- Machine Learning Engineering Open Book☆19,738Updated this week
- Minimalistic large language model 3D-parallelism training☆2,869Oct 8, 2026Updated last month
- DSPy: The framework for programming—not prompting—language models☆39,164Updated this week
- Go ahead and axolotl questions☆12,742Updated this week
- Managed hosting for WordPress and PHP on Cloudways • AdManaged hosting for WordPress, Magento, Laravel, or PHP apps, on multiple cloud providers. Deploy in minutes on Cloudways by DigitalOcean.
- Structured Outputs☆16,017Sep 21, 2026Updated 2 months ago
- Our library for RL environments + evals☆4,798Updated this week
- Train transformer language models with reinforcement learning.☆19,788Updated this week
- PyTorch native post-training library☆5,820Updated this week
- Recipes to scale inference-time compute of open models☆1,134Sep 23, 2026Updated last month
- Fully open reproduction of DeepSeek-R1☆26,485Oct 26, 2026Updated 3 weeks ago
- A system for agentic LLM-powered data processing and ETL☆4,266Updated this week
- Argilla is a collaboration tool for AI engineers and domain experts to build high-quality datasets☆5,208Updated this week
- A reading list on LLM based Synthetic Data Generation 🔥☆1,557Jun 5, 2025Updated last year
- Deploy open-source AI quickly and easily - Special Bonus Offer • AdRunpod Hub is built for open source. One-click deployment and autoscaling endpoints without provisioning your own infrastructure.
- Easily use and train state of the art late-interaction retrieval methods (ColBERT) in any RAG pipeline. Designed for modularity and ease-…☆3,976May 17, 2025Updated last year
- 20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.☆13,757Nov 10, 2026Updated last week
- AllenAI's post-training codebase☆3,942Updated this week
- 🤗 smolagents: a barebones library for agents that think in code.☆30,466Updated this week
- AI Observability & Evaluation☆12,369Updated this week
- Supercharge Your LLM Application Evaluations 🚀☆16,397Nov 14, 2026Updated last week
- Fast Multimodal Semantic Deduplication & Filtering☆979Nov 11, 2026Updated last week
- A lightweight, local-first, and free experiment tracking library from Hugging Face 🤗☆1,795Nov 12, 2026Updated last week
- structured outputs for llms☆14,130Updated this week
- GPU virtual machines on DigitalOcean Gradient AI • AdGet to production fast with high-performance AMD and NVIDIA GPUs you can spin up in seconds. The definition of operational simplicity.
- Textbook on reinforcement learning from human feedback☆2,544Updated this week
- Deep learning for dummies. All the practical details and useful utilities that go into working with real models.☆858Oct 27, 2026Updated 3 weeks ago
- Large Language Model Text Generation Inference☆10,876Mar 21, 2026Updated 8 months ago
- Welcome to the Llama Cookbook! This is your go to guide for Building with Llama: Getting started with Inference, Fine-Tuning, RAG. We als…☆18,595May 19, 2026Updated 6 months ago
- ☆714Aug 24, 2026Updated 2 months ago
- Tools for merging pretrained large language models.☆7,485Oct 31, 2026Updated 3 weeks ago
- Local UI to run and train LLMs and diffusion models. Supports GGUF, MLX, Qwen3.8, DeepSeek-V4.1, MiniMax-H3, Gemma 4, FLUX and more.☆79,895Updated this week
Are these results useful?