Sharing both practical insights and theoretical knowledge about LLM evaluation that we gathered while managing the Open LLM Leaderboard and designing lighteval!
☆2,063Dec 3, 2025Updated 2 months ago
Alternatives and similar repositories for evaluation-guidebook
Users that are interested in evaluation-guidebook are comparing it to the libraries listed below
Sorting:
- Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends☆2,314Feb 20, 2026Updated last week
- Distilabel is a framework for synthetic data and AI feedback for engineers who need fast, reliable and scalable pipelines based on verifi…☆3,108Feb 23, 2026Updated last week
- 🤗 Benchmark Large Language Models Reliably On Your Data☆429Dec 30, 2025Updated 2 months ago
- Curated list of datasets and tools for post-training.☆4,265Nov 10, 2025Updated 3 months ago
- A course on aligning smol models.☆6,587Feb 6, 2026Updated 3 weeks ago
- The LLM Evaluation Framework☆13,787Feb 23, 2026Updated last week
- Robust recipes to align language models with human and AI preferences☆5,506Sep 8, 2025Updated 5 months ago
- Machine Learning Engineering Open Book☆17,162Feb 21, 2026Updated last week
- Everything about the SmolLM and SmolVLM family of models☆3,636Jan 13, 2026Updated last month
- A framework for few-shot evaluation of language models.☆11,478Feb 15, 2026Updated 2 weeks ago
- DSPy: The framework for programming—not prompting—language models☆32,381Updated this week
- Freeing data processing from scripting madness by providing a set of platform-agnostic customizable pipeline processing blocks.☆2,903Feb 24, 2026Updated last week
- Minimalistic large language model 3D-parallelism training☆2,579Feb 19, 2026Updated last week
- A reading list on LLM based Synthetic Data Generation 🔥☆1,520Jun 5, 2025Updated 8 months ago
- A system for agentic LLM-powered data processing and ETL☆3,636Feb 2, 2026Updated last month
- Go ahead and axolotl questions☆11,335Updated this week
- Structured Outputs☆13,456Feb 13, 2026Updated 2 weeks ago
- Our library for RL environments + evals☆3,869Updated this week
- structured outputs for llms☆12,428Updated this week
- Easily use and train state of the art late-interaction retrieval methods (ColBERT) in any RAG pipeline. Designed for modularity and ease-…☆3,859May 17, 2025Updated 9 months ago
- Fast lexical search implementing BM25 in Python using Numpy, Numba and Scipy☆1,500Feb 17, 2026Updated last week
- Meta Lingua: a lean, efficient, and easy-to-hack codebase to research LLMs.☆4,752Jul 18, 2025Updated 7 months ago
- Train transformer language models with reinforcement learning.☆17,460Updated this week
- PyTorch native post-training library☆5,691Updated this week
- Argilla is a collaboration tool for AI engineers and domain experts to build high-quality datasets☆4,875Feb 23, 2026Updated last week
- Deep learning for dummies. All the practical details and useful utilities that go into working with real models.☆831Updated this week
- Optimizing inference proxy for LLMs☆3,342Jan 28, 2026Updated last month
- AI Observability & Evaluation☆8,666Updated this week
- 🤗 smolagents: a barebones library for agents that think in code.☆25,615Feb 21, 2026Updated last week
- 20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.☆13,182Feb 22, 2026Updated last week
- Evaluate your LLM's response with Prometheus and GPT4 💯☆1,046Apr 25, 2025Updated 10 months ago
- Fast Multimodal Semantic Deduplication & Filtering☆890Jan 20, 2026Updated last month
- Recipes to scale inference-time compute of open models☆1,129May 22, 2025Updated 9 months ago
- Fine-tuning & Reinforcement Learning for LLMs. 🦥 Train OpenAI gpt-oss, DeepSeek, Qwen, Llama, Gemma, TTS 2x faster with 70% less VRAM.☆52,724Updated this week
- Supercharge Your LLM Application Evaluations 🚀☆12,736Feb 24, 2026Updated last week
- ☆4,368Jul 31, 2025Updated 7 months ago
- A lightweight, low-dependency, unified API to use all common reranking and cross-encoder models.☆1,599Dec 20, 2025Updated 2 months ago
- AllenAI's post-training codebase☆3,592Feb 24, 2026Updated last week
- Welcome to the Llama Cookbook! This is your go to guide for Building with Llama: Getting started with Inference, Fine-Tuning, RAG. We als…☆18,220Nov 3, 2025Updated 3 months ago