Holistic Evaluation of Language Models (HELM) is an open source Python framework created by the Center for Research on Foundation Models (CRFM) at Stanford for holistic, reproducible and transparent evaluation of foundation models, including large language models (LLMs) and multimodal models.
☆2,967Sep 1, 2026Updated 2 months ago
Alternatives and similar repositories for helm
Users that are interested in helm are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- A framework for few-shot evaluation of language models.☆14,415Updated this week
- Beyond the Imitation Game collaborative benchmark for measuring and extrapolating the capabilities of language models☆3,245Jul 19, 2024Updated 2 years ago
- Benchmarking large language models' complex reasoning ability with chain-of-thought prompting☆2,774Aug 4, 2024Updated 2 years ago
- Measuring Massive Multitask Language Understanding | ICLR 2021☆1,626May 28, 2023Updated 3 years ago
- An automatic evaluator for instruction-following language models. Human-validated, high-quality, cheap, and fast.☆2,023Aug 9, 2025Updated last year
- End-to-end encrypted cloud storage - Proton Drive • AdSpecial offer: 40% Off Yearly / 80% Off First Month. Protect your most important files, photos, and documents from prying eyes.
- Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.☆19,710Apr 14, 2026Updated 6 months ago
- Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends☆2,579Oct 29, 2026Updated last week
- A repo for distributed training of language models with Reinforcement Learning via Human Feedback (RLHF)☆4,756Jan 8, 2024Updated 2 years ago
- 🤗 PEFT: State-of-the-art Parameter-Efficient Fine-Tuning.☆21,873Updated this week
- Toolkit for creating, sharing and using natural language prompts.☆3,041Oct 23, 2023Updated 3 years ago
- ☆4,695Apr 22, 2026Updated 6 months ago
- Train transformer language models with reinforcement learning.☆19,675Updated this week
- Human preference data for "Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback"☆1,861Jun 17, 2025Updated last year
- OpenCompass is an LLM evaluation platform, supporting a wide range of models from OpenAI, Anthropic, Gemini, Qwen, GLM, DeepSeek, etc, ac…☆7,569Updated this week
- Managed Kubernetes at scale on DigitalOcean • AdDigitalOcean Kubernetes includes the control plane, bandwidth allowance, container registry, automatic updates, and more for free.
- Robust recipes to align language models with human and AI preferences☆5,711Sep 23, 2026Updated last month
- AllenAI's post-training codebase☆3,924Updated this week
- An open platform for training, serving, and evaluating large language models. Release repo for Vicuna and Chatbot Arena.☆39,548May 1, 2026Updated 6 months ago
- The RedPajama-Data repository contains code for preparing large datasets for training large language models.☆4,999Jun 3, 2026Updated 5 months ago
- Code and documentation to train Stanford's Alpaca models, and generate the data.☆30,223Jul 17, 2024Updated 2 years ago
- ☆1,575Jul 2, 2026Updated 4 months ago
- Aligning pretrained language models with instruction data generated by themselves.☆4,617Mar 27, 2023Updated 3 years ago
- A Comprehensive Benchmark to Evaluate LLMs as Agents (ICLR'24)☆3,817Feb 8, 2026Updated 8 months ago
- Fast and memory-efficient exact attention☆25,249Updated this week
- Managed Database hosting by DigitalOcean • AdPostgreSQL, MySQL, MongoDB, Kafka, Valkey, and OpenSearch available. Automatically scale up storage and focus on building your apps.
- General technology for enabling AI capabilities w/ LLMs and MLLMs☆4,484Sep 15, 2026Updated last month
- Accessible large language models via k-bit quantization for PyTorch.☆8,537Updated this week
- The official GitHub page for the survey paper "A Survey on Evaluation of Large Language Models".☆1,615Oct 28, 2026Updated last week
- DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.☆43,302Updated this week
- Ongoing research training transformer models at scale☆18,341Updated this week
- The hub for EleutherAI's work on interpretability and learning dynamics☆2,986Nov 15, 2025Updated 11 months ago
- 800,000 step-level correctness labels on LLM solutions to MATH problems☆2,156Jun 1, 2023Updated 3 years ago
- Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities☆22,240Sep 21, 2026Updated last month
- This repository contains code to quantitatively evaluate instruction-tuned models such as Alpaca and Flan-T5 on held-out tasks.☆551Mar 10, 2024Updated 2 years ago
- Wordpress hosting with auto-scaling - Free Trial Offer • AdFully Managed hosting for WordPress and WooCommerce businesses that need reliable, auto-scalable performance. Cloudways SafeUpdates now available.
- Tools for merging pretrained large language models.☆7,436Oct 31, 2026Updated last week
- LLM training code for Databricks foundation models☆4,462Mar 25, 2026Updated 7 months ago
- A modular RL library to fine-tune language models to human preferences☆2,400Mar 1, 2024Updated 2 years ago
- Large Language Model Text Generation Inference☆10,881Mar 21, 2026Updated 7 months ago
- Code for the paper "Evaluating Large Language Models Trained on Code"☆3,422Oct 16, 2026Updated 3 weeks ago
- ☆777Jun 13, 2024Updated 2 years ago
- A high-throughput and memory-efficient inference and serving engine for LLMs☆96,465Updated this week