Compendium of over 50 benchmarks for evaluating AI agents, categorized into Function Calling & Tool Use, General Assistant & Reasoning, Coding & Software Engineering, and Computer Interaction.
☆202Oct 15, 2025Updated last year
Alternatives and similar repositories for ai-agent-benchmark-compendium
Users that are interested in ai-agent-benchmark-compendium are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- Code and data for the paper: IntentionQA: A Benchmark for Evaluating Purchase Intention Comprehension Abilities of Large Language Models …☆12Apr 27, 2024Updated 2 years ago
- ☆42Jul 6, 2025Updated last year
- Code repository for the ICML 2026 paper "Reasoning over Boundaries: Enhancing Specification Alignment via Test-time Deliberation".☆24Jun 14, 2026Updated 4 months ago
- LatentLoop: Scaling Looped MoE Models with Asymmetric Compute Allocation☆15Oct 15, 2026Updated 2 weeks ago
- A suite of AI coding agent skills that build and maintain a shadow knowledge base for any codebase.☆30Oct 22, 2026Updated last week
- Deploy on Railway without the complexity - Free Credits Offer • AdConnect your repo and Railway handles the rest with instant previews. Quickly provision container image services, databases, and storage volumes.
- ☆19Oct 19, 2026Updated last week
- A practical guide to using SonarQube Agentic Analysis and Context Augmentation with Claude Code to catch and prevent code quality issues …☆21Apr 16, 2026Updated 6 months ago
- 🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.☆11Jun 26, 2026Updated 4 months ago
- CLI and MCP server for LinkedHelper automation — let AI assistants control LinkedIn outreach☆20Sep 10, 2026Updated last month
- LLM-as-a-judge using G-eval Scratch☆15Oct 12, 2025Updated last year
- AI studying companion, powered by LlamaCloud☆23Nov 24, 2025Updated 11 months ago
- ☆308Jul 1, 2026Updated 3 months ago
- An autonomous Kubernetes SRE agent. It monitors cluster health, diagnoses issues, and applies fixes — with human approval required before…☆59Oct 9, 2026Updated 3 weeks ago
- Experimental validation of the Bayesian BM25 paper☆10Feb 16, 2026Updated 8 months ago
- End-to-end encrypted email - Proton Mail • AdSpecial offer: 40% Off Yearly / 80% Off First Month. All Proton services are open source and independently audited for security.
- ☆40Jan 1, 2026Updated 9 months ago
- General plug-and-play inference library for Recursive Language Models (RLMs), supporting various sandboxes.☆17Apr 26, 2026Updated 6 months ago
- ☆19Oct 18, 2026Updated last week
- Codes for paper: "LaMP-QA: A Benchmark for Personalized Long-form Question Answering"☆13Jun 3, 2025Updated last year
- Progressive learning guide for Pi coding agent extensions☆22Jun 27, 2026Updated 4 months ago
- Detect hallucinations, repetitive bug fix (AKA. bottomless pit) and help AI coder's with access to documentations and suggest fixes.☆14Aug 15, 2026Updated 2 months ago
- Fastest one-click OpenClaw deploy on Render. Simple, easy setup. Powered by AlphaClaw.☆22Updated this week
- A collection of scripts and tools for analyzing SWE agents.☆18May 7, 2025Updated last year
- Utilities to connect Clojure with Cloud Foundry☆12Sep 20, 2012Updated 14 years ago
- 1-Click AI Models by DigitalOcean Gradient • AdDeploy popular AI models on DigitalOcean Gradient GPU virtual machines with just a single click. Zero configuration with optimized deployments.
- VideoAtlas/VideoRLM: Navigating Long-Form Video in Logarithmic Compute☆21May 19, 2026Updated 5 months ago
- A Comprehensive Benchmark to Evaluate LLMs as Agents (ICLR'24)☆3,794Feb 8, 2026Updated 8 months ago
- MCP-first local context retrieval for AI coding tools — BM25 over markdown + SQLite learned memory. Works with Claude Code, Codex, Gemini…☆14May 5, 2026Updated 5 months ago
- Affinity server and online console application☆12Jun 13, 2014Updated 12 years ago
- Railway skill for Claude Code - manage deployments via CLI☆30Jan 22, 2026Updated 9 months ago
- ☆24Jun 16, 2026Updated 4 months ago
- Skill Self-Play: Pushing the Frontier of LLM Capability with Co-Evolving Skills☆153Jul 27, 2026Updated 3 months ago
- LiveMCPBench is a benchmark for evaluating the ability of agents to navigate and utilize a large-scale MCP toolset. It provides a compreh…☆105Dec 18, 2025Updated 10 months ago
- Events manager for a docker cluster☆13Oct 12, 2014Updated 12 years ago
- Wordpress hosting with auto-scaling - Free Trial Offer • AdFully Managed hosting for WordPress and WooCommerce businesses that need reliable, auto-scalable performance. Cloudways SafeUpdates now available.
- Get aid from local LLMs right in your PowerShell☆17May 2, 2025Updated last year
- Lemonade Pi Agent CLI Plugin☆16Sep 12, 2026Updated last month
- ☆17Jul 29, 2025Updated last year
- Framework for evaluating and improving agents☆6,574Updated this week
- ☆170Oct 21, 2026Updated last week
- Unofficial self-hosted Cloudflare OS Home with Docker Compose, project-local LiteLLM, Tailscale, and an agent-first runtime.☆14Oct 3, 2026Updated 3 weeks ago
- Simple plugin that allows to add selected text as a card to your Anki deck☆13Jul 29, 2011Updated 15 years ago