openai / SWELancer-BenchmarkLinks
This repo contains the dataset and code for the paper "SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering?"
☆1,437Updated 5 months ago
Alternatives and similar repositories for SWELancer-Benchmark
Users that are interested in SWELancer-Benchmark are comparing it to the libraries listed below
Sorting:
- OpenAI Frontier Evals☆974Updated last month
- Agentless🐱: an agentless approach to automatically solve software development problems☆1,997Updated last year
- A benchmark for LLMs on complicated tasks in the terminal☆1,350Updated 3 weeks ago
- MLE-bench is a benchmark for measuring how well AI agents perform at machine learning engineering☆1,270Updated 3 weeks ago
- [NeurIPS'25] Official codebase for "SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution"☆654Updated 10 months ago
- ☆618Updated 4 months ago
- SWE-bench: Can Language Models Resolve Real-world Github Issues?☆4,115Updated last week
- Code and Data for Tau-Bench☆1,058Updated 4 months ago
- Code for Paper: Training Software Engineering Agents and Verifiers with SWE-Gym [ICML 2025]☆613Updated 5 months ago
- An agent benchmark with tasks in a simulated software company.☆622Updated last month
- The #1 open-source SWE-bench Verified implementation☆848Updated 7 months ago
- OO for LLMs☆887Updated last week
- Official Repo for ICML 2024 paper "Executable Code Actions Elicit Better LLM Agents" by Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhan…☆1,535Updated last year
- Windows Agent Arena (WAA) 🪟 is a scalable OS platform for testing and benchmarking of multi-modal AI agents.☆808Updated 8 months ago
- Code for "WebVoyager: WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models"☆993Updated last year
- Renderer for the harmony response format to be used with gpt-oss☆4,135Updated last month
- Sandboxed code execution for AI agents, locally or on the cloud. Massively parallel, easy to extend. Powering SWE-agent and more.☆404Updated 2 weeks ago
- Post-training with Tinker☆2,719Updated this week
- [NeurIPS 2025 D&B Spotlight] Scaling Data for SWE-agents☆514Updated this week
- LDB: A Large Language Model Debugger via Verifying Runtime Execution Step by Step (ACL'24)☆575Updated last year
- 🌎💪 BrowserGym, a Gym environment for web task automation☆1,082Updated last week
- End-to-end Generative Optimization for AI Agents☆704Updated last month
- AgentLab: An open-source framework for developing, testing, and benchmarking web agents on diverse tasks, designed for scalability and re…☆494Updated this week
- Our library for RL environments + evals☆3,730Updated this week
- The 100 line AI agent that solves GitHub issues or helps you in your command line. Radically simple, no huge configs, no giant monorepo—b…☆2,530Updated this week
- Atropos is a Language Model Reinforcement Learning Environments framework for collecting and evaluating LLM trajectories through diverse …☆825Updated this week
- ☆483Updated 5 months ago
- Official repository for the paper "LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code"☆762Updated 6 months ago
- E2B Desktop Sandbox for LLMs. E2B Sandbox with desktop graphical environment that you can connect to any LLM for secure computer use.☆1,219Updated last week
- τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment☆631Updated 3 weeks ago