Leaderboard

Both benchmarks evaluate terminal agents, but they measure different things. Terminal-Bench tasks are hand-crafted by researchers, which means they reflect what researchers think developers do rather than what developers actually do, and they cannot keep up with the pace at which real-world tooling evolves. TerminalWorld tasks are derived from 80,870 real developer recordings on asciinema, capturing authentic workflows by construction. The Pearson correlation between the two is only 0.20, indicating they assess distinct capabilities.

Yes. Every task originates from a real developer who completed it successfully, so the tasks are solvable by construction. We also validate each task before inclusion by running the reference solution through three rounds of automated testing to confirm the test suite is correct. The best current result is 62.5% (Claude Opus 4.7). No model has reached 100%, which reflects genuine difficulty rather than broken evaluation.

TerminalWorld covers 1,280 unique commands across 18 categories, reflecting the authentic diversity of real developer workflows. Many of these tools and patterns are absent from curated benchmarks, so models encounter genuinely unfamiliar territory. Performance drops sharply in demanding domains like performance optimization (28.1%) and scripting (39.1%). The tasks were not designed to be easy; they were designed to be real.

By pass rate on the 200 human-verified tasks (TerminalWorld-Verified). This subset was manually reviewed and cross-verified by four authors, making it the highest-fidelity evaluation core. All models are evaluated using the Terminus-2 agent framework at default thinking/reasoning effort settings.

Each task runs inside a sandboxed Docker container. Agents cannot browse the web or consult external resources to look up solutions. The evaluation tests what the model actually knows, not its ability to search for answers. Reference solutions are not publicly released, and tasks are graded on final system state rather than specific commands, so agents are free to reach the goal by any valid path. Each task page also embeds a harbor-canary identifier to help detect if task content is scraped for training.

Run your model using the Harbor evaluation harness (see the Quickstart page for step-by-step instructions), then open a pull request to the GitHub repository with your results. Results are verified and published within a week.