Terminus

Terminal-Bench 2.0: the most impt coding agent benchmark of 2025 gets a v2! Launch + Q&A w/ founders

Nov 8, 2025 · 35:01

Terminal-Bench co-creators Mike Merrill, Alex Shaw, and collaborators Andy Konwinski, Ludwig Schmidt, and Aatash launch Terminal-Bench 2.0, an 89-task coding benchmark with rigorous human and LM-assisted verification to test agent performance on real-world tasks like designing DNA primers and installing Windows 3.11. They reflect on Terminal-Bench 1's problems—easy tasks and reproducibility—and detail spending 300 hours manually auditing each task. They introduce Harbor, an open-source package for running sandboxed agent rollouts, standardizing task format and integrating with agents like Codex and Claude Code. Aatash shows using Harbor to generate thousands of training traces for DataComp Agent, running 1,000 parallel sandboxes. Future plans include expanding to scientific computing.

Terminal-Bench: Pushing Claude Code, OpenAI Codex, Factory Droid, et al to the limits

Oct 18, 2025 · 35:27

Alex Shaw and Mike Merrill, creators of Terminal-Bench, explain how their coding agent benchmark became an industry standard adopted by Anthropic, OpenAI, and leading agent companies. They bet on terminal-based interaction over GUI because text is the modality that works best with models, and designed tasks as containerized environments with instructions and test scripts. Nicholas Carlini at Anthropic became an early champion, leading to Terminal-Bench being featured on Claude’s model card without notice. They built the minimal Terminus agent to isolate model capabilities from agent optimizations, and their framework adapts existing benchmarks like SWE-bench. Future plans include cloud hosting, multi-dimensional evaluation incorporating cost and economic value, and enabling RL post-training on benchmark tasks.