Terminal-Bench: Pushing Claude Code, OpenAI Codex, Factory Droid, et al to the limits
Oct 18, 2025 · 35:27
Alex Shaw and Mike Merrill, creators of Terminal-Bench, explain how their coding agent benchmark became an industry standard adopted by Anthropic, OpenAI, and leading agent companies. They bet on terminal-based interaction over GUI because text is the modality that works best with models, and designed tasks as containerized environments with instructions and test scripts. Nicholas Carlini at Anthropic became an early champion, leading to Terminal-Bench being featured on Claude’s model card without notice. They built the minimal Terminus agent to isolate model capabilities from agent optimizations, and their framework adapts existing benchmarks like SWE-bench. Future plans include cloud hosting, multi-dimensional evaluation incorporating cost and economic value, and enabling RL post-training on benchmark tasks.
