Terminal-Bench

[State of Code Evals] After SWE-bench, Code Clash & SOTA Coding Benchmarks recap — John Yang

Dec 31, 2025 · 17:45

John Yang, creator of SWE-bench, explains the benchmark's evolution from an ignored project to the industry standard after Cognition's Devin launch, and introduces CodeClash as a new paradigm for evaluating long-horizon coding agents via programming tournaments. Yang covers SWE-bench's expansion into nine languages (JavaScript, Rust, Java, C, Ruby) and multimodal tasks, the proliferation of independent variants like SWE-bench Pro and SWE-bench Live, and the shift from unit tests to iterative agent competitions. He addresses the Tau-bench controversy over impossible tasks, arguing they serve as cheat detection, and weighs the tension between long autonomy (5-hour runs) and interactivity (Cognition's fast back-and-forth). Yang highlights the academic data problem—companies like Cognition have rich user interaction data, while academics need user simulators or compelling products like LMArena—and positions CodeClash as a testbed for studying human-AI collaboration by freezing model capability and varying interaction setups.

Terminal-Bench 2.0: the most impt coding agent benchmark of 2025 gets a v2! Launch + Q&A w/ founders

Nov 8, 2025 · 35:01

Terminal-Bench co-creators Mike Merrill, Alex Shaw, and collaborators Andy Konwinski, Ludwig Schmidt, and Aatash launch Terminal-Bench 2.0, an 89-task coding benchmark with rigorous human and LM-assisted verification to test agent performance on real-world tasks like designing DNA primers and installing Windows 3.11. They reflect on Terminal-Bench 1's problems—easy tasks and reproducibility—and detail spending 300 hours manually auditing each task. They introduce Harbor, an open-source package for running sandboxed agent rollouts, standardizing task format and integrating with agents like Codex and Claude Code. Aatash shows using Harbor to generate thousands of training traces for DataComp Agent, running 1,000 parallel sandboxes. Future plans include expanding to scientific computing.

Terminal-Bench: Pushing Claude Code, OpenAI Codex, Factory Droid, et al to the limits

Oct 18, 2025 · 35:27

Alex Shaw and Mike Merrill, creators of Terminal-Bench, explain how their coding agent benchmark became an industry standard adopted by Anthropic, OpenAI, and leading agent companies. They bet on terminal-based interaction over GUI because text is the modality that works best with models, and designed tasks as containerized environments with instructions and test scripts. Nicholas Carlini at Anthropic became an early champion, leading to Terminal-Bench being featured on Claude’s model card without notice. They built the minimal Terminus agent to isolate model capabilities from agent optimizations, and their framework adapts existing benchmarks like SWE-bench. Future plans include cloud hosting, multi-dimensional evaluation incorporating cost and economic value, and enabling RL post-training on benchmark tasks.