The End of SWE-Bench Verified — Mia Glaese & Olivia Watkins, OpenAI Frontier Evals
Feb 23, 2026 · 27:10
OpenAI researchers Mia Glaese and Olivia Watkins explain why they are retiring SWE-Bench Verified, arguing the benchmark is saturated and highly contaminated, with models like GPT 5.2 regurgitating ground-truth solutions or task IDs. They detail how a contamination auditor agent found evidence across Claude Opus 4.5 and Gemini Flash, and how over half of unsolved problems had unfair tests (e.g., requiring specific function names not in the spec). The team is pivoting to SWE-Bench Pro from Scale, which features harder, longer tasks (1-4+ hours), more diverse repos and languages, and far less contamination. They advocate for future coding evals to measure open-ended design decisions, code quality, maintainability, and real-world product building, while also calling for more real-world usage metrics to track AI's impact on jobs and productivity.
