Scale

The End of SWE-Bench Verified — Mia Glaese & Olivia Watkins, OpenAI Frontier Evals

Feb 23, 2026 · 27:10

OpenAI researchers Mia Glaese and Olivia Watkins explain why they are retiring SWE-Bench Verified, arguing the benchmark is saturated and highly contaminated, with models like GPT 5.2 regurgitating ground-truth solutions or task IDs. They detail how a contamination auditor agent found evidence across Claude Opus 4.5 and Gemini Flash, and how over half of unsolved problems had unfair tests (e.g., requiring specific function names not in the spec). The team is pivoting to SWE-Bench Pro from Scale, which features harder, longer tasks (1-4+ hours), more diverse repos and languages, and far less contamination. They advocate for future coding evals to measure open-ended design decisions, code quality, maintainability, and real-world product building, while also calling for more real-world usage metrics to track AI's impact on jobs and productivity.

How AI is Eating Finance - with Mike Conover of Brightwave

Jun 11, 2024 · 1:05:06

Brightwave founder Mike Conover explains how his vertical AI startup builds a 'partner in thought' for finance professionals, using a systems-of-systems approach that decomposes problems into specialized subsystems rather than relying on large context windows or monolithic agents. He argues that the evolution from DALL-E's 1,024-token context to today's million-token models has not solved synthesis; smaller, focused reasoning units still outperform. Conover emphasizes that the real competitive edge lies in custom training data and human annotation, not pre-training, and that models are converging in capability—so value shifts to fine-tuning data that elicits specific behaviors. He details why Brightwave avoids spreadsheets in favor of conversational analysis, and predicts AI will automate idea generation and second-order derivative bets, but humans must remain the final synthesizers and deciders. The episode also covers hiring for vertical AI, the role of knowledge graphs, and the diminishing economic incentive for companies to train their own foundation models.