# The End of SWE-Bench Verified — Mia Glaese & Olivia Watkins, OpenAI Frontier Evals Latent Space · 2026-02-23 OpenAI researchers Mia Glaese and Olivia Watkins explain why they are retiring SWE-Bench Verified, arguing the benchmark is saturated and highly contaminated, with models like GPT 5.2 regurgitating ground-truth solutions or task IDs. They detail how a contamination auditor agent found evidence across Claude Opus 4.5 and Gemini Flash, and how over half of unsolved problems had unfair tests (e.g., requiring specific function names not in the spec). The team is pivoting to SWE-Bench Pro from Scale, which features harder, longer tasks (1-4+ hours), more diverse repos and languages, and far less contamination. They advocate for future coding evals to measure open-ended design decisions, code quality, maintainability, and real-world product building, while also calling for more real-world usage metrics to track AI's impact on jobs and productivity. ## Questions this episode answers ### Why is OpenAI no longer reporting SWE-Bench Verified results? Olivia Watkins explains that the benchmark is “effectively saturated and also highly contaminated,” so further score improvements are meaningless. Models now achieve around 80%, and many remaining failures stem from overly narrow tests—like requiring specific function names—not genuine capability gaps. Evidence also shows models recall repository details or task IDs, indicating widespread contamination across frontier models. [1:08](https://addtry.com/f0301c56-cfcd-4121-b9f8-5578223747cd?t=68000) ### What evidence of contamination did OpenAI find in SWE-Bench Verified? Olivia Watkins details finding GPT 5.2’s chain of thought referencing a later repository implementation to suggest an unmentioned argument, revealing prior exposure. A “contamination auditor agent” also uncovered multiple models—including Claude Opus 4.5 and Gemini Flash—regurgitating ground-truth solutions or knowing task IDs, which she calls “pretty clear evidence” of contamination. [5:38](https://addtry.com/f0301c56-cfcd-4121-b9f8-5578223747cd?t=338000) ### What is SWE-Bench Pro and why is OpenAI switching to it? SWE-Bench Pro, built by Scale, is a harder, more diverse benchmark with tasks spanning 1–4 hours and 4+ hours, multiple repositories, and languages. Olivia Watkins says it is not saturated, and contamination testing with their auditor agent found only “very light evidence” of familiarity with one or two repos, making it a better gauge of real coding progress. [10:57](https://addtry.com/f0301c56-cfcd-4121-b9f8-5578223747cd?t=657000) ## Key moments - **[0:00] Intros** - [1:08] SWE-Bench Verified has become saturated and contaminated, no longer effectively measuring coding progress, says Olivia Watkins. - **[1:47] Building Verified** - [2:56] OpenAI hired almost 100 software engineers to review SWE-Bench tasks and found that many failures stemmed from unfair problem setups. - [3:14] "It's hard to overstate the amount of effort" — Mia Glaese on the three independent expert reviews required per SWE-Bench Verified task. - **[4:32] Contamination** - [5:38] GPT 5.2's chain of thought recalled an argument from a later repository version, revealing contamination, reports Olivia Watkins. - [6:04] Mia Glaese: Olivia's contamination discovery triggered a company-wide investigation into contamination across frontier models. - **[6:16] Unfair Tests** - [7:26] Over half of the deep-dived SWE-Bench problems had overly narrow tests, like requiring specific function names not in the task description. - [8:08] "Just because you didn't pass a test doesn't mean your implementation wasn't a good one" — Mia Glaese on unfair test narrowness. - **[8:40] Saturation** - [8:45] Olivia Watkins: Finding unfair tests is easier when examining a smart model's failed solutions than when reviewing problems in the abstract. - [9:00] Mia Glaese: Once benchmarks hit 80%+ performance, incremental gains become meaningless, measuring trivia like function naming rather than coding ability. - [10:03] Q: SWE-Bench Verified's ceiling? Olivia Watkins: GPT 5.2 solved 31 problems suspected to require contamination, suggesting the benchmark is effectively saturated. - **[10:28] SWE-Bench Pro** - [10:57] SWE-Bench Pro is harder: 90% of Verified tasks take under an hour, while Pro includes longer 1-4 hour and 4+ hour tasks across more languages and repositories. - [11:55] OpenAI's contamination auditor agent found memorized task IDs and ground-truth solutions in Claude Opus 4.5 and Gemini Flash on SWE-Bench Verified. - **[12:31] Ideal Evals** - [12:42] Mia Glaese: The field must continually find harder and more representative problems as benchmarks like SWE-Bench Verified become saturated. - [13:23] Olivia Watkins: Future coding benchmarks should include open-ended design decisions to test if models can make reasonable choices beyond narrow specs. - [14:32] Mia Glaese: Beyond pass/fail, evals should measure design taste, code quality, and maintainability on tasks lasting hours or days, not minutes. - [15:51] Q: Should grading rely on humans or LLMs? Mia Glaese points to GDP Eval, which hired domain experts to create rubrics and gold solutions. - [16:23] GDP Eval measures real-world white-collar work across 16 professions by hiring professionals to write tasks and validate solutions, explains Olivia Watkins. - [17:35] Mia Glaese: Test-based grading like SWE-Bench is simple but fails to answer whether a pull request would actually be merged by a project maintainer. - **[18:17] Beyond Pass/Fail** - [18:53] Q: Should AI self-improvement evals be part of coding benchmarks? Olivia Watkins notes that proprietary AI codebases make realistic public evals harder. - [20:03] Public AI coding benchmarks are rare because state-of-the-art AI codebases are proprietary, limiting realistic evaluation, says Olivia Watkins. - [20:45] Mia Glaese: Dollar-denominated benchmarks like Sweet Lancer and time-based ones are proxies for task complexity, not fundamentally different. - [21:37] Meter's long autonomy eval uses time as a proxy for complexity, an approach OpenAI collaborates on and appreciates, says Mia Glaese. - **[21:49] Preparedness** - [22:36] Q: How does OpenAI's preparedness framework work? Olivia Watkins: It tracks bio risk, cybersecurity, and model autonomy, with coding evals supporting autonomy. - [23:30] SWE-Bench Verified was built for OpenAI's model autonomy track; now the company shifts toward evals for automating full research workflows. - [24:02] "We really appreciate other people and the entire field building evals and sharing them" — Mia Glaese urging more public benchmark creation. - [25:07] Olivia Watkins calls for evals of tasks taking top engineers months, end-to-end product building, and real-world AI usage and employment metrics. - [26:25] OpenAI Frontier Evals will focus more on real-world impact, measuring actual AI usage, job replacement, and augmentation, says Olivia Watkins. ## Speakers - **Mia Glaese** (guest) - **Olivia Watkins** (guest) ## Topics Benchmarks ## Mentioned Meter (company), OpenAI (company), Scale (company), ALPA benchmarks (product), Claude (product), GDP eval (product), GPT (product), Gemini (product), SWE-Bench (product), SWE-Bench Pro (product), SWE-Bench Verified (product), Sweet Lancer (product), Vending Bench (product), o3 (product) ## Transcript ### Intros **Host** [0:04] Okay. Hi, we're here in the OpenAI studio with Mia and Olivia from the Frontier Evals team, or, uh, however you want to introduce yourself. Maybe Mia, you want to introduce, uh, name what you do at OpenAI, and we can get get started. **Olivia Watkins** [0:18] Um, sure. Hi, I'm Olivia. I'm on the Frontier Evals team. **Host** [0:21] Great. Sure. And Mia? **Mia Glaese** [0:22] Hi, I'm Mia. I am a VP of research at OpenAI, um, and my teams are the Codex team, the Human Data team, and, uh, the Alignment team, and we work a lot with Olivia's team on Frontier Eval. **Host** [0:36] Yeah. Uh, very exciting. And, uh, as by my understanding, you were part of the original team that worked on SWE-Bench Verified as well. **Mia Glaese** [0:42] Yeah. Uh, Olivia's team, the Frontier Eval team, and the Human Data team collaborated on creating SWE-Bench Verified. **Host** [0:49] So you've, you've seen the evolution of coding benchmarks over time, and I, uh, I think it was roundabout sort of mid to late 2024 when you first published SWE-Bench Verified. Things have evolved a lot since then. What's the blog post that you have worked on that you're, that we're releasing today? Like, what, what is the sort of con- what is the main thesis that you're pushing out? **Olivia Watkins** [1:08] So the main thesis is that S- uh, SWE-Bench Verified has been one of the North Star coding benchmarks that the field has looked at to measure coding progress. Uh, but recently we've seen that progress has kind of stalled, and this we realized that this is because the eval is effectively saturated and also highly contaminated. So at this point, we think that, um, it's not really measuring coding performance improvements well anymore, and we think that the field should move away from this towards other benchmarks. **Host** [1:33] Like SWE-Bench Pro. **Olivia Watkins** [1:34] Like SWE-Bench Pro, yeah. **Host** [1:35] Amazing. Yeah. I, um, one of the jokes I always have is, like, there's a group chat with all the labs, and everyone just takes turns to increment, like, 0.1 on charts. And then it's like, "Okay, well, you have the best coding model, I guess, because you're 0.1% higher," but it's not super convincing at this point. ### Building Verified **Olivia Watkins** [1:50] No. **Host** [1:50] Yeah. **Olivia Watkins** [1:51] No. **Host** [1:51] So cool. I think the, uh, let's, let's sort of reset on, like, what was your init- original work that you guys did for SWE-Bench Verified, which I think was pretty substantial. Like, it was, like, a very, uh, significant investment for an OpenAI, which, like, people still don't appreciate. And then what were the dissatisfactions that we, that we found over time, right? So l- like, what, what was SWE-Bench Verified? Or should, should, um, that people should know about. **Olivia Watkins** [2:14] SWE-Bench Verified was a kind of a cleanup of original bench- academic benchmark from a lab at Princeton called SWE-Bench, and the agent is basically given a code base and a task that was sourced from a real world, uh, repository and GitHub issue, and was asked to solve a task, and is graded on whether some tests pass. And at the time, this was, uh, th- quickly became a popular benchmark because at the time the field didn't really have good real world coding benchmarks. But then when OpenAI took a look at the benchmark as part of one of the evals we wanted to track in our preparedness framework, folks, uh, started realizing that some of the cases where agents were failing were due to bad problem setups rather than just to models being dumb. So folks at OpenAI did a pretty extensive human data campaign hiring, like, almost 100, um, real world, uh, software engineers to go through, um, the problems and figure out, like, are the tasks well specified, are the tests actually fair, and kind of created a curated set of, like, 500 tasks that we thought were much better. **Mia Glaese** [3:14] It's just, it's maybe it's hard to, to overstate, like, the amount of effort that it took to, like, create that benchmark. It was literally, like, many expert software engineers reviewing the problems, like, sequentially multiple times. Uh, and to, you know, basically, like, three different experts independently decided that- **Host** [3:38] Yeah **Mia Glaese** [3:38] ... that the- **Host** [3:38] You didn't have to do that. You just tripled your cost for just- **Mia Glaese** [3:41] I mean, we had to do it. We had to do it actually because it's quite a hard task to, like, look at something like a, a problem and, and the patch and then, like, it's not just the problem and the patch, right? You have to, like, understand it in the context of the code base that the human or, or the model is entering to solve the task. So it's a very complex problem, and it was definitely needed to have three reviews, and I think, like, maybe we should have done more . But, uh, it was definitely, uh, a lot of effort to get there. **Host** [4:10] Yeah. Um, and there's, there's more, but people can read the, the blog post for that. Uh, I will note that you guys set a trend in verifying benchmarks because I just recently saw, I think Quinn had a HLE verified for Humanities License Verified. **Olivia Watkins** [4:23] Yeah. **Host** [4:23] Which, like, so now everyone's verifying everything, which is nice and good and, like, extra quality there. Um, okay, so but the, I think the, the meat of it is the, the, this, this was a lot of like, well, here's the issue or problem statements, and then here's the, here's the diffs, here's the golden, uh, tests, and here's some regression tests, right? ### Contamination **Host** [4:40] That's, that's like the rough setup of these 500 problems. Um, and there's some contamination always happens because all of SWE-Bench Verified was fully open, I think. Um, you, you did have canaries, but, like, you know, stuff, stuff leaks. **Mia Glaese** [4:52] There's like multiple avenues that, like, the problems are sourced from open source repos. **Host** [4:58] Yes. **Mia Glaese** [4:59] So it's not just like when we usually publish evaluations, we publish evaluations and then we, we add canary strings to ensure that, you know, they are easily filtered out at training time. Obviously, if you use sort of like data from like open source- **Host** [5:16] Just GitHub **Mia Glaese** [5:17] ... you don't have actually like a ca- canary string in that. **Host** [5:20] Yeah. And you- **Olivia Watkins** [5:21] And these are also, like, some of these are very popular repos, like the Django repository. So you're gonna see like many instances being used kind of throughout GitHub. **Host** [5:29] Yeah, yeah. And you, you, uh, just before recording, you were telling me that you found this in your own chain of thought with, like, GPT 5.2 also seeing that, like- They had an extra knowledge or something **Olivia Watkins** [5:38] Uh, yes. So this was an example where the task, um, asked the agent to, uh, influence something, but it wasn't told that there was this specific argument that the test was going to be looking for it using. But in the GPT 5.2 chain of thought, we actually saw instances of the model reasoning like, "Hey, I think that at some later version of this repository they implemented this particular argument. Maybe I should add it in." **Host** [5:59] Yeah. **Olivia Watkins** [5:59] So this is an example of a test that, like, would be pretty impossible to pass without this contamination knowledge. **Host** [6:04] Yeah. **Mia Glaese** [6:04] And I think you found that sort of first, right, and it triggered like a whole investigation both, like, in our own models- **Host** [6:10] Oh **Mia Glaese** [6:10] ... and also in, uh, other frontier models, like, in the market and, like, understanding how contaminated the benchmark is, like, across the industry. ### Unfair Tests **Host** [6:20] What else did you find? I mean it's... I have to double click on this. **Olivia Watkins** [6:24] So we... And when I say we, this is mostly- **Host** [6:26] Yeah **Olivia Watkins** [6:26] ... work from other folks at our team, not... I don't wanna take credit- **Host** [6:28] Not the Mario we. **Olivia Watkins** [6:29] Yes. Um, but so we did, um, some analysis on, first of all, are the tests actually fair? And so this happened by first, um, taking all the problems that o3 couldn't solve reliably, and then again, uh, getting a lot of humans to do basically another pass of, uh, kind of digging into, you know, what's wrong. **Host** [6:49] Is it the same exact analysis, or were they reading o3's output and going, "Here's where o3 went wrong"? **Olivia Watkins** [6:55] I think it was... I mean, it was definitely like a scope to the set of problems that models failed. And I believe they were able to look at, like, what the model solutions look like versus what the- **Host** [7:04] So this isn't the same work as the original- **Mia Glaese** [7:06] It's not exactly the same work. It was like a, a deeper dive. It's like, okay, which are the problems that we don't see any model solving? It's like, is there something fundamentally wrong with those problems or is there something, you know, wrong with, like, are the models just not smart enough to solve the problems? So that's kind of like what we, what we dug into. **Host** [7:24] Yeah. And you found some. **Olivia Watkins** [7:26] Oh, yes. Like, uh, in over half of the problems that were investigated in that deep dive, there was one problem or the other. I think the most common problem are, like, overly narrow tests where there's some particular implementation detail that, um, the tests were looking for but wasn't specified in the problem description, so it wasn't fair to expect that model to make that particular design choice. Like, one pretty blatant example are cases where the task asks you to implement some feature, um, and the tests are looking for you naming that argument or that function with a particular name, but if you na- chose another reasonable name, the test would fail. **Host** [8:00] Yeah. **Olivia Watkins** [8:01] And another set of, um, types of, um, bad tests are tests that are just looking for additional features that were never mentioned in the problem description. **Mia Glaese** [8:08] But, but that's significant. It's like, that means that if you pass a test, actually, like, you probably did, like, a really good job, but just because you didn't pass a FET test doesn't mean that your implementation wasn't, like, a good one, right? So it was just like we only accept, like, very narrow versions of solutions and, like, not the whole space- **Host** [8:27] Yeah **Mia Glaese** [8:27] ... of, of, like, viable and sort of, like, good solutions to the problem. **Host** [8:33] Yeah. I think it's important that you're doing this because it... in some way it is you in twenty twenty five, six going back in time and correcting your own work, right? 'Cause you could have caught all this in ver- in the original verified work. ### Saturation **Olivia Watkins** [8:45] I think so. It's definitely much harder to find a problem in the abstract than when you're looking at a very smart agent's best effort solution and trying to compare it. **Host** [8:53] It is harder or harder? Okay. **Olivia Watkins** [8:55] Sorry. It's much easier when you have it. **Host** [8:56] Much easier. Exactly. **Mia Glaese** [8:57] Yeah. **Olivia Watkins** [8:57] Sorry. **Mia Glaese** [8:57] I think, I think also, like, at the time when SWE-Bench Verified was published, I think it was, like, a very strong benchmark. It's not like we're, we're like, "Oh, this is not... This wasn't, like, a strong benchmark at the time." I think this is something that a lot of benchmarks go through, like, as an evolution, right? Like, when they start to become, like, popular and, like, viable it's because they measure something, like, important, and models maybe do, like, twenty percent correct on them, sometimes even less. And sort of, like, people have something to hold on and, and improve models on, on these benchmarks. And by the time that you hit, like, very high performance on the benchmarks, like, additional, like, point one percent improvements become sort of, like, meaningless. And so, like, at the time, I think, you know, that benchmark was, like, super valuable and it, it taught, like, us and, like, the industry a lot. It's just, like, now at the point that we are at now where models are as strong as they are now, we're kind of starting to measure not necessarily, like, what we want to measure, which is, like, coding capability of our agents, but, like, the agent's ability to, like, correctly guess how to name a specific function. **Host** [10:03] Yeah. **Mia Glaese** [10:03] And- **Host** [10:04] It's- **Mia Glaese** [10:04] ... and that isn't really what we are, like, want to measure at this point. **Host** [10:08] Yeah. I, I think that's fair. Is, is there... I mean, if I, if I asked you to ballpark it, like, uh, most models are... most frontier models are now, like, eighty something. Is there, like... What's the actual, like, number on SWE-Bench Verified that you did you guess as, like, the ceiling or ... **Olivia Watkins** [10:25] I guess it's really hard to say. Like, I... Uh, when GPT 5.2 came out, um, folks took a look and found that it was solving, like, thirty-one problems that were in the set of should be very hard to solve without contamination problems. ### SWE-Bench Pro **Olivia Watkins** [10:38] So I think it's quite possible that that number is already something that we've hit if you didn't have contamination at all. **Host** [10:44] Fair enough. **Olivia Watkins** [10:44] Hard to say, though. **Host** [10:45] Yeah. Uh, cool. We're gonna stop reporting SWE-Bench Verified, right? And then, uh, SWE-Bench Pro will, will be sort of the next one, which is an effort from Scale. What's your sort of comparison analysis? What's, what attracts you to SWE-Bench Pro? **Olivia Watkins** [10:57] Uh, the first one I think is just that it's harder. Um, for SWE-Bench Verified, uh, I think, uh, something like ninety percent of the problems are things that were estimated to take, like, an expert software engineer, um, like, less than an hour. They're, like, very well specified, very self-contained, and the, um, SWE-Bench Pro problems are just, um, bigger and harder, and there's much more headroom on the eval because it's not saturated. Um- **Host** [11:17] Yeah, like categories of, like, one to four hours and four plus, so something- **Olivia Watkins** [11:21] Yeah, and it's more diverse. Um, lots of r- r- repositories, multiple languages, m- qualitatively more different types of problems. So all that's great. Uh, on the contamination side, we also think it's better there So the way we, uh, were measuring for contamination for SWE-Bench Verified was, uh, with this little, like, contamination auditor agent, which is given the description of the task and the patch and the task ID and told to go take this target model and kind of as an open-ended, like, set of questions, try to, uh, find questions that will manage to kind of reveal what contamination might be lurking in that model. And in SWE-Bench Verified, we found many instances of contamination across, um, like, across open AI models, across like Claude, uh, Opus 4.5, Gemini Flash. And in all of these, we saw things like regurgitating the ground true solutions, things like, um, in some cases, um, giving like the task IDs and other things that are, uh, pretty clear evidence of at minimum familiarity with the repositories. **Host** [12:20] Memory, yeah. **Olivia Watkins** [12:21] Um, so we've been- **Mia Glaese** [12:21] I mean, with task ID, that's, that's- **Olivia Watkins** [12:24] Yeah. Yeah. Uh, and SWE-Bench Pro on the other hand, uh, we don't see this. I think they're, uh, the auditor agent found some like very light evidence that maybe a couple of models might be very lightly familiar with like one or two of the source repositories, but it's very different than SWE-Bench Verified. ### Ideal Evals **Olivia Watkins** [12:40] Um, so yeah, less contamination is good. **Mia Glaese** [12:42] I think there also like we should expect that at some point like that, that's not going to be like the right benchmark anymore and like it's a field we kind of have to continue to like move on and like find harder and more representative problems- **Host** [12:56] Yeah **Mia Glaese** [12:56] ... that we can measure our capabilities on. **Host** [12:58] Awesome. So let's go into that. Uh, I think that there are a lot of... I, I think we also prepped this in the, in the pre-chat was, well, people feel a qualitative difference when they're using 5.1 to 5.2 to 5.3 and it's not super expressed in the these benchmarks because they're, they are on a number of these things. What capabilities do you really want to benchmark in a, in a ideal coding benchmark? You know, I guess like agentic coding benchmark, whatever you call it, uh- **Olivia Watkins** [13:23] I mean, one thing is, uh, kind of open-ended design decisions, places where the problem maybe is a little bit underspecified and seeing if the model can make reasonable design decisions. **Host** [13:33] What's a reasonable prompt for that? Like just five code me a, a B2B SaaS and make no mistakes or you know? That's, that's the meme, but like, okay, what's like, what's like an actual usable, uh, open-ended problem like that? Like- **Olivia Watkins** [13:46] Uh, sure. I mean, maybe an example could be finding a way to speed up a particular part of a code base, but there might be like multiple different ways to speed it up. **Host** [13:54] Yeah. There are dedicated performance benchmarks. I think you guys have one, SWE 效率 , or is that, is that... Oh, no, I think that's, that's, uh, Ophir Harpaz's group. Um, but yeah. Yeah. I mean, that, that, that is a good one. **Mia Glaese** [14:04] I think there's just many, many things that people like value about working with, with software engineering agents. I think SWE-벤- SWE-Bench Verified obviously measure like some, it measures like some important capability which is like given like a description of a GitHub issue, can you produce like a patch that solves that issue, you know, satisfactorily? And like obviously there's like some issues with the, with the benchmark that means that now that we're at like eighty percent we don't really trust like further improvements on it. But like it does measure something that is like a real, real, uh, like capability of models. But I think as a field we're like moving beyond sort of, you know, can my coding agent like solve a small like GitHub issue for me, right? And so we are starting to look at like much more lo- longer term tasks, right, like that don't take like fifteen minutes but maybe like hour, sometimes days. And then beyond sort of like what kind of tasks can my a- agent solve, like there might be things that are kind of a bit harder to grasp, right? Like Olivia talked about sort of like does it have like design taste, right? Like, does it solve the problem the way that, you know, my team likes to solve problems? **Host** [15:26] Got it. **Mia Glaese** [15:26] The... Is the code nice, right? Like is it, is it well written? Like the, is it sort of like clean code, right? Like people care about this. Is it maintainable in the future? People care about a lot of these l- maybe less tangible, um, less tangible and like harder to measure frankly things that, that are still like super meaningful for people that are working with coding agents. **Host** [15:51] Yeah. So, um, I mean, these are all qualities that are obviously the no longer the low-hanging fruit like we have no idea how to evolve this. I think the, the, the simple question maybe the, the, that there's sort of two forks in the road. One is the sort of very human intensive, money intensive path which is hire a bunch of contractors and try to annotate this. The other is use an LLM to, to proxy it and try to align the LLM so that it can give you a reasonable proxy. Which of those would you want? I... You want, you wanna do both? **Mia Glaese** [16:19] I think like maybe you should talk about GDP eval as like an example. **Olivia Watkins** [16:23] Um, sure. So GDP eval is an eval that, um, was a- again produced by a collaboration between Human Data team and the Frontier Evals team and it's trying to measure whether agents can do kind of a variety of like real world white collar work. Um, that was an eval where grading is very hard. Uh, it requires, uh, kind of a lot of kind of like domain knowledge on exactly what, um, are you looking for in each different context. **Host** [16:50] Yeah. Across like fifteen, sixteen white collared jobs, uh, professions like I- that take up a significant part of GDP which is great- **Olivia Watkins** [16:57] Yeah. Kind of high level professions and then a lot of like different granular sub-professions. **Host** [17:00] Uh, I have said like I, I'm a big fan. Uh, it's- **Olivia Watkins** [17:03] Cool **Host** [17:03] ... it, it is, this is the eval for, for AGI basically. Like- **Olivia Watkins** [17:07] But partly because it was, um, so hard to... It required, uh, so much kind of like domain knowledge, um, uh, uh, that the Human Data team hired like a lot of, um, people from these professions to be very involved in creating tasks and creating the gold solutions and trying to, um, help create rubrics and so forth so we can grade it reliably. **Host** [17:27] So basically take the GDP eval which is a general listing, take that same approach to apply it to code and you roughly have like a, a rough roadmap. **Mia Glaese** [17:35] I think it's an interesting- **Host** [17:37] Yeah **Mia Glaese** [17:37] ... solution. I think what you're pointing out is an important problem which is sort of this- This like how realistic is it? And like do you know, what, what we wanted was like coding agents should write code that, you know, we think is good. And so like asking human, that's actually like a, a good way to ensure that. It's also kind of a slower, like complex way to do that. And so part of why I think, you know, SWE-Bench Verified ended up being super popular and where we are seeing like ALPA benchmarks like this being super popular is like it's very easy. It could even be easier, but like validating that a solution passes all the tests, it's like fairly trivial once you can like run the tests in, in like your, on your computer or wherever you're running them, and you can kind of like, okay, is it correct or is it not correct? ### Beyond Pass/Fail **Mia Glaese** [18:26] And you can kind of aggregate that, and that is super simple, but it doesn't tell you it's like, you know, did the model like solve the problem? Like, well, like, you know, ugly, but like what if actually like an open source maintainer of that project have like merged that PR? Like that it doesn't tell you. But there is a lot of value in having benchmarks that are both like easy to compare across the industry and also that can be sort of run really fast without human involvement. **Host** [18:53] Yeah. Amazing. Uh, your teams also put out other kinds of evals that are related, like the... I think there's an RL, uh, paper bench, um, and then they- they're sort of like the more sort of re- recursive im- self-improvement type, uh, evals. How much should that figure into mainstream coding evals? You know, like, uh, is there, is there some way in which those things join together? **Olivia Watkins** [19:17] So we were asking like, should we build e- should we also be building evals for the self-improvement evals, or are you saying do coding evals currently cover that? Am I- **Host** [19:25] Um, I think, I, I, I just think like those are some of the most advanced evals that we have, and we're not using them in the normal path, and it's just, uh, it's an interesting, uh, split between, well, here's evals f- for coding normal things, and then here's the one for machine learning that is like completely different, right? Yeah, I think you and Mia get what I mean. Uh, and that's mostly like a safety argument, I guess. Uh, but also like it's actually really useful for people to understand if the model is really good at like AI code, basically. **Olivia Watkins** [19:55] Yeah. Oh, yeah. Like my guess is that part of the reason that a lot of benchmarks so far haven't focused as much on the AI coding is just a question of like what data sets are easy to gather. **Host** [20:04] Yeah. **Olivia Watkins** [20:04] Because a lot of the like, you know, state-of-the-art, um, AI code bases are proprietary. So, um, if we make evals for that, like we're probably not gonna release them. **Host** [20:13] Okay. **Olivia Watkins** [20:13] And it's a lot harder for people in the field to make evals that kind of measure like is this a realistic research coding workflow. I do think that it's good for the field to try to measure these skills in a public way. I think it's just harder to make it realistic. **Host** [20:25] Sure. And then one more thing that a lot of people are trying to do, which is like sort of, well, in- instead of like a percentage of zero to one hundred, maybe we re-do- reddenominate in dollars, right? So you had Sweet Lancer and all that. Uh, other people are doing like Vending Bench, whatever. Any, any alpha in those or are they, are they you, you still want like a traditional academic benchmark? **Mia Glaese** [20:45] I think in a way, like there's like different ways to measure the same thing, right? If we're like, "Oh, this is like how much money it produces." **Host** [20:53] Yeah. **Mia Glaese** [20:53] It's a fairly similar thing to saying like, "Oh, this problem would take like a human, you know, two hours to solve," or something like that. Usually they, they're like fairly like correlated, right? Like however, you know, much it would take like a human to solve that problem kind of determines the value that we ascribe like a solution. And so I, I do think that is like an important thing is like h- how complex and how sort of long running are the tasks that we are like able to entrust our agents with. **Host** [21:25] Yeah. **Mia Glaese** [21:26] And so I think that that's like an important piece, but I, I think here sort of monetary value, time, or complexity, they all kind of like try to capture like a similar thing. **Host** [21:37] Yeah. Okay. So they're, they're all proxies for some amount of increasing ca- capacity that we wanna measure. I think that's a good thing. I think the only other sort of major player in this field is Meter, which has done the sort of long grass, and congrats, you guys have completely destroyed the curve for that. ### Preparedness **Host** [21:51] Any takes on that? Obviously you come up really well, so like, it, it looks good, but, uh, I don't know if like that approach is something that you wanna incorporate in your own work in making evals. Uh, this is the long autonomy task, uh, eval if you're- **Olivia Watkins** [22:04] Yeah. **Mia Glaese** [22:04] Yeah, no, we're familiar and, and, and we, we, we work with, with Meter on these evaluations, so like we, we do appreciate them. I think then they're using time, right? They're not using money. So I think like that was your question. I think like complexity, however we can sort of like quantify it, is really important to understand like where our models are, are, are getting to. **Host** [22:24] Okay. Com- complexity is the abstract thing, and then it projects down to time, projects down to story points, whatever, dollars. Uh, great. Um, one last question on just like, just the overall preparedness framework is, uh, you know, I was actually kind of looking at... People mention the preparedness framework a lot. Uh, I don't think it's well explained to a lot of people. Uh, and you actually have a nice website where it's like, uh, um, I think it's like test and like, uh, inform and, uh, teach something. And I, I feel like you, you actually do a lot of work there and, and I don't know if you wanna talk about how the preparedness framework applies. **Olivia Watkins** [22:55] So the preparedness framework is, uh, OpenAI's kind of like public framework for how we track, uh, frontier risk. So these are kind of capabilities that are typically dual use, like you can use them for good things or bad things, but we wanna at least keep an eye out for the bad things to make sure that we ha- both we as a company and like the broader society are kind of prepared to handle the potential downsides. And so at the moment, we kind of track three different categories. Um, one is kind of, uh, bio risk, another is cybersecurity, and a third is kind of, uh, research automation and model autonomy, and that's kind of what ties most into the SWE-bench- **Host** [23:30] Got it **Olivia Watkins** [23:30] ... where, uh, coding is not all of automating research, but it is one very important key component. And so, uh, we initially created SWE-bench Verified as part of our like building out evals for that model autonomy work stream. Uh, now, uh, I think, uh, for like we have to move beyond that towards looking more at like can models actually start to actually automate research workflows. **Host** [23:52] Yeah. Amazing. Great. Um, Mia, anything else to add on just the general what people should know about preparedness and, uh- How evals and human data alignment all work together in that **Mia Glaese** [24:02] I think maybe the thing that I would say is that we really appreciate... We, we work really hard to build these evals and, and so we, that's why we published SWE-Bench Ver-Verified, and that's where we're, like, sharing GDP Val, these sorts of things. We also deeply appreciate, like, other people and the entire field to kind of build evals and, and share them, and we use them. Like, SWE-Bench Pro, we're like, "Yes, that, that's a better eval now. We should use them." So would really encourage people to find more ways to create and share evals that we can, we and the entire field can use to measure, like, progress- **Host** [24:40] Yeah **Mia Glaese** [24:40] ... on, on, on, like, a variety of capabilities, including, including coding, because it's important to understand sort of where we are. **Host** [24:47] Mia had to leave, uh, but we're, we're just kind of, uh, talking a little bit about, like, the f- the future directions that we want evals to go. **Olivia Watkins** [24:53] Mm-hmm. **Host** [24:53] Um, and I, I think here, here we can dive in on, like, give us work, good work on these, these, these things, and we'll talk to you. You know, uh, uh, here's your platform to make a call for what you're looking for. **Olivia Watkins** [25:07] I think a few things that would be useful, I'd say first of all, really, really hard tasks. Like, the kinds of things that would take top-notch engineers months or teams weeks, um, would be quite good. Um, especially if grading is reliable and grading is, uh, like, you know, you have, for example, like, rubrics that have been, uh, sourced and validated by many people in the field. I think that'd be quite valuable. I think also benchmarks on kind of creating products end to end. I think as people are micro targeting more, that would be quite useful. I think, uh, a third thing that I'd say that is maybe not quite an eval, but I think is still relevant to the kind of overall mission of, like, we as a field and as a world should be tracking, like, where are these capabilities going. I'd like to see more metrics, um, tracking, like, real-world usage. Like, how much is AI actually being used in the field? Um, uh, how much is it, you know, replacing people's jobs? How much is it, um, you know, augmenting people, speeding people up? Just, like, real-world metrics. Yeah. **Host** [26:03] Yeah. The, the, the replacement thing is always, like, a sensitive one on the, on the sort of PR side of things. But, uh, you know, we create new jobs that, that manage the old jobs, and that's how it is. Um, yourself, like, uh, you know, I think i- in terms of the Frontier Evals that, that OpenAI is really going to excited to push, like, you, you put out really good work every single time. Um, what should people expect from, from OpenAI itself? **Olivia Watkins** [26:25] I'm not sure I can say what we're gonna- **Host** [26:27] General directions **Olivia Watkins** [26:28] ... I mean, general directions, I think looking at real-world impact. Like, re- real world, um, real- **Host** [26:34] GPTs, Vials 2, yeah, whatever **Olivia Watkins** [26:36] ... that kind of stuff. **Host** [26:36] Yeah. **Olivia Watkins** [26:37] Yeah. **Host** [26:37] Yeah. Amazing. Um, okay. Well, I'm excited for more real-world impact. I, I think you guys have, you know, really made a lot of progress, and I think taken a lot of industry leadership for SWE-Bench Verified and, and now moving on to SWE-Bench Pro. So thank you for doing this. Thank you for being so transparent. Uh, and I think people will respond in kind. **Olivia Watkins** [26:55] Yeah. **Host** [26:55] Yeah. **Olivia Watkins** [26:55] Thanks for your time. **Host** [26:56] Thank you. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.