Terminal-Bench: Pushing Claude Code, OpenAI Codex, Factory Droid, et al to the limits

Hey, everyone. Welcome to the Latent Space Podcast. This is Alessio, founder of Kernel Labs, and I'm joined by Swyx, editor of Latent Space.

Hello, hello. We're super excited to have Alex and Mike from Terminal Bench today. Welcome.

Yeah. Hey, guys.

Thank you.

Maybe we'll just quick launch right into it. I think a lot of people have heard about Terminal Bench. It kind of honestly came out of nowhere. I didn't really, like, see an announcement. Uh, it's just that all the frontier labs suddenly adopted it.

Uh, uh, all the agent labs also adopted it. I see Alessio's wearing the Factory AI T-shirt today. Factory just came, came out, come out claiming, uh, the top performance on the Terminal Bench. Uh, so maybe, you know, what's your self-intro and what is Terminal Bench?

Um, yeah. So I, uh, studied math and computer science at BYU, undergrad in math and then got a master's in computer science, moved to the Bay right after that, started working at Google. Spent about a year at Google before I started kind of looking around to see what might be next.

I-- Pr-pretty quickly after I arrived, I realized that maybe I wanted something that gave me a little bit more autonomy to be able to choose the problems that I work on, and definitely that I was-- had an appetite for risk that would allow me to do something that was a little bit riskier with potentially higher upside or potentially higher downside as well.

Hmm.

Met Andy Konwinski through a friend while I was kind of looking around figuring out what to do next, and I can talk more about Andy's background later, but he was a co-founder of Databricks and Perplexity, was in the middle of co-founding his next thing, Law Institute and Law Adventures, which is Law Institute is where I work now.

And yeah, so he, he invited me to come work on the K Prize, which was a one million dollar prize around Sweet Bench. And I decided I knew what Google would look like in three months, but I didn't know what the K Prize was gonna look like in three months, so I might as well go do that.

So left Google, worked on the K Prize, spent like three months in the Sweet Bench code base figuring out how they did things, building, like, continuously updating versions of it so that we could build a competition around it, and then ended up joining Law Institute full-time after that.

We teamed up with Mike and Ludwig and came up with this idea of Terminal Bench, which was kind of like a broader abstraction of the original framework that Sweet Bench had proposed that wasn't specific to, like, repos and PRs from GitHub, but really just generically anything that you can accomplish on a computer using code in a terminal.

And yeah, Ludwig had some ideas that he was thinking about. Andy had been thinking about, like, the terminal being kind of the one tool to rule them all, where you could really do very arbitrary things on a computer using the terminal.

And then I think Mike had been thinking about it a little bit as well. So all those kind of formed the-- converged into what Terminal Bench is today.

Yeah, Mike?

Yeah, yeah. Uh, I'm Mike Merrill. I'm a, a post-doc at Stanford, where I, I work with Ludwig Schmidt. Yeah, I mean, you know, all through, all throu-all throughout my PhD, I was very interested in how we can use code to do more than just write software, and in particular, like, how language models are part of that story.

So I did work on using code to help scientists do more reproducible science and, and how to use code to study time series and, you know, building language model agents to write code to help people understand their personal health data.

And so I, I've been curious for a long time about, like, what's the limit of this? You know? Like, how can we build agents that are not just writing code to build software, but are writing code to actually control systems and interact with the world?

And that was part of what was really exciting to me about Terminal Bench from the beginning, was, like, you have this, this great abstraction. You know, the, the terminal, it's, it's text-based, it's super powerful. We hear these stories about software engineers who just live inside VIM and do all their work there and don't even need a GUI.

And it kinda got me thinking, like, how, how can we build agents that are like our best software engineers? And maybe the terminal is a good place to start. And yeah, that was, I guess, about eight months ago now, something like that, when we made the first commit to the code base, and we've been growing since then.

I think, I think one of the really key moments for us was getting onto the Claude 4 model card, being one of two benchmarks that, that Dario actually mentioned while releasing the model. And since then, I mean, yeah, as you mentioned-

By the way-

... there's been adoption

... do they tell you when they do that, or do you just not get a heads up?

No. No, they don't. They don't, they don't tell you. They don't tell you.

Oh, wow. We-

We were actually on a call, Alex and I, and we were on a call with, uh, Nicholas Carlini, who's at Anthropic.

Yeah, he's a former guest. Good.

And he said... Oh, really? Oh, really? Yeah. Well, Nicholas, uh, Nicholas stopped the call and said, "Hey, you guys wanna, might wanna check this out," and pulled up the model card.

He shared, he shared his screen. He said, "Can I share my screen really quickly?" And then it was the blog post where they called out Terminal Bench.

Yeah. So no, they, they don't, they don't give you a heads up. But I, I can tell you, it does change things pretty quickly.

Well, I mean, well, Nicholas being at Anthropic probably had a hand in it, right? Is, is... He was, he was responsible for it, I guess?

Well, I, I, I think he had some involvement. I, I think maybe that's how we got on their radar, but-

'Cause he was, like, an advisor to the, to the project or something.

Yeah. He'd been involved with the project before. I, I, I don't know exactly the story of how it got used at Anthropic. You know, they're very secretive about these things.

Yeah, I th- I think that we... So we had built the initial, like, formulation of what a Terminal Bench task is and how you define one, and then the ability to, like, pick an agent and a model and see if it can complete the task.

And then Mike, I think, met up with Nicholas somehow, introduced him to the framework. Nicholas got really excited because he had all these random problems that he solved at some point in his career that he wanted to see if an agent could solve.

So he just started cranking out tons of tasks. And then my guess is, I think he started probably running it internally just to see how good the new models were, and then that kind of got word out to Anthropic.

And we were suspicious that something was up because there was, like, a lot of after-hours requests from people that we knew at Anthropic, seeing if certain ways we had formulated the benchmark and stuff. So we assumed something might be going on, but yeah, we didn't actually know until we saw the announcement ourselves.

And I think this all makes sense now with Claude Code and Codex, but you started working on this before these things got released. So what was at the time the decision there and, uh, in a way there's like the models chart the benchmarks and vice versa, you know?

So I'm curious about the design of the benchmark and how that has evolved once those products came out.

Yeah. You're right that this came out before Claude Code. Actually, Claude Code was the reason that we ended up refactoring a large portion of our harness to allow arbitrary agents to be evaluated, because we saw this great product come out and we wanted to see how it ran on the benchmark.

And I, I think, like, the f- the, the bet here that we made was that GUI-based computer use was going to be much slower to come into its own than terminal-based use. And there's a few reasons for this.

I mean, as, as I mentioned earlier, text is just the modality that works best with these models, and so having a text-based interface that allows the, the model to reason natively in text just seemed like the most performant way to get most things done.

And I think also the thing about these GUI-based systems is that they are just designed for humans, and they're designed for sort of the lowest common denominator humans at that. You know, like, like the example that I like to give is if you've ever tried to launch an EC2 instance through AWS, like clicking through their GUI, it's probably 20, 30 clicks to get that thing off the ground.

You know, collecting t- connecting to your volume, c-configuring your keys. It's far from simple. But there's just one terminal command that does that. And so, like, it just made more sense to me that this is what language model computer interaction would look like, and that was behind this original bet to focus on terminals before, say, browsers or-

Before it was cool.

Before it was cool. Before it was cool. That's right.

And then from a task selection perspective, you know, when you hear the name Terminal Bench, you assume it's like, oh, how good is it at grep, LS, all these different things, and then, uh, you look through the task and obviously there's a lot of coding, uh, there are games things, there are like mathematics things.

How did you think about structuring the task set, and like has that evolved as like most of the CLI usage of these tools has been coding, or do you think like there's still like a big interest in like non-coding things wrapping through the terminal?

I, I think that-- Well, first of all, what a Terminal Bench task literally is an instruction, a container environment, and then a test script that usually calls a set of unit tests to check whether or not the container is in a state that corresponds to what was described in the instruction.

So we call it Terminal Bench mostly because you interact with it by usually these bash commands, and sometimes they're really gnarly bash commands that, you know, GPT 5 or Claude Sonnet decides to write, but, but somehow they work.

But really, maybe in a different world, we could have called it Container Bench or something like that. And the interesting thing about, like, agents right now is agents more or less, right, o-outside of robotics are not embodied, so they can't actually do stuff in the real world.

So actually the entire effect that they have is on computers, essentially. And these containers just represent a computer in some way. So in theory, anything that you could do on a computer, you could probably represent as a Terminal Bench task.

And that's why you see things that don't obviously pop out as like a coding problem. Although I do think coding is probably the majority of the data set right now, and I do think that's probably the domain in which we're accelerating the quickest.

But yeah, even the other day, I think the head of DevRel at Anthropic tweeted and said, "What is your favorite non-coding use case for Claude Code?" And it got like a ton of replies and people were like, "Oh, I use it to automate my email drafts.

I use it to, uh, like, write journal entries based on what I did on my computer." I don't know, things that I maybe wouldn't do myself. But I think it just goes to show that it is a really powerful domain, and we tried to get a lot of diversity by, like, making it open source really soon, and then trying to build out an incentive system to get a lot of contributors involved to create tasks that they've encountered themselves.

And then it's not just like the Alex and Mike benchmark, the Alex, Mike, and Nicholas benchmark or something like that, but really corresponds to like all sorts of different tasks. And, and even now I think it's still like not as diverse as it's going to be because, like, we-- it feels like we're on an exponential in terms of con- contributions and, like, recognition.

So I, I think the benchmark will get even better as time goes on. We'll release like multiple versions of it.

So I like to show and tell a little bit. Yeah, we're on the, we're on the page. And just, you know, what-- do you have a favorite? I don't know if you, you, you obviously picked these for the landing page, but, uh, any, anyone we can sort of di-dive into.

Train FastText is a good one.

Okay.

Yeah, that is a fun one.

What's the story behind this one?

If you click on the card itself, it will keep you in the UI.

Yeah. Oh, there we go. Yeah.

Yeah. There's the instruction.

So the, the story behind this one is that Jeffrey Lee, one of our contributors here, is one of Ludwig's PhD students, and on a previous paper of theirs, Datacon for Language Models, he had to design a classifier for text.

This turns out to be a very important step of most pre-training pipelines. Yeah, there you go.

That's the point.

DCLM.

Yeah.

Yep, yep. So he's the, he's the first author of that paper. And it was a, it was a difficult engineering challenge to build this model in such a way that it worked well enough, but was also small and fast enough to scale to this large corpus that they were working on.

And so we like to joke internally that when models are reliably getting this one right, Ludwig won't have to hire any more PhD students. But unfortunately, we are getting to the point where models do reliably get this one right, and so I'm not sure what implications that has, but hey, I-I'm a postdoc.

This is rather complicated. Yeah. I mean, like the, the, the execution time for this is like, what is it? Like ten minutes?

Yeah. Yeah, that's right.

And if, and if you read the instruction itself, it's train the model to be under a certain size but get at least this much accuracy. So it's essentially like training a machine learning model under constraints, which is a very open-ended problem, right?

Yeah. Insane. Yeah, so and, and then, and then you have a, a thing about tasks here. I-is there a way that you think about grouping them? Or wh-when you say that, you know, you wanna increase the diversity, how are you thinking about increasing the diversity?

Anything of that sort.

So I think one, one way that we think about Terminal-Bench tasks is I think there's a bit of a tension between, and you alluded to this earlier, what people are currently using these agents for and what they're capable of, right?

So sure, most people are using Claude Code right now for software engineering, but as Alex mentioned, you can jack it into an MCP and get it to do all kinds of things on your computer. Or I, I saw a thread the other day where someone was dealing with some networking problems, and they just let Claude Code issue all the bash commands to figure out, you know, what was wrong with their VPN.

And we designed Terminal-Bench tasks thinking much more about the future of these agents than we do thinking about what people are using them for right now. And so the categories follow this distribution. So like one task that we have now is from a collaborator of ours who's a postdoc in biology at Stanford, and he does work on assembling DNA sequences.

And so the Terminal-Bench task is using a set of tools that are used by real scientists to solve this relatively complicated problem that he actually encountered in his day-to-day work, and we'd like to have more tasks like that, I think.

Like, like a good Terminal-Bench task is one that's hard, but not adversarially hard, just hard on its own merits and valuable. You know, someone's paid to do it in the real world and verifiable, so we can write test cases for it and actually check whether or not the agent did what we thought it did.

Do, do you feel like this is the way everything should be tested more and more? Like, especially you mentioned Claude Code and writing emails, for example, right? There's already benchmarks around writing emails and kind of like things that are not related to the terminal, but in a way it's like, well, if I orchestrate it through the terminal, maybe this is the best way to test it.

Do you feel like eventually even all of these non-code things should be run in like a Terminal-Bench-like setup? Or do you think that there's still value in like just doing things in like a chat interface because maybe a lot of people will just use the claude.ai or ChatGPT UI that doesn't have the same access?

I don't think that you need to access a terminal-- sorry, an agent through the terminal. Like, I don't think the future of all agents is people NPM installing them onto their computers and then typing commands into their terminal.

But I do think that behind the scenes, these agents are running in containers, and their tools are actually programs that they're executing. Um, in that sense, I think it's a useful benchmark, no matter where the agents actually get, like distributed to the users.

And that is a change that Mike mentioned earlier we had to make, is originally the way we formulated it, we wanted there to be one tool, which was a tmux session, so it was as close as we could get to like a headless terminal.

And we wanted all the agents to have to use it, and we were gonna like build an MCP around it, and then the agent would have to like make, like execute commands through the MCP server. But very quickly, we realized that people like felt a very core component of their agent was the custom tools that they built for it, and they wanted to install their agents into the task container and then just execute them in a headless mode there and let them, and let them run wild.

And like the way you do that in Terminal-Bench is you like actually execute the install command and then like a headless terminal command. But yeah, you could just as easily see that happening by like clicking a button on your browser, and behind the scenes, it's like spinning up a container and executing some task for you.

The other thing you have adapted other benchmarks for Terminal-Bench, so it's kind of like the meta benchmark. How do you think about the state of benchmarks? You know, so you mentioned diversity. You can be diverse by adding more tasks, you can be diverse by supporting more third-party benchmarks in it.

And then, for example, you have SWE-bench Verify, but you don't have, uh, SWELancer, you don't have SWE-bench Pro. Like, are all of those things kind of worth testing? Or is it some point, hey, look, if you're great at SWE-bench Verify, you're probably also great at SWE-bench Pro, you're also very great at SWELancer.

Do you feel there's like diminishing returns to like supporting more and more of these benchmarks and it's better to build tasks from like first principles?

So there's, there's two things there that I want to touch on. The fir- the first is our, our adapters, where we're building existing benchmarks into our evaluation harness to allow people to run their agents on all of these benchmarks in one place.

And that's something we're really excited about continuing to support. So like our vision here is you bring us your agent, you install it into our harness, and you have a one-stop shop for every academic benchmark and every future academic benchmark that's been published.

And there, I think we want this to be as big of a tent as possible. Something that I think we did really well was design our harness so that it's compatible with all of these other benchmarks, and we, we want to continue to support that.

I think there's a, there's a bigger question of what the future of evals really looks like. You know, like what's working right now and what's not going to work in the future. And we're at a really challenging point for evals because I think like all the low-hanging fruit are really taken care of.

So what made SWE-bench so powerful was that you could just go on GitHub and find all of these repositories and just set up an agent to try to do the work that you had found online, right, of resolving these GitHub issues.

And I think that as agents get better and better at solving these sort of like clear input, clear output tasks, the opportunities for creating benchmarks in this way gets slimmer and slimmer and slimmer. You can compare that to a benchmark like Terminal-Bench, where each task is unique, comes with its own environment Has a very long horizon of potential, potentially hundreds of actions that need to be chained together.

Like these are characteristics that make Terminal-Bench harder than SWE-Bench, but are also characteristics that make it much more difficult to develop, you know? Like there's nowhere you can go online and scrape a dataset of five hundred Terminal-Bench tasks.

Like this is something that we tried to do when we were building the benchmark originally, and it just, it just doesn't work. Like you can't find them anywhere. You can't even really generate good ones using a language model, although y-you open up all kinds of other potential problems if you go down that route anyway, which, which I can get, get into.

But I think like the future of evals does look much more like this, is like observing people who are doing their real jobs and then translating those real jobs into a format that allows you to evaluate language models and harnesses on them.

And it's probably gonna mean a lot more diversity, it's gonna mean each of these tasks requires more and more specialization to even generate, and more and more specialization from the models in order to do really well on. I mean, like something, you know, if you l- if you go look at SWE-Bench Verified, I think like sixty, seventy percent of the tasks in there are from Django, right?

And like every environment in Terminal-Bench is unique. And I do think that this is what benchmarks are gonna look more and more like over the next two to three years.

Can I, can I add a third point to what Mike is saying as well, which is, yeah, something that we realized very early is that the task abstraction was, was really capable of expressing lots of different types of problems, which quickly led us to realizing, oh, pretty much all of these other benchmarks can be expressed in the Terminal-Bench task formulation.

And you called out that we have certain benchmarks adapted, certain are not adapted yet, like SweetLancer. Although actually I think SweetLancer is adapted, I just haven't uploaded it to the website, to the gallery yet. But, uh, yeah, we have a contributor, Lin Shi, who is kind of like leading the adapter effort, and he's really good, and he has made like a call for contributions.

I think we have thirty adapters on their way, and we have a couple of users who are building their benchmarks directly in Terminal-Bench and like they will use that as a way to distribute it. So we've been trying to frame up Terminal-Bench as like not only its own benchmark, but actually a framework in and of itself that people use to create and host and distribute their benchmarks to other people.

And then even taking it one step further, we have like a grander vision where maybe one day people will use Terminal-Bench at their companies to create internal private eval sets for their specific custom use case. And then maybe one step further than that is they will even use Terminal-Bench tasks to do rollouts to post-train a specific model on their use case by using it as an RL environment, not just an evaluation environment.

So I do think that, yeah, there's, there's a lot of room for the, for diversity to occur within the benchmarks themselves, but then also just like a diverse framework that supports a lot of companies building in private as well.

How do you guys think about testing the agent versus the model? So you have Terminus, which is your own kind of like a research preview agent to do these tests, um, and then obviously people can submit their own agents.

Are those two things separate? Like, is there value in like just testing the model without the harness? And likewise, how, how should people think about the difference between the harnesses where it feels very nebulous? You know, like the factor we want is like the droids are the better, but it's like, why, you know?

Like I think to me that's hard to understand still.

Yeah. Yeah. It's not always easy for us to understand either, which is why we came up with Terminus in the first place. So, so Terminus is our agent, which we designed to be simple and unopinionated. And the reason for this is that if you go and use an agent like Claude Code or Codex CLI or even Open Hands, it's not clear to the, the degree to which this agent has been optimized to make up for some of the weaknesses of a particular model.

Or even in the case of these agents that come from the frontier labs, the degree to which the model has been optimized with that agent in the loop. You know, you hear about how like Anthropic might be doing RL on top of Claude Code, you know, like OpenAI may be doing something similar with Codex.

And so like if you're really interested in, in studying the ability of the model, it becomes very important to separate it from its harness. Now you, you can't do this entirely because this isn't an agentic benchmark to begin with, but you can do it by making a minimal number of assumptions about how your agent is structured.

So in, in Terminus, the only tool that the agent gets is a headless terminal. It can only operate by issuing batch commands. And so there's no special tools developed, you know, there's, there's not a whole lot of funny business going on with like context management, you know, or like tool calling or web search.

Like it's, it's purely the model being jacked directly into the terminal. Um, and this is, I think, the, these are the scores that I am the most excited to see move because I think that they are much more indicative of underlying improvements in capability than they are optimizing for the particular use case or benchmark.

So if your agent does really well on Terminal-Bench and you're using Claude Four or Five Sonnet, that's great. I think that's very interesting. But if a new model comes out and it gets like seven percent higher on Terminal Bench, that's using Terminus, that I think is like a sign that we're actually making fundamental progress in AI and less that we're just sort of like figuring out clever prompts or like slightly better tool design to make up for some of the, the areas where these models might be lacking to begin with.

Right. And if you look at the difference between the same model and the biggest spread across agent frameworks versus the same agent framework and the biggest spread across models, it's much larger across models, which I think goes to show that like improving models and model selection is gonna be your, your biggest lever to pull when you're trying to build a great agent.

So a lot of that ends up going back to the frontier labs and I guess open source labs as well. But, but agent framework seems to make a difference as well, like up to 15% or something pretty significant.

Yeah, it's a bit hard to tell here. There might be... I don't know if you've like, have a better graphic for this, but, yeah, if you can do like a group by of agent or group by of model and then see the s- see the separation, I think it'll be easier to see from here.

Mm-hmm.

'Cause I can, I can intuit what you're saying, but like, uh, visually there, there should be a little bit of work to tell that story.

Yeah, we have a lot of graphics that we need to probably put on the website, so that's a good, good reminder.

I think there's a lot of discussion, like one of the big topics of this year is really the discussion of like the, the model labs like starting to ship their own, ship their own agents, right? But then also like the ag- agents kind of like the products almost.

Like they want you to use Claude Code with Sonnet and Codex with GPT-5. Um, and like, you know, there, there, there's, and then there's, there's another f- group of companies that are all like agent labs, which are, which are like, "Okay, we'll, we'll just use like all these models and like build our own agents on top of it," which you, you see with the, the Factory Droids.

Um, I, I think it's like an independent moving thing that may become more synchronous over time because they want to sell that unit together. Um, but like, and it's, that's at odds with like kind of like the model as an API service where you want anyone to be able to build on top of it.

Yeah. Yeah. I, I think this is a very open question right now is like how much, how much do you get as a, as a agent manufacturer by training your model in the loop with it? This is also incidentally like an a- academic question that we're very interested in answering, and I think will probably be some of the follow-up work to Terminal Bench.

We're actively trying to understand this question of like what are the best practices for training a model in the loop with your agent? But the, yeah, the business model is very different, right? Like, like there is this open question right now in the AI world of how much longer are these labs going to continue to be pure purveyors of intelligence, and at what point are they g- going to shift to being something that looks much more like a product company, right?

And with the Claude Codes and the Codex CLIs and the deep researches- researchers, you're starting to see some evidence that the products are gonna be a much more central part of how these frontier labs operate.

Yeah, I mean, my, my sort of cop-out answer is they're just gonna try to do everything 'cause they have infinite money, so why-

Really? Sure. Yeah. Yeah. Why not?

Cool. Uh, yeah, we were, I think we're just gonna about to talk about the roadmap, but in our pre-chat you guys were saying that you're working on something for, for the cloud or like a hosted version. Uh, maybe I'll, I'll let you run away with like what exactly it is that, you know, what's next for T-Bench.

Yeah. Maybe I can give a, a high level overview. I mean, this essentially is the roadmap, so I guess this is the perfect time to start talking about it. Yeah, we, we originally set out just to build a benchmark, and we did intentionally approach it as, uh, and this comes I think a little bit of the LAAD thesis, the LAAD Institute thesis, which I can explain in a second as well.

We, we set out to build it to be like the best developer experience of any benchmark that you see out there, which isn't the highest bar in the world because a lot of these benchmarks are created as academic projects and they mostly wanted to get their innovation out into the world rather than make it extremely usable.

But yeah, at, at LAAD, based on kind of Andy's background, there's a huge culture around shipping research, which is like, don't stop at the research paper, but like actually put an open source project or a product into somebody's hands that makes it extremely usable.

So we tried to make it extremely usable. We saw a lot of interest and still see a lot of interest in the framework itself, so we're really trying to come at it strong from the framework angle now, which is, I think, trying to make an interactive agent benchmarking and, right, RL post-training framework that makes it really easy to create tasks as a human, so to specify them, take, like Mike was saying, your, your business workflow and describe as a Terminal Bench task instead to potentially synthesize like a ton of tasks based on some data flywheel that maybe we invent or other companies invent and put into the framework.

And then to run evaluations at scale, hopefully like in, in the cloud using cloud containers or even like in hosting the entire thing in the cloud. And then to, yeah, ver-version, h-host and distribute it to a bunch of people and potentially do like RL rollouts on Terminal Bench style tasks.

So that, those are the directions that we're heading in right now. Like I said, we'll also periodically be releasing like bigger and better and harder Terminal Bench task sets. And there, there was a ton of work, m-more, I think, than we expected, uh, in making sure that the tasks were really high quality, which specifically means like, are the things that you're testing actually the things that you described in the instruction?

And did you describe in the instruction in enough sufficient detail based on the test that you're, you're testing the agent with? And we wanna take a lot of those tips and tricks that we learn and make it super easy for other people so they don't have to reinvent the wheel every time.

Yeah. Um, I, I, I would s- I would also say, so I mean, I think that's, that's very exciting and I'm sure, uh, people will see more as, as you go. Uh, I would also say like, I, I think there's a shift in the way that we evaluate agents in a sense that like, you know, Terminal Bench is a, it's a one-dimensional eval, which is there's a, there's accuracy column.

I wonder if there's any interest in like cost, uh, time, wall clock time. I don't know. Like what are, what, like what, what is the eval meta gonna be in 2026, right? Like, uh, I, I'm definitely like thinking ahead and I'm seeing like basically my, my, my sort of banner tweet for this, which is gonna be like, "There's no more one d- one-dimensional charts."

It all should be two-dimensional. Second dimension should be something like a cost, something like a latency, something, I don't know. But what, what are your thoughts on these kinds of things?

Yeah, yeah. Uh, we definitely wanna take cost into account while evaluating these agents, and I think that we have some plans on how to do this, and you'll see it in subsequent versions of Terminal Bench. I think that this is particularly important, that is to say, taking cost into account when looking at the highest performers on the benchmark because- Since they are independently evaluated and installed, we don't actually know how many tokens are going into-

Yeah

... getting those numbers, right? And for that matter, I mean, we don't know that these are the versions of the agent that are actually being deployed to customers. You know, we rely on our partnerships to help us understand, like, how these agents relate to the ones that are being deployed in the real world.

But, like, I think there's probably a way in which you spend $1,000 on tokens for each Terminal Bench task and, like, get them all right just by scaling inference time compute. But that's, that's just not particularly interesting, right?

To ans-answer your question about what the future of evals is gonna look like in these 2D charts, I think the important thing is not, one day is not just gonna be how much did it cost to run these evals, but it's actually gonna be how much money did running these evals make, you know?

I think, I think the ultimate eval is going to be, like, actually deploying an agent to operate as a software engineer at a company for an extended period of time, and then measuring how much money it made. Like, that's gonna be the thing that actually matters at the end of the day.

And I think similarly, like, you're gonna have agents deployed in the stock market, and you're gonna say, "How much money did my, my quant agent make me this month? Like, what's the actual P&L from my agent look like?"

I'm, I'm calling in from New York, and so maybe I've got finance on the mind- ... a little bit more than usual, but, like, this is actually what people are going to care about at the end of the day.

How much money-

How much did my, my-- How much money did my AI marketing campaign make me?

That, that is the ultimate eval. So at the end state is we're all VCs.

Yeah. I don't know, I don't know. I, I think, I think that that is... Or, you know, I deployed my teaching agent. How much did the standardized test scores go up, right? Like, how much... Yeah. That's, that's gonna be the second dimension.

Yeah, I agree on the economic value thing especially. To me, the, the other eval is, like, how much more money, how much money can I throw at it and get a commensurate amount of value out of it, you know?

I feel like that's a... Like, a lot of these tasks are, like, small, so it's kind of like, yeah, I can put $50 of tokens and get something nice out of it. But, like, if I put, if I had a million dollars of tokens to spend, like, how much value could that agent actually go out and generate?

I think that's what people haven't really tested, you know? Like, today, the Sonnet 4.5 release, it was like, "Oh, it ran for, like, 30 hours." It's like, well, what can you do in 30 hours? I'm happy to pay for 30 hours, you know?

But, like, what's the difference? Is it just a lot of, like, five-minute tasks being done?

Yeah.

Or is it, like, unlocks a new type of task?

Well, also it's just incentivizing slowness. Like, you actually want it to be faster, not slower. Like, you're incentivizing everyone to just take their time, and that's bad.

I guess it depends on how important the problem is and how long you're willing to wait for it. But, like, we, we built in timeouts so an agent only has a certain amount of time to attempt a task, mostly for usability, because we didn't want to wait, like, 30, 30 hours ourselves for a benchmark to finish running.

But an experiment we do want to run, which we think would be interesting, which kind of is what Alessio is saying, would be to, like, take different lengths of timeouts and see as the timeout increases, how does the accuracy on the benchmark increase?

Because at some point you'd expect it to plateau, like giving it more time, i.e. giving it more tokens, stops improving the performance on the benchmark because there's some tasks that maybe it's just not smart enough to solve right now.

But we do want to run that experiment to see, yeah, to, to what extent can you, can you keep bumping the timeouts and keep getting incremental performance gains?

Cool. Looking forward to the updated episode once you guys release that. Um, any final call to action, like requests for contributions? Are you hiring people? Like, um, anything else you want to ask for?

Yeah, I mean, c-come join our Discord. It's a real party. I think Alex-

I'm in right now

... Ludwig, and I are still hitting the wave to say hello button for every new person who joins, so-

Aw

... come on through. And yeah, we're looking for-

It's... I'm convinced-

Go ahead

... Ludwig has a bot on his computer because it feels like he doesn't miss a single person that joins the channel, no matter how big it grows.

Mm-hmm. All right, I just waved at Shawn. Shawn can wave at me.

There you go.

Wave at me.

And then we'll do, we'll do your guys' job.

There you go.

There we go.

No, cool. And, and honestly, it's like, I think this is something that we have to interview Ludwig at some point because organizing DCL-LM as, like, a pretty distributed effort among all these institutions is, like, an immense effort. And, like, I, I always wanted to know, how do you organize online community to do something substantial like this, right?

Yeah.

So it's, it's kind of cool.

It's a challenge.

I think Mike is another person who has, I think, a, a sixth sense for that type of thing because he did it really, really well with Terminal Bench. I also think that Ludwig is probably, like, one of the best people in the world at doing these massive decentralized open source projects.

So if you want, yeah, we'll put in the good word. We'll get him on here. He can chat with you guys too.

Yeah.

I think I talked to him at one of the NeurIPS's, but, uh, yeah. Anyway, too, too much going on. Uh, yeah, I'm excited to invite you to, guys, to, to also come to AIE Code and talk more and, you know, there's lots of, uh, coding agent work that's gonna be done at, uh, AIE, so.

Um, awesome. Well, see you guys later.

Yeah. Thanks so much, guys. Good to meet you.

Yeah.

Take care.

Thanks for having us. See you guys.