# Terminal-Bench 2.0: the most impt coding agent benchmark of 2025 gets a v2! Launch + Q&A w/ founders Latent Space · 2025-11-08 Terminal-Bench co-creators Mike Merrill, Alex Shaw, and collaborators Andy Konwinski, Ludwig Schmidt, and Aatash launch Terminal-Bench 2.0, an 89-task coding benchmark with rigorous human and LM-assisted verification to test agent performance on real-world tasks like designing DNA primers and installing Windows 3.11. They reflect on Terminal-Bench 1's problems—easy tasks and reproducibility—and detail spending 300 hours manually auditing each task. They introduce Harbor, an open-source package for running sandboxed agent rollouts, standardizing task format and integrating with agents like Codex and Claude Code. Aatash shows using Harbor to generate thousands of training traces for DataComp Agent, running 1,000 parallel sandboxes. Future plans include expanding to scientific computing. ## Questions this episode answers ### What are the main improvements in Terminal-Bench 2.0 compared to the original? Mike Merrill explains Terminal-Bench 2.0 includes 89 tasks, each based on real paid work and rigorously verified with over 300 hours of manual auditing to ensure solvability, reproducibility, and quality. It eliminates easy tasks like 'Hello World,' unreproducible tasks like YouTube download, and games. New hard tasks include DNA primer design, installing Windows 3.11 in an emulator, and cybersecurity challenges. [4:09](https://addtry.com/3c596cd7-0852-423e-8db5-1116a94d3288?t=249000) ### What is Harbor and how does it help with agent evaluation? Alex Shaw introduces Harbor as a package for evaluating and optimizing agents in containerized environments. It standardizes the task format, pre-integrates popular agents (Codex, Claude Code), supports cloud deployments, and unifies the repetitive process of starting containers, running agents, and collecting rewards, thereby saving developers from rewriting infrastructure code for benchmarks and RL loops. [13:01](https://addtry.com/3c596cd7-0852-423e-8db5-1116a94d3288?t=781000) ### How did the Terminal-Bench team use co-authorship to gather high-quality tasks? Alex Shaw says they offered co-authorship on the Terminal-Bench paper to anyone creating three tasks. This incentive drew 250 task contributions from PhD students and researchers, but only 89 passed the multi-level review, which included agent runs to catch trivial solutions and manual auditing to ensure tasks were hard, solvable, and realistic. [26:53](https://addtry.com/3c596cd7-0852-423e-8db5-1116a94d3288?t=1613000) ### Which model leads the preliminary Terminal-Bench 2.0 leaderboard? Mike Merrill reports that in their preliminary testing, Codex with GPT-5 ranks at the top of the leaderboard. Interestingly, their minimal harness Terminus 2 often outperforms popular agents like Claude Code, suggesting that simpler agent implementations can sometimes better reveal underlying model capabilities. [5:59](https://addtry.com/3c596cd7-0852-423e-8db5-1116a94d3288?t=359000) ## Key moments - **[0:00] TB1 Lessons** - [0:40] CLI agents outperform GUI agents for tasks like launching EC2 instances, say Terminal-Bench creators. - [1:56] Terminal-Bench One appeared on Claude 4 model card just three days after launch, claiming state-of-the-art. - [3:16] Terminal-Bench 2.0 fixes v1 flaws: removed easy tasks like 'Hello, World', non-reproducible YouTube download, and games like Zork. - **[4:09] TB2 Launch** - [4:09] Terminal-Bench 2.0 comprises 89 tasks based on real-world paid work, with 300+ hours of verification. - [5:08] Terminal-Bench 2.0 includes tasks like DNA primer design, installing Windows 3.11 in an emulator, and cybersecurity challenges. - [5:59] Codex with GPT-5 ranks highest on Terminal-Bench 2.0 in preliminary testing. - [6:22] Terminus 2, Terminal-Bench's minimal agent, outperforms complex agents like Claude Code on many models. - **[7:01] Harbor** - [13:01] Harbor is a new package for evaluating and optimizing agents in container environments, ending repetitive harness coding. - [14:12] Harbor enables running thousands of sandbox rollouts in the cloud for evals and post-training with minimal code. - **[15:01] Harbor Demo** - [15:31] DataComp Agent project uses Harbor to generate thousands of agentic tasks for open-source model training. - [16:09] Harbor supports the full agent training pipeline: data generation, SFT, RL, and evaluation. - [18:33] Aatash ran 1000 parallel sandboxes on Daytona to collect agent trajectories using Harbor. - **[21:13] Behind the Scenes** - [21:46] Andy Konwinski's LAAD provides grants, GPUs, and engineering resources to open-source AI projects like Terminal-Bench. - [22:43] Ludwig Schmidt started Terminal-Bench in summer 2024 to measure long-horizon agent capabilities in the terminal. - [23:51] Andy Konwinski: "the terminal is one tool to rule them all because you can do almost anything on your computer." - [26:53] Terminal-Bench 2.0 used co-authorship as incentive, receiving 250 task contributions, 89 passing quality checks. - [28:14] Andy Konwinski says open-source co-authorship generated evaluation data comparable to expensive services like Scale. - **[29:20] Looking Ahead** - [30:19] Terminal-Bench co-creators regret naming it 'Terminal' as it's really about container environments, not just terminals. - [30:28] Terminal-Bench creators plan future benchmarks for scientific computing to accelerate science. - [32:44] Q: How does Harbor handle rapidly changing agent definitions? Alex Shaw: 'An agent is just a program that gets executed.' - [33:58] Mike Merrill: text-based agents are the current frontier, but Harbor is flexible enough for future GUI agents. ## Speakers - **Aatash** (guest) - **Alex Shaw** (guest) - **Andy Konwinski** (guest) - **Ludwig Schmidt** (guest) - **Mike Merrill** (guest) ## Topics Benchmarks, Coding Agents, Agent Infrastructure ## Mentioned Anthropic (company), Databricks (company), LAAD (company), Perplexity (company), Arc (product), Claude (product), Codex (product), Cursor (product), DC Agent (product), Daytona (product), E2B (product), GPT (product), Harbor (product), JePA (product), Modal (product), Open Hands (product), SkyRL (product), Suitebench (product), Terminal-Bench (product), Terminus (product) ## Transcript ### TB1 Lessons **Mike Merrill** [0:00] We're super excited today to talk to you guys a little bit about Terminal Bench Two. Woo! Yeah. Thank you. So what we're gonna be doing right now is, uh, first, we're gonna reflect a little bit on Terminal Bench One. You know, what made it work, what didn't work so well, what we learned from it. Then we're gonna dig into Terminal Bench Two, um, you know, the successor to Terminal Bench One, and share some exciting things about it that I think you all are gonna like a whole lot. And finally, we'll talk about a new project of ours called Harbor, which is our package for evaluating and optimizing agents. Sound good? Yeah. Yep. All right. So let's just get started with talking about Terminal Bench One. Uh, we really started Terminal Bench with, uh, an opinion about how agentic evaluations and agents in general should interact with computers. If you go back a few months, and, and even today, there were really two paradigms about how computer use would work. The first was by operating GUIs, you know, clicking through menus, drop-downs, selection, using multimodal models to navigate computers much like a human would. And I, I, I think we really thought that this was not going to be the most performant way for a language model to operate a computer going forward. Um, and a great example of this is if you've ever tried to launch an EC2 instance from the GUI. It's a ton of menus, takes you twenty or thirty clicks, and in comparison, there's just a single CLI command that could do this for you. So if you're really an expert at using the command line, you could do valuable work much faster than you could if you were clicking through GUIs. And so if you have an opinion about how AI should work, what's the best way to get other people to work on it? And that's by launching a benchmark. So we put out Terminal Bench, uh, I guess it was back in March. Is that right? That was- May. May was the original launch. And we've been very happy with the reception that it's gotten since then. So only three days after we announced it, it made it onto the Claude Four model card, said a new state-of-the-art. Today, we have a thousand stars on GitHub. We have a thousand Discord members, many of whom are in the room today with us, and a hundred contributors to the repo. So just thrilled to see the community grow around this thing and the adoption that we've had. It's used by all frontier labs in some capacity. It's made it onto many more model cards since that first Claude Four one. And in the meantime, we've seen this proliferation of CLI agents. So, you know, your Claude Codes, your Cursor CLI, your Codex CLI, and in general, an increase in agents writing code to operate computer systems. People also use Terminal Bench in many unexpected ways. We didn't even anticipate, uh, JePA using Terminal Bench for prompt optimization or DevRev taking the Terminal Bench harness and using it for CI/CD for their agents before they deployed them, or members of the open source community taking the Terminal Bench harness and hacking it to form RL. Didn't see any of this coming. But there's a, a few things that we definitely would have done differently. So if you look back at Terminal Bench One, and you pay close attention to the tasks in it, a few of them were maybe not at quite, quite the, the highest level of quality that we desired. Some tasks were far too easy. "Hello, World" is a task in Terminal Bench One where an agent just has to edit a text file, and this was useful for debugging, but it also meant that even very weak agents were able to get some questions right in Terminal Bench. Uh, some tasks weren't reproducible. YouTube download, on that one, there's this cat and mouse game between YouTube and anti-bot protections, and so a solution that worked for an agent once might not work at some point in the future, and we wanted to avoid this going forward. And I think the third thing that's worth highlighting is that some tasks just weren't inherently valuable. A few tasks were games, like playing Zork, which is very interesting and, I think, useful for saying something about agent performance, but not the type of task that we really want to evaluate on the long term. And so with these things in mind, we're launching Terminal Bench two point O tomorrow. ### TB2 Launch **Mike Merrill** [4:09] Terminal Bench Two is comprised of eighty-nine tasks, some of which are new, some of which are modified and improved from the original version of Terminal Bench. Every task in the benchmark is based off of real work that people do out in the world that are paid for. And my favorite part about Terminal Bench Two is the improved verification. So I estimate we spent at least three hundred hours digging through each one of these eighty-nine tasks, trying to figure out whether or not they're broken, whether or not there's some way to cheat, whether or not they're reproducible, and really obsessing over quality while building this benchmark. And as I mentioned, we're launching tomorrow. Thank you. Thank you. Thank you. I'll just quickly highlight three new tasks that we have in the benchmark to give you a sense of some of the things we're excited about measuring. Uh, one of them involves designing DNA primers to allow biologists to assemble DNA segments. This one I, I think is great because it's easy to describe and immediately obvious why it's so hard. Installing Windows three point one one in an emulator on a Voodoo box. And, uh, some cybersecurity tasks as well that I think are great for measuring the capabilities of frontier models in potentially dangerous scenarios. As I mentioned, we obsessed over quality this time around. Uh, I think we did a great job on that in the first benchmark, but e-even better verified this time around. We had multiple levels of review that went into each and every task that was merged into the final repo. We ran multiple agents, including ones that checked for trivial solutions or opportunities to reward hack. Uh, and then, as I mentioned, we manually audited every single task in the benchmark rigorously. And I think the end result is just a benchmark that has, uh, the highest possible quality standards. So really excited to see how people do on that. Here's some preliminary results. We see Codex with GPT 5 ranks at the top of the benchmark in our testing. Uh, I think something that's really interesting here is that Terminus 2, which is our harness for measuring language models, is the highest performing agent for many of these models. Um, and so, you know, you don't-- agents like Claude Code or Open Hands don't appear on this leaderboard quite as often as our, uh, our minimal agent that we put together just for the purpose of understanding language models. So pretty cool to see that work. And if you're interested about why I think that might be, happy to talk about that later. Um, so that's Terminal Bench 2.0. Really excited about that. And for the remainder of the talk, I'm gonna hand things off to Alex, who is going to share some of the things that we learned while building Terminal Bench and how we're fixing those going forward. ### Harbor **Alex Shaw** [7:01] Awesome. Thank you, Mike. Um, yeah, so we, we learned a few things by building a benchmark. Actually, by building two benchmarks. So I think actually the easiest way to, to take you guys through this process is to, uh, walk you through a day in the life of an agent developer. Um, that's probably many of the people in this room, actually. So, uh, the first thing I think to myself is I have an agent. So I, I want to know if it's good. I think, uh, most people with agents want to know if they work. Uh, most people buying agents also want to know if they work. Uh, so I think to myself, I should build an eval for my agent. Um, well, what do I need to build an eval? I need data. And what kind of data do I need? Well, my agent has to know what it's supposed to do when I'm measuring its performance, so I need an instruction. Um, and then my agent also needs a place where it can complete the instruction, right? And right now, agents automate computer work, so it should probably get a computer that it can operate in. Uh, well, we'll give it a virtual computer. We'll give it a container. And then finally, after I give my agent my instruction, I give it a container to operate in, uh, I need some way of knowing whether or not it completed the task. So in our case, uh, that's a set of tests. Um, these tests could, in theory, be like an LLM as a judge running under the hood that does a more subjective verification. Uh, but for at least for Terminal Bench, it was always a set of unit tests. Um, cool. So I have data now. I have it in the right format to evaluate my agent. Now, how do I actually evaluate all the data? So keep this process in mind for the rest of the presentation. So step one, start the container. Step two, give an agent the instruction. Step three, let the agent interact with the container. Step four, run the tests. And step five, based on the test output, return a reward. Um, and if you're anything like us, the very first thing you'll ask yourself when you run an agent on this benchmark is, is you will say that the containers are so slow. Um, and the solution to that is to run them in parallel. So, uh, if you start running them in parallel, you'll, uh, slowly bump up the rate of parallelization, and then you'll run into this problem. Uh, I'm running out of cores. We, we actually did have one contributor in Terminal Bench who went and bought a machine with a hundred and twenty-eight cores on it so that they could run every single task in the benchmark in parallel. But we don't want you guys to all have to go buy hundred and twenty-eight core machines. So instead, the solution is to run them in the cloud. So cool. We have our parallelization problem is solved now. We can run a bunch of tasks in parallel. We can do it quickly. The next question is, uh, how should my agent interface with the environment? Uh, well, the most obvious answer is you should build a tool, right? Build a tool that lets it send commands to a container so that it can execute tasks. Um, but then you might ask yourself, well, how can, uh, how can I run any agent in this environment? And, uh, the solution there is, well, agents are just programs, so install them into the container and then run them there. Um, and now you have your setup is finished. You've defined your data. You've set up your process for evaluating all these tasks, and you'll then look at your results and you'll say, "Okay, my agent can't solve this task, but I need it to because I need to sell this agent to somebody." So maybe the first thing that you try to do is you say, "I guess I'll optimize the prompt." Um, maybe Jeppa seems like a good option or one of the other prompt optimizers out there. Um, well, what do I need to optimize a prompt? Data in the same format. Uh, and how do I use the data? Well, step one, start the container. Step two, give an agent the instruction. Step three, let the agent interact with the container. Step four, run the test. Step five, return a reward and trajectory. And then step six, update the prompt. Okay, so the prompt made things a little bit better, but it still is not quite good enough. Well, how about we do RL, and we actually update the model weights now? Okay, well, what do I need to do RL? Data in the same format. And how do I use the data? Well, step one, I start the container. Step two, I give an agent the instruction. Step three, I let the agent interact with the container, run the tests, return a reward and tokens in this case, and then update the weights. So starting to feel a little bit familiar. Um, phew, my agent works great now. Uh, I think it's-- I'm successfully ready to deploy this agent to production. Um, but how can I make sure that if I modify it in the future, it doesn't negatively impact the agent performance without me knowing? Well, I should probably test it before I deploy it. What do I need for that? Data in the same format. And how do I use the data? Well, step one, I start the container. Step two, I give an agent the instruction, let the container interact with it, run the tests, and return a reward. So hopefully you guys have started to notice a pattern, and we didn't even talk about SFT or other forms of optimization or modifying agent harnesses. Um, but I think it's safe to say that there's a pattern of a type of process that everybody is coding over and over and over. In fact, every single benchmark that gets released, uh, at least, um, I would, I would say m-maybe all coding benchmarks that get released at this point, um, are some form of instruction container tests with some bespoke harness that was coded up that feels very analogous to every other harness that has been coded up. Or, or they don't even release a harness at all. So what we are here to tell you guys is you should stop writing the same code over and over again, and instead you should start using Harbor. So Harbor is a package that we've been working on based on all of our learnings from Terminal-Bench, and we're releasing it tomorrow, although you can technically go pip install it right now if you're really excited. And, um, Harbor specifically is our new package for, for evaluating and optimizing agents and models in container environments. Um, so specifically, Harbor has a standard task format, which is an iteration of the Terminal-Bench task format, um, which we found to be very flexible and can, uh, often express m-most of the existing evaluations. It, uh, has popular agents pre-integrated, so Codex, Cloud Code, all of these agents, uh, just plug and play right out of the box. Uh, it has a dataset registry with many popular benchmarks like Suitebench Verify pre-integrated, uh, integrations with popular prompt optimization and RL frameworks like SkyRL, and then out-of-the-box cloud deployments using Daytona, Modal, ETB or pick your favorite agent sandboxing solution. Um, and the goal is that in just a few lines of code, you can run thousands of sandbox rollouts in the cloud for evals and post-training. So yeah, we took everything that we learned, and we tried to make it as simple as possible for you guys to go do your agent development and to, uh, optimize your agents as necessary and understand how they work without having to manage all the infrastructure yourselves. So yeah, pip install harbor harbor run, uh, Terminal-Bench 2.0 in this case with Cloud Code and Haiku is a simple command from the command line. So to show you guys a little bit of what it looks like exactly to use Harbor, we're gonna invite Aitash to the front, who's a Stanford PhD student, uh, who's been working on a project called DC Agent, and he'll share a little bit about how it's, uh, how he's been using Harbor. ### Harbor Demo **Aatash** [15:01] Uh, yes, I think- **Alex Shaw** [15:02] Yeah, just share your screen. **Aatash** [15:03] Yeah. **Alex Shaw** [15:04] Cool. **Aatash** [15:04] Yep. **Alex Shaw** [15:08] Loud enough. All right. Can everyone see my screen? Okay, cool. Where's the button? So hello, my name is Aitash. I am a lab mate of Mike's and a student of Ludwig's at Stanford, and I've been working on with a group of large, a lot of contributors on this project called Data Comp Agent. This is a little bit of context. So as of recently, like many open source models, the frontier models have been targeting amazing benchmarks like Terminal-Bench and Suitebench, trying to target agentic behavior because it's very desirable for downstream users to have this. Um, but it's very natural to want to do this in the open. So the question is: how would you actually build such a model in the open? And right now it's very, very tricky. One of the problems is there's not enough data to do so. And so for our group and with this project is we want to be able to train an open source model to do agentic behavior in the open with a huge dataset of really high-quality agentic tasks. And so there's three main use cases of Harbor. So we have to do the whole model pipeline. So we start with data, we train the model on that data, and then we eval the model using these benchmarks. And all three use cases use Harbor. So Harbor is really, really important. So I want to walk through the first, um, examples of data generation. So let me pull up. So we operate on a slightly different scale. So we're, you know, we need thousands and thousands of tasks. So here's a finder of, uh, a specific dataset that I've created. So this is a batch textbook dataset. These are all, uh, tasks in the Harbor format. So Harbor standardizes format so that we can represent a bunch of different tasks similarly, and we can use these tasks for training. So it looks like doing instruction, which is like, you know, solve this batch script task, whatever it may be. We have a Docker file, which is maybe a very simple Docker file. Yeah, you can see it's extremely simple. Um, and with this very simple formulation of data, we can actually start generating traces from this. So let's say we want to do SFT. So we want to train to mimic a, a frontier model. So let's say I want to run GPT or Claude on this task and train on it. So I can just very much use very simple code. So Mike wanted me to show this code, so I'll show it. It's not that clean. Don't judge me. Can you zoom in? **Aatash** [17:28] Yeah, I can zoom in. It's not that clean, so don't judge me. Uh... So you can see that we create a job config just like you would in Harbor. You create an agent config, which defines your agent's like harness, the model name, and all that's taken under, taken care of underneath the hood. So you can use different agents like Open Hands or Terminus or what have you to generate the data. **Alex Shaw** [17:50] Can you zoom more? **Aatash** [17:51] Yeah. I think. **Alex Shaw** [17:53] Can you collapse your-- Yeah. **Aatash** [17:54] Yeah. Is this better? **Alex Shaw** [17:58] No. **Aatash** [17:58] More? Okay. **Alex Shaw** [18:00] There we go. Now we're talking. **Aatash** [18:04] Okay. **Alex Shaw** [18:04] Huge. **Aatash** [18:04] Okay. So yeah, we define an agent, we define an environment config. We can define how we like run the job, so like how we orchestrate it all through Harbor, and then we just run this very simple thing, job.run, and I'll run it real quick Okay. You can trust me that I've run... I'm running this, uh, traces on the Bash textbook that you just saw. See, if you go to, if I go to Daytona. So we're using... I can't run ten thousand, uh, Docker containers on my laptop. I've tried. Uh, you know, had to spend a day with Apple support. But if you look at Daytona, and you can see literally thousands of sandboxes are spinning up. So I'm running a thousand parallel sandboxes at once to collect data with GPT and running all those Bash textbooks and collecting traces, and I'll use that for data. So here's an example of what it looks like. This is one I've generated before. This is a hundred rows of, like, datasets where I've scraped. Um, I've used Harbor, uh, to interact with this task, and you can see the traces in terms format interacting with this task, and I can directly train on this. And so being able to generate thousands of traces with Harbor on thousands of different types of tasks with fine-tune models is a super nice use case for generating high-quality data. Um, and that's all for SFT. We use this throughout the entire model stack. So we do, we do RL, just like Mike mentioned. We RL on with Harbor in the back end to do a lot of good calculate rewards. And downstream, we eval on a bunch of different evals using Harbor as well. So Harbor is really critical to our entire stack. That's... They build a great thing. **Alex Shaw** [19:52] Thank you. **Mike Merrill** [19:54] Thank you, Aatash. Aatash actually set multiple records on Daytona, by the way, for most, most sandboxes ever created in a day. Their, their support loves us. **Alex Shaw** [20:10] And the bank account. **Mike Merrill** [20:13] Yeah, so, uh, I guess just to summarize, thanks so much for coming out, everyone. We have two very exciting things launching tomorrow. Terminal-Bench 2, an even better version of Terminal-Bench with, uh, increased verification, harder problems, and even more environments. And we're also launching Harbor, which is our new package for evaluating and optimizing agents and models. Um, we're gonna be announcing this tomorrow. Please follow us. Consider retweeting if you think this is something that's interesting and people might think is cool because I do, and if you're in this room, you probably do as well. And, uh, I, I also just, you know, on a personal note, I wanna say thanks so much to the community. I mean, this is a, this has been a fantastic open source effort by people from all walks of life, all corners of the country and the world coming together on Terminal-Bench and Harbor, and, uh, really wouldn't have been possible without you all. So, um, thanks so much, and looking forward to this next chapter. **Host** [21:13] But, and we also, uh, did an interview with, uh, Mike and Alex on Headspace- ### Behind the Scenes **Alex Shaw** [21:17] We did **Host** [21:17] ... a while ago. So, uh, thanks for having us back. Uh, we are joined by two new folks. I just wanted to give you guys a chance to introduce yourselves. **Alex Shaw** [21:24] Yeah. **Host** [21:24] Andy, maybe you wanna go first. **Andy Konwinski** [21:27] I'm Andy. I'm, uh, founder of Databricks and Perplexity, and this thing called LAWD. LAWD is, uh, uh, in collaboration with Ludwig and Stanford and Mike and the rest of the DC Agent and Terminal-Bench community working on th- things like this. **Ludwig Schmidt** [21:46] Well, hey, everyone. I'm Ludwig. I'm an assistant professor at Stanford, and I also work at Anthropic in the pre-training team. And yeah, together with Andy, Mike, and Alex, I was very excited to put Terminal-Bench together over the past year. **Host** [21:59] Did you work... Did you start DataComp? Did you, uh, did... What's your affiliation with DataComp? **Ludwig Schmidt** [22:05] I mean, okay, so DataComp is the series of papers that collaborators and I have done over the past few years around building datasets for AI training. So we started this back in the good old days, building datasets for clip, then for language model pre-training, then for reasoning, fine-tuning, and now for agent training data. **Host** [22:25] Yeah. Awesome. Um, okay, so the- there's, there's, like, a lot of, like, academic affiliations going around. I'm just kinda wondering, like, uh, how does something like this come together? Like, was it, was it your idea? Like, uh, you know, I, I think it was, like, kind of your research projects that you then talked to Andy about something like that, right? Like **Ludwig Schmidt** [22:43] Yeah. Okay. Um, basically on my side, around summer twenty twenty-four, I was starting to think around, like, okay, what would be a good benchmark to work on next after chatbot benchmarks? And I was wondering, okay, like, how good are language models going to be at, like, long horizon tasks? And I thought, okay, something in the terminal would be a good benchmark to do. And then I was chatting with Mike about this because at the time I was trying to get Mike to come to Stanford to do a postdoc with me. And then I think Mike mentioned, "Hey, you should talk with this guy, Andy." Uh, because Andy was thinking about something very similar, and then we decided to team up. **Host** [23:18] Yeah. This happens a lot, by the way. Andy gets a lot of the, "You should talk to Andy" calls. **Andy Konwinski** [23:23] And from my side, I was tinkering with agents for a long time, uh, and was really into SuiBench and MemGPT, and we-- And then Devin happened, and I thought, "Wow, that's an interesting demo. I wonder how much of it's smoke and mirrors." But it seemed cool that it was using the terminal, and I had this hypothesis that I put in some slide at some point that, like, maybe the terminal is one tool to rule them all because you can do almost anything on your computer. And maybe this dates back to when I was an undergrad, and I used to turn my mouse upside down in the computer lab at Wisconsin and try and do everything from the terminal, uh, 'cause I was, like, an operating systems nerd, and I wanted to, like, be a, a hacker at, like, Bill Joy, de- developing Unix. And so I had this fantasy that, like, really awesome computer nerds don't need the mouse, and we should just do everything in the terminal. And that carried with me to, oh, well, maybe agents can do everything in the terminal too. Then I ho-- I met up with Ludwig and Mike and other... Chris and others at NeurIPS, I think, is where we first connected about this idea. And Ludwig said, "I have this student that I've been talking to about this thing, CLI Bench." And I said, "Oh, I have just the slide that you are already building, so let's team up and have LAWD and Stanford, uh, bring this thing to reality." **Host** [24:39] Uh, I'll briefly mention LAAD, and then I'll double-click on Terminal Bench 2.0. Uh, so LAAD, you also announced, like, a bunch of slingshots, right? We also have another one of your, uh, supportees, I guess- **Andy Konwinski** [24:48] Yeah, I think there might be more than one in here now. **Host** [24:50] Who else? **Andy Konwinski** [24:51] Anybody who got a LAAD slingshot today. **Host** [24:53] Can you shout ... Yeah, can you shout out? Like, what, what, what was it they do? **Andy Konwinski** [24:55] On two? **Host** [24:56] So everyone knows Arc, so I'm not gonna ... I'm gonna skip over Arc. **Andy Konwinski** [24:58] Arc, and then Laksha, who is the lead on JePA. **Host** [25:01] Oh, okay. Everyone can say hi to him. **Andy Konwinski** [25:03] Yeah. Say hi to Laksha. Laksha's a rock star right there. Cool. Uh, who? Oh yeah, Lisa. Lisa, are you still here? **Host** [25:11] Somewhere. **Andy Konwinski** [25:12] She was here. She might had to leave. Too bad. Um, Berkeley Stanford ... Berkeley student in the Berkeley IRA research lab. So three, three of the slingshots. Well, how many did we announce? 12? 14? Something like that. Where's Braden? **Host** [25:25] Uh, uh, and- **Ludwig Schmidt** [25:26] Hello. **Andy Konwinski** [25:26] Yeah, so they ... These are research projects that LAAD is giving resources to. It's like a grant program. Uh, we don't only give cash, we give GPUs, we give, uh, data credits for doing data creation and labeling. We give access to industry introductions and advice around communications. We try and l- throw events, so there's community resources. We run meetups for these. And yeah, so basically it's a grant program to keep people doing awesome research in the open. **Host** [25:53] If someone here is wonder- **Ludwig Schmidt** [25:54] One addition, you also contribute, Alex, to projects. **Andy Konwinski** [25:57] That's true, yeah. **Host** [25:57] That's a, that's a big one. **Andy Konwinski** [25:58] So engineering resources. **Ludwig Schmidt** [25:58] That's a big one. **Andy Konwinski** [25:59] Yes. **Ludwig Schmidt** [25:59] Much better than GPU credits. **Andy Konwinski** [26:00] That's fair. **Host** [26:01] Alex, are you, like, a floating resource? Or what, what is this? **Alex Shaw** [26:03] Well, I'm a GPU. No, um- I, I joined LAAD to work on the K Prize originally, which was this $1 million prize around SuiteBench. Uh, and then we worked on that for about three months, launched it in December, which is the exact same time that we met Ludwig and Mike. And then, uh, after that, yeah, I kind of spent all of my time working on Terminal Bench with Mike. So we kind of just organically became the co-creators of it. **Andy Konwinski** [26:26] And Terminal Bench is the largest investment we made in w- any one of the slingshots so far. **Host** [26:30] Got it. Got it. Um, okay. Terminal Bench 2.0, um, I think a lot of, like, notable things. I really like that slide where you were, like, calling out issues that you fixed. Um, any qualitative stories behind the scenes of, like- **Ludwig Schmidt** [26:43] Behind the scenes **Host** [26:43] ... developing the new ones that you want to shout out? So as you, you said they were all backed by real work. Uh, you paid a real human to do it or something? **Alex Shaw** [26:53] We didn't pay anyone. We used an incentive that's more powerful in the world of academia, which is co-authorship. **Host** [27:00] Ooh. **Alex Shaw** [27:00] And, uh, we, we, we told people if they created three tasks for Terminal Bench, they could be a co-author on the paper that we eventually published. And I think we got maybe 250 task contributions, and 89 of them made it into the benchmark, and the reason a task might not make it into the benchmark is if it was too easy, or if we didn't feel that it quite lived up to the quality standards that we were hoping for. And it's actually very hard ... Uh, we learned a lot of lessons on what makes a good task, and it's very hard to, to, to both to build a high-quality task and to, to judge whether a task is high quality. So, um, yeah. **Host** [27:36] Interesting. Uh, yeah, I mean, I think that's, that's also, like ... Oh, I think you want to add something. **Andy Konwinski** [27:40] Well, I wanted Ludwig maybe to chime in because Ludwig has done this before, bringing together collaborators, and it's a pretty unique thing in Ludwig's research style to get so many PhD students and researchers across many different groups, uh, multiple faculty, like four or five faculty members with their PhD students coming and dedicating many hours to ch- creating this human PhD brain-generated data set, and that's the thing that the frontier labs are actually paying shit tons of money to generate with companies like Surge and Scale and the rest. **Host** [28:14] Of course. **Andy Konwinski** [28:14] And, uh, being able to do, to find an incentive structure in the open ecosystem that gets extremely high-quality training, uh, training and/or in this case evaluation data generated is, like, it's extreme ... It's, like, singular and hard to put a value on. So, uh, it's really cool. So I don't know if you wanted to say a little bit more about how you did it in past projects or ... **Ludwig Schmidt** [28:37] I mean, okay, so I think a lot of the credit really goes to Mike and Alex, so, like, shepherding the open source community and so on was really great to see so many people contribute, and this is really a big team effort. Um, I think as Alex said, like, if you make it a paper and you give everyone a share in the project, this is a little bit like in the startup ecosystem, right? Everyone gets a little bit of equity. Here everyone gets a little bit of co-authorship. And, um, yeah, I mean, I think the important thing is sort of, like, having a network of people who are interested in similar things. You find other people who are excited about the same project, start building it together, and then as the thing gains momentum, you can add more people. And yeah, I think this has worked very well for in the past with the DataComp projects, worked well with Terminal Bench, so excited about that approach in open source. ### Looking Ahead **Host** [29:20] Yeah, we've covered DataComp every NeurIPS, uh, and when I saw you guys all had the same shirts, I was like, "These guys, like, super well organized." Like, very abnormal, um, project that you might see in, in academia. **Ludwig Schmidt** [29:32] Right. **Host** [29:33] Uh, okay. Uh, I just, I can't be ... Um, you know, I can't have Greg here and not make comparisons with Terminal Bench 2.0 and ArcadeGI 2 and, like, that developments. You know, something that, uh, Greg is sort of pushing is for evaluations based on games or any, any, uh, sort of much more long-term a- agentic, uh, sort of intelligence. Any inspiration in, like, developing the, the Arc of Terminal Bench? Like, is there, is there a Terminal Bench 3? Um, what are the main directions that you're interested in pushing this? **Andy Konwinski** [30:04] I, I, I ... We haven't talked exactly about a Terminal Bench 3, but I, I think something that is very exciting about what we've landed on here is that virtually any task you can think of performing on a computer can be expressed as a Terminal Bench task. Like, I, I, I think if, if we can- **Host** [30:17] I can challenge you on that, but yeah, keep, keep going **Andy Konwinski** [30:19] ... I'd love, I'd love to hear that. **Host** [30:20] Yes. **Andy Konwinski** [30:20] But if we, if, if ... I think if we made any mistake, it was calling it Terminal Bench. It's really about- **Host** [30:25] Oh **Andy Konwinski** [30:25] ... it's really about containers. It's about environments. **Host** [30:28] Okay. **Andy Konwinski** [30:28] And some agents interact with these environments through terminals, but that's not necessarily the only way that they could go about doing it. And so with that in mind, I'd, I'd love to see a version of Terminal Bench around scientific computing. We have a handful of tasks that are in this vein already, but I think it would be great to bring the scientific community together to create, like- **Mike Merrill** [30:46] A state-of-the-art, very hard scientific benchmark where success on that benchmark would mean accelerating the pace of science. **Andy Konwinski** [30:54] A- and so j... Can you give a few short examples as to what does scientific computing mean here? **Mike Merrill** [30:57] Sure, sure. So we have a couple of examples like this already in the benchmark. You know, I mentioned we have this one that involves DNA assembly- **Andy Konwinski** [31:03] Yeah **Mike Merrill** [31:04] ... modifying strings that will then be turned into proteins to ach- achieve certain goals. Much of this goes above my head. We have great contributors who figure out these tasks and vet them for us. Um, you know, we have another task that involves understanding, uh, spectra. I, I think, I think, you know, much of a scientist's work looks like a computer scientist. It's spending time on a computer, writing code to solve their problems, and if we can make this faster and easier, we can hopefully accelerate scientific progress. **Andy Konwinski** [31:37] There's a, a professor in astronomy at Caltech that I met recently and related to a scientific grant program that we run at Law called the Moonshot Grant Program, which is trying to fund scientists using AI to make scientific progress, and this professor is really well known because she took the first picture of a black hole and- **Mike Merrill** [31:57] Oh, yeah **Andy Konwinski** [31:57] ... uh, that was in the news about- **Mike Merrill** [31:58] Katie something. **Andy Konwinski** [31:59] Yeah. Uh- **Mike Merrill** [32:00] She's a professor now? **Andy Konwinski** [32:01] I, I think so. **Mike Merrill** [32:02] Okay. **Andy Konwinski** [32:02] Uh- **Mike Merrill** [32:04] Nice. **Andy Konwinski** [32:04] Yeah. Professor, did I get that right? Okay, so fact check me. Um, so that's a great example of people using AI and models- **Mike Merrill** [32:12] Yeah **Andy Konwinski** [32:12] ... to, uh, like analyze images to, um, kind of- **Mike Merrill** [32:16] Yeah. And also- **Andy Konwinski** [32:17] ... measure and take pictures of the phenomena. **Mike Merrill** [32:19] Uh, we did the CZI, uh, Chan Zuckerberg pod today, uh, and I, I went to the image- imaging institute and they did, they do so much 3D imaging of cells, and they, they sort of slice layer by layer, and it's, uh, it's very computationally intensive. So you g- you got me, you got me some idea. Uh, my challenge was mostly that when you build something like Harbor and you, you mentioned, like, the terminus format, is that what's called? Uh, you kind of lock in an opinion on what an agent is, and that changes every year. Uh, one of the things that I was thinking about when I was looking at it was like, okay, this is very tech space, but like, you know, something that I'm seeing that, uh, you know, sitting at cognition I'm seeing is that computer use is coming back. Uh, and so n- my challenge to you is kind of y- you know, how do you know what a g- what an agent is? You're kind of like locking it down right now. **Andy Konwinski** [33:03] I, I, I, I have thoughts on this, but I see Alex is locked and loaded, so **Alex Shaw** [33:07] I- **Andy Konwinski** [33:08] Go for it. **Alex Shaw** [33:08] I, I can give my answer, and then maybe Mike will add more to it. So, uh, the thing about an agent is r- really, no matter, no, no matter what exactly it does or what exactly its implementation is, I think we even had it on one of our slides, it's a program. Like, right? It's this program that you execute, and it ultimately does something. **Andy Konwinski** [33:25] That may not stop. That may improve over time with memory. **Alex Shaw** [33:28] Yeah, exactly. Like, it could, it could run for a really long time, and it could be, like, doing all sorts of things, c- curating context and stuff like that. But, um, i- if you can just frame any agent as a program that gets executed, then that program can be executed inside the container, and you could also have, uh, the GUI running inside the container too, and the agent could be interacting with the GUI in the container by, uh, scrolling, clicking, typing, stuff like that, um, just depending on what its tools are implemented as. **Mike Merrill** [33:58] Yeah. I mean, I was just gonna say, like, we, we like text-based agents. We think right now they're the frontier of agentic performance. This, this very well may change. A- and you're right, I think it doesn't make sense to lock in too firmly on an abstraction. And so, as Alex said, something that's great about Harbor is if you can install and run your agent, then you can evaluate it inside the framework. **Andy Konwinski** [34:17] Thanks. **Mike Merrill** [34:17] So that's, that's kind of what I mean when I say I think just about any agentic task, although I guess infinitely running ones are interesting and something to think about, could be expressed in this format. **Andy Konwinski** [34:28] Okay, lovely. I don't want to take up too much time. Any parting words? Any, uh- **Mike Merrill** [34:32] Thank you. **Alex Shaw** [34:33] I, I, I, I would say one thing, which is, um, we, we really, uh, want Terminal Bench 2 to be adopted a- as widely as possible because we believe it's a good benchmark, and the number one thing that you guys can help us with is to help amplify our announcements tomorrow. So anything you guys can do, if you also want this benchmark to be widely adopted and used to improve models and agents in the future, uh, that would be amazing. So thank you so much already for, for coming tonight. **Andy Konwinski** [34:58] Yeah. Thank you. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.