# Best of 2024 in Agents (from #1 on SWE-Bench Full, Prof. Graham Neubig of OpenHands/AllHands) Latent Space · 2024-12-25 Professor Graham Neubig of CMU and AllHands AI, maintainer of the open-source coding agent framework OpenHands, delivers a talk on the state of agents in 2024, arguing that agents are now capable of automating significant portions of software development but still struggle with information gathering and error recovery. He demonstrates OpenHands solving real coding tasks live, explains key design choices like using Python code execution over granular tool calls to reduce LLM invocations, and shares that Claude currently outperforms other models for agentic tasks due to superior instruction following and error correction. Neubig discusses eight perennial agent problems including agent-computer interface, human-agent interface, model selection, planning, workflows, exploration, search, and evaluation. He predicts every major LLM trainer will focus on agent-oriented models by mid-2025, that agent benchmarks like SWE-Bench and WebArena will saturate and require harder successors, and that the biggest challenge will be designing human-agent interfaces for systems that succeed only 30-40% of the time autonomously but 80-90% with human feedback. The talk also covers workflow memory as a… ## Questions this episode answers ### How does OpenHands let agents use tools without making dozens of separate tool calls? Graham Neubig explains that OpenHands gives coding agents a few core tools (program execution, file editing, web browsing) but allows them to write and execute arbitrary Python code that calls these tools. Instead of making 30 separate LLM calls for a multi-step task, the agent writes a single program, runs it, and fixes errors. This CodeAct approach reduces complexity and significantly improves performance. [8:32](https://addtry.com/f1a1aa31-7bc2-46cd-9635-ef4912a6eb7e?t=512000) ### Which LLM is best for building coding agents, and why? Graham states Claude is currently the top model for agentic tasks like coding. It excels in instruction following, tool use, environment understanding, and error recovery—often saying “let me try a different approach” rather than looping. In their evaluation, Claude was “head and shoulders above” GPT-4.0, o1-mini, Llama-405b, and DeepSeek 2.5. [14:06](https://addtry.com/f1a1aa31-7bc2-46cd-9635-ef4912a6eb7e?t=846000) ### Why do coding agents like OpenHands underperform on real repositories compared to SWE-Bench? Graham notes that SWE-Bench issues come from popular open-source repos that were included in the models’ training data, so the models often already know the repos or specific issues—inflating benchmark scores. On private or newer repos, success drops to 30–40% because the model faces genuinely unfamiliar code and problems. [32:42](https://addtry.com/f1a1aa31-7bc2-46cd-9635-ef4912a6eb7e?t=1962000) ## Key moments - **[0:00] Intro** - [0:27] A highly competent human with a web browser, terminal, file system, and text editor can do everything. - **[1:46] Demo** - [2:15] OpenHands agent creates scatter plots of SWE-Bench score increase during NeurIPS talk. - [5:15] OpenHands agent finishes data science task, ranking itself #1 on SWE-Bench Normal, while Graham was presenting. - [6:38] OpenHands' code-generation approach to tool use reduces LLM calls and was instrumental to SWE-Bench success. - **[7:11] Computer Interface** - [9:01] OpenHands provides only 5 tools to agents: program execution, file editing, search/replace, and web browsing. - **[10:48] Human Interface** - [11:55] OpenHands' GitHub plugin lets users tag @OpenHands agent to fix issues and tests directly in PRs. - **[13:15] Model Choice** - [13:15] Claude is the best foundation model for coding agents, outperforming GPT-4o due to better error recovery and avoiding loops. - **[15:42] Planning** - [15:42] Don't Sleep on Single Agent Systems: Graham Neubig argues single prompt with light planning beats multi-agent for flexibility. - [18:22] Agent Workflow Memory achieves 22.5% absolute improvement on WebArena by self-learning from 40 successful task examples. - **[20:08] Exploration** - [20:08] Agent Lists improves coding agents by creating a repository map; Bagel learns website structure through exploration. - [21:13] Tree search for language agents expands multiple solution paths and rewinds on failure, but is harder on web than code. - **[22:12] Evaluation** - **[23:46] Predictions** - [23:46] Every large LLM trainer will focus on training models as agents by mid-2025, predicts Graham Neubig. - [25:44] Graham Neubig predicts 2025 agent benchmarks will be too easy, prompting harder evaluations; 2026 maybe AGI? - [27:00] Graham Neubig estimates OpenHands solves 30-40% of issues autonomously, but reaches 80-90% with human feedback. - [28:02] Graham argues every website may need an API because API-based web agents significantly outperform browser-based navigation. - [28:57] Agents building agents will accelerate progress even further, predicts Graham Neubig. - **[29:12] Call to Action** - [29:12] Graham highlights Duolingo's cross-subsidy model as a template for making powerful agent tools globally accessible. - **[31:16] Q&A** - [31:53] Q: Why does OpenHands solve only 30-40% of issues when SWE-Bench shows 53%? A: Training data leak artificially boosts SWE-Bench scores. - [32:27] Q: How can agents learn to ask for help when stuck? A: Current agents are poor at it; they request help at wrong times. - [34:29] Q: How do web agents interact with websites? Three methods: pixel clicking, accessibility tree parsing, and hybrid approaches. - [38:45] Q: One general agent or multiple specialized ones? A: OpenHands uses micro-agents triggered by keywords for task-specific instructions. - [40:43] Q: Will OpenHands adopt Anthropic's Model Context Protocol? A: No, it duplicates exsiting APIs and adds server overhead without clear benefit. - [42:46] Q: What do agents consistently fail at? A: Insufficient information gathering before solving the task is the top failure mode. ## Speakers - **Graham Neubig** (guest) ## Topics Agent Platforms ## Mentioned All Hands AI (company), Anthropic (company), Aider (product), Amazon Q Agent (product), Anthropic Model Context Protocol (product), Claude (product), DeepSeek (product), Devin (product), GPT-4.0 (product), GitHub (product), Grok (product), Livecode Bench (product), Llama (product), Mini World of Bits (product), OpenHands (product), Resend.com (product), SWE-Bench (product), WebArena (product), npm (product), o1-mini (product) ## Transcript ### Intro **Graham Neubig** [0:07] Okay. Uh, hi, everyone. So, um, I was given the task of talking about agents in twenty twenty-four, and this is an impossible task because there are so many agents, so many agents in twenty twenty-four. So this is gonna be strongly covered by, like, my personal experience and what I think is interesting and important. But, um, I think it's an important topic, so let, let's go ahead. So, um, the first thing I'd like to think about is, let's say I gave you, uh, you know, a highly competent human some tools. Um, let's say I gave you a web browser and a terminal or a file system and the ability to edit, uh, text or code. Um, what could you do with that? Everything. Yeah, probably a lot of things. This is like ninety-nine percent of my, you know, daily, uh, daily life, I guess, uh, when I'm, when I'm working. Uh, so I think this is a pretty powerful tool set. And what I am trying to do and what I, I think some other people are trying to do is come up with agents that are able to, you know, manipulate these things: web browsing, coding, running code, uh, in successful ways. So there was a little bit about my profile. Um, I'm a professor at CMU, chief scientist at All Hands AI, building open source coding agents. Um, I'm maintainer of OpenHands, which is an open source coding agent framework. And, um, I'm also a software developer, and, uh, I, I like, uh, doing lots of coding and, and, you know, shipping new features and stuff like this. So building agents that help me to do this, you know, is kind of an interesting thing very close to me. Um, so the first thing I'd like to do is I'd like to try some things that I haven't actually tried before. ### Demo **Graham Neubig** [1:58] Uh, if anybody has, you know, tried to give a live demo, you know this is, uh, very, very scary, uh, whenever you do it, and it might not work, so it might not work this time either. Um, but I wanna show you, like, three things that I typically do with coding agents in my everyday work. I use coding agents maybe five to ten times a day, uh, to help me solve my own, uh, problems. And so this is a first one. This is a data science task which says, uh, I want to create scatter plots that show the increase of the SWE-Bench score over time. Um, and so I, I wrote a, a kind of concrete prompt about this. Agents work better with, like, somewhat concrete prompts. And I'm gonna throw this into OpenHands, um, and let it work. And I'll, I'll go back to that in a second. Um, another thing that I do is I create new software. Um, and I, I've been using a service, uh, a particular service, I won't name it, uh, for sending emails, and I'm not very happy with it. So I want to switch over to this new service called Resend.com, which makes it easier to send emails. And so I'm going to ask it to read the docs for the Resend.com API and come up with a script that allows me to send emails. The input to the script should be a CSV file and, uh, the subject and body should be provided in Jinja 2 templates. So, um, I'll start another agent and, um, and try to get it to do that for me. And let's go with the last one. The last one I, I do is improving existing software. And, uh, in order-- You know, once you write software, you usually don't throw it away. You go in and, like, actually improve it iteratively. Um, this software that I have is, uh, something I created without writing any code. Um, it's basically software to monitor how much our, um, our agents are contributing to the OpenHands repository. And on the... Let me make that a little bit bigger. On the left side, I have the number of issues where it, like, sent a pull request. Um, I have the number of issues where it, like, sent a pull request, whether it was merged in purple, closed in red, or, um, is still open in green. And so these are, um, like... You know, it, it's helping us monitor. But one thing it doesn't tell me is the total number, and I kind of want that feature added to this, uh, software. So I'm gonna, uh, try to add that too. So, um, I'll take this. I'll take this prompt. And here, um, I, I want to open up specifically that GitHub repo. So I'll open up that repo and, uh, paste in the prompt asking it-- I asked it to make a pie chart for each of these and give me the total over the entire time period that I'm monitoring. So, um, we'll do that. And so now I have, um... Let's see. I have some agents. Oh, this one already finished. Let's see. So this one already finished. Um, you can see it, it finished analyzing the SWE-Bench repository. It wrote, um, a demonstration of... Yeah, I'm, I'm trying to do that now, actually. It wrote a demonstration of, um, how much each of the systems have improved over time, and I asked it to label the top three, uh, for each of the datasets. And so it labeled OpenHands as being the best one for SWE-Bench Normal Um, for SWE-Bench Verified, it has, like, the Amazon Q Agent and OpenHands. For the SWE-Bench Light, it has, um, uh, three here, uh, over, um, three over here. So you can see, like, that's pretty useful, right? If you're a researcher, you do data analysis all the time. I did it while I was talking to all of you and making a presentation, so that's, that's pretty nice. Um, uh, I, I doubt the other two are finished yet. That would be impressive if they-- Yeah, so I think they're still working, so maybe we'll get back to them at the end of the pre-presentation. But, um, so th-these are the kinds of, uh, these are the kinds of things that I do every day with coding agents now. Um, and it's, uh, or software development agents. It's pretty impressive. Um, the next thing I'd like to talk about a little bit is things I worry about when designing agents. So we're designing agents to, you know, do a very difficult task of, like, navigating websites, um, writing code, other things like this. And within twenty twenty-four, there's been, like, a huge improvement in the methodology that we use to do this. Um, but there's a bunch of things we think about. There's a bunch of interesting papers, and I'd like to introduce a few of them. So the first thing I worry about is, um, the agent computer interface. Like, how do we get an agent to interact with computers? And, um, how do we provide agents with the tools to do the job? And within OpenHands, um, we are doing the thing on the right, but there's also a lot of, uh, agents that do the thing on the left. ### Computer Interface **Graham Neubig** [7:34] So the thing on the left is you give, like, agents kind of granular tools. You give them tools like, um... Or, or l-let's say your instruction is, "I want to determine the most cost-effective country to purchase the smartphone model CodeAct one. Uh, the countries to consider are the USA, Japan, Germany, and India." And you have a bunch of available APIs. And so what you do for some agents is you provide them all of these tools, uh, APIs as tools that they can call. And so in this particular case, um, in order to solve this problem, you'd have to make about, like, 30 tool calls, right? You'd have to call lookup rates for, um, Germany. You'd have to look it up for the US, Japan, and India. That's four tool calls. And then you'd go through and do all of these things, um, separately. And the method that we adopt in OpenHands instead is we provide these tools, but we provide them by just giving a coding agent the ability to call arbitrary Python code. Um, and in the arbitrary Python code, it can call these tools. Uh, we expose these tools as APIs that the model can call. And what that allows us to do is instead of writing 20 tool calls, making 20 LLM calls, you write a program that runs all of these all at once, and it gets the result. And of course, it can execute that program. It can, you know, make a mistake. It can get errors back and, and fix things, um, but that makes our job a lot easier. And this has been really, like, instrumental to our success, I think. Um, another part of this is what tools does the agent need? And I, I think this depends on your use case. We're kind of extreme, um, and we're only giving the agent five tools, um, or maybe six tools. And what, what are they? Uh, the first one is program execution. So it can execute bash programs, and it can execute, uh, Jupyter notebooks. It can execute cells in Jupyter notebooks. Um, so th-those are two tools. Um, another one is a file editing tool. Um, and the file editing tool allows you to browse parts of files, um, and, uh, kind of read them, overwrite them, other stuff like this. And then we have another global search and replace tool. So it's actually two tools for file editing. And then a final one is web browsing. Web browsing, I'm kind of cheating when I call it only one tool. You actually have, like, scroll and text input and, and click and other stuff like that. But, um, these are basically the only things we allow the agent to do. Um, what... Then the question is, like, what if we want it to allow it to do something else? Um, and the answer is, well, you know, human programmers already have a bunch of things that they use. Uh, they have the requests PyPI library. They have, um, the PDF to text PyPI library. They have, like, all these other libraries in the Python ecosystem that they can use. And so if we provide a coding agent with all these libraries, it can do things like data visualization and other stuff that I just showed you. So, um, it can also get clone repositories and, and other things like this. Um, the agents are super good at using the GitHub API also, so they can do, you know, things on GitHub, like finding all of the, you know, comments on your issues or checking GitHub actions and stuff. Um, the second thing I think about is the human agent interface. So this is, like, how do we get humans to interact with agents? Um, I already showed you one variety of our human agent interface. It's basically a chat window where you can browse through the agent's results and things like this. ### Human Interface **Graham Neubig** [11:02] Um, this is very, very difficult. Um, I, I don't think anybody has a good answer to this, and I don't think we have a good answer to this. But, um, the, the guiding principles that I'm trying to follow are we want to present enough info to the user, so we want to pre- uh, present them with, you know, what the agent is, uh, doing in the form of a kind of English description. So you can see here, um, you can see here every time it takes an action, it says, like, "I will help you create a script for sending emails." Um, when it runs a bash command... Um, sorry, that's a little small. When it runs a bash command, it will, um, say, "Ran a bash command." It won't actually show you the whole bash command or the whole Jupyter notebook because they can be really large, but you can open it up and see, um, if you want to by clicking on this. So, like, um, if you want to explore more, you can click over to the Jupyter notebook and, and see what's displayed in the Jupyter notebook, and you get, like, lots and lots of information. Um, so that's one thing. Um- Another thing is go where the user is. So like if the user's already interacting in a particular setting, uh, then I'd like to, you know, integrate into that setting, but only to a point. Um, so at OpenHands, we have a chat UI for interaction. We have a GitHub plugin for tagging and resolving issues. So basically, what you do is you, um, do @OpenHands agent, and the OpenHands agent will, like, see that comment and be able to go in and fix things. So if you say, "@OpenHands agent, uh, tests are failing on this PR, please fix the tests," it will go in and fix the tests for you and stuff like this. Um, another thing we have is a remote, uh, runtime for launching headless jobs. So if you want to launch like a fleet of agents to solve, you know, uh, five different problems at once, you can also do that through an API. So we have, um, we have these interfaces. And this probably depends on the use case. So like depending... If you're a coding agent, you want to do things one way. If you're a, like, insurance auditing agent, you'll want to do things other ways, obviously. ### Model Choice **Graham Neubig** [13:15] Um, another thing I think about a lot is choosing a language model. Um, and for agentic LMS, uh, we have to have a bunch of things work really well. Um, the first thing is really, really good instruction following ability. And if you have really good instruction following ability, it opens up like a ton of possible applications for you. Um, tool use and coding ability. So if you provide tools, it needs to be able to use them well. Environment understanding. So, um, it needs-- Like, if you're building a web agent, it needs to be able to understand web pages either through, uh, vision or through text. Um, and error awareness and recovery ability. So if it makes a mistake, it needs to be able to, you know, figure out why it made a mistake, come up with alternative strategies, and other things like this. Um, under the hood in all of the demos that I did now, um, Claude, we're using Claude. Um, Claude has all of these abilities. Uh, very good. Not perfect, but very good. Um, most others don't have these abilities quite as much. So like GPT-4.0 doesn't have very good, uh, error recovery ability. And so because of this, it will go into loops and do the same thing over and over and over again, whereas Claude does not do this. Um, Claude, i-if you, if you use the agents enough, you get used to their kind of like personality, and Claude says, "Hmm, let me try a different approach," uh, a lot. So, you know, o-obviously, it's been trained in some way to, you know, elicit this ability. Um, we did an evaluation. This is old, um, and we need to update this basically, but, um, we evaluated Claude, GPT-4.0, o1-mini, um, Llama-405b, DeepSeek 2.5 on being a good code agent within our framework, and Claude was kind of head and shoulders above the rest. Um, GPT-4.0 was kind of okay. The best open source model was Llama 3.1 405b. Um, this needs to be updated because this is like a few months old by now and, you know, things are moving really, really fast. But I still am under the impression that Claude is the best. The other closed models are, you know, not quite as good and then, uh, the open models are a little bit behind that. Grok, I-- We haven't tried Grok at all, actually. So it's a good question. If you wanna try it, um, uh, I'd be happy to help. Yeah. Cool. Another thing is planning. Um, and so there's a few considerations for planning. Um, the first one is whether you have a curated plan or you have it generated on the fly. And so for solving GitHub issues, you can kinda have, uh, an overall plan. ### Planning **Graham Neubig** [16:00] Like, the plan is, um, first reproduce. If there's an issue, first write tests to reproduce the issue or to demonstrate the issue. After that, run the tests and make sure they fail, then go in and fix the tests. Run the tests again to make sure they pass, and then you're done. So that's like a pretty good workflow for, like, solving, uh, coding issues. And you could curate that ahead of time. Um, another option is to let the language model basically generate its own plan. Um, and both of these are, are perfectly valid. Um, another one is explicit structure versus implicit structure. So let's say you generate a plan. Um, if you have explicit structure, you could like write a multi-agent system and the multi-agent system would have your, uh, reproducer agent, and then it would have your, um, your bug, uh, your test writer agent and your bug fixer agent and, uh, lots of different agents, and you would explicitly write this all out in code and then, uh, then use it that way. On the other hand, you could just provide a prompt that says, "Please do all of these things in order." Um, so in OpenHands, we do very light planning. Um, we have a single prompt. We don't have any multi-agent systems. Um, but we do provide like instructions about like what to do first, what to do next, and other things like this. Um, I'm not against, uh, doing it the other way, but I laid out some, uh, kind of justification for this in this blog called Don't Sleep on Single Agent Systems. And the basic idea behind this is if you have a really, really good instruction following agent, um, it will follow the instructions as long as things are working according to your plan. But let's say you need to deviate from your plan, you still have the flexibility to do this. And if you do explicit structure through a multi-agent system, it becomes a lot harder to do that. Like, you get stuck, um, when things deviate from your plan. Um, there's also, uh, some other examples, and I wanted to introduce a few papers. Uh, so one paper I liked recently is this paper called CoAct, um, where you generate plans and then go in and fix them. Um, and so the basic idea is like if you need to deviate from your plan, you can, uh, you know, figure out that your plan was not working and go back and deviate from it. Another thing I think about a lot is specifying common workflows. So we're trying to tackle, uh, software development, and I already showed, like, three use cases where, um, we do software development. And, uh, when we do software development, we do a, a ton of different things, but we do them over and over and over again. So just to give an example, um, we fix GitHub actions when GitHub actions are failing, and we do that over and over and over again. That's not the number one thing that software engineers do, but it's, uh, you know, high up on the list. So how can we get a list of all of, like, the workflows that people are, are working on? And there's a few research works that people have done in this direction. Um, one example is manual prompting. So there's this nice paper called Step that got state-of-the-art on the WebArena web navigation benchmark, where they came up with a bunch of manual workflows for solving different web navigation tasks. And we also have a paper recently called Agent Workflow Memory, where the ba-basic idea behind this is we want to create self-improving agents that learn from the past, their past successes. And the way it works is we have a memory that has an example of lots of the previous workflows that people have used. And every time the agent finishes a task and it self-judges that it did a good job at that task, you take that task, you break it down into individual workflows included in that, and then you put it back in the prompt for the agent to work next time. And this, um, we demonstrated that this leads to, um, a twenty-two point five percent increase on WebArena, um, after, uh, forty examples. So that's a pretty, you know, huge increase by kind of self-learning and self-improvement. ### Exploration **Graham Neubig** [20:08] Another thing is exploration. Um... Oops. And, uh, one thing I, I think about is, like, how can agents learn more about their environment before acting? And I work on coding and web agents, and there's, you know, a few good examples of this in, in both areas. Um, within coding, I view this as, like, re-repository understanding, understanding the code base that you're dealing with. And, uh, there's an example of this or a couple examples of this, one example being Agent Lists, where they basically d- create a map of the repo, and based on, on the map of the repo, they feed that into the agent, so the agent can then navigate the repo and, uh, and better know where things are. And for web agents, uh, there's an example of a paper called Bagel, and basically what they do is they have the agent, um, just do random tasks on a website, explore the website, better understand the structure of the website, and then after that they, um, they feed that in as a part of the prompt. Um, part seven is search. Um, right now in OpenHands, we just let the agent go on a linear search path, so it's just solving the problem once. Um, we're using a good agent that can kind of, like, recover from errors and try alternative things when things are not working properly, but still we only have a, a linear search path. But there's also, um, some nice work in twenty twenty-four, um, that is about exploring multiple paths. So one example of this is there's a paper called Tree Search for Language Agents, and they basically expand multiple paths, um, check whether the paths are going well, and if they aren't going well, you rewind back. Um, and on the web, this is kind of tricky because, like, how do you rewind when you accidentally ordered something you don't want on Amazon? It's kind of, you know, not, not the easiest thing to do. For code, it's a little bit easier because you can just revert any changes that you made. Um, but I, I think that's an interesting topic too. And then finally, evaluation. Um, so within our development for evaluation, we want to do, um, a number of things. The first one is fast sanity checks. Um, and in order to do this, we want things we can run really fast, really, uh, really cheaply. ### Evaluation **Graham Neubig** [22:27] So, um, for web, we have something called Mini World of Bits, which is basically these trivial, um, kind of, uh, web navigation things. Um, we have something called the Aider code editing benchmark, where it's just about editing, uh, individual files that we use. Um, but we also want highly realistic evaluation. So for the web, we have something called WebArena that we created at CMU. This is web navigation on real, um, real open source websites. So it's open source websites that are actually used to serve, um, shops or, uh, like bulletin boards or other things like this. Um, and for code, we use SWE-Bench, which, um, I think a lot of people may have heard of. It's basically a coding benchmark that comes from real world pull requests on GitHub. So if you can solve those, you can also probably solve other, uh, real world pull requests. Um, I, I would say we still don't have benchmarks for the fir- full versatility of agents. So for example, um, we don't have benchmarks that test whether agents can code and do web navigation. Um, but we're working on that and hoping to release something in the next week or two. Uh, so, uh, if that sounds interesting to you, come talk to me and, uh, I, I will, uh, tell you more about it. ### Predictions **Graham Neubig** [23:46] Cool. Um, so I, I don't like making predictions, but, um, I, I was told that I should be somewhat controversial, I guess. So I will I will try to do it, uh, try to do it anyway, although maybe none of these will be very controversial. Um, uh, the first thing is agent-oriented LLMs, uh, like large language models for agents. Um, my, my prediction is every large LLM trainer will be focusing on training models as agents. So every large language model will be a better agent model, uh, by mid-twenty twenty-five. Uh- Competition will increase, prices will go down, smaller models will become competitive as agents. So right now, actually, agents are somewhat expensive to run in some cases, but I expect that that won't last six months. I, I bet we'll have much better agent models, uh, in six months. Um, another thing is instruction following ability, specifically in agentic contexts, will increase. And what that means is we'll have to do less manual engineering of agentic workflows and be able to do more by just prompting agents in more complex ways. Um, Claude is already really good at this. It's not perfect, but it's already really, really good, and I expect the other models will catch up to Claude pretty soon. Um, error correction ability will increase, less getting stuck in loops. Again, this is something that Claude's already pretty good at, um, and I expect the others will, will follow. Agent benchmarks. Um, agent benchmarks will start saturating. So right now we have, uh, WebArena, um, and SWE-Bench. Um, I think WebArena is already too easy. Um, it, it is... It's not super easy, but it's already a bit too easy because the tasks we do in there are ones that take like two minutes for a, a human, so, uh, not, not too hard. Um, and kind of historically, in 2023, um, our benchmarks were too easy, so we built harder benchmarks like WebArena and SWE-Bench were both built in 2023. In 2024, our agents were too bad, so we built agents, um, and now we're building better agents. Um, in 2025, our benchmarks will be too easy, so we'll build better benchmarks, I'm, I'm guessing. So I would expect to see much more challenging, uh, agent benchmarks come out, and we're already seeing some of them. In 2026, I don't know. I, I didn't write AGI, but, uh, we'll, we'll, we'll see. Um, then the human agent computer interface. Um, uh, I think one thing that we're, we'll want to think about is what do we do at seventy-five percent success rate at things that we, like, actually care about? Um, right now we have fifty-three percent or fifty-five percent on SWE-Bench Verified, which is real world GitHub PRs. Um, my impression is that the actual, actual ability of models is maybe closer to thirty to forty percent. So thirty to forty percent of the things that I want an agent to solve on my own repos, it just solves, uh, without any human intervention. Uh, eighty to ninety percent it can solve without me opening an IDE, uh, but I need to give it feedback. Um, so w- how do we, how do we make that interaction smooth so that humans can audit the work of agents that are really, really good but not perfect, um, is going to be a big challenge. Um, how can we expose the power of programming agents to other industries? So like, um, as programmers, I think not all of us are using agents every day in our, our programming, although we probably will be, um, in, uh, in months or maybe a year. Um, but I, I think it will come very naturally to us as programmers because we know code, we know, you know, like, how to architect software and stuff like that. So I, I think, um, the question is: How do we put this in the hands of like a lawyer or a chemist or somebody else and have them also be able to, you know, interact with it as naturally as we can? Another interesting thing is: How can we re-redesign our existing systems for agents? So we had a paper on API-based web agents, and basically what we showed is if you take a web agent and the agent interacts w- not with a website but with APIs, the accuracy goes way up just because APIs are way easier to interact with. And in fact, like, um, when I ask the, what... Our agent, our agent is able to browse websites, but whenever I want it to interact with GitHub, I tell it, "Do not browse the GitHub website. Use the GitHub API," because it's way more successful at doing that. So maybe, you know, every website is gonna need to have an API because we're gonna be having agents, uh, interact with them. Um, about progress, I think progress will get faster. Um, it's already fast. A lot of people are already overwhelmed, but I, I think it will continue. Um, the reason why is agents are building agents, um, and better agents will build better agents faster. So, um, I expect that, um, you know, if you haven't interacted with, uh, a coding agent yet, it's pretty magical, like the stuff that it can do. Um, so, uh, yeah. And I have a call to action. Um, I'm honestly... Like, I've been working on, you know, natural language processing and language models for what, fifteen years now, and even for me, it's pretty impressive what like, uh, AI agents powered by strong language models can do. ### Call to Action **Graham Neubig** [29:30] Um, on the other hand, I believe that we should really make these powerful tools accessible. Um, and what I mean by this is I don't think like, you know, we, we should have these be opaque or limited to only a set of, certain set of people. Um, I feel like they should be affordable. Uh, they shouldn't be increasing the, you know, difference in the amount of power that people have. I- if anything, I'd really like them to kind of, uh, make it possible for people who weren't able to do things before to be able to do them well. Um, open source is one way to do that. Uh, that's why I'm working on open source. Uh, there are other ways to do that. You know, make things cheap, make things, uh, you know, so you can serve them to people who aren't able to afford them easily. Like Duolingo is one example where they get all the people in the US to pay them twenty dollars a month, uh, so that they can give all the people in South America free, you know, language education so they can learn English and become, you know, uh- Like, and become more attractive on the job market, for instance. And so I, I think y- we can all think of, uh, ways that we can do that sort of thing. Um, and if that resonates with you, please contribute. Of course, I'd be happy if you contribute to OpenHands and use it. Uh, but, um, another way you can do that is just use open source solutions, contribute to them, research with them, and train strong open source models. So I see, you know, some people in the room who are already training models. It'd be great if you could train models for, uh, coding agents and make them cheap and yeah. Uh, yeah, please. Uh, I, I was thinking about you, uh, among others, so. Cool. Yeah, that's all I have. Thanks. ### Q&A **Host** [31:16] Slight-slightly controversial take is probably the nicest way to say hot takes. Um, any hot takes questions? Actual hot takes? **Graham Neubig** [31:30] Oh, I can also show the other agents that we're working if anybody's interested. But yeah, sorry, go ahead. **Guest** [31:35] Yeah, I have a couple of questions. Um, so they're, they're kinda paired maybe. Uh, the first thing is that you said that you're estimating that your, um, your agent is successfully resolving, like, something like thirty to forty percent of your issues, but that's, like, below what you saw in SWE-Bench. So I guess I'm wondering where that discrepancy is coming from. And then I guess my other second question, which is maybe broader in scope, is that, like, if, if you think of an agent as like a junior developer, and I say, "Go do something," then I expect maybe tomorrow to get a Slack message being like, "Hey, I ran into this issue. How can I resolve it?" Um, and, and like you said, your agent is, like, successfully solving, like, ninety percent of issues where you give it direct feedback. So are you thinking about how to get the agent to reach out to, like, for, for planning when it's, when it's stuck or something like that? Or like identify when it runs into a hole like that? **Graham Neubig** [32:31] Yeah. So great-- uh, these are great questions. **Guest** [32:33] Oh, sorry, the third question, which is a c- since this is the first two- **Graham Neubig** [32:37] Okay **Guest** [32:37] ... and if so, are you going to add a benchmark for that second question? **Graham Neubig** [32:42] Okay, great. Yeah, great questions. Okay. So the first question was, um, why do I think it's resolving less than fifty percent of the issues on SWE-Bench? So first, um, SWE-Bench is on popular open source repos, and all of these popular open source repos were included in the training data for all of the language models. Um, and so the language models already know these repos. In some cases, the language models already know the individual issues in SWE-Bench. So basically, like, some of the training data has leaked. Um, and so it, it definitely will overestimate with respect to that. I don't think it's like horribly, horribly off. Um, but I think, you know, it's boosting the accuracy by a little bit. So maybe that's the biggest reason why. Um, in terms of asking for help and whether we're benchmarking asking for help, um, yes. Uh, we are. So, um, one, uh, one thing we're working on now, which we're hoping to put out soon, is we, uh, we basically made super vague SWE-Bench issues. Like I'm having a, I'm having a problem with the matrix multiply, please help. Because these are like, if anybody's run a popular open source, like, framework, these are what half your issues are. Your, like, users show up and say, like, "My screen doesn't work, what, what's wrong?" Or something. And so then you need to ask them questions and how to reproduce. So, um, yeah, we're, we're working on that. I, I think it-- my impression is that agents are not very good at asking for help, even Claude. Um, so like, when, when they ask for help-- they'll ask for help when they don't need it and then won't ask for help when they do need it. So this is definitely like an issue, I think. **Guest 2** [34:29] Yeah. Thanks for the great talk. I also have two questions. I guess first one, uh, can you talk a bit more about how the web agent interacts with, uh, websites? So is there a VLM that looks at the webpage layout and then you parse the HTML and select which buttons to click on? And if so, um, do you think there's a future where there's like... So I work at Bing, at, uh, Microsoft AI. Do you think there's a future where, um, there's like the same web index but there's an agent-friendly web index where all the processing is done offline so that you don't need to spend time, um, cleaning up like, cleaning up the HTML and figuring, figuring out what to click online? Um, and any thoughts on, thoughts on that? **Graham Neubig** [35:10] Yeah. So great question. Um, there's a lot of work on web agents. I didn't go into like all of the details. But I think there's three main ways that agents interact with websites. The first way is the s- the simplest way and the newest way but it doesn't work very well, which is you take a, um, a screenshot of the website and then you click on a particular pixel value, um, on the website and like, models are not very good at that at the moment. Like they'll misclick. There was this thing about how like Claude computer use started, uh, like looking at pictures of Yellowstone National Park, um, or something like this. I don't know if you heard about this, uh, anecdote. But, um, like people were like, "Oh, it's so human. It's looking for a vacation." And it was like, no, it probably just misclicked on the wrong pixels and accidentally clicked on an ad. Um, so like, uh, th-this is the simplest way. The second simplest way is, uh, you take the HTML and you basically identify elements in the HTML. Um, you don't use any vision whatsoever. Um, and then you say, "Okay, I wanna click on this element. I want to enter text in this element," or something like that. But HTML is too huge, so actually it usually gets condensed down into something called an accessibility tree, which was made for screen readers for visually impaired people. And, um, so that's another way. And then the third way is kind of a hybrid where you present the screenshot but you also present like a s- a textual summary of the output. And that's the one- Yeah, I think will probably work best. Um, what we're using is we're just using text at the moment, and that's just an implementation issue that we haven't implemented the, uh, visual stuff yet. But that's kind of, like, we're working on it now. Um, another thing that I should point out is we actually have two modalities for web browsing. Very recently, we implemented this, and the reason why is because if you want to interact with full websites, um, you will need to click on all of the elements or have the ability to click on all of the elements. But most of our work that we need websites for is just web browsing and, like, gathering information. So we have another modality where we convert all of it to Markdown because that's, like, way more concise and easier for the agent to deal with. Um, and then can we create an index specifically for agents? Maybe a Markdown index or something like that would be, you know, would make sense. Oh, how would I make a successor to SWE-Bench? So, um, I mean, a first thing is there's, like, Livecode Bench, which Livecode Bench is basically continuously updating to make sure it doesn't leak into language model training data. That's easy to do for SWE-Bench because it comes from real websites, and those real websites are getting new issues all the time, so you could just do it on the same, uh, benchmarks that they have there. Um, there's also, like, a pretty large number of things covering various coding tasks. So, like, for example, SWE-Bench is mainly fixing issues, but there's also, like, documentation. There's, um, generating tests that actually test the functionality that you want. Um, and there, uh, there was a paper by a student at CMU on generating tests and stuff like that. So I feel like, um, SWE-Bench is one piece of the puzzle, but you could also have, like, 10 different other tasks, and then you could have, like, a composite benchmark where you test all of these abilities, not just, uh, that particular one. Um, lot, lots, lots of other things too, but yeah. **Host** [38:39] Question from across. Use your mic. It would help. **Guest 3** [38:45] Yeah. Um, uh, great talk. Thank you. Uh, my question is about your, um, experience designing agent architectures. Uh, specifically, how much do you have to separate concerns in terms of, um, task-specific agents versus having one agent to do three or five things with a gigantic prompt with conditional paths and so on? **Graham Neubig** [39:09] Yeah. So that's a great question. So we have a basic coding and browsing agent. Um, and I won't say basic, like it's a good, you know, it's a good agent, but it does coding and browsing. Um, it has instructions about how to do coding and browsing. Um, that is enough for most things, um, especially given a strong language model that has a lot of background knowledge about how to solve different types of tasks and how to use different APIs and stuff like that. Um, we do have a mechanism for something called micro-agents, and micro-agents are basically something that gets added to the prompt when a trigger is triggered. Right now, it's very, very rudimentary. It's like, um, if you fi-- detect the word GitHub anywhere, you get instructions about how to interact with GitHub, like use the API and don't browse. Um, also, uh, another one that I just added is for npm, the, like, JavaScript package manager. And npm, when it runs and it hits a failure, it, um, like, hits, uh, in interactive terminals where it says, "Would you like to quit?" It enter yes. And if that does it, it, like, stalls our agent for the timeout until, like, two minutes. So, like, I added a new micro-agent. Whenever it started using npm, it would, uh, like, get instructions about how to not use the interactive terminal and stuff like that. So that's our current solution. Um, honestly, I like it a lot. It's simple, it's easy to maintain, it works really well and stuff like that. But I think there is a world where you would want something more complex than that. **Guest 3** [40:40] Got it. Thank you. **Guest 4** [40:43] Uh, I got a question about MCP. Um, I feel like, uh, this is the Anthropic Model Context Protocol. Um, it seems like the most successful type of this, like, standardization of interactions between computers and agents. Um, are you guys adopting it? Um, is there any other competing standard? A-anything, anything thought about it? **Graham Neubig** [41:06] Yeah, I think the anth-- so the Anthropic, um, MCP is, like, a way... It, it's essentially a collection of APIs that you can use to interact with different things on the internet. I, I think it's not a bad idea, but it, it's like there's a few things that bug me a little bit about it. It's like we already have an API for GitHub, so why do we need an MCP for GitHub, right? You know, like, GitHub has an API. The GitHub API is evolving. Um, we can look up the GitHub API documentation. So, uh, it seems, like, kind of duplicated a little bit. And also they have a, a setting where it's like, uh, you have to spin up a server to serve your GitHub stuff, and you have to spin up a server to serve your, like, you know, other stuff. And so I think it makes, it makes sense if you really care about, like, separation of concerns and security and, like, um, other things like this. But right now we haven't seen, we haven't seen that to have a lot more value than interacting directly with the tools that are already provided. And that kinda goes into my general philosophy, which is we're already developing things for programmers. You know, w- how is an agent different than from a programmer? Um, and it is different obviously, you know. Like, uh, agents are different from programmers, but they're not that different at this point, so we can kind of interact with the interfaces we create for pr- for programmers. Yeah. Uh, I might change my mind later though, so we'll see. **Guest 5** [42:46] Yeah. Hi. Thanks. Um, very interesting talk. Uh, you were saying that the agents you have right now solve, like, maybe 30% of your, your issues out of the gate. Um, I'm curious. Of the things that it doesn't do, is there, like, a, a pattern that you observe of like, oh, like, these are the sorts of things that it just seems to really struggle with, or is it just seemingly random? **Graham Neubig** [43:10] Um, it, it's definitely not random. It's like if you think it's more complex, then it's like just intuitively it's more likely to fail. Um, I've gotten a bit better at prompting also. So, like, just to give an example, um, it, it will sometimes fail to fix a GitHub workflow because it will not look at the GitHub workflow and understand what the GitHub workflow is doing before it solves a problem. So I, I think actually probably the biggest thing that it fails at is, um, or that our, our agent plus Claude fails at is insufficient information gathering before trying to solve the task. And so if you provide all-- if you provide instructions that it should do information gathering beforehand, it tends to do well. If you don't provide sufficient instructions, it will try to solve the task without, like, fully understanding the task first and then fail, and then you need to go back and give, uh, you know, additional feedback. Um, another example, like I, I love this example. While I was developing the, um, the monitor, uh, website that I, I showed here, we hit a really tricky bug where it was writing out a cache file to a different directory than it was reading the cache file from, and I had no idea, I had no idea what it was going on. I, I thought the bug was in a different part of the code. But what I asked it to do was, um, come up with five possible reasons why this could be failing in decreasing order of likelihood and examine all of them, and that worked. And it could just go in and, like, do that. So, like, um, I, I think s- a certain level of, like, scaffolding about, like, how it should sufficiently gather all the information that's necessary in order to solve a task is-- like, if that's missing, then that's probably the biggest failure point at the moment. **Guest 5** [45:03] Thanks. Yeah. **Graham Neubig** [45:06] Yeah. **Guest 4** [45:08] I'm just, I'm just using this as a chance to ask you all my questions. Uh, you had a, you had a slide on here about, like, self-improving agents or something- **Graham Neubig** [45:15] Yeah **Guest 4** [45:15] ... like that with memory. Um, it's like a really throwaway slide for, like, a super powerful idea. Uh, it got me thinking about how I would do it. I have no idea how. Um, so I just wanted you to chain a thought more on this. **Graham Neubig** [45:30] Yeah. Self, self-improving. So I think the biggest reason, uh, like the, the e- simplest possible way to create a self-improving agent is to have a really, really strong language model that with infinite context, and it can just go back and look at, like, all of its past experiences and, you know, learn from them. Um, you might also want to remove the bad stuff just so it doesn't over-index on its, like, failed past experiences. Um, but the problem is a really powerful language model is large. Infinite context is expensive. We don't have a good way to index into it because, like, RAG, at least in my experience, RAG from language to code doesn't work super well. So I think in the end it's like that's the way I would like to solve this problem. I'd like to have an infinite context and somehow be able to index into it appropriately. Um, and I think that would mostly solve it. Um, another thing you can do is fine-tuning. So I, I think, like, RAG is one way to get information into your model. Fine-tuning is another way to get information into your model. So, um, that might be another way of continuously improving. Like, you identify when you did a good job and then just add all of the good examples into your model. Yeah. **Guest 4** [46:42] Yeah. So you know how, like, Voyager, um, tries to write code into a li-- skill library and then reuses the skill library, right? **Graham Neubig** [46:48] Yep. **Guest 4** [46:48] So the-- it improves in the sense that it just builds up the skill library over time. **Graham Neubig** [46:52] Yep. **Guest 4** [46:52] Um, one thing I was, like, thinking about, uh, and then, and there's this idea of, from, from Devin, your, your arch nemesis, uh, uh, of playbooks. **Graham Neubig** [47:02] Uh-huh. **Guest 4** [47:02] I don't know if you've seen them. **Graham Neubig** [47:03] Yeah. Uh, I mean, we're calling them workflows- **Guest 4** [47:05] Workflows **Graham Neubig** [47:05] ... but they're similar. **Guest 4** [47:06] Yeah. So, like, basically, like, you should-- like, once a workflow works, you hit kind of like persistent as a skill library. **Graham Neubig** [47:12] Yep. **Guest 4** [47:12] Right? Like, um, I, I, I feel like that there's a-- that's like some in between. Like you said, you know, it's hard to do RAG between language and code, but I feel like that is RAG for a-- like, I've done this before. Last time I did it, this, this worked, so I'm just gonna shortcut all the, the stuff that, uh, that failed before. **Graham Neubig** [47:30] Yeah, I, I totally-- I think it's possible. It's just, you know, not, not trivial at this point. **Guest 4** [47:35] Okay. **Graham Neubig** [47:35] Yeah. Uh, I'll explain the two curves. So basically the base, the baseline is just an agent that does it from scratch every time. And, uh, this curve up here is agent workflow memory where, um, it's like adding the successful experiences back into the prompt. Um, why is this improving? Um, the reason why is because just, um, it failed on the first few examples, and for the average to catch up, uh, it, it took a little bit of time. So it's not like this is actually improving it. You could just basically view the, um, this one as constant, and then this one is like improving like this. Basically, you can see it's, it con- it's continuing to go up. Yeah. **Guest 6** [48:16] Um, how do you think we're gonna solve the authentication problem for agents right now? **Graham Neubig** [48:20] Um, when you say authentication, you mean like credentials? Like... Yeah. **Guest 6** [48:25] Yeah. 'Cause I, I've seen a few, like, startup solutions today, but it seems like it's limited to the amount of like, like, websites or actual, like, authentication methods that it's- **Graham Neubig** [48:33] Yeah **Guest 6** [48:33] ... it's capable of performing today. **Graham Neubig** [48:35] Yeah, great questions. Um, so my preferred solution to this at the moment is GitHub, like, fine-grained authentication tokens. And GitHub fine-grained authentication tokens allow you to specify like very- On a very granular basis, um, on this repo, you have permission to do this. On this repo, you have permission to do this. Um, you also can prevent people from pushing to the main branch unless they get approved. Um, you can do all of these other things. And I think these were all developed for human developers. Or, like, the branch protection rules were developed for human developers. The fine-grained authentication tokens were developed for GitHub apps. Um, I, I think for GitHub, maybe just pushing this, like, a little bit more is the way to do this. For other things, they're totally not prepared to give that sort of fine-grained control. Like, most APIs don't have something like a fine-grained authentication token, and that goes into my, like, comment that we're gonna need to prepare the world for agents, I think. Um, but I think, like, the GitHub authentication tokens are, like, a good template for how you could start doing that maybe. But yeah, I don't, I don't, I don't have an answer. Yeah. **Guest 6** [49:46] I'll let you know if I find one. **Graham Neubig** [49:47] Okay. Yeah. Thank you. Cool. Um, I'm, I'm gonna finish up. Let, let me just see. Okay, so this one, um, this one did write a script. I'm not gonna actually read it for you. And then the other one... Let's see. Yeah, so it sent a PR... Sorry, what is, what is the PR URL? So I don't, I don't know if this, uh... Sorry, that's taking way longer than it should. Okay. Cool. Yeah, so this one sent a PR. I'll, I'll tell you later if this, uh, actually, like, uh, successfully... Oh, no, it's deployed on Vercel, so I can actually show you. But let's... Le- let me try this real quick. Sorry, I, I know I, I don't have time. Uh. Yeah, there you go. I have pie charts now, so yeah. It, it's so fun. It's so fun to play with these things 'cause you could just do that while I'm giving a, you know, talk. Things like that. So yeah, thanks. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.