# The #1 SWE-Bench Verified Agent Latent Space · 2025-04-02 Guy Gur-Ari, co-founder of Augment Code, explains how their coding agent achieved the #1 spot on SWE-Bench Verified without fine-tuning, using off-the-shelf models with techniques like sequential thinking, ensembling, and code base understanding. He shares that orientation and memories help the agent learn codebase conventions, and that MCP support allows extensibility. Gur-Ari contrasts Augment's IDE-first approach with competitors like Devin and Factory, emphasizing meeting developers where they are. He also discusses the future of IDEs, the role of custom models, and recommends the SWE-RL paper for reinforcement learning in coding. The episode features a live demo of the agent implementing a feature from a Linear ticket and creating a PR. ## Questions this episode answers ### How did Augment Code achieve the top score on SWE-Bench Verified without fine-tuning any models? Guy Gur-Ari explains they used off-the-shelf generation models and combined several techniques: a sequential thinking MCP tool that lets the model reflect, ensembles of multiple agent runs (using O1) at the end to boost reliability, and careful iteration on file-editing. These together, without custom fine-tuning, pushed them to #1. Augment has open-sourced their SWE-Bench implementation. [1:24](https://addtry.com/22ef06b2-71f9-4f44-bd41-8211f9f3c8ce?t=84000) ### What is sequential thinking MCP and how does it improve coding agent performance? Sequential thinking MCP is a tool that allows the model to reflect and plan step-by-step. Guy Gur-Ari says they found it gave a noticeable score bump on SWE-Bench, even outperforming Anthropic’s reasoning mode on Claude 3.5 Sonnet for this benchmark. He notes they don’t fully understand why it works better for code, but empirically it helped their agent achieve top results. [2:36](https://addtry.com/22ef06b2-71f9-4f44-bd41-8211f9f3c8ce?t=156000) ### How does Augment Code's agent compare to other AI coding agents like Devin or Factory? Guy Gur-Ari contrasts Augment’s philosophy: it targets developers in large, complex codebases and meets them in their existing IDEs (VS Code, JetBrains, Vim) rather than pushing a new workflow. While others may focus on zero-to-one development, Augment invests in codebase understanding and plans to let users run multiple agents concurrently to unlock efficiency, all within the familiar IDE environment. [12:27](https://addtry.com/22ef06b2-71f9-4f44-bd41-8211f9f3c8ce?t=747000) ## Key moments - **[0:00] Intro** - **[0:35] Agent Launch** - [0:36] Augment Code launches agent feature that understands codebase conventions and executes commands to produce working PRs. - [1:13] Augment Code's agent achieves #1 on SWE-Bench Verified leaderboard using off-the-shelf models. - [2:16] SWE-Bench samples often pinpoint where changes must be made, so codebase understanding adds only a small delta to scores. - **[2:36] Under the Hood** - [3:01] Sequential thinking via MCP improved agent scores more than model reasoning mode on SWE-Bench. - [3:55] Claude's reasoning mode did not yield significant improvement over sequential thinking for coding tasks. - [4:14] Building a real coding agent system demands supporting multiple clouds and models from the start to manage cost and performance. - [5:42] Guy Gur-Ari recommends starting agent development with a curated set of 10 samples before scaling to automated evals on hundreds of samples. - [7:59] String replace editing was more reliable than smart paste for agent file edits. - [8:48] As model costs decline, Guy predicts UX will become the main barrier to ensembling agents, not expense. - **[12:12] Landscape** - [12:36] Augment Code vs Devin and Factory: Augment focuses on IDE-native agents for large codebases, not a separate development environment. - [14:17] As agent usage explodes, Guy expects cost to drive Augment Code to train custom models for some features. - [15:26] Users combine Augment Code's extensions inside Cursor for features like next edit and deep codebase understanding. - **[15:46] Demo** - [22:57] Guy Gur-Ari live-demos Augment's agent implementing a new dialog tool for itself and creating a GitHub pull request end-to-end. - **[23:18] Beyond IDE** - [23:34] Guy predicts within 2-3 years, developers will spend 80% of their time outside the IDE, using web apps to manage agents. - **[24:45] MCP** - [25:09] MCP is valuable for extensibility, but Augment prioritizes seamless authentication for integrations like GitHub and Linear. - **[27:01] Research** - [27:06] Guy Gur-Ari recommends the SWE-RL paper for applying reinforcement learning to code generation. - **[28:02] Google** - [28:12] Ex-Googler Guy Gur-Ari says Google's internal culture shifted to be more startup-like and aggressive after ChatGPT launched. - [30:20] Despite models producing whole PRs, Guy says tasks still often require human breakdown, and coding agents are far from AGI. - **[30:46] Outro** - [31:07] Augment Code open-sourced their SWE-Bench approach, allowing others to see how they reached #1. ## Speakers - **Swyx** (host) - **Alessio** (guest) - **Guy Gur-Ari** (guest) ## Topics Coding Agents ## Mentioned Anthropic (company), Augment Code (company), Factory AI (company), Google (company), JetBrains (company), Magic.dev (company), Poolside (company), Sourcegraph (company), Weights & Biases (company), Cursor (product), Devin (product), Gemini (product), GitHub (product), Linear (product), Notion (product), SWE-Bench (product), Sonnet (product), VS Code (product) ## Transcript ### Intro **Alessio** [0:03] Hey everyone, welcome back to the Lit In Space podcast. This is Alessio, partner and CTO at Decibel Partners, and I'm joined by my co-host Swyx, founder of Small AI. **Swyx** [0:12] Hello, hello, and today we're so happy to have Guy Gur-Ari on the pod with, uh, from Augment Code. Welcome. **Guy Gur-Ari** [0:19] Thank you. Great to be here. **Swyx** [0:21] Uh, Guy, uh, yeah, I was actually very excited to feature, uh, an Augment talk in our New York conference, and now it looks like you're coming up with some even bigger stuff with your new agent. So why don't we just get right into it? ### Agent Launch **Swyx** [0:35] What are we announcing today? **Guy Gur-Ari** [0:36] Yeah, sounds good. So, uh, yeah, we're announcing our agent feature. This is a new feature that kind of takes coding with Augment to the next level. Previously, we had completions right around the cursor. We had next edit to, to change code away from the cursor. We had chat with code base understanding, and today we're launching agent with code base understanding, and so the agent can understand the request, understand where to make changes and how in the code base in a way that tries to be respectful of the code base's conventions and design, and then it can go and execute commands, run tests, all the way to getting to a working PR. **Alessio** [1:13] And, well, maybe you don't wanna say it to be humble, but, uh, this is gonna be the number one SWE-bench verified agent, dethroning the Weights & Biases one that we previously featured. **Guy Gur-Ari** [1:24] Yeah, we just, we just made number one on SWE-bench, so for us, uh, SWE-bench has been a useful tool for exploring how can we get the most out of agents. And so we were able to, to get the best, uh, uh, result on SWE-bench Verified right now. Yeah. **Alessio** [1:39] And all on off-the-shelf models, right? **Guy Gur-Ari** [1:43] So yes. So the generation models for, for SWE-bench, it's all off-the-shelf models. For the product, it includes our own custom-trained models that help the model with-- that help the agent with code base understanding. **Alessio** [1:55] What's the delta between, like, have you run also the private models on SWE-bench? **Guy Gur-Ari** [2:00] So we have seen a s- I would say a small delta when you include the code base understanding, and that's something that we're digging into. What we found with SWE-bench is an interesting benchmark, very useful for doing some things like refining prompts, for example, and trying out different tools. Specifically for code base understanding, in the beginning, we were hopeful that it will tell us more about how we're doing there. But looking at the actual samples in SWE-bench, they usually very often kind of pinpoint where the changes have to be made, and so we have not seen that much benefit from, from code base understanding on this benchmark. **Swyx** [2:36] To tie in to, uh, another previous guest that we had, we also had Eric from Anthropic, who built the, uh, Anthropic version of the, their sort of agent, and it looks like you also had some kind of sequential thinking MCP usage. ### Under the Hood **Swyx** [2:49] Uh, you wanna talk a little bit about that? Like, what, what I'm trying to do is also, like, promote, um, w- like the different elements of the agents that were useful that people may, may often miss when they build agents. **Guy Gur-Ari** [3:01] Yeah, definitely sequential, sequential thinking helped bump the score. There were, there were a few things that helped. So sequential thinking lets the model kind of, kind of reflect and, and do better. Uh, another thing that helped was at the end, run the agent a few times and ensemble those scores. Uh, I think we ended up using a one for that. Um, and so if you can afford that extra cost, then ensembling also helps. And then there were other things like just getting... When you start out building an agent, getting file edits to be reliable is also by itself a, a nontrivial task and one that we iterated on as well. Yeah. **Alessio** [3:37] And when you say the sequential thinking part is interesting because it's not reasoning mode, do you have, like, do you have a good way to, like, just summarize why the sequential thinking MCP performed better than just, like, reasoning mode on Sonnet-Three Point Seven? Or is it just the results were better, that's the TLDR? **Guy Gur-Ari** [3:55] Yeah. I don't think I have a good insight for that. We tried reasoning mode as well with the new Three Seven, and we didn't see that much of a bump in performance. We don't know if this is something that's code specific or not. I don't have an insight. We tried both and yeah, sequential thinking worked better. **Swyx** [4:14] I think the experimentation approach is probably the right one. It's just interesting that you have to ensemble these models. You can't just be sort of single source in your models. It would be, it would be neat because then you can sort of deploy your agent across all environments, whereas here there's some clouds that are friendly to OpenAI, some clouds are more friendly to Anthropic. You have to kinda be cross-cloud. I feel like there's a, there's an old argument in developer toolings where you have to be hybrid cloud, uh, in order to, to get maximize the benefits, and I think there's a little bit of hybrid model argument going on in the state-of-the-art. **Guy Gur-Ari** [4:47] Yeah, I think, I think we're, we're, we're at that point of when you build a real system, you have to support multiple clouds and you have to support multiple models. So our system from the beginning had multiple models because even for completions, you had a generation model for doing the generation, but for retrieval, we had separate retrieval models, and over time, we actually had multiple different retrieval models working. So we're, we've had that built into the system for a while. I think for agents, one thing that becomes, mm, uh, that, that becomes an issue pretty quickly is cost. These models are very capable, but they're also very expensive, and so that by itself is a reason to start mixing and matching different models in the system. **Swyx** [5:29] Yeah. Amazing. Just kinda curious because, uh, this is something I've been thinking a, a lot about. Is there an experimentation framework that you like or process that you like? Is it just notebooks everywhere? Have you built your own tooling? Have you seen other tooling elsewhere that you have brought in? **Guy Gur-Ari** [5:42] Uh, it, this is a question about how to evaluate models? **Swyx** [5:45] Experimentation framework. It's almost like it's, it's not just evaluations. I mean, evaluations, you know, you mentioned your own int-internal evals, but also, um, do you, do you just run a battery of tests every single time? Do you have a, a well-defined process? Do you have tooling that you, that you built that might be interesting to dig into, right? Like, I think as, as a founder of any, any AI company, basically every founder needs this. **Guy Gur-Ari** [6:09] Yeah, absolutely. So I would say the process that I like to follow is in the beginning when developing a feature, come up with a curated set of samples. Could be as small as ten, 10 samples that you run through, and then yeah, it's all notebooks basically. You run through the samples. The data set is small enough that you can keep it in your head. You become kind of very familiar with those samples, and you see how well iterations improve or regress on them. Beyond a certain point, that's not good enough anymore, and then you go to hundreds or sometimes maybe thousands or more of samples. So we have infrastructure for doing these kinds of evaluations, both with and without code execution. So code execution, like checking the correctness of solutions and running tests automatically can help. Although not-- Although you can get a lot of mileage out of evals that don't have code execution in them, that just compare against ground truth. The other thing that we found to be very useful is there is often a gap between what you do in research and what you have in production. Those systems are not always the same. And so we've worked to, to make sure that at least some of our evaluations that we use in research, we can also run the production system against them. And so whenever we deploy a new model, if it's one of our own models, we can-- We'll automatically run those evals, and then we will see if there were any regressions. That, that's what's that's meant to catch. Yeah. The last thing I can mention is we use contractors for evaluation where we cannot do automatic evaluation. So with chats and agents, it becomes way harder to do things automatically, and so we use contractors for that. **Alessio** [7:44] And then just to wrap up on the tool side, we talked about sequential thinking. I saw in your press release also, you mentioned how string replace is, like, better than smart paste. Are there any other interesting small technical wins that you had that you want to highlight? **Guy Gur-Ari** [7:59] Let me see. There was... I think that c- that plus the, plus the ensembling, I think, I think that those were the major ones. **Alessio** [8:07] Yeah. And yeah, let, let's also clarify on the ensembling. You mentioned you just did just kind of basic, uh, majority voting thing because the cost was kinda gonna, gonna go high, too high. Are there any promising directions that you saw were kind of like, you know, maybe good result-wise, but the cost was, like, too expensive that you didn't pursue, that you would want other people to, to spend more time on? **Guy Gur-Ari** [8:30] You mean besides, besides the, besides the ensembling? **Alessio** [8:33] Yeah, like even the... You know, when you mention in the... I'm reading from the blog post. **Guy Gur-Ari** [8:38] Yeah, yeah. **Alessio** [8:38] You decided not to investigate this direction because of significant extra cost. Like if it costs 10 times more, how much more of a result will we get? You know- **Guy Gur-Ari** [8:48] I see **Alessio** [8:48] ... is it, is it like if the, if the models start going down in cost, are we just gonna do more of that? **Guy Gur-Ari** [8:52] Yes, I expect we will. I think it will become less of... Over time, as cost will come down, I expect it will become less of a cost question and more of a UX question. Because with ensembling, well, what we found is that users really want to see what the agent is doing and follow along. And if you're doing ensembling, you don't have one trajectory anymore, you have several. And then the question of how the users supervise that as it's happening becomes pretty tricky. Like, you can present the results after it's done, but typically users don't, don't like waiting that long to see what's happening. They want to make sure the agent is on the right track. So I think even if the cost ends up making sense, and it might, the UX will become a problem. In terms of how much more you can squeeze out of this, yeah, I would guess you can get a few more percentage points, at least with better ensembling, 'cause what we did is, is very simple, and I know others have been experimenting with this as well. I don't think it's... It's-- Ensembling is usually not the kind of thing that just solves a benchmark, just also in other domains. You typically can get like, yeah, and this, and for SWE-Bench, probably a few percentage points is my guess. **Alessio** [9:57] Yeah. And, and then there is something that you tried doing a more multi-agent approach with like the orientation agent and the regression fixing agent. Can you maybe talk people through that and what results you had? **Guy Gur-Ari** [10:09] Yeah. So we found for agents that orientation in particular is extremely important for getting good value out of the agents. And so we-- I believe we have this o-orientation phase where we try to f- to look around the code base, and this is something we're also building into the product, look around the code base and try to understand the conventions. For SWE-Bench, this is mostly around testing and how you execute the test. That's the main thing. But for the product, there's, there is more than that. There is try to figure out testing, try to figure out what frameworks are being used, if we can fi-figure out the versions that are being used. All those things help give users a much better out-of-the-box experience when they use agents. **Alessio** [10:50] Yeah. Is that the onboarding process with Augment? Like when people sign up for your product, there's kind of like a agent that gets in there and starts to learn everything and codifies it. **Guy Gur-Ari** [11:01] So there are a few things. The onboarding, uh, that we have is meant to be kind of a quick, uh, process, and that, um, that does a little bit of orientation. The next thing after that, we've built in memories. So with memories, the agent tries to learn, uh, from its mistakes as it makes them. And so then at least if it made a mistake, it's not gonna make that mistake again, if it's something that's generally true about the code base. Things like these are the conventions of how we store tasks, or these are, this is how we run them. So the agent learns along, um, uh, with you as it's, as it's working. And we also have, uh, a, a feature that I think has not shipped, uh, yet, but we are planning to ship, which is a more thorough orientation. So this is something that will run for several minutes probably, um, and try to do a pretty thorough job of trying to figure out, um, you know, what languages do you have, what kind of how to run tests, how to do frameworks and so on. That's, that's, uh, coming. **Alessio** [12:01] Okay. Anything else on multi-agent? I, I was gonna move on to more sort of broader topics other than just the, the agent feature. **Guy Gur-Ari** [12:07] No, I think that's good. Yeah. **Alessio** [12:09] One, one thing... Okay, one way I would maybe ### Landscape **Swyx** [12:12] Put the hot potato question to you is how-- I think there are a lot of people trying to solve agents, uh, coding agents for the enterprise. Uh, let, let's just call it that. There would be Devin, there would be Factory AI, there would be Sourcegraph. How do you compare your philosophy in... Draw the landscape for us. Like, like where, where are you in the top right corner of, and what would be the X and the Y-axis? **Guy Gur-Ari** [12:36] Right. Good. Yeah. So for us, I, I can try on the fly to convert it to a graph, but let's see. So we, you know, we-- our product is meant for developers who are working in organizations that have large complex code bases. This is really what we're optimizing for. It's less about zero to one development, and it's more about working within existing large code bases. That's why we invested a lot in the context engine from the beginning, and if you use our product, you can kind of get the sense that it's more meant for, for that kind of work. Compared to, I would say, uh, Devin and Factory, our philosophy is that we, we wanna meet developers where they are. And so our product is a set of extensions. You have VS Code, we are launching JetBrains agent as well, so we have support for JetBrains. We have a Vim plugin, and so we, we really try to, um, not ask developers the way-- ask developers to change their workflow, and this is why we've built the first agent into the ID. Having said that, it's clear to us that agent usage is going to explode. The f- kind of the thing that you notice pretty quickly as you get the hang of using agents, as you get past the initial learning curve, the magic kind of wears off, and it becomes ex- it becomes obviously really slow. And not only is it slow, it also takes over your whole workspace. And so that's where we think that being able to run multiple agents and not just one is going to be the way to unlock, honestly, most of the value out of these agents, and that's what we're working toward. But it's all with a sense of we still wanna build this for developers, and we still wanna meet developers where they are and not try to change how they do things. Yeah. **Swyx** [14:17] Yeah. I think the other comparisons also would be, you said Magic, like there's Magic.dev and Poolside. Poolside also trying to go to market with, with extensions. But I, I guess those are training proprietary models, right? They're, they're investing a lot of money in that, and you've just decided to do no custom models here. **Guy Gur-Ari** [14:36] Yes. I would say for now. So for us, there's always the question of for every feature, do we use an off-the-shelf model or an external model, or do we train our own? I think for now, for agents, uh, except for the code-based understanding, we, we, we outsource the models. However, we believe that the cost of these... Or let's say, we believe that the usage of agents is going to explode, and as a result, the cost is going to explode as well. And so we feel that there is room here for training custom mo- custom models to help with that. But yes, for now, building a product and going to market quickly, this is what we've prioritized, and there's clearly a lot of, lot of demand for these things out there, so pr- we're pretty happy with that decision. Yeah. **Swyx** [15:17] And then lastly, on, in terms of, uh, we're going, uh, I think, I think we're gonna go for a demo soon. In terms of developers where they are, can I use Augment inside of Cursor? Would it make sense? **Guy Gur-Ari** [15:26] Yes, you can use Augment inside of Cursor. We actually have quite a few users doing that. **Swyx** [15:31] Yeah. **Guy Gur-Ari** [15:32] And what we found is that they like things like next edit, they like the code-based understanding. And so yes, you can absolutely do that. **Swyx** [15:38] Yeah, we would love to do a quick demo product run through just so people can see it too on YouTube. **Guy Gur-Ari** [15:44] Sounds good. Let me set it up here. **Swyx** [15:46] We can always cut the silence as well. ### Demo **Guy Gur-Ari** [15:47] The setup, right? **Swyx** [15:48] Yeah. **Guy Gur-Ari** [15:49] Yeah. **Swyx** [15:49] Yeah. **Guy Gur-Ari** [15:50] That's what I figured. **Swyx** [15:50] It's all, it's all edited, which is nice. I see it. Um, yeah, it's got a good zoom as well. You can make it full screen and also hide the, the menu bar if that helps. That works. **Guy Gur-Ari** [16:03] Works well. Let's see. Sorry, this is, this is from the trial run. I'm assuming we can cut all of this out. So this is from the trial run. The hardest part about this is that... Or the riskiest part is not the agent, but getting it to build because VS Code extensions. This is gonna be like in trying to add a feature to the agent using the agent. Yes. And so I thought that that, that would be fun. And the problem is VS Code extensions, I don't know how much you've played with them, but they are, they are tricky to build. **Swyx** [16:41] You have to reload them. Yes, I, I maintain one for one of my previous, uh- **Guy Gur-Ari** [16:45] Okay **Swyx** [16:45] ... companies. **Guy Gur-Ari** [16:46] So let me, let me destroy this and this, and let me just start this again. **Swyx** [16:50] We use Bazel. Nice. **Guy Gur-Ari** [16:51] We use Bazel, yes. Yes. Quite a few ex-Googlers at the company. **Swyx** [16:57] Right. **Guy Gur-Ari** [16:58] And so this is somewhat familiar. **Swyx** [17:01] There's a startup called Bazel.Builds, which I, I heard some buzz about when they left Google, but I'm not sure where they are. Like some Bazel-based- **Guy Gur-Ari** [17:07] Oh, actually. Interesting. **Swyx** [17:10] And there's like BuildBuddy too. **Guy Gur-Ari** [17:11] Yeah. **Swyx** [17:12] Anyway. **Guy Gur-Ari** [17:14] Okay. Looks like it's building. I think this looks promising somewhat. All right. I think worst case, yeah, this looks good. All right. Wor-worst case we will, I think we will just show the code and not show it running, which is not as good, but at least the code will, the code will show up. Um, all right. I think that's it. Let me just switch here and switch to this. All right. Shall we go? **Swyx** [17:41] Whenever you're ready. **Guy Gur-Ari** [17:42] Cool. Uh, yeah. So I'm gonna show a quick demo of the agent, uh, going from a ticket to working code. And so we're actually gonna use the agent to add a feature to itself inside the ID. And the idea is, here we're gonna do something pretty simple, which is let's give the agent a new tool that it can call to pop up a dialogue box for the user, asking it, "Okay, cancel." And then once it has that tool, it can decide to autonomously use it, uh, whenever it wants. Uh, so we use Linear for managing tickets. So I'm gonna copy the, the ticket ID, and within Augment we have, uh, we have both MCP support for complete extensibility of tools, but we, we've also built in a, a few first-party integrations and serve them as tools to the agent. So we have GitHub, Linear, Notion that, that we use, uh, internally, and then we have several others. And so here I should be able to tell it, "Implement ticket with the ticket ID." I'm going to tell it to just implement, but don't run it, because we have our own build system and I know it's going to be eager about trying to go and, like, build the code and so on, and I don't want it to do that. All right. So the idea here is that we see the agent going on the right, and it's basically hacking on its own code. This is the code of the extension that's currently running inside VS Code, and we're gonna get a new extension version out of this. So the first thing it did was it went to Linear to grab information about the ticket so it understands what that is, and then you can see it immediately reaches for the Augment context engine to get oriented and understand how to go about performing its task. And so here it asks about, "Okay, let's figure out how tools are implemented in the VS Code extension," and it's gonna keep, uh, uh... it's gonna do probably a few of these retrieval calls and then come up with a plan and then start implementing it. So this should take, I think, a couple minutes. While it's doing that, I wanna highlight a feature that I mentioned before, which is memories. So memories is where the agent learns, uh, and adapts to the user as the user is using it, and you can click on memories and you can see, uh, a little Augment, a logo that's getting animated. But below that you can see all the memories that the agent has collected, and you can edit them and add to them if you like. All right, so it looks like the agent came up with a plan, uh, which is to create a new class and then define the input schema and so on and so forth, and we can see that, that it's starting to actually go ahead and edit some files. We can follow along with what it's doing either in the drawer at the bottom or we can actually also open the version control pane and see what it's doing. But so far it made just a small change registering this new tool in the local tools that the agent has access to, and now it's going to implement probably the bulk of the work, which is, I guess, it decided to call it the Show Dialog tool. Let me make this a bit bigger. So you can see that it, it managed to understand within our whole code base what, uh, uh, what class it should inherit from, how to implement this, uh, tool interface. So it knows about language model, uh, tools, but it needs to adapt it to our own interface. We have a safety, uh, feature for tools, so it needed to implement that, calling the tool, and then integrating that also with the VS Code API to actually show the, show... All right. Let's see. I think it's should be close to being done. Okay, it's taking some liberty here of also adding it to chat and not just the agent. Yes. **Alessio** [21:39] It just wants to work. **Guy Gur-Ari** [21:40] It just wants- **Alessio** [21:41] It wants to be productive. **Guy Gur-Ari** [21:42] Exactly. All right. I think... Let's see. Oh, there's an issue with the imports. I think it's gonna be fine. **Alessio** [21:53] Nice. **Guy Gur-Ari** [21:53] All right. I think I can stop it here 'cause it's probably gonna be fine, and I'm gonna try to actually run this now and see, see if we can see it in action. All right. So this built, built the extension, the new version of the extension, uh, that's supposed to have the tool. If I open this and close this. There we go. I'm gonna switch to agent mode here, and then I'm gonna ask it to call it, and let's see if this works. All right. And here it is. Uh, this is sample dialog message, click Okay or Cancel, and if I click Okay it's gonna tell me, uh, yeah, "I understand I should use the repository information," so on and so forth. Uh, yeah. So that's it. Oh, uh, actually, uh, we can take another step and actually tell it to, okay, great, create a PR from this because we also have the GitHub integration and then it's gonna go create a branch, uh, check in everything, and then push a PR to, to GitHub because it's connected both to Linear and GitHub, and this is what allows for, like, an end-to-end workflow. So for GitHub we're being a bit more careful with the commands. You have to click, uh, uh, uh, click play to actually run these because this is an external system, uh, and it's not so easy to revert if the agent messes up. Uh, but then it goes and, like, starts creating branches and committing them, uh, and all the normal stuff for getting a PR. Yeah. **Swyx** [23:18] Yeah, that's really impressive. ### Beyond IDE **Alessio** [23:21] What, what's the, what's the plan for, like, moving it out of the ID? Do you also see a world in which just in Linear you can say, "Please implement," and then you don't have to actually open VS Code and be on top of it? **Guy Gur-Ari** [23:34] Yeah. So that's, that's a great question that we, we ask ourselves almost on a weekly basis. These things are getting powerful quickly, um, and there are... Like, if you're doing zero to one development, it seems it's already at a point where you don't need a lot of the ID or maybe not even an ID at all to get to something that works and iterating on it. What we found so far is that when you're working in complex code bases, it's still quite often that you need to use the ID to make changes. And so for now our product is going to be in the ID because of this reason. And also when we add support for multiple agents, we expect that it's still gonna be in there, and this is also based on, on feedback from early users. I think in the future at some point my guess is that the ID is going to become Less of the focal point and more like an app that you can launch when you need to dig in deeper, but you spend most of your time away from it. So maybe eighty percent of your time is in a website or an app where you control the agents, and then twenty percent of your time, maybe you go into the IDE. But I feel like for, for the developers that we're targeting, we're not quite there yet. ### MCP **Swyx** [24:45] I would say I would agree. I think you would scare a lot of people by saying, you know, uh, the IDE goes away and becomes, you know, a smaller part of your workflow. People are very religious about their IDEs. As the resident MCP guy, I know you got-- you have, uh, MCP support. I also wonder your, your personal takes on MCP. Uh, you know, what, is it, is it helpful? Is, is it, is it just a fad? What do you think? **Guy Gur-Ari** [25:09] Yeah, I... It, it's certainly helpful if you're willing to go and learn about it and get past the friction of setting up tools. But it gives users so much control over, uh, things that they can do, right? You can-- If we don't have that integration built into the product, you can connect it, uh, you can connect our agent to any of your systems, and they could be, they could be external systems that we just haven't gotten around to implementing yet, or they could be your own internal systems. We try to cater both to, to, to, let's call them more, more power users in terms of these integrations, but also make it really easy for developers who just want, like, to go through three clicks and connect their agent to GitHub or Linear or any of the other popular tools. So yeah, I think MCP absolutely has, um, a place in this ecosystem, clearly, also based on the adoption, and we'll, we will definitely continue to, to support it. **Alessio** [26:03] Are you gonna remove your custom hand integrations and replace them with MCPs, like the Linear, GitHub, the ones that you showed? At some point, they're all gonna have, have, have MCPs, right? Like, do you think there's still value in, like, just building your own thing on top of it? **Guy Gur-Ari** [26:18] Yeah. So as far as I know, I mean, I think the main thing to figure out is, again, from... It's not, it's not an agent quality thing. Those are just the same. It's more making it, making the authentication story really good for, for developers. This is, this is why we've built it ourselves, because this is... Now, if MCP has an authentication story that we can use to make it as seamless as we can ourselves, then yeah, we would definitely consider it, 'cause it's e-it's easier to standardize on one way of extending the tool set. **Alessio** [26:48] Yep. Um, awesome, man. This was, this was great. Any... I, I know you mentioned you're gonna do some RL on your custom models now to, to improve them. Any, yeah, interesting areas of research that you've seen in that space? ### Research **Alessio** [27:03] Anything net new that you wanna shout out that you've seen? **Guy Gur-Ari** [27:06] There was an interesting paper, I think it's called... Let me... Yeah. There was an interesting paper called SWE-RL, which I thought was a, was a really nice paper on how to do RL for, specifically for coding. So that's by Wei and other authors. So I'd recommend if you, if you, if you're interested in RL for coding, besides the normal things, so I think the DeepSeek paper, R1 was very good, understanding DPO and the variants, like GRPO is very popular now. These are all, like, great things to look up. And specifically for coding, recently, I really liked the SWE-RL paper. **Swyx** [27:39] Yeah. Um, yeah, a, a lot of exciting... And thanks for the pointers on the papers, actually. I'm gonna go look them up, uh, and try to add to my list. You were-- You mentioned that a lot of your team is ex-Google. Eric, uh, Schmidt is, is, is a backer. What's your take on this as a general Gemini Google revival? You know, you're on the, the PaLM 2 team. What seems to be... Uh, what, what, what should people view the Gemini progress as, right? ### Google **Swyx** [28:02] Like, you know, 2.5 Pro is out, and, you know, by some accounts, it's one of the best models in the world. It, it is the best model in the world. Uh, yeah. What do you, what do you think? **Guy Gur-Ari** [28:12] Yeah. It's been pretty interesting to see how, how this all developed. I think when I, uh, when I left Google for Augment... So actually, the week I gave notice at Google was the week ChatGPT came out, and so this was certainly, I mean, a well-known, like- **Swyx** [28:26] Not related? **Guy Gur-Ari** [28:27] Not related. No, no. Everything happened before. Uh, all, everything happened before, un-unrelated to ChatGPT. I was actually at a conference then. I think it was Europe, so I don't really, I'm not really sure. ChatGPT came out, everybody was talking about it. It just all happened to coincide, and I think it's well known that this was kind of a crisis moment for Google. From what I've hear-- from what I hear from friends, it seems, I mean, they're obviously taking it super seriously, and the, the, the culture has really shifted internally to be a lot more startup-y, fast-moving, aggressive. And at least folks I talk to are optimistic about Google's chances to, to win this. Lots of good people there, very motivated, clearly are making, like, solid progress on the models. It's still a race, but I think the, the s- the sentiment that I'm hearing seems to be pretty, pretty positive, I would say. **Swyx** [29:16] Yeah. I think the, I think people have really seen that Google has kinda woken up. I just don't know what it would take to get to the next level. Like, if Google, if Demis or whoever at DeepMind came and asked us, like, "Okay, what do we need to ship in order to, like, take mind share from here?" You already have the best scored model, LmSis. You already have the, one of the fastest, cheapest models with Flash, and then you, you already launch with, like, you know, native image generation. Like, what is the frontier that people care about? I don't know, man. What, what el- what else is left? **Guy Gur-Ari** [29:52] Well, I mean, AGI. I mean, we're- **Swyx** [29:54] Yeah **Guy Gur-Ari** [29:54] ... we're moving, we're, we're moving rapidly, but I wouldn't... I mean, some people say that, like, "Oh yeah, this is all AGI already." I, I don't feel that way. **Swyx** [30:04] Mm. **Guy Gur-Ari** [30:04] Like, I can-- Again, going back to coding, on the one hand, this stuff is brilliant and magical and does, sometimes whole PRs for me, but sometimes, like, and more often than not, I really have to take a task and break it down to bite-sized pieces- **Swyx** [30:20] Gum it more **Guy Gur-Ari** [30:20] ... and feed it. **Swyx** [30:21] Yeah. **Guy Gur-Ari** [30:21] So I think there's... I mean, if we're asking what's the headroom here, there's a ton of room for improvement. I think the, it's still day one, as far as I can tell, in terms of model quality. **Swyx** [30:30] Yeah, no, that, that makes sense. It's, it's funny 'cause, like, we won't know how bad today's models are until two years from now when we can compare and go like, "Oh yeah, of course," like, you know, "why would you ever use Gemini 2.5?" Well, uh, yeah, I think that's all the questions we had. Um, any, any, uh, call to action? ### Outro **Guy Gur-Ari** [30:47] Yeah. Uh, please go to AugmentCode.com, download the extension, uh, start using the agent and all our other features, and especially if you have a large complex code base, you will see the value right away. **Swyx** [30:59] And, and if you don't mind sharing your code for, for training purposes, individual developers can use it too, right? That's something I picked up from your previous podcast here, yeah. **Guy Gur-Ari** [31:07] So, so, uh, uh, yes, that's right. We have, we have a, we have a free tier where our code is available for training purposes. Maybe another thing is with, since we talked about SWE-Bench, we actually open sourced our implementation of SWE-Bench, so if you're curious how we got to number one, you'll be able to go see all the details of, of how we did it. **Swyx** [31:24] Oh, excellent. **Guy Gur-Ari** [31:25] Yeah. **Swyx** [31:25] Uh, actually, that, yeah, that would be a very good contribution to the community. Awesome. Well, thank you so much. Um- **Guy Gur-Ari** [31:29] Thank you, Guy. **Swyx** [31:30] Yep. **Guy Gur-Ari** [31:30] Yeah. Thanks so much. This was fun. Thanks. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.