# 🕰️ The Oral History of Windsurf (ft. Varun Mohan, Scott Wu, Jeff Wang, Kevin Hou, Anshul R)

Latent Space · 2025-07-28

<https://addtry.com/c88ae4b3-d23e-4ced-8008-9803317b0868>

This episode traces the oral history of Windsurf (formerly Codeium and Exafunction) with founders Varun Mohan, Scott Wu, Jeff Wang, Kevin Hou, and Anshul R, charting its evolution from a GPU virtualization startup to a leading agentic IDE and its dramatic acquisition by Cognition. The narrative covers the pivot to code completion in 2022, the shift to a free model to gain traction, and the launch of the Windsurf editor in November 2024 with Cascade, an agent that operates on a shared timeline with the developer. Key principles include meta-learning (auto-generated memories that adapt to each user's preferences) and scaling with intelligence (replacing hard-coded rules with LLM reasoning as models improve). The episode also details the acquisition weekend when Cognition bought Windsurf after Google acquired a stake, highlighting product synergies between the synchronous agent (Windsurf) and async agent (Devin) and the ambition to build a comprehensive platform for real-world software engineering teams.

## Questions this episode answers

### What exactly happened in the Cognition acquisition of Windsurf, and what were the terms regarding the Google deal?

Jeff Wang and Scott Wu stated that Cognition gained full and exclusive access to all needed Windsurf IP, describing it as a 'clean cut.' They did not comment on specific reported terms about Google's non-exclusive license but emphasized that the team's work remains proprietary and exclusive to Cognition. The deal was completed rapidly over a weekend.

[3:50:36](https://addtry.com/c88ae4b3-d23e-4ced-8008-9803317b0868?t=13836000)

### How will Windsurf and Devin work together after the acquisition?

Scott Wu and Jeff Wang said the two products have complementary philosophies: Windsurf is a synchronous, real-time agent for in-the-loop coding, while Devin is an asynchronous, background agent. They plan to integrate them, combining the strengths of both. Full integration will take weeks, but their overlapping roadmaps—Devin wanting an IDE and Windsurf wanting an agent—will accelerate development.

[3:43:03](https://addtry.com/c88ae4b3-d23e-4ced-8008-9803317b0868?t=13383000)

## Key moments

- **[0:00] Intro**
- **[2:18] Codeium Origins**
  - [2:18] Varun Mohan explains Exafunction was built to maximize GPU utilization for heterogeneous hardware, addressing the gap in managing GPUs efficiently.
  - [2:49] Varun Mohan defines model flop utilization (MFU) as the fraction of compute actually used, noting most workloads run at very low efficiency.
  - [4:19] Varun Mohan describes dynamic multiplexing: running multiple clients on one GPU to improve utilization, a technique from Exafunction.
  - [8:26] Varun Mohan advises companies to avoid building their own inference infrastructure and instead use generic models like GPT-3, fine-tuning only when needed.
  - [9:30] Varun Mohan says Codeium's pivot was driven by seeing the clear ROI in products like Copilot and their ability to build them quickly with in-house infra.
  - [10:41] Varun Mohan recalls Copilot flawlessly writing complex C++ templates, convincing the team that LLM code generation is not a toy.
- **[52:42] Devin Launch**
  - [52:42] Scott Wu introduces Devin, the first AI software engineer, demonstrating it benchmarking Llama on different API providers.
  - [53:49] Scott Wu shows Devin autonomously adding a debug print statement to fix an error, then building and deploying a visualization website.
- **[54:31] Context Retrieval**
  - [54:31] Kevin Hou says Codeium offers free unlimited autocomplete across 70 languages and 40 IDEs, with over 1.5 million downloads.
  - [56:58] Kevin Hou argues full-context prompting is infeasible: even a small repo exceeds 1 million tokens (~100k lines of code).
  - [58:23] Kevin Hou claims embedding search fails at reasoning over multiple items and introduces recall@50 as a better retrieval metric.
  - [1:00:10] Kevin Hou describes product-led evals: using GitHub PR commits as ground truth to measure retrieval quality across millions of repos.
  - [1:02:20] Kevin Hou says Codeium broke through the embedding performance ceiling by redefining retrieval and leveraging scale.
  - [1:04:54] Kevin Hou introduces MQuery: running thousands of LLMs in parallel on candidate documents to rank context with high confidence.
- **[1:12:14] Devin Workflows**
  - [1:12:16] Scott Wu presents the early makings of Devin, demoing a mobile-friendly speaker name game it built that morning from a TSV of faces.
  - [1:14:29] Scott Wu shares that Devin is used internally at Cognition to build features for itself, like a search bar PR.
  - [1:16:15] Scott Wu declares we are entering a new wave of AI agents beyond simple code generation, unlocking higher-order capabilities.
  - [1:16:55] Scott Wu argues code with agents works well because software engineering involves browsing, running commands, and iterating with feedback.
  - [1:19:16] Scott Wu demonstrates parallel Devins: users can run multiple Devin agents on different tasks simultaneously.
  - [1:20:10] Scott Wu shows Devin can be invoked via Slack to make PRs, expanding its surface area to async, mobile communication.
- **[1:31:04] Windsurf Launch**
  - [1:31:04] Varun Mohan launches Windsurf Editor, announcing a paradigm shift with AI flows that go beyond agents.
  - [1:32:10] Varun Mohan defines AI flow as a collaborative interaction where human and AI share state and intent, the successor to agents.
  - [1:36:07] Varun Mohan's engineers say Cascade is a fundamental leap from what was possible before, excelling on large, complex codebases.
- **[1:37:21] Windsurf Deep Dive**
  - [1:39:23] Varun Mohan reveals the new Windsurf office is the Silicon Valley TV show building, a 2-min walk from Caltrain.
  - [1:39:37] Varun Mohan says Codeium has over 800,000 developers and enterprise customers like Dell, driving the need to build an IDE.
  - [1:40:53] Varun Mohan rejects the 'text prompt to code' modality as only good for zero-to-one apps; real code evolution is iterative and messy.
  - [1:42:57] Varun Mohan details VS Code API failures: unable to show refactor suggestions and constant breakage, forcing Windsurf's fork.
  - [1:46:26] Varun Mohan defines Cascade as the agentic part of Windsurf that combines human trajectories and knowledge retrieval for multi-step tasks.
  - [1:48:56] Varun Mohan describes their eval method: stripping commits from open-source repos and having the model predict changes, measuring via tests.
- **[2:43:28] Windsurf Waves**
  - [2:45:37] Varun Mohan admires Palmer Luckey's Anduril for repeatedly pivoting and increasing revenue by 10x, inspiring Windsurf's adaptability.
  - [2:51:21] Kevin Hou says Windsurf generated 4.5 billion lines of code in its first three months after launch.
  - [2:52:45] Kevin Hou demos Windsurf's 'continue my work' feature, where the agent picks up from where the user left off using a shared timeline.
  - [2:54:42] Kevin Hou explains the unified timeline concept: human and AI actions interleaved in a single narrative, avoiding context loss.
  - [2:55:55] Kevin Hou predicts the future will have no terminal; commands will be embedded directly in the agentic timeline.
  - [2:56:56] Kevin Hou predicts future agents will look 10-30 steps ahead, writing unit tests before functions are even complete.
- **[3:03:28] Agent Evolution**
  - [3:05:27] Scott Wu presents Moore's Law for AI agents: capability doubles every 70 days, matching observed progress from single-line to full projects.
  - [3:09:55] Scott Wu says understanding Git commit history was a key unlock that enabled Devin to make multi-file bug fixes.
  - [3:12:22] Scott Wu says Slack integration became crucial for tagging Devin with tasks, fitting naturally into developers' workflow.
  - [3:15:14] Scott Wu describes the current tier: Devin can autonomously handle sprint backlogs, requiring confidence models and iterative local testing.
  - [3:16:18] Scott Wu says they are building a confidence model for Devin to decide when to autonomously execute versus ask for human help.
- **[3:18:55] Swe1 Model**
  - [3:19:57] Kevin Hou says Windsurf now generates 10 billion lines of code, a jump from 4.5 billion a few months earlier.
  - [3:23:59] Kevin Hou announces Swe1, Windsurf's in-house model trained on real software engineering workflows, not just code generation.
  - [3:24:12] Kevin Hou says Swe1 was built because Windsurf's workflow eval needs are very different from existing benchmarks, requiring conversational Swe tasks.
  - [3:25:45] Kevin Hou says Swe1 is chosen by users because it recognizes their workflow, not just how to generate code, and feeds back into product improvement.
  - [3:31:08] Jeff Wang says Scott Wu reached him through multiple paths after the OpenAI news, leading to a frantic weekend.
  - [3:32:37] Scott Wu says they saw natural product synergies: Windsurf for synchronous editing, Devin for async, long-running engineering tasks.
- **[3:34:28] CogSurf Future**
  - [3:34:54] Jeff Wang says customers were emailing about the company's future, forcing a fast deal to ensure stability.
  - [3:35:25] Jeff Wang recalls sleeping only three hours that entire weekend, working in a fugue state to close the deal.
  - [3:37:21] Scott Wu says they moved fast as a small startup, needing just board approval, with lawyers available around the clock in different time zones.
  - [3:38:15] Scott Wu: "At some point, Jeff and I concluded, 'If it's going to happen, we should just make it happen as fast as possible.'"
  - [3:39:37] Scott Wu confirms they will maintain both Windsurf and Devin branding, with no plans to merge the distinct product philosophies.
  - [3:40:32] Jeff Wang reveals Devin's roadmap included building an IDE, and Windsurf's roadmap included a Devin-like agent, accelerating both plans.

## Speakers

- **Alessio** (host)
- **Swix** (host)
- **Anshul R** (guest)
- **Jeff Wang** (guest)
- **Kevin Hou** (guest)
- **Scott Wu** (guest)
- **Varun Mohan** (guest)

## Topics

IDE & Editor Tools, Coding Agents, Startups

## Mentioned

Anthropic (company), Cognition (company), Exafunction (company), JetBrains (company), OpenAI (company), Claude (product), Codeium (product), Cursor (product), Devin (product), Eclipse (product), GPT (product), Gemini (product), GitHub Copilot (product), Grok (product), Llama (product), Swe1 (product), VS Code (product), Windsurf (product)

## Transcript

### Intro

**Swix** [0:00]
Hello, and welcome to a very special Lint Space Pod. As many of you know, rumors of OpenAI acquiring Windsurf for $3 billion have been going on for roughly March to May of this year. In July, the OpenAI deal finally fell through, and it was reported that Windsurf leadership went to Google for roughly $2.4 billion in cash.

All of this is publicly reported; there's some minor nuances, but we're not breaking any news here. One week later, Cognition acquired the rest of Windsurf for what has been estimated to be $300 million in stock. At the end of this special pod, we have an exclusive interview with Scott Wu and Jeff Wang, the CEOs of Cognition and Windsurf respectively, about the deal and the path forward.

Please note that we couldn't get Varun on the record, the former CEO of Windsurf. But the reason we're doing this pod is not really for breaking news or controversy or anything like that; it's more for the tens of thousands of you that joined us in the past year and haven't listened to our backlog.

I meet you guys every single day, and you always ask me for something or someone that we probably already have done a podcast with, and that's just because you haven't been with us the entire time. As the AI Engineering podcast, we pride ourselves on having the world's best coverage of AI coding tools and software agents, and are often the first podcast that teams like Codeium, Cursor, Fine, Replit AI have ever done.

We have a uniquely long history of covering Windsurf. I met them when they were still called Exafunction, and our first-ever guest post was from Anshul at Codeium, while our second-ever podcast episode was with their CEO, Varun, as Codeium.

I've hosted both Cognition and Windsurf at the AI Engineer World's Fairs in 2024 and 2025, and our most recent podcast was recorded last November,right after the launch of Windsurf in their offices. This episode is a little special in that it is a recap episode, or recap episode, really just running through chronologically, telling the story as an oral history from start to finish, and ending with our special interview with Jeff and Scott, which I've been affectionately calling CogSurf.

I'm excited to compile this for you to rewind back in time, and I hope you enjoy.

**Alessio** [2:18]
Hey everyone, welcome to the Latent Space podcast. This is Alessio, partner and CTO in residence at Decibel Partners. I'm joined by my co-host, Swix, writer and editor of LSpace Diaries.

### Codeium Origins

**Swix** [2:29]
Hey, and today we have Varun Mohan from Codeium/Exafunction on. I should introduce you a little bit, because I like to get the LinkedIn background out of the way. So you did CS at MIT, and then you spent a few years at Neuro, where you were ultimately tech lead manager for autonomy, and that's an interesting dive into self-driving cars and AI.

And then you went straight into Exafunction with a few of your coworkers, and that's where I met some of them and started knowing about Exafunction. And then, from out of nowhere, you cloned GitHub Copilot. That's a lot of progress in a very short amount of time.

So anyway, welcome.

**Alessio** [3:07]
That's high praise.

**Swix** [3:08]
What's one thing about you that doesn't appear on LinkedIn that is like a big part of what people should know?

**Alessio** [3:13]
I actually really like endurance sports, actually. Like, I've done multiple triathlons. I've actually biked from San Francisco to LA. Yeah, I like things that are like suffering. I like to suffer while I do sports. Yeah.

**Swix** [3:27]
Do you think a lot about code and tech while you're doing the endurance sports, or your mind is just focused?

**Alessio** [3:34]
I think it's maybe a little bit of both. One of the nice things about, I guess, endurance athletics is it's one of the few things you can do where you're not thinking about. You can't really think about much beyond suffering.

Like, you're climbing up a hill on a bike, and you see how many more feet you need to climb, and at that point, you're just struggling.

**Swix** [3:52]
That's your only job.

**Alessio** [3:53]
Yeah. The only thing you can think is pedaling. One more pedal. So it's actually like a nice way to not think about work. Yeah.

**Swix** [4:00]
Yeah.

**Alessio** [4:00]
Yeah.

**Swix** [4:00]
Yeah, maybe for the audience, you want to tell a bit about Exafunction, how they came to be, and how Codeium came out of that.

**Alessio** [4:07]
Yeah. So a little bit about Exafunction. Before working at Exafunction, I worked at Neuro, as Sean was just saying. And at Neuro, I sort of managed large-scale offline deep learning infrastructure, realized that deep learning infrastructure is really hard to build and really hard to maintain for even the most sophisticated companies, and started Exafunction to basically solve that gap, to make it so that it was much easier for companies to serve deep learning workloads at scale.

One of the key issues that we noticed is GPUs are extremely hard to manage, fundamentally because they work differently than CPUs. And once a company has heterogeneous hardware requirements, it's hard to make sure that you get the most out of the hardware.

It's hard to make sure you can get great GPU utilization. And Exafunction was specifically built to make it so that you could get the most out of the hardware, make sure that your GPU is effectively virtualized and decoupled from your workload, to make it so that you could be confident that you were running at whatever scale you wanted without burning the bank.

**Swix** [5:07]
Yeah, you gave me this metric about inefficiency,right?

**Alessio** [5:10]
Oh, okay. Like flop efficiency?

**Swix** [5:12]
Yeah.

**Alessio** [5:12]
Yeah. So basically, I think it comes down to, for most people, one of the things about CPUs that's really nice is with containers,right? You can end up having a single node, and you can place many containers on them, and all the containers will slowly start eating the compute.

It's not really the same with GPUs. Like, let's say you have a single node. For the most part, only have one container using that GPU. And because of that, people heavily underestimate what a single container can sort of do, and the GPU is left like heavily idle.

And I guess the common term now with a lot of LLM workloads is like the flop efficiency of these workloads.

**Swix** [5:47]
MFU.

**Alessio** [5:48]
Yeah.

**Swix** [5:48]
Yeah, model flop utilization.

**Alessio** [5:49]
The model flop utilization, which is basically like what fraction of the flops or compute on the hardware is actually getting used. And sort of what we did at Exafunction was not only make it so that the model was always running, we also built compiler technology to make it so that the model was also running more efficiently.

And some of these things are with tricks like operator fusion. Like, basically, you could imagine fusing two operations together such that the time it takes to compute the fused operation is lower than the time it takes for each individual operation.

**Swix** [6:18]
Oh my God.

**Alessio** [6:19]
Yeah.

**Swix** [6:20]
Yeah. And you have this technique called dynamic multiplexing, which is basically, instead of having a one-to-one relationship, you have one GPU for multiple clients. And I saw one of your customers, they went from 30 clients to just one single GPU, and they cut costs by 97%.

What were some of those learnings in hardware usage inefficiencies and how that then played into what you're building now?

**Alessio** [6:42]
Yeah, I think it basically showed that there was probably a gap with even very sophisticated teams making good use of the hardware is just not an easy problem. I think that was the main. It's not that these teams were like not good at what they were doing.

It's just that they were trying to solve a completely separate problem. They had a model that was trained in-house, and their goal was to just run it. And that should be an easy thing to do, but surprisingly, still, it's not that easy.

And that problem compounds in complexity with the fact that there are more accelerators now in the cloud. There's like TPUs, inferentia, and there's a lot of decisions that users need to make, even in terms of GPU types. And I guess sort of what we had was we had internal expertise on what theright way to run the workload was, and we were basically able to build infrastructure to make it so that companies could do that without thinking.

**Swix** [7:27]
So most teams are underutilizing their hardware. How should they think about what to own? Like, should they own the inference architecture? Like, should they use XDEPLOID to get it to production? How do you think about it?

**Alessio** [7:39]
Yeah. So I think one thing that has proven to be true over the last year and a half is companies, for the most part, should not be trying to figure out what the optimal ML architecture is or training architecture is, especially with a lot of these large language models.

We have generic models and transformer architecture that are solving a lot of distinct problems. I'll caveat that with most companies because some of our customers, which are autonomous vehicle companies, have extremely strict requirements. Like, they need to be able to run a model at very low latency, extremely high precision recall.

GPT-3 is great, but the precision recall, you wouldn't trust someone's life with that,right? So because of that, they need to innovate new kinds of model architectures. But for a vast majority of enterprises, they should probably be using something off-the-shelf, fine-tuning BERT models.

If it's Vision, they should be fine-tuning ResNet or using something like CLIP. Like, the less work they can do, the better. And I guess that was a key turning point for us, which is like we started to build more and more infrastructure for the architectures that were like the most popular.

And the most popular architecture was the transformer architecture. We had a lot of LLM companies explicitly reach out to us and ask us, "While our GPT-3 bill is high, is there a way to serve GPT-3 or some open-source model much more cheaply?"

And that's sort of what we viewed as why we were maybe prepared for when we internally needed to deploy transformer models ourselves.

**Swix** [9:05]
And so the next step was, "Hey, we have this amazing infrastructure. We can build kind of consumer-facing products, so to speak, with much better unit economics, much better performance." And that's how Codeium kind of came to be?

**Alessio** [9:18]
Yeah. I think maybe the play is not maybe for us to be just we make a lot of consumer products. We want to make products with like clear ROI in the long term in the enterprise. Like, we view Codeium as maybe one of those things.

And maybe we can talk about Codeium maybe after this. We view products like Copilot as being extremely valuable and something that is generating a lot of value to professionals. We saw that there was a gap there where a lot of people probably weren't developing high-intensive LLM applications because of cost, because of the inability to train models the way they want to.

And we thought we could do that with our own infrastructure really quickly.

**Swix** [9:55]
I want to highlight, when you say high-intensive, you mean basically generate models every key, generate inferences on every keystroke.

**Alessio** [10:00]
That'sright. Yeah. So I would say like there's probably two kinds of LLM applications here. There's an LLM application where it rips through a bunch of data, and maybe you wait a couple minutes and then you see something. And then there's an application where the quality is not exactly what you want, but it's able to generate enough, sorry, low enough latency that it's still providing a ton of value.

And I will say there's like a gap there where the number of products that have hit that Copilot spot is actually not that high. A lot of them are kind of like wait and just generate a lot of stuff and see what happens.

Because one is clearly more compute-intensive than the other, basically.

**Swix** [10:37]
Well, Codeium, I don't know if we told the whole story yet. You were going to dive into it.

**Alessio** [10:41]
Yeah. So I guess the story was, I guess, four or five months ago, we sort of decided internally as a team, we were like very early adopters of Copilot. I'm not going to sit here and say Copilot's not a great tool.

We love Copilot. It's like a fantastic tool. We all got on the beta the moment it came out. We're like a fairly small team, but we all got it, and we were showing each other completions. We ended up writing like a lot of CUDA and C++ inside the company, and I think there was probably a thought process within us that was like, "Hey, the code we write is like very high IQ.

So like there's no way it can help." And one of the things in C++ that's like the most annoying is writing templates. Writing template programming is maybe one of those things. Maybe there's like some people in the C++ standards community that can do it without looking at anything online.

But we struggle. We struggle writing variety of templates. And Copilot just like ripped through. Like, we had a 500-line file, and it was just like writing templates. And we didn't really even test it while we were running it.

We then just compiled it, and it just compiled. And we were like, "Wow. This is actually something that's not just like it's completing for loops. It's completing code for us that is like hard in our brains to reach, but fundamentally and logically is not that complicated.

The only reason why it's complicated is there's just a lot of rules,right?" And from then, we were just like, "Wow. This is " That was maybe the first LLM application for us internally, because we're not like marketers that would use Jasper, where we were like, "Wow, this is like extremely valuable.

This is not a toy anymore." So we wanted to take our technology to build maybe apps where these apps were not going to be toys,right? They were not going to be like a demo where you post it on Twitter, and then there's hype, and then maybe like a month later, no one's using it.

**Swix** [12:17]
There's a report this morning from Copilot where they were estimating the key tabs on amount of code generated by Copilot that is then left in code repos and checked in. And it's something like 60 to 70%.

**Alessio** [12:29]
That's nuts, but I totally believe it given the stats we have too. There's this flip in your head once you start using products like this where in the beginning, there's like skepticism, like how valuable can it be? And suddenly now, like user behavior fundamentally changes so that now when I need to write a function, I'm like documenting my code more because I think it's prompting the model better,right?

So there's like this crazy thing where it's a self-fulfilling prophecy where when you get more value from it, more of your code is generated from Copilot.

**Swix** [12:55]
Just to walk through the creation process, I actually assume that you would have grabbed your data from the pile, which is the Luther AI open-source code information. But apparently, you scraped your own stuff.

**Alessio** [13:06]
Yeah. We ended up basically using a lot of open, I guess, permissively licensed code in the public internet, mainly because I think also the pile is fairly a small subset. I think maybe after we started, there was the stack that also came to be.

But for us, we had a model for ourselves even before that. Was the point.

**Swix** [13:24]
Ah, okay. So the timing was just a bit off.

**Alessio** [13:26]
Yeah, exactly. Exactly. But it's awesome work. It seems like there's a good amount of work that's getting done decentrally, which is a little bit surprising to me because I'm like more bullish on everyone needs to get together in a room and make stuff happen.

Like, we're all in person in Mountain View. But yeah, no, it's pretty impressive work. Yeah. Luther in general, like everything they've done, I'm pretty impressed with it.

**Swix** [13:45]
Yeah. And we're going to talk about that because I didn't know you were that involved in the community that early on.

**Alessio** [13:49]
I wasn't involved. I was more of like I was watching and maybe commenting from time to time.

**Swix** [13:53]
So you're a very special community for sure.

**Alessio** [13:55]
Yeah. Yeah.

**Swix** [13:55]
Yeah, that's true. That's true. My impression is a bunch of you are geniuses. You sit down together in a room, and you get all your data, you train your model, like everything's very smooth sailing. What's wrong with that image?

**Alessio** [14:06]
Yeah. So probably a lot of it just in that a lot of our serving infrastructure was already in place before then. So like, hey, we were able to knock off one of these boxes that I think a lot of other people maybe struggle with.

The open-source serving offerings are just, I will say, not great in that they aren't customized to transformers and these kind of workloads where I have high latency and I want to like batch requests. And I want to batch requests while keeping latency low,right?

One of the weird things about generation models is they're like autoregressive. At least for the time being, they're autoregressive. So the latency for a generation is a function of the amount of tokens that you actually end up generating.

Like, that's like the math. And you can imagine while you're generating the tokens, though, unless you batch a lot, it's going to end up being the case that you're not going to get great flop utilization on the hardware.

So there's like a bunch of trade-offs here where if you end up using something completely off-the-shelf, like one of these serving frameworks, you're going to end up leaving a lot of performance on the table. But for us, we were already kind of prepared to sort of do that because of our infrastructure that we had already built up.

And probably the other thing to sort of note is early on, we were able to leverage open-source models, sort of bootstrap it internally within our company. But then to ship, we finally had some requirements like, "Hey, we want this model to have fill-in-the-middle capabilities and a bunch of other things."

And we were able to ship a model ourselves. So we were able to time it so that over the course of multiple months, different pieces were like working out properly for us. So it wasn't like we started out and we were just planning the launch materials the moment we started.

There was like maybe some stuff that was already there, some stuff that we had already figured out how to train models at scale internally. So we were able to just leverage that muscle very quickly.

**Swix** [15:43]
I think the one thing that you had figured out from the beginning was that it was going to be free forever.

**Alessio** [15:47]
Yeah. Yeah.

**Swix** [15:49]
Copilot costs $10 a month.

**Alessio** [15:51]
Copilot costs $10 a month. I would argue significantly more value than $10 a month. The important thing for us, though, was we were going to continue to build more great products on top of code completion. We think code completion is maybe day one of what the future looks like.

And for that, clearly, we can't be a product that's like, "We're $10 a month and we're adding more products." We want a user base that loves using us, and will continue to stay with us as we continue to layer on more products.

And I'm sure we're going to get more users from the other products that we have, but we needed some sort of a differentiator. And along the way, we realized, "Hey, we're pretty efficient at running these workloads. We could probably do this."

**Swix** [16:25]
Oh, so it wasn't planned to be free from the start. You just realized.

**Alessio** [16:29]
Yeah. We realized we could probably, if we cut and optimized heavily, we could probably do this properly. Part of the reasoning here was we were confident we could probably build a pro tier and go to the enterprise. But for now, originally when we started, we weren't like, "We're just going to go and give all pieces of software away for free."

That wasn't like sort of the goal there.

**Swix** [16:46]
And since you mentioned adoption and traction and all that, what can you disclose about user growth, user adoption?

**Alessio** [16:52]
Yeah. Soright now, we probably have over 10,000 users and thousands of daily actives, and people come back day over day. Our growth is like around 4 to 5% day over dayright now. So all of our growthright now is sort of like word of mouth.

And that's fundamentally because the product is actually one of those products where even used Copilot and used us, it's hard to tell the difference, actually. And a lot of our users have actually turned off of Copilot.

**Swix** [17:16]
Yeah. I saw it. Yeah. Most users support you guys, but also to try it out.

**Alessio** [17:20]
Yeah, exactly. So the crazy thing is it wasn't like, "Hey, we're going to figure out a marketing motion of like going to the people that have never heard of Copilot and we're going to like get a bunch of users."

We wanted to just get users that in our ownright were like a really great product. And sort of we've spent a lot of engineering time, and obviously, we co-wrote a blog post with you, Sean, on this in terms of like there's a lot of engineering work even beyond the latency, making sure that you can get your costs down to make a product like this actually work.

**Swix** [17:46]
Yeah. That's a long tail of stuff that you referenced.

**Alessio** [17:49]
Yes. Yeah, exactly.

**Swix** [17:50]
And you said something to the order of, and this maybe gets into Copilot for X, which is something that everybody is keen about because they see the success of Copilot. They're like, "Okay, well, first of all, developer tools, there's more to do here.

And second of all, let's take the Copilot idea and apply it for other disciplines." I don't know if you want to.

Yeah. There's kind of some key points that you touched on, how to estimate inference at scale and the latency versus quality trade-offs, building on first-party. So this is free forever because you run your own models,right?

That'sright.

If you were building on OpenAI, you wouldn't be able to offer it for free. Real-time, when I first used Codeium, it was literally the same speed as Copilot.

I think it's a little bit faster. I don't know how to quantify it, but.

**Alessio** [18:32]
We are faster, but it's one of those things that we're not going to like market as that's the reason because it's not in and of itself aright for you to like, I'm just going to be open with you.

It's not a reason for you to like suddenly turn off of Copilot where if our answers were trash, but we were faster. You know what I mean?

**Swix** [18:46]
But your focus was there. We used the half, I think, prem, on our Discord, Kim told us and said, "You guys should try this out." So it was really fast even then. Prompt optimization is another big thing, and model outputs and UX, kind of how you bring them together.

Which ones of these things are maybe like the one or two that new founders should really think about first?

**Alessio** [19:08]
Yeah. I think my feeling on this is unless you are you probably should always bootstrap on top of an existing API,right? Because like even if you were to, the only reason why we didn't is because we knew that this product was actually buildable.

Probably if we worked hard enough to train a model, we would actually be able to build a great product already. But if you're actually going out and trying to build something from scratch, unless you genuinely believe, "I need to fine-tune on top of terabytes of data," a terabyte is a very large amount of data, but like tens of gigabytes of data, probably go out and build on top of an API and spend most of your time to make it so that you can hit that quality-latency trade-off properly.

And if I were to go out and think about like the three categories of like an LLM product, it's probably like latency, quality, and correctability. The reality is, if I were to take a product like Copilot or Codeium, the latency is very low.

The quality, I think, is good enough for the task, but the correctability is very easy.

**Swix** [20:03]
So correctability, what is correctability?

**Alessio** [20:04]
Correctability means let's say the quality is not there. Like you consider the case where the answer is wrong. How easy is it for your user to actually go and leverage parts of the generation? Maybe a concrete example is a lot of things people are excited aboutright now are I write a comment and it generates a PR for me.

And that's like really awesome in theory. I think that's like a really cool thing. And I'm sure at some point, we will be able to get there. That will probably require an entirely new model for what it's worth that's trained on diffs and commits and all these other things that looks at like improvements in code and stuff.

It's probably not going to be just trained on generic code. But the problem with those sort of, I would say, applications are that let's suppose something does change many files, makes large amounts of changes. First of all, it's guaranteed not going to be fast because even the idea of like reviewing the change takes a long time.

So if the quality and the correctability is just not there, let's say you had a 10-file change and you modified like file two and four, and those two modifications were consistent, but the other eight files were not consistent, then suddenly the correctability is like really hard.

It's hard to correct the output of the model. And so the user interface is 100% really important, but maybe until you get the latency down or the correctability, like correctability like a lot better, it's probably not going to be shippable.

And I think that's what you got to spend your time focusing on. Can you deliver a product that is actually something users want to use? And I think this is why I was talking about like demo wear. It's like very easy to handpick something that works for a demo, exceedingly hard for something that has large scope, like a PR, to work consistently.

It will take a lot of engineering work effort to make it work on small enough chunks so that a user is like, "Wow, this is value generative to me." Because eroding user trust or consumer trust is very easy.

Like that is, it is much, much, it's very easy to erode user trust versus enterprise trust. So just be mindful of that. And I think that's probably like the mantra that most of these companies need to operate under.

**Swix** [22:05]
Have you done any analysis on what the ratio between code generated and latency is? So you can generate one line, but you could also generate the whole block. You could generate a whole class. And the more you generate, the more time it takes.

Like what's the sweet spot that you found?

**Alessio** [22:22]
So I think there was a great study, and I'm not sure if it's possible to link it, but there was a great study about Copilot actually that came out. And basically what they said was there were two ways that developers usually develop with a code assistant technology.

They're either in what's called like acceleration mode or exploration mode. And exploration mode is basically you're in the case where you don't even know what the solution space for the function is, and you just want to generate a lot of code because you don't even know what that looks like.

Like it might use some API that you've never heard of. And what you're actually doing at that point is like you're writing a clean comment, just wishing and praying that the generation is long enough and gets you far enough,right?

Acceleration mode is basically you are doing things where you are very confident in what you're doing. And effectively, Codeium gives you that muscle so that you can basically stay in flow state and you're not thinking about like exactly what the APIs look like.

But push comes to shove, you will figure out what the APIs look like. But actually, like mentally, it takes off like a load in your head where you're like, "Oh, wow, like I can just do this." The intent to execution is just a lot, a lot lower there.

And I think effectively, you want a tool that captures that a little bit. And we have heuristics in terms of capturing whether or not you're in acceleration versus exploration mode. And a good heuristic is, let's say you're inside like a basic block of a piece of code.

Let's say you're inside a block of code or an if statement. You're probably already in acceleration mode, and you would feel really bad if I started generating the else clause because what happens if that else clause is really wrong?

That's going to cause like mental load for you because you're the way programmers think they only want to complete the if statement first, if that makes sense. So there are things where we are mindful of like how many lines we generate.

If you use the product, like multi-line generations happen and we are happy to do them, but we don't want to do them when we think it's going to increase load on developers, if that makes sense.

**Swix** [24:08]
That makes sense. So Copilot for X, what are Xs that you think are interesting for people to build in?

**Alessio** [24:14]
Didn't we see some tweet recently about Harvey.ai company that is trying to sell legal assistance? That's pretty impressive. Honestly, that's very impressive. So it seems like I would really love to see what the product looks like there because there's a lot of text there.

Looking at Bing AI, like I mean, it's pretty cool, but it seems like groundedness is something a lot of these products struggle with. And I assume legal, if there's one thing you want them to be to getright, it's like the groundedness.

**Swix** [24:42]
I've actually made the analogy before that law and legal language is basically just another form of programming language. You have to be that precise. Definitions must be made and you can scroll to find the definition. This is the same thing.

**Alessio** [24:55]
Yes. Yes. Yeah. But like I guess there's a question of like comprehensiveness. So like let's say the only way it generates a suggestion is it provides like citations to other legal docs. You don't want it to be the case that it misses things.

So you somehow need the comprehensiveness, but also at the same time, you also don't want it to make conclusions that are not from the things it cites. So I don't know. Like that's very impressive. It's clear that they've demonstrated some amount of value because they've been able to close a fairly sizable enterprise contract.

It was like a firm with 3,500 lawyers. Something nuts. Honestly, very cool. So it's clear this is going to happen. And I think people are going to need to be clever about how they actually make it work within the constraints of whatever workload they're operating in.

**Swix** [25:36]
Well, so you guys are so good at trading stuff. Why don't you try cloning it?

**Alessio** [25:40]
Yeah. So I think that's.

**Swix** [25:43]
Preview the roadmap?

**Alessio** [25:44]
Yeah, yeah, yeah. No, no, no. But I'm just kidding. I think one of the things that we genuinely believe as a startup is most startups can't really even do one thing properly.

**Swix** [25:52]
Focus.

**Alessio** [25:53]
Yeah. Yeah. Usually doing one thing is really hard. Most companies that go public have like maybe a couple big products. They don't really have like 10. So we're under no illusions that give the best product experience the amount of engineering and attention to detail to build one good product is hard.

So it's probably going to be a while before we even consider leaving code. Like that's going to be a big step because the amount of learning we need to do is going to be high. We need to get users,right?

We've learned so much from our users already. So yeah, I don't think we'd go to law anytime soon.

**Swix** [26:22]
3,500 lawyers with Ellen and Overy is apparently the new headline.

**Alessio** [26:27]
That's actually really big. Yeah, yeah.

**Swix** [26:28]
Congrats to them.

**Alessio** [26:29]
Yeah.

**Swix** [26:29]
It's funny because like it seems like these guys are moving faster than Copilot. Copilot just launched, just announced enterprise, like Copilot for Teams or Copilot for Enterprise after like two years of testing.

**Alessio** [26:39]
Yeah. It does seem like the Copilot team has built a very, very good product. So I don't want to like say anything, but I think it is the case that startups will be able to move faster. I feel like that is true.

But hey, like GitHub has great distribution. Whatever product they do have, they will be able to sell it really well.

**Swix** [26:55]
Shall we go into model numbers and infra estimates?

Our favorite topic, small models.

**Alessio** [27:02]
Nice.

**Swix** [27:03]
So this is. Event to basically, I'm researching a lot of scaling law stuff. You have a lot of thoughts. You host paper discussions in your team?

**Alessio** [27:11]
Yeah. We try to like read papers that we think are really interesting and relevant to us. Recently, that's been, there's just a fire hose of papers. Someone even just curating what papers we should read internally as a company.

But yeah, I think there's so much good content out there.

**Swix** [27:27]
You guys should have a podcast. I mean, I told you this before.

**Alessio** [27:29]
Should have a podcast?

**Swix** [27:30]
Just put a mic near where you guys are talking.

**Alessio** [27:32]
We got to keep developing Codeium though.

**Swix** [27:35]
No, but you're doing this discussion anyway. You might as well just.

**Alessio** [27:38]
Oh, put the discussion on a podcast. I feel like some of the thoughts are raw,right? Like they're not going to be as nuanced. Like we'll just say something completely stupid during our discussions.

**Swix** [27:46]
I don't know.

**Alessio** [27:47]
Maybe that's exciting. Maybe that's, it's kind of like a Justin.tv, but for ML papers. Okay, cool.

**Swix** [27:53]
I watched that. So Copilot is 12 billion parameters. Salesforce Code Gen is up to 16. GPT-3 is 175. GPT-4 is going to be 100 trillion billion.

**Alessio** [28:03]
Yeah, yeah, yeah.

**Swix** [28:04]
So what we landed on with you with Chinchilla is that we now have an idea of what compute optimal data scaling is, which is about 20 times parameters. Is that intuitive to you? Like what did that unlock for you?

**Alessio** [28:17]
Yeah. I think basically what this shows is that bigger models are like more data efficient. Like given the same number of tokens, a big model like trained on the same number of tokens, a bigger model is like is going to learn more, basically.

But also at the same time, the way you have to look at it is there are more flops to train a bigger model on the same number of tokens. So like let's say I had a 10 billion parameter model and I trained it on 1 million tokens, but then I had a 20 billion parameter model at the end of it will be a better model.

Like it will have better perplexity numbers, which means like the probability of like a prediction is going to be better for like the next token is going to be better. But at the end of it, you did burn twice the amount of compute on it,right?

So Chinchilla is an interesting observation, which says if you have a fixed compute budget and you want the best model that came out of it, because there's like a difference here where a model that is smaller trained on the same number of tokens has fewer flops.

There's a sweet spot of like number of tokens and size of model. I will say like people probably like are talking about it more than they should, and I'll explain why. But it's a useful result, which is like let's say I have some compute budget and I want the best model.

It tells you what you should generate. The problem I think here is there is a real trade-off of like you do need to run this model somewhere. You need to run it on a piece of hardware. So then it comes down to how much memory does that piece of hardware have?

Let's say for a fixed compute budget, you could train a 70 billion parameter model. What are you going to put that on? Yeah, maybe you could, could you put that on an 80 gig A100? It would be a stretch.

You could do things like int8 FPA to reduce the amount of memory that's on the box and do all these other things. But you have to think about that first,right? When you want to go out and train that model, the worst case is you ended up training that model and you cannot serve it.

So actually what you end up finding is for a lot of these code completion models, they are actually what you would consider overtrained. So by that, I mean like let's look at a model like CodeGen. It's actually trained on, I believe, and I could be wrong by 100 billion here or there.

**Swix** [30:16]
I got some data.

**Alessio** [30:17]
Oh, okay. Let's look at the 3 billion parameter model. It's a 2.7. I think it's actually a 2.7 billion parameter model. It's weird because they also trained on natural language on top of code, but it's trained on hundreds of billions of tokens.

If you applied that Chinchilla optimization to it, you'd be like, wow, this is a stupid use of compute,right? Because three, they should be going to 60, anything more than 60, and they're like, they should have just increased the model size.

But the reality is if they had like the compute optimal one might not be one that's easy to serve,right? It could just have more parameters. And for our case, our models that we train internally, they might not be the most compute optimal.

In other words, we probably could have had a better model by making it larger, but the trade-off would have been latency. We know what the impact of having higher latency is. And on top of that, being able to fit properly on our hardware constraints would have also been a concern.

**Swix** [31:06]
So isn't the classic stopping point when you see like loss kind of levels off,right?

**Alessio** [31:12]
Yeah.

**Swix** [31:12]
Now you're just letting Chinchilla tell you, but like you should just look at loss.

**Alessio** [31:15]
The problem is the loss will like continue to go down. It'll just continue to go down like in a way that's like not that pleasing. It's going to take longer and longer. It's going to be painful. But it's like one of those things where if you look at the perplexity number, a difference between like, let's say a model that's like 70 billion versus 10 billion, it's not massive.

It's not like tens of percentage points. It's like very small,right? So the reality is here, like, I mean, this comes down to like IQ of like these models in some sense. Like small wins at the margins are massive wins in terms of IQ.

Like it's harder to get those and they don't look as big, but they're like massive wins in terms of reasoning. They can now do chain of thought, all these other things.

**Swix** [31:52]
Yeah, yeah. And it's apparently unlocked around the 20 billion.

**Alessio** [31:55]
Yes, that'sright.

**Swix** [31:57]
Some kind of magic.

**Alessio** [31:58]
Yeah. I think that was from the UL2 or maybe one of those SLAN papers.

**Swix** [32:01]
Any thoughts on why? Like is there.

**Alessio** [32:03]
I don't know. I mean.

**Swix** [32:04]
Emergence of intelligence?

**Alessio** [32:05]
I think maybe one of the things is like we don't even know maybe like five years from now what we're going to be running our transformers,right? But I think it's like we don't 100% know that that's true. I mean, there's like a lot of maybe issues with the current version of the transformers, which is like the way attention works, the attention layers work, the amount of compute is quadratic in the context length because you're like doing like an n squared operation on the attention blocks, basically.

And obviously, one of the things that everyone wantsright now is infinite context. They want to shove as much crap as possible in here. And the current version of what a transformer looks like is maybe not ideal for that,right?

You might just end up burning a lot of flops on this when there were probably more efficient ways of doing it. So I'm sure in the future there's going to be tweaks to this. But it is interesting that we found out interesting things of like, hey, bigger is pretty much always better.

There are probably ways of making smaller models significantly better through better data. That is like definitely true. And I think one of the cool things that the stack showed actually was they did a, like a, I think they did some ablation studies where they were like, hey, what happens if we do decontamination of our data?

What happens if we do deduplication? What happens if we do near dedupe of our data? And how does the model get better? And they have like some compelling results that showcase data quality really matters here. But ultimately, like, yeah, I think it is an interesting result that at 20 billion there's something happening.

But I also think like some of these things in the future may look materially different than what they look likeright now.

**Swix** [33:27]
Do you think the token limitation is actually a real architectural limitation? Like if you think about the tokens need is kind of like asymptotic,right? Like once you have 50,000 tokens context, like 50,000 or infinite for most use cases, it's like the same.

Where do you think that number is, especially as you think about code? Like some people have very large code bases. There's a lot. Have you done any work there to figure out where the sweet spot is?

**Alessio** [33:53]
Yeah. Look, I think what's going to really end up happening is if people come up with a clever way, and there was some research that I believe came out of Stanford recently. I think the team from the Helm group, I think, came out with some architecture that looks a little bit different than transformers.

And I'm sure something like this will work in the future. What I think is always going to happen is if you find a cheap way to embed context, people are going to figure out a way to put as much as possible in because LLMs so far have been like virtually stateless,right?

So the only thing that they have beyond fine-tuning is like just shoveling everything you can inside. And there are some interesting papers like RETRO. Actually, there are maybe some interesting pieces of thought, like ideas that have come out recently.

**Swix** [34:31]
Yeah, let's go through them.

**Alessio** [34:32]
So one of the really interesting ideas I think is RETRO. It's this paper that came out of DeepMind. And the idea is actually, let's say you send out a prompt, okay? Send out a prompt. You compute the BERT embedding of that prompt, and then you have this massive embedding database.

And by massive, I'm not talking about like gigabytes. I'm talking about terabytes. Like you actually have 10 times the number of tokens as what was used to train the model. So like let's say you had a model that was trained on a trillion tokens.

You have a 10 trillion embedding database. And obviously Google has this because they have all content that ever existed in humanity and they have like the best data set and sort of they were able to make one of these embedding databases.

But the idea here, which is really cool, is you end up taking your prompt, computing the BERT embedding, you find out the things that were nearby. So you do roughly like a semantic search or embedding search within that.

And then you take those, you take the documents that were from those embeddings and you shove those in the model too in what are called like cross-chunked attention. So you like shove them in the model with it as well.

Suddenly now the model is able to take in external information, which is really exciting actually, because suddenly now you're able to get dynamic context in and the model in some sense is deciding what that context is. It's not deciding it completely in this case because the BERT model in this case was actually frozen.

It wasn't trained with the RETRO model as well. But the idea is you're somehow adding or augmenting context, which I think is like quite exciting. There's probably two futures. Either context becomes really cheap. Right now it's quadratic. Maybe there's a future where it becomes linear in the size of the context, but the future might actually be the model itself dictates, "Hey, I have this context, you have this data source, give me this."

So the model itself is going out into your database and like being like, "I want this information." This is kind of like what Bing search is looking like,right? Or Bing chat is sort of looking like where it's like the model is probably, there's probably some model that's saying, "I want this information," and that is getting augmented into the context.

Now the model itself knows what context it sort of has and it can sort of like build a state machine of sort of what it needs. And that's probably what the future of this looks like.

**Swix** [36:32]
So you predict monster embedding database companies?

**Alessio** [36:35]
Probably monster embedding database companies or yeah, the model in some sense will need to talk to these, talk to these embedding databases. I'm actually not convinced that the current breed of embedding database companies are like ready for what the future sort of looks like because I think I'm just looking at their pricing, how much it costs per gigabyte, and it's prohibitive at the scale we're talking about.

Like let's say you actually did want to host a 10 terabyte embedding database. A lot of them were created, let's say, two years ago, two, three years ago where people were like, you know, embedding databases are small and they need to make the cost economics work.

But maybe, yeah, there's probably going to be a big workload there. I will just say for us, we will probably just build this in-house to start with. And that's because I think the technology probably isn't there yet. And I think if the technology isn't there yet, like waiting on point solutions to come up is a lot harder than probably building it out.

The way I like to think about this is probably the world looks on the LLM space looks like how the early internet days were where I think the value was accrued to probably like Google and Google needed to figure out all the crazy things to make their workload work.

And the reason why they weren't able to outsource is no one else was feeling the pain.

**Swix** [37:41]
They're just solving their own pain points.

**Alessio** [37:42]
They're just solving their own pain points.

**Swix** [37:43]
They were so far ahead of everyone else.

**Alessio** [37:44]
Yes.

**Swix** [37:45]
And just wait for people to catch up.

**Alessio** [37:46]
Yes. And that's maybe different than how things like Snowflake look where the interface has been decided for what SQL looks like 50 years ago. And because of that, you can go out and build the best database and yeah, like everyone's going to be like, "This doesn't make my beer taste better," and buy your database basically.

**Swix** [38:03]
That's a great reference, by the way. We have some friends of the pod that are working on embedding database. We'll try to connect you to Chroma and see.

**Alessio** [38:10]
Yeah. Oh, I actually know Anton. I worked with him at Neuro.

**Swix** [38:12]
Oh, there you go. What do you think about, I mean, so Chroma's pivoting towards an embedding database.

**Alessio** [38:18]
I think it's an interesting idea. I think it's an interesting idea. I wonder what the early set of workloads that they will hit are and what the scaling requirements are. This is maybe the classic thing where like the teams are great, but you need to pick a workload here that you care about the most.

**Swix** [38:31]
You could build anything.

**Alessio** [38:32]
You could build anything. When you're an infrastructure company, you can go in, "If I was selling serving infra, I could build serving for like linear regression. I could build this." But like unless you hit theright niche for the end user, it's going to be hard.

So I think I'm excited to see what comes out and if they're great, then we'll use it. Yeah.

**Swix** [38:49]
I also like how you suddenly equated yourself to Google there.

**Alessio** [38:51]
Oh, we're not Google.

**Swix** [38:53]
You're going to be the Google of AI.

**Alessio** [38:55]
No, no, we're definitely not Google. But I was just saying in terms of like if you look at like the style of companies that came out.

**Swix** [39:00]
Yeah, no, absolutely.

**Alessio** [39:01]
Or maybe we should just say.

**Swix** [39:02]
Live in the coming years. Live in the future. Yeah.

**Alessio** [39:04]
I think that's the pitch.

**Swix** [39:05]
Okay. Okay. Thanks for pitching us.

So you just mentioned the older vector embedding stores are kind of not made for the LLM generation of compute size. What does LLM ops look like? You know, which pieces need to be drastically different? Which ones can we recycle?

**Alessio** [39:22]
One of the things that we've found like in our own thing of building Codeium that's been just shows how much is missing. And this is the thing where like I don't know how much of this you can really outsource, which is like we needed to build eval infrastructure.

That means how do you build great code eval? And there are things online like human eval,right? And I was telling Sean about this. The idea of human eval is really neat for code. The idea is you provide a bunch of functions with doc strings.

And the eval, instead of being, "Did you predict next token?" It's like, "Did you generate the entire function? And does the function run correctly against a bunch of unit tests?" Right? And we've built more sophisticated evals to work on many languages, to work on more variety of code bases.

One of the issues that ends up coming up with things like human eval is contamination because a lot of these things that train models end up training on all of GitHub. GitHub itself has human eval. So they end up training on that and then the numbers are arbitrary.

**Swix** [40:16]
It's tiny, though. It's going to be tiny,right?

**Alessio** [40:17]
But it doesn't matter if it's tiny because it'll just remember it. It'll remember that it's not that it's that precise, but it will. It's like, it's basically like mixing your training and validation set.

**Swix** [40:27]
Yeah.

**Alessio** [40:28]
Yeah. But we've seen cases where like online where someone is like, "We have a code model that's like, we're like, we did this one thing and human eval jumped a ton." And we were just like, "Huh, did human eval get into your data set?

Is that really what happened there?" But we've needed to build all this eval. And what is shown is data cleaning is massive, but data cleaning looks different by workload. Like code data cleaning is different than what is a high-quality piece of code is probably different than what's a high-quality legal document.

And then on top of that, how do you eval this? How do you also train it at scale at whatever cost you really want to get? But those are things that the end user is either going to need to solve or someone else is going to need to solve for them.

And I guess maybe one of the things I'm a little bearish on is if another company comes out and solves eval properly for a bunch of different verticals, what was the company that they were selling to really doing?

What were they really doing at that point? If they themselves were not evaling for their own workload and all these other things, I think there are cases where, let's say for code, where we probably couldn't outsource our eval.

Like we wouldn't be able to ship models internally if we didn't know how to eval. But it's clear that there's a lot of different things that people need to take. Like, hey, maybe there's an embedding piece. How large does this embedding database actually need to be?

But hey, this does look very different than what classic MLOps probably did.

**Swix** [41:42]
How do you compare some of these models? Like when you're thinking about model upgrading and making changes, like what does the testing piece of it internally look like?

**Alessio** [41:51]
For us, it's like old school A/B testing. We built like infrastructure to be able to say, ramp up users from 1 to 10 to 50% and slowly roll things out. This is all classic software.

**Swix** [42:03]
Which you do in-house? You don't buy any services?

**Alessio** [42:05]
We don't buy services for that. There are good services, open source services that help.

**Swix** [42:09]
You just don't need them.

**Alessio** [42:10]
Yeah. I think that's just like not the most complicated thing for us, basically. Yeah. But I think in the future, maybe we'll, obviously we use things like Google Analytics and all this other stuff, but yeah, for things of ramping our models, finding out if they're actually better, because the eval also doesn't tell the whole story, because also for us, even before generating the prompt, we do a lot of work.

And the only way to know that it's really good across all the languages that our users need to tell us that it's actually good. And they tell us by accepting completions.

**Swix** [42:37]
So GitHub Copilot, the extension does this thing where they like they'll set a timer and then within like five minutes, 10 minutes, 20 minutes, they'll check in to see if the code is still there. I thought it was a pretty creative way.

**Alessio** [42:48]
It's a very, it's honestly a very creative way. We do do things to see like in the long term if people did accept or write things that are roughly similar.

**Swix** [42:56]
Because they could accept and then change their mind.

**Alessio** [42:57]
They could accept and then change their minds. So we are mindful of things like that. But for the most part, the most important metric is at the time, did they actually, did we generate value? And we want to know if that's true.

And it's kind of, it's honestly really hard to get signal unless you have like a non-trivial amount of usage. Non-trivial meaning you're getting, you're doing hundreds of thousands of completions, if not millions of completions. That sounds like, oh wow, like that's like a very small amount, but like it's classic, maybe like if you look at like when I used to be an intern at Quora, like, you know, now more than seven, eight years ago, and when I was there, I like shipped a change and then Quora had like millions of daily actives and then it looked like it was good.

And then a week later, it was just like way worse. And how is this possible? Like in a given hour, we get like hundreds of thousands of interactions, just like, no, you just need way more data. So this is like one of those things where I think having users is like genuinely very valuable to us, basically.

**Swix** [43:47]
Users is all you need.

**Alessio** [43:49]
Yeah.

**Swix** [43:51]
By the way, since you brought up Quora, have you tried Poe? Any thoughts on Poe?

**Alessio** [43:55]
I have not actually tried Poe. I have not actually tried Poe.

**Swix** [43:57]
I mean, it seems like a question answering website that's been around for 20 years or something would be very good at question answering.

**Alessio** [44:03]
Yeah. Also, Adam, the CEO, is like incredibly brilliant. That guy is like insanely smart. So I'm sure they're going to do great things.

**Swix** [44:10]
They may accidentally build the perfect like data collection company for QA.

**Alessio** [44:14]
Yeah. It takes a certain kind of person to go and like cannibalize your original company. Like in some sense, it was kind of stagnant for like a few years.

**Swix** [44:21]
Yeah, that's probably true. That's probably true. The observation is I feel like you have a bias towards domain-specific models, whereas most research is skewed towards general models, general purpose models. I don't know if there's like a deeper insight here that you want to go into or not, but like train on all the things, get all the data, and you're like, no, no, no, everyone needs like customized per-task data set.

**Alessio** [44:41]
Yeah, I think I'm not going to say that general intelligence is not good. You want a base model that's still really good, and that's probably trained on normal text, like a lot of different content. But I think probably one thing that old school machine learning, even though I'm like the kind of person that says a lot of old school machine learning is just going to die, is that training on a high-quality data set for your workload is always going to yield better results and more predictable results.

And I think we are under no illusions that that's not the case, basically.

**Swix** [45:10]
And then the other observation is bandwidth and connectivity, which is not something that people usually think about, but apparently is a big deal. Apparently training or grading descent asynchronous needs high GPU coordination. These are deleted notes from Sam Altman talking about how they think about training.

And I was like, oh yeah, that's an insight. And you guys have the same thing.

**Alessio** [45:26]
Yeah. So I guess for training, you'reright in that it is actually nuts to think about how insane the networks are for NVIDIA's most recent hardware. It's like for the H100 boxes, you shove eight of these H100s on a machine between two nodes.

The bandwidth is 3,200 gigabits a second, so 400 gigabytes a second between machines. That's like nuts. When you just sit and think about it, that's like double the memory bandwidth of what a CPU has, but it's like between two machines.

On top of that, within the machine, they've created this fabric called NVLink that allows you to communicate atultra-low latency that's even lower than PCIe, if you're familiar. That's like the communication protocol between, yeah, between like the CPU and the other devices or other PCIe devices.

And all of this is to make sure that reductions are fast, low latency, and you don't need to think about it. And that's because like a lot of deep learning has sort of evolved. Training has evolved to be synchronous.

In the OG days, there was a lot of analysis in terms of how good is asynchronous training, which is like, hey, I have a node, it has a current state of the model, it's going to update that itself locally, and it'll like every once in a while go to another machine and update the weights.

But I think like everyone has converged to synchronous. And I'm not exactly sure. There's not a lot of good research on asynchronous trainingright now, or maybe there is and I haven't read it. It's just that there isn't as much research because people are just like, oh, synchronous works.

And the hardware is continually up-leveled to handle that.

**Swix** [46:50]
Yeah. It was just unintuative to me because like the whole purpose of GPUs, you could train things, a lot of things in parallel.

**Alessio** [46:55]
Yes. But the crazy thing is also maybe I can give some dumb math here, which is that let's go with GPT-3, which is like 170 billion parameters. They optimize their state. So while you're training is 14 times the size of the model.

So in this case, if it's like 170 billion parameters, it's probably, I'm not great at mental math here, but that's probably around 2.5 terabytes to just store the optimizer state. So that has got to be sharded across a lot of machines.

Like that is not a single GPU. Even if you take an H100 with 80 gigs to just shard that much, that's like 40, at least 30 machines. So there's like something there where these things need to communicate with each other too.

**Swix** [47:33]
You need to vertically scale horizontally.

**Alessio** [47:36]
Yeah. You got to somehow feel like you have this massive, the ideal programming paradigm is you feel like you have this massive GPU that's like that has no communication, you know, overhead at all, but it has like infinite compute and infinite memory bandwidth.

**Swix** [47:49]
That's the OpenAI cluster. Okay. Well, we want to head to the questions.

So favorite AI product that you're not building.

**Alessio** [47:57]
Yeah. I'm friends with some of the folks at Midjourney, and I really think the Midjourney product is super cool, especially seeing how the team is iterating and the quality of generations consistently gets up-leveled. I think it's like quite neat.

And I think internally at Exafunction, we've been trying out Midjourney for like random content to like generate images and stuff.

**Swix** [48:14]
Does it bother you that they have like a style? Like, I don't know. It seems like they're hedging themselves into a particular, like, you want Midjourney art, you go there. It's a brand of art.

**Alessio** [48:22]
Yeah, you'reright. I think they do have a style, but it seems more predictably good for that style.

**Swix** [48:27]
Okay. So just get good at a domain-specific thing.

**Alessio** [48:30]
Yeah. Maybe I'm just selling, talking to our bookright now.

**Swix** [48:33]
Next question. Favorite AI people and communities.

**Alessio** [48:36]
Yeah. So I think I mentioned this before, but I think obviously the OpenAI, the OpenAI folks are insane. Like we only have respect for them. But beyond that, I think Aluther is a pretty special group, especially it's been now probably more than a year and a half since they released like GPTJ, which was like back when open source GPT-3 Curie, which was comparable.

And it wasn't like a model where like it wasn't good. It was like comparable in terms of perplexity to GPT-3 Curie. And it was trained by a university student, actually. And it just showed that, you know, in the end, like I would say pedigree is great, but if you have people that are motivated, know how computers work, and they're willing to just get their hands dirty, you can do crazy things.

And that was a crazy project. That gave me more hope in like decentral training being potentially pretty massive. But I think that was like a very cool thing where a bunch of people just got on Discord and were chatting and they were able to just turn this off.

**Swix** [49:29]
Yeah. I did not know this until I looked further into Aluther, but it was not a formal organization. It wasn't a company. It wasn't a startup. A bunch of guys on Discord.

**Alessio** [49:36]
They got a TPU research grant and they somehow just wrote some code.

**Swix** [49:39]
Yeah. Yeah. Listened to a podcast with Connor, who's the person, and basically OpenAI at the time was like, we cannot release GPT because it's like too good and so bad. And he was like, he actually said he was sick, so he couldn't leave home for like a few weeks.

So it was like, what else am I going to do? And ended up getting through the Google like research program through his university. And they were like, oh, we'll give you TPUs. And he was like, cool. And that's how.

**Alessio** [50:05]
That's amazing. I love the story. It's a great story.

**Swix** [50:09]
So a year from now, what do you think people will be most surprised by in AI?

**Alessio** [50:13]
Yeah, I think the thing people will be most surprised by is I think the models are going to get more good at special tasks for sure, but even the existing models, I think people will come up with more creative ways of leveraging them to build like world-class products.

I think that's just like human creativity is going to go wild. It seems like ChatGPT has already kind of unleashed that. Yeah, I think I'm just excited to see what the future of these products look like. I guess law was not something I expected in such a short.

**Swix** [50:38]
Totally expected. I was actually watching a different company that I thought was going to be the winner and it hardly just came out of nowhere.

**Alessio** [50:44]
Oh wow. Okay. Okay. Well, that's awesome.

**Swix** [50:46]
But yeah, so my takeaway from what you're saying is like foundational models have kind of shot way too far ahead of the apps and people need to build apps.

**Alessio** [50:53]
Yes. I think people should be building apps, but I think the reality is the model is like probably at a stateright now where it can do crazy enough things. And I think great apps will come out of this.

Yeah.

**Swix** [51:03]
AI thing you would pay for if someone else built it, personal or work.

**Alessio** [51:07]
I think if someone else built like a proper assistant, like a proper like fitness assistant, I would probably pay for that actually. I know that that sounds weird, but someone that actually tells me like, how should I end up like, you know, doing fitness today?

I ended up injuring my knee from overbiking. I ended up biking like 150 miles a week and I ended up just injuring my knee out of nowhere.

**Swix** [51:27]
So you need an app to tell you to exercise less.

**Alessio** [51:29]
Exercise less, but tell me what my training regimen is. Tell me what I should do to prepare for things. I know that this is like a big niche, but I think the fact that Strava is such a big group of people and like Zwift is a big group of people seems to suggest that I think a lot of people would be willing to pay for something like this.

Yeah.

**Swix** [51:44]
What's one thing you want everyone to take away about AI and our conversation?

**Alessio** [51:49]
Probably the most important thing to take away is there's probably a lot out there if people continue to tinker. I think that's probably like the biggest takeaway I've had. And it's, you know, being a pure infrastructure company, I think like six to eight months ago, I think it was like very hard to watch everyone tinkering and us just, you know, building infrastructure.

But I think there's going to be some crazy things to come out over the next year or two. Excited to just see what that looks like.

**Swix** [52:13]
Awesome. Yeah, man. That's it.

**Alessio** [52:15]
This was fantastic.

**Swix** [52:16]
Thanks so much. Thanks for coming.

**Scott Wu** [52:42]
Hey, I'm Scott from Cognition AI, and today I'm really excited to introduce you to Devin, the first AI software engineer. Let me show you an example of Devin in action.

### Devin Launch

**Scott Wu** [52:53]
I'm going to ask Devin to benchmark the performance of Llama on a couple of different API providers. From now on, Devin is in the driver's seat. First, Devin makes a step-by-step plan of how to tackle the problem.

After that, it builds the whole project using all the same tools that a human software engineer would use. Devin has its own command line, its own code editor, and even its own browser. In this case, Devin decides to use the browser to pull up API documentation so that it can read up and learn how to plug into each of these APIs.

Here, Devin runs into an unexpected error.

Devin actually decides to add a debugging print statement, reruns the code with the debugging print statement, and then uses the error in the logs to figure out how to fix the bug.

Finally, Devin decides to build and deploy a website with full styling as the visualization.

You can see the website here.

All of this is possible today because of the advancements that we've made in both reasoning and long-term planning. It's a really hard problem, and we've only just started, but we're super excited about the progress that we've made so far.

In the meantime, if you'd like to try out Devin on your own real-world tasks, send us a request below and we'd be happy to forward it to Devin.

**Kevin Hou** [54:31]
So my name is Kevin, and I'm going to be talking about how embeddings are stunting AI agents. So I'm going to let you in on some secrets about how we build the product and exactly what we're doing behind the scenes to improve your code gen experience.

### Context Retrieval

**Kevin Hou** [54:44]
So at Codeium, we are building AI developer tools, and we're starting with an IDE plugin. And as mentioned before, we've been downloaded over a million and a half times. We're one of the top-rated extensions across the different marketplaces.

And to reiterate, we offer free, unlimited autocomplete, chat, and search across 70 different languages and 40 different IDEs. So we plug into all the popular IDEs. We are the highest-rated developer tool as voted by developers in the most recent Stack Overflow survey.

And you'll note that this is even higher than tools like ChatGPT and GitHub Copilot. And importantly, we are trusted by Fortune 500s to deliver high-quality code that actually makes it into production. And we do this with top-grade security, licensing, attribution for some of the largest enterprises on the planet.

Our goal at Codeium is to empower every developer to have superpowers both inside of the IDE and beyond. And today, I'm going to let you in on some secrets about how we've been able to build a tool like this and why users choose us over the other AI tools on the market.

And the short answer is context awareness. So here's a quick overview about what context looks like today. We're all familiar since we're at an AI conference with the basics of retrieval augmented generation, the idea being that a user puts in a query, you accumulate context from a variety of different sources, you throw it into your LLM, and then you get a response, whether that be a code generation or a chat message.

Here's a concrete example about how retrieval can be used in code generation. So let's say we want to build a contact form in React. Now, you could go to ChatGPT, you could ask it to generate a contact form, but in reality, on a moderately large code base, this is really not going to work.

It's not going to give you things that are personalized to you. And this is really where context retrieval comes in. We need to build a contact form that, you know, is in line with our design system components. Let's say you already have buttons and inputs.

It has to be able to pattern match with local instances of other forms inside of your code base. It has to ingest your style guide. For example, if you're using Tailwind, you have to be able to detect and make the form look and feel like every other thing on your site.

And then, of course, there's documentation both locally and externally for packages and other dependencies. So the question becomes, how do you collect and rank these items so that our code generation can be both fast and accurate for your use case?

So to dive into a couple of different methods about how people are tackling this today, there's really three main pillars. The first one is long context. So this is the idea that if you expand your prompt window in your LLM, it can read more input and therefore be a bit more personal to what you're trying to generate.

This is very ergonomically easy to use,right? You just shove more items into your prompt. But this comes at the cost of latency and financial cost. So one of the most recent examples was Gemini. Gemini actually takes 36 seconds to ingest 325K tokens.

To put this into perspective, moderately sized or even small repo is easily over 1 million tokens. And that accounts for about 100K lines of code. So in this instance, most enterprises have over a billion tokens of code. It's simply not feasible to be throwing everything into a long context model.

The second method is fine-tuning. So for those that are familiar, fine-tuning is the idea of actually tweaking the weights of your model to reflect the distribution of the data that your consumer expects,right? And so this requires continuous updates.

It's rather expensive computationally. You have to have one model per customer, and it's honestly prohibitively expensive for most applications. And finally, we have embeddings. And for all of you, hopefully you're familiar, this is a relatively proven technology today.

It's pretty inexpensive to compute and store. But the difficulty that we're about to dive into is that it is hard to reason over multiple items. It also has a low dimensional space, and I'll talk about that shortly. So to dive deeper into embeddings, the whole concept is that you take your objects, you throw it through an embedding model, and then you end up with some sort of vector, some sort of array of numerical values, and this is in a fixed dimension.

And so by mapping and chunking code, we can map it to an embedding, and that allows us to quickly search over our functions, our documents, whatever you decide to chunk by. And this is what embedding search is called.

Embedding search, like I said, is not a new concept. There's a bunch of models that have tried to optimize. And in this example, we're looking at one of the kind of North Star eval benchmarks. It's become increasingly popular.

And the question becomes, how do we fit millions of lines of code into an LLM model so that we can actually generate useful results? And so it's evident through the years that we're actually hitting a ceiling on what is possible using these traditional vector embeddings.

And over time, even the biggest models are approximating to around the same level of performance. As you can see, everything's kind of within plus or minus five. At Codeium, we kind of believe that this is because fundamentally, we cannot distill all the dimension space of all possible questions, all possible English queries down into the embedding dimension space that our vectors are going to occupy.

And so at Codeium, we've thought very critically about what retrieval matters to us. Are we measuring theright things? And does semantic distance between these vectors really equate to things like function relevance in the concrete example that I showed earlier?

And so what we landed on is that benchmarks like the one that I showed you before heavily skew towards this idea of needle in a haystack. It's the idea that you can sift through a corpus of text and find some instance of something that is relevant to you.

Note, it is only one single needle. So in reality, code search requires multiple different needles,right? We showed that slide earlier. When you're building a contact form, you need all these different things in order to actually have a good generation.

And these benchmarks really don't touch that. And so we decided to use a different metric, and it's called recall 50. The idea and its definition is that it's what fraction of your ground truth is in the top 50 items retrieved.

So the idea being now we have multiple documents, and we're now looking at the top 50 documents that we retrieved. How many of those are part of our ground truth set? So this is really helpful for understanding document, multi-document context, especially again for those large, large code bases.

And now we actually have to build a data set around this. And so this is where we did a little bit of magic. We wanted to make the eval as close as possible to our end user distribution. So we had to compile our own data set.

So what we did, this is a PR that I put out a few months ago. We looked at PRs like this. It's broken down into commits. Those commits, we can extract and actually match them with the modified files,right?

So now we have this mapping from something in English to a list of files that are relevant to that change. And you can imagine we can hash this in many different ways. But ultimately, the point I'm trying to make is we are creating an eval set that mimics our production usage of something like a code gen product.

And so this message serves as the backing for this new type of eval, where now we can run at scale this idea of product-led benchmarks. It gets us closer to the ground truth of what our users are actually experiencing and what retrieval tweaks in retrieval actually mean to the end product.

And so we threw some of the currently publicly available models at this notion of retrieval, this idea of using commit messages. And we found that there is reduced performance. They're unable to reason over specifically code, but then also specifically this kind of real-world notion of English and commits,right?

And so at Codeium, we've been able to actually break through this ceiling. This is something that we've worked very hard at. We have to redefine exactly how we are approaching retrieval in order to be kind of in our class of our own so that when you are typing in your IDE, when you're chatting with our assistant, when you're generating autocompletes, we're retrieving the most relevant things that are for your intents.

So now the question becomes, how do we actually get this kind of best-in-class retrieval? And so I'm here to give you the very short and sweet answer, which is we throw more compute at it,right? But of course, that can't come with absurd cost,right?

Financial cost. So how do we do this actually? In production, how do we actually do this without recurring an unreasonable cost? And so this goes back to a little bit of Codeium secret sauce,right? We are vertically integrated. And what this means is that we train our own models.

So number one, we train our own models. This means that these are custom to our own workflows. So when you're using our product, you're touching Codeium's models. Number two, we build our own custom infrastructure. This is actually a very important point and actually connects to the whole Exafunction to Codeium pivot that we discussed earlier.

Exafunction was an ML infrastructure company. And so what we've been able to do is build our own custom infrastructure down to the metal. This means that our speed and efficiency is unmatched by any other competitor on the market so that we can serve more completions at a cheaper cost.

And finally, we are product-driven, not research-driven. Now, what this means is we look at things like actual end user results. When we actually ship a feature, we're looking at real-world usage, and we're always thinking about how does this impact the end user experience, not just some local benchmark tweaking.

And so we could spend all day talking about kind of why Codeium has done this and yada yada, but that's a talk for a different time. So I'm going to talk about something that I find very cool, and this is the reason why we've taken this vertical integration approach and been able to turn it into something that we call MQuery.

So MQuery is this way of taking your query, so similar, it's that idea of taking your retrieval query. You have your code base, and let's just say you have N different items. And because we own our own infrastructure and train our own models, we're now making parallel calls to an LLM to actually reason over each one of those items.

We're not looking at vectors. We're not looking at small dimension space. We're literally taking models and running them on each one of those items so that you can ensure, you can imagine, you run ChatGPT and tell it to say yes or no on an item, for example.

That is going to give you the highest quality, highest dimension space of reasoning. This leads into very, very high confidence ranking that we can then take into account things like your active files, your neighboring directories, your most recent commits.

You know, what is the ticket that you're working on currently? And we can compile all this to give you the top N documents that are relevant for your generation so that we can start streaming in higher quality generations, higher quality chat messages, things of that nature.

And the reason behind this is, again, it's that vertical integration. It's that idea that our computation is 1/100th of the cost of the competitors. We are not using APIs. And as a result, our customers and our users actually get 100X the amount of compute that they would on another product.

And so we're willing to do that. We're willing to spend more compute per user because it leads to a better experience.

And so, like I mentioned earlier, I lead our product engineering team. So we always want to anchor ourselves around these three different things. One, that we have to build a performant product. It has to be really fast. For those of you that have used the product, you can probably attest to this.

MQuery runs thousands of LLMs in parallel so that the user can start streaming in code within seconds, not minutes, not hours, seconds, and oftentimes milliseconds. It has to be powerful,right? None of this matters if the actual quality and the actual generations that you're building are wrong,right?

And finally, it has to be easy to use. We're building an end user product for people today that's in the IDE. Tomorrow, it might not be in the IDE. How do we actually build something that is intuitive to understand that people can grapple with and see exactly what my model is thinking?

And so because we have the benefit of distribution, we were able to roll this out to a small percentage of our users. And by small percentage, we're dealing in the order of million plus downloads. This actually reached the surprising number of people.

And what we've been able to see is that we were able to successfully reason over these thousands of files in people's mono repos and people's remote repos and select what was relevant,right? We can very accurately deem which files are relevant for the generation that you're trying to have.

And the result, as you can see, this is a real-time GIF, is both fast and accurate. So I'm asking for usage of an alert dialog. It's going through, and I think I've panned down here. This is kind of a ShadCN component that I've modified internally.

We're pulling in basically the source code of what is relevant for our generation. And ultimately, the results of this experiment with that users were happy. They had more thumbs up on chat messages. They were accepting more generations. And we were able to see that ultimately, we were writing more code for the user, which is the ultimate goal.

It's that idea of how much value are we providing to our end users? And so we built this context engine,right? This idea of MQuery, this idea of ingesting context and deciding what is relevant to your query to give you coding superpowers.

And so our users will generate today, they're generating autocompletes. They're generating chats, search messages. But in the future, they're going to generate documentation. They're going to generate commit messages, code reviews, code scanning. They're going to take Figma artboards and convert them into UIs that were built by your own components.

The possibilities are endless. But what it starts with is this bedrock, this very hard problem of retrieval. And it brings us to, again, one of the reasons why Codeium is approaching this problem a little bit differently. Our iteration cycle starts with product-driven data and eval.

So we're starting with the end problem. We're building a product for millions of people. How do we start with what they're asking for? And how do we build a data set and eval system locally so that we can iterate on the metrics that matter?

Secondly, because we're vertically integrated, we're taking that massive amount of compute, and we're going to throw it at our users. Paying or not paying, we're going to throw it out at our users so that they can get the best product experience and the highest quality results.

And then finally, we're actually going to be able to push this out to our users in real time, overnight, and be able to get a pulse check on how this is going. This is what we did for MQuery.

And when we evaluate in production, we can say, thumbs up, thumbs down, and then hit the drawing board again, back to that same cycle, repetition. And so you can start seeing how these pieces of compounding technology come together,right?

We've alluded to some of them today, modeling, infrastructure, being able to retrieve. But then it also includes things like AST parsing, indexing massive amounts of repos, knowledge graphs, parsing documentation, looking at websites online. The list can go on and on and on.

But we're confident that we're solving these problems one piece at a time using that same iteration cycle, that same idea that we're going to take the distribution and knowledge that we have and that additional compute that we're willing to afford each user to solve each one of these puzzle pieces.

And I want to leave you with a parallel analogy. So in my past life, I had experience in the autonomous driving industry. So to bring over a metaphor from that industry, in 2015, TechCrunch boldly predicted that that was going to be the year of the self-driving vehicle.

It was largely, now we're in 2024, so we can look back in hindsight, largely untrue,right? We were doing things like sensor fusion. We were decreasing our polling rates. We were running offboard models, all this in the effort of making heuristics that would compensate for the lack of compute that was available because consumer graphics cards were not as popular or not as powerful as they are today.

Fast forward today, we're seeing 100X the amount of compute available to a vehicle. You can take a Waymo around San Francisco, which I encourage you to do. It's a wonderful experience. But that means that we're actually able to throw larger models at these problems,right?

More sensors, higher frequency. And now, 2024, TechCrunch has released another article that said, will 2024 finally be the year of the self-driving vehicle? And we can now look at this pattern and say, driving performance was substantially better by throwing larger models, being able to handle more and more data.

And so at Codeium, we believe that this embedding-based retrieval is the heuristic. We should be planning for AI-first products, throwing large models at these problems so that AI is a first-class citizen. We're planning for the future. And finally, we also believe that ideas are cheap.

I could sit up here and tell you all these different ideas about how we're going to transform coding and the way that the theory behind possible solutions. But what we believe at Codeium is that actually shipping, actually showcasing this technology through a product is the best way to go.

And so if you agree with these beliefs, you can come join our team. We're based in San Francisco. And you can download our extension. It's free. I'm not, obviously, what's it called? I'm not advertising the core product nearly as much.

We're kind of talking about the technology. But you can experience this technology firsthand today by downloading our extension. It's available on all the different plugins, VS Code, JetBrains, Vim, Emacs. And you can see how this infrastructure and the way that we've approached product development has shaped the experience for you as a user.

And then, of course, you can reach out to me on Twitter. I put my handle up there. I'll be kind of floating around outside. So if you have other questions or are interested in what I had to say.

But I hope that you learned something today. I hope that you use Codeium, you try it out, and see what the magic can do for yourself. Thank you.

### Devin Workflows

**Swix** [1:12:14]
I'm Scott from Cognition AI, and I'm going to tell you guys a little bit about the early makings of Devin. We're still super, super early on, and also a little bit about kind of the space as a whole and what's coming next.

I thought it'd be nice to start with a demo first. It sounds like some of you guys have already seen some of the videos, but I brought a nice custom one here for the World's Fair today. So I'll just show that quickly.

And here I basically said, "Hey, Devin," this was this morning, by the way. This is a huge scramble. I said, "Hey, Devin, I want you to build a mobile-friendly website to play the name game." So I have a lot of trouble memorizing names and faces.

I don't know about you guys. But I basically just said, "Here's a TSV file of a bunch of names and faces. These are all the speakers here at the World's Fair this week." And I said, "Can you set up the game so that you show two different random faces and then show the names of one of them and have me guess which one is which,right?"

And I gave kind of a few instructions on how the game should work. And so Devin is a fully autonomous software engineer. And what that means is Devin has access to all the same tools that a human software engineer would have when they're building something like this.

And so the first thing that Devin's going to do is Devin's going to make a plan. And you can see here kind of a basic plan coming out. One of the interesting things about this is the plan changes a lot over time.

And so as you get new information or new feedback, you update your plan according with that too. After that, Devin's basically just running this the same way that a human would. And so if you can take a look, Devin makes a new directory for the name game website, starts a new React app, all the same primitives, works on building it out and building the code, reads the TSV file to take a look at what's going on here.

And it's just kind of generally working through it and jumping through. It comes out and deploys this first version after some minutes. And I'll just pull this up quickly. So that's what this looks like. It's close, but not quite there,right?

I mean, it's still showing the names. And I think maybe I didn't quite specify that exactly. But you can click the name and got that correct. And so I just went ahead and just gave it some more feedback in plain English.

And so I said, "Hey, can you hide the two names until I click on the answer? And also, can you probably restyle the play again button?" It's like somehow it's a little off on this page. And I kept going and just kind of gave it more and more feedback over time.

And I also asked it, "Hey, can you add a streak counter as well? Can you keep track of how many I got correct and reset to zero?" A few of these other things. And the website it ultimately deployed was this oneright here.

And so this is Justine, for example, keeps track of my streak. And you can see it's kind of ramping it up. And so if I were to, for example, if I got this one wrong on purpose, then you would see the streak would reset to zero and it would go on.

And so I actually played this game and learned the names of everyone, which was super helpful, by the way. And you guys can play it too. It'sright here if you want to try it out. This has all the speakers.

I think it was something like 170 speakers here at the World's Fair this week. So this is kind of a cool example. But I want to highlight how different the world is if software engineering is just this easy, if you can just explain exactly what you want in plain English and get that out.

And so this is obviously kind of a toy use case, and it's perhaps useful. But we use Devin all the time ourselves when we're building Devin, actually. And by the way, obviously, I didn't make this website myself. I just said, "Hey, can you build me this website with the QR code and whatever?"

And Devin built that too. But here's a quick example of Devin that we're using ourselves in production. And so if you take a quick look here, for example, there's this whole search bar, and there's all the sessions, and you can search across sessions,right?

Devin actually made that in the Devin repository. You can see here, Bryce is on our team. And Bryce was asking, "Hey, Devin, can you go into the Devin sessions list, create a search bar component? Here's what I need you to do."

And so there's a few features about this in particular that are obviously tuned for working in a production codebase. You can see here that Devin started from a snapshot. So we have a machine instance loaded where it's cloned from.

It has a playbook, so it knows a lot of the details about our repositories. And then it's also just able to generally work within our Git environment. So you'll see it just make a PR and interact with all those same tools.

And so I'll just kind of go through this quickly. So yeah, Devin says, "Absolutely." Makes the first pull request. Bryce continues. And again, you're just giving feedback in plain English,right? And you say, "Hey, this is a great start.

Now, could you add a magnifying glass and make it idiomatic? Use phosphor, Lucy. It's up to you,right?" And Devin says, "Yeah, sure. I'll build that." And Bryce says, "Oh, by the way, no need to test. I trust you."

And Devin says, "By the way, I'm dealing with a bit of an issue with the login process." It's just like you're working with another engineer,right? And Bryce says, "OK, bro." And it kind of builds it all and gets the PR.

And this PR was actually merged, and this is the search bar,right? And similarly, a lot of the API integrations that Devin has were built by Devin. A lot of our own internal dashboards and metrics tracking within Devin were actually also built by Devin.

And it's been kind of a fun one to see Devin building the company with the company as well. So cool. Yeah, I want to talk a little bit about our journey so far and about what's happening in the space as well.

And so we got started back in November. So it's been about seven months now. It's kind of funny. We started in a hacker house in Burlingame. And it was basically just a lot of us had already lived together at that point.

We'd all had our own journeys in AI, and we just knew that we wanted to build something together. And we obviously knew that we wanted to do something in code and build a coding agent. And then that hacker house in the Bay Area.

After that, there was another hacker house in New York. Then there was another hacker house in the Bay Area. So we've been going back and forth between New York and the Bay Area for basically the last seven months.

I think at this point, we are now going to settle in the Bay. But it's been going back and forth and getting a slightly bigger Airbnb each time because the team also gets a little bit bigger. But why Devin in particular?

And this is a particular question that I'm really passionate about, which is language models have been pretty big. I think that's fair to say. And the first wave of generative AI is what I generally call these text completion products,right?

And that makes a lot of natural sense if you think about it, that obviously the interface of a language model is text completion,right? You give it a prefix, and it completes the suffix from there. And so if you think about ChatGPT, if you think about a lot of these Q&A products, if you think about writing marketing copy or answering customer support or even GitHub Copilot and Cursor and products like that, obviously, a lot of these are really great products and very natural use case where you have the prefix so far, and you're asking the model to complete what's next in the suffix,right?

And it does that for you. And that's a tool that's useful,right? And I think we're entering this new wave where we're going beyond that and actually introducing some amount of autonomous decision-making. And obviously, that's typically referred to in our space as agents,right?

And there's all sorts of new things that you unlock,right? There's a lot higher bar of consistency that you require, but there's new things that you unlock with that. And so it's been an interesting one because it's both a very deep core capabilities question of getting Devin to solve these, but also a pretty interesting product design problem because I think the UX of agents is something that's extremely new.

And then why code in particular? A few different things. Obviously, we're all coding nerds as well. We're all engineers. And so the idea of teaching AI to code is one of the coolest things that we could think of.

But beyond that, I think there's a few particular reasons that code with agents works especially well. One is that obviously, there's so much more to being a software engineer than typing the code,right? A lot of the work that you're going to do is you're going to be looking into a bug.

You're going to be looking at the different files of the codebase. Maybe you're going to be running this or that command. Maybe you're going to be pulling up documentation. Maybe you're going to run the front end yourself to reproduce the bug.

You look at this thing. You make this edit. You try it again. All of this work here, obviously, that's what software engineering is,right, more so than just typing the code in the file, which leads very naturally to an agentic workflow.

Another part, which I think is closely related, is the ability to iterate with code feedback. And so what I mean by that is if you were given an entire production codebase and you were told, "Hey, this has this one bug.

I need you to fix it. Here's the bug." And let's say it's like thousands of files and hundreds of thousands of lines of code. I mean, it'd be pretty tough, honestly, for most humans. It's also going to be quite tough for AIs as well.

And obviously, the way that we do this in practice is you go and add print statements. You pull up the logs. You check the monitoring. You jump back and forth between different files. You try and diagnose it,right? Each of these things that you're doing, you're making a decision, and then you're running actual code to find out what happened.

And from that, you're able to iterate. And it just gives you a much cleaner path to solve the problem in front of you. And similarly, that kind of lends very well to agents. And the last thing I just want to mention is how fast model agentic capabilities are improving.

And so two years ago, even something as simple as this name game demo, I think, would have been almost unthinkable. And you think about where things are going and where things are going to be two years from now.

I think there's a lot of the data, theright training, and so on that's really, really rapidly improving in the space.

And then again, beyond the capabilities problem, there's actually a really deep UX problem as well. And at a high level, I think what's kind of happening here is when we're building agents, and I think all of us in the space are quite new to agents, the immediate first things, I think, to map to are how we use software today and also how we talk with other humans,right?

And so even a lot of the features in Devin are essentially looking over your own intern's shoulder. You can see their computer, and you can see what commands they're running and things like that. The thing is, I think an agent is actually pretty different from both.

There's a lot of nuances and details of parallel work, information gathering, how it manages context, et cetera, et cetera, that are super, super different. And it's actually a quite deep problem from a product perspective as well. And just to give you guys a bit of a sense of that, here's just kind of a short list of some of the features that we've built into the product.

And so obviously, there's Devin being able to use the shell, edit code, browse the web. But there's all these other things,right? Being able to fork and roll back sessions, being able to handle integrations with Slack and GitHub, being able to handle playbooks to store machine snapshots, to keep track of secrets, to be able to work with theright tools for verification.

All of this is part of the actual product iteration,right, which is on its own, I think, already an incredibly, incredibly dense problem. And I think, honestly, we're going to see actually a lot more iteration with that over time.

And I just wanted to show kind of a new feature which we just recently shipped, which is the ability to use Devin's machine, which is kind of, again, it's the kind of thing that's not always there's not necessarily a very close parallel in the software that we have today,right?

But the ability to just have a VS Code live share in Devin's machine. And if you want to collaborate with Devin and say, "Hey, oh, there's these couple of lines. You should make this edit. I went ahead and did that edit for you."

And you can just talk with Devin and do that,right? So there's a lot more room to go and a lot to iterate on in the space. One of the other things I wanted to mention too is just how much we've seen it changing our own workflow.

And so you guys saw a simple example of Devin building the search bar. But we actually handle tasks in a much more async way now. One of the cool kind of features of Devin, I'd say, is if as an engineer you're working on, let's say, four different tasks today, you just give one to Devin number one.

You give the second one to Devin number two. The third one to Devin number three. You have four Devins that are all running in parallel. And it's kind of turning every engineer into an engineering manager is almost how I describe it.

I think the Devins are very enthusiastic interns is what I'd say. I mean, they try very hard. Obviously, they don't know everything. They get little things wrong. They ask a lot of questions. But you're kind of working with each of them and having them iterate.

And so here's just kind of a fun example. I mean, this is literally from earlier today. But we were talking about some particular feature and about what we wanted to build. In this case, it was a pretty simple thing of changing the color.

But it's just as simple as just saying in Slack in the conversation, "Hey,@Devin, can you just change this thing?" And then Devin goes and makes the PR. And then you hit merge. And so we've had a lot of occasions where we're in the gym or in the car or something.

And now you can actually write code because you can tell Devin exactly what you want Devin to do. You just don't have your whole computer with you and can't type everything. But being able to just kind of describe what you want to Devin and then being able to review the code afterward actually works really well.

So what's next? I think this is a really important question. And obviously, I think the technology is extremely early today. But where do these things go in a few years? And also, what happens with software engineering? I think there's been a lot of uncertainty about that question.

And as we're using Devin more and more, I think one of the big things that we see actually is this is kind of obvious perhaps, but Devin is not the one that decides what to do or what to build.

And there's this core part of software engineering, the way I describe it, is like software engineers everywhere are doing really two jobs at once,right? And the first job is basically problem solving with code. You're given a problem, and you're breaking down exactly what is the solution you're going to build.

What is the architecture that you're going to use? What are all the flows and the details and the edge cases that might come up? And kind of architecting your exact solution. And then the second part is once you have that, you're dealing with debugging or implementing different functions or writing unit tests or all of the other things that kind of go into this implementation of something that you know you want to do,right?

And I thinkright now, the average software engineer is probably spending like 10% or 20% of the time on that first thinking part and they're spending 80% or 90% of the time on that implementation part. And what we really see is Devin actually just frees you up to do more of the first part.

And I think the future of Devin, again, it's very, very early. But I think as Devin gets better, we're going to see more of that where Devin just frees up the implementation for you where you don't have to go figure out how to set up Kubernetes.

You don't have to go debug all these APIs that are broken. You don't have to go deal with version changes or migrations or all of these other things that take up a lot of time in software engineering,right? But you actually are spending all your time on figuring out how to solve the problems in front of you.

It's a little more like a mix between a technical architect and a product manager almost,right? And so I think software engineering, the job that we call software engineering, is going to change. But I think practically, there's actually going to be way more software engineers than ever.

And I think there's a lot of precedent for that too. Programming back then used to mean punch cards. And then after that, it used to mean assembly. And then after that, it used to mean C,right? And as these things have gone on, I mean, most people aren't using punch cards anymore.

But there's actually way more programmers than before,right? And I think one of the things that's easy to underestimate is just how much more code there is to write. And it's funny to think about, I think, because obviously, we all love software here in this room.

I would say I think software has been the number one driver of progress in the world in the last 40 or 50 years. And yet, despite that, I think our demand for software to be built is actually probably a lot more than 10X what we're currently getting.

And so I think what happens is we get to open up the power of software engineering to a lot more people. And every single software engineer gets to be 5 or 10X more effective. But we actually do a lot more software engineering.

Cool. Yeah. So that's all I had. But yeah, we'd love to open the floor if there's any questions.

Yeah,right here in the front.

**Alessio** [1:27:43]
Oh, I'll just show it now. Yeah, cool. Thanks again. I've moved my mind like 80 times. How do we get access immediately?

**Swix** [1:27:51]
Great question. Great question. So we've been ramping up access. Every week, we've been letting on more and more people. We've also been sizing up with our enterprise customers. We have a lot of wait lists to get through. So we're doing it as fast as we can.

But we'd love to get you guys access as soon as possible. Yeah. Yeah. All the way in the back over there. Yeah, in the red.

**Alessio** [1:28:12]
Hey. So I'm just wondering, does Devin have access to running the code? Well, from the back end, how are you doing that analysis of correctness of the code in the generation?

**Swix** [1:28:21]
Yeah, exactly. So in our codebase, for example, Devin has all the setup that it needs. It has a machine that's basically instantiated where it can run the dev environment. It can run the server. It can run the front end.

And so if you're asking it, "Hey, I need you to debug this particular thing," it'll just pull it up itself and then reproduce it. And then it'll debug it and try it again. Yeah.

**Alessio** [1:28:39]
So it's fully executable.

**Swix** [1:28:40]
Exactly. Yeah. Yeah. Yeah. Any other questions? Right here?

**Alessio** [1:28:47]
So with all of the synchronous potentially being solved by Devin's software, how do we not completely forget about all the junior interns that don't need to learn how to code,right?

**Swix** [1:29:01]
Yeah.

It would be good for you to repeat the question.

Oh, of course. Yeah. So sorry. So someone asked, with all of these simpler tasks getting solved, what happens to all the junior engineers or the interns who obviously need to learn how to code? I think what happens, honestly, is I think that demand is going to just keep rising with supply.

And I think the training process is going to change a little bit. But I think a lot of these core fundamentals of if you think of someone as when you say someone's a really great engineer, typically, you don't mean that they type really fast, although maybe they do that too,right?

You typically mean that they have a really great understanding of problems. They know all the different architectures. They never miss an edge case, stuff like that,right? And so those are the fundamentals that I think are always going to matter.

And I think basically, I think interns or junior engineers are going to get more exposed to getting to use those fundamentals earlier and earlier. Yeah.

**Alessio** [1:29:54]
Do you think that the biggest open technical challenges you have to realize this vision of a future suite?

**Swix** [1:30:00]
Yeah. OK. So someone asked, what are the biggest challenges to realizing the vision of a future suite? There's a lot. I mean, it's basically everything, as you can imagine. I mean, there's speed. There's consistency. There's access. There's integrations.

There's theright product UX. And I think all of these things, one of the cool things, I think, is just how much of a rising tide there is everywhere. And so obviously, we're going to do our best work on it.

But every new hardware release is amazing for us. Every new foundation model that comes out is amazing. Every new piece of agentic research. And I think this is the kind of thing where I think there will be a lot of different optimizations that come in different parts of the stack that make this agentic flow better and better and better.

But yeah, it won't just be one small thing. But I think it'll be pretty fast.

**Scott Wu** [1:31:04]
Hey. My name is Varun from Codium. And today, we're launching the Windsurf Editor. Think about the last time you felt magic with technology, the first ChatGPT question, the first smartphone interaction, or the very first internet search. Magic happens when the interaction with technology is intuitive, but it's also obvious that something super powerful is happening under the surface.

### Windsurf Launch

**Scott Wu** [1:31:22]
So what does true magic look like for software development? We believe it's an editor where you and the AI can truly mind-blow with information naturally and infinitely flowing between the two without any friction, giving you instant, invaluable AI developer assistance where you want it and when you want it.

And that's the core Windsurf Editor paradigm shift.

**Kevin Hou** [1:31:41]
Oh, yeah. I like it. This sounds super high ROI.

**Anshul R** [1:31:45]
Hi.

**Kevin Hou** [1:31:47]
Up high.

**Anshul R** [1:31:49]
Yeah, Varun.

**Kevin Hou** [1:31:50]
Keep going.

**Anshul R** [1:31:51]
Copilots are collaborative but can only work on small scopes. Agents can do larger tasks but are not collaborative. Both are very useful. But we realize that this real magic will happen when the AI has the ability to be both collaborative and independently powerful, all wrapped up in an intuitive interface.

That is an AI flow, which is what we think comes after AI agents. The core motivation of creating the Windsurf Editor was to natively expose flows, starting with Cascade, the flow evolution of chat. Flows create magic by giving the AI access to three things: knowledge, tools, and human actions.

Let's start with knowledge. Here is us asking Cascade to add docstrings to all integration tests in a production codebase. To do this well, Cascade needs to rip through all the code, identify the integration tests, understand what those tests are testing, and generate contextually relevant docstrings.

That's a lot of deep codebase understanding. Most attempts from other companies have normally degraded quickly as codebases get larger and more complex. But that's where our multiple years of creating powerful context awareness systems gives Cascade the edge. Next up is tools.

We have given Cascade access to tools like List Directory, Grep, Edit File, Add File, and more. This allows Cascade to be independent. But Cascade takes it further than any other agentic system by also generating commands to be run.

Here is Cascade installing a requirements.txt file and independently realizing that it needed to use Python 3 instead of Python given my environment. Or here is Cascade suggesting to move files. Don't worry. The human is in the loop approving the commands.

But Cascade can truly take action by utilizing the broadest set of tools. Finally is human action. This is where the collaborative aspect of flows really shines. Cascade lives in your editor. So if you make some changes in the text editor, as shown here, you can just ask Cascade to continue.

And it will pick upright where you left off. Cascade truly understands your intent because you and the AI are always operating on top of the same state of the world. We brought together knowledge, tools, and human actions to create the next generation of AI systems.

**Jeff Wang** [1:34:14]
From the first prototype we built, Cascade was already great at building small, zero-to-one apps. But our goal has always been to help developers with the most ambitious projects that they could dream of, the kind of thing that takes more than a weekend to build.

So we've invested a lot of effort into making it a better partner for when you're contributing to large, sophisticated codebases. When the engineers inside of Codium started to rely on Cascade to work on our own production codebase, we knew that we had something.

Cascade is a fundamental leap from what was possible before. And we're really excited to be able to share it with you now. No wait list necessary. Of course, this is just the beginning. We know that Cascade has the potential to do more, more impressive tasks on larger, more complex codebases, predicting more of your intent by looking ahead on the joint developer-AI timeline and bringing in multiple developers into the same conversation, allowing for AI to participate in your day-to-day collaboration with your colleagues and extending the flows across more of the key work services involved in software development, such as documentation, work planning, and code review, so that the AI can stay in lockstep with you through all of your work.

This launch is just the beginning.

**Anshul R** [1:35:22]
The Windsurf Editor was built around our AI engine to assist you however you write code, whether it's Cascade or in the text editor. While you're typing, you will get all the features you have come to love from the Codium extensions.

Autocomplete ships with a blazingly powerful Fast Mode to finish your thoughts. Supercomplete, the modality that predicts the edits corresponding to your next intent, prompts you with multi-cursor edits to keep you in your flow state. Command has a brand new UX that makes it easier, faster, to perform large file modifications, full file generations, and follow-ups.

We even brought Command into the terminal, so you never have to Google that random bash command again. All of this is powered by our custom, world-class LLMs and backed by the same context-aware engine that predicts your next move.

The Windsurf Editor meets you where you are and helps you move faster so that you can build more.

**Scott Wu** [1:36:12]
We're really excited about the Windsurf Editor. The team has been working incredibly hard on it. It is a fork of VS Code. And yes, we do know the memes. But there wasn't a need to rebuild everything from scratch.

The Windsurf Editor should feel familiar and intuitive. But with capabilities like Cascade, you should also internalize that something super powerful is going on under the surface. The editor will also give us the ability to give every developer another magical moment with technology.

It's why we call the Windsurf Editor tomorrow's editor today. So what do you think?

**Anshul R** [1:36:42]
What do I think? What do I think, Varun? I'll tell you what I think. I think this Windsurf Editor

flies. It's really fast. I think this is the next big thing. Now, when can people expect it, Varun? When?

**Scott Wu** [1:36:55]
You can experience the magic today at windsurf.ai.

**Anshul R** [1:36:59]
windsurf.ai, fuckers.

Fucking Varun, man. You fucking killed it. And congrats on the new office space, man. I mean, look at this. Fucking great. Oh, the Windsurf app. How does it how does it open?

### Windsurf Deep Dive

**Alessio** [1:37:21]
Everyone, welcome to the Latent Space Podcast. This is Alessio, partner and CTO at Decimal Partners. And I'm joined by my co-host Swix, founder of Small.ai.

**Swix** [1:37:29]
Hey. And today, we are delighted to be, I think, the first podcast in the new Codium office. So thanks for having us. And welcome, Varun and Anshul.

Thanks for having us. Yeah. Thanks for having us.

This is the Silicon Valley office?

Yeah.

So what's the story behind this?

**Varun Mohan** [1:37:44]
The story is that the office was previously so we used to be on Castro Street. So this is in Valley View. And I think a lot of the people at the company previously were in SF or are still in SF.

And actually, one thing if you notice about the company is it's actually like a two-minute walk from the Caltrain. And I think we were we didn't want to move the office very far away from the Caltrain. That would probably piss off a lot of the people that lived in San Francisco, this guy included.

So we were scouting a lot of spaces in the nearby area. And this area popped up. It previously was being leased by, I think, Facebook/WhatsApp. And then immediately after that, a ghost autonomy. And then now, here we are.

And we also I guess one of the things that the landlord told us was this was the place that they shot all the scenes for Silicon Valley, at least externally and stuff like that. So that just became a meme.

Trust me, that wasn't the main reason why we moved it. But we've leaned into it.

**Swix** [1:38:37]
It doesn't hurt.

**Varun Mohan** [1:38:37]
Yeah. Yeah. And obviously, that played a little bit into your launch with Windsurf as well. Let's get caught up. You were guest number four? I think.

**Swix** [1:38:45]
I think it was two.

**Varun Mohan** [1:38:46]
Maybe it was two.

**Swix** [1:38:46]
Two.

**Varun Mohan** [1:38:47]
Might have been two.

**Swix** [1:38:48]
So a lot has happened since then. You've raised a huge round and also just launched your idea. What's been the progress over the last year or so since Latent Space people last saw you?

**Varun Mohan** [1:38:57]
Yeah. So I think the biggest things that have happened are Codium's extensions have continued to gain a lot of popularity. We have over 800,000 developers that use that product. Lots of large enterprises also use the product. We were recently awarded JPMorgan Chase's Hall of Innovation Award, which is usually not something a company gets within a year of deploying an enterprise product.

And then large companies like Dell and stuff use the product. So I think we've seen a lot of traction on the enterprise space. But I think one of the most exciting things we've launched recently is this actually IDE called Windsurf.

And I think for us, one of the things that we've always thought about is how do we build the most powerful AI system for developers everywhere? The reason why we started out with the extension system was we felt that there were lots of developers that were not going to be on one platform.

And that still is true, by the way. Outside of Silicon Valley, a lot of people don't use GitHub. This is like a very surprising finding. But most people use GitLab, Bitbucket, Garrett, Perverse, CVS, Harvest, Mercurial. I could keep going down a list.

But there's probably 10 of them. GitHub might have less than 10% penetration of the Fortune 500, full penetration. It's very small. And then also on top of that, GitHub has very high switching costs for source code management tools because you actually need to switch over all the dependent systems on this workflow software.

It's much harder than even switching off of a database. So because of that, we actually found ways in which we could be better partners to our customers, regardless of where they stored their source code. And then more specifically on the IDE category, a lot of developers, surprise, surprise, don't just write TypeScript and Python.

They write Java. They write Golang. They write a lot of different languages. And then a high-quality language servers and debuggers matter. Very honestly, JetBrains has the best debugger for Java. It's not even close. These are extremely complex pieces of software.

We have customers where over 70% of their developers use JetBrains. And because of that, we wanted to provide a great experience wherever the developer was. But one thing that we found was lacking was we were running into the limitations of building within the VS Code ecosystem on the VS Code platform.

And I think we felt that there was an opportunity for us to build a premier experience. And that was within the reach of the team. The team has done all the work, all the infrastructure work to build the best possible experience and plug it into every IDE.

Why don't we just build our own IDE that is by far the best experience? And as these agentic products sort of become more and more possible and all the research we had done on retrieval and just reasoning about codebases became more and more to life, we were like, hey, if we launch this agentic product on top of a system that we didn't have a lot of control over, it's just going to limit the value of the product.

And we're just not going to be able to build the best tool. That's why we were super excited to launch Windsurf. I do think it is the most powerful IDE system out thereright now in the capability. And this is just the beginning.

I think we suspect that there's much, much more we can do, more than just the autocomplete side. When we originally talked, probably autocomplete was the only piece of functionality the product actually had. And we've come a long way since then.

These systems can now reason about large codebases without you adding everything. When you use Google, do you say,@New York Times Post, blah, blah, blah, and ask it a question? No. We want it to be a magical experience where you don't need to do that.

We want it to actually go out and execute code. We think code execution is a really, really important piece. And when you write software, you not only just kind of come up with an idea, the way software kind of gets created is software is originally this amorphous blob.

And as time goes on and you have an idea, the blob and the cloud sort of disappear. And you see this mountain. And we want it to be the case that as soon as you see the mountain, the AI helps you get to the mountain.

And as soon as you see the mountain, the AI just creates the mountain for you. And that's why we don't believe in this sort of modality where you just write a task and it just goes out and does it.

It's good for zero-to-one apps. And I think people have been seeing Windsurf is capable of doing that. And I'll let Anshul talk about that a little bit. But we've been seeing real value in real software development, which is more to say this is not to say that current tools can't.

But I think more in the process of actually evolving code from a very basic idea. Code is not really built as you have a PRD and then you get some output out. It's more like you have a general vision.

And yes, and as you write the code, you get more and more clarity on approaches that don't work and do work. You're killing ideas and creating ideas constantly. And we think Windsurf is theright paradigm for that.

**Alessio** [1:43:03]
Can you spell out what you couldn't do in VS Code? Because I think when we did the Cursor episode, explained, then everybody on AgriNews is like, oh, blah, blah, blah. Why did you fork? You could have done it in an extension.

Can you maybe just explain more of those limitations?

**Varun Mohan** [1:43:19]
I mean, I think a lot of the limitations around APIs are pretty well-documented. I don't know if we need to necessarily go down that rabbit hole. I think it was when we started thinking, OK, what are the pieces that we actually need to give the AI to get to that kind of emergent behavior that Varun talked about?

And yes, we were talking about all the knowledge retrieval systems that we've been building for the enterprise all this time. That's obviously a component of that. We were talking about all the different tools that we could give it access to so they can go do that kind of terminal execution and things like that.

And the third main category that we realized would be kind of that magical thing where you're not out there writing out a PRD. You're not scoping the problem for the AI is that if we're actually being able to understand kind of the trajectory of what developers are doing within the editor.

If we actually are being able to see like, oh, the developer just went and opened up this part of the directory and tried to view it. Then they made these kind of edits. And they tried to do some kind of commands in the terminal.

And if we actually understand that trajectory, then our ability for the AI to just immediately be like, oh, I understand your intent. This is what you want to do without you having to spell it all out for it, that is when that kind of magic would really happen.

I think that was kind of like that into it. So you have the restrictions of the APIs that are well-documented. We have the kind of vision of what we actually need to be able to hook into to really expose this.

And I think it was that combination of those two where we were like, I think it's about time to do the editor. The editor was not necessarily a new idea. I think we've been talking about the editor for a very long time.

I think it's like, of course, we just pulled it all together in the last couple of months. But it was always something in the back of the mind. And it's only when we started realizing, OK, the models are now capable of doing this.

We actually can look at this data. We have a really good context awareness system. We're like, I think now is the time. And we went on and executed on it.

**Alessio** [1:44:58]
So it's basically not action. It's not like one action you couldn't do, but it's like how you brought it all together. It's like the VS Code is kind of like a sandbox, so to speak.

**Varun Mohan** [1:45:06]
Yeah. Let me maybe even just to go one step deeper on each of the aspects that Anshul talked about. Let's go with the API aspect. Soright now, I'll give you an example. Supercomplete is actually a feature that I think is very exciting about the product.

It can suggest refactors of the code. I think you can do it quickly and very powerfully. On VS Code, actually, the problem for us wasn't actually being able to implement the feature. We had the feature for a while.

The problem was actually even to show the feature, VS Code would not expose an API for us to do this. So what we actually ended up doing was dynamically generating PNGs to actually go out and showcase this. It was not really aligned.

We actually ended up doing it ourselves. And it took us a couple of hours to actually go out and implement this. And that wasn't because we were bad engineers. No. Our good engineering time was being spent fighting against the system rather than being a good system.

Another example is we needed to go out and find ways to refactor the code. The VS Code API would constantly keep breaking on us. And we'd constantly need to show a worse and worse experience. This actually comes down to the second point, which Anshul brought up, which is like we can come up with great work and great research.

All the work we have here is not like the research on Cascade is not like a couple-month thing. This is like a nine months to a year thing that we've been investigating as a company, investing in on evals.

Even the evals for this are a lot of effort. A lot of actually systems work to actually go out and do it. But ultimately, this needs to be a product that developers actually use. And I think let's even go for Cascade, for example, and looking at the trajectory.

**Swix** [1:46:24]
Can you define Cascade? Because that's the first time you brought it up.

**Varun Mohan** [1:46:26]
Yeah. So Cascade is the product that is the actual agentic part of the product that is capable of taking information from both these human trajectories and these AI trajectories, what the human ended up doing, what the AI ended up doing, to actually propose changes and actually execute code to finally get you the final work output.

I'll even talk about something very basic. Cascade gives you a bunch of code. We want developers to very easily be able to review this code. OK, then we can show developers a hideous UI that they don't want to look at.

And no one's going to really use this product. And we think that this is like a fundamental building block for us to make the product materially better. If people are not even willing to use the building block, where does this go?

And we just felt our ceiling was capped on what we could deliver as an experience. Interestingly, JetBrains is a much more configurable paradigm than VS Code is. But we just felt so limited on both the directions that Anshul said that we were just like, hey, if we actually remove these limitations, we can move substantially faster.

And we believe that this was a necessary step for us.

**Swix** [1:47:23]
I'm curious more about the evals side of it because you brought it up. And we have to ask about evals anytime anyone brings up evals. How do you evaluate a thing like this that is so multi-step and spanning so much context?

**Varun Mohan** [1:47:36]
So what you can imagine we can sort of do and this is like one of the beautiful things about code is code can be executed. We could go take a bunch of open-source code. We can find a bunch of commits.

And we can actually see if some of these commits have tests associated with them. We can start stripping the commits. And the approach of stripping the commits is good because it tests the fact that the code is in an incomplete state.

When you're writing the commit, the goal is not the commit has already been written for you. You're giving it in a state where the entire thing has not been written. And can we go out and actually retrieve theright snippets and actually come up with a cohesive plan and iterative loop that gets you to a state where the code actually passes?

So you can actually break down and decompose this complex problem into a planning, retrieval, and multi-step execution problem. And you can see on every single one of these axes is it getting better. And if you do this across enough repositories, you've turned this highly discontinuous and discrete problem of make a PR work versus make it not work into a continuous problem.

And now that's a hill you can actually climb. And that's a way that you can actually apply research where it's like, hey, my retrieval got way better. This made my eval get better. And then notice how the way the eval works is I'm not that interested in the eval where purely it's a commit message and you finish the entire thing.

I'm more interested in the code is in an incomplete state. And the commit message isn't even given to you because that's another thing about developers. They are not willing to tell you exactly what's in their head. That's the actual important piece of this problem.

We believe that developers will never completely pose the problem statement because the problem statement lives in their head, conversations that you and I have had at the coffee area, conversations that I've had over Slack, conversations I've had over Jira.

Maybe they're not Jira. Let's say Linear. That's the cool thing nowadays.

**Swix** [1:49:13]
We're talking about Jira.

**Varun Mohan** [1:49:13]
Yeah. So conversations I've had on Linear. And all of these things come together to actually finally propose sort of a solution there, which is why we want to test the incomplete code. What happens if the state is in an incomplete state?

And am I actually able to make this pass without the commit? And can I actually guess your commit well? Now you can convert the problem into a masked prediction problem where you want to guess both the high-level intent and as well as the remainder of changes to make the actual test pass.

And you can imagine if you build up all of these, now you can see, hey, my systems are getting better. Retrieval quality is getting better. And you can actually start testing this on larger and larger codebases. And I guess that's one thing that we honestly, to be honest, we could have done a little faster.

We had the technology to go out and build these zero-to-one apps very quickly. And I think people are using Windsurf to actually do that. And it's like extremely impressive. But the real value, I think, is actually much deeper than that.

It's actually that you take a large codebase. And it's actually a really good first pass. And I'm not saying it's perfect, but it's only going to keep getting better. And we have deep sort of infrastructure to that actually validating that we are getting better on this dimension.

**Speaker 12** [1:50:12]
Varun mentioned the end-to-end evals that we have for this system, which I think are super cool. But I think you can even decompose each of those steps, the ideas of just take retrieval, for example. How can we make eval for retrieval really good?

And I think this is just a general thing that's been true about us as a company. Most evals and benchmarks that exist out there for software development is kind of bogus. There's not really a better way of putting it.

OK, you have SweBench. That's cool. No actual professional work looks like SweBench. Human eval, same thing. These things are just a little kind of broken. So when you're trying to optimize against a metric that's a little bit broken, you end up making kind of suboptimal decisions.

So something that we're always very keen on is like, OK, what is the actual metric that we want to test for this part of the system? So take retrieval, for example. A lot of the benchmarks for these embedding-based systems are like needle-in-the-haystack problems.

I want to find this one particular piece of information out of all this potential context. That's not really what actually is necessary for doing software engineering because code is a super distributed knowledge store. You actually want to pull in snippets from a lot of different parts of the codebase in order to do the work.

And so we've built systems that instead of looking at retrieval at one, you're looking at retrieval at like 50. What are the 50 highest things that you can actually retrieve? And are you capturing all of the necessary pieces for that?

And what are all the necessary pieces? Well, you can look again back at old commits and see over all the different files that together were edited to make a commit because those are semantically similar things that might not actually show if you actually try to map out a code graph.

And so we can actually build these kind of golden sets. We can do this evaluation even for sub-problems in the overall task. And so now we have an engineering team that can iterate on all of these things and still make sure that the end goal that we're trying to build to is really, really strong, so that we have confidence in what we're pushing out.

**Varun Mohan** [1:51:54]
And by the way, just to talk one more thing about the SweBench thing, just to showcase these existing metrics, I think benchmarks are not a bad thing. You do want benchmarks. Actually, I would prefer if there were benchmarks versus, let's say, everything was just vibes.

But vibes are also very important, by the way, because they showcase where the benchmark is not valuable because actually vibes sometimes show you where criminal issues sort of exist in the benchmark. But you look at some of the ways in which people have optimized SweBench, it's like make sure to run pytest every time X happens.

And it's like, yeah, sure. You can start prompting it in every single possible way. And if you remove that suddenly, it doesn't get good at it. It's like what really matters here. What really matters here is across a broad set of tasks, you are performing high-quality sort of suggestions for people.

And people love using the product. And I think actually the way these things work is beyond a certain point because yes, I actually think it's valuable beyond a certain point. But once it starts hitting the peak of these benchmarks, getting that last 10% actually probably is counterintuitive to the actual goal of what the benchmark was.

You probably should find a new hill to climb rather than sort of p-hacking or really optimizing for how you can get higher on the benchmark.

**Alessio** [1:52:59]
Yeah. We did an episode with Anthropic about their recent SweAgent, SweBench results. And we talked about the human eval versus SweBench. And for human eval, it's kind of like a greenfield benchmark. You need to be good at that.

SweBench is more accessing. But it sounds like, I mean, your eval creation is similar to SweBench as far as using GitHub commits and kind of like that history. But then it's more like masking at the commit level versus just testing the output of the thing.

**Swix** [1:53:25]
Cool. We have some listener questions actually about the Windsurf launch. And obviously, I also want to give you the chance to just respond to Hacker News.

**Varun Mohan** [1:53:36]
Let me tell you something very, very interesting. I love Hacker News as much as the next person. But the moment we launched our product, the first comment this was a year ago. The first comment was, "This product is a virus."

And we were like.

**Swix** [1:53:48]
This was the original Codeium launch like two years ago.

**Varun Mohan** [1:53:50]
This is the original.

**Swix** [1:53:51]
Like, "I am analyzing the binary as we speak. We'll report back."

**Varun Mohan** [1:53:55]
And then he's like.

**Swix** [1:53:55]
And he was like, "It's a virus." And I was like, "Dude, it's not a virus." We just want to give autocomplete suggestions. That's all we want to do.

**Varun Mohan** [1:54:04]
Yeah.

**Swix** [1:54:05]
OK. Wow. I didn't expect that.

**Alessio** [1:54:07]
And then there was ITO drama. There's enough drama on the launch to cover. But I don't know if we want to just make this a Cascade piece. But we had a bunch of people in our Discord trial the product, give a lot of feedback.

One question people have is like, to them, Cascade already felt pretty agentic. Is that something you want to do more of? Obviously, since you just launched on IDE, you're kind of like you're focusing on having people write the code.

But maybe this is kind of like the Trojan horse to just doing more full-on end-to-end code creation.

**Swix** [1:54:36]
Devin style.

**Varun Mohan** [1:54:37]
Yeah. I think it's like, how do you get there in a real principled manner? We have obviously enterprise asking us all the time, like, "Oh, when's it going to do end-to-end work?" The reality is like, OK, well, if we have something in the IDE that, again, can see your entire actions and get a lot of intent that you can't actually get if you're not in the IDE, if the agent there has to always get human involvement to keep on fixing itself, it's probably not ready to become a full end-to-end automated system because then we're just going to turn into a linter where it produces a bunch of things and no one looks at any of it.

That's not the great end state. But if we start seeing like, oh, yeah, there's common patterns that people do that never require human involvement and just end-to-end just totally works without any intent-based information, sure, that can become fully agentic.

And we will learn what those tasks are pretty quickly because we have a lot of data. Maybe add on to that. I think that if the answer is full agentic is called is Devin, I think, yes, the answer is this product should become fully agentic.

And limited human interaction is the goal, is 100% the goal. And I think, honestly, of all usable productsright now, I think we're the closestright now of all usable products in an IDE. Now, let me caveat this by saying I think there are lots of hard problems that have yet to be solved that we need to go out and solve to actually make this happen.

For instance, I think one of the most annoying parts about the product is the fact that you need to accept every command that kind of gets run. It's actually fairly annoying. I would like it to go out and run it.

Unfortunately, me going out and running arbitrary binaries has some problems in that if it RMRs my hard disk.

**Swix** [1:56:06]
It's actually a virus.

**Varun Mohan** [1:56:07]
I'm not saying actually.

**Swix** [1:56:08]
It's a virus.

**Varun Mohan** [1:56:09]
Hacker News scream on you.

**Swix** [1:56:10]
Yeah.

**Varun Mohan** [1:56:11]
It does become a virus. I think this is solvable with complex systems. I think we love working on complex systems infrastructure. I think we'll solve it. Now, the simpler way to go about solving this is don't run it on the user's machine and run it somewhere else because then if you board that machine, you're kind of totally fine.

Now, I think, though, maybe there's a little bit of trade-off of running it locally versus remotely. And I think we might change our mind on this. But I think the goal for this is not for this to be the final state.

I think the goal for this is, A, it's actually able to do very complex tasks with limited human interaction. But it needs to know when to actually go back to the human. Also, on top of that, compress every cycle that the agent is running.

Right now, actually, I even feel like the product is too slow for me sometimesright now, even with it running really fast. It's objectively pretty fast. I would still want it to be faster. So there is systems work and probably modeling work that needs to happen there to make the product even faster on both the retrieval side and the generation side.

And then finally speaking, I think another key piece here that's really important is I actually think asking people to do things explicitly is probably going to be more of an anti-pattern if we can actually go and passively suggest the entire change for the user.

So almost imagine as the user is using the product that we're going to suggest the remainder of the PR without the user kind of even asking us for it. I think this is sort of the beginning for it.

But yeah, these are hard problems. I can't give a particular deadline for this. I think this is a big step up than what we had, particularly in the past. But I think what Anshul said is 100% true. But the goal is for us to get better at this.

**Alessio** [1:57:39]
I mean, the remote execution thing is interesting. You wrote a post about the end of localhost.

**Swix** [1:57:43]
Yeah.

**Alessio** [1:57:44]
And now it's almost like then we were kind of like, well, no, maybe we do need the internet. And people want to run things. But now it's like, OK, no, actually, I don't really care. I want the model to do the thing.

And if you were like, you can do a task end-to-end, but it needs to run remotely, not on your computer, I'm sure most people would say, yeah.

**Swix** [1:57:59]
No, I agree with that.

**Varun Mohan** [1:58:00]
I actually agree with it running remotely. That's not a security issue. I actually totally agree with you that it's possible that everything could run remotely.

**Swix** [1:58:07]
That's how it is at most big codes, like Facebook. Nobody runs things locally.

**Varun Mohan** [1:58:12]
No one does. In fact, you connect to a remote.

**Swix** [1:58:14]
It's associated to the main frame.

**Varun Mohan** [1:58:15]
You'reright on that. Maybe the one thing that I do think is kind of important for these systems that is more than just running remotely is basically when you look at these agents, there is kind of like a rollout of a trajectory.

And I kind of want to roll this trajectory back. In some ways, I want a snapshot of the system that I can constantly checkpoint and move back and forth. And then also on top of that, I might want to do multiple rollouts of this.

So basically, I think there needs to be a way to almost move forward and move backwards the system. And whether that's locally or remotely, I think that's necessary. But every time if you move the system forward, it destroys your machine.

It's probably going to be a hard system to kind of or potentially destroys your machine. That's just not a workable solution. So I think the local versus remote, I think you still need to solve the problem of this thing is not going to destroy your machine on every execution, if that makes sense.

**Swix** [1:59:00]
Yeah. There is a category of emerging infrastructure providers that are working on time travel VMs where you can kind of.

**Varun Mohan** [1:59:06]
And Varun's first episode on this podcast was an indication we like infrastructure problems.

**Swix** [1:59:11]
Yeah. OK. Allright. Oh, so you're going there. Allright. OK.

**Alessio** [1:59:13]
Oh, that's funny,right? It's like when we first had you, you were doing so much on actual model inference, optimization, all these things. And today, it's almost like.

**Swix** [1:59:21]
It's Claude. It's 4.0.

**Alessio** [1:59:23]
It's like people are forgetting about the model. And now it's all about at a higher level of abstraction.

**Varun Mohan** [1:59:29]
Yeah. So maybe I can say a little bit about how our strategy on this has evolved because it objectively has. I think I would be lying if I said it hasn't. The things like autocomplete and supercomplete that run on every keystroke are entirely our own models.

And by the way, that is still because properties like FIM, fill-in-the-middle capabilities, are still quite bad with the current.

**Swix** [1:59:48]
Nonexistent.

**Varun Mohan** [1:59:49]
They're very bad. Nonexistent. They're not good actually at it.

**Swix** [1:59:52]
Because FIM isn't actual how you order the tokens.

**Varun Mohan** [1:59:55]
Yes. It's how you order the tokens actually in some ways. And this is sort of if you look at what these products have sort of become, and this is great, is a lot of the Claudes and the OpenAIs have focused on kind of the chat assistant API where it's like complete pieces of work, message, and other complete piece of work.

So multi-turn kind of back and forth systems. In fact, actually, even these systems are not that good at making point changes. When they make point changes, they kind of are off here and there by a little bit because, yeah, when you are doing multi-point kind of conversations, exact diffs getting applied is not even a perfect science still yet.

So we care about that. The second piece where we've actually sort of trained our model is actually on the retrieval system. And this is not even for embedding, but actually being able to use high-powered LLMs to be able to do much higher quality retrieval across the code base.

So this is actually what Anshul said. For a lot of the systems, we do believe embeddings work. But for complex questions, we don't believe embeddings can encapsulate all the granularity of a particular query. Imagine I have a question on a code base of find me all quadratic time algorithms in this code base.

Do we genuinely believe the embedding can encapsulate the fact that this function is a quadratic time function? No, I don't think it does. So you are going to get extremely poor precision recall at this task. So we need to apply something a little more high-powered to actually go out and do that.

So we've actually built large distributed systems to actually go out and run these at scale, run custom models at scale across large code bases. So I think it's more a question of that. The planning modelsright now, undoubtedly, I think the Claudes and the OpenAIs have the best products.

I think LLaMA 4, depending on where it goes, it could be materially better. It's very clear that they're willing to invest a similar amount of compute as the OpenAIs and the Anthropics. So we'll see. I would be very happy if they got really good, but unclear so far.

**Swix** [2:01:37]
Don't forget Grok.

**Varun Mohan** [2:01:38]
Hey, dude, I think Grok is also possible. I think don't doubt Elon. Yeah.

**Swix** [2:01:42]
OK. So I didn't actually know. It's not obvious when I use Cascade. I should also mention that I was part of the preview. Thanks for letting me in. And I've been maining Windsurf for a long time. It's not actually obvious.

You don't make it obvious that you are running your own models. I feel like you should so that I feel like it has more differentiation. I only have exclusive access to your models via your IDE than having to drop down into Claude and 4.0 because I actually thought that was what you did.

**Varun Mohan** [2:02:08]
No. So actually, the way it works is the high-level planning that is going on in the model is actually getting done with products like Claude. But the extremely fast retrieval, as well as the ability to take the high-level plan and actually apply it to the code base, is proprietary systems that are running internally.

**Swix** [2:02:22]
And then the stuff you said about embeddings not being enough. Are you familiar with the concept of late interaction?

**Varun Mohan** [2:02:28]
No, I actually have never heard of that.

**Swix** [2:02:29]
Yeah. So this is Colbert or the guy Omar Kattab from, I think, Stanford has been promoting this a lot. It is basically what you've done. Sort of embedding on retrieval rather than pre-embedding in a very loose context.

**Varun Mohan** [2:02:41]
I think that sounds like a very good idea that is very similar to what we're doing.

**Swix** [2:02:45]
It sounds like a very good idea. I think we'd say that.

**Varun Mohan** [2:02:48]
That's like the meme of Obama giving himself a medalright there.

**Swix** [2:02:52]
Well, I mean, there might be something to learn from contrasting the ideas and seeing where the subtle opinion difference is. It's also been applied very effectively to vision understanding because vision models tend to just consume the whole image.

If you are able to sort of focus on images based on the query, I think that can get you a lot of extra performance.

**Varun Mohan** [2:03:12]
The basic idea of using compute in a distributed manner to do operations over a whole set of raw data rather than a materialized view is not anything new. I think it's just like, how does that look like for LLMs?

**Swix** [2:03:23]
When I hear you say build large distributed systems, you have a very strange product strategy of going down to the individual developer, but also to the large enterprise. Is it the same infra that serves everything?

**Varun Mohan** [2:03:34]
I think the answer to that is yes. The answer to that is yes. And the only reason why for the yes, the answer is yes. And to be honest, our company is a lot more complex than I think if we just wanted to serve the individual.

And I'll tell you that because we don't really pay other providers to do things for our indexing. We don't pay other providers to do our serving of our own customer models. And I think that's a core competency within our company that we have decided to build.

But that's also enabled us to go and make sure that when we are serving these products in an environment that works for these large enterprises, we're not going out and being like, we need to build this custom system for you guys.

This is the same system that serves our entire user base. So that is a very unique decision we've taken as a company. And we admit that there are probably faster ways that we could have done this.

**Swix** [2:04:17]
I was thinking when I was working with you for your enterprise piece, I was thinking this philosophy of go slow to go fast, build deliberately for theright level of abstraction that can serve the market that you really are going after.

**Varun Mohan** [2:04:28]
Yeah. I mean, I would say when writing that piece, looking back and reading it back, it sounds almost obvious in a sense. Not all of those are really conscious decisions we made. I'll be the first to admit that.

But it does help when we go to an enterprise that has tens of thousands of developers. And they're like, oh, wow, we have tens of thousands of developers. And does your infrastructure work for tens of thousands of developers?

We can turn around and be like, well, we have hundreds of thousands of developers on an individual plan that we're serving. I think we'll be able to support you. So being able to do those things, we start off by just like, let's give it to individuals.

Let's see what people like and what they don't like and learn. But then those become value propositions when we go to the enterprise.

**Alessio** [2:05:05]
And to recap, when you first came on the pod, it was like autocompletion is free and Copilot was $10 a month. And you said, look, what we care about is building things on top of code completion. How did you decide to just not focus on short-term kind of growth monetization of the individual developer and build some of this?

Because the alternative would have been, hey, all these people are using it. It's like we're going to make this other $5 a month plan, monetize.

**Varun Mohan** [2:05:31]
I think this might be a little bit of commercial instinct that the company has. And unclear if the commercial instinct isright. I think thatright now, optimizing for making money off of individual developers is probably the wrong actually strategy, largely because I think individual developers can switch off of products very quickly.

And unless we have a very large lead trying to optimize for making a lot of profit off of individual developers, it's probably something that someone else could just vaporize very quickly. And then they move to another product. And I'm going to say this very honestly.

When you use a product like Codeium on the individual side, there's not much that prevents you to switch onto another product. I think that will change with time as the products get better and better and deeper and deeper.

I constantly say this. There's a book in business called Seven Powers. And I think one of the powers that a business like ours need to have is real switching costs. But you first need something in the product that makes people switch on and stay on before you think about how do you make people switch off.

And I think for us, we believe that there's probably much more differentiation we can derive in the enterprise by working with these large companies in a way that is interesting and scalable for them. I'll be maybe more concrete here.

Individual developers are much more sort of tuned towards small price changes. They care a lot more. If our product is $10, $20 a month instead of $50 or $100 a month, that matters to them a lot. For a large company where they're already spending billions of dollars on software, this is much less important.

So you can actually solve maybe deeper problems for them. And you can actually kind of provide more differentiation on that angle. Whereas I think individual developers could be churny as long as we don't have the best product. So focus on being the best product, not trying to take price and make a lot of money off of people.

And I don't think we will, for the foreseeable future, try to be a company that tries to make a lot of money off individual developers.

**Swix** [2:07:17]
I mean, that makes sense. So why $10 a month for Windsurf?

**Varun Mohan** [2:07:20]
Why $10? $10 a month was actually the pro plan. So we launched our individual pro plan before Windsurf existed because I think there's we all said to be financially responsible.

**Swix** [2:07:30]
Right.

**Varun Mohan** [2:07:32]
We can run out of money.

**Swix** [2:07:34]
There's $150 million.

**Varun Mohan** [2:07:35]
I mean, there's a lot of things because of our infrastructure background. We can give for essentially free unlimited autocomplete, unlimited chat on our faster models. We give a lot of things actually out for free. But yeah, when we start doing things like the super completes and really large amounts of indexing and all of these things, there is real cogs here.

We can't ignore that. And so we just created a $10 a month pro plan mostly just to cover the costs. We're not really operating, I think, on much of a margin there either. But OK, just to cover us there.

So for Windsurf, it just ended up being the same thing. And everyone who downloads Windsurf in the first, I forget, a couple of weeks, they get two weeks for free. Let's just have people try it out, let us know what they like, what they don't like.

And that's how we've always operated.

**Alessio** [2:08:17]
I've talked to a lot of CTOs in the Fortune 100 where most of the engineers they have, they don't really do much anyway. The problem is not that the developer costs $200K and you're saving $8K. It's like that developer should not be paid $200K.

But that's kind of like the base price. But then you have developers getting paid $200K that should be paid $500K. So it's almost like you're averaging out the price because most people are actually not that productive anyway. So if you make them 20% more productive, they're still not very productive.

And I don't know in the future, is it that the junior developer's salary is like $50K? And it's like the bottom of the end gets kind of squeezed out. And then the top end gets squeezed up.

**Varun Mohan** [2:08:57]
Yeah. Maybe I'll let you do one thing that I think about a lot because I do think about this. The Percy, anything, all of this stuff, I think about a good deal. Let's take a product like Office 365.

I will say a lawyer at Codeium uses Microsoft Word way more than I do. I'm still footing the same bill. But the amount of value that he's driving from Office 365 is probably tens of thousands of dollars. By the way, everyone, Google Docs is a great product.

Microsoft Word is a crazy product. It made it so that the moment you review anything in Microsoft Word, the only way you can review it is with other people in Microsoft Word. So it's like this virus that penetrates everything.

And it not only penetrates it within the company. It penetrates it cross-company too. The amount of value it's driving is way higher for him. So for these kinds of products, there's always going to be, for these kinds of products, this variance between who gets value from these products.

And you'reright. It's almost like a blended because you're actually totallyright. Probably this company should be paying that one developer maybe like four times as much. But in weird ways, software is like this team activity enough that there's a bunch of blended outcomes.

But hey, like 20% of the four times and there are four people is still going to cover the cost across the four individuals. And that's how roughly these products kind of get priced out.

**Alessio** [2:10:02]
I mean, more than about pricing, this is about the future of the software engineer.

**Varun Mohan** [2:10:07]
We could be very wrong also.

**Swix** [2:10:08]
Yeah.

**Alessio** [2:10:10]
I think nobody knows.

**Swix** [2:10:11]
Reserve theright to be incredibly off.

**Varun Mohan** [2:10:13]
Yeah.

**Swix** [2:10:13]
I mean, business model does impact the product. Product does impact the user experience. So it's all of a kind. I don't mind. We do are as concerned about the business of tech as the tech itself.

**Varun Mohan** [2:10:23]
That's cool.

**Swix** [2:10:24]
Speaking of which, there's other listener questions. Shout out to Daniel Infeld, who's pretty active in our Discord, just asking all these things. Multi-agent, very, very hot and popular, especially from the Microsoft research point of view. Have you made any explorations there?

**Varun Mohan** [2:10:37]
I think we have. I don't think we've called it a multi-agent, which is more so like once you this notion of having many trajectories that you can spawn off that kind of validate sort of some different hypotheses. And you can kind of pick the most interesting one.

This is stuff that we've actually analyzed internally at the company. By the way, the reason why we have not put these things in actually is partially because we can't go out and execute some random stuff in Perl in the meantime on other sides.

**Swix** [2:11:02]
Because of side effects.

**Varun Mohan** [2:11:02]
Because of the side effects. So there are some things that are a little bit dependent on us unlocking more and more functionality internally. And then the other thing is, in the short term, I think there is also a latency component.

And I think all of these things can kind of be solved. I actually believe all of these things are solvable problem. They're not unsolvable problem. And if you want to run all of them in parallel, you probably don't want N machines to go out and do it.

I think that's unnecessary, especially if most of them are I/O bound kind of operations where all you're doing is reading a little bit of data and writing out a little bit of data. It's not extremely compute intensive. I think that it's a good idea and probably something we will pursue and is going to be in the product.

**Swix** [2:11:35]
I'm still processing what you just said about things being I/O bound. So for a certain class of concurrency, you can actually just run it all in one machine.

**Varun Mohan** [2:11:43]
Yeah. Why not? Because if you look at the changes that are made for some of these, it's writing out what, a couple thousand bytes? Maybe like tens of thousands of bytes on every it's not a lot. Very small.

**Swix** [2:11:53]
What's next for Cascade or Windsurf?

**Varun Mohan** [2:11:55]
Oh, there's a lot. I don't know. We did an internal poll. And we were just like, are you more excited about this launch or the launch that's happening in a month or what we're going to come out with in a month?

And it was like almost uniformly in a month. I think there's some obvious ones. I don't know how much you want to say. I don't want to speak it. But I think you look at all the same axes of the system.

How can we improve the knowledge retrieval? We'll always keep on figuring out how to improve knowledge retrieval. In our launch video, we even showed some of the early explorations we have about looking to other data sources. That might not be the coolest thing to the individual developer building a zero to one app.

But you can really believe that the enterprise customers really think that that's very cool. I think on the tool side, I think there's a whole lot more that we can do. I mean, of course, Vernon's talked about not just suggesting the terminal command but actually executing them.

I think that's going to be a huge unlock. You look at the actions that people are taking, like the human actions, the trajectories that we can build. How can we make that even more detailed? And I think all of those things and you make some even cleaner UI, the idea of looking at future trajectories, trying a few different things, and suggesting potential next actions to be taken, that doesn't really exist yet.

But it's pretty obvious, I think, how that would look like. You open up Cascade. And instead of starting typing, it's just like, here's a bunch of things that we want to do. We kind of joke that's like Clippy's coming back.

But maybe now's the time for Clippy to really shine. So I think there's a lot of ways that we can take this, which I think is the very exciting part. We're calling each of our launches Waves, I believe, because we want to really double down on the aquatic themes.

**Swix** [2:13:24]
Oh, yeah. Does someone actually Windsurf at the company?

**Varun Mohan** [2:13:29]
I don't think so.

**Swix** [2:13:29]
We're living out our dream of being cool enough to Windsurf through the product.

**Varun Mohan** [2:13:32]
Yeah. I don't think we can.

**Swix** [2:13:34]
Yeah. Allright.

**Varun Mohan** [2:13:35]
That was actually something we learned because I don't think any of us are Windsurfers. In our launch video, we have someone using Windsurf on a Windsurf.

**Swix** [2:13:42]
You saw that.

**Varun Mohan** [2:13:42]
You saw that in the beginning of the video. Someone has a computer. And we didn't realize now apparently is the time of the year where there's not enough wind to Windsurf. So we were trying to figure out how to do this launch video with Windsurf on the Windsurf.

Every Windsurfer we talked to were like, yeah, it's not possible. And there was one crazy guy who was like, yeah, I think we can do this. And. We made it happen.

**Swix** [2:14:00]
Oh, OK.

**Varun Mohan** [2:14:01]
That's funny.

**Swix** [2:14:02]
Is there anything that you want feedback on? Maybe there's a fork in the road. You want feedback. You want people to respond to this podcast and tell you what they want?

**Varun Mohan** [2:14:09]
Yeah. I think there's a lot of things that I think could be more polished about the product that we'd like to improve. Lots of different environments that we're going to improve performance on. And I think we would love to hear from folks across the gamut.

Like, hey, if you have this environment, you use Windows and X version, it didn't work. Or this language, it was very poor. I think we would like to hear it.

**Swix** [2:14:29]
Yeah. I gave Preb and Kevin a lot of shit for my Python issues.

**Varun Mohan** [2:14:33]
Yeah. Yeah, yeah. And I think there's a lot to kind of improve on the environment side. I think, for instance, even just a dumb example, and I think sort of Swix this was a common one, is like, yeah, the virtual environment, where is the terminal running?

What is all this stuff? These are all basic things that, to be honest, this is not rocket science. But we need to just fix it. We need to fix it. So we would love to hear all the feedback from the product.

Was it too slow? Where was it too slow? What kind of environments could it work way more in? There's a lot of things that we don't know. Luckily, we're daily users of the product internally. So we're getting a lot of feedback inside.

But I will say, there's a little bit of Silicon Valleyism in that a lot of us develop on Mac. A lot of people, once again, over 80% of developers are on Windows. So yeah, there's a lot to learn and probably a lot of improvements down the line.

**Swix** [2:15:14]
Are you personally tempted, as you're the CEO of the company, to switch to Windows just to feel something?

**Varun Mohan** [2:15:20]
Feel the Windows?

**Swix** [2:15:22]
You know what?

**Varun Mohan** [2:15:22]
Feel the pain.

**Swix** [2:15:23]
You know what? Maybe I should. Actually, I think I will.

**Varun Mohan** [2:15:26]
I mean, your customers, everyone says it's 89% all Windows. You live in Windows. You will never not see something that missed. So I think in the beginning, part of the reason why we were hesitant to do that was a lot of our architectural decisions to work on across every IDE was because we built a platform-agnostic way of running the system on the user's local machine that was only buildable, easily buildable on dev containers that lived on a particular time of platform.

So Mac was nice for that. But now there's not really an excuse if it's like, if I can also make changes to the UI and stuff like that. And yeah, WSL also exists. That's actually something that we need to add to the product.

That's how early it is that we have not actually added that.

**Swix** [2:16:06]
We don't have remote. Anything else about Codeium at large? You still have your core business of the enterprise Codeium.

**Varun Mohan** [2:16:14]
Yeah.

**Swix** [2:16:15]
Anything moving there or anything that people should know about?

**Varun Mohan** [2:16:18]
Anshul, you want to take that? I think a lot are still moving there. I think it would be a little bit like very kind of egotistical. It's going to be like, oh, we have Windsurf now. All of our enterprise customers are going to switch to Windsurf.

And this is the only no, we still support.

**Swix** [2:16:29]
I was going to say, you just talked about your Java guys loving JetBrains. They're never going to leave JetBrains.

**Varun Mohan** [2:16:33]
They're not. Forget JetBrains. There's still tons and tons of enterprise people on Eclipse. We're still the only code assistant that has an extension of Eclipse. That's still true years in. But that's because that's our enterprise customers. And the way that we always think about it is like, how do we still maximize the value of AI for every developer?

I don't think that part of who we are has changed since the beginning. And there's a lot of meeting the developers where they are. So I think on the enterprise side, we're still pretty invested in doing that. We have a team of engineers dedicated just to making enterprise successful and thinking about the enterprise problems.

But really, if we think about it from the really macro perspective, it's like if we can solve all the enterprise problems for an enterprise and we have products that developers themselves just truly, truly love, then we're solving the problem from both sides.

And I think it's one of those things where I think when we started working with the enterprise and we started building dev tools, we started as an infrastructure company. Now we're building dev tools for developers. You really quickly understand and realize just how much developers loving the tool make us successful in an enterprise.

There's a lot of enterprise software that developers hate.

**Swix** [2:17:36]
I want to draw this flywheel.

**Varun Mohan** [2:17:37]
But we're giving a tool for people where they're doing their most important work. They have to love it. And it's not like we're trying to convince the executives at this company also ask their developers, do you love this?

That is almost always a key aspect of whether or not Codeium is accepted or not into the organization. I don't think we go from 0 to 10 million ARR in less than a year in an enterprise product if we don't have a product that developers love.

So I think that's why we're just the IDE is more of a developer love kind of play. It will eventually make it to the enterprise. We still solve the enterprise problems. And again, we could be completely wrong about this.

But we hope we're solving theright problems.

**Swix** [2:18:12]
It's interesting. I asked you this before we started rolling. But it's the same team. That's the same engine team. In any normal company or my normal mental model of company construction, if you were to have effectively two products like this, you would have two different teams serving two different needs.

But it's the same team.

**Varun Mohan** [2:18:29]
Yeah. I think one of the things that's maybe unique about our company is this has not been one company the whole time. We were first this GPU virtualization company pivoted to this. And then after that, we're making some changes.

And I think there's a versatility of the company and this ability to move where we think the instinct we have this instinct where and by the way, the instinct could be wrong. But if we smell something, we're going to move fast.

And I think it's more a testament to, I think, the engineering team rather than any one of us.

**Alessio** [2:18:56]
Anshul, you had December 19, 2022. You had one of our guests post what building Copilot for X really takes. Estimate inference to figure out latency quality. Build first party instead of using third parties API. Figure out real time because ChatGPT and DALL·E, the RFP, are too slow.

Optimize prompt because context window is limited, which is maybe not that true anymore. And then merge model outputs with the UX to make the product more intuitive. Is there anything you would add.

**Varun Mohan** [2:19:28]
I'd give myself a B minus on that. So some parts of that are accurate, even the context, like the one that you called out. Yeah, models have larger context links now. That's absolutely true. It's grown a lot. But look at an enterprise code base.

They have tens of millions of lines of code. That's hundreds of billions of tokens of code.

**Swix** [2:19:45]
Never going to change.

**Varun Mohan** [2:19:46]
We're still being really good at being able to piece together this distributed knowledge store is important. So I think there are figures there that I think are still pretty accurate. There's probably some that are less so.

**Swix** [2:19:55]
First party versus third party.

**Varun Mohan** [2:19:56]
First party versus third party, I think we're wrong there. I think I would nuance that to be like, there are certain things that it's really important to do first party, like autocomplete. You have a really specific application that you can't just prompt engineer your way out of or just maybe even fine-tune afterwards.

You just can't do that. I think there's truth there. But let's also be realistic. The stuff that is coming out from the third model providers, like Cascade and Windsurf would not have been possible if it wasn't for the rapid improvements with 4.0 and 3.5.

That just wouldn't have been possible. So I'll give myself a B minus. I'll say I passed. But yeah, it's two hours, two years later.

**Alessio** [2:20:28]
Just to be clear, we're not grading. It's more of a.

**Varun Mohan** [2:20:31]
I'll grade myself. I would grade myself.

**Alessio** [2:20:32]
What would you where are they now? What would you have added? What would you like.

**Varun Mohan** [2:20:36]
Yeah. I mean, that first post, that was when we had literally I think that was a few weeks after we had launched Codeium. I think that's like Swix and I were talking like, maybe we can write this because we're one of the first products that people can actually use with AI.

That's cool.

**Swix** [2:20:48]
I specifically like the Copilot for X thing because everyone it is so hot.

**Varun Mohan** [2:20:52]
All the time. Everyone was just like, ChatGPT. That's all that was. But I think that we didn't have an enterprise product. I don't even think we were necessarily thinking of an enterprise product at that point. So all of the learnings that we've had from the enterprise perspective, which is why I loved coming back for a third time now on the blog, some of those I think we kind of figured, some of those we just honestly walked backwards into.

We had to get lucky a lot of the ways. We had many we just did a lot. There's so many opportunities and deals that we had that we lost for a variety of reasons that we had to learn from.

There's just so much more to add that there's no way I would have gone that route in 2022. Can I mention one thing that I think is hopefully this is not very controversial, but is true about our engineering team as a whole?

I don't think most of us got much value from ChatGPT, largely because I think the problem was and this is maybe a little bit of a different thing. It's like a lot of the engineers at the company who have been writing software for over eight years, and this is not to say they know everything that ChatGPT knows.

They don't. They'd already gone good enough at searching for Stack Overflow, invested a lot in searching code base. They can very quickly grab through the code incredibly fast, like every tool. And they've spent eight years mastering that skill.

And ChatGPT being this thing on the side that you need to provide a lot of context to, we were not able to actually get my co-founder just basically never used ChatGPT at all, literally never did. And because of that, probably at the time, one of our incorrect assumptions was probably that, hey, a lot of these passive systems need to get good because they're always there.

And these active systems are going to be behind. I think actually Cascade was the big thing. It's our company where everyone is now using it, literally everyone. Biggest skeptics. And we have a lot of people at the company that are skeptical of AI.

I think this is actually important.

**Swix** [2:22:33]
The way we'd hire them.

**Varun Mohan** [2:22:34]
No. I think here's the important thing. Those people that were skeptical about AI previously worked in autonomous vehicles. These are not crypto people. These are people that care about technology and want to work on the future. Their bar for good is just very high.

They will not form a cult of, this is awesome. This is going to change the world. They were not going to be the kind of people on Twitter that are like.

**Swix** [2:22:52]
Wagme.

**Varun Mohan** [2:22:53]
Yeah. This changes everything. Software as we know it is dead. No. There are people that are going to be incredibly honest. And we know if we hit the bar that is good for them, we found something special. And I think at that time, we probably had a lot of sentiment like that.

That has changed a lot now. And I think it's actually important that you have believers that are incredibly future-looking and people that kind of reign it in. Because otherwise, you just have this is like autonomous vehicles. You have a very discrete problem.

People are just working in a vacuum. And there's no signal to kind of bring you down to reality. You have no good way to kill ideas. And there are a lot of ideas we're going to come up with that are just terrible ideas.

But we need to come up with terrible ideas. Otherwise, how does anything good come on? And I don't want to call these skeptics. Skeptics suggest that they don't know. They're realists. They're the type of people that when they see waitlist on a product online, they just will not believe it.

They will not think about it at all.

**Swix** [2:23:41]
Kudos for launching without a waitlist.

**Varun Mohan** [2:23:42]
Yeah. Yeah. By the way, we will never launch with a waitlist. We will never launch with a waitlist. That's the thing at the company. We'd much rather be a company that's considered the boring company than a company that launches once in a while.

And hopefully, it's good.

**Swix** [2:23:53]
My joke is that generative AI has really gotten really good at generating waitlists.

**Varun Mohan** [2:23:59]
Yeah. I'll just clarify. Both of us used to work in autonomous vehicles. So it doesn't come across as.

**Swix** [2:24:03]
Oh, yeah. I'm not an astronaut.

**Varun Mohan** [2:24:06]
I love that technology.

**Swix** [2:24:07]
We love it. I love hard technology problems. That's what I live for. Amazing. Just push back on the first party thing. I accept that the large model labs have just done a lot of work for you that you didn't need to duplicate.

But you now are sitting on so much proprietary data that it may be worth training on the trajectories that you're collecting. So maybe it's a pendulum back to first party.

**Varun Mohan** [2:24:30]
Yeah. I mean, I think we've been pretty clear from a security posture perspective. I think there's both customer trust and.

**Swix** [2:24:38]
I mean, I kind of want let me off that.

**Varun Mohan** [2:24:40]
I think that there is signals that we do get from our users that we can utilize. There's a lot of preference information that we get, for example.

**Swix** [2:24:45]
Which is effectively what you're saying of our trajectories. Our trajectory is good. I will say this. The super complete product that we have has gotten materially better because of us not only using synthetic data, but also getting the preference data from our users of like, hey, given these set of trajectories, here's actually what a good outcome is.

And in fact, one of the really beautiful parts about our product that is very different than a ChatGPT is we can not only see if the acceptance happened, but if something more than the acceptance happened. And it happened even more than that.

Let's say you accepted it. But then after accepting it, you deleted three or four items in there. We can see that. So that actually lets us get to even better than acceptance as a metric because we're in the ultimate work output of the developer.

**Varun Mohan** [2:25:27]
It's the preference between the acceptance and what actually happened. If you can actually get ground truth of what actually happened, this is the beauty of being in the IDE, then yeah, you get a lot of information there.

**Swix** [2:25:36]
Did you have this with the extension? Or is this pure Windsurf?

**Varun Mohan** [2:25:39]
We had this with the extension.

**Swix** [2:25:40]
Yeah. OK. Allright. The Windsurf just gives you more of the IDE.

**Varun Mohan** [2:25:43]
Yes. So that means you can also start getting more information. For instance, the basic thing that Anshul said, we can see if a file explorer was opened. That's actually just a piece of information we just cannot see previously.

**Swix** [2:25:54]
Sure.

**Varun Mohan** [2:25:54]
Yeah. A lot of intent in there. A lot of intent.

**Alessio** [2:25:57]
Second one. How to make AI UX your moat.

**Varun Mohan** [2:26:00]
Oh, man. Isn't that funny that we now created the full UX experience in the IDE? I think that one is pretty accurate.

**Swix** [2:26:06]
That one's an A.

**Varun Mohan** [2:26:07]
I think that one I'll give myself. I think we were doing that within the extension. And I still think that's true within the extensions as well. We got very, very creative with things. We mentioned the idea of just essentially rendering images to display things.

We get creative to figure out what theright UX is doing there. We could create a really dumb UX that's like a side panel, whatever. But no. Actually going the extra mile does make that experience as good as it possibly can there.

But yeah, now look at some of the UX that we're able to build in Windsurf. And it's just like it's fun. The first time I saw because now we can do command in the terminal. You can not have to search for a bash command.

The first time I saw that, I was like I just started smiling. And it's not like Cascade. It's not like a genetic system or anything like that. But I'm like, that is just very, very cool.

**Swix** [2:26:50]
You literally couldn't do that in VS Code. Yeah. I understand that. Yeah. I've even made a 60-line bash command called please. And you can do that inside of it.

**Varun Mohan** [2:26:59]
Oh, wow. That's cool.

**Swix** [2:27:00]
Yeah. So please, English. And then it.

**Varun Mohan** [2:27:02]
That's actually really cool because one of the things I think we believe in is actually I like products like autocomplete more than command, purely because I don't even want to open anything up. So that thing where I just can type and not have to press some button shortcuts to go in a different place, I actually like that too.

**Swix** [2:27:17]
Yeah. And I actually adopted Warp, the terminal Warp, initially for that because they gave that away for free. But now it's everywhere. So I can turn off a Warp and not give Sequoia my bash commands.

**Alessio** [2:27:29]
I'm with you.

No, I use Warp. No, no. Look. I use OK, I don't know. I'm going to go on a rant. Hopefully, somebody at Warp this is like Warp product feedback. But they basically had this thing where you can do kind of like pound and then write in natural language.

But then they have also the auto infer if what you're typing is natural language. And those are different. When you do the pound, it's only like it gives you a predetermined command. When you talk to it, it generates a flow.

It's a bit confusing of a UX. But going back to your post, you had the three P's of AI UX.

**Varun Mohan** [2:28:04]
What were they again?

**Swix** [2:28:05]
Present, practical, powerful. Actually, that was really good. I liked it.

**Alessio** [2:28:09]
Yeah. And I think in the beginning, being present was enough. Maybe even when you launch, it's like, oh, you have AI. That's cool. Other people don't have it. Do you think we're still in the practical where the experience is actually like the model doesn't even need to be that powerful.

Just having better experience is enough. Or do you think really the being able to do the whole because your point was like you're powerful when you generate a lot of value for the customer. You're practical when you're basically wrapping it in a nicer way.

Yeah. Where are we in the market?

**Varun Mohan** [2:28:40]
I think there's always going to be room for practical UX, like getting it. I mean, the command terminal thing, that's like very practical UX. I do think with things like Cascade and these agentic systems, we are starting to get onto powerful.

Because there are so many pieces from a UX perspective that make Cascade really good. There's really micro things that are just all over the place. But as we're streaming in, we're showing the changes. We're allowing you to jump and open diffs and see it.

You can run background terminal commands. You can see what's running. Background processes are running. There's all these small UX things that together come to a really powerful and intuitive UX. I think we're starting to get there. It's definitely just the start.

And that's why we're so excited about where all this is going to go. I think we're starting to see the glimpses of it. I'm excited. It's going to be a whole new ball game.

**Swix** [2:29:27]
Yeah. Awesome. First of all, it's just been really nice to work with you. I do work. A number of guest posters. And not everyone makes it through to the end. And nobody else has done it three times. So kudos.

**Varun Mohan** [2:29:39]
Patrick.

**Swix** [2:29:41]
This one was more like the money one, which it's funny because I think developers are quite uninterested in money. Isn't it weird?

**Varun Mohan** [2:29:50]
Yeah. I mean, I think I don't know if this is just the nature of our company. I think there's some people said there's all the San Francisco AI companies. And everyone's hyping each other on the tech and everything, which is great.

The tech's really important. We're here in Mountain View, beautiful office. We just really care about actually driving value and making money, which is kind of like a core part of the company.

**Swix** [2:30:09]
I think maybe the selfish way of saying that, or a little more of the selfless way, is like, yeah, we can be kind of like this VC-funded company forever. But ultimately speaking, if we actually want to transform the way software happens, we need this part of the business that's cash-generative that enables us to actually invest tremendously in the software.

And that needs to be durable cash, be cash that churns the next year. And we want to set ourselves up to be a company that is durable and can actually solve these problems. Yeah. Yeah. Excellent. So for people obviously, we're going to link in the show notes.

But for people who are listening to this for the first time, I had a lot of trouble naming this piece. So we originally called it you had how to make money something.

**Varun Mohan** [2:30:51]
I

apologize. I'm super bitty. I think I was writing for that during a plane flight. So I apologize for that.

**Swix** [2:31:01]
No, you had like $3 signed in the title.

**Varun Mohan** [2:31:02]
Oh, I absolutely had $3.

**Swix** [2:31:03]
I was like, I can't do that. So it's either building AI for the enterprise. And then I also said the worst, the most dangerous thing an AI startup can do is build for other AI startups, which I think you all, both of you, will co-sign.

And I think basically the main thesis which I really liked was go slow to go fast. If you actually build for security, compliance, personalization, usage analytics, latency budgets, and scale from the start, then you're going to pay that cost now.

But eventually, it's going to pay off in the long run. And this is the actual insight. You cannot do this later. If you build the easy thing first as an MVP, it's like, yeah, I'll just ship it with whatever's easy to do.

And then you tack on the enterprise-ready.io set of 12 things that you have, you actually end up with a different product. Or you end up worse off than if you had started from the beginning. So that I had never heard before.

**Varun Mohan** [2:31:52]
Yeah. I mean, we see that repeatedly. I mean, just likeright now, where we have a lot of customers in the defense space, for example, we're going through FedRAMP accreditationright now. And people that we're working with, they saw all the fact like, oh, yeah, we already have a containerized system.

We can already deploy in these manners. We can already do we've already gone through security. They're like, oh, you guys are going to have a much easier time doing this than most companies that are just like, OK, we have a big SaaS blob.

And now we need to do all these things. It might sound like a really deep thing. I think it's just anyone who's worked for an extended period of time at a company on a certain project has probably seen this happen.

The technology just keeps on improving. And then you realize that you have to now re-architect your whole system to get something improving. Just making that kind of change, when you've invested so much effort, people have poured in hours.

They're emotionally invested, whatever it might be. It's really hard to make that change. So I'm sure we're going to hit that also. Yes, I think we've done things a little bit earlier than most companies. I think we're going to hit points where we're going to see parts of our systems where we really need to re-architect that.

Actually, we've definitely hit that already. And I think that's just at the project level, the product level, or is that your whole company? I think the thesis behind here is to some degree, your company needs to have this DNA from the beginning.

And I think then you'll be able to go through those bumps a lot more smoother and be able to drive the value. I know Vernon can talk about more.

**Swix** [2:33:16]
Can I say two points? So first point I'd like to say is this is something that me and Douglas, my co-founder, talk about a lot. It's like there's this constant thing of build versus buy. I think the answer is a lot of the time, the answer should be buy.

We're not going to go build our own sales tool. We should go buy Salesforce. That's kind of dumb. That's undifferentiated. And the reason why you go with buy instead of build is, hey, look, the ROI of what exists out there is good.

From an opportunity cost standpoint, it's better to actually go out and buy it than build it and do a shittier job. There's a company that's actually going out and focused on that. But here's the hidden thing that I think is really important.

When you go out and buy, you're losing a core competency inside the company. And that's a core competency you can never get. Or it's very hard. Startups are so limited on time. Let me just say, let's say as a company, we did not invest in, I don't know, model inference.

Yeah, we have a custom inference runtime. We give that upright now. We will never get it back. It's going to be very hard to get it back. You can't just use VLM and TensorRT.

**Varun Mohan** [2:34:13]
That would be our only option.

**Swix** [2:34:14]
Or just let me put it. If we use VLM, we would not be talking with youright now. Yeah. But the point is this is more a question of I tried to think about it from first principles. Google is a great company.

It makes a lot of money. What happens if they actually made the search index of the product something that someone else built for them? It's like they could. Maybe someone else could have done a good job. Maybe that's a bad example.

But particularly because Google is a search index. But tough luck getting that core competency back. You've lost it. And I think for us, it's more a question of what core competencies do we need inside the business. And yeah, sometimes it's painful.

Sometimes actually some of these core competencies are annoying. Sometimes we'll be behind what exists out there. And we just need to be very honest. That's where the truth-seekingness of the company matters. Are we really honest about this core competency?

Can we actually keep up? If the answer is we truly can't keep up, then why are we keeping up with the trade? We should just buy. Let's not build. If the answer is we can, and we think that this will differentiatedly make our company a better company in the long term, then the answer is we need to.

We need to. Because the race is not won in the next year. The race is won over the next 5, 10 years, maybe even longer. So that's maybe one thing. And then the second thing, actually from the enterprise standpoint, I think one of the unique parts of the company now is we have both this individual and enterprise side.

And usually, companies stick to one or the other. And I think that needs to be part of the DNA, I think, kind of early on in the company. As Anshul said, I mean, there's stories of companies like Dropbox and stuff that tried.

And Dropbox is an amazing company, fantastic company, one of the fastest growing consumer companies of all time, consumer more on the software company of all time. But yeah, when you have everyone sort of product-oriented on the consumer side, the enterprise is just it's checking off a lot of boxes that ultimately do not help the consumer at all, doesn't help your growth metrics.

And effectively, if the original group of people didn't care, it's incredibly hard to get them to care down the line. It's incredibly hard. Why do it? And you need to feel like, hey, this is an important part for the company's viability.

So I think there's a little bit of the build versus buy part. And then also the cultural DNA of the company that I think are both really important. And yeah, it's something we think about all the time. I have the privilege of being friends with you guys off the air.

I don't feel like I think I know your work history. You say cultural DNA, but it's not like you've built giant enterprise SaaS before. Right?

**Varun Mohan** [2:36:29]
Yeah. I think yeah.

**Swix** [2:36:31]
So where are you getting this from?

**Varun Mohan** [2:36:33]
Yeah. In fact, I think the only other sort of I guess when I look at my previous internships, maybe Anshul can provide some context here. It's like I worked at LinkedIn and then Quora and then Databricks. And to be honest, I was not that interested in B2B ETL software that much.

That's not what drives me when I wake up at night. So because of that, I decided to go work at an autonomous vehicle company immediately after. I think part of it comes down to maybe a little bit of the unique aspect of the company and the fact that we pivoted as a company is we want to be a durable company.

And then the question is, how do you work backwards from that? There's a lot of things about being very honest about what we're good at and what we're not good at. I think surprisingly, enterprise sales is not something that I came out of the womb knowing how to do.

I didn't really know. And because of that, obviously, a lot of sales happened between folks like Anshul and I helping partner with companies. But very soon, we hired actually a VP of sales. And we've actually been deeply involved in the process of scaling out a large go-to-market team.

And I think it's more a question of what matters to the company and how do you actually go out and build it. And I think one of the people that I think about a lot actually is someone like Alex Wang.

He dropped out of college. He was a year younger than us at MIT. And he has figured out how to constantly change the direction of the company. Effectively, it starts out as a human task interface, then an AV labeling company, then a cataloging company, then now a generative AI labeling company.

And every time, the revenue of the company kind of goes up by a factor of 10, even though the business is doing something larger than that.

**Swix** [2:37:56]
Now it's all about military contracts.

**Varun Mohan** [2:37:58]
Yeah. Now it's probably going to be military. And then after that, it might be taking over the world. He's just going to keep increasing the stakes. And there's no playbook on how this really works. It's just a little bit of solve a hard problem and work backwards.

Work backwards from that. Right? And we'll get lucky along the way. I don't think you think everything from first principles to the best of our abilities. But there's just so many variable unknowns that, yeah, we don't know everything that's happening in every company out there.

And everyone knows how fast the AI space has been moving. We have to be pretty good at adapting.

**Swix** [2:38:28]
I want to double-click on one thing just because you brought it up. And it's a rare thing to touch on, VP of sales. We don't get to actually we talk to pretty early-stage founders mostly. They don't usually have a pretty built-out sales function.

Advice, what kind of sales works in this kind of field? What didn't work? Anything you can share with other founders?

**Varun Mohan** [2:38:46]
I think one of the hard parts about hiring people in sales and I really gram Anshul can also attest. We have an amazing VP of sales at the company. One of the things is if you're purely a developer, salespeople, their job is to talk really well, from in proper.

I mean, very obvious if you hear me talk. I'm not a very polished person.

**Swix** [2:39:05]
You're a great notebook. I don't know.

**Varun Mohan** [2:39:07]
Compared to most pure, pure salespeople. So actually just checking based on the way they speak is not that interesting. I think what matters in a space like ours that is very quickly, moving very quickly, I think is intellectual curiosity is very important, intellectual horsepower, understanding how to build a factory.

I'm not trying to minimize it. But in some ways, you need to build something incredibly scalable here. It's almost like every year, you're kind of making this factory twice, thrice maybe as big. Because in some ways, you have people that are counter-carrying.

You need some number of people. And you need to make the math work. And actually, the process of building a factory is not something you can just take someone who is a great rep at another company and just make them build a factory.

This is actually a very different skill. How do you actually make sure you have hundreds of people that actually deeply understand the product? Actually, Anshul works very closely also with sales to make sure that they're enabled properly, make sure that they understand the technology.

Our technology is also changing very quickly. Let's maybe take an example on how our company is very different than a company like MongoDB. When you sell a product like MongoDB, no one at the company is interested in how the data is being stored.

It's not that interesting. I love databases. I would be interested. But most people are like, solve the application problem I have at hand. People aren't curious about how our technology works. People are curious about RAG, people that are buying our technology.

And imagine we had a sales team that is scaling where no one understands any of this stuff. We're not going to be great partners to our customers. So how do you create almost this growing factory that is able to actually distribute the software in a way that is true to our partners and also at the same time taking on all the new parts of our product?

They're actually able to expound on the new parts of our product. So sorry, that was more a statement of building a scalable sales team. But in terms of who you hire, you just need to have a sense. In some ways, this is maybe an example of talk to enough people, find out what good looks like potentially in your category, and find someone who's good and humble and willing to work with you.

**Swix** [2:40:57]
Yeah. That's just generic hiring.

**Varun Mohan** [2:40:58]
It's just generic hiring.

**Swix** [2:40:59]
I think here, there's sales for AI or sales for AI infrastructure. And then there's also the sales feeding into products in the way that we're talking about here, where they basically tell you what they need. I imagine a lot of that happened.

**Varun Mohan** [2:41:14]
I think a lot of that happened. I mean, Vernon mentioned Vernon, myself, a number of other people who are developers by trade, engineers. We're pretty involved in the sales process because there is a lot to learn. Before we went out and hired a sales leader, yeah, if all we want is neither of us had ever done a sale for Codeium in our lives and we want to try to find a sales leader, we probably would have not hired theright person.

**Swix** [2:41:36]
Yeah. We had sold a product to like 30 or 40 customers at that time.

**Varun Mohan** [2:41:39]
We had done hundreds and hundreds of deal cycles ourselves personally without I mean, we read a lot of books. And we just did a lot of stuff. And we learned what messaging worked, what did we need to do.

And then I think we found theright person. Second, Vernon Graham's amazing. And who we brought on as our VP of sales. That just has to be part of the nature. And it doesn't stop now. Just because we have a VP of sales and people dedicated to sales, it doesn't stop that we can't be involved or engineering can't be involved.

We have lots of people. We hire plenty of deployed engineers. These are people I think Palantir kind of made this really famous.

**Swix** [2:42:13]
Deployed engineers.

**Varun Mohan** [2:42:14]
But deployed engineers work very, very closely with the sales team on very technical aspects because they can also understand what are people trying to do with AI.

**Swix** [2:42:21]
As in, they work at Codeium as deployed engineers?

**Varun Mohan** [2:42:24]
Yeah. And they partner with our account executives to make our customers successful and learn what is it that people are actually getting value with AI. And that's information that we keep on collating. And it's like we will both jump into any deal cycle just to learn more because that's how we're going to just keep on building the best product.

It comes back to the same care. I don't know. And hopefully, we build theright thing.

**Swix** [2:42:47]
Cool, guys. Thank you for the time. And it's great to have you back on the pod.

**Varun Mohan** [2:42:51]
Yeah. Thanks a lot for having us. Hopefully, in a year, we can do another one.

**Swix** [2:42:53]
Yeah. You'll be 10 billion by then.

Yeah. Exactly. At this rate, then next year.

**Varun Mohan** [2:42:58]
We try not thinking about that.

**Swix** [2:43:00]
Try to not be a zero billion company.

**Varun Mohan** [2:43:02]
That's there's that. Yeah.

**Swix** [2:43:04]
Allright. Cool.

**Varun Mohan** [2:43:04]
Awesome.

**Swix** [2:43:28]
Allright. How are we doing, New York?

### Windsurf Waves

**Swix** [2:43:32]
So my name is Kevin. This is our first ever Windsurf presentation. So you could say it's the first time we're kind of spilling the beans on what the idea is all about. So thank you all for coming. I'm going to be talking about Windsurf, the first AI agent-powered editor.

So my name is Kevin Hou. I lead our product engineering team. We're a team based out of San Francisco. And thank you so much to Swix and Ben and the whole AI Engineering Summit team for inviting us here and letting us speak to you all.

It's been a pleasure talking to people in the audience at the booth and just generally talking about AI. So let's dive into it. Windsurf is an agentic editor. And we're going to talk a little bit about some of the principles that we use when we're building a product like this.

So we believe that agents are the future of software development. And you all are here. So you kind of understand the power of what agents can do both for software engineering and otherwise. But to start, I'm going to take you down a trip down memory lane.

Let's go back to 2022. Copilot was the state of the art. It just came out of beta. People were experiencing the ghost text. They were seeing their completions. And it was one of the first times that people really got to see the magic of what AI could do for developers.

It was making them more productive. And we, Codeium, decided we were going to be one of the first companies to also launch an autocomplete product. So we garnered a couple million users on our VS Code, JetBrains, Vim, Emacs extensions.

Raise your hand if you were one of those Codeium users. Nice. Nice. But we always knew that intelligence was going to get better. Back then, we were doing short completions, maybe finishing your functions. But we knew that there were going to be better models, larger models, better training paradigms, completely new RL, new tool use, all this stuff.

And so we knew that we wanted to build the best experience for devs possible. So even back then, we started looking at agents. We started thinking about what could the future of software development be if models just got bigger.

And so we built the best experience that we could at the time. And that was a chat autocomplete product. But we always knew that copy-pasting from ChatGPT was going to be a thing of the past. We also knew that people were going to probably tab less.

We were going to have LLMs that we'd be able to generate more and more. And who knows? We always think agents are the best now. But as a company, we're always thinking about the future. We're technology optimists. So if an IDE in the future, who knows?

We might not even be writing code inside of IDEs. We'll just be there building the best product for SWEES. And so this year, 2025, is finally the year where I feel like we are all recognizing the power of agents inside of software development.

Agents are here to stay. And Windsurf, I'm proud to say, is pushing the envelope of that technology. And we're going to talk about some of those features. And we're going to keep pushing that agentic future because we believe that agents are going to move software engineering in a direction that no other LLM has done in the past.

So this slide, I guess, is titled "Vibe Coding with Windsurf" or also just "Coding in Windsurf." So I'm going to give you a quick demo about this is the Windsurf product. Here, you have a sidebar. This is our agent.

And you can see that we're going to be building a Python web scraper. So what this is going to do, it's going to build a Python web crawler, give us some stats about the website. And you can see it's actually installing dependencies from pip.

It's doing so inside of the terminal that you use so you can interact with it. It's suggesting edits, setting up your virtual environment. And we give a user a very helpful accept and reject so that you can go through and have confidence that the code that it's generating works for you and your code base.

This is, of course, there's a lot more features under the hood. Some of the things that our users like to do, they like to look up documentation. We have web search enabled by default. It always looks at your code base so you can grasp through your code base.

We can generate commit messages. You can drag and drop images. The possibilities are truly endless. And I'm going to be talking about these features are powered by a handful of principles, a handful of through lines that we as an engineering team hold true as we're building.

And as a team, we always go back to the same mission, which is to keep you in the flow and unlock your limitless potential. We want to work on we want to handle the grunt work for you. We want to handle looking at your debug stack traces.

We want to handle modifying your original source code. We want to pull the correct version of documentation so that you never have to worry about pulling in the correct context. These are problems that we are trying to solve.

And we want you to spend time on things that you are good at, the things that make us all excited, which is shipping products, building great features, and generally just shipping code.

And so with that goal in mind, how do we tell what to work on? It's a game of input and output. So we want to allow users to give the least amount of explicit input possible to produce the most correct and production-ready code.

We want you to contribute less. And we want our agent to contribute more. And we do this by reducing the amount of human-in-the-loop required by doing things like background research. We are always trying to predict your next step.

And we'll make decisions on your behalf so that you can move faster. And this might all seem like a fantasy, but Windsurf launched three months ago on November 13. That date is forever branded in my memory. And these are the results that we're already seeing.

So in three months, we've been generating 4.5 billion lines of code. That is an absurd number. And since the time I started this presentation, we've actually probably sent users have probably sent thousands of messages to Cascade asking it to refactor code, to write new features, to build new pages on their website.

And also, a fun statistic since we're all engineers here, we've had 16 nights in the last 90 days where we've been woken up in the middle of the night from PagerDuty on-call because we've had some reliability issues due to us exceeding our capacity.

These problems, we've had immense success getting people onto the platform. And we've been very fortunate to have the issue of being some of Anthropic and OpenAI's largest consumers. And so with this mission and metric in mind, let's walk through some of the principles that we use when we're building this agentic editor.

And for those of you that have used Windsurf, you might learn about some of the new ways that you could use the product. And also, for my own curiosity, how many of you have heard of Windsurf just so I know who we're talking about?

Oh, let's go. That's sick. How many of you use Windsurf? OK. Everyone who put their hand down? Doors over there. Allright. Let's get into it. So the first principle, trajectories. What is a trajectory? We use trajectories to read your mind.

So unlike other editors like Cursor, the elephant in the room, our agent is deeply integrated into the editor. And we'll talk about what exactly that means. But on one half, you can imagine an agent has to understand what you're doing.

And then on the other half, it either has to understand and be able to execute things on your behalf. And this has led to features like one of my favorites, quote, "continue my work." So we are building up an understanding of the user as you're writing code, as you're executing terminal commands.

And then you can actually just go into the agent sidebar and just say, "continue my work." And it'll actually continue executing that. And it might even give you a full PR or a full commit. We also have things like terminal execution mode.

It can automatically use the LLM to decide what is safe and not safe so that if you're running something like Git, it'll just work. Or it'll probably prompt you if there's an RMRF somewhere. You probably don't want to run that automatically.

And the LLM will be like, oh, we should probably flag to the user to confirm this. These are just some of the ways that we try and let the human be in the loop but as minimal as possible.

And then finally, we also have a stellar UX, a stellar design team that's been working on how to integrate these sort of cutting-edge features into a product in a way that allows the user to feel like they're in control, to be able to accept and reject changes into their code so they can have confidence in the code that they're pushing to production.

So here is how a trajectory works. We have this notion of a unified timeline. So an agent is working in the background, behind the scenes, to understand what the user is implicitly doing. So this includes things like viewing files, navigating around your code base.

Let's say you edit a file. And then the agent will edit a file. This all kind of goes into a shared timeline of actions. And you can imagine this includes things like searching, greping, making edits, making commits. The user has this sort of holistic understanding of what you're doing.

And this entire experience is unified by this shared timeline. So you can contribute to it. It can contribute to it. And in this way, you never run into the problem where you're talking to the agent and it undoes the change that you just did or has some outdated notion of what the file state is.

So this is a first-class principle of ours. And when we decided we were going to build an editor, we were going to build it around this notion of an agent and a shared timeline.

And so here's an example of this feature in action. Here, we're adding a new function. And you're seeing the autocomplete and all the kind of bells and whistles of that feature. And in theright side, we just ask, "continue my work."

This is a new function. We probably want our form handler to use this new function. And you can see based on the context that we gave it by making edits, it's guessing. OK, we probably want to make this file change to this file, maybe some others.

And then at the end, it's actually just saying, OK, let's just run npm run dev. And it can run terminal commands on your behalf in the background in your kind of command J terminal pop-up. And in this way, we're keeping you in the flow.

Something that would have taken minutes is now taking seconds. And here's another example. The terminal is now deeply integrated into the agentic timeline. So here, if you're typing commands, the classic example is like I npm install a new package or I pip install a new package.

The agent should know, oh, you just installed this package. Why don't we go ahead and implement it into your project? And based on context that it's able to pick up around the code base, it can continue that line of work.

So we very strongly believe in a future of no copy-paste. You should never have a situation where you're in a terminal, or you're in a document, or even on a website, and you're copy-pasting text into an agent. That's just not how the way the world works.

And in the same way, we strongly believe the future is not going to be at terminal.

Here's another example of commands running inside of your terminal. I've been talking about this for a little bit. And this concept of a trajectory allows us to automatically execute things inside of a sandbox that is as similar to the way you run commands as possible.

So instead of running some shell script in the background, what we do is we put thisright inside of the place that you would actually write terminal commands. So if you pip install something or it pip install something, it's going to the same environment.

You'll never have this instance of kind of weirdness. And this is all part of our effort to bring these two sides, the agentic side and the human side, close together, as close together as possible. And you do this through building a unified product.

We believe that developers are here to stay. And if you want to work seamlessly with a developer, that means the agent has to understand what they are thinking. Windsurf has to be ubiquitous. And the agent will be reading more and more of your mind, doing things that you might not even know it's doing.

In the future, we'll be looking not just one to five steps in the future, but 10, 20, 30 steps into the future. It'll be writing unit tests before you've even finished defining the function. It'll be performing code base-wide refactors on multiple files based on you just simply editing a variable name.

All this is part of this unified trajectory concept. Now, the second principle is meta-learning. So even if Windsurf understands what you're doing in the moment, there is still an inferred understanding of your code base and your preferences and your organizational guidelines that, let's just say, senior engineers at your company have built up a notion of over time.

We call this concept meta-learning. So Windsurf, we've built from the ground up to adapt and remember these things about you and your company. So if you think about a frontier LLM, the best LLMs that exist in the world, they're very, very smart engineers, definitely more capable than I, probably more capable than most of you.

They can just write an enormous amount of code and do so correctly. And it probably runs and compiles pretty well. But what they do not have is the exposure that you've had, the education that you've had, and the ability to kind of remember and know how you personally or your company writes code.

And so what does this mean for our product? We've implemented a concept called auto-generated memories. So over time, we build up a memory bank of what you are doing. So you can say, remember that I used Tailwind version 4?

Or remember that I used React 19 instead of 18? And these things will be remembered. You say them once, and they'll be remembered forever. We also allow people to implement things like custom MCP servers. So you can plug in your favorite tools.

We can adapt to your workflow. We will also allow you to whitelist and blacklist commands, going back to that same concept. We want to keep you in the flow as least or, sorry, we want to keep you in the flow as much as possible.

But we can tell the agent, hey, never run an RM command without my approval. And so in this way, it learns about your preferences over time. And if you think about what makes a developer effective, it's because they remember things that you tell them.

And Windsurf must also model this behavior if we hope that AI should write and maintain projects for us. So in the short term, this means you don't need to prompt the agent again and again to do the same thing over and over.

But in the long term, the AI should just feel like a seamless extension of yourself. It's this idea of explicit versus inferred context.

And we always have the saying at the company, ideas are cheap. So here's an example of auto-generated memories in action. Here, we're not even explicitly telling it, remember this thing. We're just giving it an architecture overview. We're asking, what does this project do?

And it's remembering based on a couple of tool uses. It's looking at a couple of different files, looking at the routes. And now it's committed to memory. Hey, this is the project that this person is working on. Here are the endpoints that are available.

And we can reference that in the next message that we send. So in the future conversation, we can now one-shot things because we have a notion of a memory bank. In the same way, documentation is auto-learned. We know what packages you're using because of your package JSON, because you've explicitly told us.

And we're able to look up the web, look on the web for documentation that matches those versions. And we do so all implicitly. And so the dream of meta-learning is that you can have an entirely inferred sense of context based on a code base or based on the usage of the product.

And auto-generated memories are a step in that direction. We strongly believe that having a rules file we do allow users to add a rules file. We strongly believe that a rules file is a crutch. By the end of 2025, 99% of the things that you're going to put in a rules file will be interpreted or inferred based on your code base or your usage.

So our dream is that every single Windsurf instance, every single user using Windsurf, regardless of the company or the type of person, the skill of the developer, will be personalized to that user. And you'll only have to tell it one thing.

And finally, my favorite principle, which is scale with intelligence. So what does this mean? Now that Windsurf understands what you're doing in the moment, the first principle, and can improve over time, the second principle, how do we actually build an agent that will scale with the rates at which LLMs are scaling?

And while we're trying to always give you the best tool today, we recognize that new models are coming out every other week. Every day, there's some new article about some new pattern. And it's really, really hard to keep up.

But we always think at Codium, how do we stay on top of this? How do we build the best product for not just today, but three months, six months, 12 months out, three years from now? So in 2021, when ChatGPT came out, you probably like me, we all had our imaginations running wild.

We were like, OK, we're going to solve AGI, post-economy, whatever. But obviously, there's a lot of things that need to happen between then and that future. And so models at that time were, quite frankly, a little bit too dumb to be able to accomplish everything that we wanted them to do.

So we built up a lot of infrastructure. And you and I have all probably done this. We build out embedding indices. We build retrieval heuristics. We have output validating systems to make sure that the code that it's generating is good.

These are all things that we're able to help at the margin. But this is all predicated on the assumption that we were operating with a fixed notion of intelligence. 2021, 2022, these models we were operating, we were building all this infrastructure to compensate for areas and edge cases that models could not handle.

And what's very different about the way we're approaching Windsurf is that we want our product to scale with the models. So if the models get better, our product gets better. And I'll give you one such example. It kind of surprised me.

When I landed in New York, I tweeted that we deleted chat in Cascade. I was like a very I don't know. I had thoughts. And weirdly, a lot of you picked this up. And this is an example of something that we feel very strongly about.

One example of this principle in practice is that we deleted chat. So what does this mean? We only have an agent. And it's called Cascade inside of Windsurf. Chat is a legacy paradigm. And we completely replaced it. And as you can see here, users are enjoying it.

Or in fact, they might not even know the difference, but they're just enjoying the higher quality. An example of this is App Mentions. We built App Mentions. And probably you all have used App Mentions because context was not very good a year or two years ago.

Today, Windsurf can dynamically infer the relationships between bits of code and documents. 90% of the time, you do not need to App Mention something. All you need to do is let the retrieval system and the agent kind of plan out what it needs to do and then reconstruct the context automatically for you.

So App File and App Web, these are very helpful patterns when you're working at kind of the margin. But these are eventually eking out basis points. In the long term, we believe that LLMs are going to improve. And they already have improved to the point where you don't need to explicitly specify an App Mention.

The LLM should be intelligent enough to pick it up. And so in this example, previously, I was implementing Supabase inside of a Next.js app. Previously, you'd be at webbing. You'd be at docs, at code base, at this, at that.

No, just add Supabase. And it's able to infer and plan out. Let's search the web. Let's behave like a human would. And to get into this, there's also web search built into Windsurf. And what's very special about this is that it reads the web the way a human would read the web.

So instead of these hard-coded rules, we probably could have created an embedding index. But we would probably get very low-quality results. And so instead, we said, the LLMs are very, very good. Let's let the model decide what it wants to do.

Let's have it decide which search results to read, what parts of the page to read, and then finally give us an answer. And so we believe that as models will continue to get better, we're going to be continuing to do unsupervised work.

We're going to generate full PRs. We're going to read complex documentation. The possibilities are truly endless. And so here are some of the principles that we just talked about. Where are we going with this? There's a lot of ways we can take this.

The engine underneath Windsurf is really, really the secret sauce. And we believe that we're going to be 2025 is going to be a whole new world. No rules files, generating PRs, generating commits. It's going to be crazy. And we're already seeing this.

90% of our users or, sorry, all of our users, 90% of the code that they're writing is generated with Cascade. That's an astonishing number. Autocomplete was more in like the 20%, 30%. This is insane. People are using agents today to accomplish so much more than they could have in the past.

And we're all software engineers. I want to make sure that every single person in this room is armed with the best tools. And those best tools are agents. And like every good thing in the city,

I expect tips. 25% of your ticket price, which I heard was quite a lot. Here's the actual QR that you're probably curious about. This is Windsurf's download link. We offer a free tier. So go ahead and scan that.

Start using the magic today.

And then finally, we have some killer swag at our booth. You can also connect with me on Twitter. I try and stay active with the community. But thank you so much for watching. I hope that you all learned something about how we're building at Windsurf and enjoy the rest of the conference.

Yeah, well, thank you guys so much for having me. It's exciting to be back. I was last here at AI Engineer one year ago. And it's kind of crazy. I've always been I've been telling Swix that we need to have these conferences way more often if it's going to be about AI software engineering.

### Agent Evolution

**Swix** [3:03:40]
It probably should be like every two months or something like that with the pace of everything's done. But it's going to be fun to talk a little bit about what we've seen in the space and what we've learned over the last 12 or 18 months building Devin over this time.

And I want to start this off with Moore's Law for AI agents. And so you can kind of think of the capability or the capacity of an AI by how much work it can do uninterrupted until you have to come in and step in and intervene or steer it or whatever it is.

And in GPT-3, for example, if you were to go and ask GPT-3 to do something, it could probably get through a few words or so. And then it'll say something where it's like, OK, this is probably not theright thing to say.

And GPT-3.5 was better. And GPT-4 was better. And so people talk about these lengths of tasks. And what you see in general is that that doubling time is about every seven months, which already is pretty crazy, actually. But in code, it's actually even faster.

It's every 70 days, which is two or three months. And so if you look at various software engineering tasks that start from the simplest single functions or single lines, and you go all the way to we're doing tasks now that take hours of humans' time.

And an AI agent is able to just do all of that. And if you think about doubling every 70 days, I mean, basically, every two to three months means you get four to six doublings every year, which means that the amount of work that an AI agent can do in code goes something between 16 and 64x in a year, every year, at least for the last couple of years that we've seen.

And it's kind of crazy to think about. But that sounds aboutright, actually, for what we've seen. 18 months ago, I would say, the only product experience that had PMF in code was just tab completion. It was just like, here's what I have so far.

Predict the next line for me. That was kind of all you really could do in a way that really worked. And we've gone from that, obviously, to full AI engineer that goes and just does all these tasks for you and implements a ton of these things.

And people ask all the time, what

is the future interface? Or what is theright way to do this? Or what are the most important capabilities to solve for? And I think, funnily enough, the answer to all these questions actually is it changes every two or three months.

Every time you get to the next tier, the bottleneck that you're running into or the most important capability or theright way you should be interfacing with it, all of these actually change at each point. And so I wanted to talk a bit about some of those tiers for us over the last year or so.

And over the course of that time, obviously, when we got started in the end of 2023, obviously, agents were not even a concept. And now everyone's talking about coding agents. People are doing more and more and more. And it's very cool to see.

And each of these has kind of been almost a discrete tier for us. And soright around a year ago, when we were doing the last AI engineer talk, actually, the biggest use case that we really saw that was getting broad adoption was what I'll kind of call these repetitive migrations.

And so I'm talking like JavaScript to TypeScript or upgrading your Angular version from this one to that one or going from this Java version to that Java version or something like that. And those kinds of tasks in particular, what you typically see is

you have some massive code base that you want to apply this whole migration for. You have to go file by file and do every single one. And usually, the set of steps is pretty clear. If you go to the Angular website or something like that, it'll tell you, allright, here's what you have to do.

This, this, this, this, this. And you want to go and execute each of these steps. It's not so routine that there's no classical deterministic program that solves that. But there's kind of a clear set of steps. And if you can follow those steps very well, then you can do the task.

And this was the thing for us because that was all you could really trust agents to do at the time. You could do harder things once in a while. And you could do some really cool stuff occasionally. But as far as something that was consistent enough that you could do it over and over and over, these kinds of repetitive migrations that you would be doing for 10,000 files were, in many ways, the easiest thing, which was cool, actually, because it was also kind of the most annoying thing for humans to do.

And I think that's generally been the trend where AI has always done these more boilerplate tasks and the more tedious stuff, the more repetitive stuff. And we get to do the more fun, creative stuff. And obviously, as time has gone on, it's taken on more and more of that boilerplate.

But for a problem like this one, a lot of what you need to do is you need Devin to be able to go and execute a set of steps super reliably. And so a lot of this was, I would say, the big capabilities problems to solve was mostly instruction following.

And so we built this system called Playbooks, where basically, you could just outline a very clear set of steps, have it follow each of those step by step, and then do exactly what's said. Now, if you think about it, obviously, a lot of software engineering does not fall under the category of literally just follow 10 steps step by step and do exactly what it said.

But migration does. And it allowed us to go and actually do these. And this was kind of, I would say, the first big use case of Devin that really came up. I think one of the other big systems that got built around that time, which we've since rebuilt many times, is knowledge or memory, which is if you're doing the same task over and over and over again, then often the human will have feedback on, hey, by the way, you have to remember to do X thing.

Or you need to do Y thing every time when you see this. And so basically, an ability to just maintain and understand the learnings from that and use that to improve the agent in every future one. And those were kind of the big problems of the time.

And that was summer of last year. And around end of summer or fall or so, I think the kind of big thing that started coming up was as these systems got more and more capable, instead of just doing the most routine migrations, you could do these more still pretty isolated but a bit broader of these general kind of bugs or features where you can actually just tell it what you want to do and have it do it.

And so, for example, hey, Devin, in this repo select dropdown, can you please just list the currently selected ones at the top? Like having the checkboxes throughout just doesn't really and Devin will just go and do that. And so if you think about it,

it's something like the kind of level of task that you would give an intern. And there are a few particular things that you have to solve for with this. First of all, usually, these changes are pretty isolated and pretty contained.

And so it's one, maybe two files that you really have to look at and change to do a task like this. But at least, you do still need to be able to set up the repo and work with the repo.

And so you want to be able to run lint. You want to be able to run CI, all of these other things, to at least have the basic checks of whether things work. One of the big things that we built around then was the ability to really set up your repository ahead of time and build a snapshot that you could start off, that you could reload, that you could roll back, and all of these kinds of primitives as well.

So having this clean remote VM that could run all these things, it could run your CI. It could run your linter and so on. But that's when we started to really see, I would say, a bit more broad of value.

I mean, migrations is one particular thing. And for that particular thing, we were showing a ton of value. And then we started to see where, with these bug fixes or things like that, you would be able to just generally get value from Devin as almost like a junior buddy of yours.

And then in the fall, things really moved towards just much broader bugs and requests. And here,

most changes again, we're jumping another order of magnitude. Most changes don't just contain themselves to one file. Often, you have to go and look, see what's going on. You have to diagnose things. You have to figure out what's happening.

You have to work across files and make theright changes. Often, these changes are hundreds of lines. If it's like, hey, I've got this bug. Let's figure out what's going on. Let's solve it. And there are a lot of things here that really started to make sense and really started to be important.

But one in particular, I'll just point out, was there's a lot of stuff that you can do with not just looking at the code as text, but thinking of it as this whole hierarchy. So understanding call hierarchies, running a language server is a big deal.

You have Git commit history, which you can look at, which informs how these different files relate to one another. Obviously, you have your linter and things like that. But you're able to kind of reference things across files. And so one of the big problems here, I think, was kind of working with the context of it and getting to the point where it could make changes across several files.

It could be consistent across those changes. It would be able to understand across the code base. And here was really the point, I would say, where you started to be able to just tag it and have it do an issue and just have it build it for you.

And so Slack was a huge part of the workflow then. And

it made sense because it's where you discuss your issues. And it's where you set these things up. So you would tag Devin in Slack and say, hey, by the way, we've got this bug. Please take a look. Or could you please go build this thing?

This is especially fun part for us because this isright around when we went GA. And a lot of that was because it got to the point where you truly could just get set up with Devin and ask it a lot of these broad tasks and just have it do it.

But a lot of the work that we did was around having Devin have better and better understanding of the code base. And if you think about it from the human lens, it's the same way, where on your first day on the job, for example, being super fresh on the code base, it's kind of tough to know exactly what you're supposed to do.

A lot of these details are things that you understand over time or that are a representation of the code base that you build over time. And Devin had to do the same thing and had to understand, how do I plan this task out before I solve it?

How do I understand all the files that need to be changed? How do I go from there and make that diff?

And around the spring of this year, again, every gap is like two or three months. We got to an interesting point, which is once you start to get to harder and harder tasks, you, as the human, don't necessarily know everything that you want done at the time that you're giving the task.

If you're saying, hey, you know I'd like to go and improve the architecture of this. Or this function is slow. Let's profile it and look into it and see what needs to be done. Or hey, we really should handle this error case better.

But let's look at all the possibilities and see what theright logic should be in each of these. And basically, what it meant is that this whole idea of taking a two-line prompt or a three-line prompt or something and then just having that result in a Devin task was not sufficient.

And you wanted to really be able to work with Devin and specify a lot more. And around this time, along with this kind of better code base intelligence, we had a few different things that came up. And so we released DeepWiki, for example.

And the whole idea of DeepWiki was, funnily enough, is Devin had its own internal representation of the code base. But it turns out that for humans, it was great to look at that, too, to be able to understand what was going on or to be able to ask questions quickly about the code base.

Closely related to that was search, which was the ability to really just ask questions about a code base and understand some piece of this. And a lot of the workflow that really started to come up was actually basically this more iterative workflow, where the first thing that you would do is you would ask a few questions.

You would understand. You would basically have a more L2 experience where you can go and explore the code base with your agent, figure out what has to be done in the task, and then set your agent off to go do that.

Because for these more complex tasks, you kind of needed that. And so that was, I would say, kind of like a big paradigm shift for us then is understanding this is what also came along with Devin 2.0, for example, and the inIDE experience, where often, yeah, you want to be able to have points where you closely monitor Devin for 10% of the task, 20% of the task, and then have it do work on its own for the other 80%, 90%.

And then lastly, most recently, in June, which is now, it was kind of, yeah, really the ability to just truly just kill your backlog and hand it a ton of tasks and have it do all these at once.

And if you think about this task, in many ways, I would say it's almost like a culmination of many of these different things that had to be done in the past. You have to work with all of these systems, obviously.

You have to integrate into all these. Certainly, you want to be able to work with Linear or with Jira or systems like that. But you have to be able to scope out a task to understand what's meant by what's going on.

You have to decide when to go to the human for more approval or for questions or things like that. You have to work across several different files. Often, you have to understand even what repo is theright repo to make the change in if your org has multiple repos, or what part of the code base is theright part of the code base that needs to change.

And to really get to the point where you can go and do this more autonomously, first of all, you have to have a really great sense of confidence. And so rather than just going off and doing things immediately, you have to be able to say, OK, I am quite sure that this is the task.

And I'm going to go execute it now versus I don't understand what's going on. Human, please give me help, basically. But the other piece of it is this is, I think, the era where testing and this asynchronous testing gets really, really important, which is if you want something to just deliver entire PRs for you for tasks that you do, especially for these larger tasks, you want to know that it can test it itself.

And often, the agent actually needs this iterative loop to be able to go and do that. So it needs to be able to run all the code locally. It needs to know what to test. It needs to know what to look for.

And in many ways, it's just a much higher context problem to solve for is this testing itself. And that brings us to now. And obviously, it's a pretty fun time to see because now what we're thinking about is, hey, maybe if instead of doing just one task, it's how do we think about tackling an entire project?

And after we do a project, what goes after that? And maybe one point that I would just make here is we talk about all these 2Xs that happen every couple of months. And I think from a kind of cosmic perspective, all the 2Xs look the same.

But in practice, every 2X actually is a different one. And so when we were just doing tab completion, single line completion, it really was just a text problem. It is just like taking the single file so far and just predict what the line is next.

Over the last year or year and a half, we've had to think about so much more. How do you work with the human in Linear or Slack or Jira? How do you take in feedback or steering? How do you help the human plan out and do all these things?

And moreover, obviously, there's a ton of the tooling and the capabilities work that have to be done of how does Devin test on its own? How does Devin make a lot of these longer-term decisions on its own? How does it debug its own outputs or run theright shell commands to figure out what the feedback is and go from there?

And so it's super exciting now that there's a lot more. There's a lot more coding agents in the space. It's very fun to see. And I think that we're going to see another 16 to 64X over the next 12 months as well.

And so, yeah, super, super excited. Awesome. Well, that's all. Thank you guys so much for having me.

**Alessio** [3:18:55]
Hello.

### Swe1 Model

**Alessio** [3:18:58]
How are we doing?

Allright. How's the energy level? Good, good. Yes. Let's go. Let's go. Two more. Two more. My name is Kevin. I lead product at Windsurf. And I'm super excited to be back here. Thank you so much, Swix and Ben.

It's always a pleasure to come back to AI Engineer World's Fair. The velocity of our industryright now is incredible. It's like being on a kite on the ocean. And we're really excited to see where the winds are taking us.

A year ago, we didn't have Windsurf. People were coding with autocomplete. No one had heard of an agent. And now, the Windsurf editor is being used by millions and millions of people all around the world. And hopefully, this is a larger number than last time.

How many people have heard of Windsurf? And how many people have used Windsurf?

Good numbers. Good numbers. We got to improve that.

And Windsurf itself has changed immensely in the last six months since its launch in November. We retired the name Codeium because we decided to catch this new wave, which is, by the way, what we call our next generation innovations in the product.

We call them waves. And in case you missed it, we are now 10 waves in. And some of the key waves we've been really excited about: web search, MCP support, auto-generated memories oh, I was supposed to do that auto-generated memories, deploys, and parallel agents, to name just a few.

And as the waves keep growing, as do the number of people that have discovered and integrated Windsurf into their daily workflows. To this day, we are generating about 90 million lines of code every single day. And that equates to around 1,000 over 1,000 messages sent every single minute.

But today is not about growth. I'm not going to sit here and tell you about the numbers. I'm going to tell you about the why. Why do people feel connected to the Windsurf editor? And I know no AI company really wants to disclose its secrets.

But I had to come up with some content. So today, I'm going to let you in on one of ours. Our secret sauce is a shared timeline between the human and the AI. And this is what makes people feel like we're reading their minds.

And now, everything you do as a software engineer can be thought of on this shared timeline. So if we rewind way back to the dark days, this is pre-autocomplete when everyone knew how to write a for loop. AI had to do everything.

You had to edit files. You had to type every single character. Imagine that. But then, once services like Copilot, like Codeium, they launched, then devs got really excited. They started seeing a small percentage of their code being written by AI.

And we started to abstract and accelerate the number of small edits, small actions that we would do for a user. And in late 2024, with the advent of Windsurf's agent and the launch of the Windsurf editor, we saw that we could do more and more for the user.

We started being able to edit multiple files at once, perform background research across thousands and thousands of files, and execute terminal commands directly inside the editor. But at Windsurf, we're in the business of trying to change how software gets created.

And this means that the timeline is actually a little bit more complicated. It needs to handle actions taken outside of just the IDE.

And so given how much of a developer's workflow happens outside of the editor, what does this mean for Windsurf? First, Windsurf is going to be everywhere.

Specifically, Windsurf will need to be able to read and ingest context from every single source that a developer uses. And if we zoom out and think about what makes you all software engineers successful, there are a couple of different categories, the first of which coding-related: file reads, running terminal commands, seeing your history, even which tabs you have open inside of your editor.

This all informs how to generate the correct code. But it goes beyond that. There's external sources, things like going on to GitHub and viewing a past history of commits, maybe looking at a PR that is doing something similar to the feature you're about to implement, doing online searches, web searches, looking at documentation.

And then there's the third category. And this is where it gets a little bit interesting. It's called meta-learning. It's the idea of what separates a junior engineer from a senior engineer, from a staff engineer. These are the organizational best practices, the engineering preferences that all get encoded into what makes good code.

And so if we think about what this means in practice, let's say that we are going to build a new page on a Dataviz dashboard. Let's walk through step by step. So first, you would probably start in Slack, as all good things start from Slack.

You'll build context, looking at a bunch of maybe customer requests. Maybe you'll have some internal messages. You'll collect that context. And you'll start planning. And this means you're going to be in Google Docs. You're going to be writing design docs, probably working on some infrastructure designs.

You're going to be tracking tickets inside of Jira. And then you might have a designer who's actually working in Figma in parallel, putting together all this material. And then finally, the fun part, or at least this is my favorite part, which is the actual writing of the code.

And hopefully, I use something like Windsurf to do so. But you're not done from there. Once you're code complete, you still have to open the PR. You got to get reviews. You got to merge into main. You got to deploy, SEO, analytics.

The list goes on and on and on. And this is really why we've built what we've built, because we know that for you, it's extremely important that we can fetch context from your Google Docs, that we can read your Figma files, and that we can one-click connect to any MCP service so that you can access your information in things like Notion, Linear, Stripe, and countless others.

And we've spent the last 10 waves making sure that Windsurf can be ubiquitous. But we know that's also not enough. We know it's not enough just to read. We need to be able to do and write everything. We need to be able to do it all for you.

And so the AI has to take action on a wide variety of surfaces beyond just the coding surface in order to accomplish what a human software engineer would do. And so this doesn't mean just write code. This means interacting with third-party services, provisioning API keys, writing design docs, PRDs, wireframing, testing.

And the list could go on and on and on. And so for the last six months, we've oriented ourselves around how do we do everything.

And if we go back to this concrete example of building a new web app, where do we start? We start by running code-based relevant terminal commands. This is something that we launchedright at the advent of Windsurf. And what's really cool about what we can do here is that we can intelligently decide which commands we want to run automatically and which ones we want to wait and ask for explicit user approval.

Next, you'll open up Windsurf browser previews, which allows you to iterate from there. It allows you to visually iterate with the agent so that Windsurf can take control of Chrome just like you would, inspecting DOM elements, looking at your JS console, being able to do what a web developer would do.

And so now, you could say, our app is code complete. We'll use the GitHub MCP to open up a pull request. And we can use context from your other PRs to be able to inform the description and inform the test plan.

And code review is still a necessary part of any software company. And so we launched Windsurf reviews, which can automatically leave comments and suggest changes asynchronously so that you can be confident that the code that hits main is production ready.

And so now that your code is merged, you'll want to be able to deploy. And so we also released a one-click service to Netlify so that you can use Windsurf's custom tool integrations to actually just in one click, the agent will deploy what you have to the live web.

And so as you can see, we've really built the ability for Windsurf to read everything that you can and do everything or almost everything that a software engineer can. So then you might ask, what's next? It's only inevitable that Windsurf will be on all the time working for you even when you don't know it.

We pioneered the agentic, human-in-the-loop synchronous workflows back when we released Windsurf in 2024. And today, timelines are 80% to 90% agent, 10% to 20% human. But we're trying to build towards a future that gets the 99% agent and 1% human.

We only want to ask the user for final approval. And as more and more of these timelines and workflows become AI-powered, it becomes possible to have Windsurf working for you at all times, not only as you type and use autocomplete and tab, but also in the background, researching when you're working, fully in parallel, only asking you to approve.

And we want to build this future where you can code anytime. You can write software at any time. This includes your bed. This includes the toilet. When you're on the bus, voice-activated Alexa, the possibilities are endless.

And so now that we've defined the problem, it's a little bit more structured. You could say, allright, we'll throw GPT. We'll throw Gemini at this timeline problem. But then from there, where do we go? How do we improve?

And specifically, how is Windsurf able to tackle this problem of the timeline? And if we take a step back, this really doesn't look like we're writing code anymore. This looks significantly more complicated than your average competitive programming question.

Windsurf wants to revolutionize the way that software gets built. It's not just how code gets written. We are solving a broader set of tasks than just code. And while the industry focuses heavily on things like SweBench, we know that the future is not going to be tokens in, tokens out.

Software engineering workflows are going to be much messier than this. It means that you have to be able to pick up tasks mid-workflow. You have to be able to deal with messy code-based states mid-commit. And you will have to work with tools that are outside of the editor.

And so we have to be able to ingest and perform over this broad set of actions on this timeline to keep our users in the flow. We have to be able to open up PRs. We have to know when to access analytics.

We need to know how to debug your CI/CD all by itself. And this problem starts to look really, really different from what people are evaluating on. And because we have our own representation of this timeline, we needed a different system to be able to handle these types of actions than what the off-the-shelf frontier models could give us.

And so where are we going with this? The realization of this is on our brand-new software engineering model called Swe1. We realized ourselves that we could actually dream bigger and build the best software engineering model that we could.

Swe1 is trained to handle software engineering workflows, not just purely code generation. And we use two main offline eval benchmarks, the first one, end-to-end task benchmark. This is basically tackling pull requests. This is saying, given an intent, given the starting point of a code base, how do we get to the end and pass all the unit tests?

Familiar pattern. The second one is where it gets a little bit more interesting. This is what we call a conversational Swe task benchmark. And this is how well the model can assist when it's being dropped into an existing user conversation or a partially completed task.

And so this actually lends itself very nicely to the Windsurf paradigm,right? Because we're not going cleanly from start to end. We're assisting and helping you along the way mid-timeline. And so it results in this blended score of helpfulness, efficiency, and correctness, and really tests the model's ability to seamlessly integrate into the Windsurf style of working.

And this initial performance really gives us a lot of confidence in Swe1's architecture, specifically how we've been able to train for software engineering workflows. And we've been able to achieve near-frontier model results at the fraction of the cost and with a significantly smaller team.

And one of Windsurf's greatest strengths, of course, is in the value of community, real software engineers doing real work, giving real feedback. And what we found is that Swe1, it's in the little dropdown for the models. It'sright up there with the rest of the frontier models.

People are choosing Swe1 because it recognizes how they do work, not necessarily how to generate code. And it's contributing actually an even higher frequency than models like 3.7 and 3.5.

Windsurf builds at the frontier so that our users can build more with the best technology. We learn from our failure modes so that we can iterate from there. And what does this start to look like? Dare I say it?

A data flywheel. We ship the best product. Devs and non-devs use that product to level up as a skill multiplier or as a skill enabler. Users then help us find the frontier. They use things like thumbs up, thumbs down, accept, reject, constantly informing us not of what the SweBench frontier is, but what is the software engineering frontier, what tools are missing, which workflows are repeated, where does the product fall short.

And we take those insights, and we build at this frontier. We train a better model. We build more tools. We improve our agentic harness. We improve our memories, our checkpointing, with the goal of being everywhere, doing anything. And we will repeat this cycle.

We will be shipping, finding the frontier, building at the margin, and repeating. And what gets me really personally excited about this is Swe1 is really an example of this in action. We have a very small team, significantly fewer resources than the larger companies.

And we were able to achieve near-frontier model quality results with Swe1. And even more so, this is really a demonstration of what it means to build AI products in 2025. It demands this harmony of model, data, and application, where the application is actually mimicking the user behavior that you want to replicate inside of your model.

And this is how Windsurf will be everywhere, doing everything all at once. Thank you so much for listening. And I won't give you any promises. But someone made a profit. But in all seriousness, thank you so much for listening.

I want to make sure that every engineer out there is using the best possible tools. So please give Windsurf a try today. And we are also hiring across a number of different roles. We have a booth downstairs. So please come join us.

Help make this future a reality. Thank you.

### CogSurf Future

**Swix** [3:34:28]
Hey, everyone. Welcome back to another Latent Space Living Pod. This is Alessio, partner and CTO at Decibel. And I'm joined by Swix, founder of SmallAI.

Hello, hello. This is a very special episode because we have the, I guess. Until recently, the separate CEOs of Cognition and Windsurf, Scott Wu and Jeff Wang. Welcome.

**Scott Wu** [3:34:47]
Hey.

**Kevin Hou** [3:34:47]
Yeah, thanks for having us.

**Swix** [3:34:50]
OK, so we have a limited time. We wanted to really cover this. But I wanted to sort of give the opportunity for folks to get to know the new Cognition and understand how did this deal come together. Both of you have relayed a little bit in your tweets and stuff.

But I don't know. You want to take it from here as to how the relationship between the two companies started.

**Scott Wu** [3:35:14]
Yeah, I got a actually, there was a couple ways. Scott tried multiple paths to get to me. So there was a text message through a friend that put us together. And then there was also an email that had a subject matter of, it just said chat with a question mark.

And I mean, it caught our eye. And I think when we were looking at. Like Friday was like a frantic day of telephone calls, I'll say. I think I probably called at least 70 different people on Friday. And when I talked to Scott, it was just like, hey, this kind of makes sense.

There was other companies that were interested. There was actually VCs that wanted to invest. But Scott, as a person with Russell and then I was with Graham, just talking about what are the companies actually missing from each other?

And then also, what are the product synergies as well,right? So it started all from Friday. I think we met around Friday at 9:00 p.m. And I kind of thought at that moment, this is probably it. And we were very excited to keep talking with Scott.

**Kevin Hou** [3:36:07]
Yeah, it was a fun one. I mean, I think for us, it was we heard the news roughly the same time everyone else. I think we had heard some rumors in the hours before. But we didn't have the news confirmed either about the whole Windsurf deal until Friday afternoon.

And around that time, we were talking about it. And obviously, I think there are a lot of natural synergies between the products and even between the teams as well. And I think Windsurf has built just an incredible, incredible go-to-market team all the way through sales, deployed engineering, marketing, enterprise, infrastructure engineering, all of these pieces.

And on our side, we were thinking about kind of scaling up on all the other facets of the team, that way, finance, operations as well. And then similarly, in terms of the products, I think Windsurf is the agentic IDE.

And Devin is the first agent,right? And so there are a lot of natural synergies there. And we've kind of seen more and more that, especially for these harder engineering tasks, the workflows that people want to have are that they want to be able to go back and forth,right?

And so you are planning things out in a synchronous way. And you're figuring out the details of the task yourself or what exactly you want. And then once you're ready with that, then you hand it off to the agent.

And the agent starts working. And once the agent produces a piece of work and tests it and everything, then you can come in and look and review the PR. And so there were a lot of natural ways that we saw to work together as well.

**Swix** [3:37:27]
Was there any one person that you want to shout out that helped make this happen? When we had Bret Taylor, he talked about Adam D'Angelo at Quora kind of helping negotiate the same moment and everything. Any other maybe people.

Brian Armstrong also, yeah.

Yeah. Any other people in the cinematic universe of the Windsurf Cognition that you want to shout out?

**Kevin Hou** [3:37:46]
Yeah, it was a fun one because I mean, I think it was a very busy weekend for all of us. I mean, I would say the obvious ones to shout out are Russell and Graham. Russell is president of Cognition.

And Graham is president of Windsurf. And obviously, the four of us spent a lot of time basically huddled up together and talking through the details and what it would look like. But honestly, besides that, I would say a huge shout out to all the lawyers involved because initially, they were saying one week is about the minimum that you could possibly do for something like this.

And obviously, we were super excited to get something goingright away, just to ensure kind of the customers, to make sure people knew that

there was the engineering muscle to take care of the product and to take care of the contracts and the team as well. And so the sooner, the better for us.

**Scott Wu** [3:38:34]
Yeah, and I don't think it was reported, but time was not our friend. We had customers starting to email like, hey, what's going to happen to the company? And we had recruiters. I think almost every employee probably got like 10 or 12 recruiting emails as well.

So basically, we were working against the clock. And me and Scott, I don't think we I think we slept maybe three hours that entire weekend. And we really started to feel like the mental limits of what happens when you don't sleep for that long.

And I remember we were talking. And I'd be like, wait, what are we talking about? It was like the memory retention starts to go low. And we really experienced what it was like to kind of push our own bodies to the limits to get this done on time.

**Swix** [3:39:13]
Yeah, context window exceeded.

Yeah, so I think this is, for me, the key thing for people to understand. As an outsider, a lot of us were like, why is everything happening so fast? What is theurgency,right? Windsurf was a fine business. Obviously, your CEO left and everything.

But it seemed like it's been around for a while. It's going to keep continuing to be around. But I think whenever there's business uncertainty, your customers start to have some questions. I didn't even think about the recruiting angle.

People were probably trying to pick you off in terms of the sort of poaching. And then I think so, and on Scott's side, I think that the speed at which you move, you probably just move a lot faster than other people who are not going to think about a deal on a Friday night and then close it on a Monday morning.

This is the kind of thing where you're moving with a pace where public markets are moving, like public companies, because they have to announce before the market's open. But I think maybe that's the way that's the new normal.

I don't know.

**Kevin Hou** [3:40:16]
Yeah, I mean, I think there's a few things. I think for one, obviously, things are just moving faster in AI today than in all these waves before. I mean, there's just so much happening and so much changes every day and every week and so on.

But no, I think from our perspective, we really thought about it. And what are the unique advantages that we can bring? And I think there were a lot of obvious ones in terms of the product synergy and the team synergy.

And I think in many ways, I would say kind of understood the business perhaps better than almost anyone else who was not at the company themselves, at Windsurf themselves, because of course, we're in AI code. We see these tools.

We understand kind of what the traction looks like and what the business looks like. But also that, yeah, just as a small startup, that we would be able to move fast and make decisions. And for me, it was like we were calling our board, which is, by the way, is a relatively small board.

But it's basically just like saying, hey, we want to do this. And we want to be ready to roll ASAP. Can you guys just be ready to stay up in case there are any things that we have to call and discuss live on a Saturday night?

And just making sure we were ready to move with the moment. I think at some point, Jeff and I kind of together came to this conclusion that, hey, if it's going to make sense here, it's going to make sense to do this by Monday morning.

And then I think from that point onward, it was just like all a matter of basically racing the clock. What are all the things that we need to have ready to go,right? And it's the legal agreements. It's the announcements.

It's what particular things we want to say, any decisions that we need to make and determine over the course of the weekend, and just going through every single thing.

**Scott Wu** [3:41:55]
I mean, we even filmed the video as this was happening. We're like, just in case it happens on Monday, let's film the video for the launch on Sunday night. And then we were both really tired in the video.

You probably, hopefully, couldn't tell, but we were. And then I think we also had specific specialized lawyers waking up in different time zones around the globe. So it would be like 3:00 a.m. And we're like, wait a second.

We can call this person now. And then they would hop on the conference call. And it was just crazy. It was just like it was almost like a movie full of shenanigans and getting things running. But it all worked out in the end.

**Swix** [3:42:28]
That's great. And our friend Jason Liu, I saw you retweeted the MCP announcement you guys had today at Cognition. And he said, I see the Windsurf marketing team in this. So I don't know if it's true or not.

But from the outside, I think the vibes are really good. What should people know about the future of the product? So if I'm a Windsurf user, am I now getting access to Devin as well? If I'm a Devin user, should I now move to Windsurf for some of the hands-on work?

Is that happening over months? Since the deal came together very quickly, but maybe the product side is not as easy to figure out.

**Kevin Hou** [3:43:03]
Yeah, yeah. Unfortunately, shipping the entire product integration will probably take more than a weekend. But we're going as fast as we can. And we've obviously put out a lot of things already with Wave 11, with Devin MCP, and so on.

And what I would say at a high level is, I think spiritually, we really want to maintain the philosophies of both of these products. And we have no plans to change that. I think Windsurf is a really amazing kind of high-touch experience for when you want to be writing and editing code yourself.

And your hands are on the keyboard. And you want to be able to move faster and move more efficiently with the help of AI. I think Devin is very much for being able to delegate tasks and basically hand off your issue backlog to an AI and have it put out these full pull requests that you review.

But I think as time goes on, I think we're finding more and more that there's just a lot of great surface, I think, that naturally kind of interacts. And to be honest, when we were kind of going through customers, it was already the case, I'd say, that there were a lot of customers who were using both Windsurf and Devin.

And you can kind of imagine it makes sense,right, which is for the kinds of tasks that you're really focused on yourself, Windsurf is theright kind of in-line experience for that versus for kind of the kinds of repetitive or tedious kind of undertakings that you would hand off, then you have Devin work on that,right?

And now I think the fun question for us is thinking about basically, yeah, how in these really hard tasks you can use the combination of both to just go a lot faster.

**Scott Wu** [3:44:32]
It's kind of crazy. We actually looked at each other's roadmaps, like product roadmaps. And the Devin team was actually looking to build an IDE. And our team was actually looking to build a remote agent. And even the browser stuff, you saw we were starting to release that on our side.

It turns out the Devin product has a really great, very powerful browser as well. So it almost feels like we've kind of accelerated our existing roadmaps quite a lot, quite far ahead. And then now the integration piece is more like we're kind of experimenting with how seamless we can make it.

But yeah, if you were to guess on a timeline, it's probably more than a few weeks. But.

**Swix** [3:45:08]
For sure. Yeah, I think what's interesting, I was talking with Andrew Gao. And I was just remembering that Devin actually had a VS Code sort of interface inside of Devin. But it wasn't like the main sort of desktop IDE interface which you used to interact with Devin.

OK, so the products are kind of independentright now. I'm very kind of curious if there will be a merging of the agents or not because I have this theory about there being very separate kinds of needs for different agents.

So Devin is architected to be background, async, long-running, whereas Cascade and Windsurf, presumably, I mean, obviously, Cascade is quite autonomous, quite agentic, but presumably optimized for fast responses and fast and more interactions. I'm curious if you anticipate any differences there, or are you pretty clear that it's going to merge?

**Kevin Hou** [3:46:00]
Yeah, so I think that's exactlyright, by the way, which is I think in the loop synchronous agent, you're optimizing for something somewhat slightly different things than this kind of async background agent. And to just call out a few examples, a lot of the things that we spent a lot of time on with Devin are having Devin test its own code and be really good at kind of running the server locally and figuring out how to set up the repo and all these other things, which you may not need locally,right?

On the other hand, a synchronous agent is really focused on just being as fast as possible and also just making sure every step is clear to the human that's watching it so that they can see and kind of jump in whenever if there's something that needs to change.

What I would say is I think we're still figuring out a lot of this, obviously. But I think at a high level, we're finding so far that a lot of the underlying capabilities and the infrastructure and so on that we've built out are really just naturally shared.

And so I think the synchronous agent will always have kind of the trade-offs of what the synchronous agent has. But at the same time, of course, having smarter code-based intelligence or better retrieval ability or better ways to parse out certain things or the ability to learn over time or a lot of these other kind of pieces are naturally helpful there as well, even though some of the kind of design decisions on the high level are different,right?

And so we should have more to share on this soon. But we're really excited about kind of just exploring this. I think there's quite a wide space, basically, in between these two products to kind of explore what that shared experience looks like.

**Swix** [3:47:30]
Yeah, and then I think the other thing is that what we're seeing is I think there's setting out to be an epic clash between Cursor and Cognition now with Windsurf, obviously, because Cursor obviously started with IDE and now going to background agents.

Cognition started with Devin and now has one of the strongest agentic IDEs. How do you think this will go down?

**Kevin Hou** [3:47:51]
Yeah, look, I mean, we know those guys. I mean, I think the fun thing, by the way, is that it is just so positive some. And people talk about Devin's paradox. And I think nowhere is it more true than code.

One of my favorite stats is in the year 2000, there were less than a million software engineers in the world. And this was whole dot-com era. A lot of software was being built. But now in the year 2025 today, there's actually over 30 million software engineers.

And obviously, coding has gotten a lot easier, not harder, if anything. And so we're producing way, way more software. And I just think of it because it just really is a reminder for me that there's 10x more or 100x more software to build.

And I think there will be a lot of tools, honestly, that get every engineer to be more and more efficient over time. And I think we have our particular lean on the kinds of problems that we own. And I think there will be other products, honestly, that have their own lean too.

And so

a lot of these folks, I'd say, have been doing well lately. You see markets on the kind of, I'll say, the app builder side of the market,right, where I think there are a few companies doing very well. There's Claude Code and so on.

So exciting times for us all.

**Scott Wu** [3:49:03]
Yeah, I mean, when we talk to our customers, though, we do want to enter the conversation being that very comprehensive platform. I think people already buy Devin separately. And they buy Windsurf separately. But going into these customer conversations, we're already seeing how excited they are.

It's kind of an easier decision now, knowing that these multiple toolkits are kind of combined someday. And I think that's kind of the main thing we want when we talk to customers.

**Kevin Hou** [3:49:29]
Yeah, and I think in the long term for us, our kind of unique lead or our focus is really just engineering teams focused on real-world work. And I say that in particular because obviously, there's a lot of different kinds of code.

And I think there's a lot of great missions going on around code. One of the things that we've found actually nicely as shared philosophy of both of these teams is we've both spent a lot of our time thinking about how to navigate big code bases and how to make tooling that's really powerful for specifically engineering teams who are working on projects and tickets and have backlogs to go through and all these various things.

And I think that's really our sweet spot.

**Swix** [3:50:08]
OK, awesome. With the remaining time we had, we wanted to try to address some of the hard issues that were kind of thrown around in social media. Maybe offer the opportunity to clear the air. So I'll say what has been reported.

You don't have to confirm or deny or whatever. But we just want to give the opportunity to clear the air. It's very sensitive. So Google bought all Windsurf shareholders for $2.4 billion in cash, which means that the employees got paid out.

They have a non-exclusive license to Windsurf's IDE and IP. But then the employees had all the IP. And Cognition's buying all the IP. It's been reported that it's been estimated that Cognition bought the remaining of Windsurf for $300 million.

I think there's a lot of back and forth over the broken founding engineer contract in Silicon Valley. I think this is probably the biggest single if you're not involved in Cognition, not involved with Windsurf, not involved in Google, you still care about the sort of startup scene.

Can you clear the air about any of those rumors that are flying around?

**Kevin Hou** [3:51:14]
Cool. Yeah, yeah. A bunch of photos. I love the careful wording, by the way, of every single kind of individual piece there. Look, I think we can't comment too much on kind of the particulars of numbers or some of these things.

But what I would say is in terms of the deal, it's a pretty clean cut for what we built. And so we have full and exclusive access to all of the things that we work on together from here, full and exclusive access to there was no information that was given out, for example, in terms of customer data, training data, any things like that.

And so all of that is strictly proprietary and remains our exclusive access. And to your point, yeah, I mean, about founders and founding team contracts, again, I think it's a complex story. And I think it's good not to try to kind of draw any too sudden conclusions.

But I do think it's an important high-level piece of

I really do think that the biggest covenants that you make as a founder is with your team and just really making sure through all circumstances that the team is treated as well as possibly can be.

**Scott Wu** [3:52:26]
I will say also, the team or even individuals that are pulled out of companies is just one component,right, of the company. And getting Windsurf to where it was means the whole team was very good, including the go-to-market team, including the marketing team, and enterprise engineering team,right?

There was a lot of other components of Windsurf that are just very valuable. And I think if future startups this happens to, I think there is just make sure the rest of the company is also very valuable. And then this kind of situation with Emmy and Scott can happen again.

**Swix** [3:52:57]
Awesome, guys. I know we all got around. But thank you for jumping on. And then we'll do a longer episode once you are done with the product integration. But yeah, thanks for coming on. And thank you guys for creating a great outcome.

I think there was a lot of drama on Twitter and a lot of maybe despair in some circles. So I think you brought a ray of light. So thank you for that. And good luck with everything.

**Kevin Hou** [3:53:19]
Yeah, thanks for having us. Excited to chat more soon.

**Swix** [3:53:21]
Yeah, congrats, guys.

---

This library is powered by PodHood (https://podhood.com), the podcast website platform.