# Notion’s Sarah Sachs & Simon Last on Custom Agents, Evals, and the Future of Work Latent Space · 2026-04-15 Sarah Sachs and Simon Last of Notion explain how Custom Agents were rebuilt four to five times since 2022, with early attempts failing due to lack of tool-calling standards and short context windows. They shifted from few-shot prompting to tool definitions and progressive disclosure, now supporting over 100 tools. Their eval system includes frontier evals that pass only 30% to gauge model progress, and they employ Model Behavior Engineers to write evals and triage failures. The episode covers their views on MCP versus CLI, meeting notes as data capture, and the software factory concept of agents collaborating to develop codebases. They also discuss Notion's culture of low-ego, high-velocity rebuilding and the philosophy that every surface must work for both humans and agents. ## Questions this episode answers ### What is a Model Behavior Engineer at Notion? Sarah Sachs describes Model Behavior Engineers (MBEs) as a role combining data science, test product management, and prompt engineering. Originally Data Specialists, many have linguistics backgrounds; they build evals, write LLM judges, and analyze failures. Notion hires MBEs without requiring an engineering background, creating a distinct career path for these ‘misfits’ to develop instinct and taste in model behavior, rather than focusing solely on traditional software engineering. [27:22](https://addtry.com/78d79d28-4ab8-497a-bad0-b4c47e9bd040?t=1642000) ### How does Notion evaluate its AI agents' performance and track frontier model progress? Simon Last and Sarah Sachs outline three eval tiers: regression tests in CI, launch-quality evals targeting 80–90% pass, and ‘headroom’ evals that intentionally only pass ~30% of the time. These headroom evals, called ‘Notion’s Last Exam’, are designed to see where model capabilities are heading and to give better feedback to labs. A full-time team — a data scientist, an MBE, and an evals engineer — works exclusively on these failing evals. [26:20](https://addtry.com/78d79d28-4ab8-497a-bad0-b4c47e9bd040?t=1580000) ### Why did Notion's early attempts to build AI agents fail, and what changed? Simon Last explains that starting in late 2022, Notion tried building agents by fine‑tuning its own tool‑calling scheme before tool calling existed, using XML representations the models didn’t understand. Context windows were too short and models were ‘too dumb’. The reliability was never high enough for a background agent. The unlock came with Claude 3.5 Sonnet / 3.7 in early 2024, which finally provided the robustness needed. Multiple rebuilds taught them to use formats models already know, like Markdown and SQLite. [3:05](https://addtry.com/78d79d28-4ab8-497a-bad0-b4c47e9bd040?t=185000) ## Key moments - **[0:00] Intro** - **[1:33] Custom Agents Launch** - **[2:51] Early Agent Failures** - [3:16] Notion's early agent attempts failed: no tool-calling, short context, unreliable models. - [4:31] The unlock for Notion's agents was Anthropic's Sonnet 3.6/3.7 in early 2024, says Simon Last. - **[5:04] Agent Lab Thesis** - [6:07] "A coding agent is the kernel of AGI — everything is a coding agent," says Simon Last. - [7:03] Sarah Sachs: Notion's two crucial AI skills are knowing when not to fight model limitations and anticipating where capabilities are heading. - [8:55] Notion's AI product is like Datadog on AWS: they build on frontier models but focus on collaboration expertise, says Sarah Sachs. - [11:03] Notion's AI team reviews the P99 most token-consuming custom agent transcripts every Friday to debug failures, says Sarah Sachs. - **[12:08] Team Culture** - [13:33] Sarah Sachs: Engineering leadership must build a low-ego team comfortable deleting its own code because Notion has rebuilt its agent harness multiple times. - [17:30] Notion shipped image generation because engineer Jimmy from the database team prototyped it on his own initiative, says Sarah Sachs. - **[18:10] Org Design** - [19:09] Sarah Sachs predicts that a majority of Notion's traffic will come from agents, not humans. - [19:27] Notion's design team now builds fully functional prototypes with an agent, skipping static mocks entirely, says Simon Last. - [20:44] "Demos over memos" became a Notion principle, says Sarah Sachs. - **[21:50] Evals** - [22:52] Notion distributes eval ownership to each product team using a shared framework, with nightly CI runs and a custom agent that pings teams on failures, says Sarah Sachs. - [25:52] Notion creates 'headroom evals' intentionally passing only ~30% to see where models are heading, and is working on 'Notion's Last Exam' akin to Humanity's Last Exam, says Sarah Sachs. - [27:24] Notion created the 'Model Behavior Engineer' role, a blend of data science, test PM, and prompt engineer for evals, says Sarah Sachs. - [30:09] Q: Will there be a day with no software engineers at Notion? - [32:11] Simon Last: For software factories, a specification layer, self-verification loop, and bug-to-PR pipeline are key design elements. - **[33:24] Agent Demo** - [35:35] Notion's most impactful internal custom agent is bug triaging: it routes Slack messages, creates tasks, and posts in relevant channels, says Sarah Sachs. - [37:36] Simon Last built a 'manager agent' to oversee 30+ go-to-market custom agents, reducing 70 daily notifications to 5. - [41:16] Simon Last is bullish on CLIs over MCP for agents because CLIs allow self-bootstrapping, debugging in the same environment, and progressive disclosure, though MCP has a stronger permission model. - [44:12] Sarah Sachs argues that using language models for deterministic tasks is wasteful and misaligned with value-based pricing; CLI can reduce repeated token fees. - **[44:13] Pricing** - [49:25] Simon Last recounts Notion's agent rebuilds: from coding agent, to XML tool-calling, to Markdown, to SQLite queries, each driven by model capabilities. - [51:12] Notion simplified database queries for its agent by representing them as SQLite, which models excel at, instead of the complex JSON format of the Notion API, says Simon Last. - **[51:20] Rebuilds** - [53:25] Simon Last: Notion's next agent harness version implements progressive disclosure to handle over 100 tools without overwhelming the model. - [57:29] Sarah Sachs says Notion deliberately avoids over-simplifying the custom agent UX because easier use reduces interpretability and agent capability. - [1:00:06] Q: Why is Notion's 'Flippy' design important? - [1:02:59] Notion uses a credit-based pricing model for AI because token pricing can't capture costs of GPU-served fine-tuned models, web search, or sandboxes, says Sarah Sachs. - [1:05:23] Notion moved to usage-based pricing for agentic features like database autofill to prevent Opus-level costs from bankrupting the company, says Sarah Sachs. - [1:07:27] Notion's 'Auto' model picker acts as a robo-advisor for AI model selection, choosing the best model for the task, says Sarah Sachs. - **[1:18:24] Meeting Notes** - [1:20:12] Notion's standup meeting is entirely agent-driven: a pre-read agent summarizes Slack and GitHub, and a post-meeting agent files tasks and sends follow-ups, says Sarah Sachs. - [1:21:01] Notion's meeting notes now @-mention people mentioned in the summary, so Simon Last gets notified whenever someone talks about him in a meeting. - [1:22:50] Q: Would Notion build its own meeting recording hardware like a pendant? - [1:24:39] Sarah Sachs: Notion's job is to be the best place where meeting notes live, not the best tool to capture them. - **[1:24:44] Outro** ## Speakers - **Alessio** (host) - **Swyx** (host) - **Sarah Sachs** (guest) - **Simon Last** (guest) ## Topics Evals, Agent Platforms, Agent Infrastructure ## Mentioned AWS (company), Anthropic (company), Datadog (company), Fireworks (company), Notion (company), OpenAI (company), Custom Agents (product), GPT (product), GitHub (product), Gmail (product), Haiku (product), Jira (product), Linear (product), MCP (product), Meeting Notes (product), Notion Calendar (product), Notion Mail (product), Opus (product), Slack (product), Sonnet (product) ## Transcript ### Intro **Simon Last** [0:00] Broadly speaking, I'm really bullish on CLIs. I'm still bullish on MCPs in, in a certain environment. I think in, in particular, MCP is really great for when you want sort of like a narrow lightweight agent. **Alessio** [0:10] Mm-hmm. **Simon Last** [0:10] I think there's, there's definitely a lot of use cases where, where you don't want like a full coding agent with a compute runtime, and also you want it to be like more tightly permissioned. MCP inherently has a really strong permission model. Like all you can do is call the tools. MCP is just like the dumb, simple thing that works and it, it is pretty good. **Sarah Sachs** [0:27] Notion is dedicated to being the best system of record for where people do their enterprise work, so we will always support our MCP insofar as other people are using MCPs, right? So regardless of our perspective, we've put a lot of effort into our MCP, and we have a fantastic team that we're building. **Alessio** [0:43] Before we get into today's episode, I just have a small message for listeners. Thank you. We would not be able to bring you the AI engineering, science, and entertainment content that you so clearly want if you didn't choose to also click in and tune into our content. We've been approached by sponsors on an almost daily basis, but fortunately, enough of you actually subscribe to us to keep all this sustainable without ads, and we want to keep it that way. But I just have one favor to ask all of you. The single most powerful, completely free thing you can do is to click that subscribe button. It's the only thing I'll ever ask of you, and it means absolutely everything to me and my team that works so hard to bring Latent Space to you each and every week. If you do it, I promise you, we'll never stop working to make this show even better. Now let's get into it. ### Custom Agents Launch **Alessio** [1:33] Hey, everyone. Welcome to the Latent Space Podcast. This is Alessio, founder of Kernel Labs, and I'm joined by Swyx, editor of Latent Space. Hello, hello, we're back in the beautiful studio that Alessio has set up for us with Simon and Sarah from Notion. Welcome. **Sarah Sachs** [1:46] Thanks for having us. **Simon Last** [1:47] Thanks for having us. Yeah. **Alessio** [1:48] Congrats on the launch recently, Custom Agents. Finally, it's here. How does it feel? **Sarah Sachs** [1:55] We ship things slowly, so it had been in alpha for a little bit, and at the point at which is it's in alpha, um, there's a group of people that are making sure it's ready for prod, and then there's a group of people working on the next thing. So sometimes some of these launches are a bit delayed satisfaction, so it's quite nice- **Alessio** [2:10] Yeah **Sarah Sachs** [2:10] ... to remind yourself all the work you did because we do have a habit of like being two or three milestones ahead, uh, just 'cause you have to be. Uh, you know, you can't get complacent. Um, but it's been great that people understood how this is helpful, and I think that's just easier in general building AI tools today than it was two, three years ago. People kinda get it, and so that user education, um, there's just... It was our most successful launch in terms of free trials and converting people and things like that. It was really successful. So yeah. But there's a lot to build. **Alessio** [2:40] Making it free for three months helps. **Sarah Sachs** [2:44] Yeah. **Simon Last** [2:44] It was, it was definitely super exciting for me because it's probably the fourth or fifth time that we rebuilt that. **Alessio** [2:51] Yes. ### Early Agent Failures **Simon Last** [2:51] And I mean- **Alessio** [2:52] So you've been building this since like twenty twenty-two. **Simon Last** [2:55] Yeah, I mean, like it was even right when we got access to like GPT-4 in late twenty twenty-two, one of the first ideas we had is like, "Oh, okay, let's make an agent that..." I mean, we used the word assistant at the time. There wasn't really the word, the word agent yet. But, "Oh, we'll give it access to all the tools that Notion can do, and then it will run in the background, like, like do work for us." And then we just tried that many times, and it just was too early. Um- **Alessio** [3:16] I need to force you to like double-click on that. What is too early? What didn't work? **Sarah Sachs** [3:20] We were fine-tune... Like before function calling came out, we were trying to fine-tune with the Frontier labs and with Fireworks like a function calling model on Notion functions. This is right when I joined. I joined because, um, we needed a manager. Simon was, needed to be able to go on vacation. So, uh, that's, that's around when I joined, so you can speak much more to it. **Simon Last** [3:39] Yeah, we did partnerships at both Anthropic and OpenAI at different times, uh, to try to... Uh, at the time, the... I mean, when we first tried, there wasn't even a concept of like tools yet. We, we sort of designed our own like, like tool calling framework, and then we tried to fine-tune the models to, uh, to use it over multiple turns, um, and because it, it didn't work well out of the box. I think, yeah, the models are just too dumb, and the context length was also way too short. **Alessio** [4:05] Yeah. **Simon Last** [4:05] Um, and yeah, we just kind of banged our head against it for a long time. Uh, unfortunately, it was always like there was always like sort of glimmers that it was working, but, um, it never felt quite robust enough to be like a useful, delightful thing, um, until I would say, uh, the big unlock was probably like Sonnet three point six or seven, uh, early last year, and that's when we started working on our agent, which we shipped last year. Um, and then, and then, uh, uh, custom agents, kind of a similar capability, and that, that one just took longer because we, we just wanted to get the reliability up a lot higher because it's actually running in the background. **Sarah Sachs** [4:43] And the product interface of like permissions and understanding. You know, this custom agent is shared in a Slack channel with X group of people and has access to documents that are surfaced to Y group of people, and the intersect of X versus Y might not be whole. And so how do you build the product around making sure administrators understand that permissioning took multiple swings. **Alessio** [5:03] Everything is hard back at the end of the day. ### Agent Lab Thesis **Swyx** [5:05] Yeah. I'm curious, like when the models are not working, how do you inform the product roadmap of like, okay, we should probably build expecting the models to be better at some reasonable pace, but at the same time, we need to... You know. You had a lot of customers in twenty twenty-two. It's not like you were a new company with like no user base. **Simon Last** [5:22] Yeah, I mean, I think there's always the balance of, you know, like you wanna be AGI-pilled and thinking ahead and building for where things are going, uh, but also you want to be like shipping useful things. And so we always try to like, like keep a balance there. You know, we, we try to take like, like a portfolio approach. You know, we're always working on multiple projects, and we're, and we're always trying to work on, you know, maintaining things where that they've already shipped, like, like shipping new things that are like eminently working well and make them really good. And, and then we want to always have a few projects that are a little bit crazy. Um- **Swyx** [5:51] And what are the AGI pill projects that you have today? I'm curious what, uh, the... You don't have to share exactly what you're working on, but I'm curious what are things today that maybe in eighteen months people will be like, "Oh, obviously this was gonna work." **Sarah Sachs** [6:03] Eighteen months? Yeah **Alessio** [6:04] 18 months is, you know- **Sarah Sachs** [6:06] It's a lifetime and yeah **Simon Last** [6:07] Yeah, I mean, there's a number of things happening. I think one thing that's becoming more clear is I think like, like, a, a coding agent, so the kernel of AGI, sort of everything is a coding agent. **Alessio** [6:16] Mm-hmm. **Simon Last** [6:16] I think that's, that's sort of one, one direction. Um, and then, yeah, the exciting thing about that is sort of your agent can sort of bootstrap its own software and capabilities and actually debug and maintain them, and so, yeah, we're, we're thinking a lot about that. And then, yeah, like, like, another category of things that I'm, I'm really excited about is, like, uh, we call the software factory. Lots of people are using this, uh, this, this sort of word. Um, basically it just means can you create sort of like a, as automated as possible, a workflow for developing, debugging- **Alessio** [6:49] Mm-hmm **Simon Last** [6:49] ... merging, reviewing, and maintaining a code base and a service where there's a bunch of agents working together inside and like, like how does that work? **Sarah Sachs** [6:57] If you think back to your initial question, like why did this take so long, I think something Notion- **Alessio** [7:01] I didn't say that, but yes. Okay, go ahead. You- **Sarah Sachs** [7:02] Why, what, what changed over the three- **Alessio** [7:05] Exactly **Sarah Sachs** [7:05] ... half years of trying it? **Alessio** [7:06] Right. Because most people always say, like, "It didn't work yet," then reasoning models came, then it worked. I was like, "Okay, let's go a little bit- **Sarah Sachs** [7:11] That's... I mean, that's part of it. But I think the other part of it that I actually think is really what will set Notion apart for every new capability is we have, like, two skills that are crucial when it comes to frontier capabilities. One is not letting yourself swim upstream. So like, quickly realizing if you're just pressing against model capabilities versus not exposing the model to the right information, not having the right infrastructure set up, that in of itself is a skill of intuition. And then the second is to see, okay, you're not swimming upstream, which direction is the river flowing? And what is like how do we think ahead about the product and start building it even if it's not great yet, so that when it is there, we're ready for it, right? And like, those can sometimes feel like counterintuitive things. Like, we can be trying to fine-tune a tool-calling model when they don't exist yet, and the, the trick is to not do that for too long, but realize that there was something there. And we've had a lot of things which, like, um, were just, like, not swimming in the right direction with the streams. I think we had multiple versions of transcription before we got meeting notes, right? **Alessio** [8:07] Oh, I gotta talk about that. Yeah. **Sarah Sachs** [8:08] Yeah. Um, and so I- I think that, like, we, we really closely partner with the frontier labs on capabilities, and we also have to have strong conviction on as those capabilities move. Notion is about being the best place for you to collaborate and do your work, and how does that narrative change if the way that we work changes? **Alessio** [8:26] Yeah. Yeah. You told me you were a fan of the Agent Lab thesis, and this is, this is kind of it, right? **Sarah Sachs** [8:31] Right. I show that thesis to so many candidates. Like, I have it as like my Chrome autofill, um, at this point. Like, it's one of my most visited pieces. **Alessio** [8:38] Like, is, is this the here's why you should work in Notion and not- **Sarah Sachs** [8:41] I think it's like- **Alessio** [8:41] ... OpenAI? Like- **Sarah Sachs** [8:42] Here's, here's what's different about it. **Alessio** [8:44] Yeah. **Sarah Sachs** [8:44] And here's why it's not just a wrapper. I actually think more and more people understand it's not just a wrapper. **Alessio** [8:50] Yeah. **Sarah Sachs** [8:50] Um, and by the way, like, in the beginning, parts of what we build are wrappers on functionality that works well. **Alessio** [8:55] Of course. **Sarah Sachs** [8:55] But that's not really the most, um... I would say that's not the product that, that drives revenue, and that's not necessarily always what users need. **Alessio** [9:03] I mean- **Sarah Sachs** [9:03] But- **Alessio** [9:03] ... you know, Notion is AWS wrapper, but, like, the, the wrapper is very beautiful and like- **Sarah Sachs** [9:07] Right **Alessio** [9:07] ... very, very well polished. **Sarah Sachs** [9:08] So like the a- **Alessio** [9:09] So, like- **Sarah Sachs** [9:10] The analogy that I've been coming back to is Datadog and AWS. **Alessio** [9:14] Yeah. **Sarah Sachs** [9:14] So, uh, Datadog could not exist without cloud storage, right? That it's kind of fundamental that that works. Um, and AWS has like a CloudWatch product, but Datadog is an expert on understanding how people want observability on the products they launch, and we're experts on understanding how people wanna collaborate, and that's really where our expertise lies- **Alessio** [9:32] Totally **Sarah Sachs** [9:33] ... um, regardless of the tools that we use. **Simon Last** [9:35] I'm kinda curious how you think about implicit versus explicit expertise. I feel like Datadog is half and half implicit and explicit. It's like they understand across markets and industries what engineering teams usually look for. With Notion, it's almost like more of the expertise is at the edge because you as a platform, you're like so horizontal that the end user is not really the same. **Sarah Sachs** [9:57] Mm-hmm. **Simon Last** [9:57] Like, with Datadog, the end user is always like- **Sarah Sachs** [9:59] Yeah **Simon Last** [9:59] ... an engineering lead, a kind of like SRE-related person. With Notion, it can be anything. So I'm curious how you put that expertise into a product versus, you know, obviously AWS cannot build Notion. It's... That doesn't quite work in this case, but- **Sarah Sachs** [10:13] It's, it's a little bit differently shaped. I think, you know, a classic vertical SaaS, like, like Datadog is kinda like that. They understand their individual customer very deeply. It's kind of a narrow slice. Um, Notion has always been super horizontal, and our, our task has always been to sort of balance these two somewhat opposing forces of like we're listening to our customers and what they want us to build. It's a broad slice. And then also we're thinking about like, okay, how do we decompose what they want into, uh, nice primitives that are, that are really nice to use and will, will get us like as much bang for the buck as possible, and then, you know, maintain the whole system, make it all like, like super clean and nice to use. We still have user journeys. I mean, we still focus on like core... I actually think the failure of our team is when we focus too much on what are tools that are, what are tools that are- **Simon Last** [10:59] Mm **Sarah Sachs** [11:00] ... cool tools. I actually think that's when we make- have the least velocity because you still need some sort of focus on a user journey. So like for instance, we'll all sit down every Friday and look at the P99 of like the most token exhaustive custom agent transcript and just look at why it didn't do well and cut a bunch of tasks. Like, we still focus on like this has- like this should work. Email triaging should work. **Simon Last** [11:23] Mm-hmm. **Sarah Sachs** [11:23] Right? And similarly, like when we're talking about before building, um, chatting, um, before we started filming about, okay, how can I do PDF export well? That's functionality that then merits maybe we should build a tool that has access to a computer sandbox and a file system and the ability to write code, right? **Simon Last** [11:39] Right. **Sarah Sachs** [11:40] Um, but it's because we're thinking about the fact that our users to do their, to do their daily work need to export PDFs, not because we're like, "Mm, I think a computer tool could be cool. Like, let's just see what happens." **Simon Last** [11:51] Mm-hmm. **Sarah Sachs** [11:51] Like we, we have to focus on some user journeys, otherwise we just don't have like enough strategy to, to prioritize. **Alessio** [11:57] I think there's a lot of like really strong opinions that you have. Do, do you have like sort of like a Tao of Sarah Sachs? Like, you know, like what... How do you run your team? Like, I feel like you just have accumulated all these strong opinions. ### Team Culture **Alessio** [12:08] Ob- obviously- ... part, part of this is your, your token town thing. **Sarah Sachs** [12:11] I think that Tao's working with Sarah Sachs is, um- You'd have to... It depends who you ask. Um, I think it depends if you're on my team or a partner, right? Or a vendor. **Alessio** [12:22] Yeah. There are other people who want to run their teams the way that you're- **Sarah Sachs** [12:24] Yeah **Alessio** [12:25] ... you're, you're, like, running these things. **Sarah Sachs** [12:26] Yeah. **Alessio** [12:26] But then also similarly, no- uh, Simon, when you did the custom agents demo, you had like, "Well, we've been using custom agents, and here's the super long list of everything that we do. No human's ever read it." Right? That's what you said. And I was like- **Sarah Sachs** [12:35] Yeah. So I think for, for me, um, something that I learned very quickly and became very comfortable with was that my job was not to be the ideas perso- person or the technical expert. My job was to make it so that everybody understood the objective, had a resource to help prioritize what they should work on, and had an avenue to prioritize what they thought was important. And I think that's true with all, all leadership, but I think especially on the AI team, almost all of our best ideas come from prototypes, from people that have a cool idea because they saw a user problem, and it's a huge disservice if all of those ideas have to pass, like, the sniff test of what me and a product partner or Simon and Ivan decided were the direction, right? Because a lot of what we're doing is leaning into capabilities. So I think that's the first thing is, like, I don't really view, like, the role of engineering leadership as, like, uh, hierarchical, nor has it ever been, but especially now, like, very willing to change direction based on the, like, proof is in the pudding. **Alessio** [13:33] Yeah. **Sarah Sachs** [13:33] And, like, and I think we have rebuilt our harness three or four times, and when you do that, then the second role of engineering leadership is, like, you need to build a team that's comfortable deleting their own code and is very low ego and is driven by what's best for the company and, um, doesn't write design docs because they think it's their promotion packet, right? And that's a culture that Notion had long before I joined. But, like, our willingness to just swarm on different problems and, um, redo things that we've built before because something has changed, like, there's a lot of friction that can happen at companies when you do that, and it doesn't happen at Notion. And because it doesn't happen when new people join, like, they don't wanna be the ones that are saying, "We shouldn't do this. I wrote that code." So then it's, you know, you, you create a culture that everyone adopts, and that culture comes directly, I think, from Simon and Ivan, though, um, because they're very open-minded. **Alessio** [14:18] Anything that you'd, you'd add? **Simon Last** [14:20] I'm not a manager like, like, like Sarah is. Um, a lot of my role is really to try to think a little bit ahead, make sure w- that we're, we're building on the right capabilities, and then, like, the prototyping stuff. And yeah, it's really, really critical to always just be, be starting again. It's like, "Okay, this is a new thing. What does this mean? What if we just rethought everything, rewrote everything?" And so I- I'm basically just doing that in a loop every six months. **Alessio** [14:44] Yeah. Do you believe in internal hackathons for this stuff? **Sarah Sachs** [14:47] W- I think there's, like, two different versions. So one is, like, we just have a, a, a solid bench of senior engineers that come and go on what we call the Simon vortex and- ... productionizing what we've built, right? 'Cause when you're in the Simon vortex, the velocity is super high, the direction changes daily, and it's meant to be like the equivalent of a skunkworks lab. We don't need to do hackathons for that. We need to have senior engineers that we trust to come in and out of those projects. For instance, like, management boundaries are really loose. Like, you report to him, but you work for her right now. Like, that's- **Alessio** [15:17] Yeah **Sarah Sachs** [15:17] ... something that when we hire managers, it's important they don't care about because we tend to form org structures after the fact. **Alessio** [15:22] Yeah, don't be too territorial. **Simon Last** [15:24] It's, it's like an ocean now. **Sarah Sachs** [15:24] We form org structures after we ship things, not, not before. Just historically. Um, the second thing is we do have company-wide hackathons. Actually, we just had our demos day for the hackathon we had last week this morning. That's more for people that aren't directly working on the project, feeling like they have the time to pause and learn how to make themselves more productive or how they would use Notion custom agents to build something. Or part of the hackathon was actually encouraging everyone across the company to build their own agentic tool loop calling from scratch, following like an every blog post on how to do it, I think, because we want- **Alessio** [15:54] Is that the compound engineering one? **Sarah Sachs** [15:56] Yeah. **Alessio** [15:56] Yeah. **Sarah Sachs** [15:56] We want everyone to use cloud code in the company or whatever the coding agent they please and understand that fundamental. So we set aside a day and a half where all leadership encourage everyone on their teams across the company do it. So we have hackathons like that. I would say, like, kind of facetiously, like, everything we build is a little bit like a hackathon until it graduates and puts on big boy pants and has a product ops rollout later- ... and has a, a- assigned data scientist and stuff like that. But all the things- **Alessio** [16:22] Security review, enterprise stuff. **Sarah Sachs** [16:24] Actually, security review is one of the things that we bring in first- **Alessio** [16:27] Ah **Sarah Sachs** [16:27] ... because it just slows us down way more and, um, causes a lot of tension, and they build better product if they're involved early. So, um, that is probably the first person to get involved in something. **Alessio** [16:37] That's the right PR-approved answer. **Sarah Sachs** [16:39] N-no, but it- It's not just PR approved. It's like, um- **Alessio** [16:41] It's, it's actually real. It's actually real **Sarah Sachs** [16:42] ... it's like, um- **Alessio** [16:43] I'm just saying **Sarah Sachs** [16:43] ... scar tissue. **Alessio** [16:44] Yeah. **Sarah Sachs** [16:44] Because, like, you know, my background's also, I worked at Robinhood for a number of years. **Alessio** [16:48] Nice. **Sarah Sachs** [16:48] So, like, uh, compliance and things like that, um, are a little bit more you learn the hard way when it doesn't come naturally. **Simon Last** [16:55] Yeah, I think the, the hackathon is really important for uplifting the general population, but, like, if that's the only way you can build new things, you're kinda toast. **Alessio** [17:03] Mm-hmm. **Simon Last** [17:03] I mean, it, it has to be, like, the daily processes, like, you know, building these new things. Um, and it has to be about, I think, like, I think in the AI era, a lot more leverage accumulates to the most curious and excited people. And so it's like we're all about just, like, activating that energy. You know, like, if someone's prototyping something on the weekend that they're excited about and it's important, that should be the main thing that we're doing. **Alessio** [17:25] Yeah. **Simon Last** [17:25] Um, it's not a hackathon that we schedule once a quarter. It's just like- **Alessio** [17:29] Yeah **Simon Last** [17:29] ... a daily process. **Sarah Sachs** [17:29] That's how we- **Alessio** [17:30] It's part of the culture. **Simon Last** [17:30] Yeah. **Sarah Sachs** [17:30] I mean, that's how we shipped image generation in Notion now. It was always this thing that would be kind of nice to have, but it wasn't really clear where that was necessarily aligned in product priorities. It'd be a lot of work. And we had someone on the database collections team, Jimmy, who was like, "I really wanna do image generation for cover photos and inside Notion." And we're like, "If you wanna build it, like, it's... Do it, please. Like, we encourage you." We gave him all the resources of working directly with Gemini and being able to, like, track the token usage and it go- working through our end plans. We gave him eval support, everything, and then it became a full, a full project. **Alessio** [18:02] Yeah. **Sarah Sachs** [18:03] That's why you can't have, like, ego as a l- a leader. Like, that's, that's how we work. **Alessio** [18:08] What's the size of the team today, both engineering and overall? ### Org Design **Sarah Sachs** [18:11] I manage, uh, the team that's what we'll call, like, query capabilities and infrastructure. That's about 50 people. But then we have par- I partner teams that do packaging, so how it shows up in the corner chat versus custom agents versus meeting notes. That's another 30, 40 people. And then every team that has a product service at Notion that a user can interface with owns the tool that the agent interfaces with. The editor team, the team that did CRDT for offline mode is the same team- **Swyx** [18:39] Mm-hmm **Sarah Sachs** [18:40] ... that handles how two agents, um, edit competing blocks. **Swyx** [18:44] Mm. **Sarah Sachs** [18:44] Right? It's the same problem. The team that built the underlying SQL engine is the same team that owns how the agent asks it to run a SQL query, and it does it performantly. And so from that regard, anyone working on product engineering is tasked with making them work for customers that are humans and agents. Because over time, a majority of our traffic will be coming from agents using our interface, not humans. And so our objective is to make it so that the whole product org is building for agents. **Swyx** [19:09] Yeah. How has it changed internally? The activation bar is kind of lowered a lot. Like, anybody can kinda create a prototype very... somewhat easily, especially if you're, like, in an existing code base. Have you raised the bar on, like, what type of prototype people need to bring forward to gonna be taken- not, like, seriously, but, like, you know what I mean. **Simon Last** [19:27] Yeah. I think the bar is lowered in many ways. Be- like, one thing of our, uh, third team build that is really cool is our, uh, our, our design team made a whole separate GitHub repo, uh, called the, the Design Playground, and it's basically just to create a bunch of, like, like, helper components and y- uh, for, for quickly throwing together UIs. And it's become, like, actually quite sophisticated. Like, it has, like, an agent in there, and, like, uh, that's pretty fun. So, like, we pretty much... Like, they don't do mocks. They just make- **Swyx** [19:53] Right **Simon Last** [19:53] ... like, like, full, full prototypes. **Swyx** [19:55] Here it is. It works. **Simon Last** [19:56] They give you, like, a URL, and they're like, "Okay, all right, so we have to make the, like, the real production version of that." Um, and then for engineers, a prototype looks like just making it a feature flag that actually works. Like, that's sort of the bar. **Sarah Sachs** [20:07] Something to understand that's really unique about Notion, one of the reasons I joined, we're super lucky, is no one uses Notion in their job as much as people that work at Notion. **Simon Last** [20:14] Of course. **Swyx** [20:15] Mm. **Sarah Sachs** [20:15] So I think there's very few companies. Maybe if you worked on Chrome, I guess. But, like, everything that we ship, we ship internally first and get a lot of really quick feedback. And also sometimes our dev instance is totally borked, and you have to change a bunch of flags to get things done, and that's kind of, like, everyone. So people that do IT ticketing, people that do supply chain procurement, recruiting, everyone is using the same instance of Notion with, like, a lot of flags on for these prototypes people build. Um, and so we have this... Brian Loven, one of the designers on our team, I think- **Swyx** [20:44] Mm **Sarah Sachs** [20:44] ... evangelized this concept of demos over memos. **Swyx** [20:47] Ooh. **Sarah Sachs** [20:47] Um- **Swyx** [20:48] Too good **Sarah Sachs** [20:48] ... which has been, uh, very good for building demos, and I think it's put a big pressure point on us to have really strong product conviction because if anything can be demoed, you really need a strong filter of making sure that if, you know, you're doing X amount of work, you're making the... You're, you're focusing on one tower. You're not just building a really flat hill, right? That's actually where I think there has to be more conviction from our PMs, um, and our designers and, and well, the company really to have conviction of what journey we're going on. **Simon Last** [21:20] But overall, I feel like it works pretty well. Like, people, almost all the engineers have good enough taste to realize that, like, this prototype doesn't actually make sense in the product, or, or it does. So it's not that common that I would see a prototype that's like, "Oh, this makes no sense." **Swyx** [21:34] Mm-hmm. **Simon Last** [21:34] It's like, you know, people are doing reasonable things, and, and, and then it's just a matter of which things we build first, and then often just, just figuring out how to turn it on and off. **Swyx** [21:43] Right. **Simon Last** [21:43] There's, like, our, in the, in our, like, experimental chat UI, there's this, there's probably, like, a, a, like, a hundred checkboxes in there- ### Evals **Sarah Sachs** [21:51] Kill Simon **Simon Last** [21:51] ... that really just you can turn on and off. **Sarah Sachs** [21:53] But I think that... Okay, so that is kind of true, Simon, but, like, being the person that manages the evals team, like, there is a level of intensity that it adds to the platform team. So, you know, if we're gonna do image generation in Notion, all of a sudden, the way that we do attachments and the way that we, um, our LLM completion, like, Cortex talks and expects tokens back, and now it's getting images back. Like, there's a lot of platform work that we do need to, like, solidify a little bit. So sometimes it'll be in dev for a couple weeks before it makes it to prod, just because we still have to, like, make it robust, make it HIPAA compliant, ZDR compliant, figure out the right contracting with the vendor, whatever it is, and we n- need to eval it because we want the team to still maintain what they build. That's the one thing is, like, if we have a bunch of prototypes, it can't just be, like, a small group of people that then maintain whatever in prototypes. So we have invested a, a lot of people in an eval and model behavior understanding teams that we call it agent dev velocity. So your dev velocity building agents can be faster if we invest in that platform. And so we have a whole org dedicated to agent, um, platform velocity so that you can build your own eval and then maintain it once you ship a... So if a new model release comes out and we auto- **Swyx** [23:06] Every team maintains their own eval? **Sarah Sachs** [23:08] We maintain the eval framework. Every team owns their own evals, and a lot of them we've integrated to opt into CI, or we run them nightly, and we have a team, uh, a custom agent that triggers to a team to look at the major failures. It's really critical because if we have, like, all these different services, now a lot of it's on the same agent harness, so it's easier to maintain. It's just packaging of different agent harnesses. But new functionality of the agent, let's say that, like, we wanna update, like, uh, you know, they deprecated Sonnet, um, 4 or whatever it is, and we need to auto-update it. **Swyx** [23:39] Have they already? **Sarah Sachs** [23:39] I don't know. **Swyx** [23:40] That's so- **Sarah Sachs** [23:41] Yeah. **Swyx** [23:41] Okay, yeah. **Sarah Sachs** [23:41] I actually- **Swyx** [23:41] That wasn't that long ago. **Simon Last** [23:42] They were just 3.5. **Sarah Sachs** [23:43] 3.5. **Simon Last** [23:44] They care about- **Sarah Sachs** [23:45] 3.7 just got deprecated. 3.7 just- **Swyx** [23:46] They never get to 5.2 or... Yeah. **Sarah Sachs** [23:48] No, it's not 5.2 that's deprecated yet. **Simon Last** [23:50] No, 5.1, 5.1. **Sarah Sachs** [23:50] Yeah. **Simon Last** [23:51] Yeah. **Sarah Sachs** [23:51] 'Cause no. **Simon Last** [23:52] Yeah, yeah. **Sarah Sachs** [23:52] Uh, 5.4 is 40% more expensive than 5.2, so if they deprecated 5.2, you would hear- **Swyx** [23:56] They came- **Simon Last** [23:57] Oh, yeah **Sarah Sachs** [23:57] ... you would hear from me about that one. **Swyx** [23:59] Yes. **Sarah Sachs** [24:00] Um, but, uh- **Swyx** [24:01] I, I just have a- **Sarah Sachs** [24:01] ... another conversation to have. **Swyx** [24:03] I have a cheeky evals question for you. **Sarah Sachs** [24:05] Yeah. **Swyx** [24:05] Have you noticed any secret degradation from any of the major model providers? **Sarah Sachs** [24:09] Secret degradation? **Swyx** [24:10] Like during the workday- **Sarah Sachs** [24:12] Oh **Swyx** [24:12] ... when it's high traffic, it suddenly gets dumber. **Sarah Sachs** [24:15] Yeah, I mean, not just between the I mean, we definitely notice flickiness. We've definitely noticed, particularly for some providers, that things are slower during working hours, and- **Alessio** [24:25] That is a latency argument- **Sarah Sachs** [24:26] Yeah **Alessio** [24:26] ... not a quality argument. **Sarah Sachs** [24:27] No, I think the quality difference that's interesting is, um, even though companies that say they're selling the same... A-a-it's really into, like, quantis-quantization. But like- **Alessio** [24:37] Yes **Sarah Sachs** [24:37] ... companies that say they're selling the same model through different vendors, whether it be through first party or Bedrock, Azure, et cetera, we do see different quality sometimes, and that's not necessarily what's advertised. **Alessio** [24:49] Yeah, Kieny went to the point of like if you- they shipped like this, like, eval across all the providers, and it was, like, very obvious who was secretly quantizing, and it was very- **Sarah Sachs** [24:56] Yeah. But that's- **Alessio** [24:57] Very embarrassing **Sarah Sachs** [24:58] ... you know, um, we hire sub-processors to figure that out for us, so we just wanna understand where it's regressing or where it's optimized, and sometimes we're okay with regressions that optimize latency if they're the appropriate regressions. Our job is to make sure we have the evals to understand the changes that are important to us. And even, like, when we're partnering with labs on pre-releases of models, they'll send us multiple snapshots, and this is less about quantization, but more just regressions. Like, they have shipped models that were not the snapshots that we wanted, and they have changed the snapshots that they shipped based on the feedback that we give because our- **Alessio** [25:31] Yeah **Sarah Sachs** [25:31] ... feedback tends to be more enterprise work-focused and not coding agent-focused, and definitely those can be bummers. Like, you know, "Ah, we know that this wasn't the version you wanted, but we'll help you make it work." I mean, we always make it work, but that definitely happens. **Alessio** [25:44] Yeah. **Simon Last** [25:45] Do you have, um, failing evals that you're just hoping- **Sarah Sachs** [25:48] Oh **Simon Last** [25:48] ... that will have success eventually when a good model comes out? **Sarah Sachs** [25:52] Uh, y- I mean, yeah. So I think... I mean, I could talk about this for 60 minutes, so I will t- limit myself. I think it's a real issue when people say evals, and it's just like that's quality. That's like uni... I mean, it's like saying testing. It's not just unit tests. **Simon Last** [26:06] Right. **Sarah Sachs** [26:06] Right? So we have the equivalent of unit tests, regression tests. Those live in CI. Those have to pass a certain percent, you know, within s-some stochastic error rate. Then we have, as you're building a product, evals of these aren't passing right now, and this is launch quality. So we have a report card, and we need to, on these categories, you know, be at 80 or 90% of all of these user journeys to launch. And then what we have, what we call frontier or headroom evals, where we actively wanna be at 30% pass rate, and that's actually been a effort that we took in partnership with Anthropic and OpenAI in the past maybe two or three months because we actually hit a point where our evals were saturated, and we weren't able to really give insightful feedback other than it wasn't worse. And not only is that not helpful for our partners, it's not helpful for us to understand where the stream is going, you know, going back to that analogy. And so we spend a lot of time thinking about what Notion's Last Exam looks like, right? **Alessio** [26:58] Mm-hmm. **Sarah Sachs** [26:59] Not just humanity's last exam- **Alessio** [27:00] Ooh, MLE **Sarah Sachs** [27:00] ... Notion's Last Exam. And, um, there's a lot of, you know, dreams about what that would look like. I know we've talked a lot about benchmarking, um, Swix. But, uh, yeah, Notion's Last Exam is a big thing inside the company, and we have people full-time staffed to it exclusively. **Alessio** [27:15] Mm. **Sarah Sachs** [27:15] We have a data scientist, a model behavior engineer, and an full-time, um, evals engineer just dedicated to the evals that we pass 30% of the time. **Alessio** [27:24] Which you're hiring for, MBEs. **Sarah Sachs** [27:25] I am hiring. **Alessio** [27:26] What is an MBE? **Sarah Sachs** [27:27] A Model Behavior Engineer. **Alessio** [27:29] Oh. **Sarah Sachs** [27:29] Model Behavior Engineers started with the title Data Specialist before I joined when they were working with Simon on, like, uh, Google Sheets, and, like, Simon just needed someone to look through Google Sheets and say, "Yes, no, this looks bad. This looks good," right? And so we hired people with kind of diverse linguistics background. We had, like, a linguistics PhD dropout- **Alessio** [27:46] Mm **Sarah Sachs** [27:46] ... and a Stanford Comp Lit new grad, and they're amazing. And they formed a new function basically, and over time, we've built a whole team, um, with a manager who's now kind of reinventing what that role is with coding agents. So they used to be kind of manually inspecting code. Now they're primarily building agents that can write evals for themselves or LLM judges. There's a really funny day, I can send you the picture, where Simon, about a year and a half ago, was teaching them how to use GitHub, um, and they're on the whiteboard. And it was like, okay, I think it would be so much faster if our data specialists learned how to use GitHub and, like, learned how to commit these things and to code it. And, and that was then, and now I think, you know, coding has been a lot more accessible. Um, but moving forward, it's this mix of, like, data science, test PM, and prompt engineer because there's craft in understanding, like, even, like, what models can and can't do things. How do we define, like, that headroom? How do we define, like, what a good journey is? Um, is this model better or not? Why is this failing? There's some qualitative work, but then there's also, like, a lot of instinct and taste to it, and that's not necessarily software engineering. **Alessio** [28:49] Mm. **Sarah Sachs** [28:49] And so we have, like, very firm conviction, and we have had for a number of years now that that is its own career path, and we have always welcomed the misfits, so to speak. So we really firmly believe that you don't need an engineering background to be the best at this job, and that's what's quite unique about this particular role. **Simon Last** [29:05] Yeah, this is something that I've been pretty excited about recently is we made an effort basically to treat the eval system as, like, an agent harness. So if you think about it like, you know, you should be able to have an agent end-to-end download a dataset, run an eval, iterate on a failure, debug, and, and then implement a fix. And ultimately, you should be able to, you know, drive the full end-to-end process with a human sort of observing the, you know, the outer, uh, system. So yeah, we went, went pretty hard on that, and that's, that's worked extremely well so far. It's like basically just to turn it into a coding agent, uh, a problem. **Alessio** [29:39] Your coding agent or just- **Simon Last** [29:41] No, just any coding agent **Alessio** [29:41] ... whatever harness connects code- **Simon Last** [29:43] Yeah, yeah. It, it should be totally general. Yeah, I think it's, it'll be a mistake to, like, like, fix it on any, any particular coding agent. At the end of the day, it's just like CLI tools. **Sarah Sachs** [29:50] It's like the same way that you would have a coding agent write the unit test, you should have a coding agent write the eval. **Alessio** [29:54] Yeah. **Sarah Sachs** [29:54] But there's a lot of supervision in that still. We just don't believe that supervision has to come from software engineers because a lot of it is, like, um, kind of UXRE and whatever, and these are the people that also triage failures and tell us where we should be investing next. **Alessio** [30:09] Yeah. I'm gonna go ahead and ask the spicy question. Is there a day there are no software engineers at Notion? **Sarah Sachs** [30:14] Oh. What does it mean to be a software engineer? **Alessio** [30:16] Exactly. **Simon Last** [30:16] I mean, I think the way things are going is, like, we're on some continuum where- If, if you look back three years ago, humans were typing all the code, and then we had auto-complete. Uh, you're typing a little less of the code. Then we had sort of like filling... agents filling lines, and now we're getting into, like, agents doing longer range tasks, where you can debug and implement a fix, and then verify it works and, you know, get your, get your PR even, like, like merged and deployed. I think we're sort of just moving up the abstraction ladder, and then the human role becomes more about observing and maintaining the outer system. There's a stream of agents flowing through, like, merging PRs. What's going off the rails? Like, what do I need to approve? Is there, like, a learning or memory mechanism that, that works? So it's kind of a hard engineering problem. There's a, you know, there's, there's a lot to do there. I think we're just k- sort of moving up the stack. **Sarah Sachs** [31:02] The same transition machine learning engineers have made, right? Like, I haven't looked at a PR curve in a while. Um- **Alessio** [31:08] Yeah, you used to do this stuff, and now, um, auto research can do it. **Sarah Sachs** [31:11] Right. Like, I think it depends on what you define as a software engineer. **Alessio** [31:14] Yes. That's, that's changing for sure. **Sarah Sachs** [31:17] I think every software engineer at Notion this summer went through, like, this, um, shear. Um, one of our engineering leads at the company called it. Like, every software engineer is going through the m- the c- uh, identity crisis that every manager goes through, where all of a sudden they realize their ability to write code is less important than their ability to delegate and context switch. And I think that is a transition out of being a software engineer. But- **Alessio** [31:40] Yeah. **Simon Last** [31:41] Yeah. There's a critical difference to being a manager, which is that, like, it is actually very deeply technical. The problem of... You know, humans are very like, like, like fuzzy, and you can't like treat a team of humans like a, like a rigorous system where like, you know, PRs like, like flow through and can be in like a blocked status, and then what happens when they're blocked, right? **Alessio** [32:02] Mm. **Simon Last** [32:02] With a set of agents, you actually can do that. A- and, and I think it's actually... There's a lot of interesting technical rigor that, that goes into that. It's a, it's a technical design problem ultimately. **Swyx** [32:11] What is the design of the software factory that you're building? **Simon Last** [32:14] Yeah, I mean, I think we're trying a lot of different things. I mean, ultimately you want to design a system that requires a- as little human intervention as possible, but like still maintaining the invariance that, that you care about. So yeah, we're exploring a lot of different ideas there. I mean, I think I can talk about like a few things I think are important there. Like, one thing I think is really important is, um, having some kind of like specification layer. You can just commit markdown files. **Alessio** [32:43] Mm-hmm. **Simon Last** [32:43] That works pretty well, but- **Alessio** [32:44] It's nice to be Notion, man. I'm just saying, like the, the spec, like- **Simon Last** [32:48] Yeah **Alessio** [32:48] ... the natural home for specs is Notion. **Simon Last** [32:50] Yeah. **Sarah Sachs** [32:50] Right. **Simon Last** [32:51] It, it can be a database of pages. Yeah, I mean, it needs to be something that is, you know, human readable and, and viewable, and I think that's pretty key. Another really key component is like the, the self-verification loop. **Alessio** [33:02] Yes. **Sarah Sachs** [33:02] Mm. **Simon Last** [33:02] Like, you need a really, really good testing layers basically, and that's a really deep, uh, uh, problem. But, but getting that right, you know. And then, and then it's kind of like the workflow of like what happens when there's a bug? How does it flow into the system? Like, is it like a sub-agent working on it? How does it make a PR? And how does that get reviewed and merged, and then, you know. So there's like the, the flow of process. ### Agent Demo **Alessio** [33:24] Yeah, cool. Uh, o- you know, one thing we did work out before you guys came in was this demo or this- **Simon Last** [33:29] Agents **Alessio** [33:30] ... agent demo. **Simon Last** [33:31] Love. **Swyx** [33:31] So, uh- **Sarah Sachs** [33:32] Oh **Swyx** [33:32] ... every, every time we do an episode, we try the product, right? I don't think there's ever been an episode that I haven't tried. Um, so let me- **Alessio** [33:39] Yeah. And w- when try... Try is a, a big word. Like, since day one, LaneSpace has been on Notion, but this is the n- **Swyx** [33:45] Well- **Alessio** [33:45] This is the new thing. Yes. **Swyx** [33:46] So this is for Kernel Labs, which is the space we're in. So next week we're opening applications for tenants. So there's a web form. Let me... We have this form done here. Um, so, uh, before, uh, the workflow would be I get an email, then I look at the person and it's like, "Should I spend time talking to this person?" Then I respond, they respond back. So I build this. So the name, it came up for on its own. Can you maybe... H- how do, how does it come up with its own name? **Alessio** [34:11] Yeah. **Simon Last** [34:12] That's a pretty apt name. It's, it's, it's just a random, it's a random, uh, name generator. **Swyx** [34:15] Oh, okay. **Simon Last** [34:16] Yeah. **Swyx** [34:16] That's funny. **Simon Last** [34:17] Yeah. **Swyx** [34:17] It just came- **Simon Last** [34:17] The fact that it picked that is, is kind of hilarious. And so I, I'm pretty sure it's just determined- **Sarah Sachs** [34:22] Resilient Collector. I, I think I've never looked at the code for that. I've never second-guessed it. I think it's kind of like a Mad Libs situation. **Simon Last** [34:28] Yeah, I think it's- **Swyx** [34:28] Right. **Simon Last** [34:29] Yeah, it's, it's totally d- a, a determined thing, yeah. **Swyx** [34:31] Oh, I, I, I thought it was great. **Simon Last** [34:32] Yeah, it's, um- **Sarah Sachs** [34:33] Yeah. **Simon Last** [34:33] Although, although when the... If you use the AI to set itself up, it can update its own name, so- **Swyx** [34:38] Okay **Simon Last** [34:38] ... um, uh, so- **Sarah Sachs** [34:39] How did you create it? Did you just do cluster agent? **Swyx** [34:41] I, you- **Sarah Sachs** [34:41] Okay. **Swyx** [34:42] I did... Yeah. I'll say just check my inbox for applications- **Simon Last** [34:45] Yeah **Swyx** [34:45] ... for a co-working space. **Simon Last** [34:46] Okay. **Swyx** [34:46] Keep available people. So it created a database for me, which I have here, and I guess database is like a Notion table because everything is Notion. Um, and then whenever, um, an email comes in, like here, it just creates a new row for the person. **Simon Last** [35:00] Mm-hmm. **Swyx** [35:00] And then it uses web search to enrich the- **Simon Last** [35:04] Mm-hmm **Swyx** [35:04] ... the profile. So it kind of like searches the web, and it's like, "This is who this person is. This is when they say they wanna move in," and kind of updates everything else. This is... I mean, it's not AGI, but to me, I don't wanna do this work. So it feels like... I mean, it took me maybe like 15 minutes to set up the whole thing. Um, and I really like that most of the information should live here. You know? It's not like some other tool asking me- **Sarah Sachs** [35:30] Yeah **Swyx** [35:30] ... to, like, bring my stuff there. It's like I would have probably already created a Notion thing. **Simon Last** [35:34] Mm. **Sarah Sachs** [35:34] Mm. **Swyx** [35:34] So... **Sarah Sachs** [35:35] Most of our biggest use cases and gains are from that extra layer of human involvement in the process to make it so, right? And so, like, one of our biggest use cases is bug triaging. **Swyx** [35:46] Mm. **Sarah Sachs** [35:46] So if someone posts something in Slack, can you just have a custom agent that lives there that has its own routing constitution of what team this belongs to, creates a task in your task database, and then posts in that Slack channel, right? Like, that's like one of the first things that we built internally, I think, and it's completely changed the way that Notion functions as a company. Nothing falls through the... Well, most things- **Swyx** [36:06] Right **Sarah Sachs** [36:06] ... don't fall through the crack. We don't know what we don't know. But it's not replacing people, it's replacing processes. **Swyx** [36:12] Yeah. **Sarah Sachs** [36:12] Right. **Swyx** [36:13] And I'm curious how you think about composability of these things. So the other one I was working on is like a lease filler. So whenever somebody signs up as a tenant, kind of fills out the lease for them. There should probably some agent that is, like, office manager agent- **Sarah Sachs** [36:28] Mm-hmm **Swyx** [36:28] ... that can handle the requests, make the lease, and then, uh, give them all Verkada access to the office and all of that. How do you think about that future? **Simon Last** [36:36] Yeah. So I mean, there's, there's two ways you can compose. One way is by using, like, the data primitives. So you can, you know, you, you give... You could have one agent, uh, be writing to the database and with another agent that's watching the database. Uh, so that's, that's one way that they can be coordinated. That's, like, a little bit more decoupled and- **Swyx** [36:52] Mm-hmm **Simon Last** [36:53] ... works really well. Or you, you can couple them. So I, I think it's actually not released. They're releasing it, like, next week. Is, uh, in the settings for an agent, you can give it access to invoke any other agent. **Swyx** [37:02] Hmm. **Simon Last** [37:02] So you can have them just, just, uh, uh, talk directly. Uh, so you- **Swyx** [37:06] Was, was there a limit on, like, number of recursions, or just- **Simon Last** [37:09] Um, uh, probably. **Swyx** [37:11] You know what I mean? Like, you can just get an infinite loop that way. **Simon Last** [37:13] There's some kind of... Yeah. **Sarah Sachs** [37:14] I think it's- **Simon Last** [37:15] Some sort of- **Sarah Sachs** [37:15] There is actually a number somewhere, I believe. **Swyx** [37:17] I'm just, you know, like- **Simon Last** [37:18] Yeah **Swyx** [37:19] ... you're, you're... Someone's gonna screw it up. **Simon Last** [37:20] You should just try it and see. **Swyx** [37:21] Yeah. **Simon Last** [37:22] I mean, everything's gonna be paperclips. **Swyx** [37:23] Oh, yeah, yeah. **Simon Last** [37:24] But, uh, but, but that's really useful. Yeah, so we're, you know, like, I just... I, I helped, uh, someone internally the other day. They had, they had built, like, over 30 custom agents for, uh, for our go-to-market team doing all kinds of different things. You know, for example, like researching, you know, like, like filling information about, about a customer, or like, like triaging customer feedback, or, like, uh, something like that. Literally over 30 of them. And, and then, and then he gave me, like, a database of all the agents, and then he was like, "Okay, and, and now I'm getting 70, over 70 notifications per day with just the agents are blocked on various things." Uh, and then I was like, "Oh, okay, cool." You know, the obvious thing to do there is to make a manager agent- **Swyx** [38:01] Right **Sarah Sachs** [38:01] Yeah **Simon Last** [38:01] ... that's gonna sort of- **Sarah Sachs** [38:02] And blocks **Simon Last** [38:02] ... be another abstraction layer in between your, your, uh, uh, 30 agents. Uh, so yeah, we, we set him up with, like, a manager agent, and it has access to invoke all the other agents, and it's sort of, like, like watching and observing them. And then it sort of, yeah, just, just creates a layer of abstraction. So instead of 70 notifications per day, it's like, like five. **Swyx** [38:18] Yeah. **Simon Last** [38:18] And then, and then the manager agent can help, like, uh, debug and fix any, any problems with the- **Swyx** [38:23] This is, is, is a concept of, like, a inbox or something like... Basically, you're basically saying that they can message each other. **Simon Last** [38:29] Yeah. **Sarah Sachs** [38:29] Well, they use a system of record- **Swyx** [38:30] They can absorb the notification **Sarah Sachs** [38:31] ... which is Notion. **Simon Last** [38:31] Yeah, so we actually... Yeah, we didn't make any special concepts at all. **Swyx** [38:34] They're interested to me- **Simon Last** [38:35] Yeah, I mean, like- **Swyx** [38:35] ... the Notion notifications that I would have got. **Sarah Sachs** [38:37] They can just, like, write a task to a database that the other agent's tasked to listening to- **Simon Last** [38:41] Okay **Sarah Sachs** [38:41] ... or they can actually call a web up to the agent. Like, they can just @ the agent. **Swyx** [38:44] Okay. **Simon Last** [38:45] Yeah, I mean, this is something that, that we're still working on. I, I think- **Swyx** [38:47] Yeah **Simon Last** [38:47] ... we, you know, like, like generally, generally the way we do these things is, you know, you first make it possible and maybe, like, a sort of janky way. So it's... I, I think the way I set him up is, like, you know, we created, like, a new database that was sort of, like, issues- **Swyx** [38:59] Mm-hmm **Simon Last** [38:59] ... that the custom agents were, were- **Sarah Sachs** [39:01] Mm-hmm **Simon Last** [39:01] ... were experiencing, and then gave them all access to file an issue, and then the manager has access to, to read the issues. Um, and that worked pretty well. Essentially, like, like, give it its own, like, internal issue tracker just for the agents. And then, you know, if that becomes a, a concept that seems useful generally, maybe we'll think of how to package it in. But I mean, generally, we try to just keep it to composing the primitives if we can, you know? Another example of this is we have no built-in memory concept. Memory is, is just pages and databases. And so if you wanna give it memory, just give it a page and give it edit access to that page. And- **Swyx** [39:31] And a human can edit it, agent can edit it. **Simon Last** [39:33] Yeah. And so that works, uh, that pattern works extremely well. And, and, you know, depending on the use case, you can have it be just a page, or it could be an entire database with, you know, or, you know, kind of subpages. It's, it's pretty endless what you can do with it. **Swyx** [39:43] So when I was setting this up, uh, I connected my inbox and it was like, "Do you wanna use Gmail or Notion Mail?" And I'm like, "I don't wanna use either. I just want you to do it." I'm curious how you think about, you know, Notion Mail, Notion Calendar, all of these kind of- **Simon Last** [39:56] Oof **Swyx** [39:56] ... UI, UX interfaces. **Simon Last** [39:57] Full stack Notion. **Swyx** [39:58] Yeah, when, like, at the same time you have the agents abstracting them away from you in a way. You know? **Simon Last** [40:02] Mm-hmm. **Swyx** [40:02] How do you spend, like, the product calories, so to speak? **Simon Last** [40:05] Yeah, I mean, I think it's pretty important that you don't have to use Notion Mail to get access to the mail capability, so we can just connect to Gmail or, or whatever you want, uh, to use. And we're thinking of the mail service as being really great, uh, to the extent that it's really agent built, right? So maybe the mail app is just sort of a prepackaged agent that helps you- **Swyx** [40:26] Hmm **Simon Last** [40:26] ... automate your, your inbox. **Swyx** [40:28] Yeah, the auto labeling is great. I think the- **Sarah Sachs** [40:31] When we, um, integrate with Gmail, for instance, we have a series of tools available that are available via MCP or API to Gmail. When we integrate with Notion Mail, we have the Notion Mail engineering team to build us the, um, exact right tools that optimize latency, optimize performance and quality. They own that quality. Um, there's product leads there that are directly thinking about the user problems that happen in mail. So it tends to be when we build integrations and connections, we build natively first, um, and then think about, um, extending them generally just because it's also easier- **Swyx** [41:06] Mm-hmm **Sarah Sachs** [41:06] ... um, to build natively first. Um, so that tends to be how we phase things out. **Swyx** [41:11] Talking about integrations, you prompted me, so I gotta ask. MCP, CLI, what's going on? What's the- **Simon Last** [41:16] Yeah **Swyx** [41:16] ... opinion? **Simon Last** [41:17] I think, I mean, I'm, I'm definitely bullish and excited about CLIs. **Sarah Sachs** [41:21] Yeah. **Simon Last** [41:21] I think there's a few really cool things about CLIs. So one really cool thing is, like, um, is that it's in the terminal environment, so it gets a bunch of extra power. So, like, you know, for example, I can, like, like paginate and, and cursor through, like, long outputs. Um, and it has, uh, progressive disclosure inherently. Uh, so, you know, you don't see all the tools at once. It's just you see the CLI wrapper and you can, like, use the, the help commands and, and, and read files. And then I think the most important thing that's, that's super cool is that there, it's also inherently, uh, uh, bootstrapped. So if there's an issue, uh, the agent can debug and fix itself within the same environment that it uses the tool. **Swyx** [41:58] Hmm. **Simon Last** [41:58] Right? Like, you know, I think I saw a, a tweet this morning. Someone said, you know, "My agent didn't have a browser, so I asked it to make itself a browser tool." And within 100 lines of code, it gave itself a little browser, like, like wrapping the, the, the Chromium API. Um, that's pretty incredible. And then if there was a bug, it would just immediately try to fix it. **Sarah Sachs** [42:14] Mm-hmm. **Simon Last** [42:15] Right? On the other hand, if you use an, you know, if you use, like, of, of the Chrome DevTools MCP, I've had this issue where, like, sometimes the transport gets, like, messed up. If it gets messed up, the agent has no way to fix itself. It, it no longer has a browser. It's, it's now broken, right? I think that's, that's pretty fundamental. But I would say, like- A lot of the, the bad things about it can be fixed. Uh, so I think, like, the, the progressive disclosure, that can be fixed with, with the right harness. Like, it, it obviously doesn't make sense to show it all the tools all the time. That's not really inherent to the MCP protocol. It's just, like- **Swyx** [42:44] Mm **Simon Last** [42:44] ... how you wrap it and use it. Um- **Swyx** [42:45] There's many poorly implemented MCPs because we didn't know. **Simon Last** [42:47] Yeah, yeah. I, I mean- ... it, it was just early. Like, like- **Swyx** [42:50] Yeah **Simon Last** [42:50] ... the obvious thing is, uh, you know, to start with, is, is to just show it all the tools, and it's like, okay, now we have 100 tools. **Swyx** [42:54] Yeah. **Simon Last** [42:55] And, like, when tool calling actually works, so let's, you know- **Swyx** [42:56] It's a victim of your success **Simon Last** [42:57] ... give it a way to, like, like, filter to search the tools, so. **Swyx** [42:59] Yeah. **Simon Last** [42:59] I would say, like, broadly speaking, I'm really bullish on CLIs. I'm still bullish on MCPs in, in a certain environment. I think in, in particular, MCP is really great for when you want sort of like a narrow, lightweight agent. **Swyx** [43:10] Mm-hmm. **Simon Last** [43:10] I think there's, there's definitely a lot of use cases where, where you don't want, like, a full coding agent with a compute runtime, and also you want it to be, like, more tightly permissioned. MCP inherently has a really strong permission model. Like, all you can do is call the tools. A CLI is a little bit murkier. It's like, can I access the API token? Are you, like, properly sort of like re-encrypting the token so it can- **Swyx** [43:31] Mm **Simon Last** [43:31] ... like, exfiltrate it. It introduces a lot of, like, like, new issues which are real and hard to solve, and MCP is just, like, the dumb, simple thing that works, and it, it is pretty good. **Sarah Sachs** [43:40] I'll add two more perspectives, not from it working well for Notion, but how Notion, like, commits to both platforms. Notion is dedicated to being the best system of record for where people do their enterprise work, so we will always support our MCP insofar as other people are using MCPs, right? So regardless of our perspective, we've m- put a lot of effort into our MCP, and we have a fantastic team that we're building, um, to do more there. And the second thing I'll say, I think, um, we all think a lot, but lately I've been thinking a lot about making sure there's a value alignment in pricing, um, with capability. **Swyx** [44:12] Literally our next question. **Sarah Sachs** [44:13] And needing language to execute deterministic tasks feels wasteful, and requiring on a language model to interface with third-party providers seems wasteful for tasks that don't require it. And particularly because our custom agents are using usage-based pricing, we think of pricing as, like, the barrier of entry for use of our product, and we're quite committed to making sure that it's not wasteful. ### Pricing **Sarah Sachs** [44:34] Um, not just because it's a bad deal for our customers, but it's also bad business. We want to have as many buyers. Like, there's a, there's an elasticity of demand. And so if we can have our agents properly execute code that calls on CLI deterministically, it's a one-time cost, right? Versus constantly having a language model integrate with an MCP over and over and over and paying those, like, repeated token fees, and it's happening outside the cache window, then you're paying for it over and over and over. And it's just kinda unnecessary and less deterministic when it doesn't have to be. **Swyx** [45:04] Yeah, the open-endedness, I think, is, like, the main thing. It's like, well, if I could write code to just call an API, I would never use an MCP. But then you need an MCP sometimes when you know what to call, but you don't want it to restart. Versus, like, I think the... It built a browser from scratch. It's like, it's great when you're doing it on your own, but, like, if your customers were having your AI write a browser from scratch every time and you had to pay the token cost of that- **Sarah Sachs** [45:26] Yeah **Swyx** [45:26] ... you'd be like, "No, no, the Chrome DevTools MCP is actually pretty great. Just use that." I'm curious, how do you make that decision? Like, should it be just straight API call, very narrow? Should it be an MCP? Should it be super open-ended? **Sarah Sachs** [45:38] Do you mean for when we ship Notion capabilities or when we add capabilities to Notion AI? **Swyx** [45:42] Yeah, or like, I mean, you might have a capability that the only way to do it is an open-ended agent, like an agent with a coding sandbox. **Sarah Sachs** [45:49] Yeah, in Notion AI. They're not explicit- **Swyx** [45:50] Yeah, yeah, yeah. Notion. Yeah, yeah **Sarah Sachs** [45:51] ... because, like, we also ship an MCP- **Swyx** [45:53] Yeah, yeah, yeah **Sarah Sachs** [45:53] ... in Notion AI. Yeah. **Swyx** [45:54] Internally. **Sarah Sachs** [45:55] Okay. **Swyx** [45:55] Like, is there ever a discussion of like, "We're not gonna ship it because we're not able to tie it down"? Or are you happy to just, like- **Sarah Sachs** [46:02] Um, no. I mean, there are a lot of things where we choose not to use MCP because we wanna add more high touch to quality. I think Search and Agentic Find is, like, the largest instance of that, where we have, um, Slack and Linear and Jira Search in Notion that is not using necessarily the search MCP functionality that is provided by those companies, and that's because it's quite critical, we think, to how our agent trajectories work, is for us to have a little bit more control on the functionality of the search journey. And so it, it usually comes from quality. And there's a long tail of things, and that's why we built an MCP client, or an MCP server, excuse me, so that people can connect to whatever they want. There is that long tail, right? But we, for Search particularly, I would say that's, like, the primary entry point, but there are other connections as well that it's a little bit of secret sauce about when we are okay with, like, MCP functionality and user-driven auth and when we actually wanna, wanna carry a lot more ourselves. **Simon Last** [46:59] I think that there's not m- really a conflict here. There's just, like, different layers of the stack and different abstractions. I mean, if we were to, like, map it out, it's like, you know, you've got MCPs give you a, a way to... It's a protocol for gaining access to tools. It's an open protocol, so you can, you can easily get, like, a long tail of many things. So if you open up our, like, in the tool settings- **Swyx** [47:21] Oh. **Simon Last** [47:21] That's not the trigger action. Actually, that's something that MCP can do. So if you scroll down and you, and you have the, the tools and access, so you're gonna add a connection, yeah. MCP is a really great way to gain access to tools. **Swyx** [47:32] Mm. **Simon Last** [47:32] It works really well. But you just looked at the, the trigger UI, for example. There's no trigger protocol, and so- **Swyx** [47:39] Mm **Simon Last** [47:39] ... those we've got to build ourselves. And then there's, there's some integrations the- where we use MCP. Like so for example, I think the, you know, the Linear and the GitHub- **Sarah Sachs** [47:48] Mm-hmm **Simon Last** [47:48] ... use MCP. But, but the Slack, uh, uh, mail and calendar, those are actually ones they built in-house, and we spent a lot of time really fine-tuning all the tools to make the rest really good, and also, like, building out the triggers. So it's just, like, different layers of the stack. Some things make sense sometimes, and then, you know, we just have to like, like, harness the right tool at the right time. I don't think there's an inherent, like, strong conflict between these things. **Swyx** [48:08] Do you have a canonical representation of these tools internally, where, like, you've wrapped these things together, the MCP plus the custom-built? **Simon Last** [48:15] Yeah, yeah. We have, like- ... internal abstractions for, like, what is a tool? What is an agent? What is a completion call? Yeah. **Sarah Sachs** [48:23] We even have internal abstractions for, like, what is a chat archetype, whether it be from Teams or Slack. **Swyx** [48:30] Mm-hmm. Yeah. **Sarah Sachs** [48:30] Right? **Swyx** [48:30] It's like the only way- **Sarah Sachs** [48:31] What is a tasker? **Swyx** [48:32] ... to build with, with AI, 'cause everything's moving so quickly. You would have to abstract it so that you can swap things out. **Simon Last** [48:37] Yeah. Yeah, I mean, there's always a dance. We, we probably Rebuilt our, our framework, like, like I said, like, like five different times. Um, it's always a dance of, like, "Okay, how does this new thing work?" **Alessio** [48:48] Right. **Simon Last** [48:48] What should the abstraction be? Like, what is OpenAI giving us? What is Anthropic giving us? Um, you know, like we're trying to wrap over it. I think, I think we've been pretty successful with that. It- it's just a matter of like, like staying nimble- **Alessio** [48:59] Yeah **Simon Last** [49:00] ... and making sure that you always have, like, the simplest, dumbest abstraction you can- ... that, you know, that the maps are different things. Yeah, so, so we have, like, a tool, a integration abstraction, for example, and then MCP is, like, a, a type of integration. **Alessio** [49:10] Yeah. **Simon Last** [49:10] That's, that's one of them. **Alessio** [49:11] This might be a big ask, uh, uh, but I'm gonna try, uh, which is you said, you've said multiple times you rebuild a few times, like five times through... I don't know if the, what the right number is. Is there, like, a brief history of what was the each rebuild doing? And yeah, I know it's- **Simon Last** [49:24] I can try to do that. **Alessio** [49:25] Okay. **Simon Last** [49:25] I mean, yeah, there's- **Alessio** [49:27] You just, you need, you need to rag over. **Sarah Sachs** [49:28] Hierarchical. **Simon Last** [49:28] Yeah. I mean, the first version, the first version that we started building in, like, late 2022, oh my gosh. Well, there have been many versions, actually. Okay, well the- **Sarah Sachs** [49:36] It's brighter. **Alessio** [49:37] Right, the highlights. **Simon Last** [49:37] Yeah, yeah. **Alessio** [49:38] The, like, oh, wow. **Simon Last** [49:39] The, uh, the first version we built was actually a coding agent. **Alessio** [49:42] Yeah. **Simon Last** [49:42] Uh, so we're like, "Oh, instead of building tools, let's make everything be JavaScript, and then we'll just give it JavaScript APIs, and it will just write code, and that's how it speaks the tools." Um, but at the time, it just sucked at writing code. It wasn't that good. Uh, so then we moved to, uh, more of like a tool-calling abstraction. Uh, tool calling didn't exist yet, so we, we created this whole XML- **Alessio** [50:01] Mm-hmm **Simon Last** [50:02] ... representation. And a big, a big learning in that version is we were catering way too much to what made sense for Notion and Notion's data model versus what the model wants. **Alessio** [50:13] Mm-hmm. **Simon Last** [50:13] So as an example, we created this whole, uh, XML, uh, format that can losslessly map to Notion blocks, and the transformation between them is super easy to do. Uh, and then we created these sort of like mutation operations to, to edit pages. Um, but it sucked because the model didn't know the XML format. And also the- **Alessio** [50:32] And you had to prompt it in and- **Simon Last** [50:33] Yeah, you had to prompt it in- **Sarah Sachs** [50:33] Like, a lot **Simon Last** [50:33] ... and, and the tools just weren't convenient. And so, yeah, we're like, "Okay, well, it has to be Markdown." Uh, the models know Markdown, you know. So, uh, uh, we did a whole project around basically, uh, uh, creating a Notion-flavored Markdown, where, uh, you know, the whole goal was, like, it has to be just simple Markdown at the core, and, and then we can add some enhancements, and it doesn't have to be a f- a full lossless conversion. That was a big one. We did... And, and then we did a whole similar learning to, uh, the, the database layer. So, so, so querying a database. I mean, in the Notion API, the way you query a database is there's a crazy JSON format, and it's, you know, kind of limiting, but it maps nicely to, like, how we represent things internally. We scrapped all that, and we're like, "Okay, let's just make it SQLite." Everything's a SQLite database. You can, you can query it just like a SQLite query, and the models are super good at that, so. **Alessio** [51:20] Give the models what they want. ### Rebuilds **Simon Last** [51:20] That was another one. Yeah. Give the models what they want. I mean, that was... I would say that was a big learning is just, you know, really be, be savvy and really careful thinking about what the model wants in terms of, you know, its environment and, and, and cater around that. And really try so hard not to expose it to any complexity about your system that that's unnecessary. **Alessio** [51:40] Notion's underlying database is Postgres, right? Not SQLite. **Simon Last** [51:43] Yeah. **Alessio** [51:44] So I don't know if there's any mismatch there. **Simon Last** [51:46] That one was kind of a fortuitous thing because we actually already, um, had a big project, uh, going where... So, so we have this, um, when you query a Notion database, it's actually querying this, like, uh, cluster of SQLite databases. **Alessio** [52:02] Mm-hmm. **Simon Last** [52:03] That's something that we'd already been working on even before the agents came around. **Alessio** [52:06] Yeah. You know, you guys had a fantastic blog post about it in May. **Simon Last** [52:09] Yeah. **Alessio** [52:09] It's, it's actually really good database engineering knowledge to have that from you guys because where else would we get it? **Simon Last** [52:16] Yeah. Yeah. It's a, it's, it's a crazy engineering problem when you wanna have, like, millions and billions of- ... tiny databases or where, where some of them are tiny, but some of them are, are very large and you want everything to be very fast. **Alessio** [52:25] Yeah. And also, like, not that hierarchical sometimes- **Simon Last** [52:28] Yep **Alessio** [52:28] ... you know? Uh, so somewhat of a graph. **Simon Last** [52:31] Mm-hmm. **Alessio** [52:31] I do like that history because I think that shows the evolution that you guys went through and the work that went into it. **Sarah Sachs** [52:37] That, he just ended you at a year and a half ago. **Alessio** [52:39] Oh, okay, okay. **Simon Last** [52:39] Yeah. **Alessio** [52:41] Oh, I need to, I need to hit continue. **Sarah Sachs** [52:42] If you're curious- **Simon Last** [52:43] I can do another one. Yeah **Sarah Sachs** [52:43] ... I mean, we can keep going. **Alessio** [52:45] These models are lazy. They stop. **Sarah Sachs** [52:45] I'm just saying, like, that's really- **Simon Last** [52:46] That's another one. Yeah. **Sarah Sachs** [52:47] I mean- **Simon Last** [52:48] Let me think **Sarah Sachs** [52:48] ... well, no, 'cause there was tool calling, and then there was research mode, which wasn't a fully agentic tool calling. Um, then we moved- **Simon Last** [52:55] Yeah **Sarah Sachs** [52:55] ... away from few-shot prompting entirely to tool definitions. Um, and now we're thinking about Agent 2.0. **Alessio** [53:02] So no few-shot prompts ever, right? **Sarah Sachs** [53:04] Uh- **Alessio** [53:04] Okay. Now, now you're not clear, but- **Sarah Sachs** [53:05] I, I don't know if never, but- **Simon Last** [53:07] Yeah, that kind of went away. It's an interesting thing. **Alessio** [53:08] Right? **Simon Last** [53:09] Yeah, I mean- **Alessio** [53:09] So these just instruction follow really well. **Simon Last** [53:12] I would say there's been, like, a general arc where, you know, it's like you gradually strip away everything, and it, it looks more AGI-like. And so, you know, it, it, it started out as, like, it's a one-shot, one prompt. There's few-shot examples, and it became like, "Okay, actually, let's give it, let's give it tools, but it'll still have few-shot examples." And then it became actually like, "No, no, let's just give it a whole bunch of tools." One big, big shift that, uh, uh, that we- I've been working on recently that's about to ship is, um, you know, what happens when you have a lot of tools. **Alessio** [53:41] Yeah. **Simon Last** [53:42] So then- **Alessio** [53:42] Tool search. **Simon Last** [53:43] Yeah. So then, uh, uh, progressive disclosure- **Sarah Sachs** [53:45] Right **Simon Last** [53:45] ... becomes really important. So, you know, we were... We sort of hit a bottleneck where our, our agent worked really well. Um, we hit a bottleneck where, um, it, it, it became pretty hard to add new tools- **Alessio** [53:56] Mm-hmm **Simon Last** [53:57] ... and we, and we became sort of worried about it, like, like breaking the model. It's like, okay, someone- **Sarah Sachs** [54:00] No, I just heard it was, like, saying hello was, like, thousands and thousands and thousands- **Simon Last** [54:03] Yeah. Yeah. It was- **Sarah Sachs** [54:04] It was really slow. **Alessio** [54:05] I can see you're the efficiency person here. **Sarah Sachs** [54:08] Yeah. **Simon Last** [54:08] It's, it's- **Sarah Sachs** [54:08] Yeah. **Simon Last** [54:08] It was too many tokens, but also it's a quality issue, uh, because it meant that, like, any engineer could introduce this, this new tool for some, like, like, niche feature, and it would kind of like, like, nerf the overall model by, like, causing it to call the tool too much and stuff like that. And so, um, it, uh, yeah. So we, uh, we had an effort basically to, to make our harness, uh, implement progressive disclosure in, in a nice way. **Alessio** [54:27] Mm-hmm. **Simon Last** [54:28] Um, that's a big shift. **Sarah Sachs** [54:28] You said earlier, like, everyone says reasoning models was the big shift. Like, what's more there? When we went away from few shots to describing the goal of the tool in, like, goal-driven, basically moving from a DAG- To like a, a true system with feedback, that's when we could distribute tool ownership to the teams much better. Because when it was all few-shots, it was everyone truly editing one string, and things would o- would compete in, like, the order. There were all this, all these papers about, "Oh, you know, not all context is created equal. The higher up it is in your examples, like, the more the model listens." And we're trying really hard to, like, fight against the order and the selection of the few-shot, and that really had to be a center of excellence. And it didn't scale with the number of people for the need the company had. It was really just five or six people that were allowed to even touch that, or had to approve it, rather, in our code base. And then now we can actually, with the right eval setup, distribute, um, so that everyone owns their tool and their tool definition. And sometimes we have crazy things where, like, we write two tools that have the same title and the agent crashes and stuff like that. So like, you know, there are issues. Y- Actually, believe it or not, um, Anthropic couldn't take it, Sonic couldn't handle two tools with the same name, and OpenAI GPT 5.2 was like, "I can figure this out." **Swyx** [55:41] Mm. **Sarah Sachs** [55:41] So that was an interesting one that we learned by accident through a, a sev. But- **Swyx** [55:45] I mean, then, then you know the underlying representation is, that's a dict, right? **Sarah Sachs** [55:48] Right. Exa- **Swyx** [55:50] Like, clearly, like, that's a CT teenage. **Sarah Sachs** [55:51] Yeah, exactly. Exactly. **Swyx** [55:52] Like **Sarah Sachs** [55:53] Um, but so that was, like, a big shift for the company in velocity. Uh, not immediate, because the AI team that was the center-of-excellence team that owned, you know, that one file of few-shot prompts had to become a platform team overnight, and that wasn't natural. **Swyx** [56:07] Yeah. **Sarah Sachs** [56:07] But I would say that in terms of, like, the velocity of how we contribute to the agent, beyond coding tools obviously being a big velocity lever, um, being able to distribute tools and not have to all collaborate on, like, one very select string of system prompt is truly, I would say, the biggest lever on how we've scaled. **Simon Last** [56:26] We're just fighting to keep the prompt as short as possible now, and then- **Sarah Sachs** [56:28] Yeah. **Simon Last** [56:29] Yeah. It's, uh, in the latest version of the agent. I- it's not in Custom Agents yet, but it will be like, like next week- **Swyx** [56:32] Mm **Simon Last** [56:32] ... or week after or so. Um, there's now, like, over 100 tools just for all, all the crazy Notion stuff. So we're able to, to really go deep and, like- **Swyx** [56:40] Would you list those tools publicly? Is this, like, IP or- **Simon Last** [56:43] Uh, no, no, no, it's, it's totally public. **Swyx** [56:45] Okay. **Simon Last** [56:45] Uh, you can ask the- **Sarah Sachs** [56:46] You can define, just ask. **Simon Last** [56:47] Well, you can just ask the agent- **Swyx** [56:48] Yeah **Simon Last** [56:48] ... and, and, and it will tell you. **Swyx** [56:49] So I find- **Sarah Sachs** [56:50] And we're gonna post a bench- I mean, like- **Swyx** [56:51] You're gonna post a benchmark. **Sarah Sachs** [56:52] We don't think our system prompt is our secret sauce. **Swyx** [56:54] Yeah. **Simon Last** [56:55] Mm-hmm. **Swyx** [56:55] Great. **Simon Last** [56:55] We don't try to hide the tools at- **Swyx** [56:57] Yeah **Simon Last** [56:57] ... at all. I think it's, I think it's kind of important actually as an operator, you know- **Swyx** [57:01] Yeah, as a power user- **Simon Last** [57:02] Yeah **Swyx** [57:02] ... I wanna be like, "Oh, it can do this, this, this? Great." **Simon Last** [57:04] Yeah, yeah. I mean, one thing that- one phrase we say internally a lot is to, to teach to the top of the class. You know- **Swyx** [57:09] Really? **Simon Last** [57:09] ... we wanna build, like, like, the custom agent's kind of like a power tool. I mean, we try to make it as easy as possible to set up, but we want it to be pretty deep and sophisticated. And I think a huge part of that is the operator needs to be able to interrogate the way the system works. And a big part of that is like, "What are the tools? How do they work?" You know? Like, like, "How should I prompt it t- to use the tools in the right way?" **Sarah Sachs** [57:29] I'd actually say we don't try to make it as easy as possible to use, 'cause the more we do that, the more we abstract away that interpretability that Simon's talking about that basically nerfs the model or nerfs the agent- **Simon Last** [57:39] Yeah, yeah **Sarah Sachs** [57:39] ... from being super capable. **Simon Last** [57:40] Yeah. **Sarah Sachs** [57:40] So a huge, I would say, turning point, I can think about, like, the week and a half that we all came together on this as we were building Custom Agents, was that alignment that we're not trying to build for everyone here. We're not trying to build the model that, um- or build the user experience that anyone can figure out how to use. 'Cause the more we do that, the more we just diminish its capabilities. And that was a big, you know, everyone in a couple Slack messages aligned on that, that actually made us all work faster again, right? 'Cause we all were, like, more centralized on who we were building for. **Swyx** [58:09] What does the meta prompt generator look like? So I looked in the- **Sarah Sachs** [58:12] Mm **Swyx** [58:12] ... system prompt that it gen- for example, it uses emojis. That's not a, you know, obvious thing to be doing. **Simon Last** [58:19] Wait, did you just ask it, "What's your system prompt?" Oh, th- th- no, this is how to generate prompts. The prompts to generate prompts. **Sarah Sachs** [58:23] We call it a setup gen. It's a setup. **Simon Last** [58:24] Well- **Swyx** [58:24] Yeah. **Simon Last** [58:24] Well, so this is actually just the agent. So- **Swyx** [58:27] Yeah **Simon Last** [58:27] ... so one thing we did that, that I really like with the custom agents is it can set itself up. So we not only gave it access to use the tools it has access to, like send your emails or whatever, uh, but it has more tools to set itself up and to debug itself. And so when you ask it to write a system prompt, it's just your agent itself is doing that. **Swyx** [58:45] So this is just the model preference. **Sarah Sachs** [58:46] Um- **Swyx** [58:47] You're not really injecting- **Simon Last** [58:48] And then- **Swyx** [58:48] ... into the model too much. **Sarah Sachs** [58:49] I mean, we're saying what makes a good custom agent- **Simon Last** [58:52] Yeah **Sarah Sachs** [58:52] ... and- **Swyx** [58:52] Yeah **Sarah Sachs** [58:52] ... and things like that. And then, and it- and it's really nice too, because, like, if it fails, you can ask it why did it fail, and then say, "Okay, update your instructions so it doesn't fail again." Obviously, we should build product of self-healing. That's, that's next- **Swyx** [59:04] Yeah **Sarah Sachs** [59:04] ... on our roadmap. But, um, it actually, it creates a nice system. **Simon Last** [59:08] Yeah. We do essentially give it, like, a development guide. Here's, you know, here's how to make a custom agent. Here's how to, like, like, help the user test it end to end, you know, to, to help them gain confidence that it works, stuff like that. **Swyx** [59:18] Mm-hmm. Yeah. Yeah, the fixing thing worked. I mean, it wasn't automatic, but I, I miss set something up, and then there was, like, a fix button- **Sarah Sachs** [59:25] Yeah **Swyx** [59:25] ... and that just- **Simon Last** [59:26] Yeah, yeah. **Sarah Sachs** [59:26] Um- **Simon Last** [59:27] I mean, one thing we're- **Swyx** [59:27] Fix this agent and it fixes it **Simon Last** [59:29] When we- **Swyx** [59:29] So **Simon Last** [59:29] It's, it's actually a, it's an interesting sort of permission problem. So like- **Swyx** [59:33] Right **Simon Last** [59:33] ... the thing about custom agents that is that by default it has no permission to do anything, and then you have to explicitly grant it all of its permissions, and that's what lets you trust it can work in the background, right? Like, you can know, like, "Oh, it, it can read my email but not send email. Okay, I can trust that." **Swyx** [59:48] Yeah. **Simon Last** [59:48] Right? If you let it fix itself, you know, you're, you're breaking that- **Swyx** [59:52] You're not- **Simon Last** [59:52] ... that permission there. It's, it's, it's not allowed to edit its own permissions. But as, so, you know, in the current product, you can sort of click a button to fix, but now you're entering sort of an admin mode where, where, where you're in a synchronous chat and, and you can, you can see what it's doing. **Swyx** [1:00:03] Yeah. **Sarah Sachs** [1:00:03] And it, and it confirms before it changes. **Swyx** [1:00:06] Yeah. The thing that I really like that most people don't do is, like, the editing chat is the same thing as the using chat. Like, you can message the agent to both edit it and use it, versus a lot of other products are like, you know- **Simon Last** [1:00:17] I think that's really key. I think- **Sarah Sachs** [1:00:18] I think a lot of designers will feel so happy you said that- **Simon Last** [1:00:20] Yeah **Sarah Sachs** [1:00:20] ... 'cause we spent- **Simon Last** [1:00:21] Yeah. **Sarah Sachs** [1:00:21] We ca- we called this Flippy. Um, uh, yeah. **Swyx** [1:00:24] Yeah. What is this? What do you mean this? **Sarah Sachs** [1:00:25] Um, this view of- **Simon Last** [1:00:26] Well, well, the... Yeah. So if you sort of, if you close that and, like, open settings, you can see sort of... Yeah. This is... We, we called it Flippy because, you know, we started with sort of like the settings were the sort of the main page, and then you can test the agent. The AGI pill way to think about it is like, oh, it's just the agent. Everything's the agent. **Swyx** [1:00:43] Right. **Simon Last** [1:00:43] It can set itself up, it can test itself, and it can run the workflow that you, that you want it to run. Uh, so we flipped it. So- The main view I was looking at is the chat, and, and then the settings is more just like a side panel that, that's sort of previewing the changes that it's making so you can introspect on them or, or you can also make changes manually if you'd like. But, but we wanna design the experience from the get-go so you don't have to ever any of the settings manually. You can just talk to it. **Sarah Sachs** [1:01:07] And the inside baseball is like how this works was probably the launch blocking part of this build. **Simon Last** [1:01:13] Right. **Sarah Sachs** [1:01:13] Um, especially 'cause we had a lot of early adapters that were used to the old way, and that's like the benefit of adapting in public, but then changing how people think about setting up custom agents when they already had this flow in and of itself was difficult. Um- **Simon Last** [1:01:26] I mean, that's really fun 'cause the, we, we ended up sort of, uh, uh, painfully delaying the launch- **Swyx** [1:01:31] Mm-hmm **Simon Last** [1:01:31] ... by- **Sarah Sachs** [1:01:32] A month **Simon Last** [1:01:32] ... a few week- yeah, definitely, like, like a month or so. Um, but the whole team was super enthusiastic about it though 'cause it was just so much better. It was like, "Oh yeah, obviously you get the chat with it-" **Swyx** [1:01:42] Right. Yeah, yeah. **Simon Last** [1:01:42] "... to set itself up," and everyone was super bullish on that. So it was like, like painful for a second, but then everyone was like- **Sarah Sachs** [1:01:48] Right, and like back to- **Simon Last** [1:01:49] ... fun **Sarah Sachs** [1:01:49] ... you know, organization design, which I probably care about more than Simon, but like the people that built this are three engineers from three different teams because we're like, "We need to launch this and we need to fix this" And then we've just built a company where then we just put people on it and no one complains, the manager doesn't complain, and we were able to unblock and just ship it. **Swyx** [1:02:06] Yeah. Yeah, but being in a failure chat and asking it to just fix yourself is amazing, versus I gotta copy this and put in the settings chat- **Sarah Sachs** [1:02:16] Mm-hmm. **Swyx** [1:02:16] ... to do it. **Simon Last** [1:02:17] Yeah. **Swyx** [1:02:17] So. **Simon Last** [1:02:17] Yeah, it was very interesting, like, uh, trade-off in there that I wish I might explore, which is, you know, we wanna be like a business enterprise safe agent where you can delegate something and, and trust that it's gonna work, but also we wanna get some of that sort of bootstrapping power that- **Swyx** [1:02:31] Mm **Simon Last** [1:02:31] ... that you feel like when you're coding it, just making a browser like for itself, right? There's something there. I think that's, that's really important. So it's, we're trying to sort of navigate that, that, that trade-off and try to get you both. **Swyx** [1:02:41] Now it's free. **Sarah Sachs** [1:02:42] Yeah. **Swyx** [1:02:43] It's amazing. Uh, I'm worried- **Simon Last** [1:02:44] Thank you **Swyx** [1:02:44] ... about when I have to start paying. How do you think about... So you have Notion credits as a payment for this, which is like separate from the usual tokens, uh, that the model generates. How do you design pricing, value-based pricing based on the task and things like that? **Sarah Sachs** [1:02:59] So there, um, the credits and payment structure is associated with the token usage. The reason that we had to make it not just throughput of tokens is that it's not always priced that way. Like our, um, fine-tuned open source models are served on GPUs. **Swyx** [1:03:12] Right. **Sarah Sachs** [1:03:12] Web search is priced differently. You know, if we were to host sandboxes, those are priced differently. So we had to think of an abstraction above tokens, and it's also not just tokens. It's the token model, um, and serving tier trade-off, right? **Swyx** [1:03:24] Mm. **Sarah Sachs** [1:03:24] Because we can have priority tier processing. We can have asynchronous processing. The cache rate could be different, um, depending on who uses it when, right? And so we wanted to, um, from the get-go, commit to making sure that customers were getting the fair deal, not necessarily that we were making a ton of money off of it, but that customers were paying for what was reasonable. That's the fundamental of where we started, and also, you know, we're selling enterprise SaaS, so if we sell credit packs, then you get discounts if you're an enterprise and you buy a certain amount of credit packs and things like that. So it also just helped the sales motion, um, work a little bit easier. So that's the answer on the abstraction of credits to dollars. Now, was the question how we decide how to price it or- **Swyx** [1:04:02] Yeah, like I mean, I think there's all tokens are not made equal, but- **Sarah Sachs** [1:04:06] Yeah **Swyx** [1:04:07] ... we obviously get charged mostly equal. Like you can ask, uh, Codex to create you a dumb tool for like... I created one for our StarCraft II LAN for people to like find a game. Uh, but then people create it to build features in like billion-dollar companies, but the token price is the same. **Sarah Sachs** [1:04:22] Yeah. **Swyx** [1:04:22] Like for you, I can ask this to update my favorite recipes doc, and it'll do it, but I could ask it to like respond to an email from an investor, and like the value is like very different, you know? And you could charge more, but you're not necessarily doing it. So I'm curious if there was any discussion. **Sarah Sachs** [1:04:40] I think, I think that, um, that's not where the market is right now, um, number one. The second reason that we're not doing that is it ended up being kind of complicated to figure out what was complicated or not. So we, at first were like, "Let's just charge on agent runs," and you know what? You went through all the different versions that ultimately just brought you back to a lot of complexity that mapped directly to token throughput. And so it, it's also just simpler. **Swyx** [1:05:01] Right. **Sarah Sachs** [1:05:01] Um, it's quite difficult, um, to build those pricing systems. And, um, I actually think that one of the biggest reasons we want had usage-based pricing for this capability is we've had our core agent for a while with a model picker, and there were certain models, um, or certain functionality that we had margins to maintain, and if we wanted to ship this functionality, uh, you-- we couldn't afford it. It would bankrupt the company. If we let, for instance, like autofill or the database autofill feature will soon be agentic. That will be associated with usage-based pricing because if every single autofill action was an agent running on Opus on every single database cell, it would be billions of dollars, right? And so we had to find a way for the customers that wanted to do more and wanted to give us their money and pay more to find the outlet for them to do it that we didn't have to apply to the lower end of the curve. And also, not all knowledge work is equal. Like, there's different points. A lot of the agent workflows here really saturate model capabilities. Like, you don't need- **Swyx** [1:05:59] Mm-hmm **Sarah Sachs** [1:05:59] ... a complicated model for it. And so charging based on token usage, um, it- we couldn't just decide for you that you wanted your email client to be dumb or not. **Swyx** [1:06:08] Right. **Sarah Sachs** [1:06:08] Like, we want you to decide. If you want to have Opus auto triage all of your emails, we will actually give you nudges in the product to rethink if that's the right choice. **Swyx** [1:06:17] Right. **Sarah Sachs** [1:06:17] Um, because also not every user, um- **Swyx** [1:06:20] Understands **Sarah Sachs** [1:06:21] ... you'd be surprised in user interviews, people will be like, "Oh, I didn't know that." So now we actually have a little hover that tells you like if it's expensive or not. Yeah, I mean, it's also slower. So the thing that's interesting is like people don't care about speed in custom agents, and so- **Swyx** [1:06:33] Mm **Sarah Sachs** [1:06:34] ... the incentive of like, uh, Haiku being faster, people don't care when it's asynchronous. Um, and so we want to only provide the service of extra, extra benefit that people want, and the best way to do that is to incentivize them because it's their own money. **Swyx** [1:06:49] Yeah. Must be confusing for people that are not familiar. It's like, why is there no five point three? You know, you open- **Sarah Sachs** [1:06:54] A hundred percent **Swyx** [1:06:54] ... this thing and it's like, "Is there something missing on my menu?" **Alessio** [1:06:56] It's not my fault, not their fault. **Swyx** [1:06:58] Yeah. **Sarah Sachs** [1:06:58] I think it's- **Alessio** [1:06:59] That's just the world we live in now. **Swyx** [1:07:00] Yeah, it, it's- **Alessio** [1:07:01] It just randomly jumps point two. It's like Claude had that. **Sarah Sachs** [1:07:03] I mean- **Swyx** [1:07:03] Yeah. **Sarah Sachs** [1:07:04] ... but Auto is heavily... I think what's actually been hard for us is to tell, convince people that Auto is not just our cheapest, dumbest model, but actually the model that's best for the task that you wanna do. Um, but- **Swyx** [1:07:14] All right, Steve. I mean- **Sarah Sachs** [1:07:16] Exactly **Swyx** [1:07:17] ... Auto. **Sarah Sachs** [1:07:17] Nice. Um, and a lot of our job is actually figuring out Auto because it's- **Alessio** [1:07:22] This is the Agent Lab. **Sarah Sachs** [1:07:24] Um- **Alessio** [1:07:24] Every Agent Lab has an Auto. **Swyx** [1:07:25] Mm-hmm. **Sarah Sachs** [1:07:25] Yeah, and- **Alessio** [1:07:26] 'Cause that's the job. **Swyx** [1:07:27] Yeah. **Sarah Sachs** [1:07:27] Exactly, because if you think about, like, like I said, I come from Robinhood, like you could spend a lot of time keeping up with the markets, or you could have a auto investing, right? And you can have an index fund, or you can have- **Alessio** [1:07:40] Robo-advisors of- **Sarah Sachs** [1:07:41] The robo-advisor **Alessio** [1:07:42] ... of investment firms. **Sarah Sachs** [1:07:42] And so, like, at a certain point, we also can be robo-advisors, and like we have a lot of people figuring out what model is best for the right task, and right now we're not using Auto as a, as a margin maker. We're just using it to kind of reduce stress. It's not Opus, that's for sure- **Swyx** [1:07:57] Yeah **Sarah Sachs** [1:07:57] ... because a majority of the tasks people are doing aren't Opus-level, um, intelligence. **Simon Last** [1:08:02] The thing I would say is the, um, you know, unlike a lab, we aren't fully incentivized just for you to use as, as many tokens as possible. We're actually really interested in giving you the right tool for the job. A lot of the time, the right tool for the job is actually just writing code and not even using Agent at all, so that's, that's something that we're- **Swyx** [1:08:19] Mm **Simon Last** [1:08:19] ... investing in a lot is like, you know, imagine your, your agent can actually automate itself out of a job. Um- **Swyx** [1:08:25] Right **Simon Last** [1:08:25] ... we would love if that were true. **Sarah Sachs** [1:08:27] I feel very strongly about this because I don't necessarily feel like that's the SKUs that Frontier Labs give you. I feel like they are just getting more and more capable and more and more expensive, which is fantastic for the use cases of when people wanna do really complicated things on Notion. Um, what's difficult is, like, that market that I think right now is no man's land of where reasoning models were six months ago, that the Nano, Haikus, et cetera, haven't caught up to because now we're just paying more for those, um, for, like, extra capability that we didn't necessarily need, and so are our customers. **Swyx** [1:08:58] Mm-hmm. **Sarah Sachs** [1:08:59] And, um, labs aren't necessarily incentivized, um, right now with how few players there are to be meeting the market everywhere. They just need to be the cheapest. They don't need to be at value that the customer wants. **Swyx** [1:09:10] Mm. **Sarah Sachs** [1:09:10] If no one's cheaper than them, then they're the cheapest, and that's good enough, right? And so we're doing a lot to make sure that we have the right optionality, um, to switch between models and also invest in open source because the open source models actually are, um, getting to be the place where reasoning models were three, four months ago, and, um, that's what's filling that gap right now. So you'll see we offer Minimax, and, um, we're collaborating a lot with different open source labs to think about Notion's last exam and how they can do better on these types of tasks so that we can offer them for that intelligence-to-price-to-latency trade-off because, you know, in that triangle of intelligence, price, um, intelligence, price, and latency, excuse me, um, users get to choose where they are. But right now, um, there's not... The whole triangle isn't filled with models, right? **Swyx** [1:09:58] Yeah. **Sarah Sachs** [1:09:58] And the more that different models fill that triangle- **Swyx** [1:10:00] Everyone's clustered on capability **Sarah Sachs** [1:10:02] ... everyone's clustered on capability, or everyone's cluster- I mean, Haiku's not that much cheaper. No one's really in the middle. Like, people re- really tend to cluster around two. **Swyx** [1:10:09] Mm-hmm. **Sarah Sachs** [1:10:09] Like, this is really capable, and it's really fast, but it's really expensive or whatever, right? And so we just wanna make sure that that triangle is filled. Um, and we wanna offer the models that fill it, and we wanna, um, ga- guide users to understand when they need it- **Swyx** [1:10:22] Yeah **Sarah Sachs** [1:10:22] ... um, which one. **Alessio** [1:10:23] I mean, all I'm hearing is that someday you're gonna train your model. **Swyx** [1:10:26] You have lots of tokens. **Sarah Sachs** [1:10:29] I don't know if... What do you mean by train your model? **Alessio** [1:10:31] Yeah, train your own- **Sarah Sachs** [1:10:32] I don't know if we have the money- **Alessio** [1:10:33] Train your own model **Sarah Sachs** [1:10:33] ... to train a founda- I mean- **Simon Last** [1:10:35] You go raise it. **Swyx** [1:10:35] Yeah, you, you can raise it. You, you- **Sarah Sachs** [1:10:37] You can raise it. That's your job, Simon. No, I, I don't think that that needs to be our core competency. **Alessio** [1:10:43] This is usually the, the thought process that leads to, like, "Well, no one else is doing it. We, we'll take a crack," you know? **Simon Last** [1:10:47] I think, um, yeah, I mean- ... I feel like to the extent that we do anything like training, the, the area I'm actually most excited about is, um, less of, like, one big model for all the users, but, like, as, as it goes, as it becomes more possible to do, you know, to make, like, specific fine-tuning that's, like, really knows your context of, you know, your company, the people that work at your company, what's going on, I think that's, that's pretty interesting because if you, if you had a model that really knows your company, I think that would be, like, a huge quality uplift. Um- **Sarah Sachs** [1:11:16] We actually have some enterprise vendors that kind of ask about this, um, along with bring your own key. Like, if I have a model that really understands, like, my enterprise that we're training for all these reasons, these tend to be, like, quite large institutions thinking about how to let people bring their own models, but those models have to function with, like- **Swyx** [1:11:32] Right **Sarah Sachs** [1:11:33] ... understanding how to call our tools, and that's where, again, having, um, more public system prompt is, like, beneficial to Notion, right? Um, we want all models to plug into Notion as, as, as well as they can. Um, that being said, like, of course, there are certain aspects of Notion where we do fine-tune and do reinforcement fine-tuning on our own capabilities. Um, but that's not necessarily trained on user data. Um, you don't need that, that much data, um, in the first place, and that's where when we have, like, a data scientist and a, a model behavior engineer really understand where the capability gap is, that's when we invest there. **Simon Last** [1:12:07] I personally burned a lot of time trying to train models. Uh, and- **Alessio** [1:12:11] It's tempting, right? **Simon Last** [1:12:12] It's so tempting. **Sarah Sachs** [1:12:12] You trained- **Alessio** [1:12:13] It feels so cool **Sarah Sachs** [1:12:13] ... retraining every day? **Simon Last** [1:12:15] I was doing crazy amount... Yeah, I was doing a lot of different things. Um, and it, it's the- **Sarah Sachs** [1:12:18] I was the budget person that came in- ... and, and popped up. When I showed up and I heard that that was happening- **Swyx** [1:12:23] Time out. **Simon Last** [1:12:24] You know, like, a, a, a funny thing that 'cause the, the sort of an arc that, like, looped on itself is, uh, you know, back when I was doing tons of training stuff, it takes a long time to do any kind of training run. And so you end up operating, like, like, twenty-four/seven around the clock. Like, it becomes very important that before you go to sleep, like, everything is- **Alessio** [1:12:41] It's large intensive work **Simon Last** [1:12:41] ... all the experiments are, are started. And then as I stopped training, that kind of went away, but now the coding agents have totally brought this back. **Swyx** [1:12:47] Mm-hmm. **Simon Last** [1:12:47] So now every night before I go to bed, I'm like, "Okay, did I start enough agents, you know, to get them done? I get everything done." So it's, it's sort of- **Swyx** [1:12:53] It's, it's- **Simon Last** [1:12:53] ... you know, it's a discipline. **Alessio** [1:12:53] Yeah, this is interesting to, like... You have to try polyphasic sleep so you can wake up every two, three. **Simon Last** [1:12:59] I saw that. Yeah. Yeah, we, uh, yeah, I have not- ... gotten there yet, but, but my goal these days is just to, before I go to bed- The agents are running, and I'm confident that they won't be done by the time I wake up. **Alessio** [1:13:09] Really? Eight hours. **Sarah Sachs** [1:13:10] He- there is a... I won't say which coding frontier lab, but there is a point where he had, like, outlived, like, the thread length and context length- **Alessio** [1:13:17] Uh-huh **Sarah Sachs** [1:13:17] ... that that coding agent provided. And I DM- you DM'd them being like, "Hey, I need, I need more." And our account rep DM'd me directly, and they're like, "Is Simon trying to prove string theory? Like, what is he doing?" **Simon Last** [1:13:28] Yeah. **Alessio** [1:13:28] Yeah. **Simon Last** [1:13:29] I, I had a single coding agent thread going for, I think it was, like, 17 days. Uh, pretty much continuously. **Alessio** [1:13:35] Don't, don't they just compress? I mean, I- **Simon Last** [1:13:36] Yeah, yes. It, it was actually just a bug. It was a harness bug. Yeah, it had done compaction, like, 100 times probably- **Alessio** [1:13:41] Yeah **Simon Last** [1:13:41] ... overnight. **Alessio** [1:13:41] Yeah, yeah. **Sarah Sachs** [1:13:42] The other thing that, um, reminded me about fine-tuning that I think you and I have aligned on is that our tools change really frequently, and right now we spend a lot of time rethinking and building tools for capability, and fine-tuning a model, um, to understand your tool. Like, we don't have legal expertise or coding expertise, so if we were to fine-tune a model, it would either be expertise about the enterprise, and, you know, we have ZDR no-data retention offerings for those enterprises, so we'd have to really rethink how we structure if an enterprise wanted to opt into that, or it would be fine-tuning and better capability on navigating our tools. That doesn't match with the velocity with which we create new tools, and so it would actually really slow us down, um, to have a model that was fine-tuned on our tools because we'd have to retrain it and cut a new model every time we did that, and that's not how we're set up right now, um, particularly with the way that we're changing our... I, I guess we could fine-tune a model to, like, search for tools. It's just the, the amount of time it takes to do that, ship it, have the right system, you're basically making a bet against a frontier capability not serving that in the time it takes you to build it. **Alessio** [1:14:42] Mm-hmm. **Sarah Sachs** [1:14:42] And that, that timeline hasn't happened for us yet. It hasn't been a fit. **Simon Last** [1:14:45] I think, yeah, it's just the wrong trade-off, I think. It's just like you want... Yeah, we literally change our tools every single day, and if we notice an issue, we'll, we'll, we'll fix the problem. I think a, a good way to think about it, I think it's pretty fruitful, is like don't focus too much on training. I would think of that as like that's an im- implementation detail. Like, what's the outer loop, right? Like, if the outer loop is you have a model and then some harness or, or system where it's interacting with the system, that needs to work. And, you know, if it's a problem, the way to solve the problem isn't necessarily to train a model. It's like, oh, maybe there's just a bug in one of the tools, right? And actually, 99% of the time it's a bug in one of the tools. **Sarah Sachs** [1:15:23] Right. **Simon Last** [1:15:23] And so just fix the bug. And then the outer loop thing that's really fruitful to think about is, like, how can you improve your, your velocity and robustness and making really good tools, making a good harness. You know, like, like verifying it works. **Alessio** [1:15:35] Mm. **Sarah Sachs** [1:15:36] The one place that we do invest more in model training now necessarily though is actually in retrieval because, um, we're at a point right now in our business and enterprise, our AI-enabled plans, where the search load and the search traffic, a majority of it's coming from agents, not humans. And so for every query that's hitting our Elasticsearch or our vector indices, they're not coming from humans, and the queries are structured differently and what's returned has a different re- requirement. Positional ranking matters less, but top-K retrieval mode matters more, right? **Alessio** [1:16:03] Isn't top-K a form of position? **Sarah Sachs** [1:16:04] Of course it is, but, um, when you're training on, like, click-through rate, it's really, you know- **Alessio** [1:16:09] Yeah, it matters much less **Sarah Sachs** [1:16:10] ... number one through number six- **Alessio** [1:16:11] Yeah, yeah **Sarah Sachs** [1:16:11] ... is very different- **Alessio** [1:16:12] Yeah **Sarah Sachs** [1:16:12] ... than it needs to be in the top 100. **Alessio** [1:16:13] Like the slope is just- **Sarah Sachs** [1:16:14] Yeah **Alessio** [1:16:15] ... higher. Yeah. **Sarah Sachs** [1:16:15] It's a different optimization- **Alessio** [1:16:16] Flatter, yeah **Sarah Sachs** [1:16:16] ... function for a retrieval, um, model. Similarly, uh, what snippet you include matters more or less, right? So we are rethinking a lot of that functionality, um, to work with how the agents like to write queries and how, um, they wanna r- uh, receive information. **Alessio** [1:16:32] Yeah. **Sarah Sachs** [1:16:32] So we are doing, like, another kind of reinvestment into rethinking not only search for, um, how do agents do searches versus how humans do searches, um, but we're also investing in, like, indexing different things now because- **Alessio** [1:16:44] Mm **Sarah Sachs** [1:16:44] ... uh, how are, how do you index, uh, the setup generator for Notion Agent? It kind of breaks our block model entirely, um, where all blocks are nested in each other. Same with meeting notes. Um, and so we do... we- I mean, so we're hiring ranking engineers and model training engineers, but it's primarily on ranking. **Alessio** [1:17:00] Yeah. Does ranking map to REXes for you? It does, right? Recommendation systems. **Sarah Sachs** [1:17:05] Yeah. Um, yes. **Alessio** [1:17:07] Right. Okay. **Sarah Sachs** [1:17:07] Um- **Alessio** [1:17:08] Same, same this. I'm trying to promote REXes more in general 'cause I- it's weirdly unpopular. **Sarah Sachs** [1:17:13] I don't know why. Um- **Alessio** [1:17:15] Yeah **Sarah Sachs** [1:17:15] ... but the other thing is that, like, I, I was just talking about this with a peer, like, how much is ranking important versus, like, uh, being able to do parallel exhaustive queries, right? Um, so we're also- **Alessio** [1:17:26] They're both important. **Sarah Sachs** [1:17:26] They're both important, but, like, they're both two tools to the same user outcome or the same agent outcome. **Alessio** [1:17:31] Uh-huh. **Sarah Sachs** [1:17:31] Right? And so, um, that, that's something that we're also rethinking a lot, even on... We just did an experiment on, um, Notion ranking at this point. Um, for Notion retrieval, vector embeddings are less and less. **Alessio** [1:17:43] Oh, did you see that? **Simon Last** [1:17:44] Yeah. **Alessio** [1:17:45] Notion just, uh- **Simon Last** [1:17:45] It did start to return the name of- **Alessio** [1:17:47] ... Notion went so long it became dark mode. **Sarah Sachs** [1:17:49] We're working the night shifts for you, right? **Simon Last** [1:17:51] Look pretty, dude. I'm not seeing any bugs. **Alessio** [1:17:53] You know, I worked on this, like, parallel search thing where you, you fan out to eight different queries, right? **Sarah Sachs** [1:17:57] Yes. **Alessio** [1:17:57] And so you actually need to use the model to work on query diversity so that you get- **Sarah Sachs** [1:18:02] Right **Alessio** [1:18:02] ... maximum search space. **Sarah Sachs** [1:18:03] And so, like, the people that are working on, um, ranking and retrieval are the same people working on what query generation is. It's all one, uh, journey. **Alessio** [1:18:11] Yeah. **Sarah Sachs** [1:18:11] We call it agentic find, and we're actually realizing, for instance, that it's less about a selection. Like, we don't spend a lot of time trying to optimize what vector embedding we use anymore. That was a period of time, but that's just not the right level of- **Alessio** [1:18:22] Remember that **Sarah Sachs** [1:18:22] ... optimization. **Alessio** [1:18:23] Yeah. **Sarah Sachs** [1:18:23] Right. **Alessio** [1:18:24] Yeah. Okay. Uh, we've gone long. I have to talk about me- Notion Meeting Minutes, and then we'll, we'll... we can call it there. Uh, you, you, you, you just have a lot of comments. Uh, yous, yous... Uh, I don't know where you wanna start. ### Meeting Notes **Alessio** [1:18:35] Um, is it the audio side? Is it the sort of- **Sarah Sachs** [1:18:38] Oh, meeting notes? **Alessio** [1:18:38] ... summarization? Yeah. **Simon Last** [1:18:40] Uh, sort of, like, what makes it work, or? **Alessio** [1:18:41] No, just, like, anything sort of interesting technically, right? Like, I think you had- **Simon Last** [1:18:45] Sure **Alessio** [1:18:45] ... you had some, uh, book points. I always call these, like, check marks along the way when the, when the guest says something that we'll... they want to return to later. I just, like, check mark it. **Sarah Sachs** [1:18:52] Yeah. **Alessio** [1:18:52] I'm like, okay, we'll get back to it. **Sarah Sachs** [1:18:55] Um, Meeting Notes was one of those things where at first we were nervous that we'd have to teach people a different way to work, and we were nervous that that was a lot of user friction. I think one of the reasons why... I mean, they're one of our biggest growth levers. I think they're one of the most, like- In terms of virality of adoption and retention, quite strong. Um, and so we've invested more and more as we did that. I think what's really powerful about it is, again, Notion is the system of record of where and how you work. The way that I use meeting notes is every one-on-one meeting I have is meeting notes. When I do my performance review for myself, my self-review, I say, "Primarily look at all my conversations with my manager and, like, write up what I did this year." Right? Because if I didn't talk about it in my one-on-one with my manager, it probably- **Alessio** [1:19:36] Mm-hmm **Sarah Sachs** [1:19:36] ... wasn't relevant for my performance review. So it also just adds a ton of signal on prioritization that's really helpful for a good system of record, that's really helpful for, like, our agent. **Alessio** [1:19:46] Mm. **Sarah Sachs** [1:19:46] It's also, like, caused a lot of scaling for Search and for the agent. Um, and you know, it's, it's just a, an explosion of content when you have transcripts like that. Um, how we do compaction, a lot of that was triggered by meeting notes passed into context, things like that. Um, so it's been a good impetus for us to think about longer form, um, content when you think of it as, like, a priority primitive. But it's been one of the most powerful signals for our agent, um, because it- **Alessio** [1:20:12] Unsurprising, right? **Sarah Sachs** [1:20:13] Right. And so- **Alessio** [1:20:13] Like, you're capturing a whole new thing. **Sarah Sachs** [1:20:15] So it's, like, our own data... Like, we want users, like, our... They're creating their own data flywheel, right? And it's- **Alessio** [1:20:20] Like, it serves me to prefer Notion, uh, to put all my stuff because it has d- my other stuff. **Sarah Sachs** [1:20:25] Totally. **Alessio** [1:20:26] So **Sarah Sachs** [1:20:26] I mean, the way that, the way that, like, our teams run right now is, you know, there's a custom agent that does a pre-read before standup. It looks through all of Slack and GitHub and just says, you know... It, it, it creates a summary and it creates a meeting note, and it says, "Everyone do this pre-read." Then we just press play. We have the meeting. We talk through the pre-read. We talk about what needs to happen next. And then we have a custom agent integrated with our calendar and triggers that then files tasks for tomorrow or today based on what we spoke about, and, um, sends off Slack messages that we decided in the meeting needed to be follow-ups. Like, our meetings are hands-off keyboard, and we're focused on, um, the root of the problem, not the bookkeeping around the problem. **Simon Last** [1:21:01] One thing that, uh, the meeting notes team added recently that was, uh, uh, been blowing my mind is they we... Uh, uh, they made it so it actually, when it makes a summary, will actually at mention the people that were referenced- **Alessio** [1:21:11] Oof **Simon Last** [1:21:11] ... in it. So I, I, I now get notifications whenever someone talks about me in a meeting. **Sarah Sachs** [1:21:14] Yeah, I feel like that one was- **Simon Last** [1:21:15] It's like- It's like, oh, you know, Simon is working on this. **Alessio** [1:21:20] Okay, I'm gonna- **Simon Last** [1:21:20] It's actually amazing how... 'Cause then I'm like, "Oh, okay, cool. I'm gonna go talk to them about that." **Sarah Sachs** [1:21:23] Right. **Alessio** [1:21:23] What, what if there are two Simons? **Simon Last** [1:21:24] Um. **Sarah Sachs** [1:21:25] No, wait. So it- ... it's powered by the agent, so it's doing agentic. So if you look at it thinking- **Alessio** [1:21:29] Yeah **Sarah Sachs** [1:21:29] ... I don't know if this has shipped yet. It will be. When you look at it thinking, when it's doing the summarization, it's saying, figuring out who Simon is. **Alessio** [1:21:36] Most probable Simon. **Sarah Sachs** [1:21:37] Um. **Alessio** [1:21:37] Yeah. **Sarah Sachs** [1:21:37] And we also have, like, a people to people similarity cache and stuff like that- **Alessio** [1:21:40] Yeah, yeah **Sarah Sachs** [1:21:41] ... based on the attendance data. **Simon Last** [1:21:41] Yeah. **Sarah Sachs** [1:21:41] Like, there's ways that we also know that. **Simon Last** [1:21:42] Yeah, we sort of, like, we also, like, generate a profile for each person and, like, and use that. Um, but- **Alessio** [1:21:47] Yeah **Simon Last** [1:21:47] ... it, I mean, of, of course it can get it wrong, but the goal is for it not to get it wrong. **Sarah Sachs** [1:21:50] Meeting notes is just like the agent primitive packaged on top of a transcription primitive. **Alessio** [1:21:54] Yeah, yeah. **Sarah Sachs** [1:21:54] And then a vertical team. It's probably one of the only teams at Notion that's completely a vertical team around quality and product, like UX design, 'cause it's still a tiger team, um, with a fantastic manager, Zach, that joined recently, um, from Embra. But, um- **Alessio** [1:22:09] Zach Tatar, yeah. **Sarah Sachs** [1:22:10] Yeah. **Alessio** [1:22:10] I, uh, chatted with him when he was talking about, uh, when he was working at Embra. **Sarah Sachs** [1:22:13] Yeah. So he's, he's managing that team now and thinking about it as data capture. That's what- **Alessio** [1:22:17] Yeah **Sarah Sachs** [1:22:18] ... meeting notes is, is data capture. **Alessio** [1:22:19] Get all the context. **Sarah Sachs** [1:22:19] Kind of reframing, um, where meeting notes are valuable as a data capture problem, and then working inside, um... Like, the summarization used to not be agentic. **Alessio** [1:22:27] Yeah. **Sarah Sachs** [1:22:27] Now it is, because it does all the things like figure out who the right Simon is, and one day you can have a custom agent directly integrated in it that knows, like, what task database the meeting is referring to. And as you're having the meeting, perhaps update the task, and things like that. Like, there's a, there's a lot of that experience of where we do our work in meetings that we want to invest in making more seamless. **Alessio** [1:22:46] Yeah. Uh, Opening Eyes doing hardware. Uh, would you ever ship one of these? **Simon Last** [1:22:50] Yeah, probably not. **Sarah Sachs** [1:22:51] But one of the things- **Alessio** [1:22:51] But, you know, this, this is meeting notes in person. **Simon Last** [1:22:54] Yeah, yeah. **Sarah Sachs** [1:22:54] But- **Simon Last** [1:22:55] I'd be, I'd be excited about... I mean, I'm excited about that, the product category in general, for sure. Yeah. **Sarah Sachs** [1:23:00] I think it's like, it's a, it's a mechanism, and it, it... One of those needs to work really well with Notion. We would partner with whoever's- **Alessio** [1:23:07] Yeah **Sarah Sachs** [1:23:07] ... building one of those. I think- **Alessio** [1:23:09] Uh, this would be... They, they were bought by Amazon. I don't know. I, I, I can't remember who. **Simon Last** [1:23:12] Yeah. **Sarah Sachs** [1:23:12] And there's, like, there's some wild companies doing, like, really cool things that come to our partnerships team that I like to sit in on the demos of, of wearables. I always like to sit in on the demos 'cause I think they're- **Alessio** [1:23:21] Oh, okay **Sarah Sachs** [1:23:21] ... pretty cool. **Alessio** [1:23:22] Yeah. **Sarah Sachs** [1:23:22] And all of them want to make sure, not just Notion, but, like, you can imagine on the ones that talk to you- **Alessio** [1:23:26] Yeah **Sarah Sachs** [1:23:27] ... um, being able to do search and build context. So, like, if you're entering, like, a conference, um, being able to, like, do, like, look at your CRM and do things like that. Um, and you can utilize the Notion agent to do that. So we are in, like, the very beginnings of those partnerships. I think what's unique about that particular technology is it goes against what I talked about with custom agents right now, which is the more simple it is, the harder it is to have, like, advanced controls over its capabilities, right? And so that would be a great investment for data capture, but not necessarily, like, our agent- **Simon Last** [1:23:54] It's- **Sarah Sachs** [1:23:54] ... as workflows. **Simon Last** [1:23:55] It's a little bit of a different slice of the problem. I would say, like, that's gonna be deeply personal. Like, like, your company's not gonna force you to wear a wrist, wristband, right? I, I think- **Alessio** [1:24:03] I don't know. **Sarah Sachs** [1:24:03] It's good to hear that from me, from you. **Simon Last** [1:24:06] Yeah. Maybe, yeah, the, the CEO's gonna force everyone to wear a wristband. **Alessio** [1:24:09] Look- **Simon Last** [1:24:09] I mean, the slice of the problem that, that we care about is, like, you know, can the company have all the context of what everyone said at every single meeting, and then use that to- **Alessio** [1:24:18] Yeah **Simon Last** [1:24:18] ... to, to derive value for themselves. **Sarah Sachs** [1:24:20] That kind of reminds me, I remember once you very strongly reminded me our job is to not make the best harness for agentic work. Our job is to be the best place where people collaborate. It's like our job isn't to build the best wearable to capture meeting notes. Our job is to build the best place where meeting notes live, right? **Alessio** [1:24:39] Yeah. So basically you're saying everyone else can just pipe to you, and it, it's fine, right? Yeah. ### Outro **Sarah Sachs** [1:24:44] Yeah. **Alessio** [1:24:44] Yeah. That's, that's a reasonable thing. All I will say is that pe- there's people walking around with Notion tattoos on them. They, they'll wear Notion anything, so just, I don't know, do a limited run. **Simon Last** [1:24:53] Yeah, yeah. No, I mean- **Sarah Sachs** [1:24:54] Well- We have such understated swag that the idea... Like, our swag has so few Notion la- logos on it, the idea that people have Notion tattoos is pretty antithesis to our design principles- ... so that's pretty funny. **Simon Last** [1:25:06] Yeah. Hmm. **Sarah Sachs** [1:25:08] Do you have one? **Simon Last** [1:25:08] No. No, I do not. I do not have a Notion tattoo. I've, I've seen them. Yeah. **Alessio** [1:25:13] Cool. Uh, well, thank you so much. This is such a great d- deep dive. Actually, the chemistry between you two is amazing. Like, I, I can't believe, like- **Sarah Sachs** [1:25:19] We work together a lot. **Alessio** [1:25:20] Yeah. **Sarah Sachs** [1:25:21] Different jobs, work closely. **Alessio** [1:25:23] Yeah. **Simon Last** [1:25:23] Mm-hmm. That's it. **Alessio** [1:25:24] Yeah. Thank you. **Simon Last** [1:25:25] Thank you. **Sarah Sachs** [1:25:26] Thank you. **Alessio** [1:25:26] Thanks for joining us. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.