Jira

Notion’s Sarah Sachs & Simon Last on Custom Agents, Evals, and the Future of Work

Apr 15, 2026 · 1:25:37

Sarah Sachs and Simon Last of Notion explain how Custom Agents were rebuilt four to five times since 2022, with early attempts failing due to lack of tool-calling standards and short context windows. They shifted from few-shot prompting to tool definitions and progressive disclosure, now supporting over 100 tools. Their eval system includes frontier evals that pass only 30% to gauge model progress, and they employ Model Behavior Engineers to write evals and triage failures. The episode covers their views on MCP versus CLI, meeting notes as data capture, and the software factory concept of agents collaborating to develop codebases. They also discuss Notion's culture of low-ego, high-velocity rebuilding and the philosophy that every surface must work for both humans and agents.

The AI Coding Factory

May 29, 2025 · 59:23

Factory.ai co-founders Eno Reyes and Matan Grinberg join Latent Space to explain their vision for autonomous software engineering droids that handle everything from code generation to incident response for enterprises. They met at a Langchain Hackathon in 2023, and within eight days both quit their jobs to start Factory. The platform is browser-based, not IDE-based, because they argue the optimal UI for a world where humans write less code will not evolve from the IDE. They focus on enterprise codebases, citing a large migration that took from four months to three and a half days. Key design decisions include a delegation model where agents ask clarifying questions, proactive context gathering via synthetic insights, and usage-based pricing tied to token efficiency. They also discuss why SWE-Bench is dead for their use case, the challenge of model post-training biases toward CLI tools, and their need for 'junior Enos' in go-to-market roles.