Evals

DevDay 2025: Apps SDK, Agent Kit, MCP, Codex and why Prompting is More Important than Ever

Oct 7, 2025 · 44:52

Sherwin Wu and Christina Huang from OpenAI's platform team discuss the new AgentKit suite and Apps SDK, arguing that visual agent builders and prompt optimization remain critical for production AI. They detail AgentKit's components—Agent SDK, visual Builder, Evals, and Tool Registry—and how Apps SDK inverts the website-chatbot paradigm by embedding apps inside ChatGPT. OpenAI adopted Anthropic's MCP protocol for tool connectivity, and the team dogfoods these tools for their own customer support at openai.com. They explain that automated prompt optimization and evaluation pipelines are essential for scaling agents, and share how Codex is used internally for code generation and PR reviews. The episode also covers the Service Health Dashboard for real-time API SLO tracking, reflecting OpenAI's reliability investments.

GPT 4.1: The New OpenAI Workhorse

Apr 15, 2025 · 45:04

OpenAI's Michelle Pokrass and Josh McGrath join hosts Alessio, Swyx, and Sean to launch GPT 4.1, a new model family positioned as the go-to workhorse for developers with major improvements in coding, instruction following, and long context. The three models—4.1, 4.1 Mini, and 4.1 Nano—introduce a 1M-token context window, 75% prompt caching discount, and cheaper pricing than 4o. Coding gains are highlighted by 55% on SWE-bench (vs o1's 41%), while instruction following benefits from real-world API data and new evals like GraphWalk for multi-hop reasoning. The team explains that most improvements come from post-training techniques, with vision capabilities lifted by a new pre-trained base. Fine-tuning is available day one, and developers are encouraged to opt in to data sharing for future model iterations.

Building AGI in Real Time (OpenAI Dev Day 2024)

Oct 4, 2024 · 2:09:35

OpenAI's DevDay 2024 unveils the Realtime API with WebSocket-based voice and function calling, Vision Finetuning, Prompt Caching, and Model Distillation. Product leads Olivier Godement, Romain Huet, and Michelle Pokrass detail design choices like zero-code caching and WebSockets for human-level latency. Alistair Pullen of Cosine explains how fine-tuning GPT-4o on custom reasoning traces beat o1 on SWE-bench Verified, while Sam Altman and Kevin Weil discuss AGI levels (o1 as level 2 reasoner), the coming agent era (level 3), safety through iterative deployment, and the importance of building for the frontier. The episode also covers moderation policy relaxation, plans for o1 system prompts by year-end, and the competitive admiration for NotebookLM.