# DevDay 2025: Apps SDK, Agent Kit, MCP, Codex and why Prompting is More Important than Ever Latent Space · 2025-10-07 Sherwin Wu and Christina Huang from OpenAI's platform team discuss the new AgentKit suite and Apps SDK, arguing that visual agent builders and prompt optimization remain critical for production AI. They detail AgentKit's components—Agent SDK, visual Builder, Evals, and Tool Registry—and how Apps SDK inverts the website-chatbot paradigm by embedding apps inside ChatGPT. OpenAI adopted Anthropic's MCP protocol for tool connectivity, and the team dogfoods these tools for their own customer support at openai.com. They explain that automated prompt optimization and evaluation pipelines are essential for scaling agents, and share how Codex is used internally for code generation and PR reviews. The episode also covers the Service Health Dashboard for real-time API SLO tracking, reflecting OpenAI's reliability investments. ## Questions this episode answers ### Why did OpenAI adopt Anthropic's MCP protocol for its agent tools? Sherwin Wu explained that OpenAI started integrating MCP in March 2025 after seeing developer uptake. They worked closely with Anthropic, treating MCP as an open protocol; OpenAI's Nick Cooper sits on the MCP steering committee. Wu said the protocol is general and easy to integrate, and Anthropic's open approach made it straightforward to embrace. [5:39](https://addtry.com/ff90c381-11a7-489d-a43e-a7aeb2e74371?t=339000) ### How quickly can you build a customer support agent using OpenAI's new Agent Builder? Christina Cai demonstrated building a live customer support agent in 8 minutes during the DevDay keynote, with 29 seconds to spare. She used the visual Agent Builder's drag-and-drop canvas, leveraging templates for common patterns like customer support, and deployed it via ChatKit. The builder supports complex workflows, including human-in-the-loop approvals, making agent creation accessible to non-developers. [9:05](https://addtry.com/ff90c381-11a7-489d-a43e-a7aeb2e74371?t=545000) ### Why does OpenAI believe prompting is becoming more important rather than less? Sherwin Wu noted that contrary to predictions two years ago, prompting has only grown more critical. He pointed to automated prompt optimization as an active research area, and that serving custom fine-tuned models at scale is extremely difficult, making zero-gradient prompt optimization a more practical way to improve model performance. The company is investing in tools to help developers optimize prompts through evals. [20:41](https://addtry.com/ff90c381-11a7-489d-a43e-a7aeb2e74371?t=1241000) ## Key moments - **[0:00] Intro** - [1:18] OpenAI DevDay 2025 reports 4 million developers, up from 3 million last year - **[3:00] Apps SDK** - [4:26] ChatGPT has 800 million weekly active users and is among the world's top websites - [5:39] OpenAI adopted Anthropic's MCP protocol in March 2025, starting with the Agents SDK - **[8:55] AgentKit** - [11:36] Agent Builder introduces a binary approval node, with plans for complex human-in-the-loop interactions spanning weeks - [14:34] AgentKit launches with templates for common agent patterns like customer support and document discovery - **[15:07] Interop** - [15:24] OpenAI considers standardizing agent workflows and the responses API for cross-provider portability - [16:58] OpenAI's Evals now supports third-party models through an Open Router integration - **[17:25] Agent Evals** - [17:42] Agent evals are only 10% complete, says Sherwin Wu; OpenAI plans to break down agentic traces for detailed evaluation - **[19:10] Optimization & Tuning** - [19:23] Automated prompt optimization launched with GPT-5, with plans to deeply integrate with the evals platform - [20:51] "Two years ago people thought prompting would be dead, and now it's only gotten more important," says Sherwin Wu - [22:21] Sherwin Wu DM'd John Schulman "Wow, we finally launched it" when the Tinker API debuted - **[23:50] Builder Feedback** - [25:43] "Anything that can store state will eventually be used as a database," Swyx warns about AgentKit's state store - **[26:32] MCP & Auth** - [27:35] OpenAI's first-party MCP sync connectors store state to enable re-ranking, creating trade-offs in quality and coverage - [31:08] Q: Does AgentKit support bring-your-own-key to offload inference costs? A: Not yet, but it's an interesting area. - **[32:24] Platform & Widgets** - [33:50] OpenAI's customer support at help.openai.com is already powered by AgentKit - [36:43] Christina Cai likens ChatKit to Stripe Checkout, providing common chat UX that businesses don't need to rebuild - **[39:13] Codex Tips** - [39:23] New grads at OpenAI use Codex in "full YOLO mode" and achieve 30-40% one-shot feature implementations - [41:02] Christina Cai's engineering team relies on Codex PR reviews, which are "very, very high quality" - **[42:16] Service Health** - [42:27] OpenAI launches a Service Health Dashboard giving developers real-time personal SLOs for their API integrations - [44:33] OpenAI's API throughput has grown past 6 billion tokens per minute, a previously stated figure now outdated ## Speakers - **Alessio** (host) - **Swyx** (host) - **Christina Huang** (guest) - **Sherwin Wu** (guest) ## Topics Agent Platforms ## Mentioned Anthropic (company), Apple (company), Kakao (company), Open Router (company), OpenAI (company), Agent Builder (product), Agent SDK (product), AgentKit (product), Apps SDK (product), ChatGPT (product), ChatKit (product), Codex (product), Evals (product), GPT-5 (product), MCP (product), Service Health Dashboard (product), Siri (product), Sora (product), fine-tuning API (product), responses API (product) ## Transcript ### Intro **Alessio** [0:05] Hey, everyone. Welcome to the Late in Space podcast. This is Alessio, founder of Kernel Labs, and I'm joined by Swyx, editor of Late in Space. **Swyx** [0:12] Hello, hello, and we are here in the OpenAI DevDay studio with Sherwin and Christina from the OpenAI platform team. Welcome. **Sherwin Wu** [0:19] Thank you for having us. **Swyx** [0:20] Yeah. It's always- **Christina Huang** [0:21] Yeah. Great to be here. **Swyx** [0:22] Yeah. Um, it's so-- It's such a nice thing. Uh, uh, we've been-- We've covered like three of these DevDays now, um, and this is like the first time it's been, like, so well-organized that we have our own little studio, podcast studio, in, in the DevDay venue. Um, and it, it's really nice that I actually get a chance to sit down with you guys. So thanks for taking the time. **Sherwin Wu** [0:39] Yeah. I feel like we, um-- DevDay's always a process, and, like, we've only had three of them and we try to improve it every time. And- **Swyx** [0:44] Yeah **Sherwin Wu** [0:45] ... I actually-- I, I know, uh, for a fact that I think we have this podcast studio this time because the, the podcast interviews and the interviews with folks like yourselves last time went really well, and so we wanna lean into a little bit more and glad that we were able to have this studio for y'all. **Christina Huang** [0:56] Yeah. **Alessio** [0:56] We, we were kneeling on the ground interviewing, like, Michelle last year. **Sherwin Wu** [1:00] I don't know. **Swyx** [1:00] Yeah. **Sherwin Wu** [1:01] I didn't know that. I, I, I just saw it post-production. I thought it was- **Swyx** [1:04] We, we had to have people, like, cordon off the area so they wouldn't walk in front of the cameras. **Alessio** [1:08] Yeah. People would just come up, "Hey, good to..." I'm like, "We're, like, recording." It's nice. **Christina Huang** [1:12] I guess if you guys have been to three, like, what, what stood out from today or what, um, what's your favorite part? **Swyx** [1:18] I feel like the vibes are just a lot more confident. Like, you are obviously doing very well. Uh, you have the numbers to show it. Uh, you know, I just-- E-every year in DevDay, you report the number of developers. This year it's four million. I think last year it was, like, three. Um, so like, and, and I have more, more questions about that, that kind of stuff. But also, like, just, like, very interesting, very high-confidence, uh, launches and, and then also, like, I think the, the, the community is clearly much more developed. Like, I think there's, there's just a lot more, um, things to dive into a-across the API surface area of, of, uh, OpenAI than I think last year, uh, in, in my mind. I don't know about you. **Alessio** [1:57] Yeah. And we were at the OG DevDay, which was the DALL-E hack night at OpenAI in twenty twenty-two, and- **Sherwin Wu** [2:03] Oh, yeah **Alessio** [2:03] ... uh, I think Sam spoke to, like, 30 people. So I, I think it's just crazy to see the- **Sherwin Wu** [2:07] Yeah. Honestly, I think it's like-- it's kind of similar to this podcast studio, which is I think we've had a number of DevDays now. We're-- we honestly were, like, slowly figuring things out as a company over time as well, and, uh, both from a product perspective and also from a, like, how we want to present ourselves with DevDay. And at this third only-- at this point, we've had a lot of feedback from people. Um, I actually think a lot of the attendees will get, like, a, a, an email with, like, a, a chance for feedback as well, and we actually, like, do read those, and we, we, we act on those. And, um, like, one of the things that we did this year that I really liked were, uh, all of those, like, um, there was, like, some art installations- **Swyx** [2:37] Mm **Sherwin Wu** [2:37] ... and, like, the, the little arcade games that we did, which, um, was, you know, came up with an in-- via, like, engaging with the feedback from the- **Christina Huang** [2:44] Yeah, the arcade games were so fun. **Sherwin Wu** [2:45] Yeah. **Christina Huang** [2:45] I loved, like, the theme of all the ASCII art throughout. Um, this was my first SF DevDay, um, but I've been to the Singapore one. That was actually my first week. **Swyx** [2:52] Oh, yeah. That's the one I spoke at. **Christina Huang** [2:53] Yeah. I saw, I saw you there. That was my first week of OpenAI. **Swyx** [2:56] Oh. **Christina Huang** [2:56] So really in the deep end. **Sherwin Wu** [2:57] Put her on a plane to Singapore. **Christina Huang** [2:59] Yeah. **Swyx** [3:00] Yeah, that's awesome. Um, well, so you know, that's, uh, congrat- congrats on everything, and, like, kudos, kudos to your organizing team. Uh, we should talk about some developer API stuff. ### Apps SDK **Sherwin Wu** [3:08] Yeah. **Swyx** [3:08] Uh, so the-- We're, we're gonna cover a few of the things. You're not exactly working on Apps SDK, but, um, I guess what should people just generally, generically take away? What should developers take away from the Apps SDK launch? Like, how do you internally view it? **Sherwin Wu** [3:23] So the way that I think about it is I actually view OpenAI since the very beginning as a company that has really valued kinda like opening up our technology and, like, bringing it out to the rest of the world. One thing we talk about a lot internally is, um, you know, our mission at OpenAI is to, one, build AGI, which we're trying to do, and then, but two, uh, uh, you know, potentially, you know, just as important is to, uh, bring the benefits of that to the, to the entire world. And one thing that we realized very early on is that we as a company, it's very difficult for us to just bring it to every, truly every corner of the world, and we really need to rely on developers, other third parties to be able to do this, which is, you know, Greg talked about the, the, the start of the API and, like, kinda how, you know, that was formulated. Um, but that was part of, you know, uh, that mentality, which is we needed to-- we need to rely on developers, and we, we need to open up our technology to, to the rest of the world so that they can partake for us to really fulfill our mission. So the API obviously is a very natural, you know, way of doing that where, um, uh, we just literally expose API endpoints or expose tools for people to, to build things. Um, but now that we have, you know, ChatGPT with its, I don't know, like, eight hundred million weekly active users. I forgot the stat that we- **Swyx** [4:27] Yeah **Sherwin Wu** [4:27] ... shared. I think it's, like, now the fifth or, like, sixth largest website in the world. This is also- **Swyx** [4:32] And, and the number one and number two most downloaded on the Apple App Store. **Sherwin Wu** [4:35] Oh, yeah, with Sora. Yeah. Yeah, but that one, like, it moves around all the time, so it's- **Swyx** [4:38] Yeah **Sherwin Wu** [4:38] ... kind of hard to celebrate and, you know- **Swyx** [4:40] You can just screenshot it when it's good. **Sherwin Wu** [4:41] Yeah, yeah. We definitely screenshot it and shared it- ... internally when it was, uh, when it was good. Um, but, uh, my, uh, kinda going back to my main point is, like, we, we've always kind of engaged with developers as a way for us to bring the benefits of, of AGI to the rest of the world. And so I view this as actually a natural extension of this. Candidly, we've actually been trying to do this, uh, you know, a couple of times with, uh, uh, last DevDay with GPT, two DevDays ago with, um, or sorry, two Devs ago with, with- **Swyx** [5:06] GPTs and plugins **Sherwin Wu** [5:07] ... GPTs and plugins, uh, which was, I think, not tied to a DevDay. Um, so I view this as like, again, we, we love to deploy things so iteratively, and I view it as, like, just a continuation of that process and also engaging deeply with developers and helping them benefit from some of the stuff that we have, uh, which, which in this case is ChatGPT distribution. **Alessio** [5:23] Um, and when-- So, uh, Apps SDK is built on the MCP protocol. When did OpenAI become MCP-pilled? I'm sure internally you must have had, you know, design discussions before about doing your own protocol. Uh, when did you buy into it, and, um, how long ago was that? **Christina Huang** [5:39] I think it was in, in March, I wanna say. It's hard for me to remember kind of like the exact- **Swyx** [5:43] March was the takeoff of MCP. **Christina Huang** [5:44] Okay. Yeah, yeah. So we, we built the Agents SDK, and we launched that alongside kind of the Responses API in early March. Um, and I think as MCP was, was growing, that felt like a really... And, you know, we're building kind of a new agentic API that can call tools and just be much more, um, powerful. Um, MCP was kind of like the natural protocol that developers were already using to bring all the tools into their system. And, um, I think like in March is when we added an MCP to agents SDK first, and then soon after with kind of our other- **Sherwin Wu** [6:12] Yeah, I think there was like a tweet- **Christina Huang** [6:13] ... products as well. Yeah **Sherwin Wu** [6:13] ... or something we did where it was like, OpenAI, you know, is- **Christina Huang** [6:16] Yeah, there was definitely a moment **Sherwin Wu** [6:17] ... deploying MCP. **Christina Huang** [6:17] I think there was a specific moment- **Sherwin Wu** [6:18] Yeah **Christina Huang** [6:18] ... in a specific tweet. Um- **Sherwin Wu** [6:19] But what I will say, though, is like I, a- and this is honestly like credit to, um, the team at Anthropic that, that kind of created MCP, is I really do think they treat it as an open protocol. Like, uh- **Christina Huang** [6:28] Yeah **Sherwin Wu** [6:28] ... uh, we work very closely with, I think, like David and, and the folks on the like, you know, consortium. Uh, and they are not, you know, really viewing it as this like, uh, thing that is specific to Anthropic. They really view it as this open protocol. There's like, uh, it is an open protocol. The way in which you make changes feels very open. We actually have a member of our team, Nick Cooper, who is sitting on kinda like that, that steering committee for MCP as well. And so I think, um, they are really treating it as something that is easy for us and other companies, you know, everyone else to, to embrace, which I think they should because they, they do want it to be something that is, uh, very embraced by all. And so because of that, I think it makes it a little bit easier- **Christina Huang** [7:03] Yeah **Sherwin Wu** [7:03] ... uh, for us to, uh, uh, to embrace it. A- and honestly, it's a great, it's a great protocol. Like, it's very general. **Swyx** [7:07] Yeah, it's already solved. Like, why would you make a new one? **Sherwin Wu** [7:09] Yeah, yeah, it's very general. Um, there's obviously still more to do with it, but it was very easy for us to, um, you know, integrate because of how, how, how, um, how streamlined and how simple it was. **Swyx** [7:18] Yeah. Now my final comment on, on Apps SDK stuff, and then we'll move to AgentKit, is, uh, you know, like I always see like in abstractly when you sort of wireframe a website or an AI app, uh, it used to be that the initial AI integration on the website would be you have the normal website, and then you have a little chatbot app. Um, and now it's kind of like inverted, where like there's ChatGPT at the, at the top layer, and then there's like kind of the website embedded inside of it. Um, and it's, it's kind of like that inversion that I honestly have been looking for for a little bit. Um, and I, I think it's really well done. Like, uh, all, actually, all like the integrations and like the custom UI components that come up, you had like Canva on the keynote- **Sherwin Wu** [7:57] Yeah **Swyx** [7:58] ... there, and it looks like Canva. **Christina Huang** [7:59] Mm-hmm. **Swyx** [8:00] But like you can chat with it in, w- in all your, the context of your ChatGPT. That is an experience I've never seen. **Sherwin Wu** [8:06] Yeah, and I think like that's, uh, like kinda back to the iterative like learning that we've had. That, I think, was because we've learned a lot from plugins. So like when we launched plugins, I remember one of the feedback that we got, I don't know if, if, you know, people here really remember plugins. It was like March- **Swyx** [8:19] Oh, yeah **Sherwin Wu** [8:19] ... 2023. **Swyx** [8:20] Yeah. **Sherwin Wu** [8:20] Um, but like one of the points of feedback was like, oh, you can integrate. We tell, we told like, you know, all these companies that you can integrate these plugins into ChatGPT, but they really didn't have that much control over how exactly it was used. It was really just like a tool that the model could call, and you were just like really bound by ChatGPT. And so I think like you can kinda see the evolution of our product with this. And like this time, we realized how important it was for, uh, companies, for third-party developers to really own and like steer the experience to make it feel like themselves, to help them, you know, like, uh, really preserve their own brand. And so, um, and you know, I actually don't think we would've gotten that learning, uh, had we not, you know, had all these other steps, uh, beforehand. ### AgentKit **Swyx** [8:55] Awesome. Um, Christina, you were the star today on stage with the- ... AgentKit demo. You had eight minutes to build an agent. You had a minute to spare, and then you had some issues with the- **Sherwin Wu** [9:05] 29, 29 seconds **Swyx** [9:05] ... with the download time. **Christina Huang** [9:06] Yeah, I wasn't sure. **Swyx** [9:06] So- **Christina Huang** [9:06] Honestly, I like, I was like, let's do a little bit less testing, and maybe we - I don't know how much time I killed on the- **Sherwin Wu** [9:12] I was extremely stressed- **Christina Huang** [9:13] ... on the widget **Sherwin Wu** [9:13] ... when the download came. **Christina Huang** [9:14] Yeah. **Swyx** [9:14] Yeah. **Christina Huang** [9:14] I was stressed out. **Sherwin Wu** [9:15] I was like, if a UI bug is what like takes the demo down, I'm gonna be so sad. **Swyx** [9:18] Yeah, yeah. **Christina Huang** [9:18] I think it was a full screen, yeah, like focus thing. **Swyx** [9:21] Yeah, yeah, yeah. **Sherwin Wu** [9:21] Yeah. **Christina Huang** [9:21] But- **Sherwin Wu** [9:21] I heard the window wasn't in focus- **Christina Huang** [9:22] Yeah **Sherwin Wu** [9:22] ... or something. Yeah. **Swyx** [9:23] Um, maybe you wanna introduce AgentKit to the audience. **Christina Huang** [9:27] Yeah. So we launched AgentKit today, um, full set of solutions to build, deploy, and optimize agents. Um, I think a lot of this comes from working with API customers and realizing how hard it actually is to take, to build agents and then actually take them into production, hard to get kind of that confidence and the iterative loop and writing prompts, optimizing them, writing evals, um, all takes a lot of expertise. And so kind of taking those learnings and packaging them into a set of tools, that makes it a lot easier and kind of intuitive to know what you need to do. Um, and so there's a few different building blocks that can be used independently, but they're kinda stronger together because you then get the whole end-to-end system, and, um, releasing that today for people to try out and see what they build. **Swyx** [10:14] Yeah. So I, I had... I find it hard to hold all the building blocks in my head. But actually chronologically, it's really interesting that you guys started out with the Agent SDK first. **Christina Huang** [10:23] Mm-hmm. **Swyx** [10:23] And then, uh, then you have Agent Builder. Uh, you have a Connector Registry. You have Trackit. Uh, and then you have the Eval stuff. Did-- Am I missing any major components? Th- those are the main moving parts, right? **Sherwin Wu** [10:34] Yeah, I think that's it. And then, I mean, we also still have like the RFT, like fine-tuning, uh, API, but, uh, we technically group it outside of the AgentKit, um- **Swyx** [10:42] Yeah **Sherwin Wu** [10:43] ... umbrella. Yeah. **Swyx** [10:43] Got it. Got it. Got it. Um, yeah, so like it, it, it's, it's weird how it develops, and it's, it, it's, it's now become the full agent platform, right? **Christina Huang** [10:50] Mm-hmm. **Swyx** [10:50] Uh, and, uh, I think one thing that I wasn't clear about when I was looking at the demo was, uh, it's very funny 'cause like what you did on stage was, uh, build like a live chat app for DevDays', uh, website. **Christina Huang** [11:02] Yeah. **Swyx** [11:02] And- **Christina Huang** [11:02] Did you get a chance to try it out? **Swyx** [11:04] Yeah, I tried, tried it out. It was, it was awesome. Right. And, uh, actually, I kind of wanted to ask, like- **Christina Huang** [11:07] Ask it all **Swyx** [11:07] ... how to deploy- **Christina Huang** [11:07] Like, "Where's merch?" Yeah. **Swyx** [11:08] Yeah, exactly. I was like, where'd you, where'd you click the merch? But anyway, um, I, I, uh, and this is very close to home because I've done it for my conferences, and like it's, it's a, it's a very similar process. But like, um, I, I think what was not obvious is like how much is going to be done inside of Agent Builder. **Christina Huang** [11:23] Mm-hmm. **Swyx** [11:23] I see there are some actually very interesting nodes that you didn't get to talk about on stage, like user approval. **Christina Huang** [11:28] Mm-hmm. **Swyx** [11:28] That's like a whole thing. Uh, and, uh, you know, like transform and set state. Like, there's, there's like a, kind of like a touring complete machine in here. **Christina Huang** [11:36] Yeah. Yeah. So I mean, I think again, like this is the first time that we're showing Agent Builder, and so it's definitely the beginning of what we're building. And, um- Human, human approval is, like, one of those use cases that we wanna go pretty deep on, I think. **Swyx** [11:49] Yeah. **Christina Huang** [11:49] Um, the node today that I showed is pretty simple, like binary approval. **Swyx** [11:52] Approval checks. **Christina Huang** [11:53] So it's, it, it's similar to kind of what you'd see for MCP tools of, you know, approving that an action can take place. Um, but I think what we've seen with much more complex workflows from our users is that it's actually quite advanced, like human-in-the-loop interactions. Sometimes these could be over, like, the course of weeks, right? It's not just kind of simple approval for tool. There's actual, like, decision-making involved in it. Um, and I think as we, like, work with those customers, we definitely wanna continue to go deeper onto those use cases, too. **Swyx** [12:22] Yeah. **Alessio** [12:23] What's the entry point? So are developers also supposed to come here and then do the to code, um, export? Like just- **Christina Huang** [12:29] Yeah **Alessio** [12:30] ... the segment, like the use cases? **Christina Huang** [12:31] Yeah, I mean, I... So I think, like, the two reasons that you would come to Agent Builder are, one, um, kind of more as a, as a playground, right? To kind of model and iterate on, um, your systems and write your prompts and optimize them and test them out, and then you can export it and run it in your own systems using a Agents SDK, using kind of, you know, other models as well. Um, the second would be kind of to get all of the benefits of us deploying that for you, too. So you can kind of use maybe, like, natural language to describe what type of agent you wanna build, um, model it out, bring in subject matter experts so that you really have this canvas for iterating on it and getting feedback, you know, building data sets and kind of getting feedback from those subject, subject matter experts as well, and then being able to deploy it all without needing to, to handle that on your, on your own. And, um, that's a lot of the philosophy around how we're building it with ChatKit as well, right? You can kind of take pieces of it. You can have a more advanced integration where it's much more customized, um, but you also get, um, a really natural path of going live, um, without... Like, with really kind of easy defaults as well. Um, yeah. **Alessio** [13:33] Do you see it as a two-way thing? So I build here, I go to code, then maybe I make changes in code, and then I bring those changes back to the Agent Builder? **Christina Huang** [13:41] Yeah. **Alessio** [13:41] Like, what's the- **Christina Huang** [13:41] I think like in, eventually, like, that's definitely what we wanna do. So maybe you could start off in code, you could bring it in. Um, we'll also probably have, like, ability to, you know, run code in, in the Agent Builder as well. And so, um, I think just a lot of flexibility around that. **Sherwin Wu** [13:55] The, the one thing I'd say too is, uh, a lot of the demos that we showed today, I think were, like, you know, erred on the side of simplicity just so that the audience could kind of see it. But, like, if you talk to a lot of these customers, like, they're building, like, pretty complex... Like, you gotta, like, zoom out on that canvas quite a bit to kind of, like, see the, the full flow. And that, and then for us, we, you know, we were kind of, like, working with a lot of customers who are doing this. Um, and then, y- you know, if you turn that into, like, an actual Agents SDK, like, file, it's, like, pretty, it's pretty long. And so we saw a lot of, like, benefit from having the visual set up here, especially as the, as the setup grows, grows longer and longer. Um, it would've been a little difficult to kind of showcase this. **Christina Huang** [14:27] Yeah. **Sherwin Wu** [14:27] But, uh, even on, like, some of the- **Christina Huang** [14:28] Or do it in eight minutes. **Alessio** [14:28] Right. **Sherwin Wu** [14:29] Yeah, you can do it in eight minutes. But, like, even with some of the presets that we have on the site- **Christina Huang** [14:32] Yeah, exactly **Sherwin Wu** [14:32] ... with, like, the support thing. **Christina Huang** [14:33] So, so one of the thing- **Sherwin Wu** [14:33] Yeah. **Christina Huang** [14:34] Yeah, one of the things that, um, we launched today as well alongside just, like, the canvas is a set of templates that we've actually gathered from our engineers who are working in the field with customers directly of, like, the kind of common patterns that they have in the... Our own basically, like, playbooks when we're working with customers on customer support, document discovery- **Sherwin Wu** [14:53] Yeah, and so- **Christina Huang** [14:54] Um, and so kind of publishing those as well. **Sherwin Wu** [14:55] Data enrichment- **Christina Huang** [14:56] Yeah **Sherwin Wu** [14:56] ... planning helper, customer service, structured data Q&A, document comparison, that's nice, internal knowledge assistant. **Christina Huang** [15:01] Yeah. **Alessio** [15:02] Yeah. **Sherwin Wu** [15:02] Yeah. **Christina Huang** [15:02] And I think, like, w- we just plan to add more to those as we, um, can kind of build those out. **Swyx** [15:07] I always wonder if there should be... So you're not the only agent builders, uh- ### Interop **Christina Huang** [15:11] Mm. **Swyx** [15:11] But obviously by default of being an OpenAI, you are a very significant one. Uh, any interest in, like, a protocol, like, interop between different, uh, open source implementations of this kind of pattern of Agent Builder? **Sherwin Wu** [15:24] I think we, we've thought about it, uh, especially around, I'd say, Agents SDK. Um, I would actually say maybe even, like, zooming out a bit more, uh, from, from just this is like, yeah, we, we, like, we're also sitting here and kind of, like, observing, like, things being made over and over again. Um, even, like, besides, like, agent workflows, uh, we're kind of watching what the industry's trying to do with responses, like what we've done with responses API, like stateful APIs. And so, uh, you know, obviously we were the first one to launch responses API, but, like, couple other, other people have kind of adopted it. I think, I think Groq has it in their, in their API. I think I saw LM 系 just did something recently as well, but not, you know, not everyone. Uh, and so, um, unfortunately I don't have a great answer today of, like, yes or no, but, um, we are kind of, like, assessing everything and trying to see, like, hey, you know, there is a, there has been a lot of value with, um, MCP, with, um, hopefully with our, uh, um, uh, uh, with our, uh, commerce protocol as well. Um, uh, ACP, yeah, it's the... I definitely did not forget the name. Uh, and, uh, uh, and so, like, even thinking about, like, what we wanna do with agents, uh, with the agent workflow, the portability story around that, as well as the portability, I'd say even of, like, responses API, it'd be great if- **Christina Huang** [16:29] Yeah **Sherwin Wu** [16:30] ... you know, that could be a standard or something. And, and developers don't need to, you know, like, build three different, uh, uh, stateful API integrations, uh, if they wanna, uh, use different models. **Christina Huang** [16:39] Yeah, and I think that's one of the... So it's not exactly a protocol, but one of the things that we launched today with Evals too, is ability to use, like, third-party models as well and kind of bring that into one place. And so I think definitely kind of see where the ecosystem is at, which is, you know, using multi-models and kind of having- **Swyx** [16:55] Third-party models, as in non-o- non-OpenAI models? **Sherwin Wu** [16:58] Yeah. **Christina Huang** [16:58] Yeah. **Sherwin Wu** [16:58] Yeah. It'll work with, uh, Evals, uh, starting today. **Christina Huang** [17:01] Yeah. **Swyx** [17:01] Okay, got it. **Sherwin Wu** [17:02] Uh, we have a really cool setup with, uh, Open Router- **Swyx** [17:04] Mm **Sherwin Wu** [17:05] ... um, where we're working with them, and then you can bring your Open Router set up. Um, and then with that you can actually... You know, you write your evals, uh, using our datasets tool or, or use our dataset tool to create a bunch of evals, and you'd actually be able to hit, uh, a bunch of different model providers. Um, you know, take your pick from wherever, even, like, open source ones on Together, uh, and see the, see the results, uh, in our product. ### Agent Evals **Swyx** [17:25] Yeah, that's awesome. Um, uh, speaking more about evals, right? Like, uh, I think, uh, I saw somewhere in the, the release docs that you basically had to expand the evals product a little bit to, uh, to allow for agent evals. Um, maybe you can talk about, like, what you had to do there. **Sherwin Wu** [17:42] Yeah. **Christina Huang** [17:44] You have an answer. **Sherwin Wu** [17:44] Yeah, I, I was gonna say, so the-- I actually think agent evals is still a work in progress, so I think we've, like, made maybe ten percent of the progress that we need, um, here. Um, for example, I think we could still do a lot more around multimodal, um, evals. Um, but, uh, uh, the, the main progress that we, we made this time was, um, kind of allowing you to take traces. So, um, the, the agents SDK has, like, a really nice traces feature where if you run, uh, if you define things, you can have like a really long trace, allowing you to use that in the Evals product and, um, uh, uh, be able to grade it in some, uh, way, shape, or form over the, the entire, the entirety of what it's supposed to be doing. Um, I think this is step one. Like, I think it's good to, to be able to do this, but, um, uh, I, I think our roadmap from here on out is to, you know, really allow you to break down the different parts of the trace, uh, and, and allow you to eval and, like, kind of like measure each of those and optimize each of those as well. A lot of the times this will involve human-in-the-loop as well, which is why we have the, the human-in-the-loop component here too. Um, but, uh, if you kind of look at our Evals product over the last year, it's been very simple. It's been much more geared towards this, like, simple prompt completion setup. But obviously as we see people doing these longer, um, uh, agentic traces, like, you know, how do you even evaluate a 20-minute task, uh, correctly? And it's like this is a really hard problem. Um, we're trying to set up our Evals product to move in that way to, to help you not only evaluate the overall trajectory but also individual parts of it. **Swyx** [19:01] Yeah, I mean, the magic keyword is rubrics, right? Everyone- **Sherwin Wu** [19:04] Yep **Swyx** [19:05] ... wants LMS judge rubrics. **Sherwin Wu** [19:07] Yep, yep, yeah. **Swyx** [19:08] Obviously where this, this will go. **Sherwin Wu** [19:10] Yeah. **Swyx** [19:10] Okay, great. Um, the, the other thing that I think, uh, on-online I see the developer community very excited about is sort of automated prompts optimization, which is kind of evals in the loop with, with, with prompts. Uh, what is the thinking there? ### Optimization & Tuning **Swyx** [19:21] Where, where's things going? **Christina Huang** [19:23] Yeah. So, so we have automated prompt optimization, but, um, again, like, kinda, I think this is an area that we definitely want to invest more in. We, um, I think did a pretty big launch of this when we launched GPT-5 actually, because we saw that it was pretty difficult as new models come out to kind of learn all the quirks about a new model. And there's- **Swyx** [19:38] Yeah, the prompts optimization. **Christina Huang** [19:39] Right. There's like we have a big prompting guide, right, for every model that we launch, and, um, I think building out a system to make that a lot easier, um, we definitely want to tie that in, like, completely with evals. We should be able to kind of improve your prompts over time, improve your agents over time as well, if they're kind of made in the Agent Builder based on the evals that you've set up. And so I think we see this as like a pretty core part of, of the platform of basically suggested improvements to, to the things that you're building. Um- **Sherwin Wu** [20:05] I actually think it's a really cool time right now in prompt optimization. I'm sure you guys are seeing this too, is like, uh, not only are there a lot of products kind of like gearing around this, so like kind of what we're, we're thinking about, but I also think, like, there's a lot of interesting research around this, like GEPA with like... The Databricks folks are actually doing really cool stuff around, uh, this. Um, we're obviously not doing any of the, the cool GEPA optimization right now in, in our product. **Christina Huang** [20:25] Sure. **Sherwin Wu** [20:25] But, uh, would love to, would love to do that soon. And, um, also it's just an active research area, so like, you know, whatever, uh, uh, Matei and the Databricks folks, like, might think about next, what we might, you know, think about internally as well, um, uh, whatever new prompt optimization techniques come out, I think we'd, we'd love to, uh, be able to have that in our, in our product as well. Um, and yeah, and, and it's interesting 'cause it's coming at a time when people are realizing that prompt... You know, like, like I feel like two years ago people were like, "Oh, at some point prompt, like prompting's gonna be dead." **Swyx** [20:51] No. **Sherwin Wu** [20:51] And like- **Swyx** [20:51] Yeah. **Sherwin Wu** [20:52] You know? And it's like, you know- **Swyx** [20:53] It's gone up. **Sherwin Wu** [20:54] Yeah, yeah, yeah. And then if anything, it is like become more and more entrenched. Um, and, uh, I think that, you know, there's this interesting trend where like it's becoming more and more important, and then there's also interesting, cool work being done to like further entrench like prompt optimization. **Swyx** [21:06] Hmm. **Sherwin Wu** [21:07] Um, uh, and so, uh, that-that's why I just think it's like a very fascinating, you know, area to, to follow right now, and also was an area where I think a lot of us were wrong, uh, two years ago because, uh, if anything, it's only gotten more important. **Swyx** [21:17] Yeah, I would say like, uh, uh, uh, what, um, Shenyu used to work at OpenAI, now at, now he's at MSL, uh, would call this kind of like zero gradient fine-tuning or zero gradient- **Sherwin Wu** [21:26] Yeah **Swyx** [21:27] ... updating 'cause you're just tweaking the prompts. But like- **Sherwin Wu** [21:29] Yeah **Swyx** [21:29] ... uh, it, it is so much prompt that it's actually, like you end up with a different model at the end of it. **Sherwin Wu** [21:34] There's a lot of like things that make it more practical too, just like even from our perspective. Like we, we have a fine-tuning API, and like it is extremely difficult for us to run, you know, and serve like all of these different snapshots. Like, you know, Laura's great, uh, uh, MSL just, you know-- Or sorry, um, uh- **Swyx** [21:50] Thinky **Sherwin Wu** [21:50] ... Thinking Labs just, just published a... John Schulman just had a cool blog post about this. But like, man, it is like pretty difficult for us to like manage all of these different snapshots. And so if there is a way to like hill climb and yeah, do this like zero, uh, gradient like, uh, optimization, um, via prompts, like, yeah, I'm all for it. And, and I think developers should be all for it because you get all these gains without having to do any of the, uh- **Swyx** [22:09] Yeah **Sherwin Wu** [22:09] ... you know, fancy, fancy fine-tuning work. **Swyx** [22:11] Uh, since you are part of the API t-- uh, you know, you lead the API team, and since, uh, we, you mentioned Thinky, I gotta throw a cheeky one in there. What do you think about the Tinker, uh, API? **Sherwin Wu** [22:21] So yeah, it's a good one. Uh, so it's actually funny, when it, when it launched, I actually DM'd John Schulman, and I was like, uh- **Swyx** [22:26] Really? **Sherwin Wu** [22:27] "Wow, we finally launched it." So the- **Swyx** [22:28] 'Cause you used to work with him. **Sherwin Wu** [22:30] Uh, yeah, yeah. So we, um... It was a- it was actually funny. So at, um, uh, yeah, so right, right when I joined, uh, OpenAI, like this has actually been I, I, I think a, a passion project of John's. Like, he's been talking about doing something in this sh- uh, like in, in, in this shape for a while, which is like, uh, a truly like low-level, um, research, like fine-tuning library. And so we actually talked about it, um, quite a bit, um, uh, when he was at OpenAI as well. It's actually funny, I talked to one of my friends who said that when he was at Anthropic, he also, you know, worked on this idea for a bit, and now- **Swyx** [23:02] He's a man on a mission. **Sherwin Wu** [23:03] Yeah. I mean, I mean, John's like so great in this regard. He's, he's like so purely just like interested in the impact of this 'cause it's... One, it's, one, it's like a really cool problem, and then two, it also empowers builders and researchers. Like, you saw all the researchers who, who like expressed all this love for Tinker 'cause it is a really great, great product. And so, um, I, I'm just really happy to see that they, they shipped it and, uh, uh, I think he was really happy to kinda get it out there in the world as, uh, as well. **Swyx** [23:26] Yeah. It, it's, this is probably, this is very much a digression, but like it's weird a-a-as someone passionate about API design that it took this long to find a good fine-tuning API abstraction, which is effectively all he wanted. He was like, like, "Guys," like, "I don't want to worry about all the infra." Like, "I'm a researcher. I, I just want these four functions." And like it's, it's kind of interesting. **Sherwin Wu** [23:44] Yeah, yeah. **Swyx** [23:45] Cool. **Sherwin Wu** [23:46] Before the OpenAI comms team barges in the room. **Swyx** [23:48] I know. **Sherwin Wu** [23:50] Um, so what feedback do you want from people on, like, the Agent Builder? Um, for example, the thing I was surprised by was the if else blocks not being natural language and using the common expression language. I'm sure that's something already on, on your roadmap. ### Builder Feedback **Sherwin Wu** [24:03] What are other things where you're kinda, like, at a fork that you would love more input on? **Christina Huang** [24:07] I think, like, one of the things that we, um, spent a lot of time d-discussing was, like, whether we want kind of more of, like, the deterministic workflows or more LLM-driven workflows. And so, um, I think, like, getting feedback on that. Honestly, having people model existing workfl- A, a lot of what we did was kind of work with our team on, um, especially with engineers who are working with customers, like, modeling the workflows that already exist in the Agent Builder and, like, what gaps exist, like, what types of nodes are really common, um, and how can we, like, add those in? I think that was, that'd be, like, the most helpful feedback to get back. Um, and then, uh, as we expand kinda from just, like, chat-based... Like, right now the, the initial deployment for Agent Builder is through ChatKit. Um, we plan on kind of releasing more standalone, like, workflow, uh, runs as well, and kind of the types of, like, tasks that people would like to, to use in that type of API. **Swyx** [25:00] So, like, more modalities, for example. Um- **Christina Huang** [25:03] Yeah. I mean, I think, like, for sure, like, more modalities, like w- you know, I think kind of voice would be, uh... i-is already something that a lot of people have talked to us about, even today at DevDay. Um, so I think modalities for sure, but also more, like, the logical nodes of- **Swyx** [25:19] Yeah **Christina Huang** [25:19] ... what, what can't be expressed today. **Swyx** [25:21] Yeah. Um, well, you know, you're, you're building a language, right? Um, you have common expression language, which I never heard of prior to this. Uh, I thought this... Was this Python? Was this JavaScript? And then there was, like, a whole link in there. Was that a, a big decision for you guys? Was it, you know- **Christina Huang** [25:35] I think that was more just kind of like a way that we thought we could kind of represent a mix of, like, the variables and- **Swyx** [25:40] Yeah **Christina Huang** [25:40] ... I don't know, like, conditional statements. Um- **Swyx** [25:43] Yeah, yeah **Christina Huang** [25:43] ... but yeah. **Swyx** [25:43] The, the other thing I'll also mention is that you let-- Once you, um... So, uh, there's a trope in developer tooling where, like, anything that can be, that can store state will eventually be used as a database- **Christina Huang** [25:53] Mm **Swyx** [25:53] ... including DNS. Uh, so, so be prepared for your state store to become a database. Uh, I don't know if there's, like, any limits on that because people will be using it. **Sherwin Wu** [26:02] It's actually funny. Yeah, I, I, I'd heard this quote before, and there's definitely some truth to it. I, I don't know if our stateful APIs have become a database- ... I guess quite yet. But, like, who knows? Like, you know- **Christina Huang** [26:12] Yeah. I mean, conversations- **Swyx** [26:13] Well, well, you charge for it. You charge for assistance A- **Christina Huang** [26:16] Storage, yeah **Swyx** [26:17] ... you know, the storage. **Sherwin Wu** [26:17] Yeah, yeah **Swyx** [26:18] Right? So there's some limit on that. But, like- **Sherwin Wu** [26:20] Yeah, but it's very cheap. It's like- **Christina Huang** [26:21] Yeah **Sherwin Wu** [26:21] ... I remember we priced it like- **Christina Huang** [26:22] I think if you wanted to kind of, like, dump all your data somewhere, I don't know, this is, like, the most ... Like, transforming it all into this shape is like the- **Swyx** [26:28] Yeah, it's useful. It's easy **Sherwin Wu** [26:29] Yeah **Christina Huang** [26:29] ... best place for it. But yeah. **Sherwin Wu** [26:31] But also, please don't do this 'cause I think it'll- ### MCP & Auth **Christina Huang** [26:32] No. **Sherwin Wu** [26:32] ... put quite a bit of strain on, on Ventad and our info team- **Christina Huang** [26:35] Yeah **Sherwin Wu** [26:35] ... and, and, uh, what we try and do. So yeah. Um, how do you think about the MCP side? So you have OpenAI first-party connectors. You have third-party preferred, I guess, servers you will call them, and then you have open-ended ones. Do you see the, that part of registry-like functionality expanding or do you see most of it being user-driven? Auth is, like, the biggest thing. Like, if you add Gmail and Calendar and Drive, you have to, like, auth each of them separately. There's not, like, a canonical, uh, auth. What, what's the thinking there? **Christina Huang** [27:04] Yeah, I mean, I think definitely for the registry, that's why we wanna make it a lot easier for, um, like, companies to kind of manage what their, like, developers have access to, managing kind of the configurations around it. And I think in terms of, like, first party versus third party, like, we wanna support both of those. We have some direct integrations and then, I don't know, anyone can kind of create MCP servers. I think we wanna make that a lot easier, too, like establish kind of private links for, for companies to use those internally. So, um, I think, like, just really excited about that ecosystem growing. **Sherwin Wu** [27:35] Yeah, I, I think one of the coolest things observed too is just... I, I actually think we, we as an industry are still trying to figure out the ideal shape of connectors. So I mean, part of why I think the 1P connectors exist too, like we, we end up storing, uh, uh, quite a bit of state. It's, like, a lot of work for us. But, like, by having a lot of state on our side, we call them sync connectors, we can actually end up doing a lot more creative stuff on our side when you're chatting with ChatGPT and using these connectors to, to kind of boost the quality of, of how you're using it, right? Like, if you have all the data there, you can do all this, like, re-ranking. You can, like, do-- We can put it in a vector store if you want. We can put it anywhere else. Um, whereas, um... And then so there, there are some inherent trade-offs here where, like, you, you put in a lot of work to get these, like, 1P connectors working. But, um, because you have the data, you can do a lot more and get higher quality. But then, but then the question is like, oh my god, there's, like, such a long tail of other things, which is where the MCP and, like, the third-party connectors come in. But then you have the trade-off of, like, you're beholden to, like, the API shape of the MCP creator. It might actually work well, it might not work well with, uh, uh, with the models, and then what happens if it doesn't work well? Then you kinda have to, like, you know, um, you're kind of, like, at the mercy of this. And MCP, by the way, is, like, really great 'cause it already does some layer of standardization, but my sense is there's still gonna be more evolving here, and I think, you know, we wanna support both of them because we see value in both right now, especially working with, working with developers. We wanna have kind of, like, all options kind of on the table here. But it will be interesting to see how, see how this evolves over time. Yeah. When I saw about three, four months ago, when you launched the forum for, like, signing with ChatGPT interest, I think to me that's kinda like the vision, where I log in and I have the MCPs tied in, and then I sign in with ChatGPT somewhere, and I can run these workflows in that app where I'm logging in. So, um, yeah, I think Sam, you know, said in an interview that he sees ChatGPT as, like, your personal assistant. So, um, I think this is, like, a great step in that direction. Yeah. I think there's a lot more to, to go in that, in that direction. **Swyx** [29:23] But, but so far, no plan on, like- ChatGPT or OpenAI's IDP, right? Which is a different role in the S- in the auth ecosystem. **Sherwin Wu** [29:32] Yeah, it's interesting 'cause, um... So, so, um, direct answer is, like, no plans right now, of course. Um, but, um, I actually think we currently have some version of this, which is our a- uh, partnership with Apple. Uh, because with Apple, you can actually sign in, uh, to your ChatGPT account, and some of that identity does carry with you into, uh, your iOS experience with Siri. **Swyx** [29:49] Oh. **Sherwin Wu** [29:50] Right? Like, if you, uh, if you, um... I don't know if you've actually used this, uh, the, the Siri integration. I, I actually use it quite a bit. But, um, if you sign into your ChatGPT account, um, the Siri integration will actually use your subscription status to decide what type of model to use, um, when it, when it, uh, passes things over to ChatGPT. And so if you're, uh, you know, uh, just a free, uh, user, uh, you get, you know, the, the, the free model, but if you're a Plus or a Pro subscriber, you get routed to, uh, uh, GPT-5, which is I think what they updated. **Christina Huang** [30:17] I think we also recently announced the partnership with Kakao. **Sherwin Wu** [30:19] Oh, yeah, Kakao's another one. **Christina Huang** [30:20] Yeah. Where, um, I think you c- a similar thing, where you can sign in with ChatGPT. Kakao is one of the largest- **Swyx** [30:26] It's coming **Christina Huang** [30:26] ... like, messenger, yeah, apps in Korea. Um, and kind of interact with Kakao directly there. **Sherwin Wu** [30:32] Yeah, I mean, Sam's been talking about it for a while. It's a very compelling vision. We obviously want to be very thoughtful with, kind of how we do it. **Swyx** [30:37] You know, now you have a social network, you have a developer platform. **Sherwin Wu** [30:40] Oh, yeah . **Swyx** [30:41] My, you know, my ChatGPT account- **Sherwin Wu** [30:41] We have the beginnings of a social network **Swyx** [30:42] ... is very, very valuable. Yeah. **Sherwin Wu** [30:44] Yeah. **Swyx** [30:44] Yeah, exactly. Uh, okay, so and then on the other side of auth is something I was really interested to look at, and I couldn't get a straight answer. Is there some form of bring, bring your own key, uh, for AgentKit? Like, uh, when I, when I expose it to the wider world, uh, obviously, like, I'm u- by default, I'm paying for all the inference, but it'd be nice for that to have a limit, and then if you want more, you can bring your own key. **Christina Huang** [31:08] Yeah. I mean, we, we don't have something like that yet. **Swyx** [31:11] Yeah. **Christina Huang** [31:11] Um, but I think, yeah, it's definitely an interesting area too. **Sherwin Wu** [31:15] Yeah, it, it doesn't do it out of the box today. **Christina Huang** [31:16] Yeah. **Sherwin Wu** [31:16] But, um, you know, uh, developers have been asking about it for forever. **Swyx** [31:20] Yeah. **Sherwin Wu** [31:20] Like, it- **Swyx** [31:20] Yeah **Sherwin Wu** [31:20] ... it's a really cool concept because then as a developer, you, especially a new developer, you don't need to bear the burden of, of inference. **Swyx** [31:26] Yeah. I, I think, like, when you get into the business of, like, agent builders that are publicly exposed, where you have, like, an allow list of domains, like, this is, this is the, the, it rhymes with this exact pattern of like- **Sherwin Wu** [31:37] Yeah **Swyx** [31:37] ... uh, someone has to bear the cost, and like- **Sherwin Wu** [31:39] Yeah **Swyx** [31:39] ... sometimes you wanna, um, mess around with, like, the different levels of responsibility. **Sherwin Wu** [31:43] Yeah. I mean, I will say in general, like, if you kind of look at our roadmap, we, we, we engage a lot with developers. We kind of hear what is the, are the pain points, and we try and build things that address it, and, you know, ideally, we're prioritizing in a way that's, that's helpful. Um, but yeah, we, we've definitely heard from a good number of developers that, like, the cost is- Or like, uh, all of the, like, copy/paste your key, like, solutions right now, which are, like, huge security iss- like, hazards, um, because developers don't wanna bear the burden of, of inference. You know, hopefully, we make the cost cheaper so it's- **Christina Huang** [32:10] The models keep getting cheaper as well. **Sherwin Wu** [32:11] Yeah, yeah. So hopefully, you know- **Swyx** [32:12] It's weak **Sherwin Wu** [32:12] ... that, that, that helps. **Swyx** [32:13] Yeah. **Sherwin Wu** [32:13] But, uh, but, but what we realized is as we make it cheaper, you know, the demand for that goes up even more- **Swyx** [32:17] Yeah **Sherwin Wu** [32:17] ... and you end up, you know, still spending quite a bit. But, um, yeah, so we definitely heard this from a lot of developers, and it's definitely something top of mind. **Swyx** [32:24] Yeah. ### Platform & Widgets **Alessio** [32:24] Do you see this as mostly like an internal tools platform, though? Like, to me, like you've been doing a big push on like the more forward deployed engineering things. It's almost like, hey, we needed to build this for ourselves as we sell into these enterprises, might as well open it up to everybody. What drives dri- building these tools? Like, you think of people building tools to then expose or mostly on the internal side? **Christina Huang** [32:45] Yeah, I mean, and so, like, I think our, again, our first deployment is ChatKit, which is kind of one of... It, it's intended to be for external users. Um, but I think one of the things that we also did see a lot as we were working with customers is that a lot of companies have actually built some version of an agent builder internally, um, to kind of manage prompts internally, to manage templates that they're sharing across, um, you know, the different developers that they have, maybe the different product areas. Um, and we were seeing that kind of like over and over again as well and, um, really wanted to, like, build a platform so that this is not, you know, an area that every company needs to invest in and, like, rebuild from scratch, but that they can kind of have a place where they can manage these templates, manage these prompts, and really focus on the parts of agent building that is more unique to their, like, business. **Sherwin Wu** [33:30] It is interesting, too, like from a deployment perspective. It is, like, it, it has spanned both internal and external use cases, right? Like, kinda like these internal platforms, people will use it for in, like, data processing or something, which is an internal use case. But if you saw some of the demos today, like there have been a huge number of companies that are trying to do this for external-facing- **Christina Huang** [33:47] Mm-hmm **Sherwin Wu** [33:48] ... um, uh, use cases as well. **Swyx** [33:49] Customer service is one template in there. **Sherwin Wu** [33:50] Um, customer service, the, like, Ramp use case, which is- **Christina Huang** [33:52] We use this internally and externally, like our customer support help.openai.com is already powered on AgentKit, and then various- **Sherwin Wu** [33:59] Yeah, yeah **Christina Huang** [34:00] ... other, like, internal- **Sherwin Wu** [34:01] Yeah **Christina Huang** [34:01] ... use cases as well. **Sherwin Wu** [34:01] And, and one of the things that I actually think the team has done a really great job of, uh, so, like Tyler, David, and Jiwon on the team, um, they built the, the, especially the, the ChatKit components, they built it to be like very consumer-grade and, and, like, very polished. Like, you kinda look at that, there's like a whole grid of like the different widgets and things that you could create there. Like, ideally, people see it and they, like, they see it as, like, these very polished, like consumer-grade, ready, uh, external-facing things versus like, you know, you think of internal tools and like the UI is always, like, the last thing that people care about. **Swyx** [34:29] Yeah. **Sherwin Wu** [34:30] But, like you really, you know, pushed the team, and I think they did a really great job of making chat, the ChatKit experience, like, really, really consumer grade, and it should feel almost like ChatGPT or, um- **Christina Huang** [34:38] Yeah **Sherwin Wu** [34:39] ... and with like really buttery smooth animations and like really, uh, responsive designs and all of that. **Christina Huang** [34:43] Yeah, I think your point on widgets is like, definitely like really resonates, right? Because, um, ChatKit, it handles the chat UX, but, um, we're also just building like really visual ways for you to represent like every action that you wanna take and, um, that is definitely like very high polished. **Sherwin Wu** [35:00] Yeah, and when working with customers, like those have been the, the most helpful customers for us to work with because, you know, when Ramp is thinking about, you know, how, what, what they want to publicly present to people, like they have a pretty high bar, uh, as they should, um, as, as, as well as, you know, all the other customers that have been iterating on it. And so that kind of feedback from our customers has really helped us up-level the general product quality of, of the launch that we had today as well. **Swyx** [35:20] Yeah. Um, would you ever, would you open source ChatKit? **Sherwin Wu** [35:26] We talked about it. **Christina Huang** [35:27] Uh-huh. **Sherwin Wu** [35:27] We've talked about it. There are a bunch of trade-offs. Um, uh- **Christina Huang** [35:31] I think so, so- **Sherwin Wu** [35:31] Yeah **Christina Huang** [35:31] ... ChatKit itself is like an embeddable iframe, and so I think the actual- Oh, it's an iframe? Yeah. **Sherwin Wu** [35:36] I thought it was like a- **Christina Huang** [35:36] Right, right, right **Sherwin Wu** [35:37] ... API key. **Christina Huang** [35:37] And so that helps us keep it, like evergreen, right? So if you are using ChatKit and we come up with new, I don't know, a new model that reasons in a different r- way, right, or kind of new modalities, that you don't actually need to rebuild and like pull in new components to use it in the front end. I think there's parts of, you know, w- widgets for example, that is much more like a language and can definitely, um, is something that is easier to explore that for, as well as kind of the, the design system that we've built, um, for ChatKit. Um, but I think like as part of, yeah, the a- the actual i- iframe itself, I think there's a lot of value in that being, um, yeah, more- Posted ... evergreen- **Sherwin Wu** [36:11] Yeah **Christina Huang** [36:12] ... more evergreen experience that is, uh, pretty opinionated. **Sherwin Wu** [36:14] Like there'd be no point in being open source- **Christina Huang** [36:15] Right. Yeah, yeah **Sherwin Wu** [36:16] ... 'cause you want the, uh- **Christina Huang** [36:17] Then you don't get the benefits of it. **Sherwin Wu** [36:18] It, uh, you know, being Stripe alums, like Stripe Checkout, like it's, it's auto- op- optimized for you to like- **Christina Huang** [36:24] So I'm not a Stripe alum, but Christina is. And the team actually is the team that built Stripe- **Sherwin Wu** [36:29] Stripe Checkout? **Christina Huang** [36:30] Yeah, so, um, it's very similar philosophically, right? So Stripe, you know, can build elements in checkout, and not every business needs to rebuild, right, the pieces that are really common, and I think we see the same with chat. We see chat being built over and over again, um, especially as we kind of come up with new, you know, modalities like reasoning, everything. It's not really something that is easy to keep up to date, and so we should just do that, um, and leave kind of the hard parts of building agents again to- **Sherwin Wu** [37:02] Yeah **Christina Huang** [37:03] ... to the developers. **Sherwin Wu** [37:04] Does it feel... I mean, I, I know WordPress has like a bad connotation in a, a lot of circles, but to me it almost feels like the WordPress equivalent of like chat. It's like, hey, this is like drop in thing- **Christina Huang** [37:15] Mm **Sherwin Wu** [37:15] ... and then you have all these different widgets. Do you see the widget becoming a big kinda like developer ecosystem- **Christina Huang** [37:21] Yeah, for sure **Sherwin Wu** [37:21] ... where people share a widget? Is that kinda like a first-party thing, and then- **Christina Huang** [37:24] Yeah **Sherwin Wu** [37:25] ... what's like the MCP versus- **Christina Huang** [37:27] Widget forest **Sherwin Wu** [37:28] ... widget forest. No, exactly. I mean, it's kinda like i- it seems great for people that are like in between being technical and like not really being technical enough. **Christina Huang** [37:35] Yeah. Yeah, I mean, I think that's a big part of building widgets, right? Like it's real- already kind of in the language that is very, um, consumer friendly. You can use, you- in our widget builder already, already you can kind of use AI to c- create those widgets, and they look pretty good. Um, I don't know if you guys have gotten a chance to try that out yet, but definitely see kind of, I don't know, a forest, is that what you said? **Sherwin Wu** [37:54] Yeah, if you haven't, if you haven't tried out the widget studio- **Christina Huang** [37:56] Yeah **Sherwin Wu** [37:56] ... um, and, and the demo- **Christina Huang** [37:57] Yeah, yeah **Sherwin Wu** [37:57] ... like, uh, uh apps as well, yeah, they're very- **Christina Huang** [38:00] You got a custom domain, like widget.studio, which is cool. **Sherwin Wu** [38:03] Actually don't know how we got that, but- **Christina Huang** [38:05] Yeah **Sherwin Wu** [38:05] ... it was fun **Christina Huang** [38:05] ... everything's in chatkit.studio, and then we have like the playground there so you can try out what ChatKit would look like with all the customizations. We have chatkit.world, which is a fun site we built. I've- I was like spinning the globe- **Sherwin Wu** [38:15] Yeah, the globe, yeah **Christina Huang** [38:16] ... for a while this morning. **Sherwin Wu** [38:17] Yeah. **Christina Huang** [38:17] It was, um- **Sherwin Wu** [38:18] I think Kasha- **Christina Huang** [38:18] ... it was like a widget spinner and- **Sherwin Wu** [38:19] Kasha also like uploaded some of her like- **Christina Huang** [38:21] Yeah **Sherwin Wu** [38:21] ... solar system stuff and- **Christina Huang** [38:23] Yeah, yeah, yeah **Sherwin Wu** [38:23] ... all the, all the demos as well. **Christina Huang** [38:24] Yeah, and then that's where like the widget, um, builder lives as well. **Sherwin Wu** [38:27] Yeah, so, so it's like, it's really come together. Like w- it's taken like almost more than a year to like come together and like build all this stuff, but it's coming together, and it's like- **Christina Huang** [38:34] Mm-hmm **Sherwin Wu** [38:34] ... really interesting. **Christina Huang** [38:35] Yeah, yeah, it's something that we like, uh, like- **Sherwin Wu** [38:37] We definitely planned all of this upfront **Christina Huang** [38:39] ... oh, yeah, yeah. We have, we have the master plan from, you know, three years ago. Um, no, but like I think, uh, especially on this stuff, I think there was like an arc of a, a general like, you know, platform that we did want to kind of build around and, um, it takes a while to, to build these things. Obviously Codex helps speed it up quite a bit now, but, um, it, yeah, I will say it does seem great to kinda like start, start to have all the pieces start fitting together. **Sherwin Wu** [38:59] Yeah. **Christina Huang** [39:00] Um, I mean, you saw we launched Evals and we've had the fine-tuning API for a while, and, um, uh, and we laid all the groundwork for, for some of this stuff over the last year, and, um, we're hoping that we can eventually, you know, make it into this, this, this full feature platform that, that, that's helpful for people. **Sherwin Wu** [39:13] I think you have. Um, since you have, since you did the Codex mention, maybe a quick tip from each of you on Codex power user tools, uh, or, or tips. ### Codex Tips **Christina Huang** [39:23] The... So there, there's actually a, a funny one that, um, uh, one of the new grads, um, has, has I think like taught our team in general, um, and I think this is like a, a point for like, uh, just how like new grads and, and younger, you know, generation people are actually more AI native. So one of them is to like really lean into like, like push yourself to like trust the model to do more and more. So like I feel like the way that I was using Codex, um, and so for me it's most usually for my, my personal projects they, they don't let me touch the code anymore. Um, but uh, you give it like small tasks so you're like, you're, you're like not really trusting it. Like I, I view it as like this like intern that I like, like I really don't trust. Um, but what a lot of the like... So we had an intern class this year. What a lot of the interns would do is just like full YOLO mode, like- ... trust it to like write the whole feature, and it like, it doesn't work. For better or for worse. **Sherwin Wu** [40:10] It like doesn't work sometimes, but like, I don't know, like 30, 40% of the time it just like one-shots it. I actually haven't tried this with like Codec, GPT-5 Codex. I bet it, I bet it probably like one-shots it even more. **Christina Huang** [40:20] Yeah. **Sherwin Wu** [40:20] Um, but one tip that I'm like starting to like, I feel like undo this, uh, like, like relearn things here, uh, is to like really lean into like the AGI component of it and just like really let the model rip and like kinda trust it. **Christina Huang** [40:31] Yeah. **Sherwin Wu** [40:31] Because a lot of times it c- it can actually do stuff that surprises me and then I have to like readjust my priors, whereas before I feel like I was in this like safe space of like I'm just treating this, I'm giving this thing like a tiny bit of rope- **Christina Huang** [40:40] Yeah **Sherwin Wu** [40:40] ... and, uh, uh, and because of that I was kinda limiting myself with how effective I could be. **Christina Huang** [40:44] Yeah. **Sherwin Wu** [40:44] Like sure, but okay, but also is there an etiquette around submitting effectively s- you know, vibe coded PRs that someone else now has to review, right? And it's like it can be offensive- **Christina Huang** [40:55] Well, we have Codex do reviews now. Uh- **Sherwin Wu** [40:57] Okay **Christina Huang** [40:57] ... it, it actually reviews itself. Just full circle. **Sherwin Wu** [40:59] Does Codex approve its own PRs a lot more than humans? **Christina Huang** [41:02] It doesn't, it doesn't get to approve them, but it- I, I was gonna say, I think like the Codex PR r- reviews are actually one of like the things that my team like very much relies on. I think they're very, very high quality reviews. **Sherwin Wu** [41:12] Yeah. **Christina Huang** [41:12] Um, on the Codex PR side, like for the visual Agent Builder, we only started that, um, probably less than two months ago. **Sherwin Wu** [41:20] Hmm. **Christina Huang** [41:21] And that that wouldn't be possible without Codex. So I think there's definitely a lot of use of Codex, um, internally, and it keeps getting better and better. And so, um, yeah, I think people are just finding they can rely on it more and more, and it's not, you know, totally vibe coded. It's still, you know, checked and edited, but definitely as a kicking-off point, and I think I've heard of people on my team, it's like on their way to work, they're like kicking off like five Codex tasks- Because the bus takes 30 minutes, right? And you get to the office and it kind of helps you orient yourself for the day. You're like, "Okay, now I know the, the files, I have the rough sense." Like, maybe I don't even take that PR, and I actually just, like, still code it, but it helps you just context switch so much faster too, and be able to, like, orient yourself in, in a code base. **Sherwin Wu** [42:02] There's so many meetings nowadays where I have, like, one-on-ones with engineers, and I walk into the room, they're like, "Wait, wait, wait. Give me a second." "I gotta kick off my, like, Codex thing." I'm like, "Oh, sorry." **Christina Huang** [42:09] Yeah. **Swyx** [42:09] We're about to enter async zone. **Sherwin Wu** [42:10] Yeah, yeah. **Christina Huang** [42:10] It's almost like your notes, right? **Sherwin Wu** [42:11] Yeah, yeah. **Christina Huang** [42:11] You're like, "Let me..." **Sherwin Wu** [42:12] And they're, like, typing, they're like, "Okay, now we can start our one-on-one 'cause now it's ready." **Swyx** [42:16] Yeah. Um, cool. Uh, we're almost out of time. I wanted to leave a little bit of time for you to shout out the Service Health Dashboard, 'cause I know you're passionate about it. ### Service Health **Sherwin Wu** [42:23] Oh, yeah. **Swyx** [42:24] Uh, well, tell people what it is and why, why it matters. **Sherwin Wu** [42:27] Yeah, so, um, this is a launch that we actually didn't, you know, uh, it didn't get any stage time, um, today, but it was actually something I'm really excited about. So, um, we launched, uh, this, this, this thing called the Service Healths, Health Dashboard. You can now go into your, um, uh, usage or, like, your settings, uh, account and kind of see the health of your, uh, integration with our OpenAI API. And so this is scoped to your own org, so basically if you have an integration that's running with us doing a bunch of, you know, uh, uh, tokens per minute or a bunch of queries, it's now tracking each of those responses, looking at your token velocity, um, uh, TPM that you're getting, the throughput, as well as the responses, uh, the response codes. And so you can see a, kind of like a real time personal SLO, um, for your integration. The reason why I care a lot about this is, um, o- obviously over the last year we've spent a lot of time thinking about reliability. We had that really bad outage last December, uh, you know, longest like three, four hours of my life, and then had to, you know, talk to a bunch of customers. Um, uh, we haven't had one that bad since, you know, knock on wood. Um, we've done a bunch of work, uh, uh, uh, our... We have an infra team led by, uh, Venkat, and they've been working with Janna on our team, and they've just been doing so much good work to get reliability better. And so it's... Uh, we actually... Again, knock on wood. Um, we're- We think we've got reliability in a spot where we're, like, comfortable kind of putting this out there, um, uh, and, and kind of, like, letting people actually see their, their SLO. And hopefully, you know, it's, you know, three, four, soon to be five nines. Um, but, uh, the reason why I cared a lot about it is because we spent so much time on it, and, um, we feel, uh, confident enough to kind of have it, uh, behind a product now. **Swyx** [43:58] Five nines is, like, two minutes of o- outage or something. **Sherwin Wu** [44:01] Yeah, yeah. We're, we're working, we're working to get to- **Swyx** [44:03] What? **Sherwin Wu** [44:03] ... to five nines. Yeah. **Swyx** [44:04] What is, what does a- an extra nine take? **Sherwin Wu** [44:07] It's, uh, it's exponentially more work, so, you know. Uh, and then... But, like, we always, we were, you know, in the last couple of years we were talking about, like, hitting three nines, and then hitting three and a half nines, and then hitting four nines. Um, uh, but yeah, it's, it's exponentially more work. I could, I could go for a while on the, on the different, different topics, but, uh- **Swyx** [44:25] We'll have to do that in a, in a follow-up. Like, I mean, that's all, that's the engineering side, right? **Sherwin Wu** [44:29] Yes, yes, yes. **Swyx** [44:30] Like, you're serving 6 billion tokens per minute. **Sherwin Wu** [44:33] We actually zoomed past that. Yeah, that's the, that's the- **Swyx** [44:35] That's outdated. **Sherwin Wu** [44:35] Yeah. But, um, yeah, it's been crazy though, the growth that we've seen. **Alessio** [44:39] Um, awesome. I know we're out of time. It's been a long day for both of you, so, uh, we'll let you go, but thank you both for joining us. **Christina Huang** [44:45] Yeah. **Sherwin Wu** [44:46] Yeah. Thanks for having us. **Christina Huang** [44:47] Thanks. **Swyx** [44:48] Thank you. **Alessio** [44:48] That's it. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.