# The new OpenAI Agents Platform: CUA, Web Search, Responses API, Agents SDK!! Latent Space · 2025-03-11 OpenAI's Nikunj Handa and Romain Huet join host Swyx to announce their new Agents Platform, including the Responses API, Web Search, File Search, Computer Use tools, and an upgraded Agents SDK. The Responses API unifies Chat Completions and the Assistants API, offering free state storage for 30 days and acting as a strict superset for future agentic workflows; Chat Completions remains supported, but the Assistants API will sunset in first half 2026. Web Search is available both as a built-in tool in Responses and as a fine-tuned GPT-4o Search Preview model in Chat Completions, achieving 90% accuracy on simple QA compared to 30% for GPT-4o. The File Search tool gains metadata filtering, while Computer Use—powering OpenAI's Operator—enables browser automation via screenshots and action signals, described as the "GPT-2 of computer use." The Agents SDK builds on the Swarm experiment, adding type support, guardrails, handoffs, and tracing in the OpenAI dashboard, with plans to tie traces to evals and reinforcement fine-tuning. ## Questions this episode answers ### How much does OpenAI's new web search model improve accuracy over the base GPT-4o? Romain Huet states that in simple QA, the base GPT-4o model achieves 30% accuracy, while the fine-tuned GPT-4o Search Preview model jumps to 90% accuracy. This fine-tuned model, used by ChatGPT for Search, is available via the Responses API as a tool and via Chat Completions as a separate model, combining with structured outputs for real-time internet data structuring. [8:02](https://addtry.com/5c3bba76-f4a0-4189-809c-da0675bd33a1?t=482000) ### Does the OpenAI Responses API store conversation state, and does it cost extra? Nikunj Handa confirms that the Responses API stores conversation state free of charge for 30 days, aiding debugging and observability directly in the OpenAI dashboard. Developers can disable state storage by setting store=false to make the API stateless like Chat Completions. This state is a core part of the API’s unification of chat completions and assistants, simplifying multi-turn workflows without hidden costs. [6:37](https://addtry.com/5c3bba76-f4a0-4189-809c-da0675bd33a1?t=397000) ### What improvements does the OpenAI Agents SDK bring compared to the experimental Swarm library? Nikunj Handa explains that the Agents SDK adds support for types, guardrails (parallel checks that can block execution), and built-in tracing visible in the OpenAI dashboard. It retains Swarm’s handoff mechanism for multi-agent orchestration but is production-ready, supports any provider with a Chat Completions API format, and integrates with multiple tracing providers, making agentic workflow debugging and iteration simpler. [21:40](https://addtry.com/5c3bba76-f4a0-4189-809c-da0675bd33a1?t=1300000) ## Key moments - **[0:00] Intro** - [0:41] OpenAI launches Web Search, File Search, and Computer Use tools, plus a new Responses API and Agents SDK. - **[2:21] Responses API** - [2:42] Chat Completions API remains supported; Responses API designed for multi-turn agentic workflows. - [3:28] Assistance API to sunset by mid-2026, with migration path to Responses API. - [5:23] Responses API supports stateless mode by passing store=false, enabling Chat Completions-like use. - [6:44] Conversation state storage is free for 30 days in Responses API, aiding debugging. - **[7:49] Web Search** - [8:01] GPT-4o Search Preview achieves 90% accuracy on simple QA, up from 30% for base GPT-4o. - [10:11] “It’s kind of like an API for the internet” — Romain Huet, on combining web search and structured outputs. - [12:54] Q: Can search depth or breadth be tuned in Web Search tool? A: Not yet, but OpenAI is exploring it. - [15:02] Combining File Search, Web Search, and private documents yields precise, company-specific answers. - **[15:49] File Search** - **[18:17] Computer Use** - [18:40] “Computer use is like the GPT-1 of computer use right now” — Nikunj Handa on its early stage. - **[21:34] Agents SDK** - [22:17] Agents SDK adds tracing, guardrails, and type support, with dashboard for debugging agent workflows. - **[25:05] Outro** - [25:05] OpenAI plans to connect agent traces to evals for reinforcement fine-tuning (RFT) to improve agents. ## Speakers - **Swyx** (host) - **Nikunj Handa** (guest) - **Romain Huet** (guest) ## Topics Agent Platforms ## Mentioned Hebiya (company), Navan (company), OpenAI (company), Stripe (company), Agents SDK (product), Assistants API (product), Chat Completions API (product), Claude (product), Computer Use (product), File Search (product), GPT-4o Search Preview (product), Operator (product), Swarm (product), Web Search (product), responses API (product) ## Transcript ### Intro **Nikunj Handa** [0:04] Hey, everyone. Welcome back to another Latent Space Lightning Episode. This is Alessio, partner and CTO at Decibel, and I'm joined by Swyx, founder of Osmo AI. **Swyx** [0:12] Hi, and today we have a super special episode because we're talking with our old friend, Romain. Hi. Welcome. **Romain Huet** [0:20] Thank you. Thank you for having me. **Swyx** [0:21] And Nikunj, who is most famously is if anyone has ever tried to get any access to anything on the API, Nikunj is the guy. So I, I know, I know, I know your emails 'cause I look forward to them. **Nikunj Handa** [0:31] Yeah, nice to meet all of you. **Swyx** [0:33] Yeah, I think that we're basically convening today to talk about the new API, so perhaps you guys wanna just kick off, uh, what is OpenAI launching today? **Nikunj Handa** [0:41] Yeah. So, uh, I can kick it off. We're launching a bunch of new things today. Uh, we're gonna do three new built-in tools. So we're launching a Web Search tool. This is basically ChatGPT for Search, but available in the API. We're launching, uh, an improved File Search tool. So this is you bringing your data to OpenAI. You upload it. We, you know, take care of parsing it, chunking it, embedding it, making it searchable, give you this, like, ready vector store that you can use. So that's the File Search tool. And we're also launching our Computer Use tool. So this is the tool behind the Operator product in ChatGPT. So that's coming to developers today. And to support all of these tools, we're gonna have a new API. So, you know, we launched Chat Completions, like, I think March 2023 or so. It's been a while. So, so we're looking for an update over here to support all the new things that the models can do. And so we're launching this new API called the Responses API. It is, uh, you know, it works with tools. We think it'll be, like, a great option for all the future agentic products that we build, and so that is also launching today. Actually, the last thing we're launching is, uh, the Agents SDK. We launched this thing called Swarm last year, where, you know, it was an experimental SDK for people to do multi-agent orchestration and stuff like that. It was supposed to be, like, educational, experimental, but, like, people, people really loved it. They, like, ate it up. And so we are like, "All right. Let's, let's upgrade this thing. Let's give it a new name," and so we're calling it the Agents SDK. It's gonna have built-in tracing in the OpenAI dashboard, so lots of cool stuff going out. Uh, so yeah, excited about it. **Romain Huet** [2:15] That's a lot, but we said 2025 was the year of agents, so there you have it, like, a lot of new tools- **Nikunj Handa** [2:20] Yeah **Romain Huet** [2:20] ... to build these agents for developers. ### Responses API **Swyx** [2:21] Okay. I guess, I guess we'll just kinda go one by one and l-we'll leave the Agents SDK towards the end. So Responses API, I think the sort of primary concern that people have, and something I think I voiced to you guys when, when I was, uh, talking with you in the, in the planning process, was is Chat Completions going away? Uh, so I just wanted to let it, uh, let you guys respond to the concerns that people might have. **Romain Huet** [2:42] Chat Completion is definitely, like, here to stay. You know, it's a bare metal API we've had for quite some time. Uh, lots of tools built, uh, around it, uh, so we wanna make sure that, uh, it's maintained and people can confidently keep on building on it. At the same time, it was kind of optimized for a different world, right? It was optimized for a pre-multimodality world. We also optimized for kind of single turn, uh, text to prompt, uh, text prompt in, text, uh, response out. And now with these agentic, um, workflows, we, we notice that, like, developers and companies wanna build, um, longer horizon tasks. You know, like things that require multiple returns to get the task accomplished, and Computer Use is one of those, for instance. And so that's why the Responses API came to life to kind of support these new agentic workflows. But Chat Completion's definitely here to stay. **Swyx** [3:28] And Assistance API, we've, uh, have a target sunset date of first half of 2026. So this is kind of like... In my mind, there was kind of very poetic mirroring of the API with the models. This-- I kind of view this as like kind of the merging of Assistance API and Chat Completions, right, into one unified responses. So it's kind of like how GPT and the O series models are also unifying. **Romain Huet** [3:48] Yeah, that's exactly the right, uh, the right framing, right? Like, I think we took the best of what we learned from the Assistance API, especially, like, being able to access tools very, uh, very, like, uh, conveniently. But at the same time, like, simplifying the way you have to integrate. Like, you no longer have to think about six different objects to kinda get access to these tools. With the Responses API, you just get one API request, and suddenly you can sweep in those tools, right? **Nikunj Handa** [4:13] Yeah, absolutely. And I think we're gonna make it really easy and straightforward for Assistance API users to migrate over to Responses API without any loss of functionality or data. So our plan is absolutely to add, you know, assistant-like objects and thread-like objects to-- that, that work really well with the, uh, Responses API. We'll also add, like, the code interpreter tool, which is not launching today, but will come soon. And, uh, we'll add async mode to Responses API, 'cause that's another difference with, with, uh, Assistants. Uh, we'll have web hooks and stuff like that. But I think it's gonna be, like, a pretty smooth transition, uh, once we have all of that in place, and we'll give folks, like, a full year to migrate and, and help them through any issues they, they, they face. So overall, I feel like Assistance users are really gonna benefit from this longer term, uh, with this more flexible parameter. **Romain Huet** [5:02] How should people think about when to use each type of API? So I know that in the past, the Assistance was maybe more stateful, kinda like long-running, many tool use, kinda like file-based things, and the Chat Completions is more stateless, you know, kinda like traditional completion API. Is that still the mental model that people should have, or, like, should you by default always try and use the Responses API? **Nikunj Handa** [5:23] So, so the Responses API is gonna support everything that the Cha-- it's a-at launch gonna support everything that Chat Completion supports, and then over time, it's gonna support everything that Assistance supports. So it's gonna be a pretty good fit for anyone starting out with OpenAI. Uh, they should be able to, like, go to Responses. Responses, by the way, also has a stateless mode, so you can pass in store false and, and that'll make the whole API stateless, just like Chat Completions. We're really trying to, like, get this unification, uh, story in, so that people don't have to juggle multiple endpoints. That being said, like, Chat Completions just, like, the, it's our most widely adopted API. It's, it's so popular, so we're still gonna, like, support it for years with, like, new models and features. But if you're a new user, you wanna... or if you're an, like, existing user, you wanna tap into some of these, like, built-in tools or something, you should feel f- feel totally fine migrating to Responses, and you'll have more capability and performance than, than Chat Completions. **Swyx** [6:17] I think the, the messaging that I agree that I, I think resonated the most when I talked to you was that it is a strict superset, right? Like, it sh- **Nikunj Handa** [6:23] Yeah **Swyx** [6:23] ... you should be able to do everything that you could do in chat completions and with assistance. The thing that I just assumed that because you're s- you're now, you know, by default it's stateful, you're actually storing the, the chat, uh, logs or the, the, the chat state. I thought you'd be charging me for it. So, uh, you know, as to me it was, uh, very surprising that you figured out how to make it free. **Nikunj Handa** [6:43] Yeah. **Swyx** [6:44] Yeah. **Nikunj Handa** [6:44] It's free. We s- we store your state for 30 days. You can turn it off, uh, but yeah, it's, it's free. Interesting thing on state is that it just, like, makes... Particularly for me, it makes, like, debugging things and building things so much simpler, where I can, like, create a responses object that's, like, pretty complicated and part of this more complex, uh, application that I've built. And I can just go into my dashboard and see exactly what happened. Did I mess up my prompt? Did it, like, not call one of these tools? Did I misconfigure one of the tools? Like, the, the visual observability of everything that you're doing is, is so, so helpful. So I'm excited, like, about people trying that out and, and getting benefits from it too. **Swyx** [7:20] Yeah. It's, uh, it, it's really, I think, a, a, a really nice to have. Uh, but I... All, all I'll say is that, uh, my friend Corey Quinn says that anything that can be used as a database will be used as a database, so- **Nikunj Handa** [7:31] Yeah **Swyx** [7:31] ... be prepared for some abuse. **Nikunj Handa** [7:35] All right. Uh, yeah, that's a good one. **Romain Huet** [7:38] We had some of that at Stripe with the metadata that some people were very, very creative at stuffing data- **Nikunj Handa** [7:42] Yes. **Romain Huet** [7:43] ... data in every type of object. **Swyx** [7:44] Yeah. **Nikunj Handa** [7:45] And we do have metadata with responses now- **Romain Huet** [7:47] Exactly. Yeah, yeah **Nikunj Handa** [7:47] ... so it will be... ### Web Search **Romain Huet** [7:49] Let's get through all of these. So web search. I think the... When I first said web search, I thought you were gonna just expose a API that then return kinda like a nice list of thing, but the way it's named is like GPT-4o Search Preview. **Nikunj Handa** [8:01] Mm-hmm. **Romain Huet** [8:02] So I'm guessing you have... You're using basically the same model that is in the ChatGPT search, which is fine-tuned for search, I'm guessing. It's a different model than the base one. And it's impressive, the jump in performance. So just to give an example, in simple QA, GPT-4o is 30% accuracy, 4o Search is 90%. We always talk about how tools are, like... Uh, models is not everything you need, like tools around it are just as important. So yeah, maybe give people a quick preview on like the work that went into making this special. **Nikunj Handa** [8:29] Should I take that? Yeah. **Romain Huet** [8:30] Go for it. **Nikunj Handa** [8:30] Um, so, uh, firstly, we're launching web search in two ways. One, in Responses API, which is our API for tools. It's gonna be available as a web search tool itself. So you'll be able to, like, go Tools, turn on Web Search, and you're ready to go. Uh, we still wanted to give chat completions people access to real-time information, so in that Chat Completions API, which does not support built-in tools, we're launching the direct access to the, the fine-tuned model that ChatGPT for Search uses, and we call it GPT-4o Search Preview. And how is this model built? Basically, we have... Our, our search research team has been working on this for a while. Their main goal is to, like, get information, like get a bunch of information from all of our, our data sources that we use to, to gather information for search, and then pick the right things and then cite them as accurately as possible, and that's what the search team has really focused on. They've done some, like, pretty cool stuff. They use, like, synthetic data techniques. They've done, like, O- 系列 model distillation to, like, make these 40 fine-tunes really good. But yeah, the main thing is, like, can it remain factual? Can it answer questions based on what it retrieves, and can it cite it accurately? And that's what this, like, fine-tune model really excels at. And so yeah, super excited that, like, it's gonna be directly available in chat completions along with being available as a tool. **Romain Huet** [9:49] Yeah, just to clarify, if I'm using the Responses API, this is a tool. **Nikunj Handa** [9:53] Yes. **Romain Huet** [9:53] But if I'm using Chat Completions, have to switch model. I cannot use o1 and call search as a tool. **Nikunj Handa** [9:58] Yeah. **Swyx** [9:58] That's right. **Nikunj Handa** [9:58] Exactly. **Swyx** [9:59] Yeah. **Romain Huet** [9:59] I think what's really compelling, at least for me and my own, um, uses of it so far, is that when you use, like, web search as a tool, it combines nicely with every other tool and every other feature of the platform. So think about this for a second, for instance. Imagine you have, like, a Responses API call with the web search tool, but suddenly you turn on function calling and you also turn on, let's say, structured outputs. **Nikunj Handa** [10:21] Mm. **Romain Huet** [10:21] Now you kinda have, like, the ability to structure any data from the web in real time in the JSON schema that you need for your application. So it's quite powerful when you start combining those features and tools together. It's kind of like an API for the internet almost, you know. Like, you, you, uh, you, you get, like, access to the precise schema you need for your app. **Nikunj Handa** [10:39] Yeah. **Romain Huet** [10:39] And then just to wrap up on the infrastructure side of it, I read on the post that people... Publisher can choose to appear in the web search. **Nikunj Handa** [10:47] Mm-hmm. **Romain Huet** [10:48] So are people by default in it? Like, how can we get laid in space in the, in the web search API? **Nikunj Handa** [10:54] Yeah, yeah. I, I, I think, uh, the... We, we have some documentation around how, uh, websites, publishers can, can control, like, what shows up in our web search tool, and I think, uh, you, you should be able to, like, read that. I think we should, we should be able to get laid in space in for sure. **Romain Huet** [11:10] Cool. **Swyx** [11:11] Yeah. You know, I, I think... So I compare this to a broader trend that I started covering last year of online LLMs. Uh, actually Perplexity, I think, was the first to say... to offer an API that is connected to search, and then Gemini had the sort of search grounding API. And yeah, I think you guys... I actually didn't... I missed this in the original reading of the docs, but you even give, like, citations with, like, the exact- **Nikunj Handa** [11:33] Yeah **Swyx** [11:33] ... uh, sub- subparagraph that is matching, which I, which I think is the standard nowadays. I think my question is: how do we take what a knowledge cutoff is for something like this, right? Because, like, n- now basically there's no knowledge cutoff, it is always live, but then there's a difference between what the model has sort of internalized in its backpropagation and what is searching up as RAG. **Romain Huet** [11:54] I think it kinda depends on the use case, right, and, and what you want to, uh, to showcase as the source. Like, for instance, you take a company like Hebiya that has used this, like, web search tool. They can combine, like, for, for credit firms or law firm, they can find, like, you know, public information from the internet with the, the live sources and citation that sometimes you do wanna have access to, as opposed to, like, the internal knowledge. But if you're building something different where, like, you just wanna have an assistant that relies on the deep knowledge that the model has, you may not need to have these, like, direct citations. So I think it kinda depends on the use case a little bit, but there are many, uh, many companies like Hebiya that will need that access to these citations to precisely know where the information comes from. **Swyx** [12:35] Yeah, yeah, uh, for sure. And then one thing on the, on, like, the breadth, you know, I think a, a lot of the deep research, open deep research implementations have this sort of par- hyperparameter about, you know, how deep they're searching and how wide they're searching. I don't see that in the docs, but is that something that we can tune? Is that something you recommend thinking about? **Nikunj Handa** [12:54] Super interesting. It, it's definitely not a parameter today, but, but we should explore that. It's, it's very interesting. I imagine, like, how you would do it with the Web Search tool and Responses API is you would have some form of, like, you know, agent orchestration over here where you have a planning step, and then each, like, web search- **Swyx** [13:10] Yeah **Nikunj Handa** [13:11] ... call that you do, like, explicitly goes a layer deeper and deeper and deeper. But it's not a parameter that's available out of the box. **Swyx** [13:18] But it's a cool, it's a cool thing to think about. **Nikunj Handa** [13:20] Yeah. **Swyx** [13:20] The only guidance I'll offer there is a lot of these implementations offer top K, which is like, you know, top ten, top 20. But actually, you don't really want that. You want, like, sort of some kind of similarity cutoff, right? Like, some matching- **Nikunj Handa** [13:32] Mm **Swyx** [13:33] ... uh, score cut- cutoff. Because if there's only five things, five documents that match, fine. If there's 500 that match, maybe that's what I want. **Nikunj Handa** [13:40] Right. **Swyx** [13:40] Yeah. **Nikunj Handa** [13:41] Yeah. **Swyx** [13:41] But also that might, that might make my cost very unpredictable, 'cause the costs are something like $30 per 1,000 queries, right? So- **Nikunj Handa** [13:48] Yeah, yeah. Yeah. Yeah, I guess you could, you could have some form of, like, a context budget, and then you, like, go as deep as you can and pick the best stuff and put it into, like, X number of tokens. There could be some creative ways of, of managing cost. But yeah, that's a super interesting thing to explore. **Romain Huet** [14:06] Do you see people, uh, using the Files and the Search API together, where you can kind of search and then store everything in the file so that next time I'm not paying for the search again? And like, yeah, how should people balance that? **Nikunj Handa** [14:18] That's actually a very interesting question. Let, let me first tell you about how I've seen... A really cool way I've seen people use Files and Search together is they put their user preferences or memories in the vector store. And so a query comes in, you use the File Search tool to, like, get someone's, like, reading preferences or, like, fashion preferences and stuff like that, and then you search the web for information or products that they can buy related to those preferences, and you then render something beautiful to show them, like, "Here are five things that you might be interested in." So that's how I've seen, like, File Search, Web Search work together. **Swyx** [14:52] Yeah. **Nikunj Handa** [14:52] And by the way, that's, like, a single Responses API call, which is really cool. So you just, like, configure these things, go boom, and, like, everything just happens. But yeah, that's how I've seen, like, Files and Web work together. **Romain Huet** [15:02] But I think that what you're pointing out is, like, interesting, and I'm sure, uh, developers will surprise us, as they always do, in terms of how they combine these tools and how they might use File Search as a way to have memory and preferences like Nikunj says. **Nikunj Handa** [15:13] Mm. **Romain Huet** [15:14] But I think, like, zooming out, what I find very compelling and powerful here is, like, when you have these, like, neural nets that have, like, all of the knowledge that they have today, plus real-time access to the internet for, like, any kind of real-time information that you might need for your app, and File Search, where you can have a lot of company private documents, private details, you combine those three, and you have, like, very, very compelling and precise answers for any kind of use case that your company or, or your product might wanna enable. **Swyx** [15:42] It's the difference between sort of internal documents versus the, the open web, right? Like, you're gonna need both. Um- **Romain Huet** [15:47] Exactly. Exactly. **Swyx** [15:49] I never thought about it doing memory as well. I guess, again, you know, anything that's a database, you can store it in the- ### File Search **Nikunj Handa** [15:55] Yeah **Swyx** [15:56] ... we'll use it as a database. That sounds awesome. But I think also you've been, you know, uh, expanding the File Search. You have more file types. You have query optimization, custom re-ranking. So it really seems like, you know, it's been fleshed out. Obviously, I haven't been paying a ton of attention to, to the File Search capability, but it s- it sounds like your, your team has added a lot of features. **Nikunj Handa** [16:14] Yeah. Metadata filtering was, like, the main thing people were asking us for for a while, and that's the one I'm super excited about. I mean, it's just so critical once your, like, vector store size goes over, you know, more than, like, you know, five, ten thousand records, you, you kinda need that. So yeah, metadata filtering is coming too. **Romain Huet** [16:32] Yeah, for most companies it's also not, like, a, a competency that you wanna rebuild in-house necessarily. You know, like, you know, uh, thinking about embeddings and chunking and, you know, how all of that... Like, it sounds, like, very complex for something very, uh, like, um, obvious to ship for your users. Like, companies like Navan, for instance, they were able to build with the File Search, like, you know, take all of the FAQ and travel policies, for instance, that you have, you, you put that in File Search tool, and then you don't have to think about anything. Now your assistant becomes naturally, uh- **Swyx** [17:00] Mm **Romain Huet** [17:00] ... much more aware of all of these policies from the files. **Swyx** [17:04] The question is, like, there's a very, very vibrant RAG industry already, as you well know. So there's many other vector databases, many other frameworks. Probably if it's an open source stack... I, I, I'll, I'll say, like, the... a lot of the AI engineers that I talk to want to own this part of the stack. **Romain Huet** [17:18] Mm. **Swyx** [17:18] And it feels like, you know, like, when should we DIY, and when should we just use whatever OpenAI offers? **Nikunj Handa** [17:25] Yeah. I mean, like, if you're doing com- something completely from scratch, you're gonna have more control, right? Like, so super supportive of, uh, you know, people trying to, like, roll up their sleeves, build their, like, super custom chunking strategy and super custom retrieval strategy and all of that. And those are things that, like, will be harder to do with OpenAI's tools. OpenAI tool has, like... We have an out-of-the-box solution. We give you some knobs to customize things, but it's more of, like, a managed RAG service. **Swyx** [17:53] Yeah. **Nikunj Handa** [17:53] So my recommendation would be, like, start with the OpenAI thing. See if it, like, meets your needs, and over time we're gonna be adding more and more knobs to, to make it even more customizable. But, you know, if you want, like, the completely custom thing, you want control over- **Swyx** [18:07] Yeah **Nikunj Handa** [18:07] ... every single thing, then you'd probably wanna go and, and hand roll it using other solutions. So we're supportive of both. Like- **Swyx** [18:14] Yeah **Nikunj Handa** [18:15] ... uh, engineers should, should pick. Yeah. **Romain Huet** [18:17] And then we got Computer Use, which I think Operator was obviously, uh- ### Computer Use **Swyx** [18:20] Yeah **Romain Huet** [18:20] ... one of the hot releases of the year, and we're only two months in. Let's talk about that, and that's also, it seems, like, a separate model that has been fine-tuned for Operator that has browser access. **Nikunj Handa** [18:32] Yeah, absolutely. I mean, the computer use model's, uh, exciting. They... I-I-- The cool thing about computer use is that we're just so, so early. It's like- **Romain Huet** [18:40] Yeah **Nikunj Handa** [18:40] ... the GPT-2 of computer use or maybe GPT-1 of computer use right now. But, uh, it is, it is a separate model that has been, you know, the, the computer use team has been working on. Uh, you send it screenshots, and it tells you what action to take. So the, the outputs of it are almost always tool calls and, uh, you're, you're inputting screenshots based on whatever computer you're trying to operate. **Romain Huet** [19:02] Maybe zooming out for a second, 'cause, like, I'm sure your audience is, like, super, super, like, AI native, obviously, but, like, what is computer use as a tool, right? And, and what's Operator? So the idea for computer use is, like, how do we let developers also build agents that can complete tasks for the users but using a computer or a browser instead? And so how do you get that done? And so that's why we have this custom model- **Nikunj Handa** [19:23] Mm-hmm **Romain Huet** [19:23] ... like optimized for computer use that we use, like, for Operator ourselves. But the idea behind, like, putting it as an API is that imagine, like, now you wanna, you wanna automate some tasks for your product or your, your own customers, then now you can, you can have, like, the ability to spin up one of these agents that will look at the screen and act on the screen. So that means able-- the ability to click, the ability to scroll, the ability to type, and to report back on the action. So that's what we mean by computer use and, and wrapping it as a tool also in the Responses API. So now, like, that gives a hint also at the multi-turn thing that we were hinting at earlier. The idea that, like, yeah, maybe one of these actions can take a couple minutes to complete because there's maybe, like, uh, 20 steps to complete that task, but now you can. **Swyx** [20:09] Do you think, uh, computer use can play Pokémon? **Nikunj Handa** [20:12] That'll be interesting. **Romain Huet** [20:13] I guess. We should- **Swyx** [20:14] I don't think we've tried it. **Romain Huet** [20:15] I guess we should try it, you know. **Nikunj Handa** [20:18] Yeah. Yeah. **Swyx** [20:18] There's a lot of interest. I think Pokémon really is a good agent benchmark, to be honest. Like, uh- **Nikunj Handa** [20:22] Yeah **Swyx** [20:22] ... it seems like Claude is, Claude is running into a lot of trouble. Um- **Romain Huet** [20:26] Sounds like we, we should make that a new eval, it looks like. **Swyx** [20:28] Uh, yeah. Yeah, yeah. Uh... Oh, and then one more, one more thing before we move on to Agents SDK. I know you have a hard stop. There's obviously, you know, blah blah dash preview, right? Like search preview, computer use preview, right? It-- And they seem all like fine tunes of 4.0. I think the question is, are we-- are they all gonna be merged into the main branch, or are we basically always gonna have subsets of these models? **Nikunj Handa** [20:50] Yeah, I think in the early days, research teams at OpenAI, like, operate with, like, fine-tuned models, and then once a thing gets, like, uh, more stable, we, we sort of merge it into the main line. So that's definitely the vision, like, going out of preview as we get more comfortable with, uh, and, and learn about all the developer use cases, and we're doing a good job at them. We'll sort of like make them part of, like, the core models so that you don't- **Romain Huet** [21:11] Yeah **Nikunj Handa** [21:11] ... have to, like, deal with the bifurcation. **Romain Huet** [21:13] You should think of it this way as exactly what happened last year when we introduced Vision capabilities. You know- **Nikunj Handa** [21:17] Yes **Romain Huet** [21:17] ... Vision capabilities were in like a Vision preview model based off of GPT-4, and then Vision capabilities now are, like, obviously built into GPT-4.0. **Nikunj Handa** [21:27] Mm-hmm. **Romain Huet** [21:27] You can think about it the same way for, like, the other modalities like audio and those kind of like models, like, uh, optimized for search and computer use. ### Agents SDK **Swyx** [21:34] Uh, Agents SDK, we have, uh, a few minutes left. So, uh, uh, let's just assume that everyone has looked at Swarm. **Nikunj Handa** [21:40] Sure. **Swyx** [21:40] Uh, I think that Swarm has really popularized the handoff technique, which, uh, I, I thought was like, you know, really, really interesting for, um, sort of a multi-agent world. What is new with the SDK? **Nikunj Handa** [21:51] Yeah. **Romain Huet** [21:51] Do you wanna start? **Nikunj Handa** [21:51] Yeah, for sure. Uh, so we've, uh, basically added support for types. Uh, we've, we've made this, uh, like, a lot... Yeah, like we've added support for types. We've added support for guardrailing, uh, which is a very common pattern. So in, in the guardrail example, you basically have two things happen in parallel. The guardrail can sort of block the execution. It's, it's a type of like optimistic generation that happens, and, uh, I think we've added support for tracing. So you can basically look at the traces that the Agents SDK creates in the OpenAI dashboard. We also, like, made this pretty flexible, so you can pick any API from any provider that supports the chat completions API format. So it supports responses by default, but you can, like, easily plug it into anyone that uses the Chat Completions API. And similarly, on the tracing side, you can support, like, multiple tracing providers. By default, it sort of points to the OpenAI dashboard, but, uh, you know, there's like so many tracing companies out there and, and we'll, we'll announce some partnerships on that front too. So just like, you know, adding lots of core features and making it more usable, but still centered around, like, handoffs as, like, the main, main concept. **Romain Huet** [23:00] And by the way, it's interesting, right? Because Swarm just came to life out of, like, learning from customers directly that, like, orchestrating agents in production was pretty hard. You know, simple ideas could quickly turn very complex. Like what are those guardrails? What are those handoffs, et cetera. So that came out of, like, learning from customers and was initially shipped as a, like, low-key experiment, I'd say. But we were kind of like taken by surprise at how much momentum there was around this concept, and so we decided to learn from that and embrace it, to be like, "Okay, maybe we should just embrace that as a core primitive of the OpenAI platform." And that's kind of what led to the Agents SDK. And I think now, as Nikunj mentioned, it's like adding all of these new capabilities to it, like leveraging the handoffs that we had, but tracing also. And I think what's very compelling for developers is, like, instead of having one agent to rule them all and you stuff like a lot of tool calls in there, that can be hard to monitor. Now you have the tools you need to kind of like separate the logic, right? And you can have a triage agent that, based on an intent, goes to different kind of agents. And then on the OpenAI dashboard, we're releasing a lot of, uh, new user interface logs as well, so you can see all of the tracing UIs. Essentially, you'll be able to troubleshoot, like, what exactly happened in that workflow when the triage agent did a handoff to a secondary agent and a third, and see the tool calls, et cetera. So we think that the Agents SDK combined with the tracing UIs will definitely help users and developers build better agentic workflows. And just before we wrap, are you thinking of connecting this with also the RFT, uh, API? **Nikunj Handa** [24:34] Yeah. **Romain Huet** [24:34] Uh, because I know you already have... You kind of store my text completions, and then I can do fine-tuning of that. Is that gonna be similar for agents, where you're storing kind of like my traces and then help me improve the agents? **Nikunj Handa** [24:44] Yeah, absolutely. Like, you gotta tie the traces to the Evals product so that you can generate good evals. Once you have good evals and graders and tasks, you can use that to do reinforcement fine-tuning and, uh, you know, lots of details to be figured out over here, but that's the vision, and I think we're gonna go after it, like, pretty hard and, and hope we can, like, make this whole workflow a lot easier for developers. ### Outro **Romain Huet** [25:05] Awesome. Thank you so much for the time. I'm sure you'll be busy on Twitter tomorrow with all the, um, developer feedback, so... Yeah, thank you so much for having us, and, uh, as always, we can't wait to see, uh, what developers will build with these tools and, and how we can, like, learn as quickly as we can from them to, to make them even better over time. **Nikunj Handa** [25:22] Yeah. **Romain Huet** [25:22] Awesome. Thank you, guys. **Nikunj Handa** [25:23] Thank you. **Swyx** [25:24] Thank you both. **Nikunj Handa** [25:25] Awesome. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.