# Building AGI with OpenAI's Structured Outputs API Latent Space ยท 2024-09-17 Michelle, OpenAI's API tech lead, explains how the newly launched Structured Outputs feature achieves near-100% reliability for schema-following outputs by combining constrained decoding with model training, moving beyond the limitations of JSON mode and Function Calling. She details the refusal field that allows the model to decline harmful requests while maintaining parseable output, and discusses use cases like dynamic UI generation and mathematical chain-of-thought. The roadmap includes parallel function calling and custom grammars. The episode also covers the GA of GPT-4o fine-tuning, Batch API's 50% cost savings, Vision API integration with structured outputs, the Whisper API's surprising lack of diarization, and the shift toward speech-to-speech APIs. Michelle shares insights from her Waterloo co-op background and what qualities succeed at OpenAI: low ego, user-focused, and driven. ## Questions this episode answers ### How does OpenAI's Structured Outputs API work, and how is it different from JSON mode? Michelle explains that Structured Outputs combines constrained decoding with a specially trained model to guarantee outputs exactly match a given JSON schema. Unlike JSON mode, which only ensures valid JSON but can produce incorrect keys or whitespace, Structured Outputs returns a fully parsed object matching the developer's Pydantic or Zod type, eliminating parsing errors and reducing retries. It's designed for developers who need reliable, exact outputs to integrate into their systems. [11:52](https://addtry.com/a25e027b-ff18-4567-812f-4d04375f8773?t=712000) ### How does the refusal field in OpenAI's Structured Outputs handle safety and prevent errors? Michelle describes that to preserve the model's ability to refuse harmful requests under a strict schema, they added a refusal field. If the model refuses, the content field may be empty, but the refusal string appears, allowing developers to programmatically handle refusals without parsing errors. This avoids breaking the schema and saves costs since refusals aren't billed as errors, providing a cleaner developer experience. [18:13](https://addtry.com/a25e027b-ff18-4567-812f-4d04375f8773?t=1093000) ### What is the difference between function calling and the new structured outputs response format in the OpenAI API? Michelle clarifies that function calling is for when the model needs to invoke actual tools, like querying a database, and should be used when you have functions to call. The new structured outputs response format is for structuring the model's reply to the user, ensuring it matches a schema without triggering actions. It avoids hacking function calling for output formatting, yielding more natural responses and better reliability. [13:21](https://addtry.com/a25e027b-ff18-4567-812f-4d04375f8773?t=801000) ### What's the difference between gpt-4o and chatgpt-4-latest in the OpenAI API? Michelle explains that gpt-4o (API model) is tuned for developer needs like function calling and structured outputs, with pinned weights for stability, while chatgpt-4-latest is the rolling model behind ChatGPT, better for chat-style interactions but without pinned weights. Developers can choose based on whether they need stable, tool-focused output or the latest chat-optimized model. [44:47](https://addtry.com/a25e027b-ff18-4567-812f-4d04375f8773?t=2687000) ## Key moments - **[0:00] Intro** - **[6:37] Early Days** - [6:49] Michelle joined OpenAI pre-ChatGPT, impressed by GitHub Copilot's quality and daily usage. - [9:20] ChatGPT's launch caused scaling issues because accounts were tied to the API and GPU capacity was constrained. - **[9:58] Structured Outputs** - [10:09] JSON mode constrained output to JSON but didn't enforce specific schemas, causing white space or wrong types. - [11:22] Structured Outputs combines token masking and fine-tuning to ensure outputs match a given schema. - [13:21] Q: Should I use Function Calling or Structured Outputs? A: Use Function Calling for actual tool use, Structured Outputs for formatting responses. - [16:57] OpenAI may support custom grammars beyond JSON schema in the future. - **[17:08] API Design** - [17:26] The API models team at OpenAI trains models specifically for API use cases like function calling. - [18:18] Q: Why does Structured Outputs have a Refusal field instead of an HTTP error? A: Because HTTP error codes can't capture model behaviors well, and developers pay for tokens. - [19:54] "There are things that are in between your fault and my fault... the model's fault, and there's no error code for that." - **[21:23] Function Calling** - [25:17] Q: Why no parallel function calling with Structured Outputs? A: Because building the combined schema index would add latency, and they want a no-added-latency solution. - [26:25] Structured Outputs first-request latency will decrease over time as they optimize the indexing. - **[27:42] Agentic Apps** - [27:43] Structured Outputs makes agentic applications viable by eliminating parsing errors, turning 95% reliability into 100%. - [30:52] Recursive schemas enable dynamic UI generation, treating UIs as nested trees that structured outputs can construct. - [33:52] You can add a chain-of-thought array in structured outputs to get step-by-step reasoning, like in a math tutor. - [34:51] OpenAI respects descriptions in JSON schema, effectively using them as prompts for each field. - **[38:36] Roadmap & Tuning** - [38:54] Parallel function calling with Structured Outputs is coming soon; custom grammars beyond JSON are under consideration. - [40:56] Michelle recommends starting with 4o mini for cost, moving to 4o if needed, and fine-tuning with as few as 100 examples for advanced use cases. - [41:32] GPT-4o fine-tuning is now generally available, with 1M free training tokens/day until September 23. - [42:58] Many developers stop early with a model without optimizing prompts or trying other available models, missing potential performance. - [44:48] Q: What's the difference between gpt-4o-2024-08-06 and chatgpt-4-latest? A: The API model is tuned for function calling and structured outputs, while the ChatGPT model is optimized for general chat. - [45:14] chatgpt-4-latest is a rolling model mirroring ChatGPT's internal model, whereas API models have pinned weights for developer stability. - **[48:13] War Stories** - [49:50] Two hours before Dev Day, the Assistants API had an outage, but the team fixed it just before the live demo. - [51:40] Assistants API file search now supports 10,000 files per assistant with semantic search and re-ranking controls. - [52:30] OpenAI will add re-ranking controls to Assistants API, aiming for a RAG solution that works out of the box. - [55:14] OpenAI has made API tiering transparent, with fine-tuning access starting at tier 2. - **[55:48] API Tour** - [58:31] Q: What is the Batch API? A: It's 50% cheaper than standard pricing, with 24-hour turnaround, enabling cheap, async processing for tasks like user email workflows. - [1:00:10] The Vision API is now generally available and works across all OpenAI APIs, enabling extraction from images with spatial relationships. - [1:02:06] Q: Why doesn't the Whisper API offer speaker diarization? A: Whisper V3 had performance issues, and adding it wasn't prioritized over other work. - [1:04:44] OpenAI is exploring a new API shape for speech-to-speech, moving beyond request-response, and aims to make it developer-friendly. - [1:05:28] OpenAI shipped admin and audit log APIs, improved SSO, and programmatic API key management for enterprise users. - **[1:06:09] Personal Insights** - [1:08:04] Michelle recommends 'The Making of Prince of Persia' for its raw journal of the grind to build iconic software. - [1:10:53] OpenAI looks for low ego, user-focused, driven engineers who are ready to roll up their sleeves and do what it takes. ## Speakers - **Alessio** (host) - **Swyx** (host) - **Michelle Pokrass** (guest) ## Topics Developer Tools ## Mentioned OpenAI (company), Assistants API (product), Batch API (product), Chat Completions API (product), ChatGPT (product), ChatML (product), Code Interpreter (product), DALL-E (product), File Search (product), Function Calling (product), GPT-3.5 (product), GPT-4o (product), GPT-4o mini (product), GitHub Copilot (product), JSON mode (product), Pydantic (product), Structured Outputs (product), Vision API (product), Whisper API (product), fine-tuning API (product) ## Transcript ### Intro **Alessio** [0:04] Hey, everyone. Welcome to the Latent Space Podcast. This is Alessio, partner and CCO in residence at Decibel Partners, and I'm joined by my co-host Swyx, founder of Smol AI. **Swyx** [0:14] Hey, and today we're excited to be in the in-person studio with Michelle. Welcome. **Michelle Pokrass** [0:18] Thanks. Thanks for having me. Very excited to be here. **Swyx** [0:20] This has been a long time coming. Uh, I've been following your work on the API Platform for a little bit, and, uh, I'm finally glad that we could make this happen after you, you shipped, uh, Structured Outputs. How does that feel? **Michelle Pokrass** [0:31] Yeah, it feels great. Uh, we've been working on it for quite a while. **Swyx** [0:34] Yeah. **Michelle Pokrass** [0:34] So very excited to have it out there and have people using it. **Swyx** [0:37] We'll tell the story soon, uh, but I wanna give people a little intro to your background. So you've interned and-or worked at Google, Stripe, Coinbase, Clubhouse, and obviously OpenAI. What was that journey like? Uh, you know, I, I... The one that has the most appeal to me is Clubhouse because that was a, a very, very hot company for a while. You- Basically, you seem to join companies when they're about to scale up really a lot, and, uh, obviously OpenAI has been the latest. But, uh, yeah, just what, what are your learnings and your history going into all these c- um, notable companies? **Michelle Pokrass** [1:07] Yeah, totally. For a bit of my background, uh, I'm Canadian. I went to the University of Waterloo, and there you do, like, six internships as part of your degree. So I started... Uh, actually, my first job was really rough. I worked at a bank, and I learned Visual Basic, and I, like, animated bond yield curves and it was, you know, not, not- **Swyx** [1:23] Me too. **Michelle Pokrass** [1:24] Oh, really? **Swyx** [1:25] Yeah. I was a derivatives trader. **Michelle Pokrass** [1:26] Oh, okay. **Swyx** [1:26] Interest rate swaps- **Michelle Pokrass** [1:27] Awesome **Swyx** [1:27] ... that kind of stuff. Yeah. **Michelle Pokrass** [1:28] Yeah. So I liked, you know, having a job, but I didn't love that job. Uh, and then my next internship was Google, uh, and I learned so much there. Uh, it was tremendous. But I had a bunch of friends that were into startups more and, you know, Waterloo has, like, a big startup culture. And one of my friends, uh, interned at Stripe, and he said it was super cool, so that was kind of my... I also was a little bit into crypto at the time, then I got into it on Hacker News, and, uh, so Coinbase was on my radar. And so that was, like, my first real startup opportunity, was Coinbase. I think I've never learned, uh, more in my life than in the four-month period when I was interning at Coinbase. They actually put me on call. I worked on, like, the ACH rails there, and it was, it was absolutely crazy. You know, crypto was a very formative experience. Yeah. **Swyx** [2:08] And this is, uh, 2018 to 2020, kind of like the first big wave. **Michelle Pokrass** [2:12] That was my full time. **Swyx** [2:13] Yeah. **Michelle Pokrass** [2:13] Uh, but I was there as an intern in 20- **Swyx** [2:16] 16. **Michelle Pokrass** [2:16] 16. Yeah. And so that was the period where I really, like, learned to become an engineer, learned how to use Git, got on call right away. You know, managed production databases and stuff, so that was super cool. After that, I went to Stripe and kind of got a different flavor of payments on the other side. Learned a lot. Uh, was really inspired by the Collisons. And then my next internship after that, I actually started a company at Waterloo. So there's this thing you can do, it's an entrepreneurship co-op, uh, and I did it with my roommate. Uh, the company's called Readwise, which still exists, but- **Alessio** [2:43] Oh, yeah, yeah. I use it. **Swyx** [2:44] Oh, I thought you said Rewise. What? **Michelle Pokrass** [2:45] Yeah. Awesome. **Swyx** [2:45] You, you co-founded Rewise? **Michelle Pokrass** [2:47] Yeah. Um. **Alessio** [2:47] I love premium music. **Swyx** [2:49] Well, it's not even on your, your, your LinkedIn. **Michelle Pokrass** [2:51] Yeah, I mean, I only worked on it for about a year, and so Tristan and Dan are the, the real founders, and, and I just had an interlude there. But, uh, yeah, really loved working on something very startup focused, user focused, and, and hacking with friends. It was super fun. Eventually, I decided to go back to Coinbase and really, like, get a lot better as an engineer. I didn't feel like I was... You know, didn't feel equipped to be a CTO of anything at that point, and so just learned so much at Coinbase, and that was a really fun curve. But yeah, after that I went to Clubhouse, which was, like, a really interesting, uh, time. So I wouldn't say that I went there before it blew up. I would say I went there as it blew up, so, uh, not quite the starling track record that it might seem. But it was a super exciting place. I joined as, like, the second or third backend engineer and, you know, we were down every day, basically. You know, one time Oprah came on, and a- absolutely everything melted down. And so we would have a standup every morning and be like, "How do we make everything stay up?" Um, which was super exciting. Also, one of the first things I worked on there was making our notifications go out more quickly because when you join a Clubhouse room, you know, you need everyone to come in right away so that it's exciting and the person speaking thinks a lot of my audience is here. But when I first joined, I think it would take, like, 10 minutes for all the notifications to go out, which is insane. Like, you know, by the time you want to start talking to the time your audience is there, it's, like, you can totally kill the room. So that's one of the first things I worked on, is making that a lot faster and, you know, keeping everything up. **Swyx** [4:12] I mean, so already we have an audience of engineers. Uh, those two things are useful. It's keeping things up and notifications out. Notifications, like, is it a Kafka topic? **Michelle Pokrass** [4:20] It was a Postgres shop, and you had all of the followers in Postgres, and you needed to, like, iterate over the followers and, like, figure out is this a good notification- **Swyx** [4:27] Ah **Michelle Pokrass** [4:27] ... to send. **Swyx** [4:28] Okay. **Michelle Pokrass** [4:28] And so all of this logic, it wasn't, like, well batched and parallelized, and our job queuing infrastructure wasn't right, and so there was a lot of, like, fixing all of these things. Um, eventually there were a lot of database migrations because Postgres just wasn't scaling well for us. **Swyx** [4:41] Interesting. And then thing... Uh, keeping things up, that was more of a, I don't know, reliability issue, SRE type? **Michelle Pokrass** [4:47] A lot of it, yeah, just goes down to, like, database stuff. Um, everywhere I've worked- **Swyx** [4:51] It's all databases. **Alessio** [4:53] Yeah. **Michelle Pokrass** [4:54] Yeah. **Swyx** [4:54] Indexing. **Michelle Pokrass** [4:55] Actually, at, at Coinbase, at Clubhouse, and at OpenAI, Postgres has been a perennial challenge. **Swyx** [5:00] Huh. **Michelle Pokrass** [5:00] It's like the stuff you learn at one job carries over to all the others because you're always debugging a long-running Postgres query at 3:00 AM for some reason. Um, so those skills have really carried me forward, for sure. **Alessio** [5:12] Why do you think that not as much of this is practiced? Uh- **Michelle Pokrass** [5:15] Mm. **Alessio** [5:15] Obviously Postgres as an open source project is not aimed at, like, giga scale- **Michelle Pokrass** [5:19] Right **Alessio** [5:19] ... but you would think somebody would come around and say, "Hey, we're, like, the..." **Michelle Pokrass** [5:22] Yeah. I think that's what PlanetScale is doing, kind of. It's not on Postgres, I think. It's on MySQL. But I think that's the vision. It's like they have zero time, zero downtime, uh, migrations, and that's a big pain point. I don't know why no one is doing this on Postgres, but I think it would be very cool. **Swyx** [5:37] Their connection poolers like PgBouncer is, like, good enough? I don't know. **Michelle Pokrass** [5:41] Yeah. Well, even, I mean, I've run PgBouncer everywhere, and it's, there's still a lot of problems. Like- **Swyx** [5:46] I mean, your scale- **Alessio** [5:46] Yeah. **Swyx** [5:47] ... is something that not many people see, so. **Michelle Pokrass** [5:49] Yeah. I mean, at some point, every company hits the scale, every successful company gets to the scale where Postgres is not cutting it, and then you migrate to some sort of NoSQL database. **Swyx** [5:56] Yeah. **Michelle Pokrass** [5:57] And, and that process I've seen happen a bunch of times now. **Swyx** [5:59] MongoDB, Redis, something like that. Yeah. **Michelle Pokrass** [6:02] Yeah. Um, I mean, we're on Azure now, and so there's... And we use Cosmos DB. **Swyx** [6:06] Cosmos DB. Hey. **Michelle Pokrass** [6:08] At Clubhouse, we, I really love DynamoDB. That's probably my favorite database, which is, like, a very nerdy sentence, but that's the one I'm using if I need to scale something as far as it goes. **Swyx** [6:17] Yeah. DynamoDB, I, I, um, when I learned, I worked at AWS briefly, and it's kind of like the memory register for the web. Like- **Michelle Pokrass** [6:24] Yes **Swyx** [6:24] ... it, you know, if you treat it just as physical memory, you will use it well. If you treat it as a real database, you might run into problems. **Michelle Pokrass** [6:31] Right. You have to totally change your mindset- **Swyx** [6:33] Yeah **Michelle Pokrass** [6:33] ... when you're going from Postgres to Dynamo. But I think it's a good mindset shift and kind of makes you design things in a more scalable way. ### Early Days **Swyx** [6:38] Yeah. I'll recommend a DynamoDB book for people who need to use DynamoDB. But we're not here to talk about AWS. We're here to talk about OpenAI. You joined OpenAI pre-ChatGPT. I also had the opportunity to join, and I didn't. **Michelle Pokrass** [6:49] Oh, no way. **Swyx** [6:49] What was, what was your insight? **Michelle Pokrass** [6:51] Yeah. I think a lot of people who joined OpenAI joined because of a product that really gets them excited, and for most people it's ChatGPT. But for me, I was a daily user of Copilot, GitHub Copilot, and I was, like, so blown away at the quality of this thing. I actually remember the first time seeing it on Hacker News and being like, "Wow, this is absolutely crazy." Like, "This is gonna change everything." And I started using it every day. It just really... I, even now, when, like, I'm, I don't have service and, and I'm coding without Copilot, it's just, like, 10X difference. So I was really excited about that product. I thought now is maybe the time for AI, and I'd done some AI in college and thought some of those skill, skills would transfer. Um, and I got introduced to the team. I liked everyone I talked to, so I thought, thought it'd be cool. Why didn't you join? **Alessio** [7:31] It was like, I was like, "Is DALL-E it?" **Michelle Pokrass** [7:34] Mm. **Swyx** [7:34] Yeah, we were there. **Alessio** [7:34] We were at the DALL-E **Swyx** [7:35] We, we were at the DALL-E, like, launch thing. And, and I think you were talking with Lenny, and, uh- **Alessio** [7:40] Yeah **Swyx** [7:40] ... Lenny was at OpenAI at the time, and you were like, "I don't-" **Alessio** [7:42] We don't have to go into too much detail, but- **Swyx** [7:44] Yeah, no, no, but it was- **Alessio** [7:44] ... this is one of my biggest regrets of my life, I think. **Swyx** [7:46] No, no, no. **Alessio** [7:47] Um, but, but, well, I was like, "Okay. I mean, I can, I can create images. I don't know if, like, this is the thing to, to dedicate." But obviously you had a bigger vision than, than I did. Um, but- **Michelle Pokrass** [7:55] DALL-E was really cool too. I remember, like, first showing my family, I was like, "I'm going to this company, and here's, like, one of the things they do." And it, like, really helped bridge the gap, whereas, like, I still haven't figured out how to explain to my parents what crypto is. Um, my mom for a while thought I worked at Bitcoin. So it's like- ... it's pretty different to be able to tell your family what you actually do, and they can see it. **Swyx** [8:15] Yeah. Yeah. And they can use it too- **Michelle Pokrass** [8:16] Yes **Swyx** [8:16] ... personally. So you were there w- were you immediately on API Platform? You were there for the ChatGPT moment. **Michelle Pokrass** [8:22] Yeah. I mean, API Platform is, like, a very grandiose term for what it was. There was, like, just a handful of us working on the API. **Swyx** [8:28] Yeah, it was like a closed beta, right? Not even everyone had access to- **Michelle Pokrass** [8:31] Yeah, there was a- **Swyx** [8:31] ... the GPT-3 model **Michelle Pokrass** [8:32] ... a very different access model then. Um, a lot more, like, tiered rollouts. **Swyx** [8:37] Yeah. **Michelle Pokrass** [8:37] But yeah, I, I would say the applied team was maybe, like, 30 or 40 people and... Yeah, probably closer to 30, and there was maybe, like, five-ish total working on the API at most. So yeah, we've grown a lot since then. **Swyx** [8:48] It's like 60, 70 now, right? **Michelle Pokrass** [8:49] No, Applied is much bigger than that. Applied now is bigger than the company when I joined. **Swyx** [8:53] Okay. All right. **Michelle Pokrass** [8:54] Yeah, we've grown a lot. I mean, there's so much to build, so we need all- **Swyx** [8:57] I'm, I'm a little out of date **Michelle Pokrass** [8:58] ... the help we can get. **Swyx** [8:58] Yeah. **Alessio** [8:59] Any ChatGPT release, kind of like all hands on deck stories? I had lunch with, um, Evan Morikawa a few months ago. It sounded like it was a fun time to get, build the APIs and have all these people trying to use the web thing. **Michelle Pokrass** [9:10] Yes. **Alessio** [9:10] Like, how are you prioritizing internally? Like, what was the helping scaling when you're scaling now GPU workloads versus, like, Postgres bouncers and things like that? **Michelle Pokrass** [9:20] Oh. Yeah, actually, surprisingly, there were a lot of Postgres issues, um- ... when ChatGPT came out because the accounts for, like, ChatGPT were tied to the accounts in the API. And so you're basically creating a developer account- **Alessio** [9:31] Mm **Michelle Pokrass** [9:31] ... to log into ChatGPT at the time- **Alessio** [9:33] Mm **Michelle Pokrass** [9:33] ... 'cause it's just what we had. It was low-key research preview. And so I remember there was just so much work scaling, like, our authorization system, and that would be down a lot. Yeah, also GPU, you know, I never had worked in a place where you couldn't just scale the thing up. It's like everywhere I've worked- **Alessio** [9:47] Yeah. **Michelle Pokrass** [9:47] ... compute is, like, free, and you just, like, auto-scale a thing- ... and you, like, never think about it again. But here we're having, like, tough decisions every day. We're, like, discussing, like, you know, should it go here or here? And we have to be principled about it. So that's a real mindset shift. ### Structured Outputs **Swyx** [9:59] So you just released Structured Outputs. Congrats. You also wrote the blog post for it, which was really well-written, and I loved all the examples that you put out. Like, it really give the, the full story. Yeah- **Michelle Pokrass** [10:06] Yeah **Swyx** [10:06] ... tell us about the whole story from beginning to end. **Michelle Pokrass** [10:09] Yeah. I guess the story, we should rewind, uh, quite a bit to Dev Day last year. Dev Day last year- **Swyx** [10:14] Mm-hmm **Michelle Pokrass** [10:14] ... exactly, we shipped JSON mode, which is our first foray into this area of product. So for folks who don't know, JSON mode is this functionality you can enable in our chat completions and other APIs, where if you opt in, uh, we'll kind of constrain the output of the model to match the JSON language. And so you basically will always get something in a curly brace, and this is good. This is nice for a lot of people. You can, like, describe your schema, what you want in prompt, and then, you know, we'll constrain it to JSON. But it's not getting you exactly where you want because you don't want the model to kind of make up the keys or, like, match different values than what you want. Like, if you want an enum or a number and you get a string instead, it's, like, pretty frustrating. So we've been ideating on this for a while, and, like, people have been asking for basically this every time I talk to customers for maybe the last year. So it was really clear that there's a developer need, and we started working on kind of making it happen, and this is a real collab between engineering and research, I would say. And so it's not enough to just kind of constrain the model. I think of that as the engineering side, whereas basically you mask the available tokens that are produced every time to only fit the schema. And so you can do this engineering thing, and you can force the model to do what you want, but you might not get good outputs. And sometimes with JSON mode, developers have seen that our models output, like, white space for a really long time where they don't match- **Swyx** [11:28] Because it's a legal character. **Michelle Pokrass** [11:29] Right. **Swyx** [11:30] Yeah. **Michelle Pokrass** [11:30] It's legal for JSON, but it's not really what they want. And so that's what happens when you do kind of a very engineering-biased approach, but the modeling approach is to also train the model to do more of what you want. And so we did these together. We trained a model which is significantly better than our past models at following formats, and we did the end work to serve, like, this constrained decoding concept at scale. So I think marrying these two is, is why this feature is pretty cool. **Alessio** [11:52] You just mentioned starts and ends with a curly brace, and maybe people's minds go to, uh, Prefills in the Claude API. How should people think about JSON mode structured output Prefills? Because some of them are like roughly starts with a curly brace and asks you for JSON, you should do it. And then instructor is like, "Hey, here's a rough data schema you should use." And how do you think about them? **Michelle Pokrass** [12:13] So I think we kind of designed structured outputs to be the easiest to use. So you just... Like, the way you use it in our SDK, I think, is my favorite thing. So you just create like a Pydantic object or a Zod object, and you pass it in, and you get back an object. And so you don't have to deal with any of the serialization on- **Alessio** [12:28] With the parse helper. **Michelle Pokrass** [12:29] Yeah. You don't have to deal with any of the serialization on the way in or out. So I, I kind of think of this as the feature for the developer who is like, "I need this to plug into my system. I need the function call to be exact. I don't want to deal with any parsing." So that's where structured outputs is tailored. Whereas if you want the model to be more creative and use it to come up with a JSON schema that you don't even know you want, then that's kind of where JSON mode fits in. But I expect most developers are probably gonna want to upgrade to structured outputs. **Swyx** [12:56] The thing you just said, you just used interchangeable terms for, for the same thing, which is two... Function Calling and Structured Outputs. We've had, uh, disagreements or discussion before on the podcast about are they the same thing. **Michelle Pokrass** [13:08] Mm. **Swyx** [13:08] Semantically, they're slightly different. **Michelle Pokrass** [13:10] They are, yes. **Swyx** [13:10] Because I think Function Calling API came out first- **Michelle Pokrass** [13:13] Yes **Swyx** [13:13] ... then JSON mode, and we used to abuse Function Calling for JSON mode. **Michelle Pokrass** [13:18] Right. **Swyx** [13:19] Do you think we should treat them as synonymous? **Michelle Pokrass** [13:21] No. **Swyx** [13:21] Okay. Yeah. **Michelle Pokrass** [13:22] Um- **Swyx** [13:22] Please clarify. **Michelle Pokrass** [13:24] Yeah. **Swyx** [13:24] And there, by the way, there's also tool calling. **Michelle Pokrass** [13:26] Yeah. The history here is we started with Function Calling, and Function Calling, you know, came from the idea of like, let's give the model access to tools, and let's see what it does. And we basically had these internal prototypes of, of what a Code Interpreter is now, and we were like, "This is super cool. Let's make it an API." But we're not ready to host Code Interpreter for everybody, so, you know, we're just gonna expose the raw capability and see what people do with it. But even now, I think there's a really big difference between Function Calling and Structured Outputs. So you, you should use Function Calling when you actually have functions that you w- want the model to call. **Swyx** [13:55] Right. **Michelle Pokrass** [13:56] And so, like, if you have a database that you want the model to be able to query from, or if you want the model to send an email or, like, you know, generate arguments for an actual action. And that's the way the model has been, like, fine-tuned on, is to, like, treat Function Calling for actually calling these tools and getting their outputs. The new response format is a way of just getting the model to respond to the user, but in a structured way. And so this is very different, like, responding to a user versus, like, you know, I'm gonna go send an email. A lot of people were hacking Function Calling to get the response format they needed, and so this is why we shipped kind of this new response format. So you can get exactly what you want, and you get kind of more of the model's verbosity. It's, like, kind of responding in the way it would speak to a user, and so less kind of just programmatic tool calling, if that makes sense. **Alessio** [14:42] Are you building something into the SDK to actually close the loop with the Function Calling? **Michelle Pokrass** [14:47] Yes. **Alessio** [14:47] Because right now it returns the function, then you gotta run it, then you gotta, like, fake another message to then- **Michelle Pokrass** [14:52] Yeah **Alessio** [14:52] ... continue the conversation like- **Swyx** [14:54] They have that in beta, the, the runs. **Michelle Pokrass** [14:55] Yes. We have this in beta in the Node SDK. So you can basically- **Swyx** [14:59] Mm **Michelle Pokrass** [15:00] ... define- **Swyx** [15:01] Oh, not Python. Oh. **Michelle Pokrass** [15:01] It's coming to Python as well. **Swyx** [15:02] Oh. **Alessio** [15:02] That's why I didn't know. **Swyx** [15:04] Yeah, I'm a N- Node guy, so the Python guys are- **Alessio** [15:06] The JavaScript, the JavaScript mind is so advanced **Swyx** [15:07] ... so I'm like, "This, this already existed." **Michelle Pokrass** [15:09] It, it's coming everywhere. But basically what you do is you write a function, and then you add a decorator to it, and then you can... Basically, there's this run tools method- **Swyx** [15:16] Yeah **Michelle Pokrass** [15:17] ... and it does the whole loop for you, which is pretty cool. **Swyx** [15:19] When I saw that in the Node SDK, I wasn't sure if that's... Because it, it basically runs it in the same machine. **Michelle Pokrass** [15:25] Yeah. **Swyx** [15:26] And maybe you don't want that- **Michelle Pokrass** [15:27] Right **Swyx** [15:27] ... to happen. **Michelle Pokrass** [15:28] Yeah, I think of it as, like, if you're prototyping and building something really quickly and just playing around, it's so cool- **Swyx** [15:34] Yeah **Michelle Pokrass** [15:34] ... to just create a function and give it this decorator. But, you know, you have the flexibility to do it however you like. **Swyx** [15:39] Like, you don't want it in a critical path of a web request because- **Michelle Pokrass** [15:41] I mean, some people definitely will. Um- **Swyx** [15:43] Really? **Michelle Pokrass** [15:44] You know, it's just kind of the easiest way to get started. **Swyx** [15:46] Yeah. **Michelle Pokrass** [15:46] But let's say you wanna, like, execute this function on a job queue async, then, you know, it wouldn't make sense- **Swyx** [15:51] Yes **Michelle Pokrass** [15:51] ... to use that. **Alessio** [15:52] Yeah. **Swyx** [15:52] Prior art in Structure Outlines, JSON former, what did you study? What did you, you know, credit or learn from these things? **Michelle Pokrass** [16:00] Yeah. There's a lot of different approaches to this. There's more fill-in-the-blank style sampling, where you, uh, basically pre-form kind of the keys and then get the model to sample just the value. There's kind of a lot of approaches here. We didn't kind of use any of them wholesale, but we really loved what we saw from the community and, like, the developer experiences we saw, so that's where we took a lot of, uh, inspiration. **Swyx** [16:22] There was a question also just about constrained grammar. This is something that I s- I first saw in Llama CPP, which seems to be the most, let's just say, academically permissive form- **Michelle Pokrass** [16:32] For the lowest level **Swyx** [16:33] ... of constr- Yeah. **Michelle Pokrass** [16:34] Yeah. **Swyx** [16:34] For those who don't know, maybe, I don't know if you want to explain it, but they use, uh, Backus-Naur Form, which you only learn in, like, college when you're working on- **Michelle Pokrass** [16:40] Yeah **Swyx** [16:40] ... programming languages and compilers. I don't know if you, like, use that under the hood or you explore that. **Michelle Pokrass** [16:45] Yeah, we, we didn't, uh, use any kind of other stuff. Um, we kind of built, you know, our solution from, from scratch to meet our specific needs. But I think there's a lot of cool stuff i- out there where you can supply your own grammar. Right now we only allow JSON schema and a dialect of that, but I think in the future it could be a really cool extension to let you supply a, a grammar more broadly, and maybe it's more token efficient than JSON. So a l- lot of opportunity there. ### API Design **Alessio** [17:09] You mentioned before also training the model to be better at Function Calling. What's that discussion like internally for, like, resources? It's like, "Hey, we need to get better JSON mode," and it's like, "Well, can you figure it out on the API platform without touching the model?" Like a- **Michelle Pokrass** [17:22] Yeah **Alessio** [17:23] ... is there a really tight collaboration between the two teams? **Michelle Pokrass** [17:26] Yeah, so I actually work on the API models team. I guess we didn't quite get into what I do in API. **Swyx** [17:29] Yeah. **Alessio** [17:31] Yeah. What do you say w- it is you do here? **Michelle Pokrass** [17:34] Yeah. So yeah, I'm the, I'm the tech lead for the API, but also I work on the API models team, and this team is really working on making the best models for the API. And a lot of common deployment patterns are research makes a model, and then you kind of ship it in the API. But- You know, I think there's a lot you miss when you do that. You miss a lot of developer feedback and things that are not kind of immediately obvious. What we do is we get a lot of feedback from developers, and we go and make the models better in certain ways. So our team does model training as well. We work c- very closely with our post-training team. And so for Structured Outputs, it was a collab between a bunch of teams, including safety systems, to make, you know, a really great model that does, uh, Structured Outputs. **Swyx** [18:13] Mentioning safety systems, you have a Refusal field. **Michelle Pokrass** [18:15] Yes. **Swyx** [18:16] Uh, you wanna talk about that? **Michelle Pokrass** [18:17] Yeah. **Swyx** [18:17] That, that one's like a- **Michelle Pokrass** [18:18] Yeah, it's a little... it's pretty interesting. So you can imagine, basically, if you constrain the model to follow a schema, you can imagine there being, like, a, a schema supplied that it wouldn't... it would add some risk or be harmful for the model to kind of follow that schema, and we wanted to preserve our model's abilities to refuse, uh, when something, you know, doesn't match our policies or is harmful in some way. And so we needed to give the model an ability to refuse even when there is this schema. But also, you know, if you are a developer and you have this schema and you get back something that doesn't match it, you're like, "Ah, the feature's broken." So we wanted a really clear way for developers to program against this. So if you get something back in the content, you know it's valid, it's JSON parsable. But if you get something back in the Refusal field, it makes for a much better UI for you to kind of display this to your user in a different way- **Swyx** [19:04] Yeah **Michelle Pokrass** [19:04] ... and it makes it easier to program against. So really there was a few goals, but it was mainly to allow the model to continue to refuse, but also with a really good developer experience. **Swyx** [19:11] Yeah. Why not offer it as, like, an error code? Because we have to display error codes anyway. **Michelle Pokrass** [19:17] Yeah, we fluffled for a long time about API design, as we are want to do, and there are a few reasons against an error code. Like, you could imagine this being a 4xx error code or something- **Swyx** [19:27] Mm **Michelle Pokrass** [19:28] ... but, you know, the developer's paying for the tokens, and that's kind of atypical for, like, a 4xx error code. **Swyx** [19:34] D- we pay with errors anyway, right? Or no? **Michelle Pokrass** [19:37] So 4xx's don't- **Swyx** [19:39] It's not. That's a, that's a U- that's a U error. **Michelle Pokrass** [19:40] Right. **Swyx** [19:41] Okay. **Michelle Pokrass** [19:41] And it, it doesn't make sense- **Swyx** [19:42] That's a malformed request **Michelle Pokrass** [19:42] ... as a 5xx either- **Swyx** [19:44] Yeah **Michelle Pokrass** [19:44] ... 'cause it's not our fault, you know? **Swyx** [19:45] It's not your fault. **Michelle Pokrass** [19:45] It's the way the API, the model is designed. And I think the HTTP spec is a little bit limiting for AI in a lot of ways. Like, there are things that are in between your fault and my fault. There's kind of, like- **Swyx** [19:55] Oh **Michelle Pokrass** [19:55] ... the model's fault, and there's no, you know, error code for that. So we really have to kind of invent a lot of the paradigm here. **Swyx** [20:02] Make a 6xx. **Michelle Pokrass** [20:04] Yeah, that's one option. There's actually some, like, esoteric error codes we've considered adopting. **Swyx** [20:08] 328, my favorite. **Michelle Pokrass** [20:10] Um, yeah, there's, uh, yeah, there's the teapot one. Um- **Swyx** [20:12] Hey. **Michelle Pokrass** [20:14] We're still figuring that out, but I think there are some things, like, for example, sometimes our model will produce tokens that are invalid based on kind of our language, and when that happens, it's an error. But, you know, it doesn't... 500 is fine, which is what we return, but it's not as expressive as it could be. **Swyx** [20:31] Mm-hmm. **Michelle Pokrass** [20:31] So yeah, just areas where, you know, Web 2.0 doesn't quite fit with AI yet. **Alessio** [20:37] If you had to put in a spec- **Swyx** [20:38] I was gonna **Alessio** [20:39] ... to this chain- **Swyx** [20:39] Yeah, yeah. **Alessio** [20:40] What would be your number one proposal to, like, re-haul in the- **Swyx** [20:43] The HTTP committee- **Alessio** [20:44] Yeah **Swyx** [20:45] ... to reinvent the world? **Michelle Pokrass** [20:47] Yeah, that's a good one. I mean, I think we just need an error of, like, a range of model error, and we can have many different kinds of model errors. Like, a refusal is a model error. **Alessio** [20:56] 601, model refusal. **Michelle Pokrass** [20:58] Yeah. Again, like, so we, we've mentioned before that chat completions uses this ChatML format, so when the model doesn't follow ChatML, that's an error. Um, and we're working on reducing those errors, but that's, like, I don't know, 602, I guess. **Swyx** [21:11] A lot of people actually don't no longer know what ChatML is- **Michelle Pokrass** [21:13] Yeah **Swyx** [21:13] ... because that- **Michelle Pokrass** [21:14] Fair enough **Swyx** [21:14] ... that was, uh, briefly introduced by OpenAI and then, like, kind of deprecated. Everyone who introdu- who implements this under the hood knows it, but maybe the, the API users don't know it. ### Function Calling **Michelle Pokrass** [21:23] Basically, the API started with just one endpoint, the completions endpoint, and the completions endpoint, you just put text in, and you get text out, and you can prompt in certain ways. Then we released ChatGPT, and we decided to put that in the API as well, and that became the chat completions API. And that API doesn't just take, like, a string input and produce an output. It actually takes in messages and produces messages, and so you can get a distinction between, like, an assistant message and a user message, and that allows all kinds of behavior. And so the format under the hood for that is called ChatML. Sometimes, you know, because the model is so out of distribution based on what you're doing, maybe the temperature's super high, then it can't follow ChatML. **Swyx** [22:02] Yeah. I didn't know that there could be errors generated there. Maybe I'm not asking challenging enough questions. **Michelle Pokrass** [22:08] It's pretty rare- ... and we're, we're working on driving it down. But actually, this is a side effect of Structured Outputs now, which is that we have removed a class of errors. We didn't really mention this in the blog just 'cause we ran out of space. But, uh- **Swyx** [22:20] That's what we're here to do. **Michelle Pokrass** [22:21] Yeah, the model used to, um, occasionally pick a recipient that was invalid, um, and this would cause an error, but now we are able to constrain to ChatML in a more valid way, and this reduces a class of errors as well. **Swyx** [22:34] Recipient meaning, so there's, there's, like, a, a few number of defined roles like user, assistant, system. **Michelle Pokrass** [22:39] So, like, recipient as in, like, picking the right tool. Um, so- **Swyx** [22:42] Oh. **Alessio** [22:43] Oh **Michelle Pokrass** [22:43] ... so the model before w- was able to res- to hallucinate a tool, but now it's, uh, it can't when you're using Structured Outputs. **Alessio** [22:50] Do you collaborate with other model developers to try and figure out these type of errors? Like, how do you display them? Because a lot of people try to work with different models. **Michelle Pokrass** [22:59] Yeah. **Alessio** [22:59] Is there any? **Michelle Pokrass** [23:00] Yeah, not a ton. We're, we're kind of just focused on making the best API for developers. **Swyx** [23:05] A lot of research and engineering, I guess, c- comes together with evals. You published some evals there. I think, I think Gorilla is one of them. What is your assessment of, like, the state of evals for function calling and structured output right now? **Michelle Pokrass** [23:18] Yeah, we've actually collaborated with, uh, BFCL a little bit, which is- **Swyx** [23:22] Mm-hmm **Michelle Pokrass** [23:22] ... I think the same thing as Gorilla. **Swyx** [23:23] The Berkeley Function Calling Leaderboard. **Michelle Pokrass** [23:25] Kudos to the team. Those, those evals are great, and we use them internally. Yeah, we've also sent some feedback on some things that are misgraded, but... and so we're, we're collaborating to, to make those better. In general, I feel evals are kind of the hardest part of AI. Like, when we talk to developers, it's so hard to get started. It's really hard to make a robust pipeline, and you don't want evals that are, like, 80% successful because, you know, things are gonna improve dramatically. And- It's really hard to craft the right eval. You kind of want to hit everything on the difficulty curve. I find that a lot of these evals are, are mostly saturated, like for BFCL, all the models are near, near the top already, and kind of the errors are more, I would say, like, just differences in default behaviors. I think most of the models on the leaderboard can kind of get 100% with different prompting, but it's more kind of you're just pulling apart different defal-defaults at this point. So yeah, I would say in general, we're missing evals. You know, we work on this a lot internally, but it's hard. **Swyx** [24:19] Did you... Other than BFCL, would you call out any others just for people exploring this space? **Michelle Pokrass** [24:24] SweetBench is actually, like, a very interesting eval if people don't know. You basically give the model a GitHub issue and, like, a repo and just see how well it does with the issue. Which I think is super cool. It's kind of like an integration test, I would say, for models. **Swyx** [24:37] It's a little unfair, right? **Michelle Pokrass** [24:38] What do you mean? **Swyx** [24:39] A little unfair because, like, usually as a human, you have more opportunity to, like, ask questions about what it's supposed to do. **Michelle Pokrass** [24:45] Right. **Swyx** [24:45] And you're giving the model, model, like, way too little information- **Michelle Pokrass** [24:47] It's a hard job **Swyx** [24:48] ... to, to do the job. **Michelle Pokrass** [24:49] Yeah. But yeah, SweetBench targets, like, how well can you follow the diff format and how well can you, like, search across files and how well can you write code? So really excited about evals like that because the pass rate is low- **Swyx** [24:59] Mm-hmm **Michelle Pokrass** [24:59] ... so there's a lot of room to improve. **Swyx** [25:01] Yeah. **Michelle Pokrass** [25:01] And it's just targeting a really cool capability. **Swyx** [25:03] I've seen other evals for function calling where I think might be BFCL as well, where they, they evaluate different kinds of function calling, and I think the, the top one that people care about for some reason, I, I don't know personally that this is so important to me, but it's parallel function calling. **Michelle Pokrass** [25:17] Right. **Swyx** [25:18] I think you confirmed that you're-- you have don't support that yet. Why is that hard? Just- **Michelle Pokrass** [25:22] Yeah **Swyx** [25:22] ... more context about it. **Michelle Pokrass** [25:23] So yeah, we put out parallel function calling, uh, Dev Day last year as well, and, uh, it's kind of the evolution of function calling. So function calling V1, you just get one function back. Function calling V2, you can get multiple back at the same time and save latency. We have this in our API, all our models support it, or all of our newer models support it, but we don't support it with structured outputs right now. And there's actually a very interesting trade-off here. Uh, so when you basically call our API for structured outputs with a new schema, we have to build this artifact for fast sampling later on. But when you do parallel function calling, the kind of schema we follow is not just directly one of the function schemas. It's like this combined schema based on a lot of them. If we were kind of do the same thing and build an index every time you pass in a list of functions, if you ever change the list, you would kind of incur more latency. And we thought it would be really unintuitive for developers and, like, hard to reason about. So we decided to kind of wait until we can support a no added latency solution, um, and not just kind of make it really confusing for developers. **Swyx** [26:20] Mentioning latency, that is something that people discovered, is that there is an increased cost in latency for the first token. **Michelle Pokrass** [26:25] For the first request, yeah. **Swyx** [26:26] First request. Is that an issue? Is that gonna go down over time? Is, is there just an overhead to parsing JSON that used to just un- insurmountable? **Michelle Pokrass** [26:33] It's definitely not insurmountable, and I think it will definitely go down over time. We just kind of take the approach of, of ship early and often. Um, and you know, if you-- if there's nothing in there you, you don't wanna fix, then you probably ship too late. Um, so I think we will get that latency down over time. But yeah, I think for most developers it's not a big concern, 'cause you're testing out your integration, you're, you're sending some requests while you're developing it, and then it's fast in prod. So it kind of works for most people. The alternative design space that we, uh, explored was, like, pre-registering your schema, so, like, a totally different endpoint, and then passing in, like, a schema ID. But we thought, you know, that was a lot of overhead and, like, another endpoint to maintain and just kind of more complexity for the developer, and we think this latency is gonna come down over time, so it made sense to keep it kind of in chat completions. **Swyx** [27:21] I mean, hypothetically, if one were to ship caching at a future point- ... it would basically be the superset of that. **Michelle Pokrass** [27:27] Maybe. I think the caching space is a little underexplored. Like, we've seen kind of two versions of it, but I think, yeah, there's ways that maybe put less onus on the developer. But, you know, we haven't committed to anything yet, but we're definitely exploring opportunities for making things cheaper over, over time. ### Agentic Apps **Alessio** [27:43] Is AGI in agents just gonna be a bunch of structure upload and function calling one next to each other? Like, how do you see... You know, there's like the model does everything. Where do you draw the line? Because you don't call these things like an agent API, but, like, if I were a startup trying to raise a seed round, I would just do function calling and say- **Michelle Pokrass** [27:59] Yes **Alessio** [27:59] ... this is an agent API. **Michelle Pokrass** [28:00] Yeah. **Alessio** [28:00] So how do you think about the difference and, like, how people build on top of it for, like, agentic systems? **Michelle Pokrass** [28:04] Yeah, love that question. One of the reasons we wanted to build structured outputs is to make agentic applications actually work. So right now it's really hard, like, if something is ninety-five percent reliable but you're chaining together a bunch of calls, if you magnify that error rate, it makes your, like, application not work. So that's a really exciting thing here from going from, like, ninety-five percent to a hundred percent. I'm very biased working on the API and working on function calling and structured outputs, but I think those are the building blocks that we'll be using kind of to distribute this technology very far. It's the way you connect, like, natural language and converting user intent into working with your application. And so I think, like, kind of there's no way to build without it, honestly. Like, you, you need your function calls to work. Like, yeah, we wanted to make that a lot easier. **Alessio** [28:45] Yeah. And, and do you think the assistants kind of like API thing will be a bigger part as people build agents? I think maybe most people just use messages and completion and- **Michelle Pokrass** [28:55] So I would say the assistants API was kind of a bet in a few areas. One bet is hosted tools, so we have the file search tool and, and code interpreter. Another bet was kind of statefulness. It's our first stateful API. It'll store, you know, threads, and you can fetch them later. I would say the hosted tools aspect has been really successful. Like, people love our file search tool, and it's like saves a lot of time to not build your own RAG pipeline. Um, I think we're still iterating on the shape for the stateful thing to make it as useful as possible. Right now there's kind of a few endpoints you need to call before you can get a run going, and we wanna work to make that, you know, much more intuitive and easier over time. **Swyx** [29:32] One thing I'm, I'm just kinda curious about, did you notice any trade-offs when you add more structured output, it gets worse at some other thing that was, like, kind of you didn't think was related at all? **Michelle Pokrass** [29:41] Yeah, it's a good question. Yeah, I mean, models are very spiky, and RL is hard to predict. And so every model kind of improves on some things and maybe is flat or neutral on other things. **Swyx** [29:53] Yeah, like it's, it's like very rare to just add a capability- **Michelle Pokrass** [29:57] Yeah **Swyx** [29:57] ... and have no trade-offs in everything else. **Michelle Pokrass** [29:58] So yeah, I don't have something off the top of my head, but I would say, yeah, every model is a special kind of its own thing. This is why we put them in API dated so developers can choose for themselves which one works best for them. In general, we strive to continue improving on all evals, but it's stochastic. **Swyx** [30:14] Yeah. Able to apply the structured output system on backdated models like, uh, 4o May as well as Mini as well as August. **Michelle Pokrass** [30:23] Actually, the new response format- **Swyx** [30:26] Yeah **Michelle Pokrass** [30:26] ... is only available on two models. It's 4o Mini and the new 4o. **Swyx** [30:30] Okay. **Michelle Pokrass** [30:30] So the old 4o doesn't have the new response format. **Swyx** [30:33] Okay. **Michelle Pokrass** [30:34] However, for function calling, we were able to enable it for all models that support function calling, and that's because those models were already trained to follow these schemas. We basically just didn't wanna add the new response format to models that would do poorly at it, because they would just kind of do infinite white space, which is, you know, the most likely token if you have no idea what's going on. **Swyx** [30:52] I just wanted to call out a little bit more in the, in the stuff you've done in a blog post. So in the blog post, just use cases, right? I just want people to be like, yeah, we're spelling it out for you. Use these for extracting structured data from unstructured data. By the way, it does vision too. **Michelle Pokrass** [31:05] Yes. **Swyx** [31:05] Right? So that's cool. Dynamic UI generation. Actually, let's talk about dynamic UI. Um, I think Gen UI, I think is like something that people are very interested in. **Michelle Pokrass** [31:14] Yeah. **Swyx** [31:14] It's your first example. What did you find about it? **Michelle Pokrass** [31:17] Yeah, I just thought it was a super cool capability we have now. So the schemas we s- we support recursive schemas, and this allows you to do really cool stuff. Like, you know- **Swyx** [31:25] Nested this **Michelle Pokrass** [31:25] ... every UI is a nested tree that has children. And so I thought that was super cool. You can use one schema and generate like tons of, of UIs. As a backend engineer who's always struggled with JavaScript and front end- ... like for me, that's super cool. I've now... We've now built a system where I can get any front end that I want. So yeah, that's super cool. The extracting structured data, like the reality of a lot of AI applications is like you're plugging them into your enterprise business, and you have something that works, but you wanna make it a little bit better. And so the reliability gains you get here is like, you'll never get a m- like a classification using the wrong enum. It's like, it's just exactly your, your types. So really excited about that. **Swyx** [32:04] Like maybe hallucinate the actual values, right? **Michelle Pokrass** [32:06] Sure. **Swyx** [32:06] So le- let's clearly state what the guarantees are. The guarantees is that this fits the schema, but the schema itself may be too broad because the JSON schema type system doesn't say like, "I only want a range from 1 to 11." You might give me 0. **Michelle Pokrass** [32:21] Right, right. **Swyx** [32:21] You know, you might give me 12. **Michelle Pokrass** [32:22] So yeah, JSON schema, so this is actually a good thing to talk about. So JSON schema is ex- extremely vast, and we weren't able to support every corner of it. **Swyx** [32:30] Yes. **Michelle Pokrass** [32:30] So we kind of support our own dialect, and it's described in the docs. And there are a few trade-offs we had to make there. So by default, if you don't pass in additional properties in a schema, by default that's true. And so that means you can get other keys, which, you know, you didn't spell out, which is kind of the opposite of what developers want. You basically wanna supply the keys and values, and you wanna get those keys and values. And so then we had a decision to make. It's like, do we redefine what additional properties means as the default? And that felt really bad. It's like there's a schema that's predated us, like, you know, it wouldn't be good... It'd be better to play nice with the community. And so we require that you pass it in as false. You know, one of our design principles is to be very explicit, and so developers know, you know, what to expect. And so this is one where we decided, you know, it's a little harder to discover, but we think you should pass this thing in so that we can have like a very clear definition of what you mean and what we mean. There's a similar one here with like required. By default, every key in JSON scheme is optional. But that's not what developers want, right? Like you would, you'd be very surprised if you passed in a bunch of keys and you didn't get some of them back. And so that's the trade-off we made, is to make everything required and have the developer spell that out. **Swyx** [33:34] Is there a require false? **Michelle Pokrass** [33:36] Um- **Swyx** [33:36] Can people turn it off or- **Michelle Pokrass** [33:37] So- **Swyx** [33:37] ... they're just getting all the, the keys? **Michelle Pokrass** [33:38] So developers can... Basically what we recommend for that is to make your actual key a union type. And so like- **Swyx** [33:45] Null able. **Michelle Pokrass** [33:45] Yeah. So make it a union of- **Swyx** [33:47] Yeah, yeah **Michelle Pokrass** [33:47] ... int and null, and that gets you the same behavior. **Swyx** [33:49] Any other, uh, the examples you wanna dive into, math, chain of thought? **Michelle Pokrass** [33:52] Yeah. You can now specify like a chain of thought field before a final answer. This is just like a more structured way of extracting the final answer. **Swyx** [33:59] Yeah. **Michelle Pokrass** [34:00] One example we have, I think we put up a demo app of this math tutoring, uh, example, or it's coming out soon. **Swyx** [34:06] Did I miss it? Oh, okay. Well. **Michelle Pokrass** [34:07] Basically, it's this math tutoring thing, and you put in an equation, and you can go step by step and answer it. This is something you can do now with structured outputs. In the past, a developer would have to like specify their format and then write a parser and parse out the model's output, which would be pretty hard. But now you just specify like steps, and it's an array of steps, and every step you can render, and then the user can try to... and you can see if it matches and go on that way. So I think it just opens up a lot of opportunities, like for any kind of UI where you wanna treat different parts of the model's responses differently, structured outputs is great for that. **Swyx** [34:38] I remembered my, my question from earlier. So I'm basically just using this to ask you all the questions as a user- **Michelle Pokrass** [34:44] Yeah, totally **Swyx** [34:44] ... as a daily, daily user of the stuff that you put out. So one is a tip that people don't know, and I confirmed with you on Twitter, which is you respect descriptions- **Michelle Pokrass** [34:51] Yes **Swyx** [34:51] ... of JSON schemas, right? And you can basically use that as a prompt for the field. **Michelle Pokrass** [34:55] Totally. **Swyx** [34:55] I assume that's blessed and- **Michelle Pokrass** [34:57] Yes **Swyx** [34:57] ... you know, people should do that. **Michelle Pokrass** [34:57] Intentional, yeah. **Swyx** [34:58] Right? One thing that I started to do, which I don't... it could be a hallucination of me- ... is I changed the, the property name to, to s- to, to prompt the model to what I want it to do. So for example, instead of saying topics as a property name, I would say like, "Brainstorm a list of topics up to five," or something like that- **Michelle Pokrass** [35:17] Nice. **Swyx** [35:18] ... as, as like a, a property name. I, I could stick that in the description as well, but is that too much? **Michelle Pokrass** [35:23] Yeah. I would say, I mean, we're so early in AI- ... that people are figuring out the best way to do things. And I love when I learn from a developer, like a way they found to make something work. In general, I think there's like three or four places to put instructions. **Swyx** [35:37] Yeah. **Michelle Pokrass** [35:37] You can put instructions in the system message, and I would say that's helpful for like when to call a function. So it's like- You know, let's say you're building a customer support thing, and you want the model to verify the user's phone number or something. You can tell the model in the system message, like, "Here's when you should call this function." Then when you're within a function, I would say the descriptions there should be more about how to call a function. So really common is someone will have, like, date as a string, but you don't tell the model, like, do you want year, year, month, month, day, day, or do you want that backwards? And that's what a really good spot is for those kind of descriptions is, like, how do you call this thing? And then sometimes there's, like, really... stuff like what you're doing is, like, name the, the key by what you want. So sometimes people put, like, do not use, and, you know, if they don't want, you know, this parameter to be used except only in some s- some circumstances. And really, I think that's the fun nature of this. It's like you're figuring out the best way to get something out of the model. **Swyx** [36:28] Okay. So, so you don't have an official recommendation is what I'm hearing. **Michelle Pokrass** [36:31] Well, the official recommendation is, you know, how to call a model system instructions. **Swyx** [36:34] Exactly, exactly. The- **Michelle Pokrass** [36:35] Or when to call a function **Swyx** [36:35] ... hierarchy. **Michelle Pokrass** [36:36] Yeah. **Alessio** [36:37] Do you benchmark these type of things? So, like, say with date, it's like description, it's like return it in, like, ISO 8- **Michelle Pokrass** [36:44] Yeah **Alessio** [36:44] ... or if you called the key date in ISO- **Michelle Pokrass** [36:46] Mm **Alessio** [36:47] ... A6001. I feel like the benchmarks don't go that, that deep. **Michelle Pokrass** [36:50] Yeah. **Alessio** [36:50] But then all the AI engineering kind of community, like, all the work that people do is like, "Oh, actually this performs better," but then there's no way to verify. **Michelle Pokrass** [36:59] Right. **Alessio** [36:59] You know? Like, uh, even the, "I'm gonna tip you $100,000," or whatever- **Michelle Pokrass** [37:03] Sure **Alessio** [37:03] ... like, some people say it works, some people say it doesn't. Do you pay attention to this stuff as you build this, or are you just like, the model is just gonna get better, so why waste my time running evals on these small, small things? **Michelle Pokrass** [37:14] Yeah. I would say... To that, I would say we basically pick our battles. I mean, there's so much surface area of LLMs that we could dig into, and we're just moc- mostly focused on kind of raising the capabilities for everyone. I think for customers, and we work with a lot of customers, really developing their own evals is super high leverage. 'Cause then you can upgrade really quickly when we have a new model, you can experiment with these things with confidence. So yeah, we're, we're hoping to make making evals easier. I think that's really generally very helpful for developers. **Swyx** [37:43] For people, I would just kind of wrap up the discussion for Structured Outputs. I immediately implemented... We used Structured Outputs for AI news. I used Instructor, and I ripped it out, and I think it w- I saved, um, 20 lines of code. But more importantly, it was like we cut it by 55% of API costs based on what I, what I measured because of all, uh, we saved on the retries. **Michelle Pokrass** [38:02] Nice. **Swyx** [38:03] Um- **Michelle Pokrass** [38:03] Yeah, love to hear that. **Swyx** [38:04] Yeah, which, which people I think don't understand when y- you, you can't just simply, like, add Instructor or add outlines. You can do that, but it's actually gonna cost you a lot of retries to get the, the, the model that you want, but you're kind of just kind of building that internally into the model. **Michelle Pokrass** [38:18] Yeah. I think this is the kind of feature that works really well when it's integrated with, like, the LLM provider. Yeah, actually, I had folks... even my, my husband's company, he works at a small startup, they thought we were just retrying, um, and so I had to make clear to a lot of folks- ... we are not retrying, you know, we're doing it in one shot, and this is how you save on latency and cost. ### Roadmap & Tuning **Swyx** [38:36] Awesome. Any other behind the scenes stuff, just generally on Structured Outputs? We, we, we're gonna move on to the other models. **Michelle Pokrass** [38:42] Yeah, I think that's it. **Swyx** [38:43] Well, look, that's an excellent product, and I, I think everyone will be using it, and we have the full story now that people can try out. So roadmap would be parallel function calling. Anything else that you've called out as, like, coming soon? **Michelle Pokrass** [38:54] Uh, quite soon, but, you know, we're thinking about does it make sense to expose custom grammars- **Swyx** [38:59] Yeah **Michelle Pokrass** [38:59] ... um, beyond JSON schema. **Swyx** [39:00] What would you want to hear from developers to give you information, whether it's custom grammars or anything else about Structured Output? Like, what would you want to know more of? **Michelle Pokrass** [39:07] Just, you know, always interested in, in feature requests, what's not working. But I'm, I'd be really curious, like, what specific grammars folks want. I know some folks want to match programming languages like Python. There's some challenges, like with the expressivity of our, you know, implementation. And so, yeah, just kind of the class of grammars folks want. **Swyx** [39:26] I have a very simple one, which is a lot of people try to do... use GPT as judge, right? **Michelle Pokrass** [39:30] Mm-hmm. **Swyx** [39:30] Which means they end up doing a rating system, and then there's like 10 different kinds of rating systems. There's a Likert scale- **Michelle Pokrass** [39:36] Right **Swyx** [39:36] ... there's whatever. If there was an o- officially blessed way to do a rating system with Structured Outputs- **Michelle Pokrass** [39:41] Totally **Swyx** [39:41] ... everyone would use it. **Michelle Pokrass** [39:43] Yeah. Yeah, that makes sense. I mean, we often recommend using log probs with classification tasks. So rather than, like, sampling, you know, let's say you have four options like red, yellow, blue, green. Rather than sampling, you know, two tokens for yellow, you can just do like A, B, C, D and get the log probs of those. You know, the inherent randomness of each sampling isn't taken into account, and you can just actually look at what is the most likely token. **Swyx** [40:07] I think this is more of like a calibration question. Like, if I ask you to rate things from one to 10- **Michelle Pokrass** [40:11] Mm-hmm **Swyx** [40:12] ... a non-calibrated model might always pick seven- **Michelle Pokrass** [40:14] Mm-hmm **Swyx** [40:15] ... just like a human would. **Michelle Pokrass** [40:16] Right. **Swyx** [40:17] So, like, actually have a nice gradation from one to 10 would be the, the, the rough idea. **Michelle Pokrass** [40:22] Yeah. **Swyx** [40:22] And then even for Structured Outputs, I can't just say, have a field of rating from one to 10 because I, I have to then validate it and, you know- **Michelle Pokrass** [40:29] Right **Swyx** [40:29] ... it might give me 11. **Michelle Pokrass** [40:30] Yeah, absolutely. **Swyx** [40:31] So, yeah. **Alessio** [40:31] So what about model selection? Now you have a lot of models. When you first started, you had one model endpoint. I guess you had like the, the v3 and then a- **Michelle Pokrass** [40:39] Yeah, the advantage **Alessio** [40:39] But, like- **Swyx** [40:40] Yeah **Alessio** [40:40] ... most people are using one- **Michelle Pokrass** [40:42] Right **Alessio** [40:42] ... uh, model endpoint. Today, you have, like, a lot of competitive models, and I think we're nearing the end of the 3.5 run RIP. How do you advise people to, like, experiment, select, both in terms of, like, task and, like, cost? Like, what's your playbook? **Michelle Pokrass** [40:56] In general, I think folks should start with 4o mini. That's our cheapest model, and it's a great workh- work- workhorse, uh, works for a lot of great use cases. If you're not finding the performance you need, like, you know, maybe it's not smart enough, then I would suggest going to 4o. And if 4o works well for you, that's great. Finally, there's some, like, really advanced frontier use cases, um, and maybe 4o is not quite cutting it, and there I would recommend our fine-tuning API. Even just, like, 100 examples is enough to get started there, and you can really get the performance you're looking for. **Swyx** [41:26] We're recording this ahead of it, but, like, you're announcing other, some fine-tuning stuff that people should pay attention to. **Michelle Pokrass** [41:32] Yeah. Actually, tomorrow we're dropping our GA for GPT-4o fine-tuning. So 4o mini has been available for a few weeks now, and 4o is now gonna be generally available. And we also have a free training Offering for a bit, I think until September 23rd, you get one million in free training tokens a day. **Swyx** [41:50] This is already announced, right? Or was... am, am, am I talking about a different date? **Michelle Pokrass** [41:52] So that was for 4o Mini, and now it's also for 4o. So we're really excited to see what people do with it, and it's actually a lot easier to get started than a lot of people expect. They think they might need tens of thousands of examples, but even 100 really high-quality ones or 1,000 is, is enough to get going. **Swyx** [42:07] Oh, well, we might get a separate podcast just specifically on that, but, um, you know, we haven't confirmed that yet. It basically seems like every time... I think people's concerns about fine-tuning is that they're kind of locked into a model, and I think you're paving the path for migration of models. Um, as long as they keep their original data set, like they can at least migrate nicely. **Michelle Pokrass** [42:26] Yeah, I'm not sure what we've said publicly there yet, but we definitely want to make it easier for folks to, to migrate. **Swyx** [42:32] It's the number one concern, you know? **Michelle Pokrass** [42:33] Yeah. **Swyx** [42:33] I'm just... You know, it's obvious. **Michelle Pokrass** [42:35] Absolutely. **Swyx** [42:36] I also want to point people to the... you have official model selection docs where it's, it's on, it's in the guide. We'll put it in the show notes where it says to op-optimize for accuracy first, so prompt engineering, RAG, evals, fine-tuning. This was done at DevDay last year, so I'm just repeating things, and then optimize for cost and latency second, and there's a, there's a few sets of steps for optimizing latency, so people can read up on that stuff. **Michelle Pokrass** [42:57] Yeah, totally. **Swyx** [42:58] Yeah. **Alessio** [42:58] We had one episode with, um, Miguel Scarlini from DeepMind, and we actually talked about how some people don't actually get to the boundaries of the model performance. You know, they just kind of try one model, and it's like, "Oh, LLMs cannot do this," and they stop. **Michelle Pokrass** [43:11] Right. **Alessio** [43:12] How should people get over the hurdle? It's like, how do you know if you hit the model performance or like you hit skill issues, you know? **Michelle Pokrass** [43:18] Mm. **Alessio** [43:18] It's like your prompt is not good or like try another model and whatnot. Is there an easy way to do that? **Michelle Pokrass** [43:22] That's tough. Some people are really good at prompting, and they just kind of get it right away, and, and for others, it's more of a challenge. I think there's a lot we can do to make it easier to prompt our models. But for now, I think requires a lot of creativity and not giving up right away. Yeah. And a lot of people have experience now with ChatGPT. You know, before ChatGPT, the easiest way to play with our models was in the playground. But now kind of everyone's played with it, uh, with a model of some sort, and they have some sort of intuition. It's like, you know, if I tell you my grandma is sick, then maybe I'll get the right output. And, and we're hoping to kind of remove the need for that. But playing around with ChatGPT is a really good way to get a feel for, you know, how to use the API as well. **Alessio** [43:59] Will prompt engineering be here forever, or is it a dying art as the models get better? **Michelle Pokrass** [44:05] I mean, it's like the perennial question of software engineering as well. It's like, as the models get better at coding, you know, if we hit 100 on SWE-bench, what does that mean? I think there will always be alpha and people who are able to like clearly explain what they're trying to build. Most of engineering is like figuring out the requirements and, and stating what you're trying to do, and I believe this will be the case with AI as well. You're gonna have to very clearly explain what you need, and some people are better than others at it, and people will always be building. It's just the tools are going to get far better. **Swyx** [44:33] Last two weeks, you released two models. There's gpt-4o 20240806, and then there's also chatgpt-4-latest. I think people were a little bit confused by that, and then you, you issued a clarification that it was one's chat tuned and the other is more function-calling tuned. Can you elaborate just- **Michelle Pokrass** [44:48] Yeah. Totally. So part of the impetus here was to kind of be very transparent with what's on ChatGPT and in the API. So basically, we're, we're often training models and, and there are different use cases. So you don't really need function calling for user-defined functions in ChatGPT, and so this gives us kind of the freedom to build the best model for each use case. So in chatgpt-latest, we're releasing kind of this rolling model. The weights aren't pinned as we release new models- **Swyx** [45:14] Like this is literally what we use. **Michelle Pokrass** [45:16] Yeah. So it's in what's, what's in ChatGPT, so it's very good for like chat style use cases. But for the API broadly, you know, we really tune our models to be good at things that developers want, like function calling and structured outputs. And when a developer builds their application, they want to know that kind of the weights are stable under them. And so we have this offering where it's like if you're tuning to a specific model and you know your function works, you know we will never change the weights out from under you. And so those are the models we commit to supporting for a long time, and we think those are the best for developers. But we want to give it up. You know, we want to leave the choice to developers, like, "Do you want the ChatGPT model or do you want the API model?" And you have the freedom to choose what's best for you. **Swyx** [45:55] I think it's... for people, they, they do want to pin model versions. So I don't know when they would use ChatGPT, like the, the, the rolling one, unless they're really just kind of cloning ChatGPT- **Michelle Pokrass** [46:07] Yeah **Swyx** [46:07] ... which it's like why would they? **Michelle Pokrass** [46:09] I mean, I think there's a lot of interesting stuff that developers can do wh-when unbounded, and so we don't, we don't want to limit them artificially. So it's kind of survival of the fittest. Like whichever model is better, you know, that's the one that people should use. **Swyx** [46:22] Yeah. I talked about it to, to my friends as like this isn't a new thing. Like they... and basically, OpenAI has never actually shared with you the actual ChatGPT model, uh, and now they do. That's it. **Michelle Pokrass** [46:32] Well, it's not necessarily true. Actually, a lot of the models we have shipped have been the same. **Swyx** [46:37] Oh, okay. **Michelle Pokrass** [46:37] But you know, sometimes they diverge, and it's not a limitation we want to stick around. **Swyx** [46:42] Anything else we should know about the new model? I don't think there were, there were e- there was no evals announced or anything, but, but people say it's better. I mean, obviously, LMSYS is like way better evolved on everything, right? It's like number one in the world on- **Michelle Pokrass** [46:53] Yeah. We're, um, we published some release notes. They're not as in-depth as we want to be yet- **Swyx** [46:58] Yeah **Michelle Pokrass** [46:58] ... because we're still, it's still kind of a science and we're learning what actually changes with each model and how can we better understand the capabilities. But we are trying to do more release notes in the future and, and keep folks updated. But yeah, it, it's, it's kind of an art and a science right now. **Swyx** [47:14] You need the best evals team in the world to help you figure this out. **Michelle Pokrass** [47:17] Yeah. Evals are hard. We're hiring if you want to come work on evals. **Swyx** [47:20] Hold that thought on hiring. Uh, we'll, we'll come back to the end on what you want, what you're looking for because obviously people want to join you, and they want to know what, uh, qualities you're looking for. **Alessio** [47:28] So we just talked about API versus ChatGPT. What's, I guess, like the vision for the interface? You know, the, the mission of OpenAI is like build AGI that is accessible. Like, uh, uh, where is it gonna come from? **Michelle Pokrass** [47:40] Totally, yeah. So I believe that, uh, the API is kind of our broadest vehicle for distributing AGI. Um, you know, we're building some first-party products, but they'll never reach every niche in the world and kind of every corner and community. And so really love working with developers and seeing the incredible things they come up with. I often find that developers kind of see the future before anyone else, and we love working with them to make it happen. And so really, the API is a bet on going really broad. And we-we'll go very deep as well in our first-party products, but I think just that our impact is absolutely magnified by every developer that we up-uplift. ### War Stories **Swyx** [48:13] They can do the last mile where's-- where you cannot. Like- **Michelle Pokrass** [48:16] Yeah **Swyx** [48:16] ... ChatGPT is one type of product, but there's many other kinds. Uh, in fact, uh, you know, I, I observed, I think in February, basically ChatGPT's user growth stopped when the API was launched because everyone's like kinda be able to take that and, and build other things. That has n-not become true anymore because ChatGPT growth has- **Michelle Pokrass** [48:34] Yeah **Swyx** [48:34] ... has continued to grow. But then you're not confirming any of this. This is me quoting Similarweb numbers which are, have very high variance. **Michelle Pokrass** [48:41] Well, the API predates ChatGPT. The API was actually OpenAI's first product, and the first, uh, idea for commercialization, that predates me as well. **Swyx** [48:49] Wide release, like GA, everyone can sign up and use it immediately. **Michelle Pokrass** [48:52] Right. **Swyx** [48:52] Yeah, that's what I'm talking about. But, but yeah, I mean, I, I, I do believe that. Uh, a-and, you know, that means you also have to expose all of OpenAI models, right? Like, um, all the multimodal models. We'll ask you, uh, questions on that, but, like, uh, I think that a- that API mission is, is important. It's interesting that hottest new programming language is supposed to be English. **Michelle Pokrass** [49:12] Hmm. **Swyx** [49:12] But it's actually just software engineering, right? It's, it's just, you know, we're talking about HTTP error codes. **Michelle Pokrass** [49:18] Right. **Swyx** [49:18] Like **Michelle Pokrass** [49:19] Yeah. I think, you know, engineering is still the way you access these models, and I think there are companies working on, on tools to make engineering more accessible for everyone. But there's still so much alpha in just writing code and, and deploying. **Swyx** [49:33] Yeah. One might even call it AI engineering. **Michelle Pokrass** [49:35] Exactly. **Swyx** [49:35] I don't know. Yeah, so, like, th-there's lots of war stories from, from building this platform. We started at the start of your career, and then we jumped straight to Structured Outputs. There's a whole thing, like two years, that we skipped in between. **Michelle Pokrass** [49:45] Right. **Swyx** [49:46] What have become your principles? What are your favorite stories that, that you like to tell? **Michelle Pokrass** [49:50] We had so much fun working on the Assistants API- **Swyx** [49:53] Yeah **Michelle Pokrass** [49:53] ... and leading up to Dev Day. You know, things are always pretty chaotic when you have an externally, like a date- **Swyx** [49:58] It's a hosting function **Michelle Pokrass** [49:59] ... that is hard, and there's like a stage and there's like 1,000 people coming. Um- **Swyx** [50:03] You can always launch a waitlist. I mean **Michelle Pokrass** [50:06] We're trying hard not to. **Swyx** [50:07] Yeah. **Michelle Pokrass** [50:07] Um, because, you know, we love it when people can access the thing on day one. And, and so yeah, the Assistants API, we had like this really small team and just working as hard as we could to make this come to life. But even actually the morning of, I don't know if you remember this, but Sam did this keynote. **Swyx** [50:24] Yep. **Michelle Pokrass** [50:25] And Roman came up, and they gave free credits to everybody. **Swyx** [50:28] Mm-hmm. **Michelle Pokrass** [50:29] So that was live, fully live, as were all of the demos that day. Um, but actually maybe like two hours before that, we had a little outage, and everyone was like scrambling to make this thing work again. So we're, yeah, th-things are, are early and scrappy here and, you know, we were really glad. We were a bit on the edge of our seat watching it live. **Swyx** [50:47] Wow. **Alessio** [50:47] What's the plan B in that situation? If you can share, it's like, okay- **Swyx** [50:50] Play, play a video. It's just a classic DevRel, right? I don't know. **Michelle Pokrass** [50:52] I mean, I, I actually don't know what the plan B was. **Swyx** [50:55] No plan B, no failure. **Michelle Pokrass** [50:57] But we just, you know, we fixed it. We got, got everything running again and, uh, the demo went well. **Alessio** [51:02] Just hire cracked Waterloo grads. **Swyx** [51:03] Exactly. **Alessio** [51:05] Skill issues as usual. **Michelle Pokrass** [51:06] Sometimes you just gotta make it happen. **Swyx** [51:08] So I'm-- I imagine it's actually very motivating. But I, I did hear that after Dev Day, like the whole company got like a few weeks off just to relax a little bit. **Michelle Pokrass** [51:16] Yeah. We, um, we sometimes get... Like, we just had the week of July 4th off and, yeah. **Swyx** [51:21] Yeah. **Michelle Pokrass** [51:21] It's, it's hard to take vacation because people are working on such exciting things, and it's like you get a lot of FOMO on vacation, so it helps when the whole company's on vacation. **Swyx** [51:29] Mentioning Assistants API, you actually announced a roadmap there, and things have developed. Um, I think people may not be up to date. What's the offering today versus, you know, one year ago? **Michelle Pokrass** [51:40] Yeah. So we've made a, a bunch of key improvements. I would say the biggest one is in the File Search product. Before, we only supported, I think, like 20 files per assistant, and the way we used those files was like, like less effective. Basically, model would decide based on the file name whether to search a file, and there's like not a ton of information in there. So our new offering, which we shipped, uh, a few months ago, I think now allows 10K files per assistant, which is like dramatically more. And also it's a, it's a kind of different operation, so you can search semantically over all files at once rather than just kind of the model choosing one up front. So a lot of customers have seen really good performance. We also have exposed more like chunking and re-ranking options. I think the re-ranking one is coming, I think, next week or, or very soon. So this kind of gives developers more control and more flexibility there. So we're trying to make it the easiest way to kind of do RAG at scale. **Swyx** [52:30] Yeah. I think that visibility into the, the RAG system was the number one thing missing from Dev Day, and then people got their first impressions, and then they never looked at it again. So that's, it's important. The re-ranker is a core feature of, let's say, some other foundation model labs. Is OpenAI gonna like offer a re-ranking service, um, a re-ranker model? **Michelle Pokrass** [52:49] So we do re-ranking as part of it. I think we're soon going to ship more controls for that. **Swyx** [52:54] Okay, got it. And like if I'm an existing LangChain, LlamaIndex, whatever, how do you compare? Do you make different choices? Like where does that e-exist in the spectrum of choices? **Michelle Pokrass** [53:05] I think we are just coming at it trying to be the easiest option. And so ideally, like you don't have to know what a re-ranker is, and you don't have to have a chunking strategy, and the thing just kind of works out of the box. So I would say that's where we're going. And then, you know, giving controls to the power users to, to make the changes they need. **Swyx** [53:23] Awesome. I want to ask about a couple other things, just updates on stuff also announced at De-Dev Day, and we talked about this before. Uh, determinism, something that people really want. Dev Day, we announced the seed param as well as system fingerprint and like objectively, I've heard issues. **Michelle Pokrass** [53:36] Yeah. **Swyx** [53:37] I don't know what's going on. **Michelle Pokrass** [53:38] Yeah, the seed parameter is not fully deterministic, and it's kind of a best effort thing. **Swyx** [53:43] Yeah. **Michelle Pokrass** [53:44] So you'll notice there's more determinism in the first few tokens. That's kind of the current implementation. We've heard a lot of feedback. We're, we're thinking about ways to make it better, but it's challenging. It's kind of trading off against, you know, reliability and uptime. **Alessio** [53:55] Other maybe underrated API-only thing, logic bias. Uh, that's another thing that kind of seems very useful and that maybe most people are like, "It's a lot of work. I don't want to use it." Uh- **Michelle Pokrass** [54:05] Yeah **Alessio** [54:05] ... do you have any examples of like Use cases or, like, products that are made a lot better- **Michelle Pokrass** [54:10] Yeah **Alessio** [54:10] ... uh, through using it. **Michelle Pokrass** [54:11] So yeah, classification is the big one. So you logic bias your valid classification outputs and, you know, you're more likely to get something that matches. We've seen that people logic bias, like, punctuation tokens, maybe- **Alessio** [54:24] Mm **Michelle Pokrass** [54:24] ... trying to get more succinct writing. Yeah, it's generally a very much a power user feature, and so not a ton of folks use it. **Swyx** [54:31] I actually wanted to use it to reduce the incidence of the word delve. **Michelle Pokrass** [54:34] Yeah. **Swyx** [54:35] Have people done that? **Michelle Pokrass** [54:37] Probably. I don't know. Is delve one token? It- you're probably... You gotta do a lot of- **Swyx** [54:41] It's gotta be **Michelle Pokrass** [54:41] ... permutations. **Swyx** [54:42] It's used so much. **Michelle Pokrass** [54:43] Maybe it is. **Swyx** [54:43] It's gotta be. **Michelle Pokrass** [54:43] Depends on the tokenizer. **Alessio** [54:44] Are there non-public tokenizers that... I guess you cannot answer or you would admit it. Are the 100K and 200K vocabs, like, the ones that you use, like, across all models or, um- **Michelle Pokrass** [54:53] Yeah, I think we have docs that publish more information. I don't have it off the top, but I think we publish which tokenizers for which model. **Alessio** [55:01] Okay. So those are the only two. **Swyx** [55:02] Rate the, the tiering rate limiting system, I don't think there was an official, like, blog post kind of announcing this, but it was kind of mentioned that, like, you started tying, like, fine-tuning to, to tiering, uh, and, like, feature rollouts. Just on a, from, from your point of view, like, how do you manage that? And, like, what should people know about the- **Michelle Pokrass** [55:18] Yeah **Swyx** [55:19] ... tiering system and rate limiting? **Michelle Pokrass** [55:20] Yeah, I think basically the main changes here were to be more transparent and easier to use. So before, developers didn't know what tier they're in, and now you can see that in, in the play- in the dashboard. I think it's also... I think we publish, like, how you move from tier to tier, and so this just helps us do kind of gated rollouts. For the fine-tuning launch, I think everyone tier two and up has full access. **Swyx** [55:40] That makes sense. That, you know, I would just advise people to just get to tier five as quickly as possible. **Michelle Pokrass** [55:45] Sure. **Swyx** [55:46] Like, gold star customer, you know. Like, I don't know, it seems to make sense. ### API Tour **Alessio** [55:49] Do we wanna maybe wrap with future things and kinda, like, how you think about designing and everything? So you just mentioned you wanna be the easiest way to basically do everything. What's the relationship with other people building in the developer ecosystem? Like, I think at, uh, maybe in the early days it's like, "Okay, we only have these APIs," and then everybody helps us, but now you're kind of building a whole platform. How do you make decisions? **Michelle Pokrass** [56:11] Yeah, I think kind of the 80/20 principle applies here. We'll build things that kind of capture, you know, 80% of the value and, and maybe leave the long tail to other developers. So we really prioritize by, like, how much feedback are we getting? How much easier will make this, will this make something, like, an integration for a developer? So yeah, we, we, we wanna do more in this space and not just be an LLM as a service, but kind of AI development platform as a service. **Swyx** [56:34] Ooh, okay. That ties into a, a thing that I put in the, in the notes that we prepped. There are other companies trying to be AI development platform, so you will compete with them, or they... Do you just want to know what you won't s- uh, build so that they can build it? **Michelle Pokrass** [56:50] Yeah. It's a tough question. I, I think we haven't, you know, determined what exactly we will and won't build, but you can think of something, if it makes it a lot easier for developers to integrate, you know, it's probably on our radar, and we'll, you know, stack rank by impact. **Swyx** [57:04] Yeah. I... So there's, like, cost tracking and model fallbacks. Model fallbacks is an interesting one because people do it. I don't think it adds a ton of value, but, like, if you don't build it, I have to build it because- **Michelle Pokrass** [57:15] Right **Swyx** [57:15] ... if one API's down or something, I, I need to fall back to another one. **Michelle Pokrass** [57:18] Yeah. I mean, the way we're targeting that user need is just by investing a lot in reliability, and so we- **Swyx** [57:24] Oh, yeah. **Michelle Pokrass** [57:24] We have- **Swyx** [57:25] Just don't fail. **Michelle Pokrass** [57:26] I mean, we have improved our uptime, like, pretty dramatically over the last year, and it's been, you know, the result of a lot of hard work from folks. So you'll see that on our status page and, and our continued commitment going forward. **Alessio** [57:37] It's the important thing about owning the platform that it gives you the flexibility to put all the kind of messy stuff behind the, the scenes? Or, yeah, how do you draw the line between what you wanna include? **Michelle Pokrass** [57:48] Yeah, I just think of it as, like, how can we onboard the next generation of, of AI engineers, as you put it, right? Like, what's the easiest way to get them building really cool apps? And I think it's by building stuff to kind of hide this complexity or just make it really easy to inte- integrate. So I think of it a lot as, like, what is the value add we can provide beyond just the models that makes the models really useful? **Swyx** [58:09] Okay. We'll touch on four more features of the API platform that we prepped: Batch, Vision, Whisper, and then team enterprise stuff. So you wanted to talk about Batch. **Michelle Pokrass** [58:17] Yeah. **Swyx** [58:17] So the rough idea is you give a... The, the contract between you and me is that I give you, give you the Batch job, you have 24 hours to run it, and, and it's kind of like Spot Inst for- **Michelle Pokrass** [58:27] Sort of **Swyx** [58:27] ... for a- of the API. What, what, what should people know about it? **Michelle Pokrass** [58:31] Yeah. So it's half off, which is a great savings. It also works with, like, 4o mini, so the savings on top of 4o mini is, is pretty crazy. Like, the stuff you can do- **Swyx** [58:41] Like 7.5 cents or something per million. **Michelle Pokrass** [58:43] Yeah. I should really have that number top of mind, but it's, like, staggeringly cheap. And so I think this opens up a lot more use cases. Like, let's say you have a user activation flow and you wanna send them an email, like, maybe every day or, like, at certain points in their user journey. So now you can do this with the Batch API, and something that was maybe a lot more expensive and not feasible is now very easy to do. So r- right now we have this 24-hour turnaround time for half off, and, and curious, would love to hear from your community, like, what kind of turnaround time do they want? **Swyx** [59:11] I would be an ideal user of Batch, and I cannot use Batch because it's 24 hours. I need two to four. **Michelle Pokrass** [59:16] Two to four hours, okay. Yeah, that's good to know. But yeah, just a lot of folks haven't heard about it. It's also really great for, like, evals, running them offline. Like, you don't, generally don't need them to come back within, you know, two hours. **Swyx** [59:26] Yeah. I think you could do a range, right? Two to four for me, like, I need to produce a daily thing, and then 24 for, like, the, the average use case, and then maybe, like, a week, a month, who cares? **Michelle Pokrass** [59:34] Yeah. **Swyx** [59:35] Like, for people who just have a lot to do. **Michelle Pokrass** [59:37] Yeah, absolutely. So yeah, that's Batch API. I think folks should use it more. It's pretty cool. **Alessio** [59:41] Is there a future in which, like, six months is, like, free? You know? Like, is there, like- **Swyx** [59:46] What? **Alessio** [59:47] ... small- **Swyx** [59:47] Oh. **Alessio** [59:47] Is there, like, super small, like, shards of, like, GPU runtime that, like- **Michelle Pokrass** [59:52] Maybe **Alessio** [59:52] ... over a long enough timeline you can just run all these things for free? **Michelle Pokrass** [59:55] Yeah, it's certainly possible. I think we're getting to the point where a lot of these are, like, almost free. **Swyx** [1:00:00] That's true. **Alessio** [1:00:01] Yeah. Why, why- **Swyx** [1:00:01] You're already close, but- Why, why would they work on something that's, like, completely free? I don't know. Okay, so Vision. Vision got GA'd. Like, last year people were so wowed by the GPT-4 demo- **Michelle Pokrass** [1:00:10] Yeah **Swyx** [1:00:10] ... and that was primarily Vision. Um, what was it like building the Vision API? **Michelle Pokrass** [1:00:13] Yeah. The Vision API is super cool. We have a great team working there. I think the cool thing about Vision is that it works across our APIs, so there's... You can use it in the Assistants API, you can use it in the Batch API, in Check Completions, it works with Structured Outputs. I think it just helps a lot of folks with kind of data extraction where, you know, the spatial relationships between the data i- is com- too complicated, and you can't get that over text. But yeah, there's a lot of really cool use cases. **Swyx** [1:00:37] I think the, the tricky thing for me is understanding how frequent to turn Vision into from, like, i- single images into, like, effectively just always watching. **Michelle Pokrass** [1:00:47] Hmm. **Swyx** [1:00:47] And right now, I think people just, like, send a frame every s- every second. Will that model ever change? Will, will, will there just be, like, I stream you video and then **Michelle Pokrass** [1:00:55] Yeah, I think it's very possible that we'll have an API where you stream video in, and maybe, you know, to start we'll do the frame, uh, sampling for you. **Swyx** [1:01:03] 'Cause the frame sampling is, is the default, right? **Michelle Pokrass** [1:01:06] Right. **Swyx** [1:01:06] But I, I feel like it's hacky. **Michelle Pokrass** [1:01:07] Yeah, I think it's hard for developers to do, and so, you know, we should definitely work on making that easier. **Alessio** [1:01:12] Is there... In the Batch API, do you have, like, um, time, uh, guarantees, like order guarantees? Like, if I send you a batch request of, like, a video analysis, I need every frame to be done in order. **Michelle Pokrass** [1:01:22] For Batch you send, like, a list of requests, and each of them stand alone. **Alessio** [1:01:26] Mm-hmm. **Michelle Pokrass** [1:01:26] So you'll get all of them finished, but they don't kind of chain off each other. **Alessio** [1:01:29] Well, if you're doing a video, you know, if you're doing, like, a- analyzing a video, so you can have- **Swyx** [1:01:34] Oh, I wasn't linking video to Batch, but that's interesting. Okay. **Alessio** [1:01:37] Yeah. Well- **Swyx** [1:01:38] All right **Alessio** [1:01:38] ... a video it's like, you know, if you have a very long video, you can just do a batch of all the images- **Michelle Pokrass** [1:01:42] Yeah **Alessio** [1:01:42] ... and let it process. **Michelle Pokrass** [1:01:43] Well, it's a good idea. **Alessio** [1:01:44] If it's the same. **Swyx** [1:01:44] You could offer, like- **Michelle Pokrass** [1:01:44] Batch but serially **Swyx** [1:01:45] ... sequential true. **Alessio** [1:01:46] Yeah, yeah, yeah. Exactly. **Swyx** [1:01:47] You know? **Alessio** [1:01:48] Uh, but the whole point of Batch is you're just using kind of spare time to run it. **Michelle Pokrass** [1:01:52] Right. **Alessio** [1:01:52] Let's talk about my favorite model, Whisper. Oliver, I, I built this thing- **Swyx** [1:01:56] We described this on, yeah. **Alessio** [1:01:57] Yeah, built this thing called Small Podcaster, which is a open source tool for podcasters, and why does Whisper API not have diarization when- ... everybody is transcribing people talking? **Michelle Pokrass** [1:02:06] Yes. **Alessio** [1:02:06] That's my main question. **Michelle Pokrass** [1:02:08] Yeah, it's a good question, and you've come to the right person. I actually worked on the Whisper API and, and shipped that. That was one of my first APIs I shipped. Long story short is that, like, Whisper V3, which we open sourced, has, I think, m- the diarization feature, but there's some, like, performance trade-offs, and so Whisper V2 is better at some things than Whisper V3. And so it didn't seem that worthwhile to ship Whisper V3 compared to, like, the other things in our priorities. **Alessio** [1:02:32] Oh. **Michelle Pokrass** [1:02:32] I think we still will at some point, but yeah, it's just... You know, there's always so many things we could work on. It, it's, it's tough to do everything. **Alessio** [1:02:39] We have a Python notebook that does the diarization- **Michelle Pokrass** [1:02:41] Nice **Alessio** [1:02:41] ... for the pod, but I would just like... You can translate, like, 50 languages- **Michelle Pokrass** [1:02:46] Yeah **Alessio** [1:02:46] ... but you cannot tell me who's speaking. **Michelle Pokrass** [1:02:47] I feel like- **Alessio** [1:02:47] That was, like, the funniest thing **Michelle Pokrass** [1:02:50] ... there's, like, an xkcd thing about this, about hard problems in AI. I f- I forget the link. **Alessio** [1:02:55] Yeah, yeah. Exactly. **Michelle Pokrass** [1:02:55] But there's like- **Alessio** [1:02:56] It's nonlinear **Michelle Pokrass** [1:02:56] ... tell me if this was taken in a park, and, like, that's easy, and it's like, tell me if there's a bird in this picture, and it's like, give me 10 people and a research team. **Swyx** [1:03:01] Yeah. **Michelle Pokrass** [1:03:02] It's like you never know which things are, are challenging, and diarization is, I think, you know, more challenging than, than expected. **Alessio** [1:03:09] Yeah. It still breaks a lot with, like, overlaps obviously. Sometimes similar voices it struggles with. Like, I need to, like, double-read the thing. **Michelle Pokrass** [1:03:17] Totally. **Alessio** [1:03:18] But yeah, great model. I mean, it would take us so long to do transcriptions, and I don't know why... Like, Small Podcast has better transcription than, like, mostly every- **Swyx** [1:03:26] Descripts **Alessio** [1:03:26] ... commercial tool. **Michelle Pokrass** [1:03:27] Oh, yeah. **Swyx** [1:03:28] It, it, it beats Descript. **Alessio** [1:03:29] Like, it- **Michelle Pokrass** [1:03:29] Nice **Alessio** [1:03:29] ... and I'm like, I'm just using the model. I'm literally not doing anything. **Michelle Pokrass** [1:03:32] Yeah. **Alessio** [1:03:32] You know? It's just a notebook, so yeah, it just speaks to, like, sometimes just using the simple OpenAI model is better than, like, figuring out your own- **Michelle Pokrass** [1:03:39] Yeah **Alessio** [1:03:39] ... pipeline thing. **Michelle Pokrass** [1:03:40] Totally. **Swyx** [1:03:41] I think the top feature request there just would be, um... I mean, s- again, you know, using you as a feature request dump **Michelle Pokrass** [1:03:47] Yeah, totally **Swyx** [1:03:47] ... is, uh, is, like, being able to bias the, the vocab. I think there is, like, in raw Whisper you can do that, but- **Michelle Pokrass** [1:03:52] You can pass a prompt in the API as well **Swyx** [1:03:53] ... in, in a... But you pass it in a prompt? Okay. **Michelle Pokrass** [1:03:55] Yeah. **Swyx** [1:03:56] There's no more deterministic way to do it. **Michelle Pokrass** [1:03:57] So you... This is really helpful when you have, like, acronyms that aren't very familiar- **Swyx** [1:04:02] Yeah **Michelle Pokrass** [1:04:02] ... to the model, and so you can put them in the prompt, and it... You'll, you'll basically get the transcription using those correctly. **Alessio** [1:04:07] We have the AI engineer solution, which is just a dictionary **Michelle Pokrass** [1:04:10] Nice **Alessio** [1:04:10] ... with, like, all the way misspelt it in the past- **Michelle Pokrass** [1:04:12] Yeah **Alessio** [1:04:12] ... and then gsub- ... and, like, replace the- **Michelle Pokrass** [1:04:14] Listen, but if it works- **Alessio** [1:04:15] ... the thing **Michelle Pokrass** [1:04:15] ... it works. Like, that's- **Swyx** [1:04:16] Ah **Michelle Pokrass** [1:04:16] ... that's engineering. **Alessio** [1:04:17] Yeah. **Swyx** [1:04:17] Okay. **Alessio** [1:04:17] It's like, you know, Lama with, like, one L- **Michelle Pokrass** [1:04:20] Yeah **Alessio** [1:04:20] ... or, like, all, all these different things, or like LangChain, it, like, transcribes LangChain in, like- **Swyx** [1:04:25] Capitalization does im- is important **Alessio** [1:04:26] ... a bunch of, like, three or four different ways. **Swyx** [1:04:28] Yeah. **Alessio** [1:04:28] Um. **Michelle Pokrass** [1:04:28] You guys should try- **Swyx** [1:04:29] It works **Michelle Pokrass** [1:04:29] ... the prompt feature. **Swyx** [1:04:30] I love these, like, kind of pro-tip. Okay, fun question that I know we don't know yet, but I've been enjoying the Advanced voice mode. It really streams back and forth, uh, and it handles interruptions. How would your audio endpoint change when, when that comes out? **Michelle Pokrass** [1:04:44] We're exploring, you know, new shape of the API to see how it would, would work in this kind of speech-to-speech paradigm. I don't think we're ready to share quite yet, but we're definitely working on it, and I think just the regular request response probably isn't going to be the right solution. **Swyx** [1:04:58] For those who are listening along, I, I think it's pretty public that OpenAI uses LiveKit for the, for the ChatGPT app, which, like, seems to be the socket-based approach that people should be at least up to speed on. Like, I think a lot of developers only do request response- **Michelle Pokrass** [1:05:11] Right **Swyx** [1:05:11] ... and, like, that doesn't work for streaming. **Michelle Pokrass** [1:05:13] Yeah. When we do put out this API, I think we'll make it really easy for developers to figure out how to use it. Yeah. **Swyx** [1:05:18] Yeah. **Michelle Pokrass** [1:05:18] It, it's hard to do audio. **Swyx** [1:05:19] It'll be a paradigm change. Okay, and then I think the last one on our list was team enterprise stuff. Audit logs, service accounts, API keys, what should people know- **Michelle Pokrass** [1:05:26] Yeah **Swyx** [1:05:26] ... is in the enterprise offering? **Michelle Pokrass** [1:05:28] Yeah, we recently shipped our admin and audit log APIs, and so a lot of enterprise users have been asking for this for a while, the ability to kind of manage API keys programmatically, manage your projects, get the audit log. So we've shipped this, and, and for folks that need it, it's, it's out there, and happy for your feedback. **Swyx** [1:05:44] Yeah. Awesome. I don't use them, so I don't know. I, I imagine it's just, like, build your own internal gateway for- **Michelle Pokrass** [1:05:50] Yeah **Swyx** [1:05:50] ... you know, your internal developers to manage your deployment of OpenAI. **Michelle Pokrass** [1:05:54] Yeah, I mean, if you work at, like, a company that needs to keep track of all the API keys, it was v- it was pretty hard in the past to do this in the, in the dashboard. We've also improved our SSO offering, so that's, uh, much easier to use now. **Alessio** [1:06:05] The most important feature- **Michelle Pokrass** [1:06:06] Yeah **Alessio** [1:06:07] ... of any enterprise company. **Michelle Pokrass** [1:06:08] People love SSO. ### Personal Insights **Alessio** [1:06:10] All right, let's go outside of OpenAI. What about just you personally? So you mentioned Waterloo. Maybe let's just do why is everybody at Waterloo cracked, and why are people so good, and, like- **Michelle Pokrass** [1:06:20] Yeah **Alessio** [1:06:20] ... why have people not replicated it? Or any other commentary on your experience. **Michelle Pokrass** [1:06:24] The first is the co-op program. It's obviously really good. You know, I did six internships, learned so much in those. I think another reason is that Waterloo is, like- You know, it's very cold in the winter. It's pretty miserable. There's, like, not that much to do apart from study and, like, hack on projects, and there's this big, like, hacker mentality. You know? There's a... Hack the North is a very popular hackathon, and there's a lot of, like, startup incubators. It kind of just has this, like, startup and hacker ethos. Then that combined with the six internships means that you get people who, like, graduate with two years of experience, and they're very entrepreneurial and, you know, they're down to grind. **Swyx** [1:07:01] I do notice a correlation between climate and the correctness of engineers. Uh, so, you know, it's no coincidence that Seattle is the birthplace of Microsoft and, and Amazon. I think I had this, uh, compilation of Denmark, where, uh, people, like... Uh, so it's the birthplace of C++, PHP, Turbo Pascal, Standard ML- **Michelle Pokrass** [1:07:19] Wow **Swyx** [1:07:19] ... BNF, the, the thing that we just talked about, MD5Crypt, Ruby on Rails, Google Maps, and V8 for- **Michelle Pokrass** [1:07:25] Wow **Swyx** [1:07:25] ... for Chrome, and it's 'cause according to Bjarne Stroustrup, the creator of C++, there's nothing else to do. **Alessio** [1:07:30] Yeah. Well, you have- ... Linus Torvalds in Finland, right? **Swyx** [1:07:33] Yeah, yeah. **Michelle Pokrass** [1:07:33] Yeah. I mean, you hear a lot about this, like, in relation to SF. People say, you know, "New York is way more fun. There's nothing to do in SF," and maybe it's a little by design that all, all tech is here. **Alessio** [1:07:41] The climate is too good. **Michelle Pokrass** [1:07:42] Yeah. **Alessio** [1:07:43] If we had- if we also have fun things to do. **Swyx** [1:07:45] Nature is so nice. You can touch grass. Why- why are we not touching grass? **Michelle Pokrass** [1:07:48] But, like, you know, restaurants close at, like, 8:00 PM. **Swyx** [1:07:50] Yeah. **Michelle Pokrass** [1:07:51] Like, that's what people are referring to. There's not a lot of, like, late night dining culture or- Yeah. So you're- you have time to wake up early and, and get to work. **Swyx** [1:07:59] You are a book recommender or book enjoyer. Uh, what underrated books do you recommend most to others? **Michelle Pokrass** [1:08:04] Yeah, I think a book I read somewhat recently that was very formative was The Making of the Prince of Persia. It's a Stripe Press book. That book just made me wanna work hard like nothing I've ever read. It's just, like, this journal of, of what it takes to, like, build, you know, incredible things. So I'd recommend that. **Alessio** [1:08:20] Yeah. It's funny how video games are for a lot of people, at least for me, kind of like the, some of the moments- **Michelle Pokrass** [1:08:26] Yeah **Alessio** [1:08:26] ... in technology. Like, when I played The Sands of Time on PS2, it was, like, my first PlayStation 2 game. **Michelle Pokrass** [1:08:31] Mm-hmm. **Alessio** [1:08:32] And I was like, "Man, this thing is so crazy compared to any PlayStation 1 game." **Michelle Pokrass** [1:08:35] Right. **Alessio** [1:08:35] And it's like, wow. My expectations for, like, the technology, I, I think, like, OpenAI has a lot of similar things, like the advanced voice. It's like you see that thing and then you're like, "Okay, what I can expect from everybody else- **Michelle Pokrass** [1:08:46] Right **Alessio** [1:08:46] ... is kind of raised now." **Michelle Pokrass** [1:08:47] Yeah. **Alessio** [1:08:47] You know? **Michelle Pokrass** [1:08:47] Totally. Another book I like to plug is called Misbehaving by Richard Thaler. He's a behavioral economist and talks a lot about how people act irrationally in terms of decision-making, and I actually think about that book, like, once a week probably, at least when I'm making a decision and I realize that, you know, I'm falling into a fallacy or, you know, it could be a better decision. **Swyx** [1:09:06] Yeah. You did a minor in psych. **Michelle Pokrass** [1:09:07] I did, yeah. I don't know if I learned that much there, but it was interesting. **Swyx** [1:09:12] Did you... Is there, like, an example of, like, a cognitive bias or misbehavior that- **Michelle Pokrass** [1:09:16] Yeah, um- **Swyx** [1:09:17] ... you just love telling people about? **Michelle Pokrass** [1:09:18] Yeah, people... So let's say you won tickets to, like, a Taylor Swift concert. **Swyx** [1:09:23] Uh-huh. **Michelle Pokrass** [1:09:23] And I don't know how much they're going for, but it's probably, like, $10,000. **Swyx** [1:09:26] Oh, okay. **Michelle Pokrass** [1:09:27] Or whatever. Sure. And, like, a lot of people are like, "Oh, I have to keep these. Like, I won them. It's $10,000." But really it's the same decision you're making if you have $10,000, like would you buy these tickets? And so people don't really think about it rationally, like would they rather have $10,000 or the tickets. For people who want it, a lot of the time it's gonna be the $10,000, but their bias is because they want it. The w- world organized itself this way, and you should keep it for some reason. **Swyx** [1:09:49] Yeah, yeah. Oh, okay. I'm pretty familiar with this stuff. There- there's also a loss version- **Michelle Pokrass** [1:09:52] Yes **Swyx** [1:09:52] ... a loss of version version of this where it's like if I take it away from you- **Michelle Pokrass** [1:09:56] Right **Swyx** [1:09:56] ... you respond mo- more strongly than if I give it to you. **Michelle Pokrass** [1:09:58] Yes. Yeah, people are, like, really upset if they, like, don't get a promotion, but if they do get a promotion, they're like, "Okay, phew." It's, like, not even, you know, excitement. It's more... Like, we re- react a lot worse to losing something. **Swyx** [1:10:11] Which is why, like, when you join, like, a new platform, they often give you points and then they'll take it away if you, like, don't do some, some action- **Michelle Pokrass** [1:10:18] Yeah **Swyx** [1:10:18] ... uh, in, in, like, the first few days. **Michelle Pokrass** [1:10:20] Yeah, totally. Yeah. Uh, the book references people who w- like, operate very rationally as econs, as, like, a separate group to humans. **Swyx** [1:10:28] Oh. **Michelle Pokrass** [1:10:28] And I often think, like, you know, what would an econ do here in this moment and, and try to act that way. **Swyx** [1:10:34] Okay. Let's, let's do this. Are LLMs econs? **Michelle Pokrass** [1:10:39] I mean, they are maximizing probability distributions and- **Swyx** [1:10:42] Minimizing loss. **Michelle Pokrass** [1:10:43] Yeah. So I think way more than all of us, they are econs. **Swyx** [1:10:46] Whoa, okay. So they're more rational than us? Hmm. **Michelle Pokrass** [1:10:50] I think their optimization functions are more clear than ours. **Alessio** [1:10:53] Yeah, just to wrap, you mentioned, uh, you need help on a lot of things. **Michelle Pokrass** [1:10:56] Yeah. **Alessio** [1:10:57] Any specific roles, call-outs, and also people's backgrounds, like is there anything that they need to have done before? Like, what people fit well at OpenAI? **Michelle Pokrass** [1:11:05] Yeah. We've hired people with all kinds of backgrounds, people who have PhD in, in ML or, or folks who've just done engineering like me, and we're really hiring for a lot of teams. We're hiring across the applied org, which is where I sit for engineering, and for a lot of researchers, and there's a really cool model behavior role, um, that we, we just dropped. So yeah, across the board would recommend checking out our careers page, and, and you don't need a ton of experience in AI specifically to, to join. **Swyx** [1:11:31] I, I think one thing that I'm trying to get at is, like, what kind of person does well at OpenAI? I think objectively you have done well, and I've seen other people not do as well and, and, and ba- basically be managed out. I know it's an intense environment. **Michelle Pokrass** [1:11:44] I mean, the people I enjoy working with the most are kind of low ego, do what it takes, ready to roll up their sleeves, do what needs to be done, and, and unpretentious about it. Yeah, I also think folks that are, are very user-focused do well on, on kind of API and ChatGPT. Like, the YC ethos of build something people want is, is very true at OpenAI as well. So I would say low ego, user-focused, driven. **Alessio** [1:12:09] Um, cool. Yeah, this was great. Thank you so much for coming on. **Michelle Pokrass** [1:12:11] Yeah. Thanks for having me. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.