# ChatGPT Codex: The Missing Manual Latent Space · 2025-05-16 Josh Ma and Alexander Embiricos from OpenAI introduce ChatGPT Codex, a cloud-hosted autonomous software engineer that runs in its own sandbox, arguing it delivers value through single-shot, long-horizon task completion. They explain that Codex evolved from internal experiments (codenamed WHAM) and now offers a 60-tasks-per-hour rate limit with a one-hour hard cutoff, emphasizing it is not meant for interactive use but for delegation—users fire off tasks and go about their day. Best practices include using agents.md, installing linters and formatters, keeping code modular, and having good architecture. The team intentionally kept it a research preview to gather feedback on environment customization and pricing, with plans to integrate seamless transitions between cloud and CLI. ## Questions this episode answers ### What are the best practices for using ChatGPT Codex effectively? Alexander and Josh recommend setting up an agents.md file with project-specific instructions, installing linters and formatters so the agent can self-verify, making code modular and well-typed (e.g., TypeScript over JavaScript), and giving clear scope by referencing directories. They note that good architecture and intentional naming, like using a unique string never appearing elsewhere in the codebase, help the agent navigate efficiently. [15:08](https://addtry.com/aa11054c-2224-4791-8b10-b8e6b5b63aa6?t=908000) ### How is ChatGPT Codex meant to be used differently from other AI coding assistants? Wigs realized he was using Codex wrong by treating it like Cursor and watching it code. It’s designed for one-shot autonomous delegation: you fire off a task and walk away, even using it from your phone to reinforce a hands-off mindset. Josh and Alexander encourage an “abundance mindset”—sending many tasks in parallel rather than carefully crafting each prompt. [40:20](https://addtry.com/aa11054c-2224-4791-8b10-b8e6b5b63aa6?t=2420000) ### What is agents.md and why is it not just README.md? Alexander explains they considered naming it README.md or CONTRIBUTORS.md, but agents need different instructions than humans—for example, they don’t need code style guidance because they infer it from the codebase. They chose a non-branded name to keep it universal across agent tools, and because it’s distinct enough that agents can find and grep for it efficiently. [22:52](https://addtry.com/aa11054c-2224-4791-8b10-b8e6b5b63aa6?t=1372000) ## Key moments - **[0:00] Intro** - [0:52] "Feedback is a gift. Thank you." - [1:59] Alexander Embiricos worked on pair programming tool Multi before joining OpenAI. - [2:34] Alexander Embiricos defines an agent as a reasoning model with tools, environment, guardrails, and task-specific training. - [3:05] Alexander Embiricos recalls a demo where Scientist updated its own code, sparking an 'AGI moment'. - **[3:08] Origins** - [4:21] Codex CLI's full auto mode increases sandboxing for safety, says Alexander Embiricos. - [5:29] Josh Ma joined OpenAI two months ago after founding dev tool platform Airplane. - [6:01] Josh Ma vs Alexander Embiricos debated cloud vs CLI form factor, settling on needing both. - [6:53] Josh Ma predicted an agentic software engineer would emerge within two years, leading him to join OpenAI. - **[8:55] Agent Design** - [10:26] Alexander Embiricos says SWE-bench passing agents often produce ungroomed code, so Codex prioritizes style adherence. - [12:07] Codex generates PR descriptions with test status checkboxes and log citations, says Alexander Embiricos. - **[14:53] Best Practices** - [14:53] Josh Ma explains agents.md provides a hierarchy of instructions that Codex agents understand. - [16:47] Alexander Embiricos compares a base reasoning model to a college grad; Codex adds 'first few years of job experience' through training. - [20:11] Codex internal codename WHAM was chosen because the string is unique in the codebase, improving agent search efficiency. - **[21:10] Human-AI** - [22:30] Q: Why agents.md and not README.md? A: Alexander Embiricos says agents need different instructions than human contributors. - **[23:58] AI Future** - [25:52] Q: How does Codex handle long agents.md within context limits? A: Josh Ma says it fits and the agent greps for the file. - **[28:18] Planning** - [29:12] Alexander Embiricos: 'The model isn't all the product, but the model is the product' — Three-party decisions among user, developer, and model. - **[35:28] Long View** - [37:03] Q: What's the hard cutoff for a Codex task? A: Josh Ma says currently 1 hour, but average is under 30 minutes. - **[39:31] Usage Tips** - [39:37] Alexander Embiricos advises an abundance mindset: fire off many Codex tasks without overthinking prompts. - [41:25] Q: What access does the Codex agent have to the compute platform? A: Josh Ma says human sets up environment, agent runs sandboxed with no internet. - **[42:32] Compute** - [46:10] Alexander Embiricos sees Codex Research Preview as a thought experiment for purest autonomous AGI collaboration. - [47:18] Q: What features would move Codex from Research Preview to full release? A: Alexander Embiricos lists multimodal inputs, network access, and IDE integration. - **[48:01] Wrap-up** - [48:38] Q: Why is 'finding a typo' an onboarding task? A: Josh Ma says it was a meta joke that also shows agent search capability. - [49:58] Q: What feedback does the Codex team want? A: Josh Ma wants environment customization feedback; Alexander Embiricos wants workflow integration ideas. ## Speakers - **Alessio** (host) - **Wigs** (host) - **Alexander Embiricos** (guest) - **Josh Ma** (guest) ## Topics Coding Agents ## Mentioned Airplane (company), OpenAI (company), 4o (product), Aider (product), ChatGPT (product), ChatGPT Codex (product), Claude (product), Cloud Code (product), Codex CLI (product), Cursor (product), Deep Research (product), Devin (product), Factory (product), GPT-3.5 (product), GPT-4.1 (product), Linear (product), Multi (product), Operator (product), SWE-Bench (product), o3 (product) ## Transcript ### Intro **Alessio** [0:07] Hey everyone, welcome to the Latent Space Podcast. This is Alessio, partner and CTO at Decibel, and I'm joined by my co-host Wigs, founder of Small AI. **Wigs** [0:16] Hello, hello. Calling in from Singapore here, but we are in the remote studio because, uh, the OpenAI team keeps shipping, and today they just re- uh, live-streamed and released ChatGPT Codex. Welcome, uh, to Josh, um, who I think y- we've talked about-- We, we've met while you were at Airplane, right? **Alexander Embiricos** [0:34] Yeah, yeah. You know, I've been building dev tools for a bit now, and, uh, you're a guy to talk-- that I have to talk to when I'm building dev tools. **Wigs** [0:42] Uh, I mean, you know, you s- you have now seen me complain a lot, uh, when, uh, when things happen. So, uh, I, I don't know if it's like a, a good, good or bad thing. Um, and then- **Alexander Embiricos** [0:52] It's a gift, man. Feedback is a gift. Thank you. **Wigs** [0:55] Um, and Alexander, uh, we're, we're new to each other, but you've been leading a lot of the, uh, Codex testing and, and demos and stuff. **Alexander Embiricos** [1:02] Yeah. Hey, I'm Alexander. I'm on the product team here. **Wigs** [1:05] Awesome. Um, so yeah, um, uh, we- we're gonna just assume that everyone's watched the live stream. You also released, uh, a blog post with a bunch of like test demo videos. Um, basically a bunch of like-- It, it's very, it's very interesting. I noticed in the demo videos it was like individual engineers sitting by themselves, very lonely, and then they're just talking to their AI friends, coding with them. I don't know if that's the vibe you wanna go give off, but like that's how it came across. **Alexander Embiricos** [1:31] Fair. Yeah, man. We-- Those, those videos we were going for like maximum authentic, just like- ... engineers talking about how it helps them. Um- Yeah, I'll take the feedback. **Wigs** [1:41] Uh, but no, I mean, it's true. I mean, sometimes, you know, on-call is a lonely job. Like mobile engineer is a lonely job. Like there's not that many of those. So yeah, totally. Um, but anyway, so, uh, what did you guys individually do? Maybe, maybe we can kind of start there. How, how did you get pulled into the project? Um, and, and w- w- we'll start from there. **Alexander Embiricos** [1:59] Yeah. Maybe I can go first 'cause then we have a, have a fun story about how we started working together. Um, okay. So actually before working at OpenAI, I was working on a native macOS software called Multi, which was like about, um-- Like it was kinda like a pair programming tool, but we thought of ourselves as working on like, um, like human-to-human collaboration. And then basically, um, as, as ChatGPT and stuff came around, we started thinking about like, "Oh, what if instead of a human pair programming with a human, it was like a human pair programming with an AI?" Um, so I'll skip this whole journey, but that was this whole journey, and then we all ended up joining OpenAI. And I was work- mostly working on desktop software. Um, and then we shipped reasoning models. And, you know, I'm sure you guys were like ahead of the curve in terms of understanding the value of reasoning models, but for me, like it's kind of like starts off as better chat, but then when you can give it tools, you can actually like make it an agent, right? Like an agent is like a reasoning model with like tools and environment, um, guardrails, and then maybe like training on like specific tasks. So anyways, we got super interested in that, and we were just like starting to think about like, "Okay, how do we bring reasoning models, um, into desktop?" Um, and at the same time here at, at, at OpenAI, there was a lot of experiments going on with like giving these reasoning models access to terminals. Um, I wasn't working on those first experiments, uh, to be clear, but, but those were like-- that was like the first true like, "Wow, I really feel the AGI moment," that I had. ### Origins **Alexander Embiricos** [3:22] It was actually while I was ta- I was talking to, uh, David Kay, um, a designer who was working on this thing called Scientist, and, um, he showed me this demo of it updating itself. And like nowadays, like I don't know if any one of us would be like the most impressed to like change the background color. **Wigs** [3:38] Uh, modifying its own code? **Alexander Embiricos** [3:39] Yeah. And then, you know, it was like they had hot reloading set up. So- **Wigs** [3:43] Wow **Alexander Embiricos** [3:43] ... I was just like mind blown at the time. And I, I... You know, it's still a super cool demo. Um, and so we kind of were experimenting with a bunch of these, and I, I sort of joined one of the teams that was like tinkering with this. Um, and you know, we kind of realized like, "Hey, it's just like super valuable to figure out how to give an, um, a reasoning model access to, um, to a terminal." And then now we have to figure out like how to make that like a useful product and how to make it safe, right? Like you can't just let it go loose on your local file system, but that's like where people were initially trying to use it. So a lot of those, those learnings ended up becoming the Codex CLI, which shipped recently. Um, you know, a lot of the work there, like, you know, the, the thinking that I'm like most proud of is like enabling things like full auto mode and like when you do that, we actually like increase the amount of sandboxing so that's still safe for you. Um, and then so, you know, we were working on these types of things, and then we started realizing, uh, we want to let the model think for longer. We want to have a bigger model. Um, we want to let the model like do more things safely, like without having to do any approvals. Um, and so we thought, um, you know, maybe we should give the model its own computer, um, the agent its own computer. And then at the same time, we were also experimenting with like putting the CLI in like our CI so it could automatically fix tests. We did this like crazy hack to like get it to automatically fix like linear tickets in our issue tracker. Um, and so then we ended up sort of like creating this, this, uh, this project, um, that is Codex, uh, which is basically like really the concept of giving, uh, the agent access to a computer. Um, actually, I realize now, I don't know if you were asking what I personally did, but anyways, I told the story. I hope that's okay. **Wigs** [5:21] Sure. No, it's, it-- I mean, you weave your personal story into the larger narrative anyway, but yeah, uh, and I'm sure Josh has a, a part too. **Alexander Embiricos** [5:29] Yeah, yeah. So my story is, uh, somewhat different. Um, I've been at OpenAI for two months here, and it's been one of the most fun, chaotic, um, uh, two months of my life. Um, but maybe I'll start back at the company I had founded a few years back called Airplane. Uh, we were building a internal tool platform. Um, the idea is to let you build internal tools but really lean into developers, um, and make that really easy. And it, it, it sounds unrelated, but in many ways, like there were-- the similar themes started coming up. Um, what's the right form factor for doing local development? How do you deploy tooling to the cloud? How do you run code in the cloud? How do you- **Josh Ma** [6:07] Compose all these primitives of storage and compute and UI to let developers build software really quickly. Um, I like to joke that we were just, I don't know, two years too early. Um, towards the end, we were playing around with, like, GPT-3.5 and, um, you know, trying to really make... Um, it was really cool. It could actually build, like, a React view really quickly, right? Um, and I think if, you know, if we had, uh, kept going on it, maybe it would've turned into some of the, um, AI builders that you see today. But, um, that company ended up getting acquired by Airtable, where I ran some of the, um, the AI engineering teams, uh, there. And, um, for me personally, towards the beginning of this year, I saw the progress we were making in software, uh, agentic software development. And for me, it was a bit of, like, my own moon landing kind of moment that I suspected was about to happen, right? Whether or not I was involved in the next two years, I think we were g- we are going to build an ag- agentic software engineer. And so I talked to my friend at OpenAI, and I was like, "Hey, are you guys working on something like this?" And, you know, he gives me a wide-eyed look. He's like, "I'm not allowed to tell you anything, but maybe you could talk to the team." Um, and so very fortunately, this was right when, you know, Alex and folks were spinning up things and, um, I remember actually, you know, in our interview, we riffed on the form factor, right? Should it be CLI? The issues with that, waiting for it to finish and not be able to interrupt all the time, wanting to run it four times, 10 times in parallel. Um, and you know, at that point, I said, "Maybe it should be both." Uh, and we sort of are, you know, going for that right now. But, um, yeah, um, I don't know. I'll just say, like, was very excited and still very excited to just be, um, pushing this forward, and I think Codex is still real early. Excited to share it with the world, but there's a lot more to build. **Alexander Embiricos** [7:52] Yeah. **Josh Ma** [7:53] Yeah. **Alexander Embiricos** [7:53] I'll say it was, it was a very fun conversation when we first met because you came in-- I've never had this happen before. It's like, "Here's exactly, like, kind of the, the, the, the change that I see in the world and therefore the type of product that I wanna build. I know you can't confirm if you're working on it, but just so you know, this is the only thing I wanna work on." And then you-- and then I was like-- I asked just a few open-ended questions, and then we immediately got into, like, some of the core debates around the form factor of the tool. And I was like, "Okay, this is awesome to work together." **Josh Ma** [8:19] Yeah. **Wigs** [8:19] I think a dev tools person can spot another dev tools person- **Alexander Embiricos** [8:22] Yeah **Wigs** [8:22] ... like, like that. **Alexander Embiricos** [8:24] Yeah. Blink twice if you're working on this. **Josh Ma** [8:26] Yeah. So- **Wigs** [8:26] Um, uh, but for what it's worth, uh, early, early iPhone team, uh, at Apple was the same, because iPhone team members did not know if they were on the same team. They're not allowed to tell each other, so they had to, like, triangulate. **Josh Ma** [8:38] Wow. **Wigs** [8:40] Wow. **Josh Ma** [8:40] It's like a two-way- **Wigs** [8:41] Anyways, um- **Alexander Embiricos** [8:42] And talking about form factor, so you mentioned the CLI, which you already released, and I think there's other, you know, Cloud Code, Aider, a bunch of other tools out there. Should people think of Codex and ChatGPT as like a hosted Codex CLI? Like, are there big differences between the two? Um, let's talk about that. ### Agent Design **Josh Ma** [9:01] Yeah. **Alexander Embiricos** [9:01] Go for it. **Josh Ma** [9:02] Yeah. Um, I-I think of it as, um... I think that's a short of it, right? Allowing you to run Codex agents, um, in OpenAI's cloud. Um, but I think that the form factor, like, it's a lot more than just where the computer runs, right? It's, um, how, how does this bind to the UI? How does this scale out over time? Um, how do you manage, uh, caching and permissioning, and how do you do the collaboration story? And so I-- let me know if you disagree, but I-I think the, really it's like form factor is the core of it. **Alexander Embiricos** [9:36] Yeah. It's like it, it was pretty f-- It's been honestly a really fun journey. Like I-- we-- the other day or maybe last night, like in the AM. **Josh Ma** [9:44] Yeah. **Alexander Embiricos** [9:44] Josh was sleeping 'cause he had to do the live stream. I didn't have to. But anyway, a bunch of us were, like, looking back at the doc where we planned what we were gonna ship. Um, and we were like, "Man, we, uh, had a lot of scope creep." And e-effectively, all that scope creep was kinda like incrementally made sense because we kept leaning further and further into this idea that this is, like, not just, like, a model that's good at coding, but rather this is an agent that is good at, like, independent software engineering work. And, like, the more we leaned into that, the more, like, things started to feel really special. So, you know, uh, we could... I'll-- I'm gonna just label and then set aside, like, the entire conversation around, like, the compute, uh, platform that Josh has, like, you know, been leading. Um, but, uh, let's just take the model, for example. Um, you know, we don't just want it to be co-good at code, and, like, we don't just want it to, like, solve, like, say, like, SWE-bench tasks. You know, SWE-bench is an eval for those who don't know that's like, has a certain way of, like, functionally grading outputs. Because if you look at a lot of, like, SWE-bench passing, like outputs from, like, an agent, they're not really like PRs that you would merge because, like, the code style might be, like, different. Like, it works, but the code style is different. Um, so you know, we spent a lot of time, like, making sure that our model is, like, great at adhering to instructions, uh, great at inferring code styles so that you don't have to tell it. Um, but let's say that you got then a PR, that it was like the code style was good, it followed your instructions well. It still might be really hard to merge if you have this, like, enormous description, like, just, like, model, like, of, like, how it thought about building it. Um, and you, you know, you probably need to pull it onto your computer to, like, test the change and validate that it works. And, like, maybe that's okay if you're just, like, running o- like one change. But in a, in a future world that we imagine where, um, actually, um, you know, maybe the majority of code is actually being written by agents that we're delegating to, you know, doing tasks in parallel, it becomes, like, critically important that you can actually, like, integrate those changes easily as the human developer. So, for instance, some of the other stuff we started to train was, like, PR descriptions. Like, let's really nail this idea of, like, a good, concise PR description that, like, highlights the relevant things. So our model will actually write, like, a nice short PR description with, like, a PR title that, like, adheres to your, like, you know, repo f- uh, format. We have a way to prompt that more if you want with agents.md. And then in the PR description, it'll actually cite, like, relevant code that it found along the way or relevant code in its, in its PR, so that you can, like, mouse over and just see it. Um, you know, and perhaps my favorite thing is actually the way we handle testing. So the model will attempt to test its change. Um, and then it will tell you in this, like, really nice way with just, like, a checkbox kind of thing, whether or not those tests passed. And again, it will cite if the test passed, like a deterministic l- reference to the log, so you can, like, read it and be like, "Okay, I know that this test passed," right? Or if the test failed, it'll be like, "Hey, like, I- this didn't work. I feel like you need to install, like, pnpm or whatever," and you can, like, read the log and, like, see what it is. Um, so those are some of the things that I think, um... Anyway, I've lost track of the original question. But anyways, those are some of the things that we like- **Josh Ma** [12:43] Yeah **Alexander Embiricos** [12:43] ... we're really leaning into as we build this, like- **Josh Ma** [12:45] Yeah **Alexander Embiricos** [12:45] ... basically, like, software engineering agent in the cloud. **Josh Ma** [12:47] I, I think also just, like, it feels very different, right? You can, like, look at the features, but I think for me the feeling is, like, there's... It takes a leap of faith the first few times. You're like, "I'm not really sure if this is gonna work." And it goes off for like 30 minutes, right? But then you, it comes back and it's like, wow, this, this, this, this agent went out, wrote a bunch of code, wrote scripts to help code mod its own changes, right, tested this, and it really went through the full end-to-end of thinking about the change it wants to make. And I, you know, had no faith that at the start that, um, that it was gonna be able to successfully do it. And then after using it a bit, you're like, "Wow," like, it, it actually, you know, um, uh, pulled through. And so that kind of long-running independence is something that's, like, hard to really, like, summarize. You have to really try it. Um, but it finally feels very different. **Alexander Embiricos** [13:36] Yeah. **Josh Ma** [13:36] Um, and yeah, that feels special. **Alessio** [13:39] It-- Yeah, I used it. I opened a PR for it a few minutes ago. **Josh Ma** [13:42] Yeah. **Alessio** [13:43] Uh, I was in the lucky first 25% of people to get the rollout. Uh, yeah, it's very nice. Uh, you know, it kinda shortcut it because it, it couldn't figure out how to run rspec in Rails- **Josh Ma** [13:52] Yeah **Alessio** [13:52] ... and so it just checked the syntax of the Ruby file and it was like- **Josh Ma** [13:55] Just- **Alessio** [13:56] ... "Looks good to me." But I think it doesn't have the agents.md yet, so I think once I set that up, that'll be good. Um- **Josh Ma** [14:01] No, just, uh, just don't use Ruby, man. Like, the Python is fine. **Alessio** [14:04] Oh, yeah, that... It's skill, skill issues. O- once it's good enough to migrate the whole thing, then I'll do that. Um- **Alexander Embiricos** [14:10] I mean, it is funny that there's, on, just briefly on the note of, like- **Alessio** [14:13] Yeah, yeah **Alexander Embiricos** [14:13] ... don't use Ruby or not, there's, like, a bunch of things that I think teams can do to, like, make better use of AI agents. Like- **Josh Ma** [14:18] Oh, please **Alexander Embiricos** [14:19] ... as we sort of heard of today. **Alessio** [14:19] Stop using Ruby. Number two out of the team. **Josh Ma** [14:22] No, no, no. But yeah, if you could list, uh, some things out- **Alessio** [14:23] Yeah **Josh Ma** [14:23] ... that's, that's... Like, best practices. You know, I, I noted from the live stream that, uh, they mentioned pro users install linters and formatters so that basically these are in the loop verifiers that the agent can kinda use. **Alexander Embiricos** [14:35] Yeah. **Josh Ma** [14:35] Right? So, uh, which, like, you know, it turns out to be dev best practices as well, but now the agents can also auto use it. Commit hooks have always been a tricky thing for humans, because I've, I've been on teams that are like, "No, everyone, everything has to have a commit hook," and then I've also been on teams that are like, "No, like, this thing gets in the way of committing, so let's rip everything out." ### Best Practices **Josh Ma** [14:53] But actually for, uh, for agents, it's actually really good to have commit hooks. Yeah, I mean, you took the words out of my mouth. I, I think the three I was going to say would be, one, agents.md and, um, like, we put a lot of effort into making sure the agent, like, understand this hierarchy of instructions, right? You can put them in subdirectories and it'll understand which ones take precedence over, um, what, which others. Um, so over time, right? Like, I mean, we, we also have, like, o3 and 4o writing our agents.md, uh, files for us. **Alessio** [15:23] I love these, uh, I love the tips. You, you actually out- open sourced the, the, the, the prompt descriptions here. **Josh Ma** [15:29] Yeah. Yep, yep. **Alessio** [15:30] Any, anything you can highlight? **Josh Ma** [15:32] And so- **Alessio** [15:32] Yeah. **Josh Ma** [15:33] Um, I mean, I think I would start simple and not try to overdo it and, like, a simple agents.md will get you a long way rather than no agents.md. Um, and then, uh, it's more of, like, you learn over time, right? **Alessio** [15:46] Okay. **Josh Ma** [15:46] What we would really like to do is auto-generate this at some point for you based on the PRs you create and the feedback you give. But, um, you know, we figured we'd ship faster, uh, rather than later. Um, but, uh- **Alessio** [15:57] And you mentioned you have o3, o4 writing agents.md for you as well. **Josh Ma** [16:01] Yeah, like, I'll, like, give it my entire directory, right, and let it... and just say like, "Hey, produce an agents.md." Well, well, actually, these days I'm using Code One to do it 'cause it can tra-traver- uh, Codex One, sorry, uh, to traverse, like, your directory tree and generate those things for, for you. So, um, yeah, I would recommend, um, in-- slowly, gradually investing in agents.md and then, you know, you took the words out of my mouth. Like, getting very basic linting, formatting up, um, nets you really big wins 'cause, um, it's similar to how, like, if you open a new project in VS Code, right? Like, you get some out of the box checking. Um, the agent's starting, like, as a human, you're sort of starting without that, um, advantage. Um, and so this is trying to give that back, uh, to the agent. Um, and, uh, yeah, I don't know. Do you have anything else? **Alexander Embiricos** [16:47] Yeah. So, so one analogy there, and then actually I, I have just some thoughts we've observed of even, like, using other coding agents, like just any coding agent, you know, how to, how to prepare for that. But, like, you know, the analogy that I kinda like is, like, y- you... So if you start with, like, a base reasoning model actually, you basically have this, like, really precocious, like, incredibly intelligent, incredibly knowledgeable and, like, weirdly spikily intelligent, you know, um, college grad. You know? And, but we all know if, if you hire, like, that person, uh, and put them, like, ask them to do software engineering work, like, independently, like, there's just a lot of practices that they're not gonna know about. And so kind of a lot of what we've done with, with Codex One is basically give it its first few years of job experience. Uh, and, like, that's effectively what, what the training is, right? So that it just, it kinda knows more of these things. And, like, if you think about it, like, a PR description is an classic example of that, like writing a good PR description, right? And prob- possibly knowing what not to put in it, actually, right? Um, and then, uh, so that's, that's what, uh, what you get there. So now you have this, like, this, like, weirdly, weirdly knowledgeable, spikily intelligent college grad with a few years of job experience, and then every time you kick off a task, it's kind of like their first day at your company, right? And so agents.md is basically a method for you to kind of compress that, like, test time exploration that it has to do so it can, like, know more. And, and as Josh said, obviously we wanna, like... Right now it's like this research preview, so you have to, like, update it yourself, but we- there's, like, a lot of ideas we have for how to make that automatic. Um, so that's just a fun analogy. Um- **Josh Ma** [18:16] Yeah. Maybe the last one I'll say is, like, make your code base discoverable. All right? **Alexander Embiricos** [18:20] Mm-hmm. **Josh Ma** [18:20] It's like the equivalent of- Maintaining good engineering practices for, uh, new hires that you make, allowing them understand your code base faster, right? A lot of my prompts start with, like, "I'm working in this subdirectory. Um, here's what I'd like to accomplish," right? "Can you please, um, do it for me?" Um, and so giving that guidance, that scoping, uh, helps ton. **Alessio** [18:40] Yeah. **Alexander Embiricos** [18:41] Okay, I'll give you three, sorry, three things for, like, generally. So first, like, language choice. I was hanging out with a friend the other day who's a bit of a latecomer to AI, and he was like, "Oh yeah, I wanna try building, like, an agents product. Uh, sh-like, should I build it in JavaScript?" And I was like, "You're still using JavaScript? Like, no wonder, like you just, like, you know, u-use at least TypeScript. Like, give it some types." So, I mean, I think that's a basic one. I don't think anyone listening to us now needs to, like, be told this. Uh, another one is, like, you just make your code modular, right? The more modular and testable it is, the better. But you don't even have to write the test. Like, an agent can write the test, but you kinda need to design the architecture to be modular, right? Um, I saw this presentation, uh, recently by someone here who was like, bas-- they weren't vibe coding, it was, like, professional software engineer, but using, like, tools like Codex to, like, build a new system. Uh, and they got to build a system from scratch, and there was kind of this graph of, like, their commit velocity. And then their system had, like, some traction, so then it was like, okay, now we're gonna port it into, you know, the monolith that is the overall ChatGPT code base, uh, that, you know, has seen ridiculous hyper growth, and so maybe is not, like, the most architecturally pre-planned. And, like, their commit rate, the same engineers, same tooling, actually, the AI tooling continues to improve, their commit rate just, like, plummets, right? Um, and so I think the other thing is just like, yeah, architecture, like, good architecture is, like, even more important than ever, and, like, I guess the fun f-thing is, like, for now, that's something that humans are really good at. So, like, you know, kind of good, you know, important for the software engineers to do their job. Um- **Josh Ma** [20:09] I don't know. Just don't, don't look at my code base. **Alexander Embiricos** [20:11] Yeah. Well, definitely won't mind. But the last thing is just kind of a fun story, which is the code name, the internal code name for our project is WHAM, like W-H-A-M. Um, and we chose it, uh, actually, um- **Alessio** [20:24] Wake me up wham. **Alexander Embiricos** [20:25] I was working with the, uh, research lead, and he was like, "Hey, make sure you grab the code base before you chose the co- choose the code name." So we, like, searched the code base and there was... the word, the string WHAM was, like, only present, like, in a few larger strings and never present as its own string. And that means that whenever we prompt, we can be very efficient. We can just say, "In WHAM," right? And then WHAM code that is, like, you know, for our web, uh, code base or our server code base or in, like, our shared types or anywhere else, is, like, really efficient for the agent to find, right? Whereas let's pretend, like, alternatively we would've called our product, like, ChatGPT Code, you know? Not saying we didn't consider that. Uh, then it would be super hard for the agent to figure out where we wanted to direct it to, and so we'd probably have to provide, like, more relative, like, folder paths. ### Human-AI **Alexander Embiricos** [21:12] So there's a lot of this stuff, like, as you start to think ahead, like, "Oh, I'm gonna have an agent that's gonna be using terminal to grab," uh, then, you know, you can start, like, naming things intentionally. **Alessio** [21:20] Would you name-- would you start naming things less for humans readability and more for agent readability? Like, what's kind of the trade-off in your mind? **Alexander Embiricos** [21:28] Hmm. **Josh Ma** [21:29] Yeah. I-- It's interesting 'cause I, I definitely had different priors coming into OpenAI. Uh, like, I currently believe that the systems are actually very convergent. Like, there's a lot of... Maybe it's because as long as you see humans and AI writing it, like maybe there's a world where it's only AI's maintaining a code base and the assumptions change. But the moment you start to have to break that fourth wall and a human's coming in, doing code review, deploying the code, like, you nee- it has human fingerprints all over it, right? And so how humans communicate to AI, where to make the change, how humans communicate a bug that needs to be done or communicate business requirements, right? All those things aren't going to go away immediately. And so I think the whole system still feels actually very human. Um, I think there's a cooler answer I could say that it's like, oh, no, it's like this alien thing, it's completely different. But I don't know. I think it's like these are started off as large language models. There's a lot rooted in, uh, human, human, human communication. **Alexander Embiricos** [22:24] Yeah. **Josh Ma** [22:25] Yeah. **Alexander Embiricos** [22:25] Um, by the way, if you-- if there's somewhere you wanna take this, you should actually cut us off- -'cause I realize we're just kinda monologuing between each other. But it's- **Alessio** [22:30] No, no. I think, uh, I, I think this also ties to the agents.md, right? It's like, why is it called agents.md and not README.md? There's kinda like, I guess in your mind, some fundamental difference with how the agents and the human consumes the information. So I'm curious if you think that's at the class naming level, it's just at the instruction level. Like, where, where does it break down? **Alexander Embiricos** [22:50] Mm-hmm. Can I... Yeah. **Josh Ma** [22:51] Go for it. **Alexander Embiricos** [22:52] Okay. So, so like a few options for this naming, right, which we considered. So you could go for reading, README.md, you could go for CONTRIBUTORS.md, right? You could go for like codex-agent.md and then like maybe like codex-cli.md is like these two separate files, right? Um, but like that are sort of like branded. **Josh Ma** [23:09] There's also cursor rules, windsurf rules- **Alexander Embiricos** [23:10] Yeah **Josh Ma** [23:11] ... all, ev-ery- **Alexander Embiricos** [23:11] Totally **Josh Ma** [23:12] ... has rules. Yeah. **Alexander Embiricos** [23:13] Yeah. And then you could go for agents.md, right? And so like there, there are a few trade-offs here. I guess one is like openness, like, a-and, and one is specificity, I suppose. And so when we thought about it, uh, you know, we, we thought about, well, probably there's things that you wanna tell an agent that you don't need to tell a contributor. And similarly, there's things you wanna tell contributors to like really like help them set up in your repo or whatever, that you don't need to tell the agent, the agent can just figure that out. So we were like, okay, maybe this is gonna be different. And like, you know, the agent's gonna read your README anyways, so like maybe agents.md ends up being like the stuff that you need to tell the agent, uh, that it's not like automatically figuring out from the README. So we kinda made that decision. Uh, then we considered, um, okay, there are, there are like different form factors of agents, right? Like the most special thing about what we're, what we are building and, and shipping is like, it's just like a out-of-the-box way to use a cloud-based agent that can do many tasks in parallel and can like think for a long time and can use a lot of tools safely, right? ### AI Future **Alexander Embiricos** [24:11] And so we thought like, well, you know, how, how fundamentally different is, is the set of instructions that you wanna give that from an agent that you're working with more collaboratively on your computer? Uh, we had a good amount of debates about that, to be completely honest. Um, and then we ended up concluding, like, actually those sets of instructions aren't different enough that we need to namespace this file. If there is something you need to namespace, you could probably just, like, say it in plain language within the file. Uh, then the last thing we consider is, like, well, okay, how different do we think the instructions you have to give, like, our agent are to the instructions you might give to an agent running on a different model or built by a different company? And w-we just think it kind of sucks if you have to, like, create, like, all these different agents and whatever, you know? It's like part of why we made the Codex CLI open source is, like, a lot of problems, like safety issues that you need to figure out for how to deploy these things safely, and no one should have to figure these out, like, more than once. So that's why we went for, like, a non-branded, uh, name. **Josh Ma** [25:02] And I, I have one specific example of why, um, you know, README and agent.md are different. Um, for agents, I don't think you really have to tell it code style. It, it looks at your code base and just writes code that's consistent to that. Um, whereas, like, a human's not going to take its time, um... sorry, their time to, uh, to go through the code base and, uh, you know, follow all the conventions, right? So that's just one example, like, you know, at the end of the day, like, there are differences between how these two kinds of, um, uh, developers, uh, approach it. **Wigs** [25:38] Um, cool. Uh, I, I think those... that's a really good, um, set of advice. Uh, I think you just gave us our s- episode title. Like, we're just gonna call it Best Practices for Using ChatGPT Codex and, you know, I mean, I think people are gonna want best practices. I, I... so I noticed, like, something that's very interesting, right? Like, um, I think there's always a two versions in terms of building agents. One which is you try to d- be more controlling. You're, you try to make it more deterministic and then the other, you try to just prompt it and trust the model. And I think your approach is very much prompt it, trust the model. Uh, I see inside of the agents.md system prompt that if, you know, you just, you just prompt it to, like, behave the way that you want and you hope, y- I mean, you just expect the model to behave it. Obviously, you have control of the model, so you can train it if, if it doesn't do well. Um, but like, one, one thing that, um, makes me question it is, um, how do you fit everything in context? Like, what if I just have a super long agents.md? You know, I s- in your live stream you had, uh, it demoing on, like, the OpenAI model, uh, model repo, uh, which is just giant, right? Like, so, uh, how do you manage caching and, um, context windows and all that? **Josh Ma** [26:49] Yeah. I mean, would you believe me if I told you right now that it all sti- fits in the context window? Um, I'll say- **Wigs** [26:56] For, not, not the OpenAI repo. **Josh Ma** [26:58] No, but sorry. Everything that the agent needs, um, to- **Wigs** [27:01] For... right. So, so you reify the agents.md, put it at, at the top, right? It's, it's just like another system prompt. **Josh Ma** [27:06] No, actually. It's a file- **Wigs** [27:08] Okay **Josh Ma** [27:08] ... uh, that the agent knows how to, like, grep and sed for, right? Um- **Wigs** [27:12] Okay **Josh Ma** [27:12] ... and read it. Uh- **Wigs** [27:13] Okay **Josh Ma** [27:13] ... because there mu- might be multiple ones. Um, and so you okay, you can actually see it in the work log, right? It's, like, gonna look for it. It very aggressively looks for an agent.md. It's been trained to do that. Um, I'll say it's been really interesting, um, joining OpenAI and, like, seeing how, um, you know, when you're thinking about where models are going and what AI products will look like years from now, um, you know, uh, you design products in a different way, right? Um, like, before OpenAI, when, especially when you don't have access to a team of researchers and many, many GPUs, um, you're building these deterministic programs, right? A lot of scaffolding around how this operates. Um, but you don't really let the model, um, operate at its fullest, um, capacity, right? A lot of, um... it was interesting when I just joined, actually. I got a lot of pushback saying like, "Hey, why don't we just, like, hard code? Like, listen, you keep using this tool wrong. Let's just say in our prompt, 'Don't do that.'" And then the researchers will, will be like, "No, no, no, we don't do that. We're like, we're gonna do it the right way. We're gonna teach the model why this is the right way to do it." Um, and I, I think that's, like, related to this overall th-thought, like, where do you put the deterministic guardrails in and where do you really let the model think, right? ### Planning **Josh Ma** [28:23] Um, similar conversation around planning. Should we just have an explicit planning stage where it's like, think out loud first, write down what you're gonna do, and then go do it? Sure. But what if the task is really easy, right? Do you really want it thinking this whole time? What if it needs to, like, replan as it goes? Like, do you have all these, like, if/else conditions, heuristics to do that or do you train a really good model that knows how to switch between those, um, modes of thinking? And so, um, it's tough. Like, I, I definitely have advocated for, like, little guardrails here and there until, like, the next training run's done. Uh, but, um, I, I think that's really, like, we're really building for this future where the model is able to make all these decisions. Um, what's really important is that you give it the right tool, right? You give it ways to manage context, manage memory, manage, um, you know, uh, ways to explore the code base. Um, those still are really important. **Alexander Embiricos** [29:12] Yeah. That's like, yeah, super well said. Like, I think building here is, like, super fun and different and like, you know, the model isn't all the product, but the model is the product. Right? And you kind of, like, need to have this kind of, like, humility, um, in terms of, like, thinking about, like, okay, well, what are the things that, like, we-- there's, like, three parties, right? There's the user, the developer, and, like, the model maybe, right? What are the things that the user, like, just needs to decide up front? And then what are the things that, like, we, the developer, are gonna be able to decide better than the model, and then what are the things that the model can just decide best, right? And, like, that you kind of every decision is, like, just has to be one of those three. And, you know, it's not like everything's the model. Like, for instance, um, we, uh, have two buttons in the UI right now, like ask and code, and, like, you know, those, those probably could get inlined into, into, like, the decisions the model makes. But, uh, you know, right now it was just really, like, it made sense to kind of just give the user choice upfront because we spawn a different container for the model first based on what button you press. So, like, if you, if you ask for code, we put all the dependencies in. I'm, I'm gonna oversimplify here, but if you don't ask for code, if you're just asking a question, we, like, do a much quicker container setup before the model, like, gets any choice. Um, and so, you know, that's maybe a user decision. Um, there's some places, uh, where, you know, user and developer decisions kind of come together around the environment, but like ultimately a lot of like, a lot of like agents that I see are, are, are really impressive, but it's basically like part of what's impressive is it's like a bunch of developers building this like really bespoke state machine around a bunch of like short model calls. And so then the upper bound of like complexity of problem that the model can tackle is kind of actually just what can fit in the developer's brain, right? Um, and over time we want these models to capture like, or to, to solve for much more complex problems, um, you know, like just by themselves on like more and more complex individual tasks. And then eventually you could really imagine that you get like a team of agents working together maybe with like one agent that's kind of managing those agents and, you know, the complexity just explodes. And so we really want to like get as much of that complexity, as much of that state machine as possible, like pushed into the model. And so you end up with these kind of two modes of building. Like in one place you're like building product UI and rules, and in the other case you're-- you, you still have to do work to get the model to learn something, but rather what you have to do is you have to figure out like what are the right things that this model needs to see during its training to like learn something. Um, and so there's still a lot of human work to like figure out how to get that change, but it's like a very different way of thinking of like- **Wigs** [31:43] And- **Alexander Embiricos** [31:43] ... we're gonna get the model to see this. **Wigs** [31:44] But how, how do you build the product to get those signals? So if you think about the code and ask, it's almost you're basically getting the user to label the prompt in a way, right? **Alexander Embiricos** [31:53] Mm-hmm. **Wigs** [31:53] Because they say, "Ask, this is a ask prompt, uh, code, this is a code prompt." **Alexander Embiricos** [31:57] Mm-hmm. **Wigs** [31:57] Um, are there any other kind of like fun product designs like as you built this of like, okay, we think the model can learn this, but we don't have the data, this is how we architect Codex to kind of help us collect the data? **Josh Ma** [32:10] I, I think file context and scoping is-- we don't have great, um, built-in things like that right now, but it's like one of the obvious things that we need to add, um, is another example of this, right? Like you could have, um, we-- we're often usually pleasantly surprised as, oh, it was able to find the exact file that I was thinking about, but it takes some time, right? And so a lot of times you'll, you'll shortcut a bunch of, um, chain of thought by just saying, "Hey, I'm looking in this directory. Can you go?" Um, so I think that'll probably be there for a bit, um, until, um, you know, hey, you have some better architectural indexing and, uh, search capa- uh, capabilities. **Alexander Embiricos** [32:45] Yeah. I'll, I'll add to this. Like I'm actually gonna double down on my thing about like where do we, how do we think about it? Like, so, you know, one thing we might consider is like context window management, right? And like should we intervene here? Um, and so we, we could do a product intervention, right? Like write, write some code to intervene. And then kind of the next level of thinking maybe like a little bit more AGI-pilled is like, okay, let's sh- let's get the model to like see context window management stuff in this training. I can't even come up with an example now at this point 'cause I'm too AGI-pilled, but like I don't know. We could come up with something that it has to see to like learn how to manage its context. Like, but it's like s-specifically tasks related to context window. But then the most AGI pill thing to do is like to be like we don't actually need to think about this problem. The model will just figure it out. All we have to do is give it harder and harder problems, and then like it will just have an emergent property of managing its own context, uh, because that's the only way it can like solve these problems, right? So I'm kind of slightly like oversimplifying here, but like, you know, basically the model learns to manage its context. And so you-- when you were talking about like it working in the monorepo, it learns how to be like efficient, um, with the way that it spends its tokens, um, as it's like browsing and like setting and like, you know, in your example of like there's a giant agent.md, like I guess we, you know, it would just need-- we would just to show it some versions where there was that, uh, and so it learns it can- it shouldn't read the whole thing every time and should like first figure out how many lines it has, et cetera. So anyways, summarizing, I'm like we just need to keep giving it harder and harder problems and a lot of these things that we might be very tempted to like build a sub-intervention for, like it will just have to figure out. And if it doesn't figure it out, maybe it didn't matter. **Wigs** [34:16] Sure. Yeah. To- uh, I totally get, uh, that. I, I think like we don't really have online models yet, right? And that's kind of what you need for, for your vision to be real. Um, and it-- for what it's worth, I, I wasn't thinking about like a giant agent.md. I was just thinking about like hierarchical nested- **Alexander Embiricos** [34:32] Totally **Wigs** [34:33] ... agent.md with, with like a lot of code. Um the, um, and it like I think one issue where you have this s- version where the model is the product is your, your dev cycle as the Codex team, like the two of you, like you have to kinda... It, it's not as tight because you have to be like, "Okay, every time there's a bug, all right, now I need to go get data." Um, uh, and where do you get the data? I don't, I don't know. Like maybe employees use it. Maybe you have-- like you buy it from vendors and like you hire some human raters or whatever and then you have to train it in and then you have to go test it again. It, it's, it's very slow, isn't it? And like expensive too. **Alexander Embiricos** [35:09] Yeah, I think it's definitely... Yeah, I think it's definitely like from a, from a building perspective you, you have to do this when you're really willing to play the like the long-term vision of like we're gonna build a better model, maybe even a better model bespoke for a certain like functional purpose like Codex 1, and then we're gonna generalize the learnings from that model into like an even bigger model that's like getting all these other like learnings from other functional purposes and like these together will like become a really powerful thing. ### Long View **Alexander Embiricos** [35:36] And that's, that's like kind of like the philosophy we've have with training models so far. Um, and it has been working, but it's, it's definitely like a long-term play. You know, like another, uh, example, we, we do, we do do this though on occasion like for example, w- uh, recently we released GPT 4.1, like really good coding model and again that was like based on like working-- we were like, "Hey, we wanna invest better in this area. Let's hang out with a bunch of developers, understand their feedback, how things work, um, you know, create some evals." And you know, like you said, this is like it's a lot of work to do that. Um, but then we end up with a great model and even more exciting we can then like take those learnings and like put them into our mainline models and then everything benefits and you kind of the, the, the, the sort of philosophical view, I don't know if I can like factually prove it or not, maybe someone here can, is that like if you can like build-- do something very specific for like a specific purpose, actually when you bring that and you bring it to the, into the generalized model like you might even get outsized returns on that because there's like transfer from all these different domains. **Wigs** [36:38] Um, okay, cool. Uh, I think we had a couple factual things to wrap up on, on just like Codex itself, and then we wanted to double-click on the compute platform stuff, which, uh, I think, uh, Josh, you wanted to, to cover more on. Um, so I, I noticed in the details it was, um, between one to 30 minutes in length. Um, is it-- is that a hard cutoff? Um, have you had it go, go for longer? Um, any, any comment on the, the task time? **Josh Ma** [37:03] Yeah. I, I mean, I just checked the code base before this, uh, 'cause someone else had a similar question. Uh, our hard cutoff's an hour right now, although don't hold us to that. It, it may change over time. Um, the longest is I've seen two hours when, uh, in development mode and the model went off the rails. Um, so you know, I, I... But I think, yeah, 30 minutes is a great ballpark for the kind of tasks that we're trying to solve, right? These are hard tasks that require a lot of iteration, uh, and testing and, uh, the model needs that time to think. **Alexander Embiricos** [37:32] Yeah, I mean... Yeah, I, I think actually like our average is like pretty-- is significantly lower than 30. **Josh Ma** [37:37] Yeah. **Alexander Embiricos** [37:37] But if you give it a hard, if you give it a hard task, you'll, you'll end up at 30. **Wigs** [37:41] Yeah. I mean, um, you know, I think there's, there's a couple analogies here. One, uh, I think the operator team released a benchmark where they, they had the cutoff for two hours. **Alexander Embiricos** [37:49] Mm-hmm. **Wigs** [37:49] And then the other one is the meter paper, which I, I don't know if has been circulating, where they estimated that the u- the current average autonomous time is like an hour, and it's maybe doubling every seven months. Um, so like an hour sounds right, but also, I mean, that's the median, so there's gonna be some that are, that go longer than that. **Alexander Embiricos** [38:06] Yeah, totally. **Wigs** [38:08] I-is this part of the, uh... You had cutoffs for a few, like 23, um, SWE-bench verified examples that were not runnable. Was that part of it, uh, in terms of length, or was it just something else? **Alexander Embiricos** [38:21] Yeah. **Wigs** [38:21] Uh, to be h- **Alexander Embiricos** [38:23] Yeah. To be honest, I'm not exactly sure, but I feel like there's a bunch of SWE-bench cases that actually are like... I-- invalid might be too strong of a word and a little bit out, you know, not sure, but like I feel like there's like issues with running them, so they just don't work. **Wigs** [38:34] Okay. **Alexander Embiricos** [38:35] Yeah. **Wigs** [38:35] Um, and then max concurrency, is there a, a concurrency limit if I have like 5, 10, 100 simultaneous Codex instances? **Alexander Embiricos** [38:42] 5 and 10 is totally fine. Do we actually have a... I feel like we did- **Josh Ma** [38:45] We, yeah **Alexander Embiricos** [38:45] ... like introduce like a limit for fraud reasons. Like, I don't know what it is. **Josh Ma** [38:48] Yeah. I think right now it's 60 an hour, um, but- **Alexander Embiricos** [38:53] Wow **Josh Ma** [38:54] ... it all- **Alexander Embiricos** [38:54] So one per minute. I'm just gonna s- Yeah, but look, this, this is literally the point, right? Like it is... So, so, uh, like long term, we actually don't want you to have to think about if you're like delegating or like pairing with AI. Like if you imagine an AGI like super assistant, you just like ask, you just talk to it, and it just does stuff. It answers quickly if it needs to. You know, it takes a long time. And you also don't have to like only talk to it. It's also just like present in your tools, right? So that's, that's like the long-term thing. But in, in the near term, like yeah, this is a tool you delegate to. And, um, the way to use it that we see like, you know, go-going back to the, I guess the, maybe the title of this podcast of like best practices, it's like you must have an abundance mindset, and you must think of it as like, like not using your time to explore things. ### Usage Tips **Alexander Embiricos** [39:37] And so like, you know, often when something-- a model's gonna work on your computer, an agent's gonna work on your computer, you'll like really craft the, the, the prompt because, you know, then it's, it's gonna use your computer for a while and maybe you can't. Um, but the way we see people who like love Codex the most using it is they don't-- they think for like maybe 30 seconds max about their prompt. It's just like, "Oh, I have this idea." Like boom. "Oh, like there's this thing I wanna do." Like boom. "Oh, like I just saw this bug," or like this customer feedback thing like, and you just send it off. And so yeah, like the more you're running in parallel, actually, I think the, the, I mean the, the happier we are and like the happier we think like users are when they see it. Like that's just the vibe of the product really. **Wigs** [40:14] Yeah. I, I would, I will pass my own anecdote. So I was on the trusted testers team for this thing, uh, as you-- as both of you well know. **Alexander Embiricos** [40:20] Yeah. **Wigs** [40:20] Um, and I was using-- I found out I was using it wrong. I was using it like Cursor. Like I, I had my chat window open and I watched it code. **Alexander Embiricos** [40:28] Yeah. **Wigs** [40:28] And then I realized I wasn't supposed to and I was like, "Oh, like you guys are just firing the things off and like, you know, going on about your day." And, um, yeah, that, that, that was a change of mindset. Um, one- **Alexander Embiricos** [40:38] Yeah, one- **Wigs** [40:39] Yeah **Alexander Embiricos** [40:40] ... real quick, I'll keep it brief. Like one thing that's quite fun is like use it on your phone because somehow just like- **Wigs** [40:45] Oh, yeah **Alexander Embiricos** [40:45] ... being on your phone just like flips the way people think about things. **Wigs** [40:48] Yeah. **Alexander Embiricos** [40:48] So like we made the website responsive and, you know, we'll make-- we'll pull it into the app eventually. Um, so like try it. Like it's actually like super fun and satisfying. **Wigs** [40:56] Okay. **Josh Ma** [40:57] That was my first week with, uh- **Wigs** [40:58] Yeah. 'Cause, uh, it-- there, there was a, there was a voice, uh... There was one of the videos was showing the mobile engineer coding with it on his phone, but it's not available in ChatGPT's app. Okay. Yeah. **Alexander Embiricos** [41:08] Yeah, not yet. **Josh Ma** [41:10] J-just one question I got from the mobile. I got the notifications that I get. Uh, when it starts the task, it says, "Starting research," the same way the deep research notification is. Uh, is it using deep research as a tool or did you just reuse the same, uh, notification that was- **Alexander Embiricos** [41:25] We just used the same notification, yeah. **Josh Ma** [41:28] So you mentioned the compute platform. You mentioned how you share some of the infrastructure with RL. Can you maybe just give people a high level of like what the Codex has access to, what it doesn't have access to? Like it doesn't look like people can run commands, uh, themselves. They can only instruct the model to do it. Um, any other things people should keep in mind? Yeah. So... And I'll say, uh, it's a, it's an evolving discussion as we figure out what parts, uh, we can give folks access to and the agent and vice, like, and what we need to like hold back for, for now, right? And so we're learning and it's really-- we would like to give, uh, humans and agents alike as much access as possible within safety and security, uh, constraints. Um, what you can do today, right, is as a human, um, set up an environment, set up scripts that get run. Uh, these scripts typically will be installing dependencies. I expect that to be maybe ninety-five percent of the use case there. Um, and just really get all the right binaries in place for your agent to use. Um, we actually do have like a bit of an environment editing experience, where as a human you can drop into a REPL, try things out. ### Compute **Josh Ma** [42:37] Um, so, you know, um, uh, please don't abuse it, but, uh, there's definitely ways, uh, for you to interact with the environment there. **Alexander Embiricos** [42:44] Um, we, we laugh about that because, like, ear-earlier I mentioned scope creep. Like, we weren't planning on having a REPL, like, to, like, interactively update- **Josh Ma** [42:51] Yeah **Alexander Embiricos** [42:51] ... your environment. But, like, you know, anyway, we try... Joshua was like, "Yo, man, we need this," and so that was, like- **Josh Ma** [42:56] Yeah **Alexander Embiricos** [42:56] ... an example. Scope creep. Thanks for doing it. **Josh Ma** [42:58] We do have rate limits in place, and we do monitor that very carefully. But, you know, there's interactive bits in there to, like, to, to get that going. But, um, once the agent starts running, right, um, what we actually do today, and we're hoping to, like, evolve on this, is we'll, um, cut off internet access. Uh, 'cause we still don't fully understand what letting loose an agent in its own environment's going to do, right? Um, you know, for now, the safety tests all have come back very stur-sturdily. Like, you know, it's not susceptible to source of certain exfiltration attempts on prompt injection. But there's still a lot of risk to this category, so we don't know. And, um, that's why to start, we're being more conservative there, and when the agent's running, it doesn't have, uh, full network access. But, you know, I'd love to be able to change that, right, allow it to give limited access, certain domains or certain, uh, repositories. And so all this to say it's, like, something we're evolving as we build out the right systems to support that. Um, not sure that quite touches on your original question. Um, the last thing, though, that I do want to mention is, like, there is an interactivity, um, element with, like, you know, as the agent's running, sometimes you're just like, "Oh, I wanna, like, correct it, tell it to, like, go somewhere else." Or, um, you know, "Let me maybe fill this part out and, uh, then you can take back over," right? We haven't quite solved those problems either. What we really wanted to start was to, like, shoot for the fully independent, just, like, deliver massive value in one shot kind of approach. Um, but yeah, we're definitely thinking about how we can, uh, weave human and agents together, um, better. **Wigs** [44:35] Uh, I mean, uh, for what it's worth, I think the one-shot thing is, um, a good end goal. Uh, that, that the other, the other people, you know, like, uh, and this is me comparing you to your, to alternatives like Devin and Factory and all the others. Um, they are more focused on multi-shot human feedback, all these. But, like, um, you know, I... So I, I have a we-website I'm working on, and I, and I gave it a, a request, and I compared it all the others. That was my test for- **Alexander Embiricos** [45:00] Yeah **Wigs** [45:01] ... uh, Codex. Uh, and it did one-shot it. Uh, I, I posted it, the, uh, the screenshot as a tweet, uh, just earlier today. Um, and it's, uh, I, I think it's really good, especially if you're running 60 at a time. Um, so I think that, that really makes sense. But it, it's, it is a very ambitious goal because, like, um, you know, human feedback is a crutch that we like to use. Um, it also, I think, makes us write more tests, which is annoying 'cause I don't like to write tests, but now I have to write tests. Fortunately, I can-- I'm now getting Codex to write my own tests. And I really like the, on the, on the livestream as well, you can just kind of ask it to just look at your code base and just suggest stuff to do 'cause I don't even have the energy to figure out what I should be doing. **Alexander Embiricos** [45:41] Yeah, delegate the delegation. I thought that was a great line. **Wigs** [45:44] Yeah. **Alexander Embiricos** [45:44] Um- **Josh Ma** [45:45] And, and to be clear, we're not saying one form factor is better than the others, right? Like, you know, I love using Codex CLI, um, and, uh, it's really, we just want... Like, as we talked about in our interview, when I was interviewing at OpenAI, like, you really want both modes. Um, but I think what we see as, like, the role of Codex here is to really push the frontier on that sort of single-shot autonomous, um, software engineering. **Alexander Embiricos** [46:10] Yeah. Like, I kinda think of the Research Preview as, like, our, like, our thought experiment. It's like, you know, what- **Wigs** [46:16] Yeah **Alexander Embiricos** [46:16] ... what is putting agent, like, in its purest, like, most, like, AGI-pilled or scale-fill, scale-pilled form look like? And then maybe, I mean, for me personally, like, I don't know, like, part of what excites me about working at OpenAI, it's not just, like, solving for developers, but it's just really thinking about, like, how does AGI, like, benefit all of humanity, and, like, what does that feel like to non-developers as well? And so, like, for me, like, what's really interesting is, like, thinking of Codex as an experiment for, like, what it'll feel like to be in other functions- **Wigs** [46:43] Mm **Alexander Embiricos** [46:43] ... um, you know, doing work. And, like, the goal for me to, to, to build towards is this, is a vision where it's like, you know, we do the work that's, like, ambiguous or creative or hard to automate in whatever way, and, but otherwise, we just have, like, agents, like, that we're delegating most of the work to. Um, but these agents, they're, they're not, like, this, like, long horizon thing versus short horizon. They're just, like, kind of ubiquitously available, um, with you. Um, so, uh, yeah, we, we decided to, like, take the purest form to start, which we thought would be the smallest scope thing to ship, and probably isn't. Uh, but, uh, yeah, and then we're gonna bring these things together. **Wigs** [47:18] Uh, okay, I think we have time for a couple questions. Um, we, you-- I'm just gonna double-click on the Research Preview a bit. Um, you, it is a research preview. Why, what, what is left? What do you think would qualify it to be a full release? Um, you know, on, on the livestream, Greg mentioned the seamless, uh, transition between cloud and CLI. Um, is it that, or is there, is there, are there other things on your mind? **Alexander Embiricos** [47:42] I mean, to be completely honest, you know, we, part of why we believe so much in iterative deployment, like, I can give you some of my thoughts now, but, like, also we're really curious to see. Um, so because this is, like, such a new form factor. Um, but, you know, some of the items that are top of mind for me are, like, multimodal inputs. Um, you know, we've, we've talked-- Yeah. **Wigs** [48:00] Yeah. **Alexander Embiricos** [48:00] I know you- **Wigs** [48:01] Yeah **Alexander Embiricos** [48:01] ... like that, right. Yeah, like another, another example would be, like, you know, just giving it a little bit more access to the world. Um, you know, a lot of folks have, have requested for, like, forms of network access. ### Wrap-up **Alexander Embiricos** [48:11] Um, you know, I also think that, um, right now kind of the, the UI that we shipped is actually one that we iterated around. There's, like, a fun story there, but, like, um, and it's one that people find useful. But it's, it's definitely not the final form of what it is, and, like, we would love for it to be, like, much closer with the tools that developers spend time in. So, like, those are some of the themes we're thinking about. Um, but, you know, to, to be clear, we'll iterate and figure that out. **Alessio** [48:38] Um, I wanted to ask, why did you put finding a typo as one of the onboarding things? Because I used it and then I saw it, and it's literally just grepping for potential type... It's like searching, grepping for like selenium with an N or like something with like some TGN instead of NG. Uh, but it went through like fifty of these and then finally found will misspelled as W-I-L without the two thing. But it was really cool to see what it thought. Like default, uh, spelled as like D-E-F-U-A-L-T. **Josh Ma** [49:13] Yeah. **Alessio** [49:13] It's like it just grabbed all these different things and then eventually got there. Like why, why, why did you pick that task? **Josh Ma** [49:20] Honestly, Tiwo and I were talking about it and he's like, "Listen, it would be funny if I had a typo as I type this prompt out and, uh, just make it a little bit meta." Um, you know, nervous fingers on a, uh, on a live stream. Uh, so maybe optimized for the, for that. I have noticed it, um, you know, will, uh, it likes to do T-E-H to look for the, um, and so- **Alessio** [49:43] Yeah **Josh Ma** [49:44] ... it's a work in progress. **Alessio** [49:46] That, that was great. Um, any parting thoughts, call to action? Are you growing the team? Do you want, uh, specific feedback from the community? **Josh Ma** [49:58] Yeah. I, I, I think for me, like the one thing that, um, you know, um, is really on my mind for getting better at, um, over the next few months is really helping you customize your environment, uh, more, um, in a more high fidelity manner. Um, turns out the good news is the agent can do a lot of really good work with only the basics, right? Um, it's much like if your dev machine is borked and you're sort of looking at your editor, but none of the type checks are working, you... A lot of folks can still actually do a lot of good work. But, um, how do you get close that last, you know, thirty, forty percent? And it's really hard 'cause, uh, there's such a wide variety of environments out there. Um, but especially would love feedback from folks on how they would like to see their environment customized. Do they wanna just ship us a darker image? Would they rather have us support dev containers, right? Um, so the form factor of how you do the DX of how you do, um, environment customization, still very much an open question that we need to improve on. **Alexander Embiricos** [50:54] Yeah, big plus one to that. And I think for me, maybe the thing that I'm most interested in is, hey, this is, this is like a new shape of, of tool to collaborate with. Uh, and I'm just really interested for people to try working with it in as many different ways as possible and like kind of figure out like where does it work well in your workflow? Like you said, you know, you mentioned you earlier, you were trying to use it kind of like your IDE, and then you realized it was different, right? So I would just love for people to take advantage, especially now, like we're very intentionally just providing like v-very generous rate limits so people can try it. Like we just want you to try it and figure out what sticks, what works, what doesn't, how do you prompt it, and then we wanna learn from that and use that to lean in. So yeah, my, my... I guess my parting call to action is like please go try it out, um, in ChatGPT. Use it as much as you can, especially now, and then let us know, uh, like how you like to hold it, basically. **Wigs** [51:44] Yeah, I'm, I'm, uh, worried about the pricing when it happens, but yeah, I'm gonna abuse this, uh, just to- Why not? Why not? **Alexander Embiricos** [51:50] Yeah. Send us feedback on pricing too. **Wigs** [51:53] Yeah. Um, okay. Uh, you're pro... It, it's too early to talk about pricing, right? **Alexander Embiricos** [51:58] Uh, yeah, it's too early now. Yeah. **Wigs** [51:59] Okay. All right. Um, but, uh, yeah, based on Cloud code, that's the thing that people are worried about, right? Um, and Claude has started to introduce some kind of fixed pricing and variable pricing, and I think it's a, it's a huge mess. Like it's... There's no right answer. Everyone just wants the cheapest form of code attention they can get. Um, so yeah. Good luck. **Alexander Embiricos** [52:19] Thanks. **Josh Ma** [52:20] I mean, my take is, I don't know if it's gonna make it into... But like we're, we aim to deliver a lot of value, right? And it's on us to show that and really make people realize like, "Wow, this is like doing very economically valuable work for me." And I think a lot of the pricing can fall from that. Um, but I, I think that that's where the conversation should start from. Um, are we actually delivering that value? **Wigs** [52:45] Yeah. Awesome. All right. **Alexander Embiricos** [52:46] Cool. **Wigs** [52:46] Well, thank you so much. Yeah, thanks for working on this, and thanks for sh-sharing your time. Um, it is, uh... It's been, it's been a long time coming, but I, I think people can start seeing like... OpenAI in general is very, getting very serious about agents. It's not just coding, but coding obviously is the, the one loop that is self-accelerating that I think, um, every... Obviously, you guys are super passionate about. It's really inspiring to see. **Alexander Embiricos** [53:09] Yeah. Super excited to, uh, super excited to just like ship you, everyone this, this coding agent, and then, yeah, like bring it together into just like the, you know, the general AGI super assistant. Uh, yeah. So thanks for having us on. **Josh Ma** [53:20] Cool. Thank you. **Wigs** [53:20] Thank you, guys. **Alexander Embiricos** [53:21] Cool. **Josh Ma** [53:21] Thank you. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.