Cooking with OpenAI’s Research Chief: AGI, o1, Evals, and Scaling Laws — Mark Chen

Woo. Yeah, I was like-

Thank you. That, I know.

That feels like the current situation I'm in. Like... Cheers.

Cheers.

Hey, guys. Welcome to the Latent Space cooking series, where we invite founders and researchers and just let them cook. Today, we have a very special guest, the chief research officer of OpenAI, Mark Chen. Welcome, Mark.

Thanks for inviting me, Aiden.

Yeah, thank you for coming. I mean, to begin-

Mm-hmm

... this all started from the inspiration after hearing a story that Mark Zuckerberg would make soup to try to poach researchers, and in response, you brought soup to researchers. Is this true? Did this happen? Did it work?

Oh, you know-

Can you elaborate?

... it's absolutely a true story. Uh-

Okay

... and I have brought soup to our own researchers. Um, I think that Meta's calmed down a little bit.

Yeah.

I think we came out on top, but, um, yeah, still a very funny story, and the craziness of how AI has evolved. Mm-hmm.

How often do you cook? Is this something you're familiar with?

Well, you know, I, I do enjoy cooking.

Mm-hmm.

But I don't have the luxury of doing that so often, so I, uh, usually have a work dinner every night of the week, and, you know, maybe post-AGI this is gonna be my hobby. I've always joked- ... you know, I'm gonna start a noodle stand once, uh, once it's, it's all over.

Yeah, yeah. You know, post-AGI, hopefully that'll, you know, still be there.

Mm-hmm.

But great. And I guess looking at what we have in front of us, do you have an idea generally of what we'll probably making?

Uh, Korean tofu soup maybe.

Yeah, yeah. That's, that's generally what it is.

Great. Okay.

So we inspired off of the story-

Okay

... of you, you know, bringing soup to researchers.

Mm-hmm, mm-hmm.

So we're making a tofu Korean stew.

Okay.

And then we have problems that we'll be cooking. Are you ready to go?

Yeah, let's do it. Let's do it.

Great. Okay.

Mm-hmm.

So the first thing we should probably do is we'll separate the veggies, and then we can cut them.

Okay.

And basically what we wanna do is just, um, cut the dirty part off with the dirt and then, yeah, separate that across-

That one, that I know.

Yeah. Okay. And then just have that there, so-

Okay

... you could do that. And while that's going, I guess I could ask more about your background.

Mm-hmm.

So in a previous life, you were once a trader, and even Sam, I think, last year in April, also tweeted about-

Mm-hmm

... how if you're a high-frequency trader, you should consider joining OpenAI-

Mm-hmm

... because, you know, build AGI.

Mm-hmm.

So do you think there's a relation between, you know, being a trader and being a researcher, or do you think it's just, like, a very technical and competitive area where a lot of great employees can come from? Can you expand on that background?

I think really the most important thing is, um-

Hmm

... there are a lot of researchers who just, uh, started out without a formal training in machine learning or AI research.

Gotcha.

Um, we've very much believed in training people up to do this. I, I think the real hard thing is the ability to creatively solve problems and think outside of the box.

Yeah.

And it's not so much, you know, you have to do a PhD, even though that does bring a valuable skill set.

Mm-hmm.

Um, with trading in particular, I mean, I, I don't know that it's that special of a profession. Like, um-

Yeah

... I kinda think of it as, uh, you know, we've had great mathematicians join, we have had great, you know, physicists join, but trading is something where, you know, it's like, it's very unhackable.

Mm-hmm.

You know, you're, um... How... You, you can't kind of cheat the real world, right?

Yeah.

Like, uh, you know, it's a, it, it's a hard metric to optimize.

Mm-hmm.

Um, and there's also a lot of characteristics, like, uh, it's, it's a field where attention to detail really matters.

Yeah.

And, um, you know, it, it's kind of the brutal hard optimization and squeezing out the juice of a system.

Hmm.

Um, and some of those, uh, those skills transfer over.

Gotcha. Yeah.

Mm-hmm.

And I guess for people who want to get into research-

Mm-hmm

... who, let's say, don't have a PhD, what do you think are the main attributes or things that they can learn to develop research taste? Because-

Mm-hmm

... I guess that's the main part of, um, getting into this field that may be very foreign to them.

Yeah. I mean, I think it's a little bit overrated.

Hmm.

Um, it is something you have to develop, but, um, the best mechanism I've found for developing that is really just, um, replication. So I think you should take papers that you really look up to-

Yeah

... and just try to fully replicate it.

Gotcha.

Um, I, like, I, I think a lot of replications stood out in mind, my mind.

Hmm.

Um, you know, back in 2018, there was, um, uh, you know, like ResNet, there were PixelCNNs, and-

Mm-hmm

... I think I learned so much just, um, trying to replicate the training curves exactly, get to the exact amount of, you know, like, training loss or perplexity that the paper is, um, hinted towards.

Yeah.

It teaches you a lot of techniques, right-

Mm-hmm

... that, um, people don't really kinda talk about. But, you know, once, once you dive in a couple layers deeper, um, you learn those techniques. And, um, yeah, I think really the first thing, too, that got me into the field was-

Yeah

... when AlphaGo played Lee Sedol.

Hmm.

And, you know, I think that was a turning point for so many people.

Yeah.

And, yeah, I, I mean, it was, it was, it was inspirational, and it, it... The, the first big project that I, I really went after was, um, can I get a DQN working?

Hmm.

Yeah.

Yeah. Yeah, that's true.

Mm-hmm.

I think it was Move 37 or-

Yeah. Yeah

... from, like, when the games was pretty, uh, insane watching it happen, and-

Yeah. Yeah

... seeing all that develop, and see also where we have gotten to today, especially with research.

I mean, isn't it crazy that you're seeing Move 37s in almost every field now?

Yeah.

It's like there's Move 37s in, in math. There's, uh, in computer science and coding. Um-

Mm-hmm

... I think even... Yeah, just it feels like a lot of people woke up at the start of this year and were like, "Man, agents are working in my profession."

Yeah.

And, um, you know, they're, they're essentially realizing that these models can just do long-horizon meaningful work for them.

Yeah. No, that's true.

Mm-hmm.

It is, it is very impressive to see-

Yeah

... like, even just using it in my own work. But okay, the next thing we could do is just as simple-

Yeah

... just cutting the onion.

Okay. Great.

So what we have to do here is just, like, dicing it.

Yeah.

Do you think, um, there's jobs that RL maybe will have, like, a much harder time to kind of break into? So for example-

Mm-hmm

... coding may be easier since a lot of the context is accessible, whether it be the code bases or even the work you're trying to do. But-

Mm-hmm

... let's say if you're trying to do the job that a junior consultant may do-

Mm-hmm

... where all the context is a little scattered-

Mm-hmm

... maybe a little more difficult, how do you view through, like, those different scenarios? Is there a way that you kind of assess what can be the right approach?

Yeah, I mean, I, I think it's... RL's traditionally had, um, headwinds when it's come to fields that, you know, it's more, um, kind of- ... subjective than objective.

Mm.

So if you kind of think of like, you know, one, one kind of, you know, uh, example of this is creative writing.

Yeah.

Where, you know, you can take two pieces of creative writing, and two experts can have wildly different opinions.

Yeah.

So it's, it's these fields where things are hard to grade.

Mm-hmm.

Um, where, you know, RL has the least amount of ability to kind of go and, um, and directly apply there.

Yeah.

I know a lot of people are developing techniques to apply RL in these, um, these settings, but, um, for now, it's just where there's cold, hard truth-

Mm-hmm

... things like math and computer science, where-

Yeah

... you implement it correctly or wrong. Um-

Mm

... that's where you kind of see it really taking off.

Yeah. No, that actually-

Mm

... brings up a thought on in terms of evaluating-

Mm-hmm

... those fields, so-

Yep

... um, you know, as models get much pow- much more powerful-

Mm-hmm

... and even saturate, for example, solving-

Mm

... like the IMO questions.

Yeah, yeah.

Um, how do you view evaluating like superhuman intelligence, like get to a point where it's so good at things that even the top, what, .01% of humans can do?

Yeah.

But like, you know, how can we push past that frontier of intelligence?

No, it's kind of, it's kind of crazy, and I feel like, um, a lot of it centers in on in, on kind of interfacing with the real world. And-

Gotcha

... um, when, when we've thought about how to evolve past things like programming contests in the past-

Mm-hmm

... um, I think a lot of the initial direction we took was you should move it to real-world research, right? And-

Hmm

... we've seen that the models, uh, they've gotten a lot better at, uh, just kind of discovering novel theorems and, uh, pushing the frontiers of, of hard sciences.

Yeah.

But even today, right, that's no longer a surprise. I think like you- we, we almost take it for granted now that, um, these, these models can solve very, very difficult problems. They can make contributions and even kind of draw relationships between, um, fields that, you know, um, that are, that are novel and insightful.

Yeah.

So I think, um, you know, we, we think of coding, co-working, um-

Mm-hmm

... as really a, a domain for, that, that tests if our models can learn in high-context settings-

Mm

... and in real-world long horizon settings.

Gotcha.

Mm-hmm.

Okay. Yeah, that makes sense. And since you're done with all the vegetables-

Mm-hmm

... we can now do the next step, which is-

Okay, great

... sautéing it, so.

Cool.

Yeah, we can use the Impulse Stove, which-

Yep

... we've seen before, and it's very powerful. Let me-

Mm-hmm

... turn it on. Um, and yeah, so we'll just sauté it-

Great

... with some oil. So yeah, put the pan in the front burner and then yeah, all-

Super cool stoves.

Yeah. You could use oil to pour some in.

Okay, great. Mm-hmm.

And then we can also... Yeah, just a good dollup, perfect.

Nice.

And yeah, and then we could turn on the stove. So just press it, and then-

Great

... um, yeah, spin the knob.

Great, perfect. And then while that heats up-

Mm-hmm

... we can just wait and then add the vegetables. But yeah, I guess more so on views for research.

Mm-hmm.

Are there, I guess, you know, commonly accepted ideas that are, you know, you disagree with, whether it be like pre-training is dead or language models will never get us to AGI? I think there's a lot of takes out there that-

Mm-hmm

... are very ambiguous-

Yeah

... and obviously haven't been proven out yet. And I guess from your perspective as, like, the research, like leading things in OpenAI, like think through those.

Yeah. I mean, I, uh, I firmly believe in exponent- uh, being on the exponential and in scaling laws.

Yeah.

So I think any of these bear takes, um, I fairly strongly disagree with.

Mm.

Um, you know, when it comes to pre-training is dead, I... I mean, I think the, the funny thing is this narrative only started spreading more widely after, let's say, um, the last one or two years or so.

Yeah.

But in many times, uh, i- in the history of, uh, developing LLMs, people have been saying this, right?

Mm.

And, you know, um, there, there have always been some, some bottlenecks that people, "Well, you can't scale past this because of this bottleneck." Um, and we've always found some kind of technique, whether it be better engineering or some new research insight that helps-

Yeah

... you break past the boundary. And so I think it's just more and more of the same, right? Like more careful research engineering, more careful data engineering-

Mm-hmm

... more careful scaling, and it always unlocks that next ability to scale further.

Yeah.

So I, I mean, it's held for, you know, almost ten orders of magnitude, but there's no reason it should- ... not keep, keep holding.

Yeah, that's a very fair point.

Yeah. Yeah.

And I guess on research bets that have-

Mm-hmm

... helped you scale beyond-

Mm-hmm

... were there specific ideas that you can even remember in the early days that everyone was, was saying that is not gonna work? Or-

Well, yeah, I mean, I think of reasoning as one of the biggest-

Okay, reasoning, yeah

... examples of this. Yeah, and, um, you know, the, the first breakthrough that we launched to the world here was o1.

Mm-hmm.

But it wasn't easy to get that off the ground because, one, the world we were back, living in back then-

Yeah

... it was one where pre-training plus post-training, right?

Mm-hmm.

That felt like such a promising paradigm.

Yeah.

Um, and so even at a company like OpenAI-

Mm-hmm

... you would have people ask naturally, "Why do something when you have a machine that works?"

Mm.

And fundamentally, you know, it's to the credit of, you know, Jakub, Ilya-

Yeah

... many of the people who really had conviction and vision in this space-

Mm-hmm

... um, that we started pushing on this in earnest. And even then, it took a lot of steering to get-

Mm

... the whole company behind this as a, as a fundamental bet.

Gotcha.

So, yeah.

And how do you kind of develop that ability to motivate researchers? 'Cause I assume that's a big part of, you know, taking a lot of bets, and some-

Mm-hmm

... will pan out, but still building the trust in the team to know that eventually some of these will actually have, you know, power law effects.

You know, what's, what's really cool about OpenAI is, um, research, it feels like a meritocracy.

Mm.

So, um, oftentimes, the research managers are the people who, um-

Do the actual-

... who have done the best research-

Yeah

... in the past. And so-

Okay

... I think a lot of steering can come top-down, right? Like, if your manager says, "Hey, you know, I'm like really convinced this is the path forward"-

Yeah

... um, generally, people will take that into heavy consideration, right?

Yeah.

Um, it's like, you know, this person who you've respected for their research taste and execution for so long-

Yeah

... is, like, now very excited by this idea.

Mm-hmm.

Um, it's, it's definitely something that, uh, that, um, yeah, you, you... people take into account. So-

Mm

... I think there, there's good top-down steering. At the same time, you know, I think one really cool thing about OpenAI is, um, there are bottom-up elements.

Mm.

Like, we like to be convinced that-

Yeah

... um, uh- You know, that we're wrong, right?

Mm-hmm.

And, and someone can just come with cold, hard evidence.

Mm-hmm.

And many things like that have turned into core parts of our research roadmap. Just things that no one was really kind of trying to steer, but some researcher on the ground had a heavy conviction in.

Yeah.

Um, and, and that's also a really d- big delight to see.

Yeah, no.

Yeah.

Absolutely. I heard in a recent interview-

Mm-hmm

... that you gave that your internal research roadmap hasn't really changed.

Mm-hmm.

Um, even through all that we've seen-

Mm-hmm

... you know, with model development and even other companies.

Yep.

I guess, how often do you guys assess that, reassess that, even, like, act proactively? I assume it's not a lot of reactive-

Mm-hmm

... you know, decision making-

Yep

... as, like, other models come out.

Yep, yep.

But how do you, like, think through that process, especially as everything around you just continues to get better?

Yeah. So I think the thing is, um, the high level researcher map should be staying, right?

Okay.

I think people need something to ground in. People need to see a path to what we're building. Um, and I've been very happy that we've stayed the course for, for a while.

Yeah.

But the implementation details can change over time, right?

Yeah.

And I think, um, it's important to kind of... Like, the, the sequencing will matter, right?

Yeah.

The relative resourcing will matter, and it, the, the kind of exact vets on the ground will matter.

Yeah.

So what we do is, um, I think we have kind of points in time that force us to reconsider these things. So, uh, one example is when we do compute.

Yeah.

Um, one of the parts of the job is just figuring out how to allocate compute to projects.

Okay.

And, um, it's, it's a time to kind of question, like, are we really putting compute to use and people to use at the highest priority bets?

Yeah. And I guess could you clarify more what you mean by-

Need some oil.

Oh, yeah. Yeah.

Thank you.

I was like, having coffee, adding oil. But clarify what you mean by the higher level versus, like, the more implementation details. Like-

Yeah, yeah

... as high level, as general as, like, AGI.

Mm-hmm.

That's like our North Star, or is it more, like, granular than that?

Um, well, yeah, I mean, at the very highest level, right?

Yeah.

We have an org that focuses on pre-training, right?

Mm-hmm.

Which is, you know, giving models a lot of world knowledge.

Yeah.

We focus on RL, like, teaching the models how to reason with that knowledge, how to chain the little insights together.

Yeah.

And then finally, um, alignment and post-training, right?

Mm.

And, um, we're always looking at both, like, how to scale the mainline in each of these domains and also new bets that fundamentally unlock either, like, different scaling properties or more aggressive scaling properties.

Gotcha.

Mm-hmm.

And so even in that, I heard that every one to two months you go through what, like, 300 projects, like, research projects that could be, um, you know, followed through on. Is there a way that you kind of hone that decision-making, I assume, as you, like, decide what to actually double down on and what not to, since I assume there's a lot of talented researchers who provide possible ideas-

Yeah

... to pursue?

Yeah, so I think really in the spirit of, um, of focus, so one-

Yeah

... one narrative you might have heard is, you know, we're, we're really focusing our bets at OpenAI, and-

Yeah. I've heard

... um, we're also trying to do a little bit more, uh, directive compute allocation as well. So-

Okay

... um, I don't like micromanaging my managers. I think one important thing is to empower them.

Yeah.

But to just kind of give compute, big swaths of compute to the big bets you wanna make. And then, um-

That's what you mean by directive, like-

Yeah, yeah, yeah

... how to-

But, but then also give them kind of flexible pools of compute-

Mm-hmm

... which they can, you know, freely allocate to things that, that they believe in or-

Yeah

... just kind of, uh, fudge with the, the allocations that-

Mm-hmm

... um, that we prescribe. So I think, um, yeah, I, I think it's just tying, let's say, like, a small number of bets, say, three to five bets from each org-

Mm-hmm

... um, into the main research roadmap, and then really letting the, the managers, um, and org leads take things from there.

Gotcha. Okay.

Mm-hmm.

That makes sense. And I guess for rising researchers-

Mm-hmm

... so, um, let's say in an interview setting-

Yeah

... are there specific tells or ways that you can identify, okay, this person has some, you know, potential of becoming a researcher to impact an org in a specific way? Or is it, like, just looking at the previous research that they've done, and then that is what heavily dictates whether they can actually continue on?

Um, it's a hard problem before someone comes to OpenAI.

Yeah.

Um, I think that's, that's genuinely true. Um, I, I think, um, for a lot of the best research managers-

Yeah

... you know, they work with so many researchers over time-

Mm-hmm

... um, where you kind of develop an intuition. Like-

Yeah

... the things that they say, the ideas that they bring up.

Yeah.

Um-

That's important as well

... yep. Are, are those kind of... Do they hit, hit the same mark? Or, like, w- are they the things that you would be thinking about personally too?

Yeah.

And so there's this gut check of, like, you know, does their intuition match, um, the same intuition that you have?

Yeah.

Um, but it is really hard to tell out, out of the gate. Usually in, you know, let's say six months to a year, it's-

Yeah

... pretty clear who, who's, you know, ha- has the strongest trajectory and who's gonna make a lot of impact. Um, so yeah, honestly, um, I, I think it's a hard problem, but just having seen a lot of people, you know, go through research development, um, at, at OpenAI, you develop an intuition for, um, you know, who's more pee-pee in different areas.

Yeah.

And one thing to kind of, like, just mention there is not every researcher is the same.

Mm-hmm.

I think there's a lot of different types of impact.

Yeah.

There are the people who just take an idea, it's very clear, and they'll just implement it before anyone else.

Mm.

There are also the people who just come up with the kind of like crazy, almost too crazy, but-

Moonshot type of-

Yeah, but somehow not that crazy and-

Yeah

... and, and they, they really convince you in a, in a different way of seeing the world or, or, or, or another completely different type of project. So there's a lot of ways to make impact.

Yeah. No, that's helpful.

Mm-hmm.

And so I guess elaborating on that-

Yeah

... would you say there are similarities that you would see between, let's say, like top engineers-

Yep

... um, and top researchers? Like I often hear top engineers-

Mm-hmm

... even at like small companies and startups are ones-

Mm-hmm

... who can take an iota of an idea-

Mm-hmm

... all the way to production.

Mm-hmm.

I assume in research it's like coming up with the idea all the way to how it's delivered to the end user, um-

Mm-hmm

... through like the product. Or do you think it's more so they're focusing solely on the research and not considering like the end design, how it's used by the customer?

Well, I, yeah, I mean, I guess the thing about research is many times the path forward is unclear, and so what-

Gotcha

... differentiates researchers is how- Often they're pointed in the right direction.

Mm-hmm.

How, like... Like you say, taste, right?

Yeah.

I think in engineering there are certain patterns that work. Like, you know, if you want to build a product that looks this way-

Mm-hmm

...um, the engineering principles can be pretty similar.

Yeah.

Um, but for research, I think the thing that's slightly different is just this ability to, you know, have good research tastes to convince other people-

Yeah

...that, um, what you're doing is promising.

Mm-hmm.

Um, and then, yeah, um, again, to just kind of integrate it into the core research roadmap.

Gotcha.

Yeah.

Great. Okay, it seems like we're done with the vegetables.

Awesome. Mm-hmm.

So now we have to multitask.

Okay.

So we're gonna pour some water into our pots to get the base of the soup going. So in the top right. Yeah, here. And then just twisting this off.

Cool.

So pour some here. Um, you can use some as well. And while we have this simmer and we'll add the veg, we'll cook our prawns here.

Okay.

Um, yeah, so let me clean this up real quick.

Looks looking great so far. I feel like saute got some color on the, um-

Yeah. Yeah, yeah

...onions and, and mushrooms.

Yeah.

So let's turn this on. I guess one aspect or area that seems very interesting are evals.

Mm-hmm.

Um, and more specifically, have there been instances where you've seen, like through just vibe checks that it's, it was really good, but on the actual benchmarks, like performs very poorly? Or do you think it's like heavily correlated that, you know, if your Suitebench Pro is, you know, a high number, then it's like your vibe check on it doing coding tasks is also really, really high.

No, no. I mean, I think there is this phenomenon, um-

Mm

...you know, I, I think internally, I, I'm not sure if this is a externally used word, but yeah, just like benchmarking-

Yeah

...you know? Um-

Good high school

...I, yeah, I think, I think you can kind of overfit onto certain distributions.

Mm-hmm.

Um, and it, it won't be reflective how you, how well you generalize, right? Because-

Gotcha

...um, I mean, easy ways to do this are, you know, you take a benchmark and-

Mm-hmm

...you just find like very, very, very similar types of instances to the benchmark, and you overtrain on those instances.

Yeah.

Um, so I think, um, beyond that, the, the other scary thing in the field is the, the number of canonical gold standard benchmarks is low.

Yeah.

And we really are kind of in an evals crisis, right?

Mm-hmm.

Where all the really great, uh, evals that we all know, like growing up, like taking the SAT or, um-

Yeah

...those, those are all fully saturated.

Yeah.

And, um, we really need to find good new ways to benchmark the models. I think one great thing about tools like Codex is they've really enabled the fast iteration of, of evals. Like we're able to just kind of have one person just very quickly put together a very high quality eval.

Mm-hmm.

Um, another kind of interesting thing of just being able to deploy your models is you can just see them eval as people are doing things with them. Right?

Yeah.

Um, one of the great things is, you know, in math, and coding, and software, like you get a sense for like where, where they fall over, what the task horizon they can do from-

Yeah

...from this like general, very broad-based deployment.

Mm-hmm.

So.

Yeah. No, that's-

Mm-hmm

...that's helpful. Now we'll just add the prawns to-

Great

...the oil and get some color on it.

Mm-hmm. Cool.

Yeah. And so I guess double clicking onto that-

Mm

...um, how do you balance both doing well on these benchmarks-

Mm-hmm

...but also not, you know, like benchmark maxing, as you said? 'Cause I assume you want to be most honest and like-

Mm-hmm. Mm-hmm

...not kind of cheat the system.

Yep.

But if you have like lower scores, let's say than a competitor or from other models-

Mm-hmm

...to the consumer, maybe like, wait, your scores aren't that great, so the model just probably is not good. Like, how do you balance both of those dichotomies?

Yeah, I mean, I think the thing is you just really have to operate over representative mixtures of evals and-

Yeah

...um, always invest in creating new evals.

Mm-hmm.

Um, and yeah, really just like there's this philosophy of once an eval's out in the world, um, then it's, it's just already not a good eval.

Yeah. Right. Yeah.

Um, and I think one, one thing is, um, also just kind of partnering with external organizations to create evals.

Mm-hmm.

So, you know, in, in many of the kind of hard math and science evals, um, we've partnered with external organizations and, um, they've been able to kind of craft gold standards there for us.

Gotcha.

So yeah, I think, um, there's a kind of interesting philosophy of separate the teams that are creating the evals from the teams that are optimizing-

Building. Okay

...the, the models themselves.

Yeah. The models.

Because that way you don't like co-incentivize them, right?

Yeah.

Like the, the way the evals team can work is they're trying to build evals that are hard for the model. So there's this inherently adversarial process-

Yeah

...where, um, you're, you're not kind of cheating yourself, right?

Yeah.

The, the incentives are somewhat, um, aligned in the right way-

Mm-hmm

...uh, between the two teams.

Yeah.

Yeah.

And do you kind of also contribute and help in the ideation process or even deciding-

Mm-hmm

...what evals, you know, you should work with a third party on to develop?

Yeah. Yeah. So I mean, I think a, a lot of the work that Jakub and I do also involves just kind of us steering the direction the evals go.

Yeah.

I think we'll notice certain gaps, right? Or certain kind of capabilities, um, that we want, and every capability on the flip side is an eval, right?

Mm-hmm.

You need some kind of eval that measures if you've elicited that capability well. So yeah, I think, uh, yeah, it's, um, it's takes a lot of steering and just to get everyone on the same page with evals is also a lot of prep work.

Yeah.

Yeah.

No, that's, that's fair.

Mm-hmm.

I guess on Jakub-

Mm-hmm

...you said in a previous interview that he's a very funny guy.

Yeah.

Do you have any fun stories that maybe you haven't shared about working with him? 'Cause you also stated that you guys align very well, so your discussions even on research, um, are very efficient and help a lot when like driving towards the frontier.

I guess like on the opposite side of being very funny, are there things that you're, you can share?

Oh, you, you asked about like a funny story. Um-

Yeah

...well, he told me this joke yesterday, which I thought was very funny. Um, I mean, in many ways we kind of, uh, jointly manage the, the research efforts and-

Yeah

...um, you know, apparently some researcher came up to him and was like, you know, um, "It feels like I now just have an army of, you know, really dumb IOI like-" "...gold medalists."

Yeah.

And Jakub was like That feels like already the situation I'm in - ... in real life. So, uh, yeah, no, he- he's just, like, brutally sarcastic and funny.

Yeah.

Yeah.

No, that's great.

Yeah.

It's good to have humor in the workplace-

Yeah

... you know, to balance out, especially as you're pushing the frontier on very important work. Um, but that also brings to mind one kind of weird scenario of how models can perform very well on, let's say, the IMO or even-

Mm-hmm

... on the IOI, but may struggle with some more mundane tasks that a human can easily do.

Mm-hmm.

So I guess, how do you deal with that?

Yeah, yeah. I mean, u- ultimately, I think what's intuitive for the models is often not, um, that intuitive for the humans. Like, uh, there's, there's a lot made of this jagged frontier-

Mm-hmm

... analogy where, um, there's some things that the models just inherently, you know, based on maybe the data it sees or, um, kind of the, the things that we, we can teach it more easily-

Yeah

... it's just good at. Um, I actually think, you know, a lot of, lot of it boils down to also just context, right? Um-

Okay

... the models don't have a lot of context that a human has.

Mm-hmm.

Um, vision, of course, is something that's more naturally biologically wired for humans. Um, and so, yeah, I think there, there's just certain kind of jagged capabilities that models are better at than humans and vice versa. Um, but I also think, you know, context just, um, being able to take a single task, learn lessons from it, and apply them to future tasks-

Mm-hmm

... um, that capability is something that, you know, a lot of people in AI are working towards right now.

Yeah.

Um, but it's, yeah, very natural for humans.

Yeah.

Mm-hmm.

And on the context point-

Mm-hmm

... um, a very low-hanging fruit example that many people say-

Yep

... is just to increase the context window to provide more, um, examples-

Yeah, yeah, yeah

... so the model can perform.

Yeah.

But do you think... I assume there's more complexity on how to actually enable even with a large context window and a lot of context. There could be bloat or even just-

Yeah

... a lot of, like, context rot, as people have said.

Yeah, yeah.

So how do you go through that process of navigating that?

Yes. I think, I think there's kind of the canonical way you would solve for very long horizon learning-

Yeah

... which is, you know, you just naively increase your context window, right?

Mm-hmm.

And, um, I mean, that makes sense. I think there's a difference between implementing long context and implementing long context well, like you said.

Yeah.

Um, and, you know, there's a lot of kind of like needle in the haystack style evals so they can measure that. Um, but I do think beyond that, um, there are also a lot of, in some sense, like engineering and research shortcuts that you could take.

Mm-hmm.

Um, so, you know, like, uh, many, many coding products today have features like compaction, right?

Yeah.

Where, um, you can compress kind of, uh, either insights or, um, working state.

Mm-hmm, mm-hmm.

And, um, stuff like that, you know, it just shortcuts a lot of the, the very brutally difficult and expensive, um, primitives that you have to build with just native long context.

Gotcha.

Right.

Great. Okay, now we're gonna do the fun part.

Mm-hmm.

So, um, let's lower the heat a little bit.

Okay, yeah.

And then add a little bit more oil to the pan.

Okay.

Um, and then we'll torch the, the shrimp-

Amazing

... pecans.

Okay.

To get a little more flavor in there.

Yep.

So I'll first do it on, on my pan to show you-

Yeah, okay

... generally what it looks like. But-

One shot learning.

Yeah.

Yeah.

Indeed. Oh, wait. I, oh, I didn't pour any bourbon. Okay, wait. So let's pour, like, a fourth. Okay.

And then pour it in. Heat is off.

Ooh.

And then torch it.

Awesome.

Great.

All right. Okay.

And then once that's out-

I, I think I got this.

Okay, yeah. So do, do you wanna do it with your... Yeah.

There you go.

So pour it to, like, half of the fourth cup.

Okay.

And then once you have that, I can give this to you.

Perfect.

Great. So it's off. For the one sec. Oh, we can turn it back on. Perfect. Okay. And then now do you wanna hold on this and just press this button-

Yep

... to, uh, fire it up. Yeah.

Cool.

Yeah. Yeah, perfect. Great. Okay. Flambéing it. It's a little light, but yeah. Great, and then we could turn on the heat again.

Okay.

And then we'll just cook off the alcohol.

Okay, great.

But-

Great, great

... great. How, how you feeling? You know, we're-

Great, great

... basically there, cooking everything.

Yeah.

So... Okay. Awesome. Yeah, and I guess in terms of research ideas and-

Mm-hmm

... what to work towards, do you think there's still a lot of low-hanging fruit or ideas that can still be improved a lot through just optimizing small parts of already implemented work? Or do you think right now there has to be a lot of research that are completely new bets that people take?

Um, yeah, that's a really great question. I feel like there are new bets, but probably not that many.

Yeah.

Um, in some sense, like, uh, hopefully you feel like, you know, AGI is coming soon, right?

Yeah.

And, um, I think everyone sees that these models are getting really capable. And-

Yeah

... I think if you really imagine the implications of that, we're getting closer and closer to a world where the models can come up with more of the innovations on their own.

Yeah.

They can kind of do self-sustained research. This is one of the big org goals that we've set for, for our research org.

Mm-hmm.

And, and so I think, like, you know, what really matters is are there big bets before that point in time?

Gotcha.

And, um, I, I think the window is small, but there are still, like, some fairly significant ideas we're trying out.

Yeah. I mean, there have been some researchers who have stated that-

Mm-hmm

... to get to AGI, we still need, let's say, like two or three more breakthroughs. It'd be like continual learning or some other ideas.

Mm-hmm.

Um, do you follow that same view perspective, or do you think it's kind of more so, like, not as drastic as coming up with, like, three completely different paradigms for research?

Um, yeah, I mean, I, I don't know. I, I don't know if I have that same framing. Like-

Yeah

... continual learning is a, a basic primitive that you have to unlock.

Yeah.

Um, uh, there's so many different techniques. I, I don't know, um, that... Yeah, I think, you know, we're, we're trying a lot of, you know, permutations of it. I don't know what would consider as a breakthrough versus not-

Yeah

... but I think there are clearly many shots on goal, and I-

Yeah

... I am pretty sure they'll work.

Great.

Yeah.

Okay.

Mm-hmm.

Great. Okay, so the shrimp is basically done.

Awesome.

Okay, so do you wanna do the flambé thing again to get more color?

Let's do it.

'Cause I feel like...

Yeah. I'll, I'll-

Yeah, we'll turn on the heat a little bit, and then let me get some more oil.

Great.

Yeah, 'cause we wanna get, like, some dark color like Big Lion has-

Mm-hmm

... in here. Great.

Okay.

And then we can hopefully get another shot.

Perfect.

Yes.

Do you want, do you want to go first?

Um-

You can-

Same, same amount as you.

Yeah, same amount. We could hopefully get... 'Cause I think the heat wasn't as... Yeah.

Okay.

So you can put it in. Yeah, and then there you go.

Just press a button. You can...

Great. Wow.

There we go.

There it is.

It's good stress relief.

Yeah. Indeed. And it's like adds some good flavor here. Let me try it over here.

Yep.

Yeah, let's see. Great. All right, let's see. Let's go for... Whoo. There it is. All right. But we're in the final stretch.

Okay.

We have our shrimp-

Mm-hmm

... all cooked.

Yep.

With some fire.

Mm-hmm.

So now we can kind of, yeah, cook it off a little bit.

Yeah.

And then, um, we should add our veg to the water.

I'm impressed by your multitasking abilities. You know, I think that's actually one thing, um, we need our models to get better at. Like, it should just be able to do some thread like this and also just have a conversation with you in the background.

Yeah, yeah.

Yeah.

No, I, uh, that also reminds me-

Mm-hmm

... do you think images and audio and video and even text-

Mm-hmm

... like, that should all be one under one model or do you think it'll, like, break through, like, specific specialized, like, audio model or...?

Well, yeah, I mean, I think for, for a research lab-

Yeah

... I think there are a lot of advantages for it to being under one.

Under one, yeah.

So, um, you just have to maintain one infrastructure stack, for instance. Um, I think the cost of, like, maintaining and scaling many infrastructure stacks at once-

Yeah

... um, I think that's something that you shouldn't underestimate.

Mm-hmm.

So I think there are a lot of benefits to just, like, you know, you do some core research in, in your, like, in your fundamental stack-

Yeah

... and that just carries over to whatever modality or whatever thing that you want.

Mm-hmm.

So, um, I, I think there's a strong bias for us to keep it in as, as few different arch- uh, architectures as possible.

Gotcha.

Yep.

Great. No, that, that makes a lot of sense.

Yeah.

I think, like, the architecture as well is something that isn't often considered-

Yeah

... but it's very important. But one also term that I've been seeing a lot that you've also kind of mentioned is a vibe researcher. You know, we have vibe coders obviously.

Yeah, yeah, yeah.

But I guess on vibe researching, what do you think is, like, the end state? Do you think the main value add of a vibe researcher is just the research taste of coming up with the right idea, or do you think it's more so the execution of going through and following through on the actual research?

Um, yeah, so I, I, I think we're actually moving towards this world very quickly, right?

Mm.

Um, I think both at OpenAI and at other labs-

Yeah

... you're starting to see a lot of the work become mostly orchestration focused, right? Like, um-

Gotcha

... the, the researcher's coming up with the ideas, um, and the model's great enough to do the implementation execution by itself.

Mm.

Um, so I think, you know, when you, when it comes down to, like, uh, you know, is, is the value of coming up with ideas versus execution, um, yeah, both are still important-

Yeah

... but it does feel like there's, there's this market shift-

Mm-hmm

... um, towards just kind of being able to come up with a lot of ideas, and then, um, the model can actually do the, the execution-

Mm-hmm

... and orchestration for you. So, um, I think it's very much going to be the future of doing research.

Yeah.

Um, we also said earlier, you know, like, the models don't quite have the taste yet.

Yeah.

And, um, that's why you still need the researchers coming up with the ideas.

Yeah.

Um, it, it's gonna be hard to teach the models good taste.

Yeah.

Uh, we noticed that. But in terms of actually accelerating the research, there's clear tangible benefits already.

Yeah. Do you think there'll ever be parity in terms of research taste with models at some point?

I think so. I mean, when we look at our kind of three-year roadmap, right?

Yeah.

Um, the end goal that we want to reach is one where, you know, the, the models are just doing end-to-end research.

Mm-hmm.

And, um, I think a part of that problem is just being able to have the model come up with good taste. You point at some, you know, just generic benchmark or something, and it finds the right solutions.

Yeah, yeah. No, that's helpful.

Mm-hmm.

And in terms of research done by humans-

Mm-hmm

... um, at OpenAI, how do you guys go about, I guess, the postmortem process of, let's say, a research bet that didn't turn out well?

Mm-hmm, mm-hmm.

'Cause I assume that a lot of it is taking these, you know, vast bets and-

Yeah

... some don't turn out well.

Well, I would say that is a big part of OpenAI's alpha-

Okay

... because I think one thing that differentiates us from other labs-

Yeah

... is we take a lot of high-risk bets.

Mm.

And I think it's what's allowed us to stay at the frontier-

Gotcha

... uh, so consistently over time. Um, but it also means that some of the bets are not gonna pan out.

Mm-hmm.

And, um, a hard corollary of that is when a, when a bet doesn't pan out-

Yeah

... um, you have to, you know, not delude yourself into thinking that, you know, this is something that will work and, and kind of, uh, disconnect from it.

Yeah.

So I think there are certain calls that you have to make, right? Like, uh, kind of look back and be like, "Well, um, this was a promising idea at the time, but actually it's less important than we thought.

You know, it's, uh... there's some other approach that works better."

Yeah.

Or, you know, um, there's some other kind of, uh, some- something that we discovered.

Yeah.

But I, I think many of that, much of that work is also very fruitful.

Mm-hmm.

So what we realized is, like, uh, even sometimes when people fail-

Mm-hmm

... at, um, proving out a technique, uh, their write-ups are very import- important.

Mm-hmm.

Because, um, they'll kind of a lot... It, it'll often be a natural idea.

Yeah.

And you can kind of save a lot of people from going through the same pain.

Yeah. No, that's helpful.

Yeah.

So I guess when it comes to this positive view on failure, how do you balance that with, you know, a researcher, let's say-

Mm-hmm

... who takes a lot of bets, consecutive bets-

Mm-hmm

... and none of them pan out?

Mm-hmm.

'Cause I assume at a certain point you'd want a researcher to eventually have contributions that are actually beneficial-

Yeah, yeah

... compared to only taking bets that maybe pan out to being not a good taste.

You know, just through experience, I-

Yeah

... I've definitely seen some people fall into this. Um, but I've also had several cases where, you know, they... it's just, like, bet after bet, it doesn't pan out, and just when you're, like, at the brink of frustration- ...

you have something that's, like, a mega hit.

Mm.

And, um, this happened enough, so it, it really just depends on kind of are, are the ideas themselves sound? Uh-

Gotcha

... they can be ambitious- But they still have to be sound. And, um, there's a certain kind of person who will, will just take a lot of those ideas, and it's okay-

Mm-hmm

... because they're somewhat on the riskier frontier.

Mm-hmm.

But they only have to justify it once in a while for it to make sense. Like, maybe like a very trading-

Yeah.

... like, kind of lens on the world, but-

That's fair

... yeah, it's just on, on expectation, right? Like they-

Right

... may need to add value.

Yeah.

So.

No, that's, that's great.

Yeah.

Okay. So we're basically assembled. Now it's the finishing touches.

Mm-hmm.

So you can taste your soup-

Oh, great. Okay

... and then we just add soy sauce if it's not salty enough.

Great. Great. Great. Okay.

And if it's too salty, we can add some water.

Mm-hmm.

Need to lower this down, but let's see our final creation.

How is it?

It's pretty good.

Good?

Yeah.

I'll tell you. Okay.

Mine's really good. Yeah.

Mine needs a little bit of water. Could you pass the water?

Yeah, yeah, absolutely.

Great. So how was that? How did that feel?

Um-

Great naturalist?

... I think this is a student distillation. Um, you, you're clearly better than I am at this point.

No, no, no. I feel like you did a very great job, especially with, even with the shrimp and flambéing. Yeah. Smells great.

Wow, okay. That sounds good. I guess just more generally-

Yeah

... I'm kind of curious.

Mm-hmm.

Are there areas in research or topics that you think are right now overrated and underrated? Like, what would you categorize under?

Hmm. Um, well, I think if you still have a pre-training is dead view of the world-

Yeah

... um, I think, I think, uh, pre-training is definitely, uh-

Not...

Yeah, yeah, yeah.

Not a lot more.

Not, not dead. It's, it's, um, it's underrated.

Yeah.

Um, yeah. And honestly, um, I think products and kind of thinking about end uses-

Mm-hmm

... and, you know, how you tie all the primitives you build in research to, you know, real agentic use cases in the world, that's also underrated.

Mm-hmm.

I think, um, you really can't just kind of build everything in a vacuum and not connect things to utility.

Yeah.

Mm-hmm.

No, that's great. Great.

Yeah.

Awesome. I think we are ready to taste, so do we want to give it a go?

Let's do it.

We can move this.

Yeah.

And I think there should be some plating. Yeah. Do you want to take those plates over here?

Okay.

Okay. Shrimp looks great, but everything else over here. Great, and then we could just use these pots.

So we have our shrimp.

Yep.

Do you want to try it? Cheers.

Sure. Cheers.

Okay, let's see. It may be a little sweet.

Yeah.

That's a little too sweet 'cause we flambéed twice, but-

It's good for me. I have a sweet tooth.

Yeah. It's good. Okay, great. Um, Sean, do you wanna come and taste our tofu soup?

Sure.

Yeah.

That was so good, by the way. You guys can't smell it, but, um, it's-

We'll pretend you're a researcher I've tried to approach. And I'm Zuck, and he's, uh, you know, trying to get you. So do you wanna grab this spoon over here?

The soup, soup is really gonna sway the decision here. The quality of soup.

Yeah. Great.

All right.

Try both.

Right. What was the artistic, uh, direction here?

Um, artistic direction. Well, it was mimicry. I think it was great art in mimicry.

Yeah. Just letting them cook.

Mm.

Good?

Wow. Yeah.

Strong?

I mean, I think, um, o- one thing I, I... Definitely, it's, like, savory and spice that goes together.

Mm-hmm. Yeah.

But then also, like, the sort of sea- seafood-ness-

Mm-hmm. Yeah

... um, kind of really goes into it.

Great.

Mm-hmm.

Okay. Mine, mine's very balanced.

Am I supposed to pick a winner or what is, what's the, what's the deal?

No, no, no. You're just supposed to try it and then see if it-

No, you do pick a winner. Yes, you do.

Okay.

This is the evals?

Yes.

Yeah. Swift bench.

External evals.

Swift bench.

Okay. I gotta say, I feel like there's too much water in this. I think, like, the-

Oh

... the, the- ... the concentration-

I, I think it's also a pot. 'Cause this is a big pot.

Yeah. Wait, okay.

I feel like.

Um, I, I would s- I have to go for this. Um-

Okay. Okay.

Just, like, you're our respected guest, but I wanna be objective.

Of course. Of course. Yes, yes, yes.

For the, for the food.

Mm-hmm.

Um, and, like, yeah, the density, I think really, uh, uh, flavor really, um, swings it.

I- I do ha- half the water. Half the water.

Probably.

Okay. Make solid sense. Yeah.

I mean, I think it's very personal, right? Like-

Mm-hmm.

Yeah, I think it's also very personal. Taste, you know, you mentioned-

You do a lot of cooking

... research taste and-

No. No, no, no. Okay. I know a couple recipes.

Yeah.

Um, I follow them to the T. I can't... Like, if you tell me, "Oh, cook something slightly different," I have no... I'm completely lost.

Right, right, right.

Yeah.

The, the, well-

Busy cooking and doing research

... ChatGPT can tell you.

Mm-hmm. Yeah, yeah. Oh, I, I'm not gonna lie, I kind of looked up in ChatGPT a couple of things beforehand, like, um- ... just as prep, but-

No worries. But yeah.

Yeah.

It was, it was great having you. I feel like-

Mm-hmm

... you're always leading the field with a lot of research taste as well, and it's-

Yeah, appreciate you

... great seeing-

Yeah

... the work. So hopefully-

Absolutely

... this was fun.

Yeah, a lot of fun.

Thanks for coming on.

Yeah. Thanks so much.