The RLVR Revolution — with Nathan Lambert (AI2, Interconnects.ai)

Hey everyone, welcome to the Leading Space Podcast. This is Alessio, partner and CTO at Decibel, and I'm joined by Swyx, founder of Small AI.

Hello, hello, and we're excited to welcome back Nathan Lambert from AI2. Welcome.

Thanks. Fun to be here.

Um, I feel like I also have to say Interconnects and, like, the Lex Fridman Podcast and, like, you-- uh, and the IE World's Fair. Like, you've just done a lot in the last year and a half.

Not that many. Still say no to plenty of things.

Yeah. Yeah, yeah. Your first episode was als- with us was January 2024, uh, when you just joined AI2, then you released all the Olmos. You joined us again at NeurIPS, where you did the, uh, open models. Oh, well, Luka, Luka did, and you, you supported.

And then you're more recently here in, uh, in SF for IE. First of all, I wanted to congratulate you on winning the, uh, best speaker-

Oh, yeah. Thank you

... for, uh, the, the reasoning track. Here, you, here you go. I, I'm, I'm limited by Mochi.

Oh, the nice AI generated. I look too zen.

I know.

I look so zen in this AI generated photo.

So, so we had our track host, like, take photos of you while you were speaking.

Yeah.

And we turned it into Ghibli photos. But this one, your eyes were closed. Um...

It's fine.

Okay. Uh, we, we were s- we were trying to have Mochi, the reasoning Foamsky, join us, but I think she's, like, very-- getting very anxious, very, very restless.

No to crazy Mochi.

Very restless. Okay. Sure. Um, okay. So you've been, you've been doing, like, really good work. Uh, and, and honestly, like, I think one of the things that we wanted to kind of establish was Tulu and RO- ROVR. I guess, um, is that a good place to start?

Like...

Sure. It starts us in the recent journey.

Yeah.

I think that we can recap kind of the story of what Tulu 3 was aiming to be, and then kind of how it got folded into what the new narrative is.

Yeah.

What the goal is, is try to do the work to compress what are complicated industry post-training recipes into something somewhat tractable that you can modify on your own and do post-training at a what is, like, actual state-of-the-art level.

I think what we do relative to Frontier Labs is that we probably have a smaller amount of tasks. I think our post-training suite for Tulu is probably, like, ten to fifteen tasks, but I would guess post-training at OpenAI at all, you have maybe hundreds of evals.

And adding more evals is more data work and more mixing work and making sure you have these things. But on, like, core evals for our suite of models from, I think, eight, 7B, and four or 5B is based on Llama at the time.

It's like it matches or beats Meta on these core evals. I think Meta has different priorities and their things for Llama 3.1, which is a great set of models at the time, and it's just like, how do we distill what is very complicated post-training, um, explanations or diagrams from the like of this Llama 3.1 report where they have these complex feedback diagrams with many iterations and earlier signs of that from, like, Anthropic papers that have these multiple model variants and early, like, constitutional AI things for multiple years.

And it's like, what does that look like when you're doing a large scale instruction tuning into preference tuning and what else you might add? I think a lot of the core contributions of that before we talk about this reinforcement learning thing, it's like we showed to s- how to scale up preference data.

It's just, like, the academic community had been using this one dataset since, like, all the way back in the Hugging Face models of, like, Zephyr be-beta is when this ultra feedback dataset got popular, and still a year later is, like, this state-of-the-art dataset for open preference tuning.

And it's just, like, one of those obvious things that doesn't need to be the case. So it's a big trying to make more mature, um, recipes available to people. And I mentioned this either on one, I think on China Talk with Jordan.

I mentioned the origin of the RLVR thing, which is, like, realistically, when you work in the open, a lot of it is trying to match what industry has done. And we're on a different path because our infrastructure is different.

So some things that OpenAI does now that works really well for long context won't work that well for Olmo because we might not have enough flops in our base model. We might not have certain datasets for legal things.

But directionally, like, a lot of it is just trying to reproduce things. And I've, like, long tried to get John Schulman on the pod of OpenAI, Anthropic, and now Thinking Machines. And at the time, he had gotten approval to, like, chat with me.

And just like-

Ooh

... it- what he said was l- so confirming a lot of the things that I had said on instruction tuning and multitask and preference tuning, and he was like: "Oh, yeah, everyone just does RL on outputs." And that's how we got the RLVR idea and scale it into something that is a general method.

There was a lot of reasonably or very similar works at the time, like VINE PPO and QuietStar, on doing these math and coding domains for getting verifiable rewards. I think the RLVR thing was about doing it in general recipes.

Yep.

And the naming was something that stuck. I th- O- originally, we had, I think it's especially, like, Costa Huang, who's was our kind of lead RL engineer at AI2, who's doing some stealth startup now. You can hear more from him now on that soon.

I think he's founding engineer of something. And Hamish Iverson, who's still a student at UDub, were leading most of the technical work on this. And the naming was gonna be RL from ground truths, but then it's like the verifiable rewards is actually a more general notion because only, like, math questions have a ground truth where code is verifiable, precise instruction following is verifiable.

So I think it's a nice evolution of the name, which makes sense as you look at more domains, which is now why it catches on with people. Like, once Jensen started using it, it was like, okay, that's, that's set.

That wasn't really our goal, but that's, that's, that's going-

You think that's where it took off?

No, that was, like, in it being taking off 'cause it was after DeepSeek. But it's like when people like that have the acronym on the slides. And th- that's- it's also very clear of, like, RLHF is four letters.

It's like we want to evolve that and have a similar-

Yeah

... four-letter acronym. It's not that much magic to it, but there's definitely intention on these-

Sure. Sure

... on these little things.

R- RLGT may not have worked as well. I don't know why, but yeah.

Yeah. Yeah. And that's like that's what these people, like, all that were definitely thinking, and they made that name change, which works, which was fun.

You did mention, so we'll, we'll show, uh, you, you kind of mostly quoted from the Tulu paper there, but, uh, we'll, we'll show the RLVR chart. You did mention that you wanted to change it now, and we'll, we'll sort of preview a little bit of the agents discussion.

Yeah, I think, uh, when you are introduced to RLVR, it's... there's just a function really that checks if the out-- you have a string outputted from the language model. You have a relatively simple function that's like, "Is this answer from the language model correct?"

And there's no real environment because you're just looking at the generation. And now I need to figure out the right way to communicate what, um, either, like, multi-hop tool use looks like for this, which is something people are definitely doing.

Yeah.

Thinking like what is the right diagram to encapsulate how o3 is trained, which in action they take multiple actions because the next sequence depends on the feedback from the environment, which is some sort of information store. So, like, when it's searching for a niche piece of information, it...

you, you can't know what the next actions are without whatever feedback from what it... Bing searches is what they say they use. That is a step that is very much happening, and then as people try to transition to more end-to-end RL is a real strong notion of environment, which is that you're looking for a sparse signal from this multiple generations, and that's what people want to do.

I think it's debatable whether or not people are actually doing it now. I think the Deep Research blog post kind of hints that they do a bunch of small scale RL and then poof, the system works. Which I think is much more of what's happening, is people train on a bunch of small things and they do some prompting, and they see that when you put these pieces together or a couple different fine tunes of a model.

So it, it seems like Deep Research has some fine tune of o3 in it. And so you do that with some different domains of RL. It works rather than Deep Research being trained on the outcome, which I think makes a lot of sense for it not working in Deep Research because doing outcome-based RL for Deep Research would be RLHF again, 'cause you have to have two humans and you're like, "Which, which generated report is better?"

I th- you can definitely do that and you... the whole sycophancy thing at OpenAI showed that they have so many different reward models and reward signals in their post-training, but that's just one of them, and I think a lot of the progress in making it exist is doing RL on a bunch of information retrieval and editing and search tasks.

We talked with Noam about, Noam Brown, uh, about this, uh, Deep Research and kinda like the verifiable rewards. He mentioned obviously that's an example of like non-verifiable thing, having RL work on them. And in one of your recent posts you also talked about how the big labs have all this data that they can find long tail things to RL on, and then kind of when you put them all together, that fixes it.

Do you feel like what we're able to verify is like a big bottleneck? That like the verifications are only done in kind of like these smaller atomic things, and so we cannot really scale that?

I think my comment was on making... So in this post I was like reflecting mostly on the question of what will agent progress look like relative to modeling progress. So we've had almost three years of modeling progress, and we're pretty used to the messaging on that.

And it wasn't just about being with the RL on small things, but do any post-training to fix a weird behavior. And RL is a very data efficient way-

Mm-hmm

... if you can get the right signal, but you could also just say like it does this weird non-verifiable thing. Let's create 100 or 1,000 instructions to include in post-training so that the model does this types of information extraction correctly or like soft extraction.

It's a space that I wanna flesh out more with more examples of tasks. It's just if you watch Claude code going, it's like what is it doing in the background?

Mm-hmm.

It's a lot of reading files and even just the compressing context. Like that's not-- I don't think that's really a verifiable thing, but that being messed up, like that's a super crucial skill for long context actions and long- longer tasks, is just compressing well.

And that's-

Mm

... gonna take some training novelty on how do you... you can effectively modify your training data instead of having all the multi-turn context. You just insert the summary and you wanna make-

Yeah

... the performance stay as well, 'cause it's also a cost saving to have shorter context. There are just a lot of new domains like that.

But do you feel like you can figure out what these things are before you release, or do you think the labs have like a big advantage because they have so much user data that they can kind of like inspect this at inference?

I think it's mostly looking at real world data at this point. To the extent that there are clear benchmarks, you can use them in the open, but I, I mean, we see the industry consolidated around data in different forms, and I, I think that's a real important touch point for people.

I'm curious who's like still collecting reliable sources of open data that everyone uses.

There's a lot of action in the space, but hard to get traction.

Yeah.

So I think for a long time preference data has been something where people understand that it'd be very good to have large repositories of it. If you want that, you can an- like annoy me to try to release all...

like for Tulu, like we have a final data set, but we have completions and ratings for more models. Like I'm talking to the student as let's figure out how to mark this down 'cause we just have so much completions, and LLM is a judge AI feedback data that we don't know how to clean.

That's one thing. The, the problem is that I think a lot of it is task and model specific, so this notion of like on policy to adopt a RL word for just this preference data and preference modeling, which is that you want the sequences that you're training this reward model on, the sequences of generations to look like the model that you're starting to fine tune.

Yeah.

That is something that has made it hard to kind of grab off the box and it, it's for example, like this ultra feedback that I mentioned is just has a lot of models in it. So most of the models that people are fine tuning, there's some signal for it to improve on.

And I don't know how long that lasts, and we still don't have the answered question on how important human is versus AI feedback. Every time I check in with people at Frontier Labs, they're like, "Yeah, we still use human preference data."

And I'm like, "Okay. I don't have access to that and I don't know how to measure how much it gives you really." It might be most of the benefit is on the... What's the right adjective to describe Chatbot Arena?

It's like people are down on Chatbot Arena, but it might be that the human data helps boost retention time and general preference a, a lot, where most academics were doing multi-skill and, like, Alpaca Eval type things, which it's just- it's not as crucial to everybody's fighting in the attention co- economy.

The attention economy. Uh, your quick- I mean, since we're there, you mentioned Sycophancy, you mentioned LM Arena. That was one of your posts on Interconnects that I really enjoyed. Are they cooked? Uh, is there a future for arenas?

Like, how does this play out? You know, they got $100 million now, like, what are you gonna do?

I don't know what the money does for them- ... but I think the, the eval is still valuable.

Yeah.

Especially at the frontier, people are very cynical, but in the compression race of how much cheap- like, what is the cheapest model you can have that does pretty good at this is still so useful to a lot of people.

Like, chat is king.

Yeah.

You know?

Everyone chats with these things. It's-

Yeah

... it's why I use... And GPT-4.5 isn't as good on Chatbot Arena. I think it's, it's higher on, like, Yup, which is a new competitor into this. It's like they have, like, a vibe category, which-

Sorry, Yup?

Yeah. There's, like, yup.ai. There's... You can look it up.

Yeah.

It's a competitor, another startup. They have, like, a cat- they all these companies have categories, and one of their categories is vibes, and GPT-4.5 is on the top, and I'm like, "Okay, there's something with those tracks." Um, but-

It's a frontier model

... yeah. It's just like I- I... That stuff intangibly is very nice. The leaderboard is established. People still should use it. It's kind of a focusing function for the community across different batches from industry-

Mm-hmm

... to academia.

Yeah.

I'm not gonna try to solve their monetization problems for them, but having clear norms and things that could be hill climbed forever is very good. Like, having this idea of an Elo linking for models-

Yeah, that you cannot saturate.

Yeah. You just can't-

It's kinda cool

... it's, it's a great, it's, like, it's a great problem. Like, what is-

But you can game it. Uh, so I think that's the, that's the issue.

Yeah. But every- everyone evaluates on multiple things now.

Coherer came out. Like, Sarah Hooker, I've never seen her so public about any of her... Like, she has gripes, but she doesn't really go public like that.

Yeah.

Artificial Analysis also has one, which I, I think is kinda cool. Uh, the other thing I think is relevant to this discussion is a lot of the data actually is, like, single test, like a single round.

Yeah.

Like, it's not, it's not multi-turn. And I wonder how to create proper multi-turn arenas because you have to switch the models. That's the whole premise of LM Arena.

It depends on how valuable the user data is.

Well-

If the user data keeps being equally val- uh, equally or more valuable than the inference, there's gonna be a platform to keep pushing this into more and more expensive things.

Yeah.

So they're gonna set up a deep res- they're- I mean, they're probably setting up a deep research arena where, because that's the data that-

Oh

... I mean, if I was OpenAI working on deep research, that's the data that I want, and there are competitors. And LMSYS is the entity that has the market placement to set it up.

Right.

I mean, it's almost like how I see scale. It's like scale kept climbing the edge-

Right

... of what AI data processes is, and because they're the name brand, they keep climbing the incremental evaluation game, and a lot of them have longevity.

Yeah. Yeah.

It's, that's a, it's a network effect in, in some ways.

You mentioned scale, which is another hot topic, but, like- ... we'll, we'll put, we'll put all the sort of hot takes at the end. But I do wanna, like, you know, focus, try to be technical up front. Try to...

You know, you're u- you're still writing the RLHF book. I... Is it RL, RLVR book now?

I can give my spiel on it. Uh, ultimately, RLVR is not mature enough, nor is it as interesting of a book. So I'm like-

Okay

... on two... So those are the two fronts of why I don't want to rebrand. And there's also some personal career strategy, but that should be independent on, like, what is objectively a good book.

Yeah.

Because RLVR is gonna be changing so much in the next 18 months. We've already seen it. There's all these new algorithms, but I think there's a lot more under the hood on how you do the right pre-training for it and what the data is, how tool use emerges.

All of this stuff is, uh, core to what RLVR will be seen as. I'm watching to see if o3 is, like, a niche model or becomes the path that everybody needs to follow on its kind of different style of tool use that you see particularly with search.

Okay.

And we don't know how OpenAI did this, and these are the things that I think is kind of core to an RLVR book that we don't have.

Mm.

Whereas RLHF is a more interdisciplinary... In the same way that Chatbot Arena can never be saturated, RLHF can never be solved. And we kind of know these problems of alignment and over-optimization and what the pipelines to getting data that people are using are.

And yes, I can add more RL algorithms to the book, which is nice for me to study, but that's not really changing... It's not changing, like, oh, what reward modeling is and the different ways that people implement these today, whether it's a value function or reward model and stuff like this.

So I think the, the breadth on RLHF is nice, and I think... I, I would tell a lot of academics that I think RLHF problems are gonna be foundational and kind of just have a much more steady study rate, where we're on this massive spike of RLVR.

But it might just be solved, and then it just goes back to zero academically. It's not... It's, it's an embellishment, but there could just be a best practice for getting 100% accuracy on any problem that you want-

Right

... and then, and then it's solved to where the debate on what is a preference is gonna go on forever.

Yeah. Because it's verifiable, b- there, there is a right answer.

Yeah.

Sorry, what, what do you mean by, by, like, over the next 18 months there'll be a lot of changes? Like, what, what do you foresee? Actually, let's just catch up. Like, what's already happened, you know, in, like, the sort of recent history?

Yeah. So there's two categories of information that we have, which is what are the models doing and what are the researchers doing.

Yeah.

I think the models provide a lot of inspiration in terms of what the, like, what's actual frontier is, and that's things like o3, Gemini 2.5, Claude. These are a mix of just... o3, I think, is the most scaling RL approach, and then Claude and Gemini 2.5 are very similar with- ...

hybrid reasoning models that you can turn on and off. They, they rolled it out in different ways, so Gemini didn't have hybrid reasoning at launch, but they're giv- they've, they've brought it in, and Claude had it at launch.

One of the most important questions has gotta be is, is the o3 path of just a reasoning model or hybrid reasoning models, like, more useful? Do they diverge in their methods for training them? I think the NVIDIA Llama Nemotron reasoning paper is probably the most detailed paper on a hybrid reasoning thing, and then DeepSeeker R1 is still the canonical recipe on a, like, reasoning-only model.

And those are very different approaches, and I don't know if one will win out or not. And then there's just a lot of work on data side and RL methods. I think there's a list... There's a whole list of kind of GRPO complaints that are out there-

Yeah

... where the math doesn't make sense for certain things.

To me, every, every paper I see come out always has, like, some fix to GRPO. It's kinda cool that, like, people are, you know, taking variations on it, but also, I don't know if DeepSeek's gonna come out with R2 and just blow away everyone with whatever is next.

Yeah. I definitely don't think the algorithm tends to be the most important thing. Like, I think I had this in my AI Engineer World Fair talk, which was kind of a snarky of, like, how do you train a reasoning model, which is like you get a starting data set, you incrementally improve the data set.

You do that until you're running out of time or your performance starts going up-

Right

... and then you try all of these switches from all the papers, or you turn all the... You do a whole bunch of binary tests of all these various algorithmic-

Yeah

... changes, and you do a grid search and see what works.

Yeah. Like, candidly, that's why I dismissed GRPO when it first came out because I s- it was sold as an efficiency thing.

Yeah.

And I was like, "Okay, fine," like, but, like, you know, I, I, I was... I've been trained to not care about efficiency because it's just a matter of resources.

Yeah. The GRPO advantage estimate is very well-suited to verifiable rewards.

Right.

But the other thing is kind of a intangible, works better on the infrastructure type argument. And when it c- like, when it came out for DeepSeek Math, which was well before the RLVR phase-

Yeah

... so it was really marketed on as that.

When you talk about hybrid models, how do you reconcile that with OpenAI saying they wanna move away from the model selector to just having a unified interface? Do you feel like they feel pressure to, like, "Hey, look, when I have all these different classes, we wanna route them to the right thing"?

Or do you think there's something else?

I would think that OpenAI wants to have a model that knows how hard the question is. I think that has to be the North Star for most people working on reasoning, which is ID- the model will just spend the right amount of tokens on it.

And if you look at a compute-level discussion, it see- and, like, what inference time scaling means-

Mm-hmm

... I think in plenty of ways, like, hybrid reasoners might just be aged out except for niche-

Right

... applications because quality is so much more important than having 100X less inference tokens.

Mm-hmm.

It's like you just pay for it and compute, and that'll get better. I think it's, like, really, like... That was something, like, Jensen said in his most recent... I think, like, Stratechery highlighted it or had the interview with him, and it was like, "Yeah, everything's gonna be a reasoning model because it's gonna get so cheap, and they're better."

And I was like, "That, that's why it's like the-

That tracks

... hybrid reasoning thing is a little bit weird, and it's like I always just will turn reasoning on unless it's a really-

Right

... silly query. Like, oh, I, like, what is this thing? So it's like, okay, like, in two years, that kinda tracks, which, like, I think o3 is also just burning money on us. I mean-

Right. Yeah, yeah, yeah

... it searches 80 websites for me asking some- what paper it is. Like- ... that's a lot of tokens, but it seems directionally like if that's the thing that works, that'll be the default-

Yeah

... at least in all of these high... Most of the things that people that we talk to, whether it's coding or very high-end information economy, those thing, the value is, is there.

I wanted to double-click on something that you seem to be coming back to a lot. You, you, you seem to assert that o3 does something very different by using t- search a lot, much more than basically everyone else.

Yeah.

Do model- do all models come with a search engine now? Is that, like, a must-have?

It depends on your use case.

Yeah.

If you're doing general information retrieval or understanding it, yeah.

Yeah.

There's old papers that we can try to find the links. I think... I don't know if Sam Altman was talking about it, but there's just this retro paper from DeepMind and other architectures that people have been pulling in the discussion again, which is like you have a very small model with a very big context length and a very big retrieval store, which I'm not one to bet against the transformer architecture and just figuring out long context and stuff like this.

But those are ideas that people are bringing back, which is search, search is better. You look at all the evals from reasoning models, and one of the trends is that, like, um, simple QA numbers all drop. It's like DeepSeek-R1 to the new R1, it goes down.

It's like all the new, like, Qwen 2.5 to Qwen3, simple QA goes down, at least when you're evaluating these without tools. And simple QA is, like, a v- what is considered to be a very nice, fairly numerically robust, like, long-tail knowledge evaluation, and all of these, the raw models, they're all going down.

Right.

But it just ma- like, long-tail information, just to have this search behavior makes a lot more sense.

Okay. The counterargument for this, just, um, I have been through this journey too of like, oh, why don't you make, like, a m- model that doesn't know anything but search, right? You can search up anything that you want and learn just in time.

Uh, the problem is you'd need to know what the search terms are. You need some baseline intelligence to, uh, make all this work.

Yeah, that makes sense. That's a, that's a good way to put it.

I think it's important because there's this thesis of, like, LMS becoming just online LMS, like, permanently, and it hasn't been super pursued. Like, Perplexity was one of the first to put it on my radar as, like, they were like, "We'll attach the search engine to the LLM," and that's what you get now.

And I think, like, more and more people are starting to offer it as part of their default services. Like, Gemini has, like, a, a search grounding thing as well.

I mean, it's what people say a big limitation of Anthropic is 'cause it uses Brave Search, which returns a bunch more, like, SEO slop than-

Is that proven? Uh, 'cause I, I, I don't know. I th- I thought they had their own index.

Okay. So I don't, I don't have-

Okay

... I haven't done a detailed look-

Yeah, yeah, yeah

... so I'm dealing with rumors. But I, I think they'll all do, end up doing their own index, and it should... It's one of those things that's like Google should have an advantage again.

Yeah.

But who knows if they do? I also hinted at this in my post, but it's like Hamish had tried to set this up, the same student from RLVR playing with like search and an RL model, and it's very easy to get the model to do tools if you prompt it to, but it's very hard to get the, like, RL model to learn that the tool is useful.

And that's why it's... To go through these things where it's like 80 failed tool uses and it still gets it or, like, it stops or it gets it on the 81st-

Okay

... is just the RL behavior that feels emergent from having a very nice way of, like, getting the model to learn to use the tool, and it's not, like, you can't SFT this model to do this. Like, it just really feels like they set up the environment right and it plugs into this deep research kind of line of work that they did, and they broke down the problem into these sub-RL tasks, and then it kind of lets it-

Yeah

... do this thing.

Interesting.

I don't wanna be an OpenAI shill all the time, but I just think... I tell people to play with o3 all the time because it's, it's weird.

It's excellent. I would say, like, the amount of work you're imputing on the deep research team when, like, as far as I know, it's three people did it. It was Isa and, like, the two other collaborators that she had.

I don't know if they did that much on top of o3. Like, every indication I've had from OpenAI is that deep research is more or less a thin wrapper over what is o3.

Yeah. It's probably like one or two small things- ... that it... They're like, "Oh, we can make our... We, we can make deep research work by adding this small amount of data to the training thing," and then it just s- just works.

That is... That would be how I describe it.

I mean, so, I mean, what is it? Gwern, the anonymous person-

Mm-hmm

... he replied to my, like, Q* post on Twitter the other day, and he was like, "Why was this all wrong?"

Yeah.

And I, and I'm... It's obviously, like, simple things don't scale. There's a lot of complexity 'cause there's a lot of other exciting things in the AI field at the time, and OpenAI kind of sends out a lot of things that confuse people.

Yeah.

But this would fit into that, which is deep research is a minor change from an existing RL trajectory of what was, like, o3 probably they had already figured out that search was gonna be better, and then we're like, "Okay, we can repackage this."

And it's a, it's a simple thing that makes a big difference.

Yeah.

And most of the things are like that once you have traction.

Yeah.

I think once... Trying to get the initial takeoff on the sigmoid is the hard Q* thing, but then once it's like, once it's like this, a lot of things in the middle feel obvious. Which is why I describe one of the things that we work on for Olmo, it's like a lot of it is just having motivation to do things that feel somewhat obvious, but they're still hard.

Like, it's, it's hard to get different recipes or it's hard to get a full reasoning recipe off the ground. It's just, like, a huge change 'cause you have all this inertia on this eval suite, and then you have to figure out if you branch your recipe or do you start from, like, do we just take, like, open reason or zero and start from scratch?

Which is like, uh, it's a whole other headache of, of things. It's just hard to move these projects that are anywhere above five to 10 people with inertia to get stuff done. But then once you're hill climbing, things can seem really obvious.

Yeah. Okay. You covered a lot there.

Before my next question, just to close the Brave thing, our friend Simon Willison wrote a post that Anthropic added Brave Search as one of the sub-processor in their product.

Yes.

So that's where the thing came from.

Yeah. Well-

Now, to what extent it gets used, we don't know. But-

We don't know. I, I, I would, I would just kinda comment on a couple of things that he said and then, and then we'll go into your, your question.

Yeah.

There's a very good post on just on the retrospective of Q*. There's a very good post, uh, that you had, which was... That I wanna send people to is... Which is, like, is, was open- was o1 a psyop?

Right. That does imply the question of, like, if o1 was a psyop, what, what else could be psyops now?

Yeah. There's definitely a psyops out there. I mean, the whole inference time scaling plot is such a psyop.

Why?

Where you, like, put these two things next to each other with an X-axis, and it just looks like it's easy to control... Whenever you see an X-axis, you think it's easy to control it. Whereas, like, for training on r- the left one was training.

Yes.

And training makes a lot of sense. So if you haven't, if you... Especially even if you go to really old RL papers, RL learning curves are a f- non-log X-axis usually, and they look like this. They look like these, like, whatever, like, logarithm or exponential rise.

And then if you take one of these and you make it a log X, it's, it's a straight line. So, like, that side is like, "Oh, okay. We've seen this before with RL." But with inference time scaling, it being an X-axis is why people are like, "Oh, there's a knob.

I can turn search up a lot."

Yeah.

Which is, like, what breeds all these weird ideas. The core of that article is just they're taking points from within training or there's a natural variance, and then you line them up, and if you line them up, then you get this nice inference time scaling behavior.

Which is... And now people, a lot of people have reproduced this plot on inference time scaling, and it's, it's, it's much clearer now, but at the time it's like I, I see why I thought it was a knob.

And it's like, "Oh, look." "It's im- they call it inference time scaling. You control it."

I think the most interesting... Well, you have a lot of interesting things in your blogs. Uh, but one that stood out was about RL and tool use. You said that it's easy in RL experiment to tell the model to try searching, but then if it doesn't get results with the tool, it's gonna stop using the tool very rapidly.

Can we impact that? So can there be a good tool that the model doesn't know how to use, and then it kind of fails, and then it stops using it? Can there be a bad tool that should be improved before giving up on it?

How should people think about designing the tool, improving the model, and kinda like where to intervene?

This is definitely on the newer side for my things that I wanna work on or have worked on. It, I think particularly in 2026, it's, like, the, especially in the open sides, all the infrastructure models will caught up a lot where I want to go deeper on this-

Mm-hmm

... in terms of, like, deep research style things or very inference-heavy multiple calls. And to answer your question that there definitely can be bad tools, and there definitely can be, like, the model just us- using them wrong. And something that I would want to see in a model is kind of not necessarily creativity, but, like, an openness that it doesn't know exactly what it'll get out of all of its tools and this uncertainty to just try a few different things.

Which almost seems classical, like, RL behavior.

Mm-hmm.

But if you think about what a language model does, they're always very con- they're not necessarily confident, but they have, like, a path and, like, a direction in their answer. Whereas that's a big change in these reasoning tokens is to have the notion of backtracking and, and things like that, which is some sort of, like- Openness to the tools, having things that are unknown in it seems like a really nice thing for the model to have, which is like, "Oh, what if I try this?"

Like, what does it get?

Yeah.

Especially in the, on the open model side, which is if w- if this is gonna work where people want to use open models with tools, it's gonna be because people have private data stores and stuff. So if you were to train an open model that is gonna be a good reasoner like O3 but on private records of some sort that'll never get sent to the cloud, like it needs to be thinking of like-

Mm-hmm

... I can try some things-

Yeah

... with this to get a sense for it before saying that I have to give up. And if you look at tool, like tool use m- right now looks m- seems much more similar to like code execution, or it's just, it's just a part of a sequential path that you need to get to, which is like, I have a plan, and if it fails at a certain step, I might have a backup.

But it's not like this iterative of I need to fiddle with the environment in order to-

Yeah

... come up with my plan. It's just that I- it's something that people probably are gonna have to train into these models, which is like, you might just tell it. Like you don't know what is in this, but your answer might be in it, which is like a very odd prompt, but maybe it'll help.

Yeah, when we had Eric Schluntz from Anthropic who worked on the Claude agent before Claude Code, he mentioned they spent basically like majority of the time on like the tool design to get to the model, and then you just kinda learn how to do it.

Are you usually... Well, I don't know how much you worked on actual this stuff, but are you putting the tools one by one in the RL process? Do you think that helps or do you usually give... Is it better to give all the tools and let the model explore?

I don't really know. Like we haven't gotten this to work. I would say it would probably depend on the model and your starting point. If your starting point is already good at tools, it can probably generalize more.

Yeah.

But if you're doing this weird base model RL and you have to have this kind of curricul- like if you scale RL long enough, you're gonna need a curriculum of things getting harder. And like that's pretty obvious. So in that case, it might be tools get added when things become too hard for it to solve certain questions.

Which would be- which sounds very intuitive, but also just really hard to manage and practice.

Yeah, yeah.

'Cause what is your automated signal on your training run that it's time to do that?

That's why video games are so good, because they're designed to k- unlock things as you progress. But I think like with things like search, it's like, you know, if you're given access to a small data store or you're given access to all knowledge on the internet, you know, it's like-

Good, good feedback for the Arcagi people for the V3 benchmark is like have things where the language model needs to learn to use-

Yeah

... new, new actuators in the world after a certain threshold.

Wow.

Yeah.

That would be Arcagi 4 then.

Yeah, probably. I don't know. They're, they're cranking them out.

They're cranking them out. They're actually doing a launch party, I think like in a couple weeks. So I'm actually really... Like it's fun to play Arcagi. I don't know if you tried.

Oh, I haven't.

Uh, it's pretty fun.

Yeah.

Like these are IQ tests. I used to be like, oh, like they weren't that relevant, but like actually now that we have a gradient where like LLMs are actually significantly climbing them, now it's actually really more intres- like interesting to compare your own intelligence to the LLMs.

I'm, I'm with Noam, Noam on no, no harnesses.

No harnesses? Yeah.

Yeah. I mean, harnesses are cool, but they're gonna... They're, they're a handicap that's changing the learning dynamic substantially. So it's good, it's good demos, but I feel like the core thrust has to be no harnesses.

To me, it's always like, is it wrong to say that these are just inductive biases, right? Like, they're not in the model, sure. But like anything where you're just like looking at the results contaminates.

It's just a different task. I think do- it or I mean, I've t- I think I talked with Greg about this at Arcagi, which I told him like do harness and no harness. You just have both-

Oh, right

... different categories. So it's like, uh, you're trying to be transparent and build targets for Frontier Labs, just do both. Like I don't think it dilutes that much. It'll... The no harness is gonna obviously be harder, but then you just get more bang for your buck on your benchmark.

Mm-hmm. Yeah. It's the same data set. Staying on the topic of tools while we're at it, you had a really good summary of like recent work in multi tool RL, which was- which had like LOOP and Retool and Torl and all these other things, and I think that this is just like an area that's super rich for research right now.

I just wanted to give you the space to like highlight what are your favorites, what do you think that people should explore?

I could share what my moderate ambition, what would be fun research project things is, is you want to create some sort of competitive dynamic or eval, and it has to be so much narrower than what industry is doing.

So I, I, I told you this at lunch, which is like, um, deep research, but only archive papers. So you like don't have to do-

Yeah

... a full index. You have a limited domain. You have to figure out how to measure it or something or some... I think like ac- it's good for academics to work on academic tools 'cause they have very high domain expertise.

They already know what's going. And just like figure out how to make that something that is either very useful to users if it's gonna be good enough at that, or something you get whole climb on. And I don't know if this is like, like brainstorming on the fly of like take related works out of papers, just look at the text and break all the links and make an eval which is filling in hundreds of related works with archive links.

Like that's a fun-

Oh

... deep research style idea. See if you can do it with open models on a set data store with tools. AI2 has gone through a lot of discussions with this, which is you, if you're trying to have impact in AI right now, it's as an academic, you have to like level up out of papers to artifacts, which is models, data sets, evals.

Data sets and evals are easier for people to have impact on. And then the next thing is like, what do people actually use? In AI2 especially in this like semantic scholar team that's now working on like information agents of different types, there's another thing that I'm like distance in, so I don't have all the names.

But it's can we make open models do that side of thing better? It's like, can you make something that people actually care about? And then you're... That's a whole level of impact that's much higher if you have actual users.

It's, it's hard for academics and small institutions to do that.

Mm-hmm.

But if you're working on agents like- ... dog feeding is viable. It's like, can we make ourselves a good Slack summary bot that we like or something? And just making these agents really tractable. I mean, that- that's one direction.

Another direction is just hill climb on humanity's last exam with tools. I just think it's kind of unlikely that we're gonna win as a academic and a sto- state-of-the-art number because they're gonna start spending millions of tokens per query.

And it's just a lot of, it's a lot of compute burn. Like the getting, beating that on the flop equivalence is gonna be so hard. Unstructured thoughts is something that I'm mostly like, "Okay, I'll get to this." Like I have, I have more things to figure out on the modeling and s- what I call like skills level, which is just how do you do reasoning to induce inference time scaling and get high eval numbers.

And once you know you can do that-

Yeah

... you can take your knowledge with you to do it in more specific domains.

There's skill and there's skill acquisition, right? I think the ArcAGI definition of AGI-

I quoted it. What is it? It's like efficient... Yeah, skill acquisition efficiency, 'cause he just- he described it as three words.

Right.

Yeah.

Your emphasis on skills in, in your recent talks that you've done, do you wanna sort of reiterate that, that thesis for people to pick up on?

Yeah. So I've been thinking about mostly I'm trying to get ahead of what OpenAI, et cetera, are doing probably now if it's not in their models. And with all the agents it seems that planning is a very critical task.

So it's kind of how do you come up with a taxonomy for different types of things you need to train into reasoning models for when it'll be a bottleneck. And the found d- so I came up with four and the foundational one was skills, which is what I would say that we have already done with o1 and r1, which is you do a lot of RL, you show the inference time scaling works and you get really high benchmark numbers.

And then the next three are kind of what comes next and most of them are around planning. So what I had as three and four on my list were abstraction and strategy, which is trying to not use planning because planning is a word that people already use a lot.

Or strategy would be the direction the model should go in and like technically like what are the steps of its plan. And then abstraction is how does it break it down into things it can actually solve. And then the fourth last thing is calibration, which is just not wasting compute and knowing when to like give up and ask the user things because like overthinking is obviously a problem.

It's easy to keep getting your eval scores to go higher by using more inference time scaling, but eventually like that's not what people want in their models. They want, they want a smarter training regime where the model is actually getting proportionately better for its training and not...

There, there's a lot of papers on overthinking and stuff like this, which I think is like OpenAI wants it 'cause they have to foot the GPU bill. Like if o3 just infinite loops itself for a bunch of people, like that's not good.

Does it actually?

I don't know, but it might.

Okay.

I mean, like these reasoning methods definitely can make the models just kind of unstable and just, yeah.

Yeah.

So it's like... But it's also the GPT-5 idea which is how do you get a model that just routes the question to the right.

Mm-hmm.

Uh, not, maybe not necessarily a router, but just knows if it needs to do a plan or if it can just answer. If you look at DeepSeek-R1 and you ask it a hard math question, it's not like, "Here's my plan of attack," it just starts.

And having a model that knows when to be like, "Okay, here's my plan of attack. I might need to make myself a memory store. I might need to take like a Claude Code approach for this query. I'm gonna build a memory store and spin up some parallel searchers and then come back."

And conceivably this is all something you can train into a model 'cause the, the searches or the parallel models could be like tools in that case. The simple way to describe it is we have something like thinking tokens and then answer tokens, and it's the model should be able to optionally have like plan tokens before, before thinking or before using tools.

It's like, okay, like here are the table stakes. I need to do these things and these sorts of tasks will be harder versus easier. It seems more tractable than some far out ideas for AI. It's like I, like a language model can write a good plan, it just needs to be asked to do so, which I'm, would bet that Claude Code and Deep Research are doing this.

Like you get a user prompt and first the model is like... Yeah, there's a plan tool in Claude Code and they first, they break it down and it's like that-

Yeah

... is something they've trained into the models. Like Deep... I don't think DeepSeek has, doesn't have it built in, but it, it probably could do it. And just thinking about that interface between like if the model needs it to be able to do the task end to end on its own-

Yeah

... like can it do that sort of thing.

I think that my, my challenge with this whole reconciling this approach with the no harnesses thing is that I think a lot of the way that people, especially engineers, want to model it is that the plans and, and the memories are tools, and there are no special plan tokens, there are no special memory tokens, it's just context or it's just, you know, whatever.

Specifically for planning because then you can do fan out to other agents for tool calls and stuff so it doesn't have to be sequential. But I'm just like, is this a fork in the road? Like or, you know, do we have to make a, a real choice here as to do we outsource things to tools or do we keep it native within the model's tokens?

I don't think it's a subjective difference. I think mostly the planning idea is to make the point that people don't get things for free. And the planning improvements might be kind of mundane-

Yeah

... which is like we were prompting Claude and its plans were bad in this way. Let's give some data where its plans are more detailed or break things down into more steps so that it's easier for them to do it.

Yeah.

'Cause it's, it's in a black box effectively so if it hasn't been targeted, it's unclear of what the performance will be.

Yeah.

Or on the like open model side it might just be the idea of having different models for different parts of it is then, then you're really training a model to just be good at planning and like that's, that's data that you need to come up with and then you j- you only use that model for that one part of it.

Does it feel like plans are much more reusable and should maybe not be generated every time? I feel like especially in coding for certain sets of tasks-

Mm.

-you wanna have similar types of plans. So maybe it's not the right way to ask the model to regenerate a plan every time. There should almost be like plan blueprints as like tools, and then the model fills it in.

Like where do you think the balance should be?

I think they're reasonable. A plan is obviously an intermediate goal. I just-- it seems likely that there's like failures on this kind of planning level. I mean, the same thing goes for these rubrics that are popular, where it's a lot of the technique that is popular for so-called rubric things is you have a prompt, and you have a language model generate a rubric for that prompt, which is a few specific things it needs to get right.

Mm-hmm.

And that's conceptually very similar to making a plan for, for every task, I think. Whether or not it's... Like grading is a-- you're gonna have a different type of abstraction than executing. But I think in what people are seeing is that it's cheaper relative to the effectiveness-

Right. Yeah.

-to just generate it. So like plans are not super long-

Makes sense

... and they probably d- they're not that many tokens, so it's probably just kind of like, "Okay, we do this." Like, uh, putting it in my taxonomy might be overselling it, where it just needs to be a prompt, and you just need to make sure that- ...

your model's not too weird at that prompting stage.

I think your taxonomy is super useful, by the way. So, uh, skills, calibration, strategy, abstraction. I feel like maybe abstraction might be the most underrated one or hardest to solve. The way that you introduced it, it was different than how you wrote in your blog post.

You said it, it was basically not to overthink. That's calibration.

Yeah.

Abstraction is about breaking things down.

Yeah. I think both of these, strategy and abstraction, make the most sense on the hardest tasks-

Okay

... that we don't know if the model can do them.

Right.

So is if like if you're assigning a task to a model that you don't know if it can implement it, the strategy is very important because it, it needs to be very specific and narrow. Where if it's doing mundane code like or, or deep research, the plan is actually not that interesting of a thing.

Yeah.

But when you're at the frontier of if it can, like, I don't know, some GPU-implementing thing, you could buy in, buy into the OpenAI and Anthropic narrative, which is, "Help me implement this research idea in our complex like mul- distributed GPU thing."

My God.

It's like this is a task that's hard for a human, and for an AI to come up with the right plan to debug and do this is very narrow path. So therefore, the strategy is pretty important of does it start with certain tests, and how does it actually build this out to complexity?

It's obvious that I need to come up with more better examples for this, but I think as you push it, it's more natural to see that there's only a few plans that actually get it done. And then abstraction is just important as your task becomes so big.

It's like a prompt engineering thing almost.

Yeah. And it's like you only have 100K tokens you can generate. Like- ... you need to make sure the model breaks it down, so it's not just spawning a ton of infinite processes under itself. Which I do agree that abstraction is an, is an interesting one, especially when you start to think about these models that could call in other models to do some tasks for it or parts that could be parallelized with-

Yep

... like multiple searches or just more compute. I think that kind of folds into abstraction, which is just like how do you approach a certain nugget of the problem? And I'd definitely say like I don't have experience building this.

It just feels like if you're going to visualize AI doing the hardest software or other tasks, it's something that humans are very good about.

Yeah.

So it's like how do you come up with a research plan in 10 weeks? Like there's a lot of how do you prioritize which experiments to do? It's like there's a lot of inductive biases that go into that that I don't...

Like a language model would not do well at that right now.

Probably memory would be helpful there, so you can just get... Like the way we do this in real life is we accumulate experience.

Yeah.

One thing I, I did want to dive in on, on was, uh, just parallelism in general. There's one case where with o1 and sort of the, the, the sort of Q* ideas, there was one case where it was sort of overhyped in some sense.

But now it's coming back with o1 Pro and DeepSeek. The theory is at least, and c- and correct me if I'm wrong, but basically they run o1 eight times, and then they have a reward model rate it, and then give you the best of, of the eight, yeah?

Something like that.

Something like that. DeepSeek also the same. I-- we don't know any, any details beyond that. I think there's a lot of people exploring that, at least on the infra provider side of like, you know, how do we parallelize, uh, search and planning and, and all that.

And I'm worried about getting too hyped about it. I think it makes a lot of logical sense, and this is one of those things where MCTS also made a lot of logical sense, and we were fooled.

Well, I don't think we're using parallel compute in a way to search over like low-probability tokens. We're using it to get robustness. If you... Like o1 Pro is-- it was so nice because it just had a, a very predictable depth to it even on niche topics where like sometimes models just fail out.

Yeah. You, you had some numbers that it went to like from like ten to like ninety-five percent or something.

I don't remember-

Okay.

... the exact numbers. But that's what it feels like. It doesn't feel like you turn on o3-Pro to make it ten times more likely to find some niche piece of information. Like, it, like, maybe it'll be a bit more likely, but we're not getting that type of like search-y notion of getting more breadth or depth into our tree.

So I think there's value, there's value to it where we wanna use this parallelism on the what are either like the most important tokens that we're generating or like, okay, I know this part is crucial. Let's, let's just spend a bit more so that those tokens are better.

But it's not a like transformative thing. The part that's potentially interesting on the transformative side is like if you can get much better verifiers. So I think of verifiers of changing the slope of inference time scaling.

Yeah.

You spend more tokens at inference, the better verifier you have. If you're doing parallel, it can extract a rare occurrence.

Okay.

So like right now, our verifiers are only like they're good at like human preference. It's like, okay, we don't need to, we don't need to crank that up very much. But if we are doing really diverse generations and your verifier is better, it'll get better.

It'll do better. I think you could look at the extreme between a reward model and an oracle where it's like- The oracle is the more you search, eventually it works, so the slope is, is good. But a reward model is, like, there's really a capped signal-

Mm-hmm

... out of it. At, at least if you're doing this preference type of thing. So the slope is pretty minor, and it kinda has diminishing returns.

Mm-hmm.

So I do think that like if you could fill that with more interesting verifiers, there's potentially more to get out of parallel compute, but I, I don't think it is, like, as transformative right now on my outlook. It's more like parallel agents makes more sense of, like, if you could break down abstraction nice, like, as a throughput engine if our tasks-

Yeah

... are taking a long time, rather than a, like, at peak performance engine.

Okay.

Yeah.

Which kind of fits with the whole agent versus model thing, where agents are much more about, like, getting it done at all, like, being robust and being fast. Where, like, this model is one generation that's like, "Can you get the answer right?"

Yeah. We'll spend a little bit more time on this and, and happy to move on. My pushback or counter to this is that it's a way to pull forward a hypothetical future model that you can then distill from.

Yeah.

Which is nice.

Well, I bet people... I mean, they, they surely will use these for synthetic data. It's just, like, the marginal gain on synthetic data is always very high. Or just, like, Amanda Askell will say, like, better prompting will effectively make it seem like you have the next generation model, where, like, most people don't put effort into their prompts.

Oh my God. Okay.

Or she had said something of those lines in one of her-

Yeah

... Anthropic interviews. Which was just like-

Yeah

... if you can really figure out how to kind of get into the certain states of the model.

Yeah, yeah. Well, anyway, that, that, that's my pitch for, like, why this is worth doing at all and, like, you know. I have a science fiction story that I wanna write about quantum models. In a world where, like, we- you could explore cheaply multiple universes then, like, you know, sort of pull forward the right one, that would work.

This sounds too science fiction-y, but I feel like in a world where we could control quantum computing well enough to explore this and scale it up enough, it could be kind of cool.

It also could be that parallel compute is grounds for interesting types of innovation. Like, I... Like, I don't know, like, what does it mean to have parallel compute with diffusion language models that generate all their tokens at once?

Like, does that meaningfully change some sort of application? I don't really know. I think it would be... Like, the diffusion language model would be fun if it works-

Yeah

... you'd have much more control over inference time scaling. I mean, like, Gemini has one and it's, like, hard to suss out what it changes. But once we have all these knobs, I'm hopeful that it helps build some interesting types of, um, innovation.

'Cause, like, the parallel stuff is new, architectures can change. We'll see.

I've been using the Codex best of them thing, and I feel like most of the generations are, like, you know, 5% different from each other.

Because, because you use Ruby.

No, no, no. I had a JavaScript one. I have a JavaScript one, so I should be good at that. I don't know if it's, like, just how the RL encoding works. One thing that I've noticed, these models always wanna do if statements when there's, like, a missing end variable so that it doesn't fail when it runs, and I feel like that, to me, that's just, like, a symptom of the RL.

Yeah.

The code is terrible. Like, no, you should not write code. Like, it shouldn't silently fail if there's missing variable. It should just raise an error.

Yeah.

But I, I feel like the RL is, like, pushing the code in this direction. And then all the generation have the same pattern. You know, I generate fourth thing, all of them use the if statement, just in different pieces.

Yeah. That, that is something I- we will definitely get over.

Yeah.

That's just, like, the labs are trading off massive gains in performance for small detriments in usability.

Yeah, yeah.

And it's... Like, do you ship that model? Yeah.

Right.

Like, you just ship it and deal with it later. But I'm sure they can fi- I'm, I'm sure that's a fixable thing.

I think, like, to me, that's the question is, like, you know, you talk about how you have gains in, like, pieces of the thing but not in the full trajectory sometimes. Do you feel like these are examples of that?

Or do you feel like as we get better, if we did a longer trajectory where instead of just writing this piece of code you have to think about how you're gonna maintain it later and, like, how it's gonna run, that's gonna fix it or...

It's hard for me to grasp.

Yeah. The software stuff is not easy because it's almost... Like, maintainability almost feels like a human preference type-

Right. Yeah, yeah

... issue again. Where somebody could look at it and be like, "Yeah, that's not as good." But adding the heuristic in training seems very messy.

Yeah.

So, so maybe it, maybe it is. I, I don't know. There's a lot more to dig into that, I mean, like, this is what Anthropic says they're doing and just what are the actual frontiers in making... Like, they say they're working on code only and what does that actually mean?

Mm-hmm.

A bunch of it is gonna be design trade-offs and, like, how much autonomy the model has versus these potential, like, side effects from training longer-

Yeah

... that we don't know how to get rid of. I mean, that, that definitely could be the sort of... A behavior like that is what I would say is, like, a simple thing to remove, where it might just be obsessed with some code format that fails when you revisit it or something.

If it- even if it's... Like, everyone has seen it with just bypassing test cases. I think they'll be a bit more nuanced than that, but they could probably be-

Yeah

... super simple.

This topic has a similar semantic content address for me as over-optimization, which is something that you've written about.

It is over-optimization-

Is it, right?

... with a different, with a different reward function.

I, I know. I, okay. Well, I wa- I made that link. I want to verify-

Yeah

... that, uh, we, we are thinking on the same wavelength. Uh, I just wanted to go over, again, specific topics on things that you've, you've spent some time thinking about. You write that there are three types of over-optimization.

First was RL for control, second was RLHF, and third is RL, RLVR. They always happen. Obviously RL is no stranger to reward hacking. But, like, y- maybe, like, do you wanna elaborate on how things are evolving in, in terms of how we're learning as an industry?

Yeah. So that three things breakdown is for people to p- put the pieces together for what has happened historically. Um, all of these over-optimizations are a... Just the model optimizer is strong enough where it can manipulate- The agent with respect to the environment or manipulate the environment in, in, in a useful...

in a way that's useful to its target signal. Also, like for context, I think with what we're doing with language models in RL in general is that if there's something that can move its reward signal up, it'll move the easiest thing, the most direct things to move that signal up.

So that's part of the story that I said on sycophancy, which is this reward model for user feedback was probably so obvious that humans just like to s- like stuff that is... like people press that thumbs up button when they're-

Long emoji-filled bullet points.

Yeah, like all those things have just been really easy for the model to extract.

Yeah. Love that.

So like once they added it, the model changed a lot and the score went up a lot, and it was easy for the RL to find that. In control, the oldest RL, the environment is normally a simulator that is fixed.

There's no feedback. So the overoptimization looks like unphysical and nonsensical behaviors. There's the motorboat example going in circles. There's like example is a project I was middle author on was like effectively overoptimizing like Half Cheetah, which is this MuJoCo thing.

Instead of running it did cartwheels off into the sunset and got like infinite numbers. It's like obviously not the intended purpose. It looks like a glitch. So it's just kind of manipulating the, the agent interface with the environment.

RLHF is kind of a classic case where the model will just break down because the reward model is imperfect. So like the environment is really imperfect in the RLHF case where-

It's so sparse. It's like very artificial.

Yeah. Yeah. It's a very artificial environment, so it makes sense that these actions which are generated tokens will do things like reduce into just repeating one token over again, and it'll be like... I think one of the early examples we had playing with this at Hugging Face was the model would just say JavaScript.

It would be JavaScript, JavaScript, JavaScript. It was like, uh, some toy dataset, and it's very obvious when you see it. It's probably harder to see when you're at the top and making deci- decisions on when to stop training if you're doing a lot of RLHF.

But that was kind of the phase that people have gone through, and now we're in the RLVR phase, which is we're giving the model reward when it does something, quote-unquote, "right" for math. It's a bit harder to overoptimize, I think, unless you have tools and the model learns to search and cheat instead of learning math which I'm sure somebody could see that out in the world.

But just like, "Oh, just find the... You're train-" It's like the model's like, "Oh, you're training me on Stanford's problem set for CS," whatever that it's seen a thousand times. So it's like, "I'll just go get the solution manual," which I'm sure, uh, there's...

somebody can find an example where that has truly happened.

Mm. Yeah.

But on code and maybe information retrieval, it's easier to fudge. So the code thing is like the easiest way to get a unit test to pass is just put a pass in it. And like that is not too surprising that a model can learn how to do that.

And there for code, you need more reward design, which I think would be a nice for like a substantial academic work is like what is reward design in code for balancing this sort of like understanding this overoptimization of test cases or avoiding failures or something like this.

I'm sure there's... it's not just only gonna be a controlled environment 'cause these models are complicated, but I would guess you can reproduce that in, in some ways.

Just to double-click, reward design means like, for example, giving credit, partial credit for partially correct work.

Yes. Or like giving the model a slight penalty for doing the unit test thing if you can detect it.

Cheating.

Yeah, for cheating.

Yeah.

Which is it adds a lot of complexity to training these models compared to math, which is just if the answer is right. This is a, I mean, you can look at the GRPO math and partial credit is weird in that because it's kind of normalized per batch.

I don't know if I have a whole spiel ready on it for that, but it's also just it becomes very complicated if you're mixing domains and it's like is partial credit in code better than partial credit in math or all of these things.

It's like reward design becomes very complicated and that's what you're incentivizing the models to do different things.

Yep. Is there any literature or hypotheses about mixing these things? So let's say you have the, the one for code, you have the one for math, you have whatever other verifiers you can come up with, and individually they work.

Do they conflict?

I think part of the intuition of RLVR is that the model is good at knowing which prompt area it is, which is why the models like don't get worse on knowledge benchmarks if you're training on like just math or precise instruction following.

So the model just kind of develops an intuition for like where the different prompts are in space. So the gradient updates will be different depending on your batches, which is partially why people will just say do big batches.

So like a lot of the model is activated, and you have a less noisy signal with RL. But a lot of the intuition is that the model just kind of handles that, and there's interesting questions on sequencing. Like, do you do large scale math and code RL to get the sequence length and then add in more general stuff?

Yeah.

Which DeepSeek mentioned, but that's one thing to go... the DeepSeek report is like math and code to more general RL. There's a question on where do you do tools if you're gonna do like, um, code execution and search within this.

So I don't know if that's interweaved or if it's a second stage.

Got it. Yeah. I, I, I don't have comments there. It's just like it's surprising how much, um, is not known, and you just need a lot of compute for ablations.

The inference, high inference length generations definitely just like kind of breaks all infrastructure 'cause there's just so many tokens.

Yeah.

It's more opportunity for out of memory or other things go wrong.

Yeah.

So it's like just on a default, all of your training jobs need way more GPUs for the memory of inference.

Sure.

And, or just like training, but it's just, it's just makes it-

Yeah

... more of a pain.

Yeah, that's a cost thing. Um, you know, one of the maybe controversial takeaways from the known prod which you listened to was that there, there's also just wall clock time of just getting r- feedback from the environment, whatever that is, especially if it's like a real world thing.

And I'm just like, yeah, there, I mean, there's some point at which your training runs cannot take longer than like a human life. Like so, uh, to me, that was the wall. Like he, he, he disagreed with that, but like that, that was what I meant by it.

Like at, at some point, you long inference you- You do want it to terminate within some reasonable amount of time regardless, just as a user.

Yeah.

We have to find a way to accelerate internally within the training time faster than the passage of time in the actual universe.

Yeah. We're not c- I'm not worried about that problem-

Okay

... but I agree with you in principle.

Right. Okay. And so I'm, I'm, I'm stretching this out too far. I get it. I get it.

As we kind of start wrapping up, what are other interesting ideas that people should pursue? Like in your AI talk, you said, "What I'm thinking about for scaling RL," you had big multi-domain data sets, uh, difficulty filtering, long run times.

Is there anything specific that if there's people out there that are either doing research or they wanna do a company or whatever, these are like interesting things that you don't wanna do, that you want other people to explore?

Most of them I think are not in the reasoning space, which like if the talks have been about reasoning. So I've been long talking about like character training is something that I think is, um, under-indexed on and been advising a student that's been-

Character level?

Um, like personality training-

Okay

... and how that, like, like different ways of changing the personality of the model from prompting activation or fine-tuning.

Okay.

Or like data engineering. So stuff that like Joanne Jiang does for OpenAI. So like, like, like how much does that matter? What are the fundamental research things? Hopefully can share more that I've been advising a student on that.

So I've been saying that for a while.

Do... Uh, just as a side, side note, do you like the model spec stuff that she's doing?

Yeah.

Okay.

Yeah.

That, that, that trajectory.

Yeah. So I've, I've been a early fan of that. I mean, that's how she final- That's how like she noticed me, as like the only person that covered it when they first released it. I think it was like over a year ago.

No, I li- I liked it. I-

Yeah.

I don't know.

Well, not many people did, so.

Okay. All right, all right. You were first.

I don't know. I don't know, but like that's what she said to me.

Well, we had a, you know, we had a model spec talk close the whole conference, right? Like, uh, that was my sign of like the, pay attention to this, guys.

But it's, it's real because of what it sends to like develop... It has a developer benefit of like where your model's going, and then also just like regulatory. I think it is very important to like what is like an intentional behavior versus just like a training error?

Okay.

So I, I think for model transparency, it's really fantastic, and I've said that like the model spec is much more useful than a constitution. 'Cause a constitution's like an intermediate training artifact that you give to the training algorithm in order to get the model that you want.

Yeah.

It is not necessarily like what model did we... Like we don't write down our goals of the model-

Yeah

... in a constitution form.

By the way, have you looked at the Constitution?

Not recently.

They, they, they talked about it. Uh, they, they put in like Apple's like design guidelines-

Yeah

... but then also like the UN like declaration of-

Yeah. So at this level I've seen it.

Okay, yeah.

I don't even know if they've updated it. That's very odd. I hope that Anthropic would write a model spec. I'm not too optimistic. But th- they're the next domino to fall.

Well, so my take on that... Actually, I, I pushed for this too late because OpenAI already approved the talk and all that, but the... I was gonna ask them to compare the OpenAI model spec to the Cloudflare system prompt, which is their closest thing to the model spec.

It's the system prompt is incomplete because OpenAI has mo- things in the model spec that their model doesn't currently do, or especially when they started. It's like we want to... When they first released it, it was like we want the model to be enga- able to engage on like s- uh, sensitive subjects and maybe like even NSFW was in their model spec.

Yeah.

Which is they're just signaling of what they want it to do, and they say like, "This is very hard to implement because there's all these obvious risks of doing this. But it's like in an ideal model where we can solve every problem, this is what we do."

Which I think is good, as I said, for many different stakeholders. So I mean, mostly my thing is like there hasn't been a good like foundational research paper on that. There's just a lot to do. It also runs into personalization and personality are similar, which is like if open models are to win, part of it could be just like everybody can have exactly the model they want.

We're serving GPT 4.5. It's kind of its thing. You can prompt it. But if fine-tuning is more effective than prompting, everybody can have the model that they want. So it's a good... It's like a, an academic problem or an open ecosystem problem where people are fighting on the turf that it feels more likely to win.

Yeah.

Which is good.

Is this somewhere where you, like as speaking as AI2 omo, you, you want to win? Or is this you're just advising a grad student on it?

Um, I don't think it's a differentiating factor nec- yet, but I'm very open to working on it.

So I think like open models have a strong, you know, role play use case and, you know, like character personalization, all that stuff, right? Especially because people, like they find their waifu, they wanna keep their waifu, and like that's the-

Yeah

... derogatory term for it, but like-

I would say that we've definitely discussed it, and I want to... Part of omo should be that it is a base model that's easy to take in directions that you want, and we will have an opinion that is probably slightly conservative on personality.

I mean, I've gone through the OpenAI model spec, and it's like most of these we, we agree with and like be conservative on anthropomorphization.

What do you, what do you disagree with?

Uh, I don't remember. I did it a couple months ago. But a lot of it is like openness or transparency, which is like if we're training an open weight model personality, like we're not gonna withhold anything.

Yeah.

And we have a different hierarchy. So most of them are on like that type of information exchange rather than be kind. Like OpenAI's model spec is pretty agreeable and if you, you read through it, and it's like-

It's weird

... treat the user with respect and all these things. So-

I'm raising kids that way, is read the spec.

Yeah. It sounds kinda stupid. Um, but then the last thing is for people doing research, it's like wacky model routing things where you figure out like a bunch of different models to off Hugging Face to route to, because an open model tool thing could use way more models more easily than any OpenAI product.

'Cause OpenAI is restricted-

Mm

... to the OpenAI's models-

Mm

... where if like maybe, I don't know, Open Router's like, "I'm gonna make a product out of this," which is a router. Like Open Router actually does it, and they're like, "Our chat window knows the best model based on all this usage that we have-

Yeah

... for your query."

There's people that started other way, like Martian, Not Diamonds. I, I don't know who else is... He would know.

There's, there's a bunch.

There's a bunch.

Yeah, so I don't, I don't know if that would work. Hugging Face should work on it. It's like, like-

Mm.

Right

... it's, it's a moonshot idea.

Yeah.

You don't know when it'll appear.

Given your Hugging Face background, what is, what is... How does Hugging Face make money? This is a very common meme question.

I think, uh- ... mostly like enterprise deals.

That's what they say.

Which is like- They're doing their thing. They're supporting their people

I mean, yeah, look, they're, they're great, they're big, they're profitable. Uh, it's just not that obvious to most people.

Yeah.

I like the router idea for media models. I feel like there's, like, so m- there's, like, a long tail of, like, a-

Like generative media

... background remover-

Yeah

... like a style-

Yeah

... applier. Like, that, that is actually hard to find. On the tech side, I feel like just use the big mo... U-unless-

Yeah

... you're, like, under some, like, latency or price constraint, you should just use the best model. Even when we're doing thumbnails, I'm like, "Okay, I'm trying to remove a background of somebody," and it's like, I go on Replicate and there's like 55 background remover models.

Yeah, I just use Adobe because it's a website and it's easy.

Well, but that doesn't work. Like, the- ... Photoshop model is bad-

Oh, okay

... on some things. But, but again, it's like... Or I wanna generate, uh, a diagram to, like, mimic something, and it's like, "Well, which model is better at diagrams?"

Yeah.

You know? And it's like those are not easy to find because none of the benchmarks are like narrow.

Part of the argument is that if distillation works really well, we could just keep making the target for distillation smaller and smaller, which is you have models that are very narrow-

Right

... and they're mimicking these huge models on something that's, like, pre- I don't know, like reformatting tables. It's like can you do a table reformat or from Markdown to LaTeX in a 100-million parameter model? Like, uh, like if you get it small enough-

Yeah

... that is really economically feasible 'cause it's effectively free at inference-

Yeah

... and instantaneous.

My pushback is, on this is just if you're doing image editing, 40 should do it, do it all of it.

Well-

You know?

Yeah. Well-

Like-

But I think it does. Like-

It's just, we're just not there yet. Like, give it five years, it'll do it, right? There's so- so why work on a router at all? You just scale up 40.

I guess I-

I-

Yeah. Yeah.

Right?

I don't know.

Like, like tell me where the, the logic h- is here. Like, this is, like, a temporary thing.

On, on, on device.

On device.

Like, the local modeling community I think is much smaller than people give it credit for because most of the use for open models is still in APIs. It's like DeepSeek API.

It's convenient. It's convenient, yeah.

And it's like if there aren't that many models, somebody's gonna host it for cheaper than most people doing it themselves. That's pretty realistic. But there-

Yeah

... there is a small community that need local.

Yeah.

The best outcome is if open models can compete on not just long tail things, but that takes the most transformation.

Uh, side note, so I resisted by building my own, like buying my own GPUs, building my own cluster for, for this reason. I'm like APIs will, will solve most of it. Like, people are losing money to serve me models, why am I, you know, having those?

Except for the fact that 4090 prices have doubled in the last year, so actually you made money doing local models.

How does that make you money? 'Cause your, your investment goes up?

Yeah, your hardware investment.

You can sell it in card and it'll make you feel better.

So as used 4090s goes up. Interesting.

Should've bought a 4090. I got a 4070. Damn.

I was like, w- what is this? Well, then, then, then it puts me on tilt. Like should I buy, you know, a 5090 if, if it ever, you know, is, is widely available?

Well, at GDC they were doing the drops.

Yeah, I know. I know.

It was crazy. People were, like, running to-

I know

... to the camper to buy it.

Um, any, any other topics before I give a closing question? Just generally your work, RLVR, like, or topics of the day.

I think companies should keep considering re-releasing open models mostly for PR and onboarding. It seems like the way it's going if OpenAI is releasing it. That's-

Are you excited about that? Do you feel like it's, like, a-

I think-

... psyops again?

... the OpenAI model will be good. I expect it to be.

No, they, they're pretty serious. It's not a psyop.

It'll be be- it'll be best in class for some sized category and some subset of tasks. That's like... OpenAI only does things-

Yeah. Yeah. Yeah

... like that. You have to give them the respect they deserve.

But they won't release it on massive subset.

Yeah. They will, they're like... That is a big, like, like open wins when more people are doing it, so, like, that's a, that's a win. And-

Yeah. Well, I mean, hopefully they are actually open about the techniques and not just the weights.

Do we think the size of the open model is... tells us anything about the hardware that they're gonna build?

No.

What? No.

No, they're so secretive about this. That's, like, that's why they haven't released GVT 3.5 or anything 'cause it's too revealing about internal stuff or plans.

Oh, okay. No, so I, I, I... You're talking about Stargate or what do you- what, what kind of hardware?

No, the Johnny AI thing.

The, the... No. Yeah, I think that's, that's a different-

That can run on device

... form factor. Yeah, that's it. Yeah, yeah.

I think that thing will run on the cloud. I don't think that'll run local anyways.

Uh, well, okay, we have to talk about it. Like it, it seems like every podcast we talk about it. Uh, so apparently the news from today, which I think you were looking at, uh- ... was that it was, like, a ear device that they sued, they, they got sued over or whatever.

But, like, I think the ear form factor is pretty good. Like, I actually did get there with B in terms of, like, where, where does this ultimately go? Like you want something... You want the AI to hear what you hear, and where do you hear what you hear?

On the ear. Like, that's pretty much it. I don't know if you guys have, like, thoughts on wearables and where that goes.

I tried the B. I, I think it just knows too much.

And-

That's my-

Yeah

... that's really my-

But you want to give it context

... that's really my thing.

Yeah, I have false privacy hopes. I think, uh, like a lot of people, I mean, that's the whole thing is, like, people don't actually care about privacy.

It's just note-taking, you know. It's just really good memory.

I think the Meta Ray-Ban form factor is good. I don't think it's as mass market. It's like if you get it in a AirPod size form factor, it's a way bigger market for obvious reasons.

Yes.

But the, the, like, sunglasses form factor is a thing that works, I think.

Okay.

I don't use them for AI, but they can fit the AI to work it.

Like yeah, empirically, yeah, it, it obviously works.

Yeah.

Cool. The, well, the last question I was saving up was this whole, uh, what, what is Meta doing? Uh, you know. You had a... You actually had a pretty interesting post back in... When was this? In April. Uh, you said, "Llama 4.

Did Meta just push the panic button?" I feel like back then it didn't actually push the panic button, but now they really pushed the panic button.

That's fair. I think the panic button at the time was the whole LMSYS model not being the model that they released thing, along with a bunch of weirdities about, like, the day of the week they released. But, uh, to be a model that claims to be open and then not release the model that is your leading claim is just like a...

That, that is, like, bad execution. Bad execution. Yeah, yeah, which is fine. And then the recent stuff I think mostly can be boiled down to talent is cheaper than GPUs by a dramatic margin, and at the end of the day, it's like, okay, like, if we're spending this much, it's ac- They go to the room, and they stare in the mirror, and they're like, "Wait, it might not actually be that ridiculous to spend this money on the top people-

That's right

... and it's like, might as well try."

They already spend it on VR.

Somebody was bound to do this eventually.

Yeah.

And it makes sense that it's like the m- um, it's like if Apple were to some way, somehow decide like, "We're gonna do this", they're gonna come in and do exactly what Meta is doing.

They need a founder mode CEO who's like, "Screw it," like, you know-

Yeah

... "We'll, we'll take the L." The, the thought that oc- did occur to me is, uh, is, um, you know, Meta, instead of spending on VR, they should spend on RLVR and, uh ... Well, I- I think my question is like, I think a lot of...

Some researchers, like most people will take the payday and happily move to Meta.

Ev- everybody has a bribe number.

Right.

It's just, just the number really big.

Yeah. But like I think some researchers are uncomfortable with the idea that this is a, sort of the great man theory of, of research, that like you have to pay this much to get this level of talents and-

The talent is definitely distributed.

Yeah, right.

A lot of the people that they would be paying this much have the confidence to redo things or to just do some of the same things and just like, whether you call it feeling the AGI or just drive to build things or like the feeling the AGI is not that different than a lot of things that have existed in Silicon Valley more in the past.

It's just people with the vision that are willing to execute on it, and they see something coming, and like those people make a big difference.

Yeah.

I think you have those people and you remove bureaucracy, getting technical talented researchers is actually something that Meta has a lot of or has the ability to get a lot of. So it's like the, it's a lot of recycling, which is very hard on individuals and morale of an organization.

But they, like, understand the approach.

Yeah, for sure. Um, cool. That's all I had.

Um-

Any parting thoughts on how you're gonna build the American DeepSeek? That was a nice tweet.

Yeah, mostly if I have to look at like what my... And the... If you were asking me like what my 10-year goal is, and it's like I only will have like a two to five year goal, where I think as models are shifting more towards agents, I think that like scaling is slowing.

It's like there's side of a fixed cost and a fixed path to getting towards something like American DeepSeek. Or mostly just I would say it doesn't have to be American if it's fully open, if you have everything and you can modify it, which is like there's a few things that need to fall.

A lot of it is just more resources, but it's like, like Olmo 32B is if you squint like original GPT-4 level and fully open. And it's like there's a few levels that you need to go through, like that's obviously a dense model, it needs to be taken to sparse MOE, and you need to scale it.

You need to have a lot more GPUs, and then you need to do like large scale reasoning. It's like that's the goal that I want to do. There's a lot... Like, that's what I want to do.

Yeah.

There's a lot of complexity in navigating like, like how to work with AI. Like, what does AI2 do to get there? It's very hard.

Mm-hmm.

I think that, I mean, it's a nonprofit, it's hard to get the resources, and building a model is a lot of aligning a lot of different people. That's the DeepSeek story is they have great people. OpenAI has kept a lot of really good people for a long time.

Anthropic has gotten a lot of good people right now, and it's like, it's a lot of incremental hard technical problems that you need to stack up. And like that's what I would like to do and make work in the next couple of years, but it's not easy to get there.

Yeah.

So that, that's the pitch is like AI2's best case scenario is AI2's gonna do other things. Like, you can't just run a nonprofit or a company that says, "Our goal is in three years to have an American DeepSeek."

Like, no one's gonna keep paying the bills on that 'cause you have to tell a better story. But that's like what I would like to do in that, and I'm sure AI2 will do many more interesting things along the way.

Like product stuff?

I don't think it's necessarily product, but like what are more... Like, what are cutting edge things in AI that we can make a new architecture for certain things.

Okay.

Or like what are demos of open models working better, whether you have like private data or something, or just far out ideas that could take you off the transformer trajectory. I think that like you, you still need to be doing these to kind of lead in AI.

Thank you for working so hard on truly open source AI.

Yeah. It's fun. I think it's... I mean, it makes it easy to align like values with what you're doing.

Yeah.

It's like, like it'd be better for the world if more things are open, and therefore it's like a lot of it is just willing it into existence. And I take seeing like what OpenAI does as or is saying they're gonna do as like hopefully a win coming soon.

Yeah.

Like DeepSeek was the most unexpected win that made some other dominoes fall. But like I, I think that is the path forward and see what it takes.

Thank you so much.

Thanks for coming on.