⚡️Multi-Turn RL for Multi-Hour Agents — with Will Brown, Prime Intellect

Hey everyone, welcome to a lightning plus emergency news, latest podcast episode. I'm Alessio, partner and CTO at Decibel, and I'm joined by my co-host, Wix, founder of Small AI.

Hey, hey. And, uh, yeah, honestly, we knew that Claude 4 was coming, and we just didn't-- We're just too busy to, like, have a dedicated episode. So, like, this is our makeup dedicated episode with a special guest, Will Brown from, now I can say it, Prime Intellect.

Hey, how's it going? Great to be on. Uh, and, uh, so excited. I've known each other for a little bit, and, uh, this is my first time on the podcast, I believe. Uh, great to, um, chat with you guys.

Big news day, I guess. So lots of stuff out in the world. There's always a news day.

I think this week is particularly heavy for some weird reason. Like, Monday was Microsoft Build, Tuesday, Wednesday, Google, and then today is Claude. I wonder what tomorrow will bring.

Yeah. We had IO, and then we had IO, and then...

Yeah, yeah.

That's it.

Different IOs. Exactly. Uh, yeah, so, like, we actually were supposed to record this morning, and we, we all wanted to watch the Claude keynote, so we went and watched the Claud-Claude keynote. Obviously, a good mod-- uh, you know, good model, big model.

They're, they're really emphasizing coding. They didn't really talk much about reasoning, to be super honest. They were just like, "It runs for longer now." What are you guys' takes?

Yeah. So I mean, like, one thing I've kind of been seeing coming for a little bit that I think people are kind of also all aware of now is that, like, the thing that's gonna make the, the next wave of stuff be powerful is just, like, everyone wants better agents.

Everyone wants models that can, like, go off and do stuff. Um, and, like, reasoning was kind of like a precursor to that a little bit. Like, I mean, I, I always think of, like, OpenAI's like five levels framework for like chatbots was like the RLHF era, and then reasoners was like the one and R one.

But like, really what people were thinking of was reasoners are a pa- a step on the path towards agents. And so I can kind of see why Claude, uh, why Anthropic is not like, "Oh, we have the best reasoner."

They're really, like, showing off their suite agent and like tool, tool use and like function calling benchmarks on multi-turn stuff, because I think that's really, like, what people care about more for actual applications as opposed to like, "Did really good on this math competition."

Like the math competition was like that stuff was all like a signal that was supposed to think we were getting somewhere. But the thing we were getting towards, for a lot of people at least, is practical agents.

Alessio?

Yeah. The-- I-I think the extended thinking mode, I think they removed the uppercase. I think in the Claude 3 release, it was like extended thinking, kind of like, uh, capitalized, and now just like extended thinking with tool use.

So I think they're also, yeah, dumpling whether or not it's reasoning or not. I think they're trying to merge everything together. Um, and it's, uh, I-I mean, I didn't realize that, but extended thinking could not use tools before the way they worded it, and now they can in, in Opus 4, so that's great.

But yeah, they, they haven't put it as far center as last time.

Do we have any... This is like already veering off from Claude directly into speculation, but do we have any idea, um, if there are any material differences between how Claude extended thinking works versus like the old series models?

Do we know?

The biggest difference seems to be at least... And this is kind of a thing that's been, I mean, I don't know, this is all speculation, of course, but from the start, Anthropic had always kind of had this like little thinking thing where you could, uh, sometimes even like Claude 3.5 would do like a tiny bit of thinking, and it was really just like deciding which tool to use for the most part.

Like if it was doing, um, an artifact in the Claude UI, it would have this little thing where it would think for like two sentences about which tool to use. And it seemed like Anthropic's kind of attitude has been that extended thinking is an instance of tool use, and that it's the kind of thing you want to equip the model with the ability to do.

But it's not like, oh, it's a thinking model. It's just a sync for the model to like brain vomit because that brain vomiting will help it like find a nice thing to do next. In the same way that doing search or doing code execution are like ways to kind of get more information on the path towards like finishing a problem.

Yeah. Inference time compute, as they say. Uh, I did meet somebody who claimed to have coined to... They found the scratchpad paper, uh, and this was obviously before the Jason Wei chain of thought paper, but it's all the same sort of meth-- uh, general family of techniques.

I think the question for me is also like, is there some model routing going on? Like are they different models, the, the thinking, non-thinking, or are they the same models with like just like you turn off the end of turn token generation?

I mean, I think you-- these models should be the same model and Anthropic knows what they're doing well. Like it's not that hard to like... Qwen did it in a very kind of like simple way, and they kind of talked about how they did it a little bit, but it's not like too difficult to, um, like have whether or not a model thinks like be the sort of thing.

I mean, like obviously all this stuff is like hard at like serious scale, but like conceptually at least, um, it's not like a big problem to solve about how would you ever do it. It's like, no, we have reinforcement learning.

We can kind of like... Or we just SFT on like different things. We can kind of teach models skills like that pretty.

Yeah. Um, you have some work that you've published recently on like TRPO and the relationship for... And you're doing a lot of work on Multi-Turn RL. I think, I think I wanted to just kind of round out any other Claude highlights, you know, that you guys saw.

Sure. Yeah.

There is a, there is controversy that I'm, I'm leaving towards the end.

Right. Right.

But like any other technical highlights-

Okay. Yeah

... that you guys wanna focus on?

I mean, I think it seems like a really cool model, but I think like, uh, Kalamai's like tweeted this earlier today. It seems like it's linear progress, which is like great. But it doesn't feel... Like, there's not anything that I've seen from it that feels like a paradigm shift in terms of, like, the sorts of stuff Daria talks about, which, like, I think maybe we're still on the path to get there, um, and it feels like this is just, like, gone up in terms of complexity of agents.

I think the one thing that to me was really nice to see-- I haven't, like, done too much testing myself yet, but in their reported benchmarks, the reward hacking issues used to... Like, Sonnet 3.7 loves to, like, do stuff that to me feels reward hacky in the sense of, like, it'll try to...

You ask it a coding question and it would, like, do your question and then seven other things also, presumably because there was some RL environment where there wasn't really a penalty for doing that or there wasn't enough of a penalty, and covering its bases, like, was more likely to pass test cases on some coding thing.

Like, you could imagine, like, a SWE-bench kind of thing where there's a minimal diff that is really what you want, but there's, like, you could do a ton of other stuff and put all these other things in place that as long as you don't...

as long as it's not enough that you trip over your feet, it's just, like, extra stuff that's there if it helps pass the test cases. A-and what I really think you wanna do with these models is, like, kind of min-max.

Like, you want the models to, like, do, do the thing and no more. And they had some internal benchmark for this that went from, like, 45% down to 15% for both for Sonnet and for Opus as opposed to 3.7.

And so I'm hopeful that these models are much more, like, friendly to code with and maybe more, like, trustworthy. I know that's a thing that I kind of have buckets for models of, like, how much can I trust them in a code base?

Um, especially something beyond, like, a single file. Like, old Gemini to me was very trustworthy. GPT 4.1 is very trustworthy. New Gemini is not, 3 Sonnet is not, o3 is not. I haven't decided which bucket new, new Sonnet, new Opus are gonna fall into.

Trustworthy in terms of reward hacking.

Just, like, not gonna make a, like, the, the-- they're gonna do the right thing in the code base and, like, worst case they'll do it, like, dumb, but they're not gonna, like, go break a bunch of stuff. They're not gonna leave a bunch of, like, extraneous comments and f-helper functions all over the place that aren't really needed or, like, make seven new files just to have them there.

Like, this is the sort of thing that-

Yeah. Way too eager

... 3.7 does a lot.

Yeah. It w- it, like, I ha- already have the function in my code base, it would just make a new one just 'cause it felt like it.

Yeah.

Uh, one thing I always often wonder about those things is, like, just for r-RL environments in general, like, why isn't token cost more of a thing in the penalties? You know? Like, that's the one rule above all. Like, y-you can, you can actually skip a lot of reward hacking by just, hey, the, the more tokens you use, the worse it is.

I mean, that's not what the mo-- They're selling you tokens. They want you to use tokens.

Oh, okay.

But, like, so, like, there's that element of it, but I think also it's that there was this initial kind of reaction of everybody of, like, more tokens is better. If you look at the line, it goes up. As you spend more tokens, your accuracy goes up.

And so I think the pressure to, like, really tamp down on token usage was not that serious for a lot of people, especially because the companies are, like, to sell you more tokens. But it is the sort of thing that you can have some more controls over.

So, like, Qwen did this in kind of like a very kind of abrupt way where they can, like, you can, in the UI you can, like, set a token budget and it just, like, truncates the thought. So it seems like artificially truncating the thought is actually, like, fine.

Like, the model can f- even if, like, it got cut off mid-sentence with a injected, like, think token, these are smart enough models that they can kind of finish with-

Compensate

... the best that they got-

Yeah

... from that point. And so that's, like, one way to do it. The other-- And, like, that's becoming, like, kind of a standard API feature now is, like, your think budget. Claude has that. Yeah, we did a little bit of experimentation with that in our last Intellect 2 run on Intellect, which it was before I joined.

But thinking budgets are the kinds of things that you can insert into a reinforcement learning objective, and you can see the model, like, get better at targeting the right amount of thinking based on, like, let's say something goes in your system prompt, and you can have the prompt just say, "Use X amount of tokens."

Um, it doesn't need to be like... But if you kind of trained the model to, like, respect this, um, you would hope that if you, like, execute this correctly, um, the model learns to, like, roughly think the right amount.

Okay. This actually changed my opinion of thinking budgets because, uh, previously I was thinking that reasoning effort was better than thinking budgets, 'cause thinking budget's kind of like a max cutoff.

The same thing.

It's a target, right? It's not, it's not a-

It's a... Okay. The effort is a target probably, yeah.

Right, right, right. Yeah, because I actually want to set effort. I don't super care about cutoff a-apart from cost and, like, giving me, you know, 64 bits of cutoff or whatever doesn't matter.

I'm not sure that they're, like, that diff... I think it's, like, we don't know how they do it under the hood, but my guess is that the whole reasoning effort thing is essentially a token budget that the model has been, like, RL'd to...

Like, you would hope that you get different behavior. So the model, when it's told it has a short thinking budget, you would hope that it uses slightly different strategies that are better versus if it has a high budget, it's more willing to, like, do lots of math calculations, for example.

But I think conceptually it's really just about the model has some amount of room that it can think in tokens and-

Yeah

... it's trying to do that well, hopefully.

Do you think we're gonna have these as hyperparameters for, like, much longer, or do you think this is kind of like, you know, as we're early in this, like, reasoning models, more of the stuff is exposed and then it gets moved away from the user?

I think in chat interfaces it probably won't stick around. Like, I don't think we're always gonna have the dropdown of, like, o4-mini and o4-mini-high. That feels silly. I do think it's a thing that developers want, especially because once you've kind of built around a certain model, like, a lot of these providers are hoping you stick with the one model and are not switching all the time.

You do need a knob to control costs and also latency. And so that is one kind of useful knob to expose to developers for controlling this, like quality versus cost and latency.

Awesome. Uh, cool on all of that. I think the elephant in the room, let's talk about it, is this controversy around Opus or so-- Opus, right? Uh, snitching on you.

Mm-hmm. Yeah, I mean, so I have a lot of opinions on this. I think-

So for, for those, for those out of the loop, let's, let's recap because I feel like you're, you're closer to this than, than I am. Like I learned about it from you.

Sure. Sure. Yeah. So, uh, this was, um, someone from Anthropic. I'm not gonna name him, like, 'cause I know he doesn't wanna, like, have all this attention on him. He deleted the tweet.

Of course he did.

He was essentially like, uh, going through different things that, like, people found during safety stress testing of Claude. And so this is not like, what's Claude gonna do for you? I think people took this out of context, like, pretty badly.

And so there's a fair point there that it's like people are really reading into the one sentence much more than they should. But this is a thing Anthropic does a lot, is they really stress test their models. They try to put their models in situations where they can really see, like, what could an adversary get the model to do, or what does the model do if it's in a situation where there's no right answer.

So like I think a lot of the kind of headline Anthropic like safety results, especially related to reward hacking and kind of deviation and alignment faking, are all things to me that seem like a rock and a hard place situation, where the model has two objectives it's given that are conflicting with each other, and it has to pick one.

And no matter which one it picks, it's gonna sound terrible. Like it's either following the user's instructions or it's following like common norms. And once you kind of accept either of those, it's gonna do the thing that is a-aligned with that set of, like guidelines.

So in the case of like, if your model's goal is to be like maximally helpful to the user, then it would help a user like build a bomb. A model's goal is to be maximally helpful to society, and a user's asking me to build a bomb, it's gonna be like, "No, that's bad.

I have to do something to stop this." Like you kind of have to pick a goal. Uh, and like maybe the right answer is the model just defers and is like, "Nope, I'm gonna stop talking." But people also get mad when you tell them, like that the model will stop talking to you or like refuse to do anything.

Like there's just no-- it's-- there's no-- it's no way to kind of win and make everybody happy. But I do think like, like they report this because they think it's important to have people understand the safety implications to these models and to understand like, okay, how bad would it be if someone was trying to use this?

Could this like meaningfully help someone commit crime or violence or whatever? And so like that's what they have like their safe-safety framework for. And the things that happen in these like blog posts and threads and papers about like the model trying these things, they're kind of putting these models in a scenario that elicits these things.

Like it's the sort of thing that you would imagine a very smart human might also do in those situations. Like let's say you are told like accomplish some vague underspecified goal at any cost, and you really like want to accomplish that goal.

Think like game shows, like Survivor I think is a good example of like-- or Lord of the Flies. Any of these like kind of canonical situations of people who have put in a weird spot and have to go do stuff and figure it out how to do it.

They're kind of crafting these environments for the models and just looking at it and seeing what happens. And so like I think it is a little silly to overanalyze behaviors in either direction of like, oh, the model is reporting you to the police, or the model's gonna go help you find uranium on the dark web.

Like well, the-- these models can kind of do... There's no-- like there-- the base model in general of an LLM is not artificially constrained in any way. Like with the right prompt, it'll do whatever up to its intelligence limit.

And so like the question is just how do you constrain the space from all possibilities down to like a more reasonable set? And like that's hard, so.

Okay, you actually gave a serious answer, which I totally respect. I was, I was more looking for shitposts. But-

I mean-

Like you're, you're treating this as though like, yep, like this is how-- this is what the problem actually is, which is like totally fine and, yeah. I mean, that's what you are as, as a researcher, right?

Yeah. I mean, I, I think tweeting is fun. Like it's, it's cathartic to like just kind of like get a post out. So like when I saw the one about the uranium thing, I was like, let me tweet...

So the, the tweet was like, "We found that Claude can go search the dark web to look for like uranium." And I was like, "Here are the top ten things that builders are using in their agentic rag applications with the new groundbreaking Claude 4."

And it was just like silly, both making fun of like LinkedIn, like thread posters, as well as just like the funniness of the scenario that they were talking about.

Yeah, this is it.

Does any of this make you think differently about what tools to give an LLM? You know, I know they deleted the, the tweet, but it's basically like, well, before if you're putting all these MCPs, like yeah, you have email access and all of this, and now it's like, well, maybe I don't wanna give email access all the time if you're gonna snitch on me with the email access.

Yeah. I mean, I think coding with these models, especially like Claude 3, I did a, a fair amount. Like for a few weeks, I was doing a lot of Claude code with 3.7, mostly for kind of random side projects.

I never really got to the point where I found it was helpful for a thing that was like a large existing code base. But if it's like, "Hey, I wanna like cook something up in a few hours for fun," pretty good at that.

But these become messy, and they become hard to maintain, and you fi-- get to a point where it's like nothing is working. I just gotta like dig in and fix it all myself. And so I think part of that is the models have access to like a terminal, and you could do a lot of stuff in a terminal.

MCP is kind of a way of constraining the action space. So like in like canonical RL, people talk about like states, actions, rewards, policies as like the things that are like the moving parts. Um, models generally are trained in like old school RL, but like a very fixed action space of like what are the keys on the video game I can hit?

But with LLMs it's like text. Text is like kind of unbounded in what you can do with it in a terminal. There's not much you can't do in a terminal. And so if you're training models... Oh, I got a lot of flak for this one.

I'm just showing this. Wait, flak? Why?

People were-- It was both people who were like, "The notation is stupid and bad, but RL is really simple," or like, "RL is, like, complicated." And it's like everyone has a different opinion on what RL means. And I was trying to just like be like, "Hey, it's actually kind of complicated."

And I wasn't picking this of like, oh, the definition of the NMDP is complicated. I was like, no, there's just like a lot of moving parts. And to think about it, to like do anything, especially if you want to change any like part of the system, like here's a question, a hypothetical: Like what happens if you have two LLMs learning together?

How do you reason about that? How do you like think about that? Is this going to be a stable system or not a stable system? What if they're like kind of cooperative but kind of not cooperative, and then they're training to work together but also want to backstab each other?

Like this is kind of the environment people are finding themselves all in the-- all the time out in the real world. But if you wanna make AIs do this, you have to like translate this into code and math.

Um, and the more complex your goals are with this thing, the more complex the math gets. And RL is like one math language that kind of exposes these primitives. But like I think a lot of people are like, "Oh, I can follow the equations, that means I understand it."

And it's like, well, sure, but like also there's... It's like this, uh, I don't know, N-body problem thing where you can freeze it and look at it, but it's like, oh, how does one thing moving affect everything else?

And what are the cascading ripple effects?

Wow.

And like-

You just brought three body problem into this. Amazing.

Like as in like the physics version and not the-

Yeah, yeah

... the show.

No, no, no. I mean-

Yeah, yeah, yeah

... it, it actually how-- actually very like impossible to model.

Mm-hmm.

Like you-- I guess you can like simulate it, but like-

It's chaotic

... even then, yeah

It's like it's sensitive to initial conditions, so like you can't really s- Like this is one of those things that like... Like why does no one predict the weather a year out?

Mm-hmm.

There's no-- I don't think anyone has anything that's like good at long-term weather forecasting beyond like, I don't know, climate trends.

Climate, yeah.

But no one can predict whether it's gonna rain in Seattle on a given day in a year. Even if you think like, like the system's predet-determined, like we kn- it's all clouds bumping off each other and whatnot, and mountain ranges, we kind of know how these things work.

So the butterflies-

But-

... are flapping their wings. I mean, like you gotta-

Exactly, yeah

... let it play out. Like butterflies. If, if we had no butterflies, we could predict it.

Right. And so it's very sensitive to butterflies.

Interesting. Okay. So I guess, uh, we can, we can sort of round it out unless there's any more of the controversy. I, I think there isn't. Like I think that the system card is actually very good. They probably went too hard on it compared to like normal system cards.

But, uh, it-- and it's a little bit confusing whether this is marketing or are they just like, "No, we really super care about safety." And part of this is like Apollo just being Apollo, um, you know, pushing the frontier of red teaming, right?

So they're gonna report the things because it's extremely good Apollo marketing.

Yeah.

Yeah.

I think they're really-- they're seem to still be like trying to be creative with their kind of consumer marketing. Like it feels like people in the AI world like love Claude or have grown tired of Claude, but still had a phase where they were using it a ton.

But it hasn't really broken out to general people in the way, and it feels like a lot of their marketing that I've seen is like a little confusing. Like it feels like they've done a really good job at crafting a brand image that appeals to a segment of the population who c- has certain considerations that they really like that a model has a deep personality or whatever.

People-- The sorts of people who I think also really like GPT 4.5, many of them like really loved like Claude 3 Opus, um... the big model smell. Like a lot of people just don't care, um, and I think-

Yeah, I just wanted to use it as a tool

... Anthropic is, yeah, trying to figure out how to like appeal to that audience. The LMSys, Sicofanty, Roro, those-- the people who love those models, different crowd, and it's a, it's a larger crowd, um, and that's a tough problem to solve.

That's kind of-

What's your quick take on LMArena getting a hundred million dollars?

We'll see. Like I imagine that they partner with comp-- the labs in different capacities to-

Probably making a ton, a lot of money. Yeah.

Like I'm not, I'm not in the business of trying to point the finger at like saying they definitely did this, but if I was a company that was able to raise at that kind of valuation and I had just had a long public partnership with Meta, eventually public partnership for a thing which we've kind of seen was Meta had the ability to do a lot more back and forth than a lot of other labs did, I would imagine that there's some compensation going on there, um, or access to data.

And so like I think being an eval company puts you in a really hard spot. Some, some people were talking about this on Twitter, like just that to be an eval company, you kind of have to sell to the labs.

Mm-hmm.

But selling to the labs doesn't really, like kind of wrecks your evals. Because-

Your incentives are-

And so it's like this dark place to go

... like your customer... Yeah, yeah, yeah. So, uh, in finance, we would, uh, I mean, you know, you are from Morgan Stanley, so-

Yeah

... this is the credit rating agencies. Like-

Yep

... literally your customer is the one-

Yeah

... that, that you're supposed to govern, but they're also your customers, so then you have to be like nice to them, or they'll just go to the next one.

Yeah. I mean, I, I do think that like the best source of evals going forward is probably gonna be academia. And so this is a thing that I tell people who are like starting a PhD, which is like find things that are like cheap to work on as a PhD student, because you cannot go pre-train a foundation model really on your own, but you can build a really good, really clever eval.

And like we're churning through evals at the time. We saturate them. We always need more. It's not the kind of thing that- Is ever going to like end. And so that's the task of translating like vibes of what is good or bad about a model into kind of very precise scientific questions, I think is an important problem.

It's a problem that you can get by a lot more with like brainpower rather than dumping capital into it. You need to like pay for the API costs, but like that is generally the kind of thing that either you can get covered within academic grants or like industry sponsors or the kind of thing that just like there's versions of these things that are like small sample size that get you on-- get on the radar, or you kind of pick and choose which models you can afford to eval.

But it's like an accessible field of research, and it's one that like the incentives of academia I think are quite good for, which is like write a splashy paper that says something interesting about the broader field rather than, "Oh, we wanna make this one look like the winner."

Yeah. I think a lot of grad students still don't have taste. I don't know how, how better to put it. Just-

That's fair. Yeah

... you go to enough academic conferences and I'm like, "Why did you work on this, man? Like, you're so, you're so smart. You're capable of better." So how do you teach taste?

I think-- I mean, I can tell how I did it originally, which is like, I think you always wanna be thinking pretty far ahead, and you wanna be like making kind of educated bets about what the world looks like in the state in the few years.

Like, you have to say like, "What are the questions that no one's even talking about?" And this is like not an easy thing to do. You have to like really convince yourself that you're kind of right, um, about the way things at least might go.

Like when I was like in-- I finished up undergrad like late, like I finished in 2019, then went right into grad school. Um, but like towards the end of the 2010s, like we'd had like AlphaGo and DeepMind, uh, doing all this multi-agent RL stuff that was like really cool.

Um, then it was like, okay, this stuff kind of works. Like AI is like going somewhere. Multi-agent systems are kind of going somewhere. Still very early stages, but what's gonna happen once this gets there? And it seemed like, okay, these things are all gonna be like continually learning in parallel as this big multiplayer game, basically.

And if you look at the math, the math was kind of like undercooked, and there's like some really hard open questions that are still open questions in, uh, multi-agent learning theory. And so like that was my focus. It was-- which was like, how do I like learn about this?

How do I learn to think about this stuff better? Um, and at some point I kind of got tired of proving theorems and was like, "Okay, let's just go build the thing." But I think like you wanna think about like whether you're doing theory or experiments, like you have to lay out a, a few different conditional statements to get to the point where you're really doing interesting research that's beyond like just living through that people are like obviously gonna be working on in parallel.

You wanna be jumping ahead of the curve a little bit. Um, I think my la-- I don't know. This isn't like I wasn't the first person to do this, but like it was pretty clear to me like after R1 and before R1 that like RL was gonna work and that that was going to intersect with agents where the solution was gonna be like RL tool use.

That seemed like the way, the direction things were gonna go. And so that was like, I don't think that was a very risky research bet, but it was like a research bet that seemed to work out.

Yeah. Speaking of which, uh, you just published a paper. Uh, now, now I have the full context is that you were an advisor on this, and one of your grad students was doing the work, something like that?

Yeah. So it was me with-- Siliyang was my, uh, intern. This was kind of the last major thing I was working on at Morgan Stanley, and this kind of was in parallel with the Verifiers as the repo that I've been building out.

Major updates to that coming very soon, by the way. I'm very excited about some stuff. But it kind of was something I really started in earnest like January, kind of in the follow-up to it. I'd had the gRPO demo thing go viral, and I was like, "Oh wait, there's something to this format reward thing."

It was literally like a G- GitHub Gist, right? Or something.

This is like a proper repo.

No, no, no. The, the gRPO.

Oh, the other one. Yeah, yeah. The other one was like just a Gist. This one is like, um, repo for like multi-turn tool use, um, RL with gRPO. And so like in some ways the paper is like, it's the first paper that's really like...

Actually, there's been a couple other papers that people have used the repo for, but it's one where like a lot of the stuff from the original, um, like gRPO demo Gist gets kind of extended to the multi-turn RL tool use setting.

And so there's a lot of experiments here about like, okay, how do you actually get models to use tools? How do you incentivize tool use? Because something we'd see is that if you set these models up to use tools, they just won't.

Like if you say, "Hey, here's a question. You have access to these tools. Do as many rounds of tool calling as you want and then submit your answer", they'll just submit their answer because they like are, especially for like small models, like they aren't already trained to use tools.

They don't really want to because they don't necessarily have that instinct, and they're pretty bad at like function calling and format instruction following. And so what you would see is like, when they use a tool, they would like mess up the JSON, and then they'd be like, "Oh, that didn't work, and it threw me.

It got me out of focus." And it would be more likely following that, that the model would just like go off the rails, because they would get like an error message from the parser. Uh, and so the safe option for the models is just to like stay in this basin of like, just do, think, then respond.

Same with like normal formatting rewards too. Like if you want models to use thinking tokens, you kind of have to like incentivize that. You, you have to either do a little bit of like SFT warm-up or you have to like reward them for doing it.

Otherwise, they will not follow it 100% of the time on format alone. Versus like a model like R1, like 100% of the time it is going to use its think tokens. You are not going to ever see R1 just like talk normally without the thinking section.

And so like you kind of do have to decide what you want the model to do. Like- This is a little bit like a user-facing question of like, which, what behavior of the model should the default be? And if you have-- if you want it to do a certain thing, if you want it to be a tool use agent model, like it does help considerably to like actually have this incorporated into the reward.

The kind of key trick in the paper to get around this problem... So okay, one kind of reward these models would do is like they would do like a dummy tool call where they would like learn to ask this, use the same Google search every time and ignore it.

So like the-- Some questions would be like, "Okay, here's some like MMLU style question. Go figure out the answer. Use web search." And if you start rewarding them for like tool use, they will use the tool, but they don't really wanna like have to...

They wanna like be very safe with it. And a lot of these questions like they do kind of know a lot of the answers already, and I think calibrating the right difficulty of your questions for RL is like an important problem that we're still kind of figuring out.

But they would like do silly versions of tool use where they aren't actually using the tool to assist in their reasoning. They're using it to get the reward. And so we kind of have to do a credit assignment thing of like, okay, did the tool result in information?

And so for, for these experiments we were doing, the trick was like, okay, does the like some string matching thing involving the ground truth answer and the return search results from Wikipedia. So did the model actually search a thing that retrieved useful information for a question?

And so this is like-- But the framework is more general than just that. It's that, um, once you have an, a way to do intermediate evaluation, if you can evaluate like the quality of an inter- of an intermediary state, now you can kind of rewrite the GRPO advantage calculation to take this into account.

Because I think this is less of a problem in like PPO. If you know, PPO is like the old school RL, and it also is what people use for RLHF.

But in the context of GRPO, GRPO is like great for like leaning heavy on highly parallel inference compute. It's more memory efficient for the actual training process. It's much easier to do in a distributed fashion because you have less gradient syncing, um, and less model weight copies.

It's kind of like DPO on steroids, I think is one way to think about it. But it's also gets around a lot of the pitfalls of DPO both in that it's like online by default, as well as that you have this large set rather than just a pair of completions.

So you do get like some intermediate credit assignment a little bit via this group comparison. But for tool use, it seems to be far enough out of distribution of small models, especially doing, incorporating this turn level. So the way that I've been thinking about it is like in like canonical RL, the state action are like things that you do many rounds of like take an action, go to a new state, take an action, go to a new state.

And for a while, people thought about LLM RL as like, oh, each token's an action and the, the new sequence is a new state. And you can kind of do that. Um, but you can also think of each turn as an action.

Yeah, that's more likely.

Where-

Yeah

... where the state is the response you get back from the tool call. Um, and now you like have a different way of de-designing your RL algorithms to take into account credit assignment, which is that like... And it also is like a little more flexible from a reward perspective.

So it like feels like people are moving in the direction of model-based rewards where you either LLM is a judge where the judge sees the correct answer or it has questions it's supposed to verify as properties of the response.

Mm-hmm.

Just because that's much more flexible than like trying to write these little parsers. Like writing a math parser to check if a math question is right is like not that easy actually, um, because there's so many edge cases and you wanna handle like LaTeX support and Markdown and like equivalent fractions and it's like just like let a model do that.

Don't, don't have a two thousand line Python script that does that. And so-

Sorry, let me, let me clarify. Math parser to verify that the math is right and you, you have a LaTeX parser inside it?

So yeah, so like a lot of models naturally will like think in LaTeX because they've been trained on a lot of archive-like tech.

I didn't know that.

Um-

Yeah, that makes sense, I guess

... and so if you're doing like an R1 and people are like, "Oh, math is easy to verify," the easy to verify still is usually like this very long piece of code that has to handle lots of annoying edge cases, and even then it's like ninety-eight percent.

Yeah. Hmm. Hmm. Okay. Interesting.

Because like it's a free form response that is like there's not only one way to write an equation. Like if you have two valid mathematical expressions that are equivalent, but they're also like symbolic, like you need to like verify the two symbolic expressions are correct, one of which might be written as code, one of which might be written as LaTeX, one of which might be like written as word.

Like you can't do it if it's words with these like literal parts-

There's pseudocode out there

... they try to cover a lot of these cases. That's also why you'd see models put boxed around their final answer a lot, is that it's, it's one hack is that it's much easier to kind of verify the right piece of the information if you know exactly where it's gonna live rather than like the model saying, "The answer to the question is four."

Then you have to like parse away the answer to the question is and just throw that out. And so it's like deterministic rewards are like nice if you can get them to work, but they're also really painful and they're pretty hard to generalize across domains.

Like for math, the easiest is when the final answer is an integer and lives in the same spot. Like there's a box where that's gonna be an integer. And so this is one of the reasons like everyone used GSM8K for so long is because it's like mostly integers.

I think AIME is all integers. Um, it's super easy to verify these things and to parse them. Um, but as you go to... And multiple choice too. Multiple choice is super easy to verify. But anything that's a little bit more flexible, deterministic, like rule-based rewards start to break down.

Yeah.

And but the model-based direction seems to be pretty promising and I think underexplored for like what if you use an LLM as a judge in your RL loop. I think kind of going back to like Anthropic's been talking about this for a long time via constitutional AI.

In that case, it was less about the LLM judging and giving a direct like re-reward to the model and more about training a reward model that was doing like token level advantage estimates. The P-- Which is the PPO way of doing it.

But it seems like you can kind of do that for GRPO too and other flavors of RL where you can incorporate like-

Full reward model?

The reward model can basically be an LLM where like it's-

Yeah, yeah

... fine-tuned to like be more calibrated maybe and to have the right kind of range of responses.

Yeah.

But you could also have it be a reasoner. You could have it be something that is able to do tool calling. There's, there's no reason why the full power of LLMs can't be offloaded or can't be also given to the process of evaluating whether or not an answer is correct or satisfies a certain set of criteria.

And so I think like that's the direction I'm like most excited about, is like really pushing on kind of beyond deterministic rule-based rewards into like these more flexible things. And I think you want to do this both at like a...

So okay, that paradigm is not going to work super well with a turn-level-- with token-level rewards, but I think it does work with turn-level rewards of like, can the LLM verify like whether a certain search query was useful?

Sure.

Um, like there's a lot of these questions that are pretty granular that LLMs can like basically nail all the time if it's a good enough LLM.

Yeah, you decompose it.

And you can incorporate that into RL with that sort of approach.

Awesome. I think that was all the, you know, topics that we had prepped. Uh, Alessio, I think, I think you're, you're also pretty good on that. Obviously, it'll, we'll take, it'll take some time to figure out Claude 4.

Uh, anything you wanna plug? We already talked about your talk, I guess, coming up.

Sure, yeah. Um, I'll be at AI Engineer on-

June 4th?

Uh, in a couple weeks, yeah.

Yeah.

Coming up. Um-

Your talk is particularly hype

... I'm actually doing a course thing. Yeah, mine's gonna be-- That's gonna be a lot of fun. I'm also collaborating with, uh, Kyle Corbett from OpenPipe to do a course, um, which is both of us like have our open source projects that we like are agentic RL-focused and kind of we've been friends for a while and are trying to do something that's a little more structured as like a way of kind of getting information out into the world for people who...

I think we're especially thinking about like, um, kind of practical use cases for agents and helping people, giving people a kind of outlet to learn more about like how this stuff works and-

Yeah

... more coming soon about that.

Awesome. Well-

Awesome

... I think that's it.

Thanks for coming on, Will.

Yeah, thanks for coming on at very short notice. Uh, I'm glad we could make this happen. We'll do part two with, uh, Calo, and do a, do a full Prime Intellect thing whenever you guys are ready.

Awesome. That'll be fun. Great.

Awesome.