# ⚡️Open Questions in Agentic RL — Will Brown (Prime Intellect) Latent Space · 2025-05-09 Will Brown of Prime Intellect maps the frontier of open-source agentic reinforcement learning, arguing that training models like o3 requires multi-turn tool use, intermediate reward verification, and efficient context management. He outlines challenges such as scaling tool calls to hundreds (e.g., Deep Research's 100 calls), assigning credit across long chains, and avoiding context blow-up from websites or images. Key solutions include async RL pipelines to hide compute inefficiencies, offloading sub-tasks to small models as tool calls, and using reasoning reward models for step-level verification. Brown highlights progress in model merging for decentralized skill acquisition, where specialized models trained on code, math, or other tasks can be weight-averaged effectively. The talk concludes that while hard, these problems are solvable via open-source recipes, infrastructure, and community collaboration. ## Questions this episode answers ### What is "ten-minute AGI" as used to describe O3 models? Will Brown introduces "ten-minute AGI" as a way to think about O3's capabilities, meaning that any task a human can complete in ten minutes, O3 can perform reasonably well. This framing helps define the goal for open-source agentic RL: building models that can autonomously handle short-horizon tasks with tool use. [1:49](https://addtry.com/e172870f-a0e0-4a7f-93a2-a5a596d07f56?t=109000) ### What are the key challenges in scaling up multi-turn RL for agentic models? Will Brown highlights several challenges: handling many tool calls (Deep Research does up to 100) requires intermediate verification and turn-level rewards for credit assignment; verifying actions outside math/code is hard; context length explodes with website reading; and efficient, async infrastructure is needed to manage compute resources without inefficiency bubbles. [3:06](https://addtry.com/e172870f-a0e0-4a7f-93a2-a5a596d07f56?t=186000) ### How can using smaller models as sub-agents help in multi-turn RL training? Will Brown suggests using smaller models as sub-agents to offload tasks like reading websites. Instead of putting full website content in the main model's context, you delegate to sub-agents that answer specific questions, then return condensed responses. This reduces context length, enables parallel reasoning, and lets the main model be trained to prompt these sub-agents effectively, bypassing the complexity of full multi-agent RL. [8:26](https://addtry.com/e172870f-a0e0-4a7f-93a2-a5a596d07f56?t=506000) ### Why is asynchronous reinforcement learning important for training agentic models like O3? Will Brown argues that asynchronous RL is crucial because it allows tool calls, reward modeling, and training to happen in parallel with some lag, preventing compute inefficiencies. By training on slightly stale data (off-policy), you can scale these components without waiting for synchronous updates. Prime Intellect's work shows this approach is robust if hyperparameters are tuned, enabling the scaling needed for agentic models. [13:29](https://addtry.com/e172870f-a0e0-4a7f-93a2-a5a596d07f56?t=809000) ## Key moments - **[0:00] Intro** - [0:20] General-purpose agentic models must handle long-horizon tasks like web browsing, coding, and image analysis. - **[1:07] Multi-turn RL** - [1:50] "10-minute AGI" is anything a human can do in ten minutes, o3 can do reasonably well. - [2:28] Multi-turn RL iteratively trains models to use tools in service of solving problems, rewarding final output. - [3:04] Deep Research models can make 100 tool calls, necessitating turn-level rewards for credit assignment scaling. - **[3:06] Challenges** - **[5:23] Tool scaling** - [6:25] O3 uses programmatic image manipulation tools like crop, zoom, and rotate for efficient multimodal reasoning. - **[7:43] Sub-agents** - [8:26] Sub-agents as tool calls offload reasoning tasks, enabling parallelization without end-to-end multi-agent training. - **[9:45] Context & verification** - [12:06] Reward models as reasoners use inference scaling to break down tasks into subcriteria and verify intermediate steps. - **[13:29] Async RL** - [13:29] Async RL pipelines train on stale data with replay buffers, robustly scaling tool calls and reward modeling. - [14:24] Model merging by averaging weights of code, math, and language models creates capable general-purpose agents. - **[15:44] Roadmap** - [15:58] Roadmap to decentralized N-minute AGI requires solving multi-turn RL, async training, and model merging. - **[16:40] Outro** ## Speakers - **Will Brown** (guest) ## Topics Reinforcement Learning, Agents ## Mentioned Cognition (company), Cohere (company), DeepSeek (company), OpenAI (company), Prime Intellect (company), ChatGBT (product), Command A (product), CrewAI (product), Deep Research (product), InstructGPT (product), KernelBench (product), MCP (product), o1 (product), o3 (product), r1 (product) ## Transcript ### Intro **Will Brown** [0:00] Yeah. Okay. Um, today I'm going to be talking to you all a little bit about, uh, the frontier directions that we're all thinking about in terms of open source agentic RL. Um, and what basically sketching what the roadmap might be towards training something that looks like an o3 model. What does it take to get something that is like a very good general purpose agent for doing long horizon tasks? Um, and we want a thing that can go off and browse the web, and write code, and look at images, and do all this stuff in service of sar-- solving increasingly important problems. Um, and so, like, we know it's doable because OpenAI has done it. Other people have done things like this, but there really isn't currently an open source model that behaves in the way that these models do. We have things like R1, which are great at kind of the single-turn math and code reasoning problems, but they are not in the general purpose agents world yet. Um, and so I think there's a lot of things we need to solve along the way, but I think these problems are things that are solvable and that we also kind of like have a, a sketch of how to do it. ### Multi-turn RL **Will Brown** [1:07] And so I'm gonna, in this talk, be mapping out, like, what are these questions? What are the roadblocks, and what are, like, the plausible, uh, things we have to do that will actually unlock this? And so what we want is models that can go off and do stuff. Um, and for LLMs, doing stuff, it seems like, uh, is best done with some kind of tool. So this is one of the reasons, like, everyone went crazy about MCP, is so it's like a standardized way to, like, give LLMs tools. But that's really what it is. You want models that are really good at doing multi-turn tool use. Um, that's kind of the o3 magic, is lots and lots and lots of multi-turn tool use with multimodal in-input in very general settings beyond just math and code. Um, and whether or not you want to call this, like, N-minute AGI, I kind of like the phrase ten-minute AGI for just, like, how to think about o3, um, is that anything that you can do as a human in ten minutes, o3 is usually gonna be able to do reasonably well. Um, and that, if you can crack that, uh, scaling direction of, like, pushing the boundary of how long these models can go out and do these things for, that is a plausible path towards things that become RLMs. Um, and so o3, you can, like, give it hard problems, and it, like, does a pretty good job after thirteen minutes of thinking at, like, sketching out a pretty good version of the solution. And so the table stakes version of this, um, which is kind of the area that I've been doing some open source work in, is multi-turn RL. So, like, rather than just, like, you train the model to, like, guess the right answer, you train the model with tools in the loop to be able to use these tools in service of solving some problem, and then you give it rewards based on the final, uh, output of this model after the sequence of tool calls. Um, and so we've kind of moved beyond the single turn RL agent world of, like, InstructGPT, ChatGBT sort of things, as well as the single turn RLVR world and models like o1 and R1. And we wanna, like, have these things become more agentic, and it seems like the path is to incorporate reinforce learning into this process. Um, some of the pro-- uh, the challenges are, like, the versions of these models that are quite good could do, like, many, many tool calls. Like, Deep Research can do, like, a hundred tool calls. Um, you probably want some kind of intermediate verification where you're not just waiting for the final answer at the end, but you want something like turn-level rewards potentially, where you want to be able to ensure that the model is getting credit for the t-- the moves it makes along the way, because it's no longer just, like, one thing it's doing. ### Challenges **Will Brown** [3:33] It's doing a hundred things, and the credit assignment problem becomes very tricky as you scale the, the length of these, uh, tasks. Um, you might want to do this thing, these for things outside of math and code, where verifying whether an answer is right or wrong is not necessarily straightforward. Um, you want to avoid, like, blowing up your context length. Like, if you have a model like reading websites, uh, and if you put all the website in context, and the model is, like, doing this in RL, this, like, blows up your context limit really quickly if you have it reading lots and lots of websites. Um, you want infrastructure for managing the resources of these rollouts, uh, in terms of compute environments. If you're having models that are, like, doing data science, you want them to be able to, like, do data science quickly as part of this training process. Um, and you also ideally want to, whether you're doing this centralized or decentralized, move in the direction of, like, everything being async and overlapped. Um, because that is, like, otherwise you have these compute, uh, inefficiency bubbles that, like, pop up all over your, uh, compute structure and your training structure, and you ideally want to be able to, like, hide all of that by having everything going with some lag in parallel. Um, and so that's kind of what, uh, the, the path forward seems to look like. Um, and so this is just a little sketch of some work recently about kind of, like, working towards this, where you kind of can do multi-turn RL by incorporating tool calls into the loop. Um, there's been a big sequence of papers of people doing, like, many applications of, like, coding agents and search agents, and, uh, people are kind of in the public sphere documenting a lot of their learnings of this process. And I think there's a lot of value that is kind of floating around in scattered papers where we just need to, like, understand the lessons learned, uh, build from them, and aggregate this into kind of, uh, training recipes that can become more scalable, uh, by, uh, crowdsourcing the answers to some of these questions. Um, some of the popular agents people are using, the, the number of tool calls scales quite dramatically. Um, Deep Research, as I said, can go up to a hundred. And we also do see that, like, for really hard problems, doing more tool calls, searching the web more, gives you a nice scaling curve where you get better answers by putting in more effort, by, like, spending more time browsing the internet, essentially. ### Tool scaling **Will Brown** [5:44] Um, and so this is a thing that we want to have a process for scaling. Um, we want, uh, technology, we want infrastructure, we want recipes, and experiments to push along these trajectories. Um, multimodal is also one of the big unlocks in o3, um, where I think for a while some people were, like, speculating like, "Oh, what if you have the model, like, generate images in its chain of thought reasoning where everything is like-" Uh, end-to-end multimodal input and output. And it seems like we don't really need to do that because for a lot of the, like, multimodal things that are useful for reasoning, it's actually just more efficient and easier to have models use, like, programmatic tools to manipulate images. So, like, O3 can, like, crop an image and zoom in and rotate. And I think there's a lot of mileage in using this sort of thing, where the ability to work with these images, um, allows you to understand them in different ways by, uh, shifting the way the model has them in its context window to emphasize certain things more. Um, and so I think just, like, leaning into thinking of modes of intelligence as tool calls is a very powerful, like, mental model and, uh, practical trick for having models do the things that you want them to do. Um, and once you have these kind of, uh, things that exist in the tool call format, which I think there are several other examples of, um, then you start being able to have a way to train the model to do the thing. Um, and so, like, GeoGuesser, for example, or, like, like, the model is, like, zooming in to, like, look at the license plate. There's all sorts of these things that humans do, uh, kind of instinctually, for example, by, like, focusing our attention. So what does that look like for an LLM? How do you, like, refocus your attention? The si- the simple way is just crop the image and zoom in. Um, and so that is one piece of the puzzle we probably do want if we're working towards, uh, scaling up these kinds of systems, and we want to, we want to replicate those sorts of models. ### Sub-agents **Will Brown** [7:43] Um, additionally, um, I think people were, like, excited about multi-agent systems for a while, like a CrewAI sort of thing of, like, "Oh, I'm gonna put my coder agent and my finance agent in a group chat." And, like, a lot of these are just kind of silly. They don't actually work very well because these models are not trained to do that. They're not, uh, ever trained in a way that actually reinforces those patterns. Um, multi-agent RL is hard. I did five years of it in grad school. It's, like, not easy. Um, and to, uh... The algorithms don't really even exist for the things you would really want to do. Um, but there are some kind of shortcuts where you do get very focused versions of multi-agent learning, um, that have good synergies with the scaling paradigms that we're currently doing. So some of these are just, like, allowing other models to be tool calls. Um, so let's say you're browsing the internet, and you want a model that can, like, go read a hundred websites. You probably don't want all hundred full websites in your context window of the main model. But what you might want to do is have sub-agents whose only job is to answer a question about a website. Um, and once you have this, now you can do things much more efficiently. You can parallelize aspects of the reasoning chains, and especially for things that can be done with, like, off-the-shelf small models, you don't need to train these end-to-end because they're already, like, decent at, at being an LLM, and there's a lot of passive reasoning that can be offloaded to just, like, any good enough LLM, at which point you now can just train the main model to learn how to prompt these models well, to use them well. Um, this is also something that we might want to do with things like video summarization. You probably don't want that all in context, um, for most cases. Navigating the web, it might be the case that, like, uh, having really, uh, large images every second in your context window, like, blows things up really quickly. Um, but if you can offload this with, like, kind of more condensed, uh, interfaces to using, uh, VLMs as tool calls, then you can have much more, uh, logic exist in your reasoning chain within your context window, and you can then scale to much longer chains of tool calls without needing as many tokens as it takes to have an image of the website appear in your context window at every single step. ### Context & verification **Will Brown** [9:45] Um, and so this is going to be a problem, especially if you're doing this in an open source, centralized way. If you, like, scaling to, like, millions of different contexts is, like, really, really hard. Um, there are-- I don't think there are real, like, open source replications of millions of different context scaling beyond, like, tiny, like, academic model sizes. Um, and if you want to do that type of reasoning, um, you want to kind of use every trick you can. Um, and I think offloading parts of this to sub-models is, like, a very effective one that you can, uh, specialize with RL because once the sub-routine is solved, you kind of freeze that version of the, the helper model as a tool call and incorporate this into your model. Um, another thing on context management is, like, these models benefit from thinking a lot. Um, but the thinking takes a lot of tokens, and so if you're doing multi-step learning, you have all these thinking chains that blow up your context length, and some of these models, like for inference, they are not actually keeping those thinking chains around. They're, like, essentially doing the thinking per step, throwing away the thinking tokens, and then continuing on. And we kind of need new algorithms for doing this. Like, the existing paper people are writing about multi-turn RL are not actually incorporating this, and it kind of, like, breaks all the math where you need to, like... The recent paper, uh, blog post, like, two days ago from, uh, Cognition for their, like, KernelBench, uh, AI does this in kind of, like, a simple form where each step is essentially like a final answer, but they allow it to, like, do many final answers in a row. Um, and they have, like, their one verification system. To do this efficiently, you kind of do need to do, like, true, uh, turn-level training where you have this big forking process where you need to be able to assign rewards to each one. But sometimes if these are, like, actually intermediate reasoning steps, you don't have a good verifier off the shelf, which, uh, is gonna need both, like, new algorithms research as well as, uh, the ability to do verification for things like, "Was this a good Google query? Was this a good, uh, crop of the image? Was this thing helpful in service of the final goal?" So that's a question that we need to be a-able to answer efficiently, repeatedly in this kind of training. Um, and the most, one of the most promising ways, I think, towards doing this is having the reward models also be able to answer harder questions by themselves being reasoning models. So there's a recent DeepSeek paper on this sort of thing where you, uh, train models to be good, like, generalist reward models by doing inference scaling, where they break down tasks into subcriteria, rubrics, and then they can evaluate these numerically based on following the rubric in reference to the thing that they are scoring. Uh, and this, a-among other tricks, there's lots of, like, clever re- reverse engineering things you can do- To kind of have other measures of similarity or quality with respect to like a ground truth answer. Um, I won't go into all the details there, but the core idea there is if you think of reward models as reasoners, uh, who then you can also benefit from inference time scaling to do harder and harder tricks, uh, harder and harder problem-solving, uh, step verification. You have a way of giving these rewards throughout the process, which you kind of need to do if you want to scale to long contexts. Um, and, but that also introduces the bubble of like, okay, now reward modeling might take a minute for validating one term, which really means you want to push really heavily in the direction of everything being async and, uh, like happening later, which might mean your model's policy is being trained on, uh, terms that happened twenty minutes ago. Um, which is kind of like in old school RL days, people did this all the time with like replay buffers and whatnot. Um, you really want both your tool calls and your inference and your reward modeling and your training to all be happening in parallel with whatever, as much of a lag as you can tolerate, as well as your communication. Um, and so that is, I think, one of the big reasons why like pushing on async RL is really huge, not just for centralized training, but for things like scaling reward modeling, for things like scaling tool calls, um, scaling resources. ### Async RL **Will Brown** [13:42] Um, and I think Prime Intellect has done some very cool work showing that this is like a direction you can push. You can go off policy. You can allow your models to be trained on like slightly stale data, and it still works because these things are pretty robust. And if you set everything up right, and you tune your hyperparameters, and you ensure that, uh, the reasoning process, the training process is like stable otherwise, then you have a paradigm which allows you to do all of these things. You can scale on tool calls. You can scale on reward modeling. You can scale on generality of task. Um, and everything still fits into this nice async, uh, pipeline where everything is flowing from one to the next thing, uh, without like blowing up your training bubbles. Um, you can also decentralize your skill acquisition. You can train on many things in parallel, and it seems like model merging like works weirdly well. That was one thing that I kind of updated on, which is like you can have a model trained on code and a model trained on math and a model trained on Spanish, and you can literally average the weights, and it works. Which on one hand sounds crazy, but on the other hand, it's like this is kind of what is being done by dialog-style pre-training at smaller scales anyways. And it seems like the updates made to model weights are orthogonal enough for specialized tasks that this is actually like totally fine. Things are nice and linear in most cases. Things are nice and orthogonal, and you can get away with a lot of, uh, async, uh, updates to models that are then merged even without bulk communication. Um, and so Cohere did this very extensively in their latest, uh, flagship model Command A. Um, and it seems like we have a lot of ways where you can split up different pieces of the process of learning to do X, Y, or Z, or games, or coding agents, or math, and these can all go off and happen and then come together and merge. Um, and if you can do this, then you can train decentralized models with lots and lots of capabilities because the capabilities themselves can be learned in centralized manners. Um, so what does this mean? Like, it's gonna be hard. ### Roadmap **Will Brown** [15:44] It's not gonna be easy to like put all of these things together and make it work, make it work well, and really train these kinds of models super efficiently. But it seems like it's doable. Um, we kind of know the sorts of problems that we have to solve to do it. Um, we kind of know what things are gonna be really annoying and how we can kind of shortcut some of them by like using kind of, uh, fallback cheap tricks because these cheap tricks are like unreasonably effective. Um, and this does kind of give us a roadmap and a recipe of what questions we need to answer, what things we have to build, uh, and what we have to accomplish in order to, uh, train decentralized, uh, N-minute AGI. Okay, we have a special guest, um, that we're pretty excited about. Um, so here we go. ### Outro All right, actually, we're gonna pause for one second and just do another transition, so. This is Moji. Hi. Thank y'all. Thank y'all. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.