# [NeurIPS Best Paper] 1000 Layer Networks for Self-Supervised RL — Kevin Wang et al, Princeton Latent Space · 2025-12-31 Kevin Wang, Ishaan Javali, Michał Bortkiewicz, and Benjamin Eysenbach won the NeurIPS 2025 Best Paper award for RL1000, defying conventional wisdom by scaling reinforcement learning networks to 1,000 layers deep. They show that self-supervised RL—learning representations of states, actions, and future states via contrastive classification—scales where value-based methods collapse, and that critical architectural tricks (residual connections, layer norm, classification instead of regression) unlock performance that multiplies beyond a critical depth. Scaling depth is more parameter-efficient (linear growth) than width (quadratic growth), and Jax GPU-accelerated environments enable collecting hundreds of millions of transitions in hours, providing the data abundance that makes scaling possible. The paper argues that RL is finally ready to scale like language and vision, not by throwing compute at value functions, but by borrowing self-supervised objectives, with implications for robotics via goal-conditioned RL without human supervision or demonstrations, and deployment via deep teacher, shallow student distillation. ## Questions this episode answers ### How did the RL1000 paper manage to train reinforcement learning networks with 1000 layers? The team used self-supervised RL, which learns state–action representations via a contrastive loss (a classification task) instead of maximizing rewards. Kevin Wang explains that depth alone degraded performance, but combining this objective with architectural components—residual connections and layer normalization—caused performance to skyrocket, especially after around 50 million transitions. Benjamin Eysenbach stresses that the key shift is using a scalable objective, not just making networks bigger. [5:07](https://addtry.com/27be5d37-4f05-492c-988c-45845e7f54ae?t=307000) ### What is self-supervised reinforcement learning and how does it differ from traditional value-based RL? Self-supervised RL, as described by Kevin Wang, learns representations by pushing states from the same trajectory together and different trajectories apart—a binary classification problem—without any reward signal. This contrasts with standard RL, which regresses noisy TD errors to estimate value functions. The classification objective is more akin to losses used in language and vision, making it far more scalable and enabling deep networks to learn effectively. [4:29](https://addtry.com/27be5d37-4f05-492c-988c-45845e7f54ae?t=269000) ### What is the “critical depth” phenomenon observed in the RL1000 experiments? Kevin Wang and Michał Bortkiewicz note that performance did not improve monotonically with depth. After adding residual connections and layer normalization, and only when using 50 million or more transitions from GPU-accelerated environments, doubling depth at certain points—e.g., from 16 to 32 layers—led to performance multiplying dramatically, a behavior they call “critical depth.” [6:19](https://addtry.com/27be5d37-4f05-492c-988c-45845e7f54ae?t=379000) ## Key moments - **[0:00] Intro** - [0:41] Kevin Wang learned via email that their paper won Best Paper at NeurIPS 2025. - [1:36] Princeton undergraduate research seminar led by Benjamin Eysenbach sparked the RL1000 project. - **[2:15] The Challenge** - [2:25] Benjamin Eysenbach: 'I was kind of skeptical it was gonna work' when students proposed scaling RL networks deep. - [4:52] Kevin Wang explains self-supervised RL as learning representations of states, actions, and future states via contrastive learning. - **[5:25] Architecture** - [6:42] Ishaan Javali notes scaling depth grows parameters linearly, while scaling width grows quadratically, making depth more parameter-efficient. - **[7:49] New Objective** - [7:54] Benjamin Eysenbach: 'Our code doesn't have a line of code saying, Maximize rewards here'—the real shift is the self-supervised objective. - [9:46] Kevin Wang argues shifting RL from Q-learning regression to a classification-based cross-entropy loss is key to scalability. - **[11:47] Efficient Robotics** - [11:47] Q: How could RL1000 impact robotics? Kevin Wang says goal-conditioned RL can train robots with no human supervision or demonstrations. - [16:24] Kevin Wang says using Jax GCRL, they can collect hundreds of millions of transitions in a few hours by running thousands of environments in parallel. - **[16:55] Scaling Data** - [18:31] Kevin Wang draws parallel between self-supervised RL's next-state classification and language models' next-token prediction, calling it an 'implicit world model'. - [21:04] Q: Could deep teacher, shallow student distillation work for RL? Kevin Wang confirms it's a listed future research direction. - **[22:05] Next Steps** - [24:10] Kevin Wang reveals all RL1000 experiments, including 1,000-layer networks, run on a single 80GB H100 GPU. - [24:41] Ishaan Javali is actively exploring vision-language-action models for robotics, leveraging pre-trained VLMs. - **[26:43] Wrap** - [27:22] Kevin Wang says the general reaction to RL1000 is that it 'could be like a really cool direction' to push the frontier of RL scalability. ## Speakers - **Benjamin Eysenbach** (guest) - **Ishaan Javali** (guest) - **Kevin Wang** (guest) - **Michał Bortkiewicz** (guest) ## Topics Reinforcement Learning, Robotics ## Mentioned H100 (product), JAX (product), Jax GCRL (product), PyTorch (product) ## Transcript ### Intro **Benjamin Eysenbach** [0:03] Light in space life 22.5. Break ups. Light in space life 22.5 **Host** [0:13] So welcome to Lane space. Uh, we are basically trying to provide the best optimal sort of podcast experience of NeurIPS for people who are not here. Uh, and congrats on your paper. How does it feel? **Kevin Wang** [0:23] Yeah, it was very exciting. Um- **Host** [0:25] Yeah **Kevin Wang** [0:25] ... we had a poster yester- uh, yesterday, and then today we'll have a oral talk. **Host** [0:28] Were you just, like, mobbed? **Kevin Wang** [0:30] Oh, yeah. There was a lot of people. It was like three hours straight of, like, you know, like, waves of people to, like-- that we were trying to assist too, but- **Host** [0:36] So I've never received the best paper. D- did you just find out on the website? Like, what, uh- **Kevin Wang** [0:41] Oh, I just, like, woke up one day and, like, checked my email and then- **Host** [0:45] Ah, they just tell- they just like- **Kevin Wang** [0:46] Yeah. There was like, oh, like that's pa-- like I just saw email. Oh, you award-- like been awarded best paper. All that. **Host** [0:51] But maybe you know from the reviews as well, right? Is there anything- **Kevin Wang** [0:55] Yeah, we know from the reviews that we did well, um, but there's a difference between, like, doing well on the reviews and getting best paper. **Host** [1:01] Right. **Kevin Wang** [1:01] So that part we didn't actually know. **Host** [1:02] Yeah. Okay. So I, I, I skipped a little bit. Uh, maybe we can go sort of, um, one by one and, and sort of introduce, um, you know, who you are and what you did on, on, on, on the team. **Kevin Wang** [1:12] I'm Kevin. I was an undergrad from Pr- from Princeton, and I just graduated. And yeah, I guess I led the project, like started the project and then was, was very happy to collaborate with Ishaan and Michal and Ben also, um- **Host** [1:25] Right. And were you in like the same research group? Like how do you- **Kevin Wang** [1:28] Yeah. So, yeah. **Host** [1:28] What's your- **Kevin Wang** [1:28] Yeah. So, uh, we were- **Host** [1:29] The social context **Kevin Wang** [1:30] So, so yeah. So we're all from Princeton. **Host** [1:32] Yeah. **Kevin Wang** [1:33] Um- **Host** [1:33] With, and thanks to Alan for- **Kevin Wang** [1:35] Yeah **Host** [1:35] ... hooking you guys. **Kevin Wang** [1:36] So this project actually started from like an IW seminar, so, uh, like, like independent work research seminar- **Host** [1:41] Mm **Kevin Wang** [1:41] ... uh, that Ben was teaching. And this was like, actually like, like one of my first experiences in like ML research. Um, so it was really valuable to like get that experience. And then Ishaan was also in that seminar and working on adjacent things, so we collaborated, um, a lot during that seminar. And then, yeah, the project turned out to have some pretty cool results. And then later on also like Michal working on sort of similar things also, um, joined in on the project and became like a good collaboration. **Host** [2:06] Yeah. And, um, I, I don't know if any of you guys wanna, wanna chime in on, on like other elements of coming into, like deciding on this, uh, problem. **Benjamin Eysenbach** [2:15] So it's like broadly my lab works on deep reinforcement learning, but historically deep meant like two or three or four layers. ### The Challenge **Host** [2:23] Not one thousand. **Benjamin Eysenbach** [2:25] When Kevin and Ishaan mentioned they wanna try really deep networks, I was kind of skeptical it was gonna work. I've tried this before, it doesn't work. Other papers have tried this before and it doesn't even work. So I was very, very skeptical starting out. I don't know if I conveyed this at the time, but that was my prior going in because- **Host** [2:41] But do you, do you view your job as like screening or like, "Hey guys, this is probably isn't gonna work. You should try a different idea." You know, like... Or should you be encouraging even if it's dumb? **Benjamin Eysenbach** [2:51] It's selecting bets. **Host** [2:53] Yeah. **Benjamin Eysenbach** [2:53] And this was a bet I was willing to make. **Host** [2:55] What, what made you willing to make a bet? Like- **Benjamin Eysenbach** [2:58] It seemed relatively low cost, uh, in that we, Michal in particular, had spent the past year developing infrastructure that made it a lot easier to run some of these experiments. And the precedent was deeper networks could do a whole lot better. Like that's what the deep learning res- revolution's been over the last decade. **Host** [3:15] Yeah, I know. Why do we stop making them deeper? **Benjamin Eysenbach** [3:18] Uh, and reinforcement was like this one anomaly where we continue to use these really shallow networks. And that's particularly true in the settings that we were looking at, where you're starting from scratch, you're starting from nothing. **Host** [3:28] Any other perspectives you guys wanna chime in with? **Kevin Wang** [3:30] I guess maybe I should just go over like an overview of our project. **Host** [3:34] Yes. Okay. Sorry. Yes. Uh- **Kevin Wang** [3:35] So the way that I kind of view our project, um, is that if you look at the landscape of deep learning, you know, you have NLP like language, vision, and then RL. And like as Ben kind of alluded to, you know, like in, in language, in vision, we've sort of converged to these like paradigms of scaling to massive networks, right? Like hundreds of billions of parameters, trillions of parameters. And there's been, you know, a lot, a lot gained from, in deep learning from, from that, right? And then, but then it seems like in the third sort of branch of deep learning, in deep RL, that has not yet been the case. Um, like I was very surprised like coming into some, like, you know, Ben's class and seminar when I was looking at the networks. Oh, why were you just using like a simple like two-layer MLP for like these frontier sort of, you know, state of the art RL algorithms? Um, and so I was very curious like, can we design RL algorithms? Can we sort of put together a recipe for RL that can allow it to scale in potentially, you know, analogous ways that language and vision might scale? And so what we did is that we know that traditional RL, like let's say like value, uh, value-based RL doesn't really scale, right? This is pretty clear from the literature. So we tried a different approach to RL, um, called self-supervised RL, where instead of learning like a value function, we're learning representations of states, actions, and future states such that the representations along the same trajectory are pushed together, the representations along different trajectories are pushed apart. And this is just like a different approach to, uh, RL that's allows us to learn in a self-supervised manner. So there's the, we can solve task reach goals without any human crafted reward signal. And so we know that self-supervised learning is scalable in these, uh, different areas in d- of deep learning. So can self-supervised RL scale in similar ways? Um, when we first tried it, it actually didn't work. Like we made the networks deeper, the performance like totally degraded. But then we also tr-- but then I separately was like, there's also some other work like, um, in, in our literature, like we tried like residual connections. ### Architecture **Kevin Wang** [5:25] Um, and then there's other, a few other architectural components that we had to put into the recipe. And then all of a sudden, like one day, like I ran this experiment and there was like this one environment in which there was like, like going from like, like doubling the depth didn't really do anything. But like doubling the depth again with these different components suddenly like skyrocketed performance in this one environment. **Benjamin Eysenbach** [5:46] Getting this to work was very non-trivial in the sense that like usually when we think about doing hyperparameter opti- optimization, we try changing A, see if it makes it better, try changing B, see whether it makes it better. And if we just made the depth bigger, makes it worse. If we just add residual connections, didn't make it better. And it was really this combination of factors that Kevin and Ishaan figured out that really made this work. **Kevin Wang** [6:07] And as a precursor to that, we also tried scaling along different dimensions. So scaling the batch size, uh, scaling the, the width of the networks or the hidden layers. **Host** [6:15] M effect. **Kevin Wang** [6:16] It-- Yeah, pretty much kind of similar to just scaling depth naively. **Host** [6:19] Yeah. **Kevin Wang** [6:19] Um, and then once we started introducing residual connections, Layer Norm, these specific architectural choices, that's when we saw these significant jumps in performance, like these critical depths at which performance multiplies by a pretty huge factor. And that's where we really noticed, like unlocking some significant performance gains, uh, as opposed to scaling just along width, which did yield some performance improvements. Um, but when you look at the number of parameters that your network has as you grow width, it's roughly a quadratic as opposed to something like growing depth. So it's more, in some sense, it's more parameter efficient, also more sample efficient from the experiments that we conducted. **Host** [6:55] Nice. Um, in some ways you're sort of replicating stuff that is seen in the wild, but on, on a very small model that you can study. Is that-- Would you, would you say that's- **Kevin Wang** [7:04] Yeah. So to kind of add to what Kevin said earlier, we saw these huge performance improvements in language models, image generation models by making them larger, making them deeper, which seems very intuitive. **Host** [7:14] Yeah. **Kevin Wang** [7:14] Uh, and so that's why our work we, we draw from like foundational research, right? Like, uh, residual networks, which employ residual connections to avoid, uh, vanishing gradients. And that's something that we show in some of our ablations in our, in our paper, like further down, uh, probably in the appendices, where we did experiments without these residual connections. And so it's sort of borrowing these concepts that have existed in other fields and applying them to this setting with, uh, RL and sh-showing that it works. **Host** [7:41] Before Ben has to, has to go, I'll, I'll leave the sort of last word, uh, to him. What additional work does this inspire that like, that, that you want to push on next? ### New Objective **Benjamin Eysenbach** [7:49] I think there's one thing I'd clarify about the paper, and then w- I'll directly answer the question. **Host** [7:54] Yes. **Benjamin Eysenbach** [7:54] I think the thing I might clarify about the paper is I think a lot of people reading the title are like, "Wow, big networks, they're great." I'll take big networks and- **Host** [8:00] You solved it now. We can just go. **Benjamin Eysenbach** [8:01] Yeah, you just take big networks from the-- add them to PPO, add them to SAC, add them to your favorite reinforcement learning algorithm. But I think that's like not the main conclusion. I think the main conclusion is that using big networks not only requires these architectural tricks, but also, as Kevin mentioned before, it requires using a different objective. This objective doesn't actually use rewards in it. And so there's another word in the title, reinforcement learning, that also might be a little bit of a misnomer because we aren't directly trying to maximize rewards. Our code doesn't have a line of code saying, "Maximize rewards here." And so is, at the end of the day, this a reinforcement learning method? I don't know. It looks much more similar to the self-supervised methods in other areas of machine learning. And so I think that the method and the work really stands in some sort of interesting intersection of reinforcement learning and self-supervised learning research. **Host** [8:52] Mm-hmm. **Benjamin Eysenbach** [8:52] And we had this little figure on the bottom left of the poster, which was the screenshot of a slide from Yann LeCun talking about how to build intelligent systems and whether that's gonna be done by unsupervised learning or supervised learning and reinforcement learning. And I think what our paper really suggests is that the boundary between these things is really blurry, and maybe the key to building intelligent systems is gonna be leveraging insights from all of them. **Host** [9:16] Yeah, the layer cake. **Benjamin Eysenbach** [9:18] Exactly. **Host** [9:20] Uh, well, thank you for your time. I know, I know you have to go soon. **Benjamin Eysenbach** [9:22] Yeah, unfortunately, I have to run. Thanks. **Host** [9:23] Yeah, thank you so much for coming. I, I think that the insight of like blurring things is interesting. I, I don't know if you, like you, you were just talking about sort of like, uh, the abstraction layer of representation learning. I don't know if, if that triggers anything in terms of like the mix between self-supervised and reinforcement learning. Is that, is that something fundamental that you've discovered or that we've-- that people don't understand when they, when they read the paper? **Kevin Wang** [9:46] Yeah. I think the best way that I would explain it is that we know that standard RL is not super scalable, and so like why can this different approach or different objective RL be scalable? I think it's because we're fundamentally shifting the burden of learning from something like Q, like Q learning, or like regressing to like TD errors, which we know is quite spurious and noisy and biased, to fundamentally like a classification problem. We're trying to classify whether a future state is along the same trajectory or along a different trajectory. And we do this with representation learning, right? And we know that classification, cross entropy loss, and representation learning is scalable in the deep learning literature, right? If we think about language, um, a-and like some of the objectives there. So in some sense, we're kind of blurring the, the lines. We're doing re- reinforcement learning. It's still an actor-critic reinforcement learning algorithm. It's like a goal condition reinforcement al-algorithm. But the objective, the burden of like learning, of the, of, of solving the RL task shifts to something that's more similar to s- some objectives that you might see in language and vision that we know have scaled so much. And so I think, yeah, I think that's like one of the fundamental insights that we've seen is that, um, it seems like by approaching RL in this different approach, we're able to like s- get so much more out of-- We were able to scale our networks like significantly beyond what was like standard used in RL. **Michał Bortkiewicz** [11:03] Can I jump in? I will just give a bit of more of context about the architecture because, uh, yeah, we, we use another objective, uh, the, uh, influence here, so the contrastive loss. However, the architecture is, uh, quite similar to the previous works, uh, of previous, uh, papers like BRAW or, or, uh, SIMBA, SIMBA V1 and SIMBA V-V2, uh, SIMBA V1, SIMBA V2. So we, we also tweaked a bit of this, uh, architecture. However, it's not that we like, uh, invented the wheel for the first time. It, it's the merging between the architecture and the objective that makes the scale, uh, really, uh, like go up a-and, and performance follow the, the scale. **Host** [11:47] I think that's something that we should, uh, probably mine deeper. Um, do you think, I guess like what domains, what industry, like you, you've applied it on, on multiple different, uh, types of networks that or, or datasets. Is there a particular affinity that you think like has, is like kind of low-hanging fruit? ### Efficient Robotics **Kevin Wang** [12:04] Yeah. So actually if you look at a lot of our tasks, they're particularly sort of like robotics tasks. Um, so this is, uh, personally, I've, I'd be very curious about how a work like this could- Impact like the robotics field. Like my understanding of robotics is that a lot of robotics are now this kind of multi- a few different approaches. Like one approach is we want to train robots using imitation learning, so we try to collect like an insane amount of data. We have a ton of huper- human siv- su- supervision, and we try to scale up this data, and we're like learning with imitation learning. Like, but on the other hand, potentially like perhaps there's another approach, which is like, for example, like goal condition reinforcement learning, where we can actually train robotic agents and train RL agents to solve meaningful tasks with absolutely no human sup- supervision, no demonstration- **Michał Bortkiewicz** [12:46] And it's much more scalable. Yeah. **Kevin Wang** [12:47] So yeah, so this could serve as an alternate approach, and perhaps instead of like scaling data, like scaling manual, like human supervision, which, which is, you know, not super scalable, if there are ways to sort of make goal condition reinforcement learning scalable, and like we can just scale the architecture or we can scale- **Michał Bortkiewicz** [13:02] Because you're focused on other objectives. Yeah **Kevin Wang** [13:04] ... right, with, with certain different objectives. I think that could be very exciting and see how-- to see how that can affect a field like robotics, for example. **Michał Bortkiewicz** [13:10] Yeah. Uh, double-click on, on just one more, one thing on the efficiency, which you, you guys were talking about. I would expect the very deep mo- the deeper it is, it should be quadratically worse. I, I'm not familiar with like the, the preexisting literature. I'm just like sort of working out intuitions. **Kevin Wang** [13:25] Mm-hmm. **Michał Bortkiewicz** [13:26] But, um, basically, uh, what are the trade-offs that you've found that I think you might wanna warn people about? Because, because you, you are the guy who mentioned efficiency, so. **Ishaan Javali** [13:36] Uh, sure. S-sure. Yeah. So I was referring to like one of the figures on our poster, also in our paper, where we compare like the number of parameters that models have as we scale along the axis of depth and as we scale along the axis of width- **Michał Bortkiewicz** [13:47] Yeah **Ishaan Javali** [13:47] ... from our baseline architecture. The most baseline one would be like a width of two fifty-six, like the hidden layers have two fifty-six neurons, and then the depth is four, four layers or hidden layers. Um, and so the point I was making there is that when you scale along depth, your, the number of parameters that your model has is gonna grow roughly linearly. Uh, whereas with width, you're making your network outputs wider, and then the input to the net- next network is also growing as well. And so the, the number of parameters your network's then gonna have grows approximately quadratically. And so one of the experiments we did was sort of examining as we grow the number of parameters in our model by scaling along these two different choices, which one for the same like approximate number of parameters yields a better performance. And the depth curve kind of goes like this. It jumps up pretty fast. That's like present throughout our paper. For width it grows a little bit more slowly. And so that-- the kind of takeaway from that is that if you are a bit more resource constrained, scaling along depth might be better because with fewer parameters with a smaller model to a small or a number to learnable parameters- **Michał Bortkiewicz** [14:47] Width is expensive **Ishaan Javali** [14:48] ... width is expensive, exactly. And in general, of course, like more parameters is also gonna be more expensive, so that's just like another consideration to, to think about when using these networks, I suppose. **Michał Bortkiewicz** [14:57] Yeah. Any other sort of rules of thumbs like that that I can extract that this is just the most basic one that I could think of? **Ishaan Javali** [15:03] Yeah. **Michał Bortkiewicz** [15:04] Uh, I don't know if there's any others. **Kevin Wang** [15:06] Yeah, I guess like to your original question of like the trade-offs, um, like one of the trade-offs, one of the limitations that we say is like, obviously, if you make the networks bigger, the, it, it will takes longer to run, right? **Michał Bortkiewicz** [15:15] Mm-hmm. **Kevin Wang** [15:16] So if you like double the depth, at some level of depth, you, you-- it might take like twice as much to like take, make a forward pass th-through the network, right? However, this is not-- So like within our paper, like for most environments, um, we are able to like saturate, like get to like almost perfect performance within just, you know, we don't even need to get to like a thousand layers, like maybe just sixty-four layers, for example, is sufficient. Um, and i-in this regime, like, like the latency of the network is not necessarily actually even the, uh, not necessarily like a, a significant bottleneck. Like you can imagine there's a lot of tasks in which e-especially in RL, that like collecting data might be the bottleneck, right? And making forward passes through our network may not be the bottleneck. And so in our environment, we sp-- in our research, we specifically used the Jax GCRL environment, which is a Jax based GPU accelerator environment. So we can collect like thousands of like environment trajectories like in parallel at the same time so that we are able to like make, uh, like- **Michał Bortkiewicz** [16:11] Oh, this is built-in. **Kevin Wang** [16:12] Right, this is built-in so that we can collect, you know, like, like thou- a thousand trajectories at the same time a-along all these environments. And so, um, ma-makes that-- makes sure that like we have enough data to like saturate the learning from- **Michał Bortkiewicz** [16:23] Wow. **Kevin Wang** [16:24] Yeah. **Ishaan Javali** [16:25] That's like work they had in columns. **Michał Bortkiewicz** [16:26] Okay. And you-- I don't know if you wanna ex-explore, expound upon that on, on the Jax GCRL. Uh, maybe I- **Kevin Wang** [16:32] A-a-and you know, most people are familiar with PyTorch and may less familiar with Jax, so. **Michał Bortkiewicz** [16:35] Uh, with Jax. I think Jax- **Kevin Wang** [16:36] Okay, back up **Michał Bortkiewicz** [16:37] ... kind of is getting the, uh, the traction, especially in RL field because the, in-- for online reinforcement learning, getting as much data as you can is, is the most important. **Kevin Wang** [16:48] There's, there's gotta be a PyTorch equivalence. But anyway. **Michał Bortkiewicz** [16:50] However- **Kevin Wang** [16:50] Any ti-tips for other people ex-- also exploring this kind of, uh, rollout? Yeah. **Michał Bortkiewicz** [16:55] Yeah. So I think I can also recommend like, uh, for, for goal-conditioned RL, I-I'm recommending Jax GCRL, but there are also like multi-agent Jax, uh, implementations and others. So going back to our paper, if you look at the plots, we only see this like huge performance increase when we cross like fifty, uh, millions of, uh, transitions, uh, gap. ### Scaling Data **Michał Bortkiewicz** [17:19] So, so I think the data is crucial like here. **Kevin Wang** [17:22] Yeah. I guess even to build on that, like I, I like drawing analogies to like successes in other areas of deep learning. Like for example, in large language models, the reason why we're able to scale to such large networks is that we found a paradigm in which we can leverage the entire internet scale of data- **Michał Bortkiewicz** [17:38] Yeah **Kevin Wang** [17:38] ... to learn, right? And so data in RL traditionally has been hard to come by. Um, but now with these like GPU accelerator environments, we can collect hundreds of millions of time sets of data within just a few hours. And so I think that this serves as like a really good test bed for us to be able to also find ways to scale up, um, like network capacity and get similar kind of gains. **Michał Bortkiewicz** [17:58] I think I asked about- **Ishaan Javali** [17:59] Are you saying that you have a difference-- you would do pre-training differently in LLMs? Like, what's the, what's the difference, uh, objective now? Um... **Michał Bortkiewicz** [18:09] Yeah, yeah. Ve-very simply, very simply, the, the paradigm that you're referencing is next word- **Ishaan Javali** [18:14] Yeah **Michał Bortkiewicz** [18:15] ... or next token, right? It's very robust. **Kevin Wang** [18:19] Yeah. Yeah. **Michał Bortkiewicz** [18:19] How do you change that? **Kevin Wang** [18:20] Oh, I'm not saying that we're changing. I wanna leverage insights from that To apply to RL **Host** [18:24] Oh. I, I feel like you should go the other way **Kevin Wang** [18:27] You think we should go the other way? **Host** [18:28] Yeah. **Kevin Wang** [18:28] Maybe. I mean, that, that would be a very interesting research direction too. But actually, yeah, even on that point, like, one of the things I was thinking about is that the way that our RL objective works is in some se- it's not exactly next word prediction, but it's kind of like next state prediction, right? You imagine you're at some current state and you're at some, a current action, and we wanna predict whether or not this future state, this, this certain state is a future state along the same trajectory or a different trajectory. And so in some sense, we are actually doing some sort of like- **Host** [18:56] Implicit world model **Kevin Wang** [18:57] ... implicit like, uh, you know, like in, in- **Host** [18:59] I don't know if that's a bad word these days **Kevin Wang** [19:01] Or, or like in language you, you do a cross-entropy loss to classify the next token, right? And here we're just doing a binary classification of like whether or not some next state is some- **Host** [19:09] Yeah, yeah, yeah. It's a classification. Yeah, yeah, yeah **Kevin Wang** [19:11] And so I do see that there are some like sort of parallels here that perhaps we should dig into deeper and see like what is the core to, of what enables deep learning to scale, and then how can we, like, leverage that? How can, how can we distill those, like, insights and then apply those across, like, all different fields, whether it's language or reinforcement learning. **Host** [19:29] Yeah. Uh, did you, did you get my, my meaning about the world model stuff? **Ishaan Javali** [19:32] Yeah. Yeah. Actually, a- and I, I heard, I think I might have heard Professor Eysenbach yesterday talking about this at a poster, and he's explaining to a couple people that because this is, like, doing representation learning and trying to learn these meaningful representations, uh, for a given state and action, but for a given goal, in some sense you can think of it almost like learning a model of the environment, learning a model of the world, but without having to do any sort of like next frame prediction or stuff like that that's a little bit more high dimensional and complex. Yeah. **Host** [19:59] Yeah. A- and I would think, like, um, the, the angle that I'm trying to think about and push is instead of learn the next world, the- they're, they're basically like generate a number of candidate possible worlds and classify them, uh, to your point. Uh, which is exactly how I do things. Let's say I'm playing poker and I'm trying to classify what hand you have. Well, there's a range of hands based on what you're, you're doing, and the more information I get, the more I resolve to, "Oh, I know exactly what hand you have based on what you're showing," you know, or you're bluffing. But that's a, that's a different thing. But you know, you know what I mean? Like I, I, I feel like that is the ultimate sort of end goal of representation- **Ishaan Javali** [20:35] Mm-hmm **Host** [20:35] ... which is a, a world. But I don't know if that is too vague compared to the more concrete types of world models that, let's say, the VideoGen people are doing. A- and then I, I guess o- one, one of the things like I, I'm also exploring, I-- you mentioned like the deep models being slower or more expensive. Yeah. That, that is a trend in the inference world of making models shallower, right? **Ishaan Javali** [20:57] Mm-hmm. **Host** [20:57] And I wonder if this, like, short catchphrase I was thinking about, like deep teacher, shallow student- **Ishaan Javali** [21:04] Mm-hmm **Host** [21:04] ... would be a good deployment paradigm. **Ishaan Javali** [21:08] Yeah. **Host** [21:08] W- like you push the frontier capabilities with d- with, with F and then you distill- **Kevin Wang** [21:12] Distill it back. I, yeah, actually, this is a good point. Like, if you go onto our website, like this is one of the future directions that we list at the very bottom of like- **Host** [21:19] Oh, okay. **Kevin Wang** [21:20] Yeah. Uh, we, we, we, we would love to see if we could get similar performance. Like we pushed the, you know, like we do achieve state-of-the-art performance on, uh, goal-conditioned RL and Jax GC RL by a significant amount. And so it was very exciting to see the, like the, the sort of frontier of the ability to train RL agents, uh, sort of pushed. Um, and if we can do that in a way that also sort of is just as efficient as a standard, uh, you know, networks, that would be very cool. So, you know, like is- **Host** [21:46] Yeah, because training, uh, doesn't have to be the s- the same thing that you deploy at inference. **Kevin Wang** [21:51] Right. **Host** [21:51] You know what I mean? Like the- **Kevin Wang** [21:52] Yeah. So, yeah. So if there's ways to like distill down to a smaller model or improve the model and maybe not, and still retain performance, that's a very interesting research direction that we should- **Host** [22:00] Let's talk about other, uh, future directions. What, what, what else is your personal passions? ### Next Steps **Michał Bortkiewicz** [22:05] Yeah. So, uh, currently I'm, I'm pursuing a direction of, uh, stitching in reinforcement learning. So we are trying to generalize, uh, reinforcement learning from shorter sub-behaviors so that they are stitched, merged, uh, during the test time. And, uh, yeah, I think this is one of my, uh, last papers that I will tackle during the PhD. **Kevin Wang** [22:28] Personally, I would-- I'm very curious of like w- can we, like what's the r- like real, like can we push... I'm, I'm, I'm curious about like advancing the frontier as much as possible. Um, so if you actually look at our paper, we focus on scaling depth, but we notice that, we see that scaling width actually also improves performance. And we also find that actually by scaling depth, we actually unlock the ability to scale along batch size as well. Um, so this is- **Host** [22:51] Ah **Kevin Wang** [22:52] ... one of-- Yeah. Uh, so, so okay, so I guess- **Host** [22:53] It's like collinear, like yeah. **Kevin Wang** [22:55] Right. So like, okay, I guess for context, like in traditional RL, like value-based RL, scaling batch size is not super effective. But there's-- we also can see there's also other work in other areas of deep learning that show that scaling batch size is only most effective when there's like a large enough network capacity to take advantage of, of the scaled batch size. And we actually find that, you know, perhaps, so one hypothesis might be like perhaps the reason why scaling batch size isn't that effective in traditional RL is 'cause like we've been using these tiny networks that haven't been able to capture that. And one of our experiments is that like because we are enabled successful training of deep network, we actually were able to-- this is a great test bed for, you know, like testing this hypothesis, and we find that indeed, as we scale the network capacity, we also unlock this different dimension of scaling batch size. And so all of that to say is that I'm very curious for someone like with enough compute to like take some of these environments, scale up batch, uh, scale up depth to the maximum capability, also scale along width, also scale along batch size, and let, let's like basically like in the same way that in language we're, we're scaling along so many different axes, can we unlock different dimensions of scaling as well, and what capabilities and how far can we push the frontier of training these RL agents from doing that? **Host** [24:04] Before we pass to Ishaan, uh, when you say enough compute, what kind of compute budget did you have? How, how does it, how- **Kevin Wang** [24:10] Yeah **Host** [24:10] I just wanna- **Kevin Wang** [24:10] That's, that's a- **Host** [24:11] ... see what you guys got **Kevin Wang** [24:12] ... good question. So we, we wanted to make sure that this is-- we, we wanted to make it such that like, uh, you know, it's quite accessible. So actually the nice thing is that all of our experiments, even the thousand-layer networks, can be run on one single eighty gigabyte H100 GPU. **Host** [24:25] Mm. So that's just dollars. Yeah. **Kevin Wang** [24:28] Right, right, right. So everything can be run on one GPU. Um, but in theory, if we had, you know, like a distributed training setup and, like, can just like blast compute through this and really wanted to push the frontier, it'd be very interesting to see how things go. **Host** [24:40] Yeah, cool. **Ishaan Javali** [24:41] Uh, and I've actively been trying to learn as much as I can about vision-language-action models, um, role models at NeurIPS, and going to a lot of- **Host** [24:48] Machine language action models? **Ishaan Javali** [24:49] Uh, vision language. **Host** [24:50] Vision language. **Ishaan Javali** [24:50] VLMs. **Host** [24:50] VLMs. Yeah, yeah. Okay. **Ishaan Javali** [24:51] Um, and yeah, curious about applications of representative- **Host** [24:54] Robotics **Ishaan Javali** [24:55] ... learning for these. Yeah, exactly. **Host** [24:56] Yeah. **Ishaan Javali** [24:56] For robotics. Um, actively trying to explore more in that area, so just reading a lot of literature, talking to as many people as I can. **Host** [25:03] Yeah. We just released our episode with, uh, general intuition- **Ishaan Javali** [25:07] Oh, okay **Host** [25:07] ... um- **Ishaan Javali** [25:08] Awesome **Host** [25:08] ... where it's-- if you know a bit about their history, they started as a gaming clipping company. **Ishaan Javali** [25:13] Yeah. **Host** [25:13] And, uh, they basically have a vision language action model. **Ishaan Javali** [25:16] Yeah. **Host** [25:16] Which, um, I, I saw, I saw a preview. It was, it was very impressive. I'm not sure exactly how transferable it is to embodied use cases, but it doesn't have to. Like, screen is fine, you know? Like, yeah, I, I don't know if you have any takes on that side. **Ishaan Javali** [25:31] Yeah, it's an exciting research direction, definitely. **Host** [25:34] Yeah. I, I, I think, um, the, the, the, the, the concept of actions as, as something that you are outputting is actually not that popular in industry, right? O- only because text has completely dominated the last three years, and tool calling and-- which is, uh, just another form of structured text. Uh, and, and I, I feel like the, uh, action research is, is kind of like, I don't know how-- I don't know what needs to happen in order to unlock the next phase in, in that. I don't know if you- **Ishaan Javali** [26:06] Yeah, I think it's- **Host** [26:07] ... have seen anything interesting out here, shot, shot it out. **Ishaan Javali** [26:09] Yeah, there's a lot of cool, like, work on, like, leveraging pre-trained VLMs and- **Host** [26:13] You freeze it, and then you apply- **Ishaan Javali** [26:15] Yeah. **Host** [26:15] Yeah. **Ishaan Javali** [26:15] Exactly. And then you- **Host** [26:16] Very classic **Ishaan Javali** [26:16] ... keep creating on top of that, like- **Host** [26:17] Yeah **Ishaan Javali** [26:17] ... some sort of experts to output actions. Um, also like systems for doing like hierarchical planning, maybe outputting some higher level plan that-- and this is like a larger network that takes a long time to-- a little longer to do inference, and so it outputs its plans with less frequency, like some sort of chunk, and then from there, there's like some sort of, uh, second system that operates a bit more fast. I think there's quite a bit of interesting research in that direction, so that's sort of what I'm looking forward to. ### Wrap **Host** [26:43] Cool. Final question. Uh, hardest question you were asked at the poster session, or just favorite encounter, anyone famous that you met? **Kevin Wang** [26:51] So I actually haven't gotten a chance to go to the conference that much. I'm actually working full-time now, so- **Host** [26:55] Oh, damn. **Kevin Wang** [26:56] Yeah. Uh, so, so far I, I, I actually literally just got my badge like a few moments before- **Host** [27:01] Oh, man **Kevin Wang** [27:02] ... before my poster session. So I, I guess I wouldn't be the best to answer that question. **Host** [27:05] No, no, no, like w- because you-- like people ask you stuff, right? **Kevin Wang** [27:08] Oh, oh, oh. At my, at my poster. **Host** [27:09] People asking you- **Kevin Wang** [27:10] Yeah **Host** [27:10] ... or meeting you- **Kevin Wang** [27:11] Yeah **Host** [27:11] ... and like, you know, just, just give a vibe of like what people are saying and- **Kevin Wang** [27:16] Yeah **Host** [27:16] ... saying about- **Kevin Wang** [27:16] I think people were very-- I think it, it's sort of like a very eye-opening. I think that the general question is that people thought it was a very eye-opening paper because, like, the objective is quite simple. It's quite elegant. And for us to be able to like, you know, like, I, I don't want to say like overturn, but like sort of challenge the conventional wisdom that like RL is not super scalable and push it to such limits like 1,000 layers deep and see continued impr- pr- improve performance. I think the general impression that I've gotten is that, you know, this, this, this could be like a really cool, like if, if we can sort of build along this direction and that like we can really scale along th- all these different dimensions and push the frontier of the ability for RL. I'm very curious to see how that plays out. **Host** [27:58] All right. Well, thank you so much for dropping by. Uh, congrats on the paper again. **Kevin Wang** [28:01] Yeah. **Host** [28:02] And, uh, good luck in your future work. **Ishaan Javali** [28:03] Thank you. Thanks for having us. **Host** [28:04] Yeah. **Kevin Wang** [28:05] Yeah. **Host** [28:05] For sure. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.