The Origin and Future of RLHF: the secret ingredient for ChatGPT - with Nathan Lambert

Hey everyone, welcome to the Latent Space Podcast. This is Alessio, partner and CTO in residence at Decibel Partners, and I'm joined by my co-host Swyx, founder of Smol.ai.

Hey, and today we have Dr. Nathan Lambert in the house. Welcome.

Thanks guys.

Uh, you are- you didn't have to come too far. You, uh, got your PhD in Berkeley, and, uh, it seems like you've, you've lived there, uh, most of the time in recent years. Um, you, uh, worked on robotics and model-based reinforcement learning on your PhD, and you also interned at FAIR and DeepMind.

Um, you were- you bootstrapped the RLHF team at Hugging Face, and you recently joined the Allen Institute as a research scientist. Um, so that's your quick bio. Uh, what, what should people know about you that maybe is not super obvious about you on, on your LinkedIn?

Um, I stay sane in various, um, insane sport, I... In ultra-endurance sport activities that I do.

What's a ultra-endurance sport activity?

Um, like long distance trail running or gravel biking.

Nice. Nice.

I'll try to unplug sometimes, although it's harder these days.

Yeah. Um, well, you know, just the Bay Area is just really good for that stuff, right?

Oh, yeah. You, you can't beat it, and it's... I have a trailhead like 1.2 miles from my house, which is pretty unmatchable in any other urban area.

Yeah. Yeah. Uh, pretty excellent. Um, you also have, uh, a incredible blog, Interconnects, which I, which I'm a fan of. Um, and I also just recently discovered that you have a new podcast, Retort.

Yeah, I do. I've been writing for a while, and I feel like I've s- finally started to write things that are understandable and fun after- ... a few years lost in the wilderness. If you ask some of my friends that I made read the earlier blogs are like, "Oh, this is yikes."

But-

Yeah

... that was... It's, it's coming along, and the podcast is with my friend Tom, and we just kinda like riff on what's actually happening on AI and not really do news recaps, but just what it all means and have a more critical perspective on the things that really are kind of funny but still very serious happening in the world of machine learning.

Yeah. Awesome. Uh, for people who are new to your work, like what would you highlight as like your greatest hits so far on like Interconnects at least?

So like the ones that are most popular are like timely and/or opinion pieces. So the first real breakout piece was in April when I also just wrote down the thing that everyone in AI was feeling, which is like we're all feeling stressed- ...

um, that we're gonna get scooped and that we're overworked, which is like behind the curtain what it feels to work, like work in AI. And then a similar one, which we might touch on later in this, was about my recent job search, which wasn't the first time I wrote a job search post.

I love that.

But people always love that stuff.

It's so open.

It's... I mean, it's like easy for me to do i- in a way that it's very on brand, and it's very helpful. Like it's... I, I understand that until you've done it, it's hard to stare- share this information.

And then the other popular ones are various model training techniques or fine-tuning. There's an early one on RLHF, which is... This, this stuff is all just like when I figure it out in my brain. So I wrote an article that's like how RLHF actually works, which is just the intuitions I had put together in the summer about RLHF, and that was pretty well.

And then I opportunistically wrote about Q*, which- ... you hate that you have to do it, but it is pretty funny. I found it's like g- from a literature perspective and like OpenAI publishes on work that is very related to mathematical reasoning.

So it's like, oh, you just poke a little around what they've already published, and it seems pretty reasonable. But we don't know. They probably just got like a moderate bump on one of their benchmarks, and then everyone f- lost their minds- ...

it doesn't really matter.

Like this is why Sam Altman was fired. Uh, I don't know. Anyway, um, yeah, we're, we're here to talk about RLHF 101. Um, you did a presentation, and, uh, you- I think you expressed some desire to re-record it, and that's why I reached out on Twitter saying like, "Why not re-record it with us, and, and then we can ask questions and talk about it?"

Um-

Yeah. Sounds good. I think it's-

Yeah.

I try to do it every six or 12 months- ... is my current, is my estimated cadence-

Yeah

... just to refine the ways that I say things, and people see w- that we don't know that much more, but we have a b- better, better way of saying what we don't know.

Yeah. Awesome. Um, we can dive right in. I don't know if there's-

Yeah

... any other, uh, topics that we wanna lay out as groundwork.

No, you have some awesome slides, so for people listening on podcasts only, we're gonna have the slides on our show notes, and then we're gonna have a YouTube version, uh, where we run through-

Like and subscribe

... everything together.

Sounds good.

Um-

Yeah. So I think to start skipping a lot of the like what is the language model stuff-

Yeah

... everyone knows that at this point. I think the quote from the LLaMA 2 paper is a great kind of tidbit on RLHF becoming like a real deal. There was some uncertainty earlier in the year about whether or not RLHF was really gonna be important.

I think it was not that surprising that it is. I mean, with recent models still using it, the signs were there. But the LLaMA 2 paper essentially reads like a bunch of NLP, NLP researchers that were skeptical and surprised, so the quote from the paper was, "Meanwhile, reinforcement learning, known for its instability, seemed a somewhat shadowy field for those in the NLP research community.

However, reinforcement learning proved highly effective, particularly given its cost and time effective- effectiveness." So you don't really know exactly what the costs and time that Meta is looking at 'cause they have a huge team and-

Mm-hmm

... a pretty good amount of money here to release these LLaMA models. But like h- this is just the kind of thing that we're seeing now. I think any major company that wasn't doing RLHF is now realizing they have to have a team around this.

At the same time, we don't have a lot of that in the like open and research communities at the same scale. I think seeing that converge would be great, but it's still very early days. And another thing on the slide is like some of Anthropic's work, but everyone knows Anthropic is kind of the masters of this, and they have some of their own techniques that we're gonna talk about later on.

But th- that's kind of where we start.

Can we do just a one-second, um-

RL diversion. So you come from a robotics background, which RL used to be, or maybe still is, state-of-the-art, and then now you're seeing a lot of LLM plus RL. So you have the Jim Fan's Eureka, you have, um, Imbibe, which we had on, on the podcast.

They went... They started with RL, now they're doing RL plus LLMs. Um, yeah, any thoughts there on how we got here, like, uh, maybe how the, the pendulum will keep swinging?

I really think RL is about, like, a framing of viewing the world through trial-and-error learning and feedback, and really just one that's focused on thinking about decision-making and inputs in the world, and how inputs have reactions. And in that, a lot of people come from a different back- a lot of different backgrounds, whether it's physics, electrical engineering, mechanical engineering.

There are obviously computer scientists, but compared to other fields of CS, I do think it's a much more diverse background of people. You know, like my background was in electrical engineering and doing robotics and things like that. It, it really just changes the worldview.

I think that reinforcement learning as it was back then, so to say, is really different because it's like you're looking at these toy problems and the numbers are totally different-

Mm-hmm

... and this, like, we, everyone went kind of zero to one at scaling these things up. But like people like Jim Fan and other people that were-- You saw this transition in like the decision transformer and papers, and when people are trying to use transformers to make decision ma- to do decision-making for things like offline RL, and I think that was kind of like the early days.

But then once language models were so proven, it's like everyone is using this tool for their research. I think in the long run it will still settle out where RL will still be a field that people work on just because of these kind of fundamental things that I talked about, that it's just viewing the whole problem formulation different than predicting text really, and so there needs to be that separation.

And the view of RL in language models is pretty contrived already. So it's not, it's not like we're doing real RL. I think the last slide that I have here is like how is a way to make RLHF more like what people would think of with RL, so like actually running things over time.

But it's a weird lineage of tools that happen to get us to where we are, so that's why the name takes up so much space. But it could have gone a lot of different ways.

Cool. We made it one slide be- before going on a tangent.

Yeah, I mean, that's kind of, it's kind of related. This is a-

Yeah, so we have a history of RL.

Yeah, so I recently, uh, this... And give the context, this paper really started 'cause I have this div- div- more diverse background than some computer scientists, which is like trying to understand what the difference of a cost function or a, a reward function and a preference function would be without going into the, all of the details.

Like, costs are normally things that control theorists would work with in these kind of closed domains, and then reinforcement learning has always worked with rewards that's central to the formulation that we'll see. And then the idea was like, okay, we now are at preferences, and each step along the way there's kind of different assumptions that you're making.

We'll get into these, and those assumptions are built on other fields of work. So that's what this slide is going to say, is like RLHF, while directly building on tools from RL in language models, is really implicitly impacted and built on theories and philosophies spanning tons of like human history.

I think we cite Aristotle in this paper, which is fun. It's like going pre, going pre-BC. It's like 2300 years old or something like that. So that's the reason to do this. I think we kind of list some things in the paper about summarizing what different presumptions of RLHF could be.

I think going through these is actually kind of funny. It's fun to talk about these and the, 'cause they're kind of a grab bags of things that you'll see return throughout this podcast that we're talking about it. Like, the core thing of RLHF that in order to b-be a believer in this is that, like, RL actually works.

Mm-hmm.

It's like if you have a reward function, you can optimize it in some way and get a different performance out of it, and you could do this at scale, and you could do this in really complex environments, which is, like I don't know how to do that in all the domains.

Like, I don't know how to exactly make ChatGPT. So it's kind of will overshadow everything. And then there's go from something kind of obvious like that, and then you re-read the Von Neumann-Morgenstern Utility, Mor- Morgenstern Utility Theorem, which is essentially a economic theory that says you can, like, weight different probabilities of different people, which is a theoretical piece of work that is the foundation of utilitarianism, and trying to quantify preferences is crucial to doing any sort of RLHF.

And it, if you look into this, all of these things, there's way more you could go into if you're interested in any of these. This is kind of like grabbing a few random things. And then kind of similar to that is the Bradley-Terry model, which is the fancy name for the pairwise preferences that everyone is doing.

And then all the things that are like, that Anthropic and OpenAI figured out that you can do, which is that you can aggregate preferences from a bunch of different people and different sources, and then when you actually do RLHF, you extract things from that data, and then you train a model that works somehow.

And we don't know. There's a lot of complex links there. But if you want to be a believer in doing this at scale, these are the sorts of things that you have to accept as preconditions for doing RLHF.

Yeah, you have a nice chart of like the sort of intellectual history of RLHF, um, that we'll send people to refer to either in your paper or in the YouTube video for this podcast. Uh, but I like the other slide that you have on like the presumptions that you need to have for R-RLHF to work.

You already mentioned some of those. And, uh, y- I don't know. Is it-- Do you think like any one of them are, are like sort of, um, which, which one's underappreciated? Like, this is the first time I've come across the VNM utility theorem.

Yeah, I know. This is what you get from working with people. Like, like to my co-host on the podcast, the rhetor is a sociologist by training.

Okay.

So he knows all these things and like who the philosophers are that found these different things-

Uh-huh

... like utilitarianism. But- There's a lot that goes into this. Like, essentially, there's even economic theories that, like, there's debate whether or not preferences exist at all, and there's, like, different types of-

Oh, God

... math you can use-

Yeah

... with whether or not you actually can model preferences at all. So it's pretty obvious that RLHF is built on the math that thinks that you can actually model any human preference, but this is the sort of thing that's debate- been debated for a long time.

So all the work that's here is, like, and people hear about in their AI classes, so, like, Jeremy Bentham, like hedonic calcul- hedonic calculus-

Wow

... and all these things. Like, these are the side of work where people assume that preferences can be measured. And this is, like, I don't really know. Like, when you look at-- This is where I kind of go on a rant and I say that in RLHF, calling things a preference model is a little annoying because there's no inductive bria- bias of what a preference is.

It's like if you were to learn a robotic system and you learn a dynamics model, like, hopefully, that actually mirrors the world in some way of the dynamics. But with a preference model, it's like, "Oh, like I don't, I don't know what this model...

Like, I don't know what ChatGPT encodes as any sort of preference or what I would want it to be in a fair way." Anthropic has done more work on trying to write these things down, but even, like, if you look at Claude's constitution, like, that doesn't mean the model believes these things.

It's just trained and to prioritize these things.

Mm-hmm.

And that's kinda what the later points I'm looking at, like, what RLHF is doing and if it's actually, like, a repeatable process in the data and in the training. That's just unknown and we have a long way to go before we understand what this is and the link between preference data and any notion of, like, writing down a specific value.

Did this connection between more, you know, sociology work versus computer work already exist, or is it, like, a recent cross-contamination? Because when we had Tree Dao on the podcast, it's a flash attention came to be because at AC they have so much overlap between systems engineer and, like, a deep learning engineer.

So, like, is it the same in, in this field?

There are a lot of pe-- So I've gone to a couple workshops where these pop- the populations of people who you'd wanna include this, like, are. I think the reason why it's not really talked about is just because the RLHF techniques that people use were built in, like, labs like OpenAI and DeepMind wh- where there are some of these people.

They have, they, they just-- These places do a pretty good job of trying to get these people in the door when you compare them to, like, startups or normal startups. But, like, they're not bringing in a, like, economic ac- like academics from economics, um, like social choice theory.

There's just too much.

Mm-hmm.

Like, the, the criticism of this paper that this is based on is like, "Oh, you're missing these things in RL or at least this decade of RL," and it's like well, like, it would be literally be bigger than the Sutton and Barto book if you were to include everyone.

So it's really hard to include everyone in a, in a principled manner when you're designing this. It's just a good way to understand and improve the communication of what RLHF is and, like, what is a good reward model for society.

It really probably comes down to what an individual wants.

Mm-hmm.

And it'll probably motivate models to move more in that direction and just be a little bit better about the communication, which is a recurring theme in kind of my work, is, like, I just get frustrated when people say things that don't really make sense, especially when it's gonna, like, mani- manipulate individuals' values or manipulate the general view of AI or anything like this.

So that's kind of why RLHF-

Mm-hmm

... is so interesting. It's, it's very vague in its actual, in what it's actually doing, while the problem specification is very general. So reinforcement learning, I kind of mentioned this. It's a trial and error type of system. Um, the diagram in the slides is really this classic thing where you have an agent interacting with an environment.

So it's kind of this agent has some input to the environment, which is called the action. The environment returns a state and a reward, and that repeats over time, and the agent learns based on these states and these rewards that it's seeing, and it should learn a policy that makes the rewards go up.

That seems pretty simple. The, then the, if you try to mentally map what this looks like in language, which is slide seven, is that, like, the language models don't make this easy. I think with a language model, it's very hard to define what an environment is.

So if the language model is the policy and it's generating, it's like the environment should be a human. But setting up the infrastructure to take tens of thousands of prompts and generate them and then show them to a human and collect the human responses and then show that, shove that into your training architecture is very far away from working.

So we don't really have an environment. We just have a reward model that returns a reward, and the state doesn't really exist when you look at it like, um, an RL problem. What happens is the state is a prompt, and then you do a completion, and then you throw it away and you grab a new prompt.

Where really in, like, RL, you would-- as an RL researcher, you would think of this as being like you take a state, you complete, get some completion from it, and then you look at what that is and you keep kind of iterating on it.

And all of that is in here, which is why you'll hear RLHF referred to as a bandits problem, which is kind of like you choose one action and then you watch the dynamics play out.

There's many more debates that you can have in this if you get the right RL people in the room, then kind of like this is an RL even when you zoom into what-

Mm-hmm

... RLHF is doing.

Does this change if you, as you think about, um, chain of thought reasoning and things like that, like does the state become part of the chain that you're going through?

There's work that I mentioned on one slide called process reward models that essentially rewards each step in the chain of thought reasoning, which it doesn't really give the part of interaction, but it does make it a little bit more fine-grained where you can think about, like, calling it at least you have many states from your initial state.

That formulation I don't think people have fully settled on. I think there's a bunch of great work out there, like even OpenAI is releasing a lot of this and let's verify step by step is their, like, pretty great paper on the matter.

I think in the next year that'll probably get kind of made more concrete by the community on, like, if you can easily draw out, like, if chain of thought reasoning is more like RL.

Um, RLHF for decision making. Uh, you have a slide here that compares, uh, pre-deep RL versus deep RL.

Yeah, this is just to say that th-this is getting into the history of things, which is showing that the work that people are using now really came from well outside of N-NLP, and it came before deep learning was big.

And the, the step from this paper, TAMER, which is from two thousand and eight, some names that are still really relevant in kind of human-centric RL, um, Bradley Knox and Peter Stone. Uh, they've-- If you have an agent take an action, you would just have a human give a score from zero to one as a reward rather than having a reward function, and then with that classifier, you can do something with a policy that learns to take actions to maximize that reward.

It's a pretty simple setup. It works in simple domains. And then the reason why this is interesting is you compare it to the paper that everyone knows, which is this Paul Christiano et al., "Deep Reinforcement Learning from Human Preferences" paper, which is where they showed that learning from human preferences, you can solve, like the basic RL tasks at the time, so various control problems and simulation.

And the, this kind of like human preferences approach had higher rewards in some environments than if you just threw RL at the environment that returned a reward. So the preferences thing was you took two trajectories. So in this case, it was tr-- like complete trajectories of the agent, and the human was labeling which one is better.

And you could see how this kind of comes to be like the pairwise preferences that are used today that we'll talk about. And there's also a really kind of interesting nugget that is the trajectory that the humans were labeling over has a lot more information than the RL algorithm would see if you just had one state, which is kind of why people think that it's like why the performance in this paper was so strong.

But I still think that it's surprising that there isn't more like RL work of this style happening now. 'Cause this paper is in twenty seventeen, so it's like six years later, and I haven't seen things that are-

Mm-hmm

... exactly similar, but it's a great paper to understand where stuff that's happening now kind of came from. And that's what the next few slides kind of go into.

Just to, just on the, on the Christiano paper, um, you mentioned the performance being strong. I don't remember, uh, what, what results should I have in mind when I think about that paper?

Um, it's mostly like if you think about a RL learning curve, which is like on the X-axis, you have environment interactions, on the Y-axis, you have performance. You can think about different like ablation studies of between algorithms. So I think they use like A2C, which I don't even remember what that stands for, as their baseline.

But if you do the human preference version on a bunch of environments like the human preference labels, the, i-the agent was able to learn faster than if it just learned from the signal from the environment. Which means like the setup does-- It, it's, it's happening because the reward model has more information than-

Human's a more signal

... than the agent would.

Yeah.

But like the fact that it can do better, I was like, that's pretty surprising to me 'cause RL algorithms are pretty sensitive. So I was like, I was like, okay. Yeah.

Which, it's just one thing I, I do want to establish as a baseline for our listeners. Um, um, like w- we are updating all the weights, right? Like this is, this is, um, i-in some sense, the next token prediction task of training a language model is a form of resor- reinforcement learning, except that it's not from human feedback, it's just, uh, on self-supervised learning from a general corpus.

Yeah.

Um, there's one di-distinction which I love, which is that you can actually give negative feedback, whereas in a, in a general sort of pre-training, uh, situation, you, you cannot. Um, and maybe like the, the order magnitude of feedback, like the Likert scale that you're gonna talk about in, in future slides, um, that actually just gives more signal than, uh, a typical training process would, would do in a language model setting.

Yeah. I don't think I'm the right person to comment exactly, but like there is-- You can make analogies that reinforcement learning is self-supervised learning as well. Like there are a lot of things that'll point to that. I don't know, like whether or not it's a richer signal, I think that's could be seen in the results as there's-- But I think it's it's a good thing for people to look into more, is like try to-- It's like as reinforcement learning is so much less compute, like it is a richer signal in terms of its impact, 'cause if they could do what RLHF is doing at pre-training, they would.

Yeah.

But they don't, they don't know how to have that effect in like a stable manner.

Yeah.

Other-otherwise everyone would do it like that.

So for on a practical basis, like as someone fine-tuning models, I have often wished for negative fine-tuning, which like pretty much doesn't exist in OpenAI land, um, and, uh, it's not the, the default setup in-

How does this work in like diffusion models and stuff? 'Cause you can give negative prompts to something, uh, to like-

Yeah

... stable diffusion or whatever.

That's, that's for, it's for guidance. That's for clip guidance. Um-

Is that just from like how they prompt it then? I don't-- I'm just wondering if we could do something similar. It's another tangent.

Right. Right. Um, anyway, so like, uh, I, I do wanna sort of spell that out for people in case they've haven't made the connection between RLHF and the rest of the training process, they might, they might have some familiarity with-

Yeah

... from our previous episodes.

So like these, these coming slides can really-

Yeah

... dig into this, which is like this twenty eighteen paper that was a position paper from the some of-- bunch of the same authors from the Christiano paper and from the like OpenAI work that everyone knows, which is like, um, some-- they write a position paper on what a preference reward model could do to solve alignment for agents.

Uh, it's kind of based on two assumptions. The first assumption is that we can learn user intentions to a sufficiently high accuracy. That doesn't last with me because I don't, like I don't know what that means. But the second one is pretty telling in the context of RLHF, which is for many tasks we want to solve, evaluation of outcomes is easier than producing the correct behavior.

And this is the whole thing, is like we can compare it to poems that the model generates, and it can be viewed as i-i-improving a po-- like liking a positive example, or it could be viewed as really disliking a negative example.

And that's what I think a lot of people are doing in like the harm space, is like a harmful response to a language model, whether or not you agree with the company's definition of harms, is that it is just, it's a really bad negative example, and they down-weight them by preferring something more benign in the RLHF process, among other ways of dealing with safety.

So this is a good way of saying, it's like this is core This kind of like comparison and positive or negative example is core to all of the RLHF work that has continued.

Yeah. Uh, uh, maybe I'll try to put a more colloquial reph- restatement of this. Uh, people often say, "I don't know what I want, but I'll know it when I see it." This is that expressed in reinforcement learning terms.

Yeah, it is. Yeah, it is. That's, that's what everyone's doing in the preference modeling stage that we'll get to.

Yeah.

Yeah, and you can see there are more papers. This is really just to have all the, um, links for people that go deeper. There's a Zig- Ziegler et al. paper in 2019 which shows that you can do this RLHF process on language models.

This familiar diagram started to emerge in 2019, and it's just to show that this goes really far back. I think we can kind of breeze through some of these. And then 2020 is the first OpenAI experiment that I think caught people's eyes, which was this learning to summarize experiment.

It has this three-step process that we'll go to into more when I kind of go into the main concepts, but this is like the first time you see this diagram that they reuse with Instruct GPT, they reuse with ChatGPT, and the types of examples that they would have, I don't think I need to read these exactly, but one that I have read a whole bunch of times is like they took these prompts from Reddit that was like explain like I'm five or get career advice, and people really poured their heart and soul into these.

So these are like multi-paragraph pieces of writing, and then they essentially do comparisons between a vanilla language model, like I think it was, uh, what's the timeline? Either GPT-2 or GPT-3. I always get the exact years.

Uh, three was early 2020, so that's about right.

Yeah, so this was probably done with GPT-2. Uh, it, it doesn't really matter, but the language model does normal things when you do few-shot, which is like it repeats itself. It doesn't have nice text. And what they did is this was the first time where the language model would generate like pretty nice text from an output.

Mm-hmm.

It was restricted to the summarization domain, but I think that-- I, I like this is where I wish I was paying attention more 'cause I would see the paper, but I didn't know to read language model outputs and kind of understand this qualitative sense of the models very well then because you look at the plots in the papers, these like sum- learning to summarize into Instruct GPT have incredibly pretty plots just with like nicely separated lines with error bars and they're like s- this super far as fine-tuning works, the RL step works.

But if you were early to see like how different the language that was written by these models was, I think you could have been early to like things like ChatGPT and knowing RLHF would matter. But that's now I think-

Obvious. Yeah

The good, the good people know to chat with language models, but not even everyone does this.

Yeah.

Like people are still looking at numbers and I think OpenAI probably figured it out when they were doing this how important that could be, and then they had years to kind of chisel away at that and that's why they're doing so well now.

Yeah, I mean arguably, you know, it's well known that ChatGPT was kind of an accident, that they didn't, they didn't think it would be that big of a deal.

Yeah. So maybe they didn't. Maybe they didn't, but they were getting the proxy that they want, that they needed.

I, I've heard off the record from other labs that it was i- in the air. If, if OpenAI didn't do it, someone else would have done it.

Yeah.

Um, I-- so you've mentioned a couple of other papers that are very seminal to this period, and I, I love how you say way back when in referring to 2019.

It feels like it in my life.

Um, so how, like how much should people understand the relationship between RLHF, instruction tuning, PPO, um, KL divergence, anything like that? Like how would you construct the level of knowledge that people should dive into? Like what, what should people know at the high level and then if people want to dive in deeper, what-- where do they go?

Like, um, is instruct tuning, um, important here? Um, is, is or is that part of the overall process towards modern RLHF?

I, I think for most people, instruction tuning is probably still more important in their day-to-day life. I think instruction tuning works very well. You can write samples by hand so that makes sense. You can get the model to learn from them.

You could do this with very low compute. It's easy to do almost in like no code solutions at this point, and the loss function is really straightforward. And then in-- if you're interested in RLHF, you can kind of learn from it from a different perspective, which is like how the instruction tuning distribution makes it easier for your RLHF model to learn.

There's a lot of details with like depending on your preference data, if it's close to your instruction model or not, if that matters. But that's really at the RLHF stage, so I think it's nice to segment and just kind of understand what your level of investment and goals are.

I think instruction tuning still can do most of the what you want to do, and it's like if you want to think about RLHF, at least before DPO really had taken off at all, it would be like, do you wanna have a team of at least like five people if you're really thinking about doing RLHF?

I think DPO makes it a little bit easier, but that's still really limited to kind of one data set that everyone's using at this point. Like everyone's using this ultra feedback data set and it boosts AlpacaEval, MT-Bench, TruthfulQA, and like the qualitative model a bit.

We don't really know why, and it's like it might just be that data set combined with the mo- ... with the method. But you've got to be ready for a bumpy ride if you're wanting to try to do RLHF.

I don't really recommend most startups to do it unless it's like gonna provide them a clear competitive advantage in their kind of niche.

Yeah.

Because you're not, you're not gonna make your model ChatGPT like better than OpenAI or anything like that. You've got to accept that there's some exploration there and you might get a vein in your specific, like a vein of benefit in your specific domain.

But I'd still like go be careful going into the RLHF can of worms. You probably don't need to.

Okay. Um, so there's a bit of a time skip in what you mentioned. DPO is like a couple months old.

Yeah.

So we'll leave that towards the end.

Yeah, yeah, yeah.

Um, the, the I think the, the main result that I think most people talk about at, at this stage, we're talking about September 2020 and then going into I guess maybe last year, uh, was Vicuna, uh, as, as one of the more in- interesting applications of instruc- instruction tuning that, uh, pushed LLaMA 1- From like let's say a GPT-3-ish model to a GPT 3.5 model in, in pure open source with not a lot of resources.

I think, I mean, they, they, they said something like, you know, they used like under $100 to, to make this.

Yeah. Like instruction tuning can really go a long way. I think the claims of ChatGPT level are long overblown in most of the things in open source. I think it's not to say... Like Vicuna was a huge step, and it's just kind of showing that instruction tuning with the right data will completely change what it feels like-

Yeah

... to talk with your model. And-

Yeah. From text completion to actually chatting back and forth-

Yeah

... multi-turn.

And like yeah, instruction tuning can be multi-turn. Just having a little bit of data that's like a couple turns can go a really long way.

Mm-hmm.

And it's, it's I think people-- That was like the story of the whole first part of the year is like people would be surprised by how far you can take instruction tuning on a small model. I think the things that people see now is like the small models don't really handle nuance as well-

Yeah

... and they could be more repetitive if even if they have really good instruction tuning. But if you take that kind of seven to 70 billion parameter jump like this, the instruction tuning at the bigger model is like robustness, little things make more sense.

But that's still just with instruction tuning and, and scale more than anything else.

Yeah. Makes it. Uh, shall we go to technical overview?

Yeah. This is kind of where we go through my own version of this like three-phase process. We talk about instruction tuning, which we've talked about a lot. It's, it's funny because all these things, instruction tuning has the fewest slides even though it's like the most practical thing for most people.

We could save the debate for like if the big labs still do instruction tuning for later, but that's kind of, that's, that's a coming wave for, for people. And then like preference data and training and then kind of like what does reinforcement learning optimization actually mean?

We talk about these sequentially because you really have to be able to do each of them to be able to do the next one. You need to be able to have a model that's chatty or helpful, instruction following.

Every company has their own word that they like to assign to what instructions mean, and then once you have that you can collect preference data and do some sort of optimization.

When you say word, you mean like, like angle bracket inst, or, or do you mean something else?

Oh, I don't even know what inst means. But just saying like-

Instruction

... they, they, they use their adjective that they like. Like I think Anthropic also like steerable is another one.

I see, I see, I see. Just the, the way they describe it.

Yeah. Yeah, so like instruction tuning, we've covered most of this. It's really about like you should try to adapt your models to specific needs. It makes the models, makes models that were only okay extremely comprehensible. A lot of the times it's where you start to get things like chat templates.

So if you wanna do system prompts, if you wanna ask your model like, um, "Act like a pirate," that's one of the ones I always do which is always funny. But like whatever you like act like a chef, like anything.

This is where those types of things that people really know in language models start to get applied. So it's good as a kind of starting point because this chat template is used in RLHF and all of these things down the line.

But as a basic pointer, it's like once you see this with instruction tuning, you really know it, which is like you take things like Stack Overflow where you have a question and an answer, you format that data really nicely, you push it through the model.

The model then kind of knows what to do when somebody asks a question. There's much more, like there's surely kind of more tricky things that people do, but I still think the vast majority of it is question answer.

It's like, "Please explain this topic to me, generate this thing for me." That hasn't changed that much this year. I think people have just gotten better at kind of scaling up the data that they need.

Yeah, this is where this talk will kind of take a whole left turn into more technical detail land. Um, I put a, a slide with the RLHF objective, which I think is good for people to know. I've started going back to this more, just to kind of understand what is trying to happen here and what type of math people could do.

I think because of this algorithm we've mentioned, this, it's in the air, direct preference optimization. But everything kinda comes from an equation of trying to learn a policy that maximizes the reward. The reward is some learned metric. A lot can be said about what the reward should be subject to some constraint, which the most popular constraint is a KL constraint, which is just a distributional distance.

Essentially in language models, that means if you have a completion from your instruction or RLHF model, you can compare that completion to a base model. And looking at the log probs from the model, which are essentially how likely each token is, you can see a rough calculation of the distance between these two models just as a scalar number.

I think what that actually looks like in code, you can look at it. It'd be like a, a sum of log probs that you get right from the model. It'll look much more simpler than, than it sounds, but it is just to make the optimization kind of stay on tracks.

It's a guardrail that's-

Make sure it doesn't overfit to the-

Yeah. It's like-

... RLHF data

... because we have so little data in RLHF, overfitting is really something that could happen. I think this just, it'll fit to specific features that labelers like to see, that the model likes to generate, um, punctuation, weird tokens like calculator tokens.

Like it could overfit to anything if it's in the data a lot and it happens to be in a specific format, and the KL constraint prevents that. There's not that much documented work on that, but there's a lot of people that know if you take that away, it just doesn't work at all.

So it is important, but it I think it's something that people don't focus on too much. But this objective, as I said, it's just kind of you optimize the reward. The reward is where the human part of this comes in.

We'll talk about that next. And then subject to a constraint, don't, don't change the model too much. The real questions are how do you implement the reward, and then how do you make the reward go up in a meaningful way?

So like a preference model, the task is kind of to design a human reward. I think the key, the equation that most of the stuff is work- based on right now is something called a Bradley-Terry model, which is like a pairwise preference model where you compare two completions and you say which one you like better.

It'll show a Interface Anthropic uses here. And the Bradley-Terry model is really a fancy probability between two selections, and what's happening in the math is that if you look at the prob-- you're looking at the probability that the chosen completion, the one you like better, is actually the better completion over the rejected completion.

And the, what these preference models do is they, um, assume this probability is correlated to reward. So if you just sample from this probability, it'll give you a scalar, and then you use that reward later on to signify, like, what piece of text is, is better.

I think. I don't know. The-- I, I'm kind of inclined to breeze through the math stuff because otherwise it's gonna be not as good to listen to.

No, no, no. The-- Peop- I think people wanna hear it, you know?

Yeah.

I think there's a lot of, like, higher level explanations out there, so.

Yeah. Yeah. So the real thing is you need to assign a scalar reward of how good a response is, and that's not necessarily that easy to understand because, like, if we take back to the, one of the first works I mentioned, this Tamer thing for decision-making, like, people tried that with language models, which is if you have a prompt and a completion and you just have someone rate it from zero to 10, could you then train a reward model on all of these completions in zero to 10 ratings and see if you could actually change, like, get some-- can you get ChatGPT with that?

And the answer is really kinda no. Like, a lot of people tried that. It didn't really work. And then that's why they tried this pairwise preference thing, and it happened to work. And this Bradley-Terry model comes from, like, the '50s.

It's, it's really, it's, it's from these fields that I was mentioning earlier, and it's wild how much this- ... happens. I mean, this, this screenshot I have in the slides is from the DPO paper. I think it might be the appendix, but, like, it's, it's still really around in the literature of what people are doing for RLHF.

Yeah.

So it's a fun one to know.

Uh, I'll point out one presumption that this heavily re-relies on. You, you mentioned this as part of your six presumptions that we covered earlier, which is that you can aggregate these preferences. Um, this is not exactly true among all humans, right?

Like, I have a preference of one thing, you have a preference of a different thing. Um, and actually coming from economics, you mentioned economics earlier. There's a, there's a theorem or a name, uh, for this called Arrow, Arrow impossibility-

Yeah

... which I'm sure you've come across.

Yes. Yeah. It's a, it's a, it's one of the many kind of things we throw around in the paper.

Right. Do, do we just ignore it?

Yeah.

We just, yeah, just aggregate-

Yeah.

Yeah. Okay.

Yeah. Yeah. I think the reason this really is done on a deep level is that you're not actually trying to model any, like, contestable preference in this. Like, you're not trying to go into things that are controversial or anything.

Yeah.

It, it's really the, the notion of preference is trying to stay around, like, correctness and style rather than any meaningful notion of preference, 'cause otherwise-

That's true

... these companies really don't wanna, they don't wanna do this, like, at all. And I think that's just how it is. And it's like if you look at what people actually do. So I have a bunch of slides on the feedback interface, and like-

And they all publish this. Uh, it's always at the ap-appendices of every paper.

Yeah.

It's pretty interesting.

Yeah. There's something later on in this talk which is like, but it's good to mention in this, is when you're doing this preference collection, you write out a very long document-

Yes

... of instructions to people that are collecting this data, and it's like, this is the hierarchy of what we wanna prioritize, something among, like, factuality, helpfulness, honestness, harmlessness. These are all different things. Every company will rank these in different ways, provide extensive examples.

It's like, if you see these two answers, you should select this one and why, and all of this stuff. And then my kind of, like, head-scratching is like, why don't we check if the models actually do these things that we tell the data annotators to collect?

But I think it's because the mo- Like, it's hard to make that attribution, and it'll be really-- It is hard to test if a model is honest and stuff. It would just be nice to understand the kind of causal men-mechanisms as a researcher, like if our goals are met.

But at a simple level, what it boils down to, I have a lot of, a lot more images than I need. It's like you're having a conversation with an AI, something like ChatGPT. You get shown two responses or more in some papers, and then you have to choose which one is better.

I think something you'll hear a lot in this space is something called a Likert scale. Likert is a name. It's a name for probably some research in economics, decision theory, something. But essentially, it's a type of scale where if you have integers from, like, one to eight, um, the middle numbers will represent something close to a tie, and the smallest numbers will represent one model being way better than the other, and the biggest model, the biggest numbers will be, like, the other model is better.

So in the case of one to eight, if you're comparing models A to B, if you return a one if you really liked option A, you return eight if you really liked B, and then, like, a four or a five if they were close.

There's other ways to collect this data. This one's become really popular. We played with it a bit at Hugging Face. It's hard to use. Filling out this preference data is really hard. You have to read, like, multiple paragraphs.

It's not for me. Some people really like it, I hear. I'm like, "I can't imagine sitting there and reading AI-generated text and, like, having to do that for my job." But a lot of these early papers in RLHF have good examples of what was done.

The one I have here is from, uh, Anthropic's, um, collection demo. It's 'cause it was from slides that I did with Anthropic. But you can look up these in the various papers. It, it looks like ChatGPT with two responses, and then you have an option to say which one is better.

It's nothing crazy. The, the infrastructure is almost exactly the same, but they just log which one you, you think is better. I think places like Scale are also really big in this, where a lot of the labeler companies will help control, like, who's doing how many samples.

You have multiple people go over the same sample once, and, like, what happens if there's disagreement? I don't really think this disagreement data is used for anything, but it's good to know, like, what the distribution of prompts is, who's doing it, how many samples you have, controlling the workforce.

All of this is very hard. A last thing to add is that a lot of these companies do collect optional metadata. I think the Anthropic example shows a rating of, like, um, how good was the ta- how good was the prompt or the conversation from good to bad, because things matter, like if you have- There's kind of a quadrant in preference data in my mind, which is you're comparing a good answer to a good answer, which is, like, a really interesting signal, and then there's kind of the option of you're comparing a bad answer to a bad answer, which is, like, you don't like, you don't wanna train your model-

They're both terrible

... on too terrible answers.

Yeah.

It's like this is why we did this at Hugging Face, and it was like our data was like, we, like, don't know if we can use this- ... 'cause a lot of it was just bad answer to bad answer-

Yeah

... 'cause you're, like, rushing to try to do this real contract. And then there's also good answer to bad answer, which I think is probably pretty reasonable to include. You just prefer the good one and move on with your life.

Yeah.

Those are very different scenarios. I think OpenAIs of the world are all in good answer, good answer and have learned to eliminate everything else, but when we- when people try to do this in open source, it's probably, like, what Open Assistant saw is, like, there's just a lot of bad answers in your preference data-

Yeah

... and you're like, "What do I do with this?"

Yeah.

Metadata flags can help. I threw in the, um, slide 28. It's, like, the InstructGPT metadata. You can see how much they collect here and, like, everything from the model fails to actually complete the task, hallucinations, different types of offensive or dangerous content, moral judgment, expresses opinion.

Like, I don't know if exactly if they're doing this now, but you can kind of see why doing RLHF at scale and prioritizing a lot of different endpoints would be hard because these are all things that you-- Like, I, I'd be interested if I was scaling up a big team to do RLHF and, like, what is going into the preference data and what happens.

You do an experiment and you're like, "Okay, we're gonna remove all the data where they said the model hallucinates. Like, does that... And then retrain everything. Like, what does that do?"

Yeah, so hallucination is big, but some of these other metadata categories, uh, w- and I've seen this in a lot of papers, uh, is, like, does it contain sexual content? Does it express a moral judgment? Does it denigrate a protected class?

That kind of stuff, very binary. Um, w- should people try to adjust for this at the RLHF layer or should they put it as a pipeline where they have a classifier as a separate model that is, uh, that grades the model output?

Do you mean for training or, like, a deployment?

Deployment.

I do think that people are doing it at deployment. I think-

Right

... we've seen safety and other things in the RLHF pipeline. Um, like, LLaMA 2 is famous for kind of having this, like, helpfulness and safety reward models. Deep in the Gemini report is something that Gemini re-re- has, like, four things which is, like, helpfulness, factuality, maybe safety, maybe something else.

But places like Anthropic and ChatGPT and Bard almost surely have a classifier after which is like, "Does, is this text good? Is this text bad?" And that's not that surprising. I think we could-- because you could use, like, a hundred times smaller language model-

Yeah

... and do much better at filtering than RLHF. But I do think it's still so deeply intertwined with the motivation of RLHF to be for safety that some of these categories still persist.

Yeah.

I think that's something that'll kind of settle out, I think.

I'm just wondering if it's worth collecting this data for the RLHF purpose if you're not gonna use it in any way because you're just gonna use a separate model to-

Yeah

... sort it .

I don't think OpenAI will collect all of this anymore, but I think for research perspectives it's very-

Yeah

... insightful to know. But it's also expensive, so essentially your preference data scales with how many minutes it takes for you to do each task and every button is like it scales pretty linearly.

Yeah.

So it, it's not cheap stuff to do this.

Can, can we, uh, since, since you mentioned expensiveness, um, and I think you may have joined one of our, one of our spaces back in, uh, when LLaMA 2 was released. We had an estimate from you that was something on the order of LLaMA 2 costs $3 to $6 million to train GPU-wise and then it was something like $20 to $30 million in preference data.

Uh, is, is that-

Yeah

... something that's still in the ballpark? Like, I don't need precise numbers.

I think it's still in the ballpark. I know that there's the $20 million was off by a factor of four because I was converting from a prompt number to a total data point. So essentially-

Ah

... when you do this you have, if you have multi-turn setting, each turn will be one data point and the LLaMA 2 paper reports like 1.5 million data points which could be like 400,000 prompts.

Yeah.

So I would say still say like 6 to 8 million is safe to say that they're spending if not more. They're probably also buying other types of data and/or throwing out data that they don't like. But it's very comparable to compute costs.

Right.

But the compute costs listed in the paper always are way lower because all they have to say is like what does one run cost but they're running tens or hundreds of runs so it's like okay like they-

Yeah, it's kind of a made up number

... it's like a made up number. Yeah.

This is the point.

The data number would be more interesting.

Right. Right, right, right.

What's the depreciation of this data?

Ooh.

You know?

It depends on the method. Like some methods people think that it's more sensitive to the-- This is what I was saying is like does the type of instruction tuning you do matter for RLHF? So like depending on the method some people are trying to figure out if you need to have like what is called like, it's very confusing, it's called like on-policy data which is like your RLHF data is from your instruction model.

I really think people in open source and academics are gonna figure out how to use any preference data and any model just because they're scrappy. But there's been an intuition that to do like PPO well and keep improving the model over time and do like what Meta did and what people think that OpenAI do, does is that you need to collect new preference data to kind of edge, like edge the distribution of capabilities forward.

So there's a dis- a depreciation where like the first batch of data you collect isn't really useful for training the model when you have the fifth batch.

Mm-hmm.

We don't really know but that's something. It's a, it's a good question and I do think that if we had all the LLaMA data we wouldn't know what to do with all of it. Like- ... probably like 20% to 40% would be pretty useful for people but not the whole data set.

Like a lot of it's probably kind of gibberish 'cause they had a lot of data in there.

Yeah.

S- so do you think like the open source community should spend more time figuring out how to reuse the data that we have or like generate more data? I think that's one of the

I think people are kind of locked into using synthetic data, which I wish I had more slides on it, but we could just talk about it. Essentially, people also think that synthetic data is like GPT-4 is more accurate than humans at labeling preferences.

So if you look at these diagrams, like humans are about 60 to 70% in agreement, or like that's what the models get to. And if humans are about 70% agreement or accuracy, like GPT-4 is like 80%. So it is a bit better, which is like in one way of saying it-

Like humans don't even agree with humans-

Yeah

... 50% of the time.

Yeah. So like that's the thing is like the human disagreement or the lack of accuracy should be like a signal. But how do you, how do you incorporate that? It's really tricky to actually do that. I think that people just keep using GPT-4 'cause it's really cheap.

It's one of my like go-to, like I just say this over and over again, is like GPT-4 for data generation, all terms and conditions aside, because we know OpenAI has this stuff, is like very cheap for get it and pretty good data and compared to compute or salary of any engineer or anything.

Right.

So it's like tell people to go crazy generating GPT-4 data if you're willing to take the organizational like cloud of should we be doing this. But I think most people have accepted that you kind of do this, especially individuals.

Like, yeah, they're not gonna come after individuals. I do think more companies should think twice before doing tons of OpenAI outputs also just 'cause it's the data contamination and what it does to your workflow is k- probably hard to control at scale.

Uh, and- and we should just mention, uh, at the time of recording, we've seen the first example of OpenAI enforcing their terms of service. Uh, ByteDance was caught, or reported to be training on GPT-4 data, and they got their, uh, access to OpenAI revoked.

Um, so that was one example.

Yeah.

I don't know if you have a comment on that.

I don't expect OpenAI to go too crazy on this 'cause they're just gonna- there's gonna be so much backlash-

There's- everyone's doing it

... against them.

Yeah.

And like everyone's gonna do it anyways.

Yeah. And- and what's at stake here, to spell it out, is like, okay, let's- let's- let's like cost $10 to collect one data point from a human. Uh, it's gonna cost you like a tenth of a cent-

Yeah

... uh, in, in, with OpenAI, right?

Yeah.

So like it's just orders of magnitude cheaper and therefore people are just gonna do it.

Yeah. And it's like the signal you get from humans is from preferences is not high. The signal that you get from humans for, um, instructions is pretty high-

Yes

... but it is also very expensive. So like the human instructions are definitely like by far and away the best ones out there compared to the synthetic data. But I think like the synthetic preferences are just so much easier to get some sort of signal-

Yeah

... running with, and you can work in other... I think people will start working in other goals there between safety and whatever. But that's, that's something that's taking off, and we'll kind of see that I think in 2024 at some point, people will start doing things like Constitutional AI for preferences-

Yeah

... which will be pretty interesting. I think we saw how long it took RLHF to get started in open source. Uh, instruction tuning was like the only thing that was really happening until maybe like August, really. I think Zephyr was the first model that showed success with RLHF in the public, but that's a long time from everyone knowing that it was something that people are interested in to having any like check mark.

So I accept that and think the same will happen with Constitutional AI. But once people show that you can do it once, they continue to explore.

Yeah. Excellent. Um, just in the domain of, uh, human preference data suppliers, uh, Scale AI very happily will tell you that they, they supplied, uh, all that data for LLaMA2. Um, you, uh, the, the other one is probably interesting, LMSIS from Berkeley.

Um, the- what they're running with Chat Arena is perhaps a good store of human preference data.

Yeah. They released some toxicity data. They, I think, are generally worried about releasing data because they have to process it and make sure everything is safe, and they're a really lightweight org. I think they're trying to, they're trying to release the preference data.

I have-- If we make it to evaluation, I'd pretty much say that Chat Arena is the best limited evaluation that people have to learn how to use language models, and like it's very valuable data and trying to get...

They also may share some data with people that they host models from. So like if your model is hosted there and you pay for the hosting, you can get the prompts because you're pointing the endpoint at it, and that gets pinged to you-

Mm-hmm

... and your, any real LLM inference stack saves the prompts that you get. So like that is some signal. I don't know if the shared preferences. They're tr- I, I do think they're trying to. They're trying to do all the right things.

They're just very strapped, and moving data comes with other like legal and liability concerns in some, some cases.

Awesome.

So kind of looping back a little bit from that very valuable digression on like what preference data is. So we're talking about the actual loss function because it's kind of like this classifier approach that might not make too much sense to people.

You take a language model and you chop it into pieces a little bit at the end so that it outputs one number. It's like, uh, in technical level, it's a logit that corresponds to the probability that we talked about earlier.

But in order to train this, you can't just have like prompt and completions. You need to have these pairs because we talked about scalers don't really work. So in order to train it, you use the magical batching of all language model, all deep learning architectures, and you put in the chosen prompt and the rejected prompt at the same time, and then you end up with two numbers.

And then there's this fun loss function, and you essentially have to increase the difference between these two predicted numbers. It's always fun when you think about like automatic differentiation. It updates the same parameters to, to kind of separate these two numbers at once.

And there's this loss function that you'll see in OpenAI, Anthropic, and everyone's papers. What it looks like is this like some log of a scaler with an exponential that's the difference between these two predicted rewards. It's just some fancy math around a difference, a subtraction between the reward of the rejected prediction and the reward of the, the predicted reward for the rejected completion and the predicted reward of the chosen completion.

Fun fact is that these loss functions look different in Anthropic and OpenAI's papers, but they're just literally just log transformed. So if you start like exponentiating both sides and taking the log of both sides, you'll like converge on one of the two, the both the two papers end up being the same thing.

Nice.

People don't know how to train preference models particularly well now. I think if you zoom into any of the details to look at like the agreement number, so how if you look at a test set, you'll have a chosen and rejected, and you can take the reward model you're training, pass in those completions, and you see if the chosen predicted reward, so the scalar number, is higher than the rejected predicted reward.

And this is the agreement numbers in all of these datasets is like that, where you see they have the sixty-five to seventy-five percent agreement. This just means that like these scalar numbers were ordered correctly, and that's a pretty low number.

It's not gonna get to a hundred percent. That goes to show the kind of like deep questions at play here. People are playing with different loss functions, ensembles, different models to try to address this, but it's really a fundamental issue.

It's like th-it goes back to like what does it mean to do RLHF? And we're not gonna answer that now, but it's good to know that like this sixty-five to seventy-five percent agreement, you'll see these numbers everywhere. It's like we don't have a hundred percent agreement with the reward model and the data, and that's fine.

That's just where we're at. And we essentially take this model and then we start throwing RL, RL at it, I think. PPO, proximal policy optimization, it's pretty complicated compared to what you really need to know. It really just does RL under the hood.

Things like PPO, it learns a value function, and then it uses the value function to update the model. You could look at-- If you actually look at like a feedback diagram, it's more of like a systems problem than an RL problem.

So you'll see things like you need to have two copies of the language model. This is for the KL constraint that we talked about before. You need to have the reward model, which is either a separate reward model or value head on your base model.

And then you need to have your like, um, RL code that actually learns a value function and updates all the parameters. I think it just is really messy to actually set up, but if you dig into it, most people could understand what each of the components are.

And then the hard parts are like, how do we actually make a language model that works out of this? Which is not something that people know that well. I think things that I talk about a lot is just like, okay, like what is the signal flow?

How do you access the reward model? The reward model is used in RLHF exactly what you would think. You have a prompt, the language model generates a completion, and then that completion is given a score. That score gets plugged into the whole RL stuff, and it learns, then it updates the parameters.

That's kind of the core of it. There's a lot of different things like zooming in on where exactly you put this distance penalty between the base model and the RL model. Most people say that you just deduct it from the reward.

So like if you go all the way back to like RL as an agent acting in the world, the reward from that world would be a combination of the reward model in any constraints like KL that you put on it.

There's a lot of different ways to do this because a lot of RL algorithms like PPO actually have a KL constraint built into them, so it's confusing because you hear KL twice, but those are different KLs. One of them is about the text, and one of them is about the value function distance or the policy distance or something like this.

So those are different. It really ends up being kind of like gibberish that I think is less important now because it's more about data and infrastructure than RL details than like value functions and everything. A lot of the papers have different terms in the equations.

I think InstructGPT does something where they like try to get the RL model to match these, the instruction tuning model or the instruction tuning dataset because they were really happy with that dataset to constrain the distribution. LLaMA does some different things, but I think these are all small gains over just getting the deep understanding of the data in the infrastructure set up.

It's kind of-- It's like this is why we say it's like so little RL. It's like now you're getting to the point where you don't even really need this to get a good model. So that's why it's like, okay, the RL is such a small part of the actual like doing RLHF.

Like RLHF is a metaphor for like all language model adaptation, and RL is one tool used at one point in the time. So that's kind of where I wrap up like the core overview in my mind to say like, RL doesn't really do as much as people think, but you could put up flashy equations and do all sorts of stuff if you want to.

It's just like I think it's kind of misleading even because I don't think about those equations on a regular basis.

But what if we called it Q star?

Yeah.

So i-in your mind is the takeaway for this kind of next generation of people working on models maybe the underlying theories is less important than actually getting good data basically?

Yeah, I think it's getting good data. And we'll see, like I have this like advanced topics thing in the slides which it starts with the VALs and then it talks about different way-- a lot of different ways that people are using reward models or constructing training signals really.

And I think that it's like about understanding what your information flow is and like if your reward signal is good and like if your language model is generating right, like zooming in on the tokens it's generating-

Mm-hmm

... and kind of understanding how those things change over time. I think like I have a slide that I, in here I think we can, like this is something we could also talk about in evaluation, but it's really like RLHF is not that shown to improve capabilities yet.

I think one of the fun ones is from the GPT-4 technical report. They essentially listed their kind of bogus evaluations because it's a hilarious table because it's like LSAT, AP exams like- ... and then like AMC 10 and AMC 12 are like kind of reasonable evals in language model land.

But they just showed that like RLHF doesn't improve their evaluation metrics. We don't know if internally they have other ones. They probably do, but from what OpenAI has shown us externally, like RLHF improves some metrics, it decreases some metrics.

No one could really see. I do think they, it, it does things that they care about, but it's like RLHF is not an easy tool to make numbers go up with. It's, it's a powerful tool to change your language model, but like as we've seen with LLaMA and safety RLHF, like that doesn't always mean that people are gonna be happy with those changes or it's gonna do exactly what you want.

It's like-

Well, I think this is intuitive. Like a lot of these quest- a lot of these tests are multiple choice, and RLHF isn't necessarily, uh, intended to improve your multiple choice reasoning capabilities.

Yeah. Yeah.

So-

I, I think that it's like It is reasonable, but I don't think a lot of people have, like, connected the dots there. Oh, okay. And, a-and like what is it in a preference point? Like what if your preference data was between a correct and a wrong answer?

Like it could conceivably do it, but I just don't think that it is remotely what it is actually doing. Yeah. It, it's, uh, much better at being a sommelier, apparently. Yeah. That was the weirdest one that was included into GPT-404.

Yeah, I did. I just see that the last three down there. That's really funny. Can't even taste it. Just can't taste it. It's just like... Anyways. Um, cool. Uh, emerging directions. Yeah. So this is essentially how to use RLHF-like things to make the model better without using PPO, because PPO is kind of a nightmare to scale.

The first thing that I started with is kind of the thing- ideas of rejection sampling and best event sampling. I think best event sampling is what people often encounter first, which is the idea of y-you take a prompt, you generate like 10, 20 responses through it, you pass it through a reward model.

The reward model assigns a scalar for each of them. You pick the one with the highest number, and that's the one you answer the question with. It seems pretty logical to people 'cause it's just spending more inference time compute to make your outputs better, and it works in a lot of things.

This let's verify step-by-step paper that I talked about from OpenAI, they use it. Lots of papers use it. It, it's just kinda like a good thing to know that you can do. You can spend more inference compute based on a preference data set to make your answers better.

The interesting thing that people are confused about more is rejection sampling, 'cause Meta talked about it in LLaMA2. Essentially, a rejection sampling is putting something like best event sampling in a feedback loop, and instead of just returning the best answer to a user, you take the best few answers, and then you apply instruction tuning on that data set.

And then you, like, you do the instruction tuning, and then you can collect more preference data, do a new reward model, and then you rank some new outputs, and you do instruction tuning again. So essentially, like LLaMA started their RLHF process with this to get some signal out of preference data.

That preference data went into a reward model, and then the reward model did a good enough ranking that it was like essentially super-powered instruction tuning based on rewards. Works very well, much easier to implement than PPO- Mm-hmm ...

'cause you can use it in all of your kind of like it's still instruction tuning, so it's the same autoregressive loss. It's easy to plug into things like transformers and stuff like that. A lot easier to start with than whatever freaking mess doing RL at scale is gonna be.

So that's one. A quick nod that offline RL is something that people talk about for RLHF, essentially because your model doesn't have to generate. In that case, you just kinda look at, you look at data, and it backpropagates through your reward model directly.

So in PPO, you have the step of like needing to generate everything and passing it through the reward model. How offline RL essentially works is that all of this is kind of just done in one big data set where it's just like, I'm not an expert in this, but essentially you need- you do much less inference costs during the RLHF process if you do offline RL.

There's a few pap-papers that people have published. Not a lot of traction. I think it could take off. Some people that I know in the RLHF area really think a lot of people are doing this in industry just because it makes the kind of training process simpler in the number of things you have to have running.

Um, different feedback types are probably gonna, um, come into play. There's papers like wr-written feedback or labeling multiple scores or multiple pairwise preferences for every completion. That's coming. It's also kind of related to what we mentioned in process reward models, where you're get- you're labeling each step in the chain of thought reasoning just to kinda make the problem more specific.

It seems very likely that different feedback will be used for different domains. Chain of thought reasoning is great for math, and that's where these process reward models are being designed. Probably not great for things like poetry, but as any tool gets better, it gets more specific.

Then kind of get into more of a talking point, which I think is fun. The next one I have is Constitutional AI. I think this is something that people really don't kn- like, I think just kind of misunderstood.

I mean, like I think most people thought that Constitutional AI was doing something where it's like created the preference data based on the specific principles in some way, where it's like, I, like I don't-- Like what did you two think of Constitutional AI?

Yeah, I'll, I'll be the, you know, dumb person and you correct me. Um, a-as far as I understood, Anthropic came out and said that the best way of doing, uh, of generating this sort of preference data or alignment is give a second model a constitution to evaluate, uh, the, the, the first model's outputs.

Yeah. And first-- The, the, the constitution is unspecified, but like this is draws from like the UN Declaration of Human Rights and, and, and the Apple Terms of Service for some reason. Yeah, and this leads into the question is like what is the other model evaluating and like how is it evaluating in a way that you can train on?

And that's what I mean. It's like people didn't think about this. A lot of the CAI paper was actually talking about instruction tuning, which is if you have an instruction, you then have a language model that critiques the instruction based on principles, and then your instruction responses are closer to the constitutional principles.

This was the first half, which is like they have some acronym for all of this. The, the diagram in their paper is wild in this one. I think their papers are sometimes pretty funny 'cause they're, they're not capabilities papers.

They're like alignment papers. So like they don't make everything super clear. So the first half of Constitutional AI is fine-tuning your com- instructions based on principles. So that's one half, and then the second half is what people really thought that they knew, which is like how do you use these, this other model to provide a critique based on principles?

And in the paper, they list essentially they like say what their prompt was, which is like for the synthetic feedback for generating new preferences, which is essentially like pick between these two answers based on this principle. So they're kind of sampling from the principles in their constitution and from kind of A, B, like two options of completions, and then- The AI model is essentially given the context of a certain principle to, to pick the A or B preference, and then that's a new preference dataset.

It's just the two completions without the context of the principles. So with this kind of like sampling idea, they're sampling from like 30 principles and a wide dataset of two candidate completions across different prompts. So it's like, to me, it's a very like loose-- like the values are not explicit in this.

It's just kind of how they're guided and let's say it's very, it's a very machine learning-y approach because it is relying on averages and scale to get the principles in there. But it is way less explicit than I thought it was gonna be.

I kind of thought there was this like feedback thing in the preference data where it like checked to see if the principles were satisfied or anything like this. But it is, it's really just like a modification to the RLHF setup that we've talked about with instruction tuning and preference data collection, where there's an AI model providing critiques, and a lot of those critiques are based on like sampling of constitutional values.

So it, it, it almost sounds more tractable in that way, but I, I would also guess while I just like say like, "Oh, look, I figured it out," I'm guessing they've done s- they do different things than they said in the paper.

Like this paper is in around 2022. It's a pretty old paper, or, and like they're surely doing more. But it's good to know like where they started at least-

Yeah

... in this, in this case.

I thought the communication around, um, the Pareto optimal improvement was helpful i-in understanding that, uh, you do actually want, uh, it to be more helpful and honest w-while maintaining the same level of harmlessness or something like that.

Yeah.

Right?

Yeah. So they h- that, that figure right at the top of the Constitutional AI paper is worth seeing if you don't have it immediately pop into your head.

Yeah.

Where they essentially compare like Constitutional AI to other RLHF that they're doing internally at different... And something that most RLHF papers don't do is like they have little dots on the lines to indicate intermediate checkpoints. It'd be really great to see more RLHF papers kind of showing like how per epoch or per half epoch of training, because most RLHF is only a few epochs, at least-

Mm-hmm

... in the open models, like what is happening there.

Yeah.

People release checkpoints, but like that's how we should be thinking about it because the optimizer is so strong, and it's like we don't know what's happening-

Yeah

... in this kind of intermediate land.

Uh, I don't know if this, uh, is a relevant comparison for you, but, uh, OpenAI also re-recently released a weak-to-strong generalization paper where they actually talked about a few checkpoints, intermediate checkpoints for GPT-4. Uh, any comments on the comparison between Constitutional AI and weak-to-strong generalization?

I didn't see the paper. I think that I saw people criticizing it for like just being like safety washing from the fact that they're like talking about GPT-2 still, which is such a kind of like odd model to focus on.

I didn't really look at the paper.

Had it lying around. I mean

So it's like I think that it's the thing with OpenAI, it's like they're sharing less than they know.

Mm-hmm.

So I think they probably have things that are pretty cool that they're doing-

Yeah

... internally and-

Yeah. So I'll, I'll summarize for listeners-

Yeah

... who may not have seen the paper because, you know, it's impossible to keep up on everything. Um, I do think that what Constituti-Constitutional AI and RLAIF represents is that we are starting to come to a point where it's just impossible for manual human preference data collection to scale.

Uh, and the only way to scale this is to trust our AI overlords to , to model our human preferences. And Constitutional AI was the first version of this. What the second version or what weak-to-strong is, is that anticipating a future of super intelligence or s- or, or the need for super alignment where the, the thing that we're trying to control is smarter than us.

Um, so you take GPT-2 and try to use GPT-4 to teach it to be smarter than, than itself because, uh, this is what we're gonna have to do in the future as well when we are not, we're no longer fully in control.

Is, are, are we the metaphorical GPT-2 or is-

No.

Okay.

We're, we're like not even in the process anymore-

Okay. Okay

... at, at, at the point of super intelligence. Um, so they're, they're just basically s- they're prepping. They're, they're preppers, and they're -

Yeah

... and they're saying this will happen, and humans will be like so far out, like in the dust that we just like have no o- no, uh, no say in this debate. How do we still control systems then?

Um, and weak-to-strong generalization seems to be the answer. And I see a lineage from Constitution- Constitutional AI to this thing.

Y- yeah, the Constitutional AI and the super alignment is like very conceptually linked.

Yeah.

It's like a group of people that has like a very similar intellectual upbringing, and they worked together for a long time-

Yeah

... like coming to the same conclusions in different ways.

Right.

And I understand the argument, and I-

Yeah

... I mostly just don't. I think they're just waiting to see more from the super alignment team because I just didn't really put it together in my brain quickly looking at weak-to-strong generalization of like exactly how it all fits, but I'm also not a safety researcher.

Yeah.

But I think that could be feedback for them. It's like, like what is, like how do, like I understand what synthetic data means in all of this. It's like how could they communicate that a little bit more specifically in this context?

Yeah.

'Cause like I wanna know what they think about that.

Which is why I like that Pareto optimal thing because it, it links, it, it's takes, steers the debate away from X risk to like, no, like this makes language models more useful. And, and we can all get behind that.

Yeah. Yeah, yeah. I agree. I think the last kind of emerging direction that I have might just be like this debate that you can control how long we talk about this, which is about direct p- preference optimization.

DPO.

Um, you could go read my blog post on this. I had tried to summarize this already, but, um, essentially DPO is a different class of algorithms. I still call it RLHF because RLHF is so vague in how it's defined.

I think DPO is closer to RLHF than RLHF is to RL. You can unpack that if you need to need to. But what DPO is doing is essentially deriving a optimal reward function from the preference data, where the preference data is the same thing that we've talked about, and then the clever math in the paper, um- Emerges a implicit, like, the optimal policy to that based on an implicit reward function that's a ratio of, like, log probs.

It's very odd. Like, the difference between what a DPO reward is and a classifier reward is very different, where, like, the classifier is trained to output a scalar value based on this kind of like contrastive like loss, where DPO is purely based on, like, the difference between two log prob ratios.

So the reward there is the ratio between, like, the policy generation likelihood and the base model generation likelihood. I don't have intuitions for what that means yet, but, like, what the reward actually is is very different. The data starting point in principle could be the same, and I think, like, we've seen a lot of successes in open source with it.

It's way simpler to implement and to work with in that regard, which is why I think we'll keep seeing a lot of success with it in the short term. I think there'll be a lot more-- They'll-- We'll keep seeing DPO models for the time being, but we won't really answer, like, what the fundamental differences are because it, like, depends on your data.

It depends on your infrastructure. Rumors seem to be that people still think that PPO-like methods or other RL methods have a, like, higher top end. But I don't necessarily think, like-

What-- Sorry, what is top end?

Just, like, the absolute best model you could get.

I see.

So, like-

I see

... Google and OpenAI-

Yeah

... aren't using DPO because they could do something more complicated, but, like, that's not what academics and open source people really care about. They care about, like, being able to improve on their methods and understand where to, like, iterate the models and kind of work off of each other.

So, like, I, in a lot of ways, I think DPO still will be what people see, but, like, in some ways, it's probably, like, slightly more constrained. There's other ways that you could think of PPO, like, working nicely in code, where it's like if your code runs is the score that you give it, and it's-- you have to generate, like, you have to kind of do canned things to get DPO to have the same data.

So there are specific cases where, like, the DPO formulation is a little bit harder, but I expect to see more DPO models than anything else in the next six months. That's probably, like, what most people need to know unless they're an RLHF expert.

And, like-

Mm-hmm

... I would love to learn more about PPO and a lot of the authors in this space from the DPO authors who are great to talk to. You can reach out to all three of them. Uh, so-

So as of time of recording, we actually about to publish our newest recap where we talk to the authors.

Yeah. So, like-

Uh, so for people who are listening to this in the future, you can refer to that episode.

Yeah. So, like, Raphael, Eric, and Archit, I've talked to all of them at good length, and they're all fantastic. And it's like-

Yeah

... they all say similar things, and they'll also defend their method because it's an awesome paper. Like, if you want to learn how, like, a good math-- like, a kind of mathy but still experimental paper in language models is, like, the DPO paper-

Yeah

... is a really good one to spend more time on. Yeah.

Well, when I asked them questions about it, they just kind of gestured at the p- poster and said, "Look at the equation. Just stare at it, and you'll see it."

Yeah, that's my, that's my criticism for them is like they-

Like, what?

Yeah, they're a little-- They're still in the academic world where some of their answers reflect that, but I've done it enough with them that I understand what they're saying.

Yeah, yeah. Uh, I will say, like, it does remind me of flash attention a little bit in the sense that, like, it, it's, like, kind of an equivalent, uh, thing to the thing it's replacing, and it's just faster, cheaper, just better in every way.

It's a very different optimization tool.

Yeah.

There's-- Essentially, the thing in my mind that I can't get past is the difference between the control you get in training a reward model and then training a policy because essentially everything you want your reward model to do might not be everything that you train the policy to do in the RLHF step where you have, like, these two different prompt distributions.

Mm-hmm.

But with DPO, you're doing both at once, so you don't control that.

Mm-hmm.

And we don't know if you have fancy engineering ac- i- like, if you have fancy engineering abstractions and, like, test your reward model to do different things if that separation is really important. And I think that's what-- where this, like, benefit at the absolute biggest scale and most investment could come from.

But you, like, can't-- DPO is one update. Like, it is one model. You can't separate that. So, like, that's a thing to know. Probably doesn't matter for most people, but it, it is very different. And, like, I was asking somebody who was on some of those earlier OpenAI papers that's not at OpenAI anymore, and they were like, "I wish we had thought of that."

So, like, it is a really cool idea, and, like, that's the type of thing that academia still can do and-

Mm-hmm

... and can do really well and hopefully continues to do. Yeah.

One thing I wanted to make sure I cover before, uh, we leave this topic. Yeah, DPO, you know, o- one of the DPO models that, uh, were trained, uh, apart from Zephyr and Mixtral, which is, uh, two of the more high-profile ones, is Tulu from the Allen Institute.

Uh, and you're one, uh, you're one of the few people maybe placed to explain-

Yo, this is so funny.

... what, like, maybe, maybe, like, what's Allen Institute doing here-

Yeah

... and, like, you know, what's g- what's the backstory?

Yeah. So the Allen Institute for AI is, I think, the 10-year birthday is in January. There's a special-

Mm-hmm

... event for that.

And also, like, people should know this is Paul Allen from Microsoft.

Yeah, yeah. Paul Allen just owns everything in Seattle. Not li- not literally. I mean, his, his, his passed and his estate is still operating in a lot of great ways. But the Allen Institute is mostly known as being, like, a super academic lab where they have more resources than academia and publish, like, hit after hit of research paper, and they're trying to move more in the direction of releasing models.

And this is part of why I joined. It's, like, talking with the CE- new, new CEO, Ali Farhadi. I don't know if I pronounced the last name right. But he's trying to move from an org that does papers only to something that does papers, releases models, is active in policy, maybe is, like, helping work with these for-profit institutions that don't have a middle, like, an established place where they could all go through to do things.

So they're really trying to expand the scope. It's part of why I joined, and, like, the Tulu 2 model is the kind of thing I've joined, and they were talking about this, and I was like, "Okay, we should just train it and release it because no one has done this direct preference optimization at a scale of, like, a really, like, seventy billion parameter scale."

And this experiment is hilarious. This is, like, classic of, like, everything kind of works right now in ML. Like, I showed up and the grad student, Hamish- Iveson and my, I need to learn how to pronounce last names better, but he had some JAX DPO code built on this EasyLM framework, and we have, like, have D- TPUs that we could access for research purposes.

So it's like, okay, we have a huge TPU. It's like, let's just try the Zephyr recipe on 70 billion parameters, and it's literally, like, the first run. It's like we did no ablations, didn't change any parameters , we just copied them all over, and like, that's the model that people have been working with, is like, that goes to show that there's a lot of runway and understanding and improving on this.

It's like we took the same data and just took it to a different JAX implementation and scaled it up 10X, and it still returned a model that was pretty good. It's like on benchmarks and in people using it.

So let's say it's like 2024-

Mm-hmm

... we'll be busy in the space as we do, like, we're running data ablations to try to understand what's best. Then Allen Institute is training, pre-training language models or pre-training, like, open language models where we'll be able to share, like, data, code, everything, the kind of horn that everyone likes to get annoyed about these days.

It's like, "Llama not releasing data." So that'll come in the new year, and then things like Tululu 2 are the recipes that we will apply to that. And it'll- we'll kind of keep doing both. As the pre-trained models get better, those will probably become more of a priority, but, like, starting pre-training is very hard, so it's like you still wanna, you still wanna l- learn from Llama 2 and Llama 3.

So that's fun. Uh, I think DPO releases are kind of becoming expected because Mistral, Mistral released a DPO model as well. I think the slide after this is just like, there's a ton. It's like Intel releases DPO models, Stability releases DPO models.

At some point you just have to accept that that's where we're going, whether or not you care about the whole, like, DPO debate. And that's why I find it so funny because there's really interesting, like, debatable questions between DPO and other RL methods, but we just won't have the answer, and it'll look like there isn't a debate because everything that is published is with DPO, but that doesn't mean that anything is answered in the time being.

Yeah, uh, the kind of last of the stuff is evaluation, and these slides were prepared kind of last minute, but I think the question is how do you evaluate these models and what you should be doing. I think the PSA is like, don't trust your numbers and actually talk to models.

It's very hard to do if you're an engineer or a researcher 'cause you have your specific thing that you're zoomed in on, and it feels like a waste of time to just go play with ChatGPT or go play with Chat Arena, but I really don't think it is.

It's something that I... This is like me telling myself what I should be doing. But there's the question of, like, is the Hugging Face leaderboard good for open source? A- and then what else can people do? The Hugging Face leaderboard came out of the team that I was on there.

We were trying to build a framework to automatically evaluate the models that we were training and the models that people were releasing, and then have them in a central place where it could be like, "Look, here's the evaluation score.

Is this what we're competing with?" It obviously blew up. I think it's very good for companies trying to operate in the open LLM space to build businesses around it. I think it's bad for people building LLMs that they think are the best because it's easy to overfit if you're training and focusing them as, on them as a developer.

But it's, it's good to have distribution of models when there's so many people training them. The- but it's like now it has six evaluation tools. I can't even name all of them off the top of my head. It's like ARC, HellaSwag, MMLU.

There was Drop on it at one point, but they dropped Drop, which was pretty funny. TruthfulQA, and then I think maybe-

MT-bench

... some other math. I don't, I don't know. But in-

Is, is there... So, um, th- this benchmark question is, uh, is something that everyone's talking about because there's a lot of gaming that it seems to be going on. Is there some discussion about sort of held-out, um, benchmarks that Hugging Face could hold onto?

Mostly it's who's gonna pay for it. 'Cause we've talked, but we have this, uh, we're thinking about this at Allen AI too, is like improving on, we're specifically thinking about improving on Opaco Eval, which is-

Who's gonna pay for running the evals?

Who's gonna pay for running the evals?

Right now-

And what models

... Hugging Face is just running every eval every day.

Yeah. So they have like 1,000 GPUs. At one point they were gonna do more training. It was gonna be used for that. But now they have less training, and they do, they run a good amount of GPUs.

Yeah.

In one of their blog posts they said how much compute it was. I don't think it's a ton to run these, but it is a, like-

I mean-

You have to have hundreds of GPUs to maintain this leaderboard.

Uh, sh- so, uh, one, one technical question. Uh, like some of these are open source models that they don't change, so you just have to run them once.

Yeah.

Okay. All right. So that, that, it's not that crazy, I don't think.

No.

You see-

It's tractable for-

It's only the closed source models that need to be re-evaluated.

Yeah, so if you look at the, like, Chat Arena, they take specific dates.

Yeah.

So then there's this whole controversy of like is GPT-

Mm

... is ChatGPT from March better than ChatGPT from June?

Yeah.

So like in the, on like one of these future slides, it's slide 58, is like the, um, Chatbot Arena leaderboard if you're looking later, which Chatbot Arena is this thing from LLMsys that we were looking at, and then like on the X-axis is models, and you can see the GPT-4 from March has a higher score and the same...

Uh, it's like this is not a perfect comparison, but there are signs that are pretty funny there that like there are things-

Oh. Yeah

... there are things cooking and but you don't know who's collecting this data, what prompts they're doing, and like what the... But it's like-

Okay

... it's such a funny timeline.

So, so for those listening, uh, GPT-4 March 14th is 40 ELO points higher than GPT-4 June 13th.

Yeah, it's like outside of the error bars-

That's pretty high

... on the LLMsys thing.

Yeah, that's pretty high.

And, and the other piece of context is that GPT-4 Turbo is also notably ahead of the other GPT-4s, which-

Yeah, that's weird

... it kind of showed up immediately once they added it to the leaderboard-

Yeah, that's weird

... or to the arena, and I was like, that, all the GPT-4.5 memes aside, it seems like this is effectively a bump in the model. If it's that-

Yeah

... if it's like clear for... If you zoom into this, the leaderboard- Is very close for many s- like strata of models. So there are levels where you can get your model to, and it'll be really close to your peers.

So in the open source, there's things like, um, like Mixtral Instruct, Tool 2.70 B, which is effectively a hu- it's a way bigger model than Mixtral. Give mix- mix- Mixtral's the mixture of expert model. Like I'll due credit, it's a very good model, and that's gonna be like the next level once people get better at fine-tuning it.

Like Yi-34B Chat, like this is one level, and then there was like a level with like the Alpacas and the Vicunas. But all of these open source models, there's then another step up to GPT-4, and then there's another step up to GPT-4 Turbo.

So it's like the difference from these tur- the GPT-4 Turbo to like the GPT-4 that was first released is bigger than the difference from Toolu2 to GPT-4. So that's just like there's something good going on there, and I was like, "Okay, that's a new model by my standards, but they're not gonna tell us about it."

Hmm. Yeah.

Like they did in DevDay. They said, "It's our new model," but they weren't like, "This is our-

Mm

... new best performing model."

Yeah.

Because it's like the benchmark scores are probably the same, but they made it so that people like using it more.

There's some hints that 4.5 might drop at some point. Uh, we don't actually know how true those things are, but I don't know.

I don't think it really matters. It's like they could call anything. They, they're retraining these models-

Yeah, it's just like-

And they could call any of them-

Marketing label

... GPT-4.5.

Yeah.

I think the two tools that I talk about most in research domains on RLHF is like AlpacaEval and MT-Bench.

Okay.

They're two academic-maintained leaderboards for evaluating chat capabilities. Evaluating chat is really hard, and what they both do is they have GPT-4 provide some sort of feedback. MT-Bench is called MT for multi-turn, and they have a prompt and a follow-up question.

So what they do is they ask GPT-4 to score both the, um, both the like the initial response and the second response and provide the average. Kind of given up on following the slides. This is all on the slides if you look for it.

And then AlpacaEval is a little bit different, where you're comparing a candidate model, so the model we've trained. So like when we're training Toolu, we compare the, we submit this, and what it's doing under the hood is comparing the new model to DaVinci-003, which is one of, um, OpenAI's older instruction models, and calculating the win rate that GPT-4 sees between the new model and DaVinci.

So that's kind of like it, it has many more prompts than MT-Bench. MT-Bench has custom prompts that they made to just kind of like take a stance on what is a good chat model. AlpacaEval sources theirs from Self-Instruct, which is a popular paper from AI2, Open Assistant, Vicuna, Koala, Anthropic's helpful harmful list.

So like An- AlpacaEval is from sources that people know and love. MT-Bench is its own thing. We were more focused on MT-Bench at Hugging Face. At AI2, we're a little bit more focused on AlpacaEval, but it really can go either way.

These are kind of like table stakes to saying that you have a good RLHF model is like you should be able to have a pretty good score on both of these, and then the kind of proof is in people actually talking to it.

So I think like the Zephyr model from Hugging Face was a kind of step change in people's percei- perception of open models that got integrated into a bunch of products within a few weeks. Like You as You.com was experimenting with it, and, uh, someone else, like I saw some Substacker was using it as like a writing feedback bot as instead of ChatGPT.

Uh, but like that's what happens when a good open release is, is there now. It's like it, the evaluations are good, and people pick it up, and the evaluations are just enough to like say like, "Okay, we're in the right ballpark," but you never really know if the model is the one or one of these big ones without talking to it.

So it's like no- however much you talk about evals, that's still where we're at, that you can't, you can't prove anything definitively. And Google seeing that, and like until Gemini Ultra comes out, like we don't know. It's probably a great model, but we, we don't know-

We don't know

... what they have.

Yeah. Gemini Pro didn't do so great on the o- other stuff too.

Yeah. I wanna know if Gemini Pro is just like some intermediate checkpoint-

Yeah, we don't know if it's exactly comparable

... or if it was like a major deliverable for them or not.

Yeah. Yeah, we don't know.

Which if it wasn't a major deliverable, it's probably a strategy headache for Google- ... but that's not my problem.

Um, you have a bunch of open questions here. One of our lightning round questions always was the-

Yeah, we'll just do inverted lightning round?

Yeah, yeah, exactly. It's like-

Uh, you asked the people open questions. Oh, I mean, there, there's so much to do here. They're kind of like summarization of things that will be hinted at in the, in the talk to this point, which is like I split it up in my work between like data training and model, which is essentially like how do we evaluate what's happening at the model level with RLHF.

I think big labs are so over index... are indexed on their own base models, so they don't know like what swapping between Claude Base or GPT-4 Base, how that would change any notion of preference or what you do with RLHF.

I think in the open we could do that. We could swap between LLaMA2 and Mixtral and kind of see like does RLHF work the same for both of those. Do they both get AlpacaEval bumps when you use the same dataset and the same framework down the line?

That'd be good to know if like how sensitive RLHF is. On the data, we talk a lot about aggregation. On the research side, there's a lot of interesting things as like does getting your data from Scale or a Discord army change the quality of the data based on like professional contexts?

And like-

The results of this might really affect Scale.

Yeah. They probably should do it internally. They should do like internal market analysis-

Yeah

... on that li- on that line.

Oh, oh, we should also mention, uh, there has been a report that a lot of these labelers use ChatGPT to do their work.

Yeah. I mean, I'm not surprised. So it's like, it's, it's a lot of messy grounds in RL these days. And then there's more training questions, which is like what happens at the end of the day. I mentioned what I call a qualitative alignment e- earlier on, which is like do the models get better in ways matching the preference data preferences.

So if you like collect two batches of preference data With different priorities, like what are the, what is the downstream model change? I don't know if it does anything. Should all data be equal? Like, if you have, like, healthcare questions, should it be the exact same as, like, write me a joke?

Like-

Mm-hmm

... this is all implicit to deep learning. Like, deep learning just scales and aggregates and, like, I think we are gonna be on that ride, but it's not necessarily what some people would call fair or good. And then the kind of last slide that I have is fun, which is just, like, John Schulman talks about this in his ICML talk.

His ICML talk on proxy objectives for RLHF is public now. They pub- made it public, like, three months after the conference or some-

Oh

... weird timeline. But he talks about things like ChatGPT being verbose and have self-doubt, refusals, things that are really, like, in vogue in the conversation right now, and, like, how those can emerge in the process of continually trying to adjust the RLHF process based on what users are seeing in the model.

And this is, like, a sort of outer loop optimization that no one in the open is even remotely qualified to talk about, but OpenAI does monitor, and they'll, like, rerun RLHF and train a new reward model with a mixture of their curated data and user prompts to try to make it work better over time.

And, like, that's the different model versions. And while there's a lot of critiques about this, they're definitely, like, intentional in trying to fix. I feel like it's probably Whac-A-Mole, where they're like, "Oh, there's this problem. We have the data.

We can fix this." And then it, like, pops up some new problem after doing RLHF, and they, they're studying this, and if you could really figure it out, this is where things start to look more like RL. You could automate it.

Things are just, like, longer timeframe of optimizing the model. It would be cool, but we're-- I feel like I'm years away from ever actually working on this, but we can try to get details from people who are.

Yeah. Excellent.

Awesome. Uh, yeah. Anything else that, that we missed? I think we covered, um, a lot of it.

I mean, I'm good. I would ask you guys about if you know companies that are doing this and things. Like, I know some that are in the, like, the RLHF as a service space will become busy, I think for good reason, just because, like-

There's companies doing RLIF as a service?

Yeah, both of them are. It depends if synthetic data is gonna win over human data. If human data is the real winning feature in the end, like, it's a big capital investment, so it kinda makes sense as a VC model anyways.

But there's gonna be both of them for a while.

Yeah.

It'll be cool.

Yeah.

Yeah. Do you see a lot of people, because I know Luis Castrejón is starting a company. Is there a lot of ambition in this field to start companies, or is this more such a research-driven part of the stack that maybe it just stays there and-

There definitely is, 'cause I know my, my former colleague, Naznin Rajani-

Mm-hmm

... from Hugging Face is also starting a company in this space. The peop- the, the Falcon team who left Hugging Face I think is also working in this space.

Oh.

I don't, I don't really know. I d- like, I don't know exactly what... I haven't talked to them since, like, ICML, so I don't know what they're doing. Startups change a lot, but there are definitely a lot of people looking at this, this space.

I mean, Scale's probably trying to do it.

Mm-hmm.

If I was Scale, they would wanna do it. I think they've historically had trouble keeping, like, technical ML talent, but they've started a new research lab, so that should help. It's, it's, it's a busy area.

Cool.

Yeah. Lots going on.

Um, yeah. Awesome, Nathan. Thank you-

That was a masterclass.

Yeah.

I think this is the first 201 that we've ever had, and you set the bar very high.

Thank you.

All right. Bye, everyone.

Bye.

Bye-bye.