# [Paper Club] Who Validates the Validators? Aligning LLM-Judges with Humans (w/ Eugene Yan) Latent Space · 2024-09-28 Eugene Yan presents the paper 'Who Validates the Validators? Aligning LLM-Judges with Humans' by Shreya Shankar, introducing EvalGen, a framework that helps developers iteratively align LLM evaluators with human labels by grading outputs, refining criteria, and tracking coverage (recall) and false failure rate (1−precision). Yan demonstrates his own prototype 'Label', which forces users to label 20 samples before unlocking evaluation mode and 50 samples plus one evaluation run before optimization, gamifying the alignment loop. The discussion covers binary vs. pairwise evaluation, the subjectivity of criteria, and Shreya Shankar's plans to integrate natural language feedback into EvalGen's next version. Yan advocates that labeling data is essential before deploying LLM evaluators in production, turning the process into prompt fine-tuning rather than vibe checks. ## Questions this episode answers ### What is the chicken-and-egg problem when creating criteria for LLM evaluators? Eugene Yan explains that to grade outputs you need evaluation criteria, but the process of grading helps define that criteria. Creating criteria without seeing data often leads to expectations divorced from reality, like wanting no toxicity or slang the LLM can't produce. The paper argues you must examine LLM outputs before finalizing criteria, and iteratively refine both. [2:55](https://addtry.com/d14a99a3-b457-456b-addb-80177bf0fada?t=175000) ### How does EvalGen measure alignment between an LLM evaluator and human judgment? EvalGen uses two metrics: coverage (recall of catching bad responses) and false failure rate (one minus precision, i.e., how many innocents are wrongly failed). It iteratively refines candidate criteria—either code assertions or LLM prompts—by having users label samples as good/bad, then checking how well those criteria align with the labels. The alignment report card helps decide when the evaluator is usable. [8:54](https://addtry.com/d14a99a3-b457-456b-addb-80177bf0fada?t=534000) ### What tool did Eugene Yan demo that forces users to label data before improving LLM evaluators? Eugene built a prototype called 'Label: LLM Assistants for Better Evaluation Labels.' It requires users to label at least 20 rows of data to unlock evaluation mode, where they can run an LLM evaluator and see alignment metrics like recall and precision. After labeling 50 rows and running evaluation, optimization mode unlocks to automatically improve the evaluation prompt. This enforces looking at data before iterating on criteria, tightening the feedback loop. [30:33](https://addtry.com/d14a99a3-b457-456b-addb-80177bf0fada?t=1833000) ## Key moments - **[0:00] Intro** - **[1:47] EvalGen** - [4:09] It is impossible to determine good evaluation criteria without first looking at LLM outputs. - [5:54] How to derive the ideal evaluator prompt from 100–400 golden samples remains an unsolved problem. - [7:30] EvalGen offers three initial ways to create an evaluator: infer criteria from the prompt, write criteria manually, or grade some responses first. - [8:54] Coverage (recall) and false failure rate (1 – precision) are the two metrics for LLM evaluator alignment. - **[10:36] Results** - [13:27] EvalGen achieves 33% recall and 90% precision on the medical pipeline, matching Spade with fewer assertions. - [14:58] The user is not writing code or evaluator prompts; labeling just 16–20 samples enables LLM evaluator creation. - [15:47] Q: Is the LLM evaluator fine-tuned? A: No, alignment comes from selecting criteria that match user labels, not fine-tuning. - **[17:30] User Study** - [19:13] EvalGen provides a great starting point, with alignment feedback in minutes instead of weeks. - [21:17] Participant two said, 'You should enforce that we look at at least twenty samples first.' - [22:46] Most people lack confidence in LLM evaluators because they haven't looked at data to validate their criteria. - [23:58] Jeff Bezos said, 'When the data and the anecdotes disagree, I tend to trust the anecdotes.' - [28:10] Eugene now believes LLM evaluators can be viable in production if throughput and latency requirements are manageable. - **[28:46] Questions** - [29:07] Future work: automatically improving evaluation prompts based on results, like using DSPy. - **[30:30] Demo** - [31:40] Eugene's prototype forces labeling of at least 20 rows to unlock evaluation mode. - [34:12] Optimization mode unlocks after labeling 50 rows and running evaluation once, providing a baseline for prompt tuning. - **[35:58] Q&A** - [38:39] The prototype gamifies labeling with milestones: 20 rows for evaluation, 50 for optimization. - [41:58] Q: Can most LLM evaluation tasks be reduced to binary predictions? A: Eugene says yes, e.g., summarization to relevance, factuality, comprehensiveness. - [46:29] The labeled data is used to fine-tune a prompt, making the prompt the deployable artifact instead of a trained model. - **[47:05] Re-Rankers** - [50:37] Speaker 7 vs Eugene: Can pairwise comparisons with Elo scores work for factual LLM evaluation, given gray areas in practice? - **[55:08] Shreya Joins** - [55:52] Q: Is there feedback on the information utility of specific labels for improving alignment? A: Shreya says V2 will include natural language feedback. - [57:15] Q: How does task decomposition interact with evaluation? A: Shreya says each node gets standalone assertions, enabling causal analysis of downstream failures. - **[58:35] Wrap-up** ## Speakers - **Eugene Yan** (guest) - **Shreya Shankar** (guest) ## Topics Evals ## Mentioned Anthropic (company), Scale AI (company), Vercel (company), AI SDK (product), Chainforge.ai (product), ChatGPT (product), Claude (product), Cursor (product), DSPy (product), EvalGen (product), FastAPI (product), GPT-4 (product), Next.js (product), SQLite (product), v0.dev (product) ## Transcript ### Intro **Eugene Yan** [0:00] I mean, the, the, the con- the, the, uh, the co- the question was, do we record the whole thing or just the, just the presentation? But then the whole- the presentation sometimes off- extends to the whole thing. So, uh, if you want private question and answers, you can write them in the chat. They will not be reflected in the YouTube recording. Uh, or we will do it afterwards when we stop recording. Okay. **Host** [0:21] Yep. And people have brought up, can we save the chat transcripts? So I think like for a write-up is what I save the chat transcripts, because there's a lot of good questions in there. Now, if people are interested, if we post them, that's good to know, because been saving them sometimes, but they've, they haven't been shared. **Eugene Yan** [0:39] Yeah, it could be. **Host** [0:40] Uh, and, and just general thing for everyone that joins, there's always great questions that come up in chat. So- **Eugene Yan** [0:46] Yeah **Host** [0:46] ... always, always open. **Eugene Yan** [0:49] Yeah. I think the chat is the main one. That's, that what, that was what was nice about Discord hosting it because all the chat just stays in Discord, the space. I have it pulled up to the side, but if I miss any of them, um, please just, you know, just raise your hand and just ask me to stop. Okay, cool. So we're gonna talk about this paper, Who Validates the Validators? Um, I don't have any slides as per normal. Uh, but I think that there are a few key interesting ideas in this paper that I wanna quickly go through in, um, hopefully in just, uh... Give me a second. Let me try to see if I can actually share my entire desktop. I wanna quickly go through this, uh, paper in maybe about 30, 45 minutes. Then I also wanna share about, um, an app that I built that I try to show all of this with evaluation mode, um, labeling mode, as well as optimization mode. ### EvalGen **Eugene Yan** [1:47] Um, I had given Shreya a sneak preview of this the last time. Um, so we'll see. Okay. Now on to who validates the validators. Um, so what, what is this paper about? So essentially this paper is about, okay, everyone's using LLM evaluators, uh, but the question is how are we actually gonna evaluate it? So obviously the-- I think that the way to do this is to label some data, label some data, and then match how it matches against that labeled data. And we'll, we'll see that that's how, that's how they try to do it as well. Um, just to confirm, you can actually see my screen, right? You can actually see the yellow and blue highlights? **Host** [2:27] Yep. Yes, yes, yes, yes. **Eugene Yan** [2:27] Okay. Yeah. So I'm just-- because I'm sharing the whole desktop. So we'll go through a couple of key ideas here. But one key idea is this idea that's very interesting, uh, and these are scattered throughout in blue. So firstly, the observation that Shreya and her authors made is that to grade outputs, we need to have some evaluation criteria, but the process of grading outputs help us define that evaluation criteria. So it's a chicken and egg problem. Um, across many tasks in terms of LLM evaluation, I've seen that trying to grade the outputs without actually having seen your data actually leads to criteria. Trying to create criteria without actually seeing the data actually leads to criteria that's very divorced from reality. Like people will say, "Oh, I don't want any toxicity. I don't want any, um, spelling errors," et cetera, et cetera. But the thing is, well, you know, those are actually not problems. Um, or they could say that, "Oh, you know, I want the LLM to speak like how it's speaking to a friend." Well, the thing is, if you actually try it and you try to look at the data, you realize that it actually cannot create that kind of, create that kind of text. It can't create, um, text that is, uh... It can't really return slang. Like for example, things like romantasy, which is a f- a mixture of romance and fantasy, like this kind of genre, or like grumpy sunshine, um, or reverse harem. So things like this is not really quite part of its vocabulary. It's, it's very hard to prompt it to get it written that kind of data. So, so that, that leads to undue expectations, right? These are expectations that just cannot be met. So therefore, what she's proposing is that it-- what they are saying is that it is impossible to completely determine good evaluation criteria without actually looking at LLM outputs. So essentially the point is you have to look at the data before you even come up with evaluation criteria. Um, unfortunately, this, this is not the main part of the paper, but it was a, a very key thing, and I think this was actually one of the biggest, uh, aspects of the paper. So essentially what, what she's proposing is that, okay, here's the typical evaluation pipeline, um, that's right on top. Okay, we start with, uh, prompting, we get outputs, then we just use some kind of evaluator LLM without really checking it. We just write some code. Oh, Vipul, you're not muted. Okay. Thanks. Thanks, Vipul. **Host** [4:52] I took care of it. **Eugene Yan** [4:53] Yeah. Yeah. Okay. So what, what we're seeing is that, okay, it's, uh, on the top, that's how we're doing it right now, but we're not actually, um, checking the LLM evaluator. So what they are proposing is that, okay, we put it through an LLM evaluator. We have some kind of candidate criteria, and the human also does need to grade the output in terms of yes or no. And then we check how well the candidate criteria aligns with the human grader output. And you know, this is an iteration process. Um, only once it's aligned enough, and that's why they have an alignment report card, we consider the LLM evaluator as aligned and usable both in development and in production. So essentially, u-users iterate through the process of refining criteria and grading so that you align the LLM together with the-- to, to your criteria. I think there are three big-- there are three main problems, uh, when trying to use an LLM e-evaluator. I think the first-- there are two of them that are-- we just solved. One is at the end, which is given the ideal prompt, can an LLM evaluator achieve sufficient level of accuracy? I believe the answer to this is yes. Second one is how do we get humans to label a hundred to four hundred samples of data with sufficient high quality? I think the answer to that is also solved Uh, essentially you just motivate them, incentivize them the right way. But the second, the-- what's in the middle is that n-now that you have this hundred to four hundred golden samples, how do you come up with some kind of prompt or some kind of input or output, uh, some kind of prompt or instruction or a few short samples that can actually achieve that ideal, that ideal prompt that can align well? So this part that is in the middle, I think is somewhat an unsolved problem, at least for me. Uh, and we, we see how, how they try to solve this. Um, so EvalGen de-design, essentially, how can we assist developers in creating evaluators? So EvalGen in design, how do we assist developers, uh, in creating evaluators to grade the LLM outputs, right? So essentially, instead of having them to use the bite track, how can I assist them to initialize the LLM evaluator once, and then we can move on. So they have a workflow here. I think it's useful to-- for me to just talk through this. I have had the chance to actually play with this, uh, live, and it's very nice. And a lot of it is, uh, what inspired my own, uh, prototype. So essentially, at the start, you can see here there's a prompt node, uh, where you're just writing a prompt. Essentially, these are tweets. You'll be doing name, entity, recognition on tweets. So this is the generation prompt. Now over here, this, uh-- still in box A, you can see that here over here, the multi-evaluator, this is the evaluation prompt. So in order to generate an evaluation prompt, they actually provide you three ways, which is you can infer the criteria based on their prompt. Uh, infer the, uh-- You can write criterias manually, or you can grade some responses first. So that's part two. So after you've done this, it will generate the criteria itself. So essentially, you're just getting GPT-4 to generate some kind of criteria. And this kind of criteria can be in the form of code, or it can be in the form of an LLM prompt. So there's two kinds of criteria here. One is code and one is LLM prompt. So you can imagine things like no hashtags, it's really just a regex search for hashtags. Uh, whereas no made up entities, uh, this is probably something that you need to rely on LLM on. So that's box C over here, creating criteria. And then as you are creating criteria, as you try to run it, you-- when you're running it, running these criteria through the LLM, um, as you're running it, you have the option to grade some, uh, grade some of the sample data, whereby you, you can say that, okay, is this good or is this bad? Uh, unfortunately, th-this is very broad. Is it, is it good or is it bad? It, it's not clear enough. I think, and the authors actually do mention that it is helpful to be more specific in some of their future lessons. And then finally, in box E over here, you can see, uh, what-- uh, how well is the LLM aligned. So I think this one, this box C is, uh, box E, oh, sorry, box E is, is very useful. So they use the term coverage and false failure rate. So to simplify things, coverage, what it really means is recall how many of the bad responses can we actually catch. And false failure rate is actually one minus precision. So it took me a long time to try to understand this. And I, I had several chats with Shreya to really unders-- uh, clarify what was she trying to indicate. False failure rate means that we are actually failing this, but they should not have been failed. So essentially, how much of it has been wasted? Uh, how many innocents are we just excluding? So it's, it's actually one minus precision. Um, okay. So essentially, this is the entire workflow, the entire UX that you're thinking of. Come up with-- LLM will help you create criteria. You will cr-- Uh, LLM will help you create criteria in box B. You will grade some responses in box D. And then eventually we check how well they align in box E. And if it's aligned enough, that's it. We have an L-- our LLM evaluator. So, um, well, I, I don't have to go through all this. Essentially, that was all, uh, indicated in the, in the diagram. And of course, the main thing is that the users can edit everything, and the users are also asked to grade the things. Um, I think I will pause here to check if there are any questions. The chat hasn't been showing up. **Host** [10:20] There's not much in chat. **Eugene Yan** [10:22] Okay. Sure thing. No, no questions then. Uh, I will proceed. Okay. So now we know how the EvalGen, EvalGen workflow looks like. Let's see how well it works and what the developers say, because they also had a small study on developers. ### Results **Eugene Yan** [10:37] So essentially, um, how is it done? Criteria suggestion. GPT-4 just proposes various criteria. Um, cr- candidate assertion, essentially, as they create this criteria, they will use code and then they will assert it. Either use code or prompt to assert it and then return the scores. And then after that, they, they will, they will sample some of the grades to get the users to give binary feedback. I think this is really interesting whereby you try to sample the rows that make the biggest difference. Um, I, I couldn't quite figure it out how they were actually doing this in a smart way that made a difference other than just going through the entire list. Maybe Shreya and Shreya is here, we can actually ask her that. So they, they had a lot of, uh, selectivity of candidate assertions. Essentially, what this means is that they will select the assertions that have a higher rate of passing it. And then they also have confidence scores, et cetera. Um, so one-- Here's how it compared to a previous paper by Shreya as well. Essentially, it was called Spade. Essentially, the idea of Spade is that in a nutshell, you version control the prompts, and every time there's a prompt change, the assumption there is that every time there's a prompt che- prompt change, it's usually to add an extra instruction to prevent a failure case. So essentially, by versioning control the prompts, by checking all the different prompt changes, you can create test cases out of it, and you can create an LLM evaluator out of it. So that's what they were comparing against. They have two datasets. One is a medical pipeline, essentially, uh, transcription doctor-patient calls. So the goal is to extract specific information without revealing any PII data. The other one is a pet It's an e-commerce data set, essentially, um, a hundred Amazon products. And the goal is to write SEO-optimized product descriptions without mentioning the negative reviews. So without ne- mentioning the negative reviews. So medical and product, uh, and e-commerce. Uh, so what I like s- what I like about these data sets is that it's fairly balanced, you know. Um, the defect rate. Okay, so essentially seventy percent and fifty percent of it was good. Uh, essentially, if you flip it, the defect rate is actually thirty percent and fifty percent, which is what we actually really want to catch. So here's our two data sets, product, uh, medical and product, with thirty percent and fifty percent defect rate. Um, so essentially, here's-- let's just focus on the results. What we can see over here is that for the medical pipeline, we were able to identify, uh, recall essentially was thirty-three percent. It's a bit on the lower side. Uh, I don't, I don't-- That said, I think they didn't really do a lot of work to try to optimize it and try to improve recall. But essentially, off the, right off the bat, recall is thirty-three, uh, thirty-three percent. So this is equivalent to spade, and the precision is also ninety percent. Remember, false failure rate is one minus precision, so precision is ninety percent. Uh, and this is, this is pretty good precision. Uh, that said, so you might say, "Hey, you know, what's the difference between EvalGen and Spade?" Well, the difference here is this. EvalGen only needed three assertions, be it code assertions or LLM prompts. So essentially, it needed less than Spade, which needed five, uh, to get, uh, comparable results. Now, when we look at the product pipeline, uh, we see that EvalGen only requires four assertions, which is half of what Spade needed. Um, essentially four different, uh, crite- uh, four different criteria, four different, uh, LLM prompts or four different code prompts, essentially up to four. And it was able to achieve pretty good recall, which is close to seventy-five percent. Um, in production, ideally, you want to try to get to eighty, eighty percent. Eighty percent is really hard to achieve in production. But eighty percent, ninety percent is pretty good. Uh, but what's not so good is the precision rate. Precision over here is sixty-one percent, right? Because, again, false failure rate is one minus precision. Uh, eventu- long story short, EvalGen does outperform Spade in terms of getting good, uh, recall and precision. Uh, in, in terms of achieving, uh, similar or better recall precision, but it's still a w- a few ways to go, uh, before we can use it in production. Uh, that said, this, uh, this idea was actually pretty good in the sense that... Again, remember, what's happening here is that the LLM is just seeing the prompt, either inferring some criteria or the user is just manually creating some criteria. The user is not writing the code. The user is not writing the evaluator prompt. Um, but just by getting the user to label, I don't know how many the user actually label, I think maybe sixteen to twenty in those two data sets, we were actually able to just come up with LLM evaluator problems just, just like this. Without the user having... Imagine a user who is a product manager or an SDE or software engineer who doesn't know how to write prompts, doesn't know about few-shot prompting, doesn't care about chain of thought. Um, and they shouldn't, right? All they should do is to provide you, "Hey, what's my criteria?" And then, "Can you get it aligned?" So I think that's a pretty good result. So any questions here before I go into user study? Okay, I guess not. Uh, how do I get the chat? **Guest** [15:47] Um, when, when, when you're talking about these, these prompts, are they, are they iterative prompts? Like when you say getting it aligned, you're talking about exchanges with the model. This isn't fine-tuning the model, right? **Eugene Yan** [15:59] No, no, it's not fine-tuning the model. What I mean by getting aligned is that, um... So imagine if-- So, um, okay, so let's look at box D over here, right? So the, the user is just saying, "Is it good or bad?" So now there could be many, many reasons why it's good or bad. It could be that because there's too many hashtags. It could be that it's not using bullet points. It could be that, um, it's making up entities. Again, the model just doesn't know. And I guess in this, in this study, the, they all just, just simplified it as good or bad. So there could be many, many different, uh, criteria, many different evaluation prompts. So now how-- So what happens is that, again, in box C you see here, the LLM is coming with many, uh, possible criteria. It's trying to guess what the user, what, what the user needs, right? Um, based on, either based on the prompt or based on what the user has typed, maybe just some free text string. So it's coming up with these evaluators. So eventually, what we're gonna do is we're gonna run these evaluators and try to see how well it aligns, aligns with the data that the user has provided, which is up or down. And, and I'll, I'll have a example demo of this, uh, in a bit where, where we can see how, how, how well alignment is. And so what is gonna happen is that then there's an algorithm on the back end, uh, essentially just, uh, computes which of these evaluators should we use, combine, and which of these should we drop to have the highest correlation with the human input, good or bad labels. ### User Study **Eugene Yan** [17:30] Does that make sense? And of course, you know, if the correlation is bad, you could try to optimize the prompt, rewrite the prompt in a different way to fi- find a better correlation. So what-- that's what they mean by alignment, which is aligning to the user-provided labels of good and bad. **Guest** [17:45] Got it. Thank you. **Eugene Yan** [17:47] Yeah. So essentially like this, this is what they really care about. Coverage. They want to have high coverage of bad responses and low false failure rate. Okay, so what's really interesting is that they also had this, um, they also had a user study with nine, uh, nine industry practitioners. Uh, I was one of them, so I, I had a chance to play with this, and I was actually fairly, uh, impressed. So you can see that, okay, you'll be... The, the prompt is you'll be doing name entity re-extraction, which is the example there. Extract up to three well-known entities, uh, and then write a sen-- Basically, they have to make up a sentence describing a personal entity, um, and answer is a book- bulleted markdown list. So you can see that there's quite a few, there's quite a lot of instructions here, right? Extract entities, um, write a short, one single sentence description and then return it as a markdown lis- markdown list, and don't extract an, uh, hashtags. So you can see, you c- you can imagine this being split into four different criteria, and it's just easy to... Some of these can be, uh, solved by code, such as no hashtags, uh, and then bullet points like this. Some of these probably have to be like, uh, do not make up, do not make up, uh, any entities. Some of these could be probably like, uh, only an LLM. So then what we found is that participants found that, uh, EvalGen was a great starting point. Which is that it's a great starting point to make it fun for people to label data and get immediate feedback on how their labeled data is helping to align the LLM evaluator, right? This, this level of feedback. And this level of feedback is like in the order of minutes or at most an hour instead of, um, days or weeks, right? Sometimes if you're not labeling the data yourself, if you're using something like Scale AI, um, the, the, the data might not come back so quickly. So the iteration loop is, it really kills your product launches or your feature development, especially when you're using, uh, trying to build an LLM-powered, uh, customer experience. So here's what participants usually did. Um, there's a hundred LLM outputs. So they sort of viewed it, like just quickly scan through it, make sure it looks right, and then partners, uh, click to start on it. So there were six participants that just used AutoGen, auto-generate criteria. One participant wrote the criteria, and then two other participants who actually wanted to look at the data. Amazing. I think that's great. Um, so after... Then after that. So after that, they get some criteria suggestions from EvalGen. Uh, participants actually really wanted to play the criteria and try to, uh, s- push the boundaries of it. So they removed some suggestions, added some criteria of their own to see how far EvalGen could go. Um, so EvalGen as a starting point, what did the participants say, right? Uh, firstly, it's really hard, uh, to come up with criteria on what is a good summary or what's a good translation or what's a good extraction, right? So by prompting an LLM, and you can probably do this with ChatGPT UI or, uh, Anthropic Claude UI, um, it was helpful that it was able to get the criteria we want. And, you know, this is nothing, nothing outstanding. Um, then participants who actually started grading before actually creating criteria said that the grading process was helpful, right? And we all know this, by looking at the data, and th- this was the, the key point, the key points that I highlighted in blue, which is by looking at the data, first, you understand the data, you feel for the data a little bit better, and that helps you write better criteria, right? Um, so that's one. I think this was me. Participant two actually said, "You should enforce that we look at at least twenty samples first." And that's what w- I will actually show this enforcement in the demo I'm gonna show later. So, so because, you know, while LLM is doing the grading, especially if there's chain of thought, uh, it takes some time. So but the, a great use of a time is that, okay, participants, maybe you wanna go get a coffee, but a great use of time is for the participants to actually provide labels themselves. So this was what participants were sharing, that they were happy to grade while waiting. And here's how the output looks like, right? So you can see, here's the full text of the tweet, and here's the response. Okay, let's, maybe let's look at a second one. Uh, great fire department at twelve NYC. So it extracted great fire department, extracted NYC, extracted FDNY. Uh, this is interesting. I was not aware of this. And then, so you can see we have four evaluators here, whether it's a bulleted list, whether there's no made-up entities, whether there's no hashtags, and whether it's in a single sentence. And of course, the single sentence, two of these actually fail, uh, b- because they, they're not made up of a single sentence. So based on this, you can actually see this. And of course, all this data is just computed in the back end to create the alignment scorecard. Um, participants who like looking the coverage and false failure rate, looking at this made it very easy, which one was, um, was actually look, uh, which one actually looked like. And here's what I think is the meat of it, which is alignment is iterative, right? The problem, I think the main problem, and I think this is very right, is that most people don't have very strong confidence in an LLM evaluator because they are-- e- even they themselves are not confident of the criteria. And the thing is, this is not explicitly vocalized, but most of the time they're not confident of the criteria because they don't know if their criteria is actually helping. And the reason is that, be-- for that is because they haven't even actually looked at the, the data. Sometimes the people that provide criteria are like, uh, PMs or like, uh, business leaders who may not have a chance to look at data. They don't, uh... Maybe it's the data is too complex, it's just not in an easy consumable format, and therefore they don't know whether it actually works or not. But I have found that by getting people to look at the data, even just ten or twenty samples, right? Of course you have to pick representative samples with both defects and non-defects. It helps them understand a little bit better, right? So if you think of data in general, there's quantitative data where you aggregate everything into one or two metrics, that gives you very broad scope. And then there's anecdotes, right? Where you actually look at specific samples. And Jeff Bezos has this quote saying that, you know, when the data and the anecdotes disagree, I, I tend to trust the anecdotes. That's what we're, that's what, uh, this is suggesting here. Look at some of the data samples. Look at some of the anecdotes. Um, yeah. So again, here's the chicken and egg problem. We need to think through criteria. We need to write it down in order to grade outputs. But when we grade the outputs, we also f- figure out new data, uh, new criteria. It's the same thing, like it's the same thing as writing. Uh, when you start writing, you probably only know half of what you actually eventually finish writing. As you start writing, you start digging a rabbit hole deeper and deeper, and you find more things. So this is what this proposes, which is that feedback loop. Looking at data, creating criteria, creating criteria, looking at data, and creating more new criteria. That's very helpful. Um, yeah. So but one interesting thing here is that one participant mentioned that they gave it a bad grade, not because the output was bad, but they wanted to be consistent with their previous grades, right? Previously, they thought something was bad, uh, and then they gave it a bad score. But after, there could have been other things that were more egregious. And, you know, on, on hindsight, this is really just a minor paper cut. It's not as, as critical. But they continued to be consistent with their previous grade instead of updating their previous grade. So I personally don't think... I think that this is a very difficult problem to solve in the sense that you want to be internally consistent with what you have done in the past. But y- what that means is that this perpetuates the bad data and the bad labels. Uh, so I think there's a point here, uh, which is I, I, I wrote to remind people that my design decision here is that if you find that your criteria has drifted, instead of trying to maintain the same criteria and aligning to the, to the previous grades, instead what we should do is we should revisit those previous grades and fix it, um, because it's an iterative process. Um, what constitutes alignment? I think there was a question there about what was alignment. Uh, it's very subjective, right? Uh, I think that when it's really just binary, yes or no, good or bad, I think that's a little bit easier to measure what alignment is. In a sense, it's really just recall and precision or Cohen's kappa in terms of correlation metrics. Uh, but sometimes it's just very difficult, and sometimes, uh, there was this slight disconnect when the, the participants provided grades, right? They provided labels, but it had little impact on their chosen assertions. Uh, I, I think this is just, it's just not clear how the grades actually, the actual grading actually tried to optimize this. And actually, during that demo, they actually didn't do any optimization or creating new evaluation prompts. So that, that may be why there was some disconnect there. The other thing is that users really want to have control over both the evaluation type as well as the prompts itself, right? And we have seen this, is that a lot of times, uh, we will come up with an new prompt that actually works well on eval, but sometimes for better or for worse, for whatever reason, people just want to include just this one sentence to make sure that that defect doesn't happen, when that defect actually doesn't happen. And when you try that, you know, sometimes the evals actually perform worse. Uh, but unfortunately, you know, sometimes these are just policy decisions that are a little bit disconnected from the data. Um, finally, they s- they found that, you know, as developers, they iterate over the criteria, they find that, and as they engage with, as they look at the data, essentially look at the LLM outputs, it helps them refine their criteria even better. So essentially, the, the point here is that tighten the feedback loop in terms of creating the LLM evaluator, and once that LLM evaluator is well aligned, we can just use it for, uh, use it in production. That said, uh, some participants were actually quite skeptical of how LLM-based assertions, LLM-based evaluations might be helpful for monitoring LLM outputs in a production pipeline. Uh, a year ago, I might be, I, I would have been in this camp as well. I think now I'm also partially in this camp, but partially not in this camp. Like, I, I, I'm beginning to see that I think it can be viable if the throughput and latency requirements are not too, uh, demanding. So participant eight actually said, "I cannot begin to think about how LLM as validators in production can work. I'm very skeptical." Um, so yeah. I think that's, that's all I wanted to focus on. Uh, essentially, it's the workflow here that I think is very valuable. Uh, and they also had future work, um, like one, one thing here they actually didn't, uh, implement here, which is users want their prompts to automatically improve based on evaluation results. So this would be something like DSPy, et cetera, et cetera, uh, that you can think about, which we saw in some demos to automatically improve our evaluation prompts. ### Questions **Eugene Yan** [29:07] Um, but so yeah, that's it. That's all I had to share. Any questions? Just gonna go back to the screen here. Oh, there's a few questions. Um, okay. I am gonna start from the one in the bottom. Um, is there feedback on the informational utility of specific labels for improving alignment? This is not clear to me. Um, I don't know. And I don't know if Shreya has joined yet. She may not be able to join. Uh, she may not have joined yet. Yeah, she has not. I think this is a question that maybe I will wait for Shreya. Let me just check Twitter if she's, she has responded. I did share it for the Zoom link. **Guest 2** [30:08] Eugene, you said you made a prototype based on, uh, uh, like the-- I think there was like a diagram earlier. Like, can you t- talk more about the prototype if, if no one else has questions for the paper? **Eugene Yan** [30:19] Yeah, sure. I can share more about the prototype. Um, where is it? Okay, here we go ### Demo **Eugene Yan** [30:33] Share. So here's a, here's a very small, simple prototype. Essentially, the question, the question that I have for myself is, how can I help users, um, label data and evaluate data and improve their evaluation prompts easier? So here's a small toy that I made. Uh, it's called Label: LLM Assistants for Better Evaluation Labels. Um, so to get started, we have to upload a CSV file, right? So you can see a CSV file. Essentially, I think these are fairly reasonable constraints: ID, input, output and a label, and the label can be empty. So what am I gonna do is I'm gonna input a- upload a CSV file. So this is a CSV file. It's the factually inconsistent. This, this is the factual inconsistent, um, data set. So what I'm gonna do is let me just start from scratch, um, upload. So here's how the CSV profile looks like after we've uploaded. That's the input, that's the output. The goal is to try to make it easy for the users to read, right? And we can enter labels. So the thing is, currently, right now, we are in evalu- we are in labeling mode. So to unlock evaluation mode, you have to label at least twenty rows. And I try to provide feedback that, you know, you have to label at least twenty rows. Essentially, I'm trying to gamify this to help users get to the next level. So but I have a data set that, uh, already has some rows uploaded. Um, so this, this has, uh, this is evaluation mode. So in, uh, in order to achieve evaluation mode... Uh, wait a minute. Have I already labeled so many, so much data? Let me just... Okay, maybe my data set wasn't prepared correctly. Uh, but anyway, this is, this is now evaluation mode. After we've labeled twenty out of twenty, it will show that evaluation mode has been unlocked. So you see evaluation mode unlocked. Scroll to the top to add your prompt. So once the evaluation mode is unlocked, you can enter the prompt, and it'll allow you to use the prompt to try to do that. So evaluation mode, you can just run evaluate. It will evaluate all of the data, uh, evaluate on the data. So you can see we start getting results as it comes in piece by piece. Essentially, what this is happening is just running on the back end. Uh, big thanks to Swix for teaching me how to do this, the right way to do this. And what we're doing is this. It's running on the back end. It's all storing in the database. But at the same time, we're also polling it once a second to show the results. And then what we see in the top left corner here, these are metrics, right? These are metrics for pre- recall, precision, Cohen's kappa. And we also have true positive, true negative, uh, false positive and false negative. So this is, uh, gonna take some time, and it's gonna evaluate the data. And along the way, we can see how well it aligns. So we can actually see that, okay, when alignment is there, you see essentially the label, the prediction matches your label. That's great. And then when alignment is not there, the prediction doesn't match the label, it's, uh, it'll be highlighted in red. And you can look through it. You can look through it. You can try to update the chain of thought. You can try to understand why it's giving this prediction based on the explanation, and you can see the explanation here. And as we scroll down, oh, wow, this, this is pretty good. You can see a lot. Most of it is largely aligned. Essentially, what this is doing, this is just a chain of thought followed by the final label. And it's just gonna go through it. And once it goes through all fifty of it, um, again, this is, uh... So currently we're in evaluation mode where the LLM is evaluating it. So to unlock optimization mode, uh, you have to label at least fifty rows, and you have to run evaluation mode once. Essentially, we sort of need an existing metric for how well-- We need a baseline for how well this basic evaluation prompt works. And you can see the prompt is actually very short. Just evaluate if the output summary is factually consistent with the input article. If it's consistent, return one. If it's inconsistent, return zero. So now we are at, uh, row number thirty-eight. Oh, seems to have broken. Nope, it's still running. It's just taking a bit of time. Uh, row, row number forty-one. So once we have at least fifty rows, I actually think the right number should be maybe a hundred to four hundred if you want to be optimizing based on this. Uh, and after we've run evaluation mode run once, uh, we can actually run optimization mode. So optimization mode is gonna come up in maybe a couple of seconds. So you can see right here, it's just-- I just want to try to figure out how, a way to give users fast feedback. All right. So now that we've labeled fifty, uh, ev- and evaluated all of them, it shows that optimization mode is unlocked. So we can scroll to the top and click on optimize evaluation prompt. So optimization, what it, it does is that on the back end, we're just gonna try to figure out how to optimize the evaluation prompt. Uh, unfortunately, clicking on this number, clicking on this button doesn't do anything. I just implemented this yesterday. Um, but I tried to, um, well, with Next.js, it's very easy to try to make it something fun and funky. Of course, I'm sure Swix, uh, will do something way better. But essentially, this is, this is what the next step is. Clicking on this button, we'll optimize it on the back end. We'll see a different, we'll see a different page, which has, like, maybe ten rows and all the different optimization scores. Essentially, it's just hyperparameter tuning. You can think of it hyperprompt tuning. Um, so yeah, that's my short demo. Any questions or feedback would be nice, yeah. Um- **Guest 2** [35:58] That's fire. ### Q&A **Eugene Yan** [35:59] Yeah. **Guest 2** [35:59] So, so of, of all the tested frameworks, you ended up going with Next.js and not like, um- **Eugene Yan** [36:03] Oh. **Guest 2** [36:03] Because it was like HTMX- **Eugene Yan** [36:04] How do you know this is Next- How do you know this is Next.js? **Guest 2** [36:07] You, you mentioned it a couple minute, uh, seconds ago, I think. **Eugene Yan** [36:09] Oh, okay. Yeah. This is Next.js. I-- It was really easy to build this with Cursor and v0.dev. **Guest 2** [36:16] Mm. **Eugene Yan** [36:17] Um, and I went with it. The back end, uh, is going to be FastAPI. I, I, I believe the back end will be FastAPI. Uh, or I could try to do it, uh, all in TypeScript and Next.js. I don't know. Uh, but I think the prompt optimization back end, I'll, I'll just have it in FastAPI and then, you know, Next.js, this app will just call FastAPI- Uh, you know, here's the, here's the labeled data set. Or rather, the labeled data set is all in a DynamoDB. Or rather, it's just in a SQLite DB. And then the, the FastAPI backend will also have access to that and just optimizing it and computing scores. **Guest 2** [36:59] Um- **Eugene Yan** [36:59] So this is... Oh, sorry. Go ahead. **Guest 2** [37:03] No, I was just gonna say that, that was really cool. I, I'm now trying to tie it back into like, uh, the validator paper that we just read and understand like the implications that you were talking about for that diagram at the beginning. **Eugene Yan** [37:12] Exactly. Exactly. That's great. Thank you, Flo. Um, so what I was trying to do was this, this green section over here, right? I'm trying to force pe- so you can see that there, there's a... It's, it's very specific, right? I'm trying to force people. I really think that you should be evaluating at least 20 samples. Evaluate at least 10, 20 samples. Um, I'm not trying to create candidate criteria. Um, I'm not trying to get an LLM to create initial criteria. The thing is, I need the human, I need the human to just provide the criteria, and it could be bullet points. So you can assume someone who doesn't know anything about prompting, they're just gonna put bullet points. They're just gonna take whatever SOP they have, they're just gonna paste it in there, and we just get S- we'll just get the LLM to run it, right? And then we, we're just gonna check. And then we can check the alignment report card, which is this thing over, on the, here on top. Probably will have a separate screen on its own. Um, I don't know. Uh, but this thing is just a floating screen. It's just, I think it's fun to see. I think it give, gives instantaneous feedback that, "Hey, you know, this thing is doing some work. Your work is, um, your work is valuable. You doing the labeling is valuable, and therefore, uh, I want you to label more data." And of course, every step of the way, I try to give you milestones. After you've labeled, um, I'm gonna, I'm gonna delete all of this. Or rather, maybe not. I'm gonna give... Okay, so if you, if you wanna unlock evaluation mode, you have to label at least 20 rows, okay? So try to get into, to the first milestone of labeling is 20 rows of data. After you've labeled 20 rows of data, you can run evaluation mode. And if you want to unlock optimization mode, you have to label a- another 30 rows. Um, so give them the next milestone. And then after they've done that, okay, now we can do optimization mode. I think, I think for this demo, the numbers were actually low. I think 20 and 50 is a bit too low. I think maybe it should be 25 and 100. Um, but I'm, I'm still trying to play with that. Uh, but yeah. That's, that's the, that's the... This is the loop. This green box here is the loop I'm trying to, um, improve on. Make, make tighter. Okay. No more questions, I guess. Yeah, it's, it's a fairly easy and straightforward paper. Oh, actually, there's a lot of stuff in the chat. Is the code open? Um, is the code open? It's not open yet because it's very ugly. Um, I will... This code is not open, but the previous one on the evaluation frameworks, it is open. Um, so you can look at it. This, this one is not open. I still have quite a bit of work to do to make it a little bit cleaner. Um, you should experiment with the Writer framework to see how it does and handle front and back for you. Yeah. Oh, kind of like FastHTML. Wow. Yeah, I, I, I know you pinged me that, Sam. Uh, I haven't had a chance to look at it. Um, the AI model is tightly coupled. Yeah. Um, um, yeah. The back-- Like, this app, I was really... Again, when I was trying to build it, before I was building it, I was like, "Okay, maybe let's try, uh, dig into Next.js again or TypeScript frameworks again." So in this over here, uh, shoot. I switched over to a different screen. Um, in this framework over here. So long story short, I went through FastHTML, Next.js, Civet Kit, and even had a Frankenstein of Civet Kit, Svelte with Fast API in the back end. Uh, this was not good. Um, yeah. It-- I really had to hack a lot of things and Frankenstein it. I, I really don't like how it works, but I think this is probably the best of both worlds. Um, yeah, SDK might be fun to try. I think I did use the Vercel AI SDK for this. Um, but I can't confirm. Let me, let me take a look right now. Uh, well, so here's this ugly code. Um, and the evaluate route. Oh, I'm just using an, I'm just calling Anthropic directly. Yeah. AI SDK. I think I initially used AI SDK from Vercel, but I think what was useful there was streaming, and I didn't have streaming. So I just simplified it and just deleted, um, stuff to, yeah, to simplify it. Yeah, the wallpaper is leadership principles. Oh, so it's- **Guest 2** [41:54] Um- **Eugene Yan** [41:54] Oh, someone. **Guest 2** [41:55] Eugene, if I may? **Eugene Yan** [41:57] Yeah, yeah. **Guest 2** [41:58] Um, okay. So in your experience, were all or at least most tasks distillable down to a binary prediction, or is that just the ideal goal? **Eugene Yan** [42:11] I think most tasks can be. I think if you think about it, if summarization, it's really relevance, factuality, comprehensiveness. Or maybe you can give me an example of something that you don't think can be distilled down. **Guest 2** [42:28] Um, well, off the top of my head, 'cause I don't have a specific use case in mind, but maybe, um, I know, multiple class classification. But I guess that can be distilled down to pairwise 1v1, uh, sort of framework. **Eugene Yan** [42:41] So- When you think about multiple class, are you thinking about does this output meet, not meet policies for toxicity, bias? **Guest 3** [42:54] Multiple choice questions where multiple answers, uh, uh, are required for some questions. **Eugene Yan** [43:01] Yeah. So that is not the task of an LLM evaluator, right? That's the task of an LLM classifier. **Guest 3** [43:07] Oh, I see. Okay. **Eugene Yan** [43:08] Yeah. **Guest 3** [43:08] Okay. Okay. You're targeting specifically the, uh, ability of an LLM to evaluate, uh- **Eugene Yan** [43:15] Whether something is good or bad **Guest 3** [43:16] ... got it. Okay. **Eugene Yan** [43:17] Uh. **Guest 3** [43:17] Makes more sense. **Eugene Yan** [43:18] So you're, you're right. For multitask, like, you know, or maybe like, um, let's say given some kind of product description and title, can you say is it, uh, is it FMCG? Is it clothes? Is it furniture? Yeah, that, that, that is completely different. That, that would be, that would be a separate thing. But right now I'm just really focusing on evaluating it. Like- **Guest 3** [43:43] So, so the goal here- **Eugene Yan** [43:43] Is this, is this classificat- is this classification correct or wrong? Simple. That's binary. **Guest 3** [43:49] And the goal here is to have a, uh, let's say, a solid justification for using an LLM as an evaluator. **Eugene Yan** [43:58] Uh, no, not really. Um, I know that we have to use an LLM as an evaluator. There's no way around it. If we wanna scale, I think that's the only way. The question is how can I tighten the alignment? How can I tighten the feedback loop to make it easier to use the LLM as an evaluator? **Guest 3** [44:16] I see. **Eugene Yan** [44:16] And I'm taking a lot of, um, inspiration from- **Guest 3** [44:20] Shortcutting the loop in order to be able to scale to, uh, more instances, uh, fast- faster and, uh- **Eugene Yan** [44:26] Exactly. **Guest 3** [44:27] A higher volume of data. **Eugene Yan** [44:27] So you can... Exactly. So you can imagine that, uh, uh, maybe an AI engineer might be able to look at the data, understand the criteria, talk to PMs, understand the criteria, look at the user experience, and then write the prompts. And then they will do all the evaluation themselves, right? But now let's imagine that we don't have that AI engineer, or that AI engineer is extremely bottlenecked, and we wanna use AI everywhere, right? We're gonna have LLMs doing classification, summarization, translations, extraction everywhere. Um, now there will be a certain level of... There, there-- At some point in time, there'll be a certain thing, there'll be certain level of risk tolerance whereby simple things like extraction and classification, if it's on the back end, we probably don't need an evaluator. But then there may be other things like if you're summarizing a financial document, or if you're summarizing a medical document, if you're extracting a medical document, there may be certain things. Yeah. Essentially, the, the key thing is I'm trying to force people to look at their data. Oh my God, I think, Swix, this is like a, um... You should just post this on Twitter and tag Hammo and me, um, and who knows what. I mean, like- **Guest 3** [45:36] Yeah **Eugene Yan** [45:36] ... this is like, um, brilliant. **Guest 3** [45:38] Yeah. And also, it might not have been your main intent, but I think it makes for a good framework, uh, to, um, uh, I don't know, maybe to answer skeptics about LLM evaluators. Uh- **Eugene Yan** [45:52] It could be **Guest 3** [45:52] ... to show that there's a systematic way of, uh, ensuring at least some, some robustness or at least some reliability in LLM evaluation before actually deploying it in production. **Eugene Yan** [46:03] Exactly. Right. I think the metrics here, that's what we're trying to do with the metrics here. Hey, we can say that, "Hey, no, you gave me... Again, I don't want to use 50. You gave me 100 to 400 rows of data. I am able to do well on that." And that's no different from machine learning, right? Instead of using the data to train a model, is what we're using now is we're using the data to fine-tune a prompt, which is the model now, which is now the artifact, that prompt. That prompt married to that, coupled with that LLM that you're using, is now the artifact that you can then use to optimize against the labeled data. Essentially, it always starts with having labeled data. Um, so I'm trying to force people to label data, uh, instead of just using vibe checks. When you are doing vibe checks, you're also, you can also just take the extra step to just label the data, and then with that labeled data, I don't even, I don't need to bother you anymore to do the vibe checks. You give me the labeled data, I try to optimize against it. **Guest 3** [46:56] Completely agree. Thank you. **Eugene Yan** [46:58] You're welcome. Um, so Nicolay, uh, have you played around with using re-rankers? What do you mean, Nicolay? ### Re-Rankers **Guest 3** [47:05] Um. **Guest 4** [47:06] Yeah, no, I talked to the guy from MixBread today, and he mentioned they're using the re-rankers a lot in evaluations, but also in test time. For example, to evaluate something like factuality and just training the re-rankers with a small data set. **Eugene Yan** [47:23] Hmm. Sorry, I, I confess. I was trying to ping Shreya, and I can't multitask. So you're using re-rankers to try to u- to evaluate? What does that mean? **Guest 4** [47:36] Yeah. Basically, as opposed to training a classifier to predict, like, the label, you are using re-rankers. And, um, so for example, for factuality is, like, an, an obvious one, and using the scores plus a threshold, which apparently seems to work better than classifiers in their cases. **Eugene Yan** [47:57] So y- okay. Just to understand, understand, just to make sure I'm understanding you well. You're using the re-ranker to rank further things down? **Guest 4** [48:07] Um, n- no. In the end, like, the, the re-ranker is trained on, like, you train it on pairs. So as you have in factuality checks when you have NLI models. So in an NLI model, you basically, you have a hypothesis, and you have, like, the ground truth, and you basically try to determine whether the hypothesis and the ground truth aligns. **Eugene Yan** [48:32] Yeah. **Guest 4** [48:32] And you could use that basically with the label to train the re-ranker on it and then basically set a threshold above the score of the re-ranker. It's determined basically as the class. Like 1, it's passing. 0, it's not passing. **Eugene Yan** [48:48] I see, I see. So what you're saying is that, okay, we have some references, we have some golden samples, and then- **Guest 5** [48:53] Yeah **Eugene Yan** [48:53] ... we compare. We train a re-ranker to rank the golden samples higher. **Guest 5** [48:58] Yeah. **Eugene Yan** [48:59] That could work. I mean, if you have a threshold, that could work. Yeah. I think what is, um, what is hard for me to understand sometimes is that if you're using a re-ranker and you pass in two pairwise preferences and it returns-- es-essentially, actually, your re-ranker is actually returning a score. It's not really a, it's not really pairwise preference. So that, that was what tripping me up. So if your re-ranker is returning a score, that's actually also you can think of it as a classifier. Yeah. The, the, the thing in-- previously, when I've tried to do this, several folks, including from, um, s-some of the labs, they sh-suggested doing pairwise preferences. So the, the reason why pairwise preferences cannot work is that if you give two things that are both factual, um, or if you give two things that are both non-factual, you will say that one is better than the other, but it still doesn't meet the bar of being factual enough. But in your case, uh, I mean, the, the case of re-ranker that you mentioned, there's a score involved, so you use the score to cut the threshold, and that would work. Yeah. I, I, I, um, so to answer that, I haven't, uh, played around with using re-rankers as, as opposed to a classifier. Yeah. Okay. I think Shreya might not be able to join. Uh, her seminar has been running late. Um, too bad. Uh, but any other questions? Or any other ideas on how we can make this more fun, um, for people? **Guest 5** [50:32] Sorry. I just had a follow-up to what you just said regarding the pairwise cannot be used- **Eugene Yan** [50:37] Yeah. **Guest 5** [50:37] Uh, for optimized evaluator. So, um, what if you took like a batch of input outputs, and then you did the pairwise preferences, computed Elo scores for each one of the input outputs, right within the particular batch. And then I guess you could have some kind of a threshold on, okay, anything which crossed this Elo rating is probably good enough. Like, couldn't you use that method? 'Cause like, yeah, you can then turn that pairwise thing into like a per data point threshold- **Eugene Yan** [51:08] That's- **Guest 5** [51:09] ... kind of thing. **Eugene Yan** [51:09] I think that's a great point. I think the un- the way to answer this question, I-- the answer to this is that, um, let's look at... Man, Slack, there's just-- I mean, Zoom just gives me no feedback when I'm sharing my screen or not. Um, so the way to answer this is the question is, is your task objective or subjective? So if your task is factual, like something more objective, like toxicity or factuality, like is, is, is a clear one or zero. Then if you're comparing two zeros and it's saying that one is better, um, that you, you-- that doesn't make sense. Or if you're comparing two ones and saying that one is worse, well. But if your task is more subjective, like which of this is more persuasive? Which of this has a better tone? Which of this is better clickbait? Which of this is more relevant? Yes, uh, a pairwise comparison could work. So that's how I think about the task. So a lot of times it's like, okay, we have control. Control is our current one. Then we have something new. Let's say we have newer, better, newer or better recommendations. It's the same thing as like we have a newer or better prompt. We have newer or better output. We have a newer or better way of creating the summary. And then we just do a... You can just-- in that case, pairwise comparisons can work. And I know, uh, AI news does a lot of pairwise comparisons. Almost every time there's a new model that comes out, it does pairwise comparison, and it actually involves users in the pairwise comparisons as well and share, shares that saying that, you know, these are different models. That can work. On the other hand, there are certain things that are just more objective. Uh, I think pairwise comparisons doesn't really make sense. **Guest 5** [52:40] Um, I, I think just one minor pushback. I think the differentiation on the objectivity versus subjectivity is not as clean in practice. Like when, for example, here you have instruction following as like a objective metric. I think to an extent it's objective, right? If you have something like don't exceed these many words and, um, don't, uh, use this kind of language. But like even just within like, uh, two do- two domain experts, right? Let's say it's like a medical task. Um, I feel like there is some level of subjectivity. It's like taste is what I think it is, where each, like, domain expert is like, "Well, I think a better way of following the instructions would have been this way." Um- **Eugene Yan** [53:21] Mm. **Guest 5** [53:21] I don't know. I, I think that's all I- **Eugene Yan** [53:23] I see. **Guest 5** [53:23] That, that's like where I think in practice this kind of breaks down. Uh, what I've seen is like, um, even, even with like factuality, like, um, yeah, there's that gray area of how you interpret the data. Um, so yeah, I think maybe that's an issue on the margins. Maybe it does not end up being as effective. But yeah. **Eugene Yan** [53:41] Yeah, that's a good point. There's definitely gray areas, right? Uh, for example, is, is the show My Little Pony should be, should be tagged as horse or should it not be? **Guest 5** [53:50] Yeah. **Eugene Yan** [53:50] I mean, people might search for, search for horses, and they're looking for equestrian or like Olympics equestrian. Definitely not My Little Pony. **Guest 5** [53:57] Yeah. **Eugene Yan** [53:57] But well, My Little Pony is a horse. So that, that's a great point. So there's definitely a huge chunk of gray area in the middle. **Guest 5** [54:04] Yeah. **Eugene Yan** [54:05] Um, that's just hard. That's just hard to get, get precise on. **Guest 5** [54:09] Yeah. 'Cause I just feel like if AI is going to get very intelligent, it's going to do more things like that. **Eugene Yan** [54:15] Yeah. And I think even for that, we can just make a call to say that, okay, in this case, we actually want it to be one, and we can write an instruction for it. **Guest 5** [54:22] Yeah. **Eugene Yan** [54:22] And we can make the task simplified to binary. **Guest 5** [54:25] Yeah. **Eugene Yan** [54:25] As opposed to, is this a better summary than this? Is-- Which one has more relevance? Which one is more comprehensive, et cetera? For me, that is really hard to... You can try to break it down to binary. You can say relevance is composed of these criteria. Um, comprehensiveness is composed of these criteria. But I think you are usually better off with a pairwise comparison. If you're trying to compare control versus treatment, which one is better? **Guest 3** [54:50] Thank you. **Eugene Yan** [54:52] Welcome. Mm. Okay, I guess that's it. I guess Shreya didn't have a ch- is not gonna be able to have a chance to join us. Um, it's okay. Uh, she's... I think she's giving a seminar right now, and she just cannot join. Well- **Shreya Shankar** [55:08] No, no, I'm here. ### Shreya Joins **Eugene Yan** [55:09] Yeah. **Shreya Shankar** [55:09] I, I was here for five minutes, sorry. **Eugene Yan** [55:12] Took your, took your time. You took your time, Shreya. **Shreya Shankar** [55:15] Sorry. No, I was at a seminar that ended at 12:50. I'm so sorry. **Eugene Yan** [55:20] It, it was not clickbait. I, I did legitimately invite Shreya. Uh, I was hopeful she would be able to join. Um, I don't know. So now that Shreya, the pa- author of the paper here, is here, anyone have any questions for her about the paper? **Guest 3** [55:35] I think we all just asked you the questions earlier and you answered that. **Eugene Yan** [55:40] Yeah. Actually, there, there's one question- **Shreya Shankar** [55:43] Sure **Eugene Yan** [55:43] ... uh, that I see from Venu. Uh, Shreya, is there feedback on the information utility of specific labels for improving alignment? **Shreya Shankar** [55:52] We do not have it in the paper's prototype, um, but the V2 that will hopefully be pushed in the next couple of weeks, it has to be before the conference, um, that does consider natural language feedback. **Eugene Yan** [56:07] Nice. So essentially it's like, is it like chain of thought feedback and then the LLM will use it to update the prompt or update the, the, the assertions? **Shreya Shankar** [56:15] If user gives a feedback maybe on why something is bad, just a few words, and then that is included in the prompt for that validator. **Eugene Yan** [56:25] Mm. **Shreya Shankar** [56:25] For that task. **Eugene Yan** [56:26] Ah. **Shreya Shankar** [56:26] Or sorry, for that criteria. **Eugene Yan** [56:28] Oh, essentially as a few short example. **Shreya Shankar** [56:30] Yes. **Eugene Yan** [56:31] Oh, no, nice. That's, that's a, that's a great idea. Yeah. Um, any other... **Guest 6** [56:39] Sh- Shreya, I have a q- quick question. Are you, are you planning to publish this framework in some way as a open source project? I didn't see if you had already done that or what's going on. **Shreya Shankar** [56:49] Not yet. It's not out yet, but we will... So the conference is in three weeks. We have to have it out by then, but it'll be implemented in Chainforge.ai, which is an open source LLM pipeline building tool. **Guest 6** [57:02] Right. Okay, awesome. **Shreya Shankar** [57:03] Yep. **Guest 6** [57:04] Great. Thanks. **Guest 3** [57:07] Um, Shreya, if I may, um, you had a tweet recently about, uh, good task decomposition and how it can 1000x- **Shreya Shankar** [57:15] Mm **Guest 3** [57:15] ... uh, the accuracy of an LLM pipeline. Uh, how do you think that plays, that interacts with, uh, an evaluation, uh, framework? Would you then have to evaluate each component separately, uh, or does it all tie in in some end of, end-to-end kind of way or some hybrid of both? **Shreya Shankar** [57:35] Great question. We're exploring this as a research topic, but the idea is to have each node in your graph kind of be a standalone, do a standalone thing that you can have standalone assertions for. And if you think about, you know, infinitely many inputs flowing through your pipeline, there's gonna be some fraction of inputs that fail at each step. Then you can track error throughout the pipeline and do some form of causal analysis to see like which is the most, which node is the biggest reason for why the most downstream node failed. Uh, so that's something that we're working on now, but the idea is that you should do decomposition in a way that like you can also in- validate the intermediates. Um, they should be somewhat of a unit task. **Guest 3** [58:19] So basically it would involve some bottleneck analysis of which nodes, uh, like you said, are more, most responsible for the overall performance and focus on those. Okay. Got it. **Shreya Shankar** [58:29] Yeah. **Guest 3** [58:30] Thank you. **Host** [58:35] All right. I think that's the hour. Uh, we're a little bit, uh, slightly distracted in the dis- Discord chat with, uh, Mira Murati getting... Well, she says she resigned, but we don't know. ### Wrap-up **Host** [58:48] Um, not much speculation we can do. Uh, yeah, I think that's it. Thanks, Eugene. **Eugene Yan** [58:53] Wait, what's happening next week? **Host** [58:56] Uh, yeah. We have, uh, people signing up to do... I, I think Sam signed up to do a paper. He didn't set a date. **Eugene Yan** [59:03] Yay, Sam. **Host** [59:04] Would really like to dive into function calling, and the Berkeley team, I guess we're, we're just mainlining all the Berkeley PhDs. Uh, Berkeley team put out V3, if Sam, you're interested. Yeah, I can try. **Eugene Yan** [59:19] You got this. You got this, Sam. **Host** [59:20] Go easy on me. It'll be my first one. The eval- **Guest 7** [59:23] Wait, what? V3 what? Uh, I missed it. **Host** [59:25] Yeah, the B- BF eval. **Eugene Yan** [59:27] Berkeley. **Host** [59:27] Or Berkeley function- **Eugene Yan** [59:28] Function calling leaderboard. Nice. **Host** [59:29] So you want me to do the Gorilla paper and then the V3 of the V- FCL or, or just the V3? Yeah, I, I actually think the Gorilla paper is not that important anymore. Uh, it- Okay ... but they just, they just don't have papers for V2 and V3 of the function calling leaderboard, which they put out in the last two months. And this is like the industry standard now for function calling, uh, which I, I've been meaning to dive into, but if you want to take it up next week, it's yours. Yeah. I'll do it. **Guest 7** [59:56] Uh, yeah, just to, I guess as a hint, if you drop by either the Gorilla server or the MemGPT server, both, uh, uh, Charlie and Shishir are generally, like, usually available in there if you have questions, so. **Host** [1:00:06] Yeah. Both run servers. Uh, Berkeley should just have- Oh, cool ... 'cause there, there's too many servers. **Guest 7** [1:00:10] They should, right? Like, just consolidate it into one. **Host** [1:00:13] We, uh- **Guest 7** [1:00:14] Small guys **Host** [1:00:14] ... we're actually talking to LMSIS, uh, next week as well. Um, so we're trying to make a Berkeley day, um, to have like vLLM, like LMSIS, all, all these people. **Guest 7** [1:00:26] Hell yeah. **Host** [1:00:27] I, I don't know if we'll actually do it, but whatever. Okay. Thank you. That was amazing. I guess we... Eu- Eugene, we, uh- **Eugene Yan** [1:00:38] Thank you, everyone. Yeah. **Host** [1:00:40] Thank you. **Eugene Yan** [1:00:40] And we have, we have new, we have new, we have... Uh, I'm glad that we have a paper next week. Yeah. **Host** [1:00:45] Yes. Awesome. Okay. **Eugene Yan** [1:00:46] Thanks. **Guest 3** [1:00:47] All right. **Eugene Yan** [1:00:47] Thanks to Shi- Shi and Shreya for all the feedback on the initial prototypes. See you, everyone. **Guest 7** [1:00:52] See ya. **Host** [1:00:53] Bye, everyone. **Guest 3** [1:00:55] Yeah. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.