[Paper Club] DocETL: Agentic Query Rewriting + Eval for Complex Document Processing w Shreya Shankar

Okay, thank you everyone for joining us for Paper Club today. Today, we will be talking about DocETL, documentation, uh, or rather Document ETL. Uh, this is a paper by Shreya. Big fan of her work, very practical work. Um, and of course, looks like there's another solid Eugene on this paper, so should be a great paper.

So the entire... I'm, I'm gonna try to go through it in a, in a few-- There, there's a lot of concepts to this paper. There's a lot of examples and a lot of results. I'm gonna, gonna try to run through it very quickly and try to finish at a twelve forty-five mark, and then we can, we can discuss what we learned and what we thought was useful.

So essentially, the, the premise of the paper is that we can use LLMs to process documents for us in a fairly hands-off-the-wheel way. Um, but the problem is, is that for fairly complex tasks and data, LLM outputs for what we want it to do, to do is fairly inaccurate.

So a lot of times what we want it to do, we just write a prompt, right? We just say, "Hey, you know, LLM, you know, given this archive, here are these ten archive papers. Find me everything that's relevant to hallucinations."

You know, we, we sort of do it in a very, uh, crude, ad hoc way. We-- every, every archive paper, what is the snippet relevant to hallucinations? We pull it out, and after we try to combine it, and then we try to extract some themes from it.

We may include citations, et cetera, et cetera. So what this paper is trying to do is, is try to provide formal vocabulary to try to do this. Um, and, and we'll see what a formal vocabulary is. So in, in a nutshell, they have an agent-based framework.

Uh, let's put the agent-based framework aside for now. We won't focus on that. Um,

so what they try to do is that they will define operators, and they will define directives. And I'll define what operators and directives are. And then they will try to optimize this. So the optimization is really called logical rewriting and agent-guided plan evaluation and, of course, optimization algorithm.

I mean, I'm oversimplifying it, but all of this is essentially trying to optimize the overall pipeline to try to get what you want. And they introduce directives and, um, and operators as well, which we'll get into right now.

So there's a lot of intro over here, and there's this very big diagram over here. Uh, I won't actually go into this right now. I feel like we need some vocabulary before we can go into this. We need to understand what the pinks mean, what the greens mean, and what the different maps, all, all these different bolded words mean.

So I'm just gonna jump right into it. I'm gonna skip the programming model. Essentially, what this is trying to do is it's trying to take database, database concepts and pandas dataframe concepts and try to apply them to shapeless, uh, documents.

Is Shreya on the call yet? Not yet. Uh, oh, Shreya is. Hey, Shreya. If I, if I say anything wrong, you know, just, just, uh, stop me and just, and just, uh, jump, uh, just jump in. Uh, I'm just starting to get into the operators right now.

So essentially, the operators are fairly straightforward, right? And I have a note over here. It's really trying to think about how can we apply data pipeline and database operations to LLM tasks. And the LLM task over here is really documents, right?

Um, and so at the very simplest level, we have the map operator. Essentially, the map operator and there's this language here that's a little bit, uh, not so straightforward. Applies an LLM-powered projection, known as a semantic projection, to each document and data-dataset.

What this means, and my interp-interpretation of it, is that given a document, we create new features out of it. So y-you can, you can look at this example here. Given a document, what are all the instances of police misconduct, and what is the name of the officer involved and a new, new description?

Essentially, given a PDF document, I want new columns for officer involved and new columns for a brief description of the misconduct violation. So you can think of map. Map is infinitely, uh, flexible. Given a document, extract the relevant text.

Given a document, s-extract your summarization. Given a document, translate it. Given a document, add a classification, et cetera, et cetera. So map is the, the workhorse of the operators here. Um, and then they also share about parallel maps, you know, where you can run maps in parallel.

So you can see one prompt could be extracting misconduct, while another prompt could be summarizing policies. Essentially, you just do a map on every single thing. Then after you do a map, you have to do a reduce. Essentially, okay, so let's say I've extracted all the different officer names, all the different incident dates.

I want to return it as a report, like write it as reports, right? Then we have to do a reduce. So map would spin up a lot of new columns. Reduce would take all those columns and try to generate something that's coherent and easy for a human to read.

Um, and then they also talk about batch folding. Essentially, they, they talk about, you know, sometimes a lot of this is bigger than your context length, and that's why we use the map reduce paradigm, where you chunk it up and then you do reduce.

But when they reduce, uh, sometimes it's larger than the context length for the LLM, so they do folding. They, they, they talk about folding and, um, and hierarchical aggregation, uh, but they apply f- uh, batch folding. Essentially, imagine here we have these six boxes and then we suddenly need to aggregate them.

Maybe we aggregate them two at a time and, and then we do something like that. And I think that's all very straightforward. It make, makes a lot of sense. Now, there's the third operator, which is, in my opinion, a very important operator, but a very difficult operator to get right, which is the resolve operator.

In a nutshell, this is the duplication. You know, Six has the name of Sean Wang, has the name of Six, has the name of YX Wang. How do we know that they are all Six? This is extremely, extremely difficult.

So compare the following two officer records from the off-- police documents, right? And then how, how do we know that they are actually the same name?

And then, you know, resolution has many different aspects of it. It could be sci-fi, science fiction- Sci-fi fantasy. Sci-fi fantasy is not really sci-fi, but depending on your context, you could consider sci-fi and sci-fi fantasy as the same thing.

So there's a bit of domain nuance to this as well. I think this is really, really, um, difficult to do. Uh, I have actually have a... So one, several things I've seen in production is that when you do, um, resol- resolving and deduplication, if your deduplication is too loose, everything deduplicates to a single huge mega cluster.

And that's when you get, um, that's when you have severity issues and, and so this is actually very d- very tricky to get right. And, um, in the paper, they actually talk about different ways, um, to do this, and they actually talk about a cheap way to do this, which we have also implemented as well, which is using semantic and code base, which we, we will talk about later.

But long story short, this is very difficult. Um, and then they have other operators, uh, very standard. Filter, essentially given a document, maybe we perform some kind of classification. Do we want to filter doc- do- the document? Or maybe you do some kind of map to extract the relevant sections and filter everything out that makes sense.

Uh, equi join is, again, um, I think it's, uh, really difficult. In a sense, it, it's sort of like resolve. Can you join these two concepts together? And, and then all, all of this all relies on schema. And then there's, they have a few auxiliary operators.

Essentially auxiliary operators means that it doesn't even rely on LLMs. Unnest is one of them. So for example, given a document, maybe you have created a lot of... Given a, maybe given a blog post, who are the, what are all the papers, uh, re- referenced in this blog post?

It'll return like an array, a list. Unnest basically splits that nest into individual elements. Okay? Uh, and then split is essentially, you can think of it as chunking. Uh, in the most naive way, it could be chunking in terms of number of tokens.

In a smarter way, it could be chunking based on paragraphs or chapters or sections or whatever. Or it could be even LLM sp- based splitting. And then gather is essentially gathering all the individual chunks. So for example, imagine you are summarizing an archive paper.

Imagine you're summarizing section by section, right? Like the previous section was the operator section, and now we're in the rewrite section. So it could be that when you're summarizing the rewrite section, you actually do need context about the operator section, right?

It could be it, it mentions certain abbreviations that the LLM would never know, but you have to provide that kind of context. So what gather does is by providing this peripheral information. So it could be as you're summarizing a document section by section, you collect some kind of map on all the various abbreviations as they've shown up so that you provide that context to the LLM so the LLM doesn't try to hallucinate what those abbreviations mean.

So those are the basic primitives of DocETL. I'll pause here. Any questions about the operators? I think it's essential to understand map, reduce, and resolve. Everything else builds on this. Um-

So, uh, this is a classic more, uh, not so much a question, more of a comment. My, my question, my real question is, was there a rank operator? I think, and the answer is no. Um, you could maybe compose the rank operator with, um, some kind of split gather thingamajig, but it's not a formal ranking system.

Uh, and it's, uh, and I even before AI, um, I had always had this strong view that, um, filtering, ranking, sorting all this stuff is, is kind of like the same layer in the API stack. Um, and, and should be, you know, kind of done together.

Um, and it... This is my number one problem with, um, with, uh, AI news right now. It, it, which is that all filtering is effectively a rex's and I have to rank.

Yes.

Um, so the, you only f- you, you filter just based on metadata, like hard numbers because you don't have any other better stuff. Like, if you could create a ranking formula, you would use the ranking formula rather than any other filter basis.

Um-

Well, um, there, there, there is a filter op, right? So when, like section two point-

There is another

... two in other ops, the filter operation independently retains documents from the input data set based on conditions specified through an LLM prompt. In some sense, that, like it's under, under ops if you scroll down a little. Um, the, this filter one seems to, you know, if you can specify conditions through an LLM and retain docs, that's a, that's a form of ranking, no?

That's how I interpreted this one.

That's-

As opposed to like... Yeah.

Yeah. I, I mean, uh, uh, I think there's a, there's a question of sort of filter then rank or rank then filter. Uh, I think in my strong-

Yeah

... view, rank then filter. Um, rank up-

Yeah. You, you, you can think of it filtering as in dropping items from an array, whereas ranking is me pushing things down or pushing things up. So I think it's slightly different. And, you know, that's a question for you, uh, Shreya.

Is there a reason why they're split into operators and auxiliary operators?

Oh, very simply, we just wanted to put the primary LLM-powered operators in one section, the one that people will probably spend a lot of time tinkering on, and the auxiliary ones are there to stitch them. So a lot of people want to do, for example, uh, reduce after map.

Well, they want to unnest after map and then do reduce there.

Yep. Okay.

Uh, the, the comment about ranking, we're def- we're working about, we're working on a sort operator. We haven't found, maybe Swix, we should chat about this. We haven't found a really compelling use case from any of the users yet.

Really?

And a lot of... Well, a lot of-

Right

... people's use cases can simply be solved by, you know, applying like embedding based similarity to some query vector and then ranking by that similarity. I would, I wanna know kind of- I, I think Swix's case is interesting.

Like, in what cases does it make sense to have this pairwise comparison interface, uh, be the primary way to, to steer that operator? So maybe like ranking articles in terms of what is most interesting to a human could be such of a case.

Yeah. So-

I, I think there's a lot of-

Sorry. Yeah, go ahead.

Go for it. Go for it.

Uh, I mean, like, uh, yes, yes on the what's, uh, you know, what's most interesting to a human. I mean, that-- this is standard Rexes. Um, I'm not convinced that it's has to be pairwise. Uh, I, I feel like pairwise maybe it's easy to, it- it's easy to do data entry, so maybe that's why.

But, uh, I don't know.

I'm actually strongly convinced that it should be pairwise. Uh, and we can debate that and see how it works. And I also think that there is-

Say more, Yujie. Why, why do you think it should be pairwise?

I think it's just more reliable and stable that way.

Hmm.

I, I think you, you don't have to go through all... I mean, if we do everything pairwise it's quadratic, right? But we don't have to go through all of that. We can sort of do it smartly. I think that-- and, you know, in Shreya's paper, they actually cover how they use embeddings and code base, uh, re-resolve, right?

You can think of that as also you can apply that if you tweak that a bit, that can be applied to ranking. But you can also tweak a, tweak it a bit. You can say that ranking, we actually have confidence levels.

If the similarity score is strong enough or weak enough, like if it's really good, like zero point nine, we know it's strongly related. If it's really bad, like zero point one, we know it's poorly related. But then there's a lot of stuff that's in the middle that then we can use an LLM power ranking.

We can go into that a bit, but this paper is huge and I want to try to go through as much of it as I can. Then, now, so now that we've covered operators, um, they propose rewrites. So actually I have a question for you, Shreya.

When you say rewrite, do you mean rewriting the pipeline or do you mean rewriting the document? What does rewrite in rewrite directive stand for?

Rewrite... This is a good question. No one's ever asked this. Um, rewriting the pipeline.

I see.

So say you have a pipeline that is just a map operator. Many people have this pipeline. They have a document set, and they want to extract a bunch of metadata from each document. Like, I have a bunch of legal contracts, and I want to extract as many clauses of interest as possible from them.

That would be, you know, you can program that as a map operation, a single map operation to go through every document and extract, you know, all your fields that you want. Um, when we say rewrite, we acknowledge that this operation might not work when you-

Makes sense

... execute it. What if the document's too long? What if you're extracting too many fields? It's too hard. Um, so you want to write it into more complex sequence of operators.

So is my mental model right, that the rewrite directives are somewhat in the realm of optimization already?

Yes.

Okay.

Exactly.

All right. So now, now we're gonna talk about rewrite object, ob-ob-objectives. So let's again take our example of this archive PDF. We want to extract what the key themes are. A noob like me would just upload to Claude AI and Claude AI, what are key themes?

Now, we can rewrite that to be smarter. We can rewrite it into a map, basically map the entire document into alt text, and then we can do split. We saw the split operator. And then on the splits, we can do a map on extracting the key themes, and then we can do a reduce, right?

To try to reduce all these key themes. Again, the, the point here is that the, if we give the LLM smaller chunks, it is better able to pay attention and therefore come up with a more rich, more comprehensive and more factual summary.

But you can imagine the search space of creating such a pipeline is infinite, right? The search space of chaining together all the different Lego blocks, of putting together all the different AWS services is infinite. So what they propose is a methodology to help us try to do that, um, semi-automatically.

So I won't focus on all the different things that they do in rewrites, but I'm going to call out what I think is absolutely essential. So rewrite directives. The first one is data decomposition. So essentially, um, when you have very large documents, uh, and there are many documents, and then you just have to decompose it.

The very standard pattern they share is document chunking, which everyone has been doing. This is standard. Uh, that said, I actually do want to share, um, this example whereby I was asked to help with a pipeline and I was able to improve, um, downstream metrics significantly by twenty to fifty percent by removing chunking.

So every now and then it's, it's, it's good to rethink, "Hey, do we actually need chunking?" In, in such cases, I think Shreya's example here is that it's way beyond the context si- window size, and so you definitely do need it.

So you can see they do chunking, so you can map, split, gather, map, reduce. And you can see the optimization. There's many, many different ways to optimize. I'm just going to pick, uh, B here. So you can, you can imagine we could split it and then we could gather all of it, right?

After we split it, we gather all of it. But what they're proposing is that we split it, we create a summary, right? And then we create, I can't remember what H stands for. Um, hierarchical information. Hierarchical information. We create a summary and create hierarchical information.

So we enrich the splitted chunks, and then we gather it, right? So you can imagine essentially all pipelines now are LLM pipelines. LLM is enriching the data, LLM is cleaning the data, LLM is filtering the data. It's going to be a bit lossy, it's going to be a bit stochastic, but I think we will figure it out in the next one to two years to get it to a more reliable state.

So they actually share, um, some very insider ghost knowledge. When splitting a document-

Yeah

... there are some kinds of context that are very useful. I fully agree with all of this. Document-level metadata. What are all the different abbreviations? Who's the author, et cetera. Hierarchical information. Summaries of neighboring chunks. Summaries of previous chunks are actually very valuable, so the LLM has enough context and doesn't make shit up.

Um, and then they also have, um, different patterns here, document-level metadata. Uh, maybe I'll just go through one of them. So example, in this case, okay- Before you split it, you extract metadata that's relevant to all chunks. So by extracting this abbreviation metadata, author metadata, you make sure that all chunks, when you're doing map on all of these chunks, you actually have all this context.

The LLM has all this context and is able to do a more reliable job. Uh, and then they have many, many different other patterns that you can consider. Now, the next thing is multi-level aggregation. So you need to aggregate it.

Essentially, you have... Let's imagine you have hundreds of chunks. You are summarizing a movie, and you have all the different movie cut scenes. And imagine we j- don't have Gemini to do this for us and just upload the entire movie.

Imagine you have to do it. Uh, we have to chunk it up. So what they propose here is to aggregate, is to chunk it up, uh, aggregating the data at a finer granular. So essentially, it's like, okay, we have scene one, uh, or le- let's put it another way.

Imagine you're trying to summarize a series of... Oh, wow, this is a bit close. Uh, no. Imagine you're trying to summarize the Harry Potter movies. So you could first summarize the first Harry Potter movie, first summarize every scene in the first Harry Potter movie, roll it up to a summary of the first Harry Potter movie, and then roll it up to a summary of the entire Harry Potter movies.

So that's hierarchical aggregation. And then they have several things here, like LLM-centric improvements. Um, there's this term here, gleaning. So it's prompted with the previous inputs and outputs and asked to improve the inputs. I mean, to simplify... Oh, sorry, Shreya has a hand raised.

Oh, I didn't want to interrupt you. Sorry. But-

No, go. Uh

... your point about chunking I thought was very interesting in that sometimes it's beneficial to chunk and sometimes you should not chunk. And we have observed this in a number of workloads, and the insight that we've gained is that we will never know what...

It's all task specific and data specific.

Exactly. I think so.

And, and we are so further convinced that you kind of need some optimizer to explore these different choices for you and come up with something reasonable. But anyway, it's just a meta, meta comment on it's not the rewrite rule that, you know, if you apply it, it always works.

It's-

Yeah

... that if you apply it, it sometimes works. It sometimes works a lot better, and you need some way to kind of try a lot of these things automatically for you.

Agreed. Vips, you have a hand.

Yeah, a little follow-up to that point. Um, somewhere, I think on like the first page where it talks about chunking and performance, there was just like a handful of dumps of citations of different papers. So like, you know, uh, there was this section, I think, on the first page, "Recent work has shown that LLM performance degrades considerably as length increases," citing a paper.

"They can be distracted," citing a paper. "Pay attention more to certain," paper, cite a paper. Or, "Fain- fail to gain holistic understanding," and then there's like four citations. So pretty cool if anyone wants to dig into that. There's like eight citations on the first page that talk about this.

Yeah.

And then, um, another little comment, actually. So you skipped over a few of the chunking strategies.

Yep.

Um, so there's stuff like document level extraction header. I thought the third one was kind of interesting, where you have chunk filtering. Uh, I haven't seen this approach too often, so just if people haven't read it, it's an interesting little one that popped up.

So as you chunk, whether it's semantically or whatever it is, you can filter out sections. Uh, not many other preprocessing handlers seem to do this, but the example here is pretty clear, right? So as you break up chunks, you can filter out stuff.

So for example, in an archive research paper, you can filter out citations and references that aren't really relevant to the core information. It's just one of those chunking strategies that you don't see too often, so, you know, figured I'd add that in.

Yeah, 100% agree. All of these bolded mini headers here, they're all v- very valuable. I just don't have the time to go through them.

Yeah.

But you should, you should read through them. I think these are practices that they have gained from working on very, very difficult real-world problems to, to just get the level of performance up. Um, and you can take a lot of inspiration from this.

Shreya, you have another hand raised.

Oh, s- small comment again. For the police misconduct data, often we have like 1,000-page records, documents where it's just page on page of like random image of like a street sign or something. And if you just drop those, your processing, your accuracy just stays the same and you save so much on cost.

So that's where the filtering came from.

Yep, very valuable. Um, now they have LLM-centric improvements. The, this one here is gleaning, which is prompted, prompted with the previous inputs and outputs and ask it to improve it. You can think of it another way. If you have an, a valid...

So that's one way. If you can, you can imagine have a validator in the loop or essentially like code. You just copy, "Hey Claude, here's my error message. How do I fix it?" Uh, this is very similar to that.

Given the previous input and output and error message or some kind of feedback LLM, get it to improve on it. So this can, you can iteratively improve on this. And I think, uh, behind, later we have an experiment result where they try to do this four times, and you can see how it tapers off.

But for some, in some instances, it just keeps improving, uh, as, as many times. So initi- initial... So how this is, how this is done... Oh, actually, they do have an evaluator in the loop. Okay, so how this is done.

Okay, you pro- pre-process, you process it. You get the op- original map operation, like summary of the, um, paper, and then you have a validator try to evaluate it and provide some feedback. Now you provide the previous summary and the feedback and maybe the context, and then try to refine on it.

And, and then you just do through this many times. So you can imagine now all your SQL pipelines, all your data pipelines now have intelligence sprinkled in where it can, it can, it can map, where it can map on all this.

We have a validator in the loop, so users can have a more capable model to be the validator. Yeah. Yeah, I, I, I think I missed, uh, the part about being a validator in the loop, um, because I thought it was just asked to improve the outputs.

Yeah, but again, fully agree. I think this cannot be done without an evaluator or vali- validator. It's absolutely essential. Um, then they have this thing, duplicate key resolved. Uh, this is the problem, right? LLM outputs are not canon- canonicalized.

So for the same, for the same person six, they may have very many different names out there. So you have to canonicalize it. Um, so they take semantic equivalent values of the key. Um, and I think they try to, uh, reduce the search space in the pa- I, I can't remember where it's mentioned, but they try, maybe they try to reduce the search space with embeddings, and they try to resolve this.

Uh, long and short of it, this is a very, very hard problem and you have to do this with fairly high precision. If not, you could be saying that, uh, Eugene Chia and Eugene Yen are actually the same thing, and Eugene Yen maybe just rai- raised ten million dollars when it's really Eugene Chia.

Uh, I actually don't know how m- how much Eugene Chia raised. I'm just making an example where precision is quite important. Oh, Vip, you have a hand raised.

Yeah. So when I read the secti- uh, section about gleaning, I saw the same thing. So I understood the... Basically on the page you're at, that-

Yeah

... there's a, there's a validation model. So there's a, there's a validator. But then it also says that, um, that first paragraph, they employ a separate validator and data generation. So, uh, the one below this. Next page.

Mm-hmm.

Yeah, I didn't really see where this data generation LLM is. So I understand that, you know, you, you do some pre-processing. You have a validator to check how it worked. Then you have N number of steps to do this generate, like, you know, do your whatever processing-

Mm-hmm

... validate process, validate. But I don't get where this data generation LLM is. And then it seems like data generation is just, that's the only reference of it. So I didn't know if this is like... I just didn't see it too much in this-

No, it's ambiguous from us. I just meant the data processing LLM, the normal LLM that you use. So-

Yeah

... how does a traditional operation work? Look at figure four. You just have the output that comes from the data processing LLM, nothing more. What gleaning does is it adds that validation agent and it creates a loop of, you know, some finite number of iterations or whether the validation agent said it was good.

So you can just bound the number of times it refines and then the final output. That's all.

So i- is it using that original step again? So it says it's a separate validator and data generation model.

It's... You can specify two different models. You could also specify the same model architecture. But the point is, the validator agent has a different prompt that has a specific instruction of, you know, does this contain all the officers or instances?

And the validator agent's job is to say yes or no with optional feedback. The data processor LLM's job is to give the answer. That's why we dis- have a distinction between the two.

Got it. Got it. Got it. Okay.

Good question.

And Shreya also has an interesting point that it's much easier to verify the output than generate it. I actually observe the opposite. Or maybe for some-

It depends on the task.

It depends on the task. Like for classification tasks, it's, yes, it's easy. For like factuality or comprehensiveness or relevance, whatever, it's a lot harder.

A lot of the synthetic data gen papers like Orca 3, Wizard LLM, they show that models are better at verifying output than generating output. So-

Yeah

... you can have a weaker... Yeah, so it's shown in other work too. But in- interesting note, you know. It, it does depend on the task.

So next we have project synthesis. Well, this is, um, tricky. I think it, what it means is that it's hyper-pipeline optimization. So essentially, we don't really know what is going to be good. So you can imagine that we have very many different ways on try- how to do this.

Um, and we can find that, you know, agents are... It, it's really hard for, if you have a pipeline generating agent, it's really hard for an agent to try to figure out which of, which pipeline is going to work, uh, in the first place.

And there are different, there are infinite ways that you could build this pipeline. Uh, so they propose, they, they propose several ways to do this. And of course, the first thing is chaining. It's like iteratively, uh, chain LLM calls to get the ta- to simplify the task.

Essentially, this is what people say, break up your task so that every prompt only does a single thing. So it could be extracting entities, summarizing key points, and then generating recommendations. Um, another one could be isolating. So for example, maybe, uh, what es- maybe a question would be, what sections of this paper talk about hallucination?

Um, and then so you could reduce the scope of work to be done, uh, on the doc, whereby you could just classify the documents, maybe just classify sections, classify paragraphs as related documents, uh, hallucination or not, and then just flag it up.

So there's a lot there. Um, and, and, and I won't go through all of this. Uh, but anyway, that is the end of the rewrite section. Any questions there

on rewriting? Because now we're going to go into the optimization step.

I'm happy to explain projection synthesis, if that's helpful.

Yeah, go for it.

Yeah. So this is very similar to the chain of thought mindset that you have in LLM prompt engineering, where if you have a very complex task, often what can help is if you break that down into two LLM calls.

Now, you can generalize this to say all this means is you have a map operation that does partial work and then your original operation that does the task to maybe make it easier. And you could do this for any operation.

You can have a map before, uh, another map. You can have a map operation before another reduce, before another filter. Um, in this police misconduct case we've used, we've, we've seen this very helpful around whittling down, like summarizing the document into the relevant points needed to then answer the hard question of misconduct.

So you'll take a very long document and the first map operation that's synthesized will just extract information that is relevant to police officers at all. Kind of, you know, making it a smaller document and then extracting, summarizing the instances of mis- misconduct is much simpler to do on that very focused document.

So that's kind- that's an example of projection synthesis and projection means map in the data world, so you can apply that to kind of any operation. And you can have the LLM write that. You can have a LLM take a task and try to break it down into multiple steps and synthesize that map projection.

Thank you, Shreya. Um, so any questions on rewriting directives?

Okay. So now we're going to go into the optimization aspect. So the key aspect here in the optimization aspect is that, okay, you sort of declare p- an initial pipeline and then DocETL will try to run that pipeline, get initial evaluation scores and then, uh, it will try to optimize that pipeline for you.

So there are two key agents here. There's the generation agent. So what a generation agent does is that the generation will actually generate rewrites. So when, when, when we talk about generating here, the generation agent is going to generate the rewrites and the alternative pipelines, the pipeline options.

And then you have validation agents. And again, this is, this is, uh, another example, right, of how evals and validation are just fundamental to this because without the validation agent, you would not be able to assess and measure the different, the different pipelines, how, how the different pipelines do, and then how to compare between them.

So there's two things. You generate pipelines. Second, you evaluate each step of the, maybe you evaluate each step of the pipeline and you evaluate the overall pipeline, and then we try to see which one performs well. So, there's a lot of algorithms here, but I found, I find that the paragraphs here actually explain it.

I found it far easier for me to understand. So essentially, you can imagine that there's a pipeline, right? You have a pipeline. Again, let's, let's just stick to the police document, uh, police example pipeline, all the different, uh, examples of misconduct.

So what you first do is you use a validation agent to create a custom validation prompt, right? And I, I don't know if we, we do need inputs and labeled outputs here. Um, Shreya, do we actually need inputs and labeled outputs here-

No

... to get the validation prompt?

Nope. I mean, you could. We, we don't support that because nobody has ever come to us with labeled examples. The other thing that's hard is sometimes there are newly synthesized operators as part of rewrite rules, and those are certainly not going to have any labeled examples.

Yeah, I mean, in, yeah.

Um, but one of the things we are exploring is we have an interface for DocETL and we're exploring kind of interactive optimization of pipelines. So if we synthesize a new operator, we might ask a human to label, um, some examples and then use that for the validation prompt.

But that's, you know, in the works.

I think that's, I think that's a good idea and, and that's, that's the entire crux of Align Eval, which I shared with you. Um, I'm, yeah, I'm of a slightly different take. I feel like we do need some set, set of seed human labeled data.

Um, okay, so first thing, we s- use the validation data. We have a custom validation prompt and then what a validation agent will do is that you will sample some outputs from the initial pipeline or from chunks of the initial pipeline to see if there's room for improvement.

Okay. If there is room for improvement, what we do is we rewrite, we-- They have rewrite rules where they apply to the sub pipeline or the individual operation or the entire pipeline itself. So then what that means is that, and then they have this thing which they call recursive improvement, in the sense that when you create new optimizations for, for a rewrite, you immediately optimize them before co- uh, before continuing the rest of the optimization.

So what it means is that from a pipeline, imagine a pipeline goes from left to right, you, you, you eat the, you start upstream, you iterate on that, and you go downstream. And this can take as much time, right?

Compute is free. You can just run it overnight. Um, and of course it'll just cost a few hundred dollars, but that's fine, uh, compared to how much human time it would take. So now you have multiple candidate plans.

Let's just say you have 10 candidate plans, uh, and then you take all these 10 candidate plans. You first execute each plan on a sample of the data. You don't want to execute it on all your thousands of data sets.

You just execute on maybe 50, and then you use the validation agent to rate each output. So now after you rated this sample, these different sample plans, uh, then you take the top K, uh, and then you do, I think, full evaluation.

Oh, no, it does pairwise comparisons across these top plans and the plan that wins most is the winner. And of course, you select the optimized plan to do that. I think let's go through an example. I think an example here is helpful.

So here is the police misconduct data set. So, um, where is... So the current, the current, the current pipeline is this, right? The team has a domain-specific clustering algorithm and then human annotation to de-duplicate officer names, right? And, and this is, this is their base, the, this is the baseline pipeline that the team has.

So what they did was that DocETL, uh, synthesized various pipelines. The first one was to extract misconduct summaries. Uh, in, in, extract the name and the mis- and the summary of the misconduct before trying to de-duplicate on the officer name, before trying to resolve the operation, operate, uh, officer name, and then it summarizes it.

Um, then you can see here's a new...

Was this synthesized by Doc- DocETL, Shreya? DocETL T?

Uh, good question. So we are redoing this entire eval, by the way. So for the, we're submitting this to VLDV December 1st. You'll have, you'll see an entirely new eval for the next version. Um, but for this case study, the DocETL STO Are all synthes-- like, candidate plans from DocETL's optimizer.

Yeah.

And when we were exploring them, we were like, "Oh man," like, "I don't even know which one is gonna be better." Truly. Like, you read them, you have no idea. Um, so we compared them all. Uh, DocETLO was the one that was actually selected.

So the, you can see that they generated several plans. I don't know how many hundreds of plans they generated. Uh, they employed, uh, evaluated, the author evaluated fifty. Um, I recall they had certain numbers here. So, okay, yes.

Uh.

Okay, should I-- I don't know. Okay, so they had two hundred and twenty documents, and the baseline was two dollars, while DocETLS each cost a little bit more money. Uh, running the optimizer, right, they used the optimizer. I don't know how many plans the optimizer generated, but it cost approximately a hundred dollars and less than half an hour.

Just go get lunch, and you come back, and you get your optimized pipeline. Um, and the optimization cost was just a hundred dollars. How many pipelines? Oh, perfect. Here it is. Um, it generated two hundred pipeline variants, and all of this was just evaluated with the validator prompt.

Um, yeah. So I think that's an example of how the optimization step runs. During a pi-- a bunch of pipelines, validate it. And of course, uh, vali- optimize the subcomponents within the pipeline and the pipeline itself. So now that we've gone through this, we can now discuss figure one.

Uh, we had to go through a lot of it, but I think now we can discuss figure one. Figure one is, you can imagine all, all of these are police documents, and we want to extract, um, the names of officers and the misconduct.

So you can see this is the user-defined map, right? Okay, and maybe if I'm a user, my initial, um, prompt is very basic. Given a PDF, just extract any cases of misconduct. So now we apply the rewrite directive.

Uh, we apply a rewrite. And the rewrite could... Of course, we have the baseline, which is no change. We could do projection synthesis, which by we, we extract name, extract summary, and then after that, do another one. Or we could do decomposition, where we split the document up together and map.

And then all of these, um, all the pink boxes are where the agent will automatically, uh, rewrite it. And then the green boxes are when the plan is selected. And NC here stands for NSync. No, I mean, it stands for no change.

Essentially, if it's no change, we just accept it. And y-y-you can see we just, we just go through, uh, from left to right. And that is just for extracting the any cases of misconduct. After you've extracted these cases of misconduct, you can see Officer A, Sergeant B, and Officer C, um, then now we need to reduce it.

And again, we apply the same thing, uh, down the stream. So we start from left to right. Every time, every time we generate something new, we validate it, and if it passes, if it's good enough, we move on to our next one.

And so it just iteratively goes down the entire stream. So now that is, um, the section on op-operators, rewrite directives, and optimization. And that's all summarized in this document. Any questions?

I can answer RJ's question. It's easier than typing it out.

Yeah, go for it.

Yeah. So RJ asked a question, "I get how you can generate good evaluation prompts for the individual operators, but why do we trust an end-to-end auto-generated trust?" Or sorry, "an end-to-end auto-generated judge?" I, too, was very skeptical, but then I saw the kinds of tasks that the police, the journalism team here at UC Berkeley was running, which was, you know, extract all cases of misconduct, where they have defined misconduct in the prompt.

Very precision recall-oriented tasks here, um, which you can have an LLM judge pretty ambiguously determine whether one output is better than another. Like, did one output extract more names than an other? LLM as judge is super good here.

Um, so I was, I was-- I think what really sold me was having an optimizer like this that off the bat improved performance for the actual people who are working on these tasks, who are gonna be authors in the eventual submission just because they've contributed a lot to kind of helping out with the evaluation here, um, and kind of guiding what they find useful.

But it's, it's... Once you kind of see it work, it's kind of hard to go back. I know Eugene Yan also has a experience with LLM as judge here.

Uh, I, I strongly, I strongly believe it could work. I mean, last year, s-this time last year, or maybe a few months, like in June last year, I didn't think it could work, but I've been debating Shreya a lot.

I've been trying a lot. I think when we simplify it to binary classification metrics, I think it can work. And I think a lot of things can be simplified to binary. Like Shreya mentioned, precision recall. I think a lot of things can be simplified to binary classification metrics.

And I've seen evidence of it working. Well, you may have to apply chain of thought. You may have to include a few few shot. You may have to do a bit of self-consistency or ensembling. But I think the technology is getting there.

Eugene Chia, you have a hand raised.

Yeah. I think the other important aspect to consider in this scenario is that what is the failure condition? So in this case, right, uh, if let's say, if let's say, uh, this pipeline fails, right, it still re-reaches the, the journalist and becomes a human-in-the-loop kind of situation.

Like, the failed documents ca-can be ignored safely because th-they, they achieve their goal with the, the successful documents. Uh, well, in this, in the case of, like, using this pipeline in a more Q&A, uh, agent style for help desk, right, a failed answer i-impact is a lot more drastic.

So depends on the, the use case.

Agree. Nibbu.

Yeah, a quick, uh, comment there. So you mentioned you've seen examples of LLM as a judge working if you break it down to a binary problem, and then you might need to include few shot examples and stuff. And you said ensembling.

Um, can you explain a bit more about what ensembling helps for LLM as a judge?

Yeah. When I talk about ensembling, the main ensembling I'm thinking about is, oh, gosh.

Uh, I wish I could find a paper as easily as-

No, but just at, at a, at a high level, you know?

Yeah. The Po- Po essentially, the, the Po paper essentially talks about, um, just combining multiple use- using multiple weaker LLMs. You can get, uh, as, as good as, uh, an LLM judge as, um, GPT-4.

Is that-

This is the paper from... Mm.

Okay. So is the benefit there trying to save on performance of a large model or, like, optimization that you can't run a large one? Like, mixture of agents kind of shows how from, um, together shows how you can use mixture of agents of small models to match performance of a larger model.

Yes.

Which we've deployed 'cause it has to be on-prem. So ensembling models leads to smaller models being able to do efficient judging than larger models? Is that the takeaway?

Yes, that's correct. So in this paper here by Cohere, what they did was they have a reference, um, model, and this reference model, GPT-4, and then essentially what they did was they ensemble Command R, Haiku, and GPT-3.5. And I can't remember what the op- op- the, um, uh, I think the ensemble was just majority voting.

And they were able to have higher Kappa score, higher correlation to humans compared to the strong reference, uh, the large language model.

Interesting. Makes sense.

Yeah. Uh, I, I think it makes sense that, you know, all these smaller calls can be parallelized and then you can do ensembling, yeah, so. And it's faster and cheaper.

So sometimes they can, sometimes they can't be parallelized. For us, we see a lot of, um, parallel, not parallel, sequential chaining-

Mm. Yeah

... add significant performance. It just adds performance bottleneck in terms of time and latency end to end. But, like, it's a necessary trade-off for what we-

Yeah. Okay, someone asked, "Can you link to this paper?" I don't know if I can link to this paper. You can, if you imag- if you, if you've been in-

I can find out

... our paper clubs, Eugene Yen just does not multitask well, as you saw while I was struggling to find this paper. Uh, but I've pasted you the link. If you search it up on Google, I'm sure it will lead you to the archive.

I'll go to-

Um, okay. Okay, back to DocETL.

I did have a-

Oh, go for it.

I had a, I had a quick high-level question since we're approaching the end. Uh, more so for Shreya. So i- if you can give a sort of high level how we go from datasets to extracted whatever. So, like, currently you can break up documents by chunking, extracting stuff.

There's some basic filtering. Just high level, what happens here? And then in that optimization step, it didn't really click for me where you have, like, the cost go from a couple dollars to $100 and you're generating a bunch of pipelines to optimize this stuff.

But at a high level, like, one, what is the DocETL? And then, two, w- what are these optimizers doing?

Let me... I'll answer the second question. The optimizer's goal is to rewrite the user's pipeline into something that is more accurate. And in this process, we explore, the optimizer explores a lot of plans using LLMs to create those plans, to verify, um, to evaluate those plans.

So that's where the cost goes up. The reason it was $100, if, if I ran the optimizer with GPT-4o Mini as the LLMs, it would be $10. But we used GPT-4o just because I think we did this at a time where Mini hadn't come out yet.

Um, anyways, uh, but LLMs are trending to be cheaper and cheaper. This project is a very high-budget project, but I think that in the future as LLMs are much cheaper, as open source LLMs get much better, um, we should, this cost should not be an issue for such kind of, um, approach.

And then your first question was what is, like, the high level, like, what's the vision for the project? Or...

No, I mean, just like what's the input outputs? Like I, from the title when I first started reading it and went through all this section 1, 2, 3, I was like it, it, I thought it was more so single document to better extraction.

Uh-huh.

Like pre-processing a document for a RAG system. And then once I get to the end, so, like, my whole background was, let's say I've got like a, you know, 10,000 documents that I want referenced in a RAG system.

Am I doing this on all of them? And then it turned into you can throw this at a dataset of a couple hundred docs. So, like, there was a bit of a jump there.

Yeah. So it's, this is very different from traditional RAG or Q&A or document processing for a chatbot. Like, the kinds of queries that people are, people want to use DocETL for, um, can be expressed as ETL style, um, sweep and harvest kind of, I wanna look at my entire dataset.

I would hire an, an analyst to look at my entire dataset and then just tell me things, um, generate me reports that I can then learn from. So for example, this officer and misconduct thing is much more digestible than, say, looking at all these, uh, millions of PDFs.

So in, in that sense, I mean, you can certainly apply chunking or, like, maybe a strategy used for processing one of the documents could be used for RAG systems, but the focus is not RAG. It's like building a kind of semantic layer on top of your unstructured data is the way that I like to think about it.

Got it. Got it. Thank you.

Okay, cool. Um, and I know we don't have time to go through the examples at the end, uh, but definitely do go through the examples at the end. Um, and you can see, okay, let's just, for the police example one, you can see they had three binary criteria.

Did, did this binary criteria come from the team that was doing the work?

Yes

Oh, okay. Like whether each officer name referred to a real person, if the summary included dates, whether each identified misconduct was, uh, was-

This is a very weak evaluation. So part of the reason we want to rerun this evaluation is just this task here doesn't have any ground truth. So when you ask the people, like, what do they care about? It's, they look at some of the outputs, they look at their data you've made, and they're like, "Oh, the officer names are fake."

So that becomes a criteria for them, because sometimes it'll be officer A, officer B, officer C. That's not, not real. That's not what they want extracted. Um, they'll read some of the summaries and say, you know, "Some of them aren't exhaustive.

I want them to include date, location, names, uh, summary of what happened." So I think like the, the criteria emerges from looking up the data and it's not necessarily a reflection of like how well the LLM did this task.

Um...

Gosh, I think Shreya must have seen this red Post-It note and preempted me.

I know.

No matter, no, no matter-

Yeah, but, but that, that's the question, right? From an academic standpoint, this is a underwhelming evaluation because there's no ground truth, right? So we're redoing our evaluation to be on datasets where we actually have ground truth from human annotators, but that's just not a practical setting.

Like, nobody's coming to DocETL with the ground truth, so.

That's why we do AlignEval to get them to provide ground truth and gamify. And then you can imagine, right, let's just say police misconduct. You can imagine there's two officers, Eugene Chan and Eugene Yen, and then we resolve them to Eugene.

And then now it becomes, um, it becomes a hi- uh, the risk is higher, right? Eugene Chow, Eugene Yen conducted some offense and then it's... So that's, that's, that's the problem where resolution is very tricky.

Yeah.

Not just on re- I, I think somewhere you mentioned resolution, you have a metric of recall. I actually think it's precision is actually more... It, it depends on your use case. I actually think precision is actually something that we want to be careful a-about.

We don't want to mess-

Yes.

Yeah.

Yeah. We, um, we do measure precision in the updated eval that is not on archive.

Yeah. Um, so yeah, they also have two other, um, examples here. Again, very valuable. This is really people doing the work, sharing the work they did, uh, what were the pain points and how they improve on it. Um, uh, just, just definitely do read this end to end.

You'll definitely learn a lot. And you can see this is the number of... Again, this is the number of iterations. You can see, um, be like, it just keeps getting better even on the fourth iteration or on the third iteration, though it may be expensive.

Whereas on some, some things like distinct games, uh, maybe after one or two iterations is good enough. Again, depends on your cost and how much, how much latency you want to incur. Um, so yeah. I think that's all.

Key takeaways, we can think about the operator operations we do as a few key operators, map, reduce, resolve, filter, rank, maybe. We can think about how we want to rewrite these operators together in a different way. And we, we also have a methodology on how to optimize these rewrites, uh, by generating pipelines and of course validate.

We need a validator, um, to validate every step of the way. Um, Shreya, anything I missed? Any, any key takeaways that I missed?

Crushed it. Wow. I'm so-

Well, thank you. Thank you, Shreya, and thank you for joining us.

Thank you for all spending so much time on this work. This work has been two years in progress, by the way, so.

Really? Wow.

Oh, yeah. And even longer.

I didn't know it was that long.

Yeah.

Really? Shoot.

Yeah.

I-

That makes me very happy that you enjoyed it

... I only knew about it now. Yeah. I, I, I'm super excited to seeing the full paper. Yeah.

Um, the, the EvalGen and all my other papers actually were, uh, part of this broader project. I've been working on this for so long-

No way

... that my advisor was like, "What are you doing?" You know, "Let's come up with something." So I was like, "What's our biggest problem right now?" About a year ago, a year and a half ago, I was like-

There you go

... "Validation. We need to come up with these validation agents. I don't know how to do it." So that's how-

EvalGen came about. Yeah, I, I love the EvalGen paper. I mean, I built an entire app inspired by it. Uh, but Eugene Chia, you have a Henry's.

Yeah. So I'm just going to do the shout-out for, for the three plus one, I guess. Uh, for newer folks who are not aware, and also for the recording, uh, the amount of meta in this room, right, and the active participants, right, is ridiculously high on this topic of RAG/LLM as a judge/Eval.

Shreya is the paper author and has been doing multiple pa- uh, banger papers on this, top-tier papers, uh, re- on this topic regarding RAG, Eval, and everything in between. Uh, if you want a list of her writing, you can probably just ask Eugene Yen and he'll produce a list for you.

Um, Eugene Yen himself, uh, undersells himself, but he writes extensively on LLM as a judge and helps teams build these solutions in the world's largest bookstore. Um, so he's probably seen some of the wo- biggest production use cases for, for this kind of deployments.

Vibu works at MindsDB, right? Uh, which is all about embedding such enterprise RAG solutions built on top of it, and has seen a lot of real-world productions for all MindsDB cl- clients. Uh, and so he provides more breadth across multiple use cases, uh, respectively.

And Sean, a-as you may know, uh, works on Small AI as well, uh, which does summarization and such on this kind of thing. So lots of high-level meta. Hence, you can see why they were the most active in their conversations as well.

Yeah, and I've dropped, I've dropped a few links about this. I think that you should definitely read Shreya's previous paper. Um, this paper is really, really, really good. Um, and what I really like about this, it actually has, um, UX, uh, UX research.

They actually had participants who... So you can hear what other people are saying. That's very useful. I also dropped another link to, um, so, a compilation I did on different LLM-as-a-judge, uh, techniques. So I think that's, it's helpful.

It's a lot, I realize, but it actually references a lot of other papers, so th- this can be helpful for you. Okay, and next week we have Eric. Thank you for volun- volunteering, Eric. Uh, we look forward to that.

And, um, you know, drop the paper that you're, you, you'll be taking us through in the Discord channel so that we have time to pre-read. Yeah, will do. Thank you, everyone. Thank you, Eric. Thanks. Thank you, Shreya, for joining us.

Thank you for being here.

Yep.

Bye.

Bye. Okay, Swix, do yous need to stop the recording or we just leave?

Uh, I would say y'all just leave. Uh, he, he sent, he made me the, he made me the host, but, uh-

Oh, he made you host. Can you stop the recording?

Uh, yeah, but then the thing is, uh, my session kind of bugged out and I need a refresh. So-

Oh, okay

... I think the host will, will, will time out eventually.

All right. All right. I'm-

This session is not the host.

Oh, shoot. Okay, I'm going to have to leave then. I have another meeting. All right, see you, man. Bye.

I, I might... I have a backup recording, I think. So if we lost it, then I can, uh, provide that. We'll see. Yep, yep, yep, no worries. Cool. Hey, was that you, Yics? Yeah, yeah. Are you the one with the backup recording?

Okay. Yeah, yeah. Cool.

All right, everyone, feel free to leave. I'm just waiting for the system to time out. Oh, no, Sean left his computer, uh, on on this.