[Paper Club] DocETL: Agentic Query Rewriting + Eval for Complex Document Processing w Shreya Shankar
Nov 29, 2024 · 55:53
This episode of Latent Space Paper Club discusses DocETL, a framework by Shreya Shankar for agentic query rewriting and evaluation in complex document processing with LLMs. The paper introduces map, reduce, resolve, and auxiliary operators, along with rewrite directives like data decomposition and gleaning to improve accuracy. The optimizer generates 200 pipeline variants, costing $100, to rewrite user pipelines automatically. Key debates include when to chunk documents (sometimes harmful) and the reliability of LLM-as-judge for evaluation, with guests Eugene Yen and Vibhu arguing that binary metrics, ensembling, and context-dependent thresholds make it viable. Shreya emphasizes that validation agents are critical, and the work builds on her prior EvalGen paper, spanning two years of development.
