Shreya Shankar

[Lightning Pod] Evals: How to Improve AI Consistently — with Hamel Husain and Shreya Shankar

Mar 13, 2025 · 27:34

Hamel Husain and Shreya Shankar join the Latent Space podcast to argue that systematic evaluation (evals) is the critical missing piece for moving AI applications from demo to production. They estimate 75% of evals in the wild use LLM-as-judge, but 80% of those are not helpful without proper validation against domain experts. Husain advocates building custom annotation UIs (using tools like Lovable or Cursor) to speed error analysis, while Shankar emphasizes synthetic data generation that balances real-world data and LLM outputs, drawing on social science methods. They preview their free lightning lesson on March 21 and a four-week paid course covering the eval lifecycle: synthetic data creation, LLM-as-judge calibration, error analysis, and iterative improvement, with hands-on coding assignments. The episode also touches on trends like dedicated judge models (e.g., Haizelabs' Verdict) and the value of basic data literacy (e.g., pivot tables) for eval participation.

[Paper Club] DocETL: Agentic Query Rewriting + Eval for Complex Document Processing w Shreya Shankar

Nov 29, 2024 · 55:53

This episode of Latent Space Paper Club discusses DocETL, a framework by Shreya Shankar for agentic query rewriting and evaluation in complex document processing with LLMs. The paper introduces map, reduce, resolve, and auxiliary operators, along with rewrite directives like data decomposition and gleaning to improve accuracy. The optimizer generates 200 pipeline variants, costing $100, to rewrite user pipelines automatically. Key debates include when to chunk documents (sometimes harmful) and the reliability of LLM-as-judge for evaluation, with guests Eugene Yen and Vibhu arguing that binary metrics, ensembling, and context-dependent thresholds make it viable. Shreya emphasizes that validation agents are critical, and the work builds on her prior EvalGen paper, spanning two years of development.

[Paper Club] Who Validates the Validators? Aligning LLM-Judges with Humans (w/ Eugene Yan)

Sep 28, 2024 · 1:00:55

Eugene Yan presents the paper 'Who Validates the Validators? Aligning LLM-Judges with Humans' by Shreya Shankar, introducing EvalGen, a framework that helps developers iteratively align LLM evaluators with human labels by grading outputs, refining criteria, and tracking coverage (recall) and false failure rate (1−precision). Yan demonstrates his own prototype 'Label', which forces users to label 20 samples before unlocking evaluation mode and 50 samples plus one evaluation run before optimization, gamifying the alignment loop. The discussion covers binary vs. pairwise evaluation, the subjectivity of criteria, and Shreya Shankar's plans to integrate natural language feedback into EvalGen's next version. Yan advocates that labeling data is essential before deploying LLM evaluators in production, turning the process into prompt fine-tuning rather than vibe checks.