Eugene Yan

[Paper Club] Who Validates the Validators? Aligning LLM-Judges with Humans (w/ Eugene Yan)

Sep 28, 2024 · 1:00:55

Eugene Yan presents the paper 'Who Validates the Validators? Aligning LLM-Judges with Humans' by Shreya Shankar, introducing EvalGen, a framework that helps developers iteratively align LLM evaluators with human labels by grading outputs, refining criteria, and tracking coverage (recall) and false failure rate (1−precision). Yan demonstrates his own prototype 'Label', which forces users to label 20 samples before unlocking evaluation mode and 50 samples plus one evaluation run before optimization, gamifying the alignment loop. The discussion covers binary vs. pairwise evaluation, the subjectivity of criteria, and Shreya Shankar's plans to integrate natural language feedback into EvalGen's next version. Yan advocates that labeling data is essential before deploying LLM evaluators in production, turning the process into prompt fine-tuning rather than vibe checks.

[LLM Paper Club] Llama 3.1 Paper: The Llama Family of Models

Jul 29, 2024 · 1:23:46

This episode examines Meta's Llama 3.1 paper, detailing the 405B dense model, its scaling laws grounded on the ARC reasoning benchmark rather than perplexity, and the decision to train on 15 trillion tokens. Vibhu explains the training infrastructure: 16,000 H100s over 54 days with 419 interruptions, 78% from GPU hardware failures. Eugene Yan walks through the synthetic data pipeline—using Llama 2 for filtering, stepwise reward models, and Monte Carlo tree search to improve reasoning traces. Hassan shares building LlamaTutor.com with Together API, serving 4,000 visitors and 5,900 requests for about $12. The group also discusses quantization trade-offs (larger models degrade less), inference provider variability (Groq's non-deterministic temperature zero), and compares Llama 405B's performance to GPT-4o and Claude.