[Lightning Pod] Evals: How to Improve AI Consistently — with Hamel Husain and Shreya Shankar
Mar 13, 2025 · 27:34
Hamel Husain and Shreya Shankar join the Latent Space podcast to argue that systematic evaluation (evals) is the critical missing piece for moving AI applications from demo to production. They estimate 75% of evals in the wild use LLM-as-judge, but 80% of those are not helpful without proper validation against domain experts. Husain advocates building custom annotation UIs (using tools like Lovable or Cursor) to speed error analysis, while Shankar emphasizes synthetic data generation that balances real-world data and LLM outputs, drawing on social science methods. They preview their free lightning lesson on March 21 and a four-week paid course covering the eval lifecycle: synthetic data creation, LLM-as-judge calibration, error analysis, and iterative improvement, with hands-on coding assignments. The episode also touches on trends like dedicated judge models (e.g., Haizelabs' Verdict) and the value of basic data literacy (e.g., pivot tables) for eval participation.
