Ankur Goyal

The Great Evals Debate — Ankur Goyal & Malte Ubl

Dec 7, 2025 · 34:33

Ankur Goyal (Braintrust) and Malte Ubl (Vercel) debate whether offline evals are essential infrastructure or premature optimization for AI coding agents, arguing that the best teams deliberately invest in multiple feedback loops—offline evals, A/B tests, and vibe checks—to build effective AI products. They explain that modern evals are not about manufacturing golden datasets but pulling real user failures from production logs into eval suites to iterate faster. The conversation highlights how evals provide a 'first derivative' that enables aggressive shipping without regression fears, akin to unit tests in traditional software. Coding evals are uniquely verifiable (e.g., 'does it compile?') yet underutilized; Vercel uses them in RL pipelines to fine-tune models that fix trivial errors 100x faster than agentic loops. They also discuss how product managers encode domain expertise through rubrics and LLM-as-judge scoring, and why proprietary evals are competitive moats while public benchmarks serve marketing. The debate concludes that RL environments are a promising frontier for computer-use agents but require specialized expertise to avoid reward hacking.

Production AI Engineering starts with Evals

Oct 11, 2024 · 1:56:16

Ankur Goyal, founder and CEO of Braintrust, argues that evaluation is the core workflow of production AI engineering and shows how his platform makes evals accessible to software engineers. Drawing from his experience at SingleStore and Impira, he details Braintrust's evolution from an eval tool into an end-to-end AI development platform used by Stripe, Zapier, Vercel, and other top AI teams. He shares market data: OpenAI handles over 95% of production workloads, fine-tuning is declining, and open-source models account for under 5% due to reliability issues. Ankur explains why he avoided building a vector database—the real challenge is permissions and joins, not vector search—and predicts o1-style reasoning will replace complex agent frameworks. He reveals Braintrust's differentiators: hybrid on-prem, TypeScript-first SDK, and declarative eval structures.