Braintrust

The Great Evals Debate — Ankur Goyal & Malte Ubl

Dec 7, 2025 · 34:33

Ankur Goyal (Braintrust) and Malte Ubl (Vercel) debate whether offline evals are essential infrastructure or premature optimization for AI coding agents, arguing that the best teams deliberately invest in multiple feedback loops—offline evals, A/B tests, and vibe checks—to build effective AI products. They explain that modern evals are not about manufacturing golden datasets but pulling real user failures from production logs into eval suites to iterate faster. The conversation highlights how evals provide a 'first derivative' that enables aggressive shipping without regression fears, akin to unit tests in traditional software. Coding evals are uniquely verifiable (e.g., 'does it compile?') yet underutilized; Vercel uses them in RL pipelines to fine-tune models that fix trivial errors 100x faster than agentic loops. They also discuss how product managers encode domain expertise through rubrics and LLM-as-judge scoring, and why proprietary evals are competitive moats while public benchmarks serve marketing. The debate concludes that RL environments are a promising frontier for computer-use agents but require specialized expertise to avoid reward hacking.

⚡️Using RFT to Build Clinical Superintelligence

Jul 29, 2025 · 26:58

Brendan Fortuner, Head of Eng at Ambience AI, explains how the healthcare startup uses OpenAI's reinforcement fine-tuning (RFT) to build clinical AI assistants that help doctors automate note-taking and ICD-10 coding, saving up to two hours per day. Ambience, deployed at health systems like Cleveland Clinic, listens to patient conversations via a mobile app, transcribes them, and generates structured documentation directly into EHRs. Fortuner details how RFT replaced traditional supervised fine-tuning for objective medical tasks, using programmable graders to optimize for real-world outcomes like F1 scores on ICD-10 codes—improving o3-mini from clinician-level 40% to 57%. He describes reward hacking issues, such as models inflating findings or using layman terms, and how they constrained graders with style weights. The episode also covers domain expert vs. ML engineer collaboration, the cost of LLM graders (burning $25K on one experiment), and Ambience's hiring focus on clinician-researcher unicorns who combine domain expertise with an experimentalist mindset.

Production AI Engineering starts with Evals

Oct 11, 2024 · 1:56:16

Ankur Goyal, founder and CEO of Braintrust, argues that evaluation is the core workflow of production AI engineering and shows how his platform makes evals accessible to software engineers. Drawing from his experience at SingleStore and Impira, he details Braintrust's evolution from an eval tool into an end-to-end AI development platform used by Stripe, Zapier, Vercel, and other top AI teams. He shares market data: OpenAI handles over 95% of production workloads, fine-tuning is declining, and open-source models account for under 5% due to reliability issues. Ankur explains why he avoided building a vector database—the real challenge is permissions and joins, not vector search—and predicts o1-style reasoning will replace complex agent frameworks. He reveals Braintrust's differentiators: hybrid on-prem, TypeScript-first SDK, and declarative eval structures.