SQLite

Long Live Context Engineering - with Jeff Huber of Chroma

Aug 19, 2025 · 57:01

Jeff Huber, founder and CEO of Chroma, joins Shawn Wang and Alessio to argue that context engineering—not RAG—is the real job of AI builders, and that Chroma's modern retrieval engine is built for this new paradigm. Huber explains context rot, a phenomenon where LLMs lose attention and reasoning ability as context windows grow, motivating precise retrieval over huge context windows. Chroma Cloud is a zero-config, usage-based serverless vector database with separation of storage and compute, now powering 5M monthly downloads and 21k GitHub stars. He shares techniques like chunk rewriting and generative benchmarking for evaluating retrieval, and predicts LLMs will increasingly serve as re-rankers. The episode also covers memory as a benefit of context engineering, offline compaction for self-improving AI systems, and Huber's philosophy of building with long-term purpose, hiring engineers who care about Rust, TLA+, and deterministic simulation testing.

[Paper Club] Who Validates the Validators? Aligning LLM-Judges with Humans (w/ Eugene Yan)

Sep 28, 2024 · 1:00:55

Eugene Yan presents the paper 'Who Validates the Validators? Aligning LLM-Judges with Humans' by Shreya Shankar, introducing EvalGen, a framework that helps developers iteratively align LLM evaluators with human labels by grading outputs, refining criteria, and tracking coverage (recall) and false failure rate (1−precision). Yan demonstrates his own prototype 'Label', which forces users to label 20 samples before unlocking evaluation mode and 50 samples plus one evaluation run before optimization, gamifying the alignment loop. The discussion covers binary vs. pairwise evaluation, the subjectivity of criteria, and Shreya Shankar's plans to integrate natural language feedback into EvalGen's next version. Yan advocates that labeling data is essential before deploying LLM evaluators in production, turning the process into prompt fine-tuning rather than vibe checks.