DeepSeek-R1

Long Live Context Engineering - with Jeff Huber of Chroma

Aug 19, 2025 · 57:01

Jeff Huber, founder and CEO of Chroma, joins Shawn Wang and Alessio to argue that context engineering—not RAG—is the real job of AI builders, and that Chroma's modern retrieval engine is built for this new paradigm. Huber explains context rot, a phenomenon where LLMs lose attention and reasoning ability as context windows grow, motivating precise retrieval over huge context windows. Chroma Cloud is a zero-config, usage-based serverless vector database with separation of storage and compute, now powering 5M monthly downloads and 21k GitHub stars. He shares techniques like chunk rewriting and generative benchmarking for evaluating retrieval, and predicts LLMs will increasingly serve as re-rankers. The episode also covers memory as a benefit of context engineering, offline compaction for self-improving AI systems, and Huber's philosophy of building with long-term purpose, hiring engineers who care about Rust, TLA+, and deterministic simulation testing.

The RLVR Revolution — with Nathan Lambert (AI2, Interconnects.ai)

Jul 31, 2025 · 1:19:00

Nathan Lambert from AI2 explains the RLVR (Reinforcement Learning with Verifiable Rewards) revolution, arguing that verifiable rewards for math, code, and instruction-following are scaling more reliably than human feedback, and that open models like Tulu 3 can match frontier labs on core evals with just 10-15 tasks vs. hundreds. He traces overoptimization through three phases—control, RLHF, and RLVR—and warns that models learn to cheat unit tests unless reward design penalizes it. He analyzes o3's search-heavy approach (e.g., 80 websites per query), hybrid reasoning models like Gemini 2.5 and Claude, and predicts that pure reasoning models will become the default as inference costs drop. Lambert introduces a four-skill taxonomy for agent models—skills, calibration, strategy, abstraction—and says planning improvements often come from mundane data fixes. He advocates for fully open models, citing AI2's Olmo 32B as approaching GPT-4 level, but notes that building an 'American DeepSeek' requires massive resources and nonprofit constraints.

The Unreasonable Effectiveness of Reasoning Distillation: using DeepSeek R1 to beat OpenAI o1

Jan 24, 2025 · 23:35

Bespoke Labs founders Mahesh, Ryan, and Trung explain how they distilled DeepSeek R1 into Bespoke-Stratos-32B, a reasoning model that beats OpenAI o1-preview on math and code benchmarks using only 17,000 training examples—47 times fewer than DeepSeek's own distill. They achieved this in a 48-hour sprint leveraging their data curation library Curator. Unlike Sky-T1, which required rewriting unreliable QWQ traces, R1's coherent reasoning allowed them to skip re-annotation and simply filter for correctness. The team argues data quality matters more than quantity, and that smaller models (like their 7B variant) can also improve with better teacher data. They see this as evidence that reasoning can emerge purely from supervised fine-tuning on high-quality traces, without complex search algorithms.