Reasoning

Cooking with OpenAI’s Research Chief: AGI, o1, Evals, and Scaling Laws — Mark Chen

Jun 25, 2026 · 41:18

Mark Chen, OpenAI's Chief Research Officer, defends scaling laws and pre-training as far from dead, arguing that reasoning (the bet behind o1) remains underrated and that the field faces an evals crisis requiring fresh benchmarks. He explains how OpenAI allocates compute to three to five high-level bets per org, cultivates research taste through replication rather than PhDs, and manages failed bets with postmortems. Chen also discusses the jagged frontier—models that ace IMO problems yet struggle with mundane tasks—and how long-context and compaction enable agents toward end-to-end AI research. Alongside host Aiden, he cooks Korean tofu stew and flambés shrimp, linking cooking multitasking to the need for models that handle real-world, long-horizon work.

Scaling Past Informal AI - Carina Hong, Axiom Math

Jun 3, 2026 · 1:33:04

Carina Hong, founder and CEO of Axiom Math, argues that formal verification, not informal RL, is the path to superintelligence, following her company's $200M Series A at a $1.6B valuation and a perfect 120/120 on the 2024 Putnam exam. Axiom's system uses Lean theorem prover data and reinforcement learning to produce verified proofs, achieving a 99% pass rate on the Verina code-with-proof benchmark (187 of 189 problems). Hong contends verification is about 'scaling brilliance' — not fixing hallucinations — and that only verified generation can compound AI reasoning. She explains Axiom's open-source Axle API for Lean at scale, addresses why frontier labs like OpenAI have deprioritized formal math (team departures, strategy shifts), and outlines a vision where verified reasoning transfers from math to code, hardware, and eventually AGI through self-improvement. She also discusses the Earth sciences challenge of autoformalization, the difficulty of search in mathematical literature (citing the Erdos controversy), and why fragmentation in the AI math field is a bottleneck.

Marc Andreessen introspects on Death of the Browser, Pi + OpenClaw, and Why "This Time Is Different"

Apr 3, 2026 · 1:16:20

Marc Andreessen argues that AI is finally different from past boom-bust cycles because of four compounding breakthroughs: LLMs, reasoning (o1, R1), coding agents (OpenClaw), and recursive self-improvement. He calls this the '80-year overnight success'—decades of neural network research now paying off. Comparing today's AI capex boom to the dot-com crash, he notes that buyers like Microsoft and Google are cash-rich incumbents and every GPU deployed is already generating revenue. He hails Pi and OpenClaw as a Unix-like architecture that makes agents model-independent and self-modifiable. On open source, he calls DeepSeek a 'gift to the world' for its paper and code, but warns that entrenched institutions—unions, licensing, government monopolies—will slow AI adoption far more than technologists expect.

⚡️ Reverse Engineering OpenAI's Training Data — Pratyush Maini, Datology

Feb 10, 2026 · 27:02

Pratyush Maini from Datology reveals how the seahorse emoji question exposes that frontier models like GPT-4.1, GPT-5, and OLMo 3.1 now exhibit self-correction loops—a sign that reasoning traces are baked into mid-training data, not just post-training. By tracking response length surges four months after o1's release and corroborating with OLMo's data, Maini argues that foundation models must contain core capabilities like self-reflection, reinforcing the 'Fine Tuner's Fallacy': you cannot simply fine-tune for a capability; it must be in pre-training. He also highlights Datology's BeyondWeb project, which uses a source-rephrasing paradigm to transform internet data at trillion-token scale, achieving Nemotron-level performance in 2.7x less compute—making specialized pre-training accessible for enterprises. The episode concludes that 2026-2027 will see a shift toward domain-specific pre-training, as the cost amortizes when a smaller pre-trained model outperforms a larger fine-tuned one.

Captaining IMO Gold, Deep Think, On-Policy RL, Feeling the AGI in Singapore — Yi Tay

Jan 23, 2026 · 1:32:05

Yi Tay, who leads Google DeepMind's Reasoning and AGI team in Singapore, explains how Gemini Deep Think achieved IMO Gold by abandoning symbolic AlphaProof for an end-to-end RL-trained model. He details the on-policy RL philosophy—models learn from their own generated outputs rather than imitating others—and the critical role of self-consistency through parallel sampling and internal verification. Tay describes the IMO effort: four co-captains in different time zones, a one-week training sprint, a live competition in Australia where researchers punched in problems as they were released, and the tension of waiting for human scores to determine the gold threshold. He discusses why the team believes one model must subsume everything for AGI, the data efficiency gap compared to humans, and his hiring focus on raw talent and research taste. Tay also shares his personal fitness transformation—losing 23 kilos and improving HRV—as integral to research productivity.

[State of RL/Reasoning] IMO/IOI Gold, OpenAI o3/GPT-5, and Cursor Composer — Ashvin Nair, Cursor

Dec 30, 2025 · 45:13

Ashvin Nair, now ML lead at Cursor, traces his path from Berkeley robotics and an OpenAI Dota-era internship to OpenAI's reasoning team (which grew from a dozen to 300+ people) and explains why IOI Gold in 2022 felt like solving AI but didn't change the world—because RL doesn't generalize beyond training distribution. He argues most RL research from 2017-2022 overfit to benchmarks, rewarding complex ideas over simple ones that scale. At Cursor, he sees a unique opportunity for continual learning with policy updates every two hours and product-model co-design, keeping engineers in the loop instead of context-switching. His bet is that the next paradigm shift is continual learning with infinite memory: models experience something once and never forget it, storing millions of deployment tokens in weights without overloading capacity.

Greg Brockman on OpenAI's Road to AGI

Aug 15, 2025 · 1:08:37

Greg Brockman, OpenAI co-founder and president, explains how GPT-5's hybrid reasoning and aggressive pricing advance the road to AGI, while releasing GPT-OSS for open source. He traces reasoning evolution from GPT-4 via reinforcement learning, noting Dota's pure RL success and IMO gold's direct transfer to IOI gold. Brockman details GPT-5's router that hides model choice from users, and says pricing has dropped ~10X per year since GPT-4. He argues compute remains the bottleneck, but connecting AI to real-world domains like healthcare is still wide open. Brockman adds that ARC Institute's DNA models are identical to language models, and that more problems emerge over time—it's never too late to contribute.

The RLVR Revolution — with Nathan Lambert (AI2, Interconnects.ai)

Jul 31, 2025 · 1:19:00

Nathan Lambert from AI2 explains the RLVR (Reinforcement Learning with Verifiable Rewards) revolution, arguing that verifiable rewards for math, code, and instruction-following are scaling more reliably than human feedback, and that open models like Tulu 3 can match frontier labs on core evals with just 10-15 tasks vs. hundreds. He traces overoptimization through three phases—control, RLHF, and RLVR—and warns that models learn to cheat unit tests unless reward design penalizes it. He analyzes o3's search-heavy approach (e.g., 80 websites per query), hybrid reasoning models like Gemini 2.5 and Claude, and predicts that pure reasoning models will become the default as inference costs drop. Lambert introduces a four-skill taxonomy for agent models—skills, calibration, strategy, abstraction—and says planning improvements often come from mundane data fixes. He advocates for fully open models, citing AI2's Olmo 32B as approaching GPT-4 level, but notes that building an 'American DeepSeek' requires massive resources and nonprofit constraints.

⚡️Math Olympiad gold medalist explains OpenAI and Google DeepMind IMO Gold Performances

Jul 24, 2025 · 33:07

Dr. Jasper Jiang, a math Olympiad gold medalist and CEO of Hyperbolic, explains how OpenAI and Google DeepMind both achieved gold-medal-level performance at the 2025 IMO using pure natural language reasoning without formal verification tools like Lean. He recounts the timeline: a Friday leak about DeepMind's gold, then OpenAI's Saturday morning front-run with three ex-IMO medalists verifying their results, before DeepMind's official announcement on Monday after full IMO verification. Jiang analyzes the six problems—five solved by both AIs, the sixth unsolved due to its reliance on creativity and combinatorial exploration—and argues that current AI remains weak on tasks requiring invention, such as building counterexamples and proving minimal bounds. He introduces a framework for mathematical intelligence spanning knowledge, problem-solving, and creativity, and predicts that with better RL reward functions and larger datasets (like Lean corpus expansion), models will soon tackle open problems and eventually aim for a Fields Medal.

⚡️ARC-AGI-3: The Interactive Reasoning Benchmark

Jul 18, 2025 · 39:41

Greg Kamradt, president of ARC Prize Foundation, previews ARC-AGI-3, an interactive reasoning benchmark of 100 novel 2D games designed to measure AI's sample-efficient learning and generalization—matching human learning efficiency. The benchmark moves from static grids to interactive environments where agents must explore, plan, and adapt to new game mechanics each level. Five games preview with three public and two private for a 30-day agent competition offering a $10,000 prize pool. Kamradt argues that AGI will be declared via an interactive benchmark, not a static one, and that human-level generalization remains the target: 'when machines can match human learning efficiency and generalization, that is AGI.' He recounts xAI's Grok 4 achieving 16% on ARC-AGI-2 and pitching Elon Musk on V3, which aims for a 36-month durability before being solved. The ARC Prize Foundation, with three FTEs and a team of game dev contractors, plans to release the full 120-game set in Q1 2026.

Scaling Test Time Compute to Multi-Agent Civilizations — Noam Brown, OpenAI

Jun 19, 2025 · 1:17:47

Noam Brown, OpenAI researcher behind Cicero and reasoning models, joins hosts Alessio and Puix to discuss how test-time compute scaling drives multi-agent AI and the limits of the system 1/2 analogy. He explains that his Diplomacy bot Cicero reached top 10% of humans in 2022, and later he won the 2025 world championship. Brown argues that reasoning models like o3 succeed in unverifiable domains (e.g., Deep Research) and that harnesses and routers will be washed away by scale. He reveals OpenAI's multi-agent team is pursuing a principled approach, not heuristic, and envisions AI civilizations collaborating. He also shares his coding stack (Codex, Windsurf) and predicts test-time compute will hit cost and wall-clock bottlenecks.

The Utility of Interpretability — Emmanuel Amiesen

Jun 6, 2025 · 1:53:02

Emmanuel Amiesen, lead author of Anthropic's Circuit Tracing paper, presents the newly open-sourced tools that let anyone inspect how language models compute, revealing that models like Gemma 2-2B engage in genuine multi-step reasoning rather than mere pattern matching. Amiesen demonstrates that the model's internal features causally drive outputs—for instance, suppressing the 'Texas' feature in a capital-of-state query changes the answer from Austin to Sacramento. He explains how sparse autoencoders uncover features shared across languages and modalities, and how attribution graphs map feature interactions. The episode covers practical insights: why jailbreaks succeed (the model prioritizes grammatical completion over refusal) and how planning in poetry generation involves deciding a rhyme before the line begins. Amiesen calls for more researchers to explore interpretability, citing the low barrier to entry with open models and tools.

[AIEWF Preview] Gemini in 2025 and Realtime Voice AI

Jun 2, 2025 · 24:29

At Google I/O 2025, Latent Space host Swyx and TWiML's Sam Charrington speak with Logan Kilpatrick, Shrestha Basu Mallick, and Kwindla Hultman Kramer about Gemini's reasoning upgrades, real-time voice AI, and the infrastructure challenges of building production voice agents. Kilpatrick announces thinking budgets and thought summaries for 2.5 Pro, now live in Flash and coming to Pro in June, while Mallick highlights native audio output with multilingual code-switching and proactive voice activity detection. Kramer discusses the need for sub-500ms latency and WebRTC packet routing, and the team debates single-model vs. componentized architectures, with Mallick noting that eventual convergence will bring capabilities like diffusion-based generative UI into the main Gemini model. The episode also covers implicit context caching, URL context for research agents, and the Live API's session length and function calling improvements.

Gemini 2.0 Flash and Flash Thinking: the new SOTA models for the agentic era

Feb 28, 2025 · 28:21

Logan Kilpatrick, Google AI Studio product lead, returns to discuss Gemini 2.0 Flash and Flash Thinking, the new models that balance frontier capabilities with cost efficiency for the agentic era. He explains the pricing strategy: Flash at 10 cents per million tokens (simplified from tiered pricing), Flash Lite preserving the 7.5-cent narrative for low cost, and Pro pushing the frontier. Flash Thinking, co-led by Noam Shazeer and Jack Rae, scales inference-time compute and already shows rapid improvements, with base model advances coupling with RL-based reasoning. Swyx reports that Gemini Flash outperforms o3-mini on long-context summarization, calling it a 'reporting model.' The multimodal live API enables real-time voice and vision interactions, and a memory layer mimicking Astra is in development. Grounding with Search as a Tool is also highlighted for agentic search use cases.

S1: the $6 DeepSeek R1 Competitor (ft. Entropix)

Feb 26, 2025 · 17:43

Tim Kellogg joins hosts Alessio and Swyx to break down his viral blog post on S1, showing how Stanford researchers cloned DeepSeek R1's reasoning for $6 by inserting a 'think token' and fine-tuning on just 1,000 examples. Kellogg argues the real innovation isn't the cost but the technique's simplicity—making advanced inference-time scaling accessible. He explains Entropix, a dynamic sampler that uses model entropy and varentropy to adjust generation on the fly, and notes its creators are starting a company. The conversation contrasts supervised fine-tuning's targeted results with RL's broader reasoning growth, using an analogy: 'RL causes a lot of growth, then SFT trims it.' Kellogg predicts that embedding introspection signals into RL training could solve agent 'doom loops' and unlock higher agency—a key challenge behind products like OpenAI's Deep Research.

Why is everyone cloning Deep Research?

Feb 18, 2025 · 1:00:10

In this episode, hosts Alessio and Swyx interview Aarush Selvan and Mukund Sridhar, the lead PM and tech lead for Gemini Deep Research, about how Google created the deep research agent category and the technical decisions behind it. They finetuned a special version of Gemini 1.5 Pro for the feature, which autonomously browses the web for about five minutes to produce a research report. The team built an asynchronous platform to handle long-running tasks and developed an ontology of research behaviors—from broad shallow exploration to deep focus—to evaluate performance. They found that users value the visible browsing process even when it takes minutes, contrary to typical latency norms. The discussion covers iterative planning, trade-offs between model memory and web grounding, and future directions including thinking models, multimodality, and personalization.

The Unreasonable Effectiveness of Reasoning Distillation: using DeepSeek R1 to beat OpenAI o1

Jan 24, 2025 · 23:35

Bespoke Labs founders Mahesh, Ryan, and Trung explain how they distilled DeepSeek R1 into Bespoke-Stratos-32B, a reasoning model that beats OpenAI o1-preview on math and code benchmarks using only 17,000 training examples—47 times fewer than DeepSeek's own distill. They achieved this in a 48-hour sprint leveraging their data curation library Curator. Unlike Sky-T1, which required rewriting unreliable QWQ traces, R1's coherent reasoning allowed them to skip re-annotation and simply filter for correctness. The team argues data quality matters more than quantity, and that smaller models (like their 7B variant) can also improve with better teacher data. They see this as evidence that reasoning can emerge purely from supervised fine-tuning on high-quality traces, without complex search algorithms.

OpenAI o1 isn’t a chat model (and that’s the point)

Jan 17, 2025 · 31:59

Ben Hylak and Dan McAteer explain why OpenAI’s o1 is not a chat model and how mastering its prompting turned Ben from a skeptic into a proponent. Ben’s prompt structure—goal, return format, warnings, context dump—emphasizes describing what you want rather than how to think, a shift from earlier models. Dan uses o1 for coding by feeding it full code context, achieving one-shot implementations that reach 100% where previous models maxed out at 95%. Ben notes o1 is the most capable but hardest model to use, with experiments costing $20 per try, and argues model routing will let humans or systems trade off cost, speed, and intelligence. Dan also applies o1 to scientific research, feeding it papers from multiple fields to find novel connections that humans would miss.

The State of Reasoning — from Nathan Lambert, Interconnects/AI2 [LS Live @ NeurIPS 2024]

Jan 2, 2025 · 16:22

Nathan Lambert argues that language models do perform reasoning, contrary to skeptics, and that embracing chain-of-thought and reinforcement learning (RL) is key to advancing their capabilities. He explains OpenAI's o1 as large-scale RL on verifiable outcomes, noting that post-training flops exceed pre-training, and highlights relatives like DeepSeek and Qwen which are narrower. Lambert details OpenAI's new reinforcement fine-tuning API, which uses the same infrastructure as o1 and requires only dozens of labeled samples, and contrasts it with process reward models or Monte Carlo tree search. He presents his own project using RL on math evaluations (GSM8K, MATH, AFeval) to show gentle RL fine-tuning can boost specific capabilities without degrading general performance. The talk concludes that reasoning is worth pursuing and that new, less-human-like forms of model reasoning are emerging.

2024 Year in Review: The Big Scaling Debate, the Four Wars of AI, Top Themes and the Rise of Agents

Jan 1, 2025 · 1:51:07

In their 100th episode, hosts Alessio and Swyx recap 2024 in AI, arguing that pre-training scaling has hit a wall—backed by Ilya Sutskever and others at NeurIPS—and that inference-time compute (o1, o3) is the new frontier. They dissect the "four wars": data quality (lawsuits vs. synthetic data), GPU haves vs. have-nots (with the middle class dying), multimodality (Sora, Veo 2, Gemini 2.0's native image output), and the LLM OS/agents stack (LangChain, E2B, memory). Market share shifted from OpenAI's 95% to 50-75% as Anthropic and Gemini gained ground; prices dropped ~3 orders of magnitude for same ELO. The episode predicts 2025 as the year agents finally enter production, driven by models like o1 and tools like Devin, and warns that AI will set the skill floor for roles.

[Paper Club] 🍓 On Reasoning: Q-STaR and Friends!

Sep 18, 2024 · 47:12

This episode of Paper Club surveys three reasoning papers — STaR, Quiet-STaR, and V-STaR — arguing that while STaR is foundational for bootstrapping reasoning via rationales and rationalization, Quiet-STaR's attempt to generate rationales at every token yields only marginal gains (5–10% on GSM8K and CQA), and V-STaR's verifier trained with DPO on both correct and incorrect solutions delivers the most practical improvement, beating majority voting. The host explains STaR's two-loop process of generating rationales and rationalizations from wrong answers, highlights examples like a filtering-straw question where human raters evaluated reasoning quality, and notes that STaR on GPT-J 6B achieved human-like step counts in math problems. V-STaR's verifier selects among candidate solutions, scaling with K candidates, and is compared to process reward models from OpenAI's "let's verify step by step."

[LLM Paper Club] Llama 3.1 Paper: The Llama Family of Models

Jul 29, 2024 · 1:23:46

This episode examines Meta's Llama 3.1 paper, detailing the 405B dense model, its scaling laws grounded on the ARC reasoning benchmark rather than perplexity, and the decision to train on 15 trillion tokens. Vibhu explains the training infrastructure: 16,000 H100s over 54 days with 419 interruptions, 78% from GPU hardware failures. Eugene Yan walks through the synthetic data pipeline—using Llama 2 for filtering, stepwise reward models, and Monte Carlo tree search to improve reasoning traces. Hassan shares building LlamaTutor.com with Together API, serving 4,000 visitors and 5,900 requests for about $12. The group also discusses quantization trade-offs (larger models degrade less), inference provider variability (Groq's non-deterministic temperature zero), and compares Llama 405B's performance to GPT-4o and Claude.

How AI is Eating Finance - with Mike Conover of Brightwave

Jun 11, 2024 · 1:05:06

Brightwave founder Mike Conover explains how his vertical AI startup builds a 'partner in thought' for finance professionals, using a systems-of-systems approach that decomposes problems into specialized subsystems rather than relying on large context windows or monolithic agents. He argues that the evolution from DALL-E's 1,024-token context to today's million-token models has not solved synthesis; smaller, focused reasoning units still outperform. Conover emphasizes that the real competitive edge lies in custom training data and human annotation, not pre-training, and that models are converging in capability—so value shifts to fine-tuning data that elicits specific behaviors. He details why Brightwave avoids spreadsheets in favor of conversational analysis, and predicts AI will automate idea generation and second-order derivative bets, but humans must remain the final synthesizers and deciders. The episode also covers hiring for vertical AI, the role of knowledge graphs, and the diminishing economic incentive for companies to train their own foundation models.

LLM Asia Paper Club Survey Round

May 22, 2024 · 55:25

The episode surveys four recent papers on LLM reasoning, uncertainty, interpretability, and efficiency. 'Let's Think Dot by Dot' shows that filler tokens (dots) inserted between input and output enable hidden computation, outperforming no-token baselines on tasks like 3-Sum and 2-Sum Transform. 'Uncertainty Estimation' trains a random forest on hidden-layer activations to predict response confidence, achieving higher AUC than unsupervised methods on Q&A and translation. 'Monosemanticity' uses sparse autoencoders to identify interpretable features (e.g., a DNA-detection feature) in a toy transformer, advancing mechanistic interpretability. 'Medusa' attaches multiple prediction heads to the final hidden state to speculate future tokens, enabling faster decoding without a separate draft model and training in five hours on 60K samples.

Supervise the Process of AI Research — with Jungwon Byun and Andreas Stuhlmüller of Elicit

Apr 11, 2024 · 1:05:28

Andreas Stuhlmüller and Jungwon Byun, co-founders of Elicit (formerly the nonprofit Ought), have built an AI research assistant that automates literature review and reasoning by breaking complex tasks into transparent, step-by-step processes. Their philosophy—"supervise the process, not just the outcome"—led them to start with human simulations before GPT-3 enabled a product pivot. Elicit now uses both open-source and closed models (e.g., GPT-4, Claude Haiku) for summarization, data extraction, and uncertainty flags, and recently launched computational notebooks for scalable, reusable workflows. The company transitioned from nonprofit to a Public Benefit Corporation, reached $1M revenue in four months, and now employs 12 people, focusing on senior software engineers to build reliable orchestration from unreliable components.

Beating GPT-4 with Open Source Models - with Michael Royzen of Phind

Nov 3, 2023 · 1:18:54

Michael Royzen, co-founder and CEO of Phind, explains how his company built a GPT-4-beating open-source model for developer Q&A. Royzen recounts founding SmartLens in high school, shifting to NLP after a Hugging Face demo, and creating an internet-scale LLM-powered RAG system in January 2022. He details Phind’s pivot to programmers, the Hacker News launch that gave it 1,500 points, and Paul Graham’s role in naming the company and introducing Ron Conway, who then connected Phind to NVIDIA for GPU access. Royzen argues that Phind’s model, fine-tuned from Code Llama 34B with extra data, closes the gap with proprietary models—especially on code reasoning—and that open-source will win the enterprise because the delta to GPT-5 will be small. He shares how Phind handles multi-step conversations via a pair programmer mode where users can pin messages, and reveals plans for reinforcement learning to reduce hallucination and improve correctness.