o1

Cooking with OpenAI’s Research Chief: AGI, o1, Evals, and Scaling Laws — Mark Chen

Jun 25, 2026 · 41:18

Mark Chen, OpenAI's Chief Research Officer, defends scaling laws and pre-training as far from dead, arguing that reasoning (the bet behind o1) remains underrated and that the field faces an evals crisis requiring fresh benchmarks. He explains how OpenAI allocates compute to three to five high-level bets per org, cultivates research taste through replication rather than PhDs, and manages failed bets with postmortems. Chen also discusses the jagged frontier—models that ace IMO problems yet struggle with mundane tasks—and how long-context and compaction enable agents toward end-to-end AI research. Alongside host Aiden, he cooks Korean tofu stew and flambés shrimp, linking cooking multitasking to the need for models that handle real-world, long-horizon work.

Marc Andreessen introspects on Death of the Browser, Pi + OpenClaw, and Why "This Time Is Different"

Apr 3, 2026 · 1:16:20

Marc Andreessen argues that AI is finally different from past boom-bust cycles because of four compounding breakthroughs: LLMs, reasoning (o1, R1), coding agents (OpenClaw), and recursive self-improvement. He calls this the '80-year overnight success'—decades of neural network research now paying off. Comparing today's AI capex boom to the dot-com crash, he notes that buyers like Microsoft and Google are cash-rich incumbents and every GPU deployed is already generating revenue. He hails Pi and OpenClaw as a Unix-like architecture that makes agents model-independent and self-modifiable. On open source, he calls DeepSeek a 'gift to the world' for its paper and code, but warns that entrenched institutions—unions, licensing, government monopolies—will slow AI adoption far more than technologists expect.

[State of RL/Reasoning] IMO/IOI Gold, OpenAI o3/GPT-5, and Cursor Composer — Ashvin Nair, Cursor

Dec 30, 2025 · 45:13

Ashvin Nair, now ML lead at Cursor, traces his path from Berkeley robotics and an OpenAI Dota-era internship to OpenAI's reasoning team (which grew from a dozen to 300+ people) and explains why IOI Gold in 2022 felt like solving AI but didn't change the world—because RL doesn't generalize beyond training distribution. He argues most RL research from 2017-2022 overfit to benchmarks, rewarding complex ideas over simple ones that scale. At Cursor, he sees a unique opportunity for continual learning with policy updates every two hours and product-model co-design, keeping engineers in the loop instead of context-switching. His bet is that the next paradigm shift is continual learning with infinite memory: models experience something once and never forget it, storing millions of deployment tokens in weights without overloading capacity.

How Zyphra went all-in on AMD + Why Devs feel faster with AI but are slower — with Quentin Anthony

Nov 3, 2025 · 1:00:02

Quentin Anthony, head of model training at Zyphra and advisor at EleutherAI, explains why his company moved all training to AMD MI300X GPUs, outperforming Nvidia H100s on certain workloads thanks to 192GB VRAM and higher memory bandwidth, and describes his kernel development approach of writing directly in ROCm or GPU assembly rather than using Triton. He also shares his experience in the METR study on AI coding productivity, where he was one of the few developers with measurable speedup, and offers tips: timebox AI use, avoid the slot machine effect, maintain context hygiene, and use direct API over tools like Cursor. Additionally, he argues that open source AI research benefits from siloed focused teams with guaranteed funding over grand collaborations, and notes that kernel datasets alone won't solve GPU programming due to evaluation challenges.

Why RL Won — Kyle Corbitt, OpenPipe (acq. CoreWeave)

Oct 16, 2025 · 1:08:23

Kyle Corbitt, co-founder and CEO of OpenPipe (acquired by CoreWeave), explains why reinforcement learning has replaced supervised fine-tuning for training reliable AI agents. He argues GRPO is a dead end due to its requirement for perfectly reproducible parallel rollouts, which is extremely hard in practice. Instead, OpenPipe’s RULER uses relative LLM-as-judge rewards, achieving state-of-the-art performance even with a weak judge. Corbitt reports that 90% of AI projects remain stuck in proof-of-concept due to reliability issues, and that LoRAs are underrated for production while GEPA failed in his tests. He predicts continuous RL from real-world experience can unlock 10x more inference demand.

⚡️Ranking Agentic LLMs — Pratik Bhavsar, Galileo

Jul 14, 2025 · 34:10

Pratik Bhavsar from Galileo presents their Agent Leaderboard, which evaluates LLMs on tool calling across multiple benchmarks, revealing that Gemini models top the charts for performance and cost-efficiency. The leaderboard, built on datasets like BFCL, XLAM, and Tau Bench, uses an LLM-as-judge metric called Tool Selection Quality (TSQ). Surprising findings include reasoning models like o1 struggling with multi-tool outputs, Mistral Small excelling as the best open-source model, and Llama models performing poorly. The episode also previews V2 of the leaderboard, which introduces domain-specific, harder, multi-turn scenarios with a user and tool simulator, and a new Action Completion metric to measure whether all user requests are accomplished.

Information Theory for Language Models: Jack Morris

Jul 2, 2025 · 1:18:13

Jack Morris, a Cornell PhD student advised by Sasha Rush, discusses his information-theoretic research on language models, including embedding inversion (recovering text from embeddings with 90% accuracy), the universal geometry of embeddings (aligning different models' latent spaces), and measuring model memorization capacity at 3.6 bits per parameter. He argues that paradigm shifts in AI, from AlexNet to instruction tuning, stem not from new architectures but from new datasets, and that the next breakthrough will likely emerge from a novel data source. The conversation also covers the shift from academia to industry, practical advice for grad students on distributed training, and the implications of embedding inversion for privacy and model alignment.

Scaling Test Time Compute to Multi-Agent Civilizations — Noam Brown, OpenAI

Jun 19, 2025 · 1:17:47

Noam Brown, OpenAI researcher behind Cicero and reasoning models, joins hosts Alessio and Puix to discuss how test-time compute scaling drives multi-agent AI and the limits of the system 1/2 analogy. He explains that his Diplomacy bot Cicero reached top 10% of humans in 2022, and later he won the 2025 world championship. Brown argues that reasoning models like o3 succeed in unverifiable domains (e.g., Deep Research) and that harnesses and routers will be washed away by scale. He reveals OpenAI's multi-agent team is pursuing a principled approach, not heuristic, and envisions AI civilizations collaborating. He also shares his coding stack (Codex, Windsurf) and predicts test-time compute will hit cost and wall-clock bottlenecks.

⚡️Open Questions in Agentic RL — Will Brown (Prime Intellect)

May 9, 2025 · 17:13

Will Brown of Prime Intellect maps the frontier of open-source agentic reinforcement learning, arguing that training models like o3 requires multi-turn tool use, intermediate reward verification, and efficient context management. He outlines challenges such as scaling tool calls to hundreds (e.g., Deep Research's 100 calls), assigning credit across long chains, and avoiding context blow-up from websites or images. Key solutions include async RL pipelines to hide compute inefficiencies, offloading sub-tasks to small models as tool calls, and using reasoning reward models for step-level verification. Brown highlights progress in model merging for decentralized skill acquisition, where specialized models trained on code, math, or other tasks can be weight-averaged effectively. The talk concludes that while hard, these problems are solvable via open-source recipes, infrastructure, and community collaboration.

What is an RL environment? w/ Nous Research's Roger Jin

Apr 29, 2025 · 15:27

Roger Jin of Nous Research explains why reinforcement learning (RL) environments are critical for training open-source language models, arguing that RL overcomes supervised learning's limitations—handling non-differentiable rewards, multi-step objectives, and negative feedback—while enabling models to surpass expert labelers. He motivates a standard environment abstraction to scale to millions of environments, mirroring the data-scaling era. Nous's infrastructure separates trainer, inference, and environment manager microservices, with environments as independent microservices running async. The minimal interface combines `get_item` (data loading/curriculum) and `collect_trajectories` (fused inference + scoring) to support multi-turn and multi-agent setups. Environments return raw tokens, allowing custom chat templates, token-level advantage overrides, and extensibility like per-environment attention masking. This design aims to unify open-source RL training and foster a shared pool of environments.

Sleep-Time Compute — Letta AI (Charles Packer, Charlie Snell, Kevin Lin)

Apr 21, 2025 · 34:02

This episode covers Letta AI's new paper on Sleep-Time Compute, a scaling direction that applies compute during model idle periods (sleep time) rather than only at test time. Charles Packer, Kevin Lin, and Charlie Snell explain that sleep-time compute precomputes inferences from static context before queries arrive, yielding Pareto improvements on math benchmarks like GSM8K by shifting the accuracy-to-token curve leftwards. They distinguish it from test-time compute by emphasizing that sleep-time tokens incur no user-latency cost, and show that the benefit is largest when questions are predictable from context. The team also ties the concept to stateful agents and memory systems (building on MemGPT), releasing two implementations: one for low-latency chatbots and another for document-driven agents.

GPT 4.1: The New OpenAI Workhorse

Apr 15, 2025 · 45:04

OpenAI's Michelle Pokrass and Josh McGrath join hosts Alessio, Swyx, and Sean to launch GPT 4.1, a new model family positioned as the go-to workhorse for developers with major improvements in coding, instruction following, and long context. The three models—4.1, 4.1 Mini, and 4.1 Nano—introduce a 1M-token context window, 75% prompt caching discount, and cheaper pricing than 4o. Coding gains are highlighted by 55% on SWE-bench (vs o1's 41%), while instruction following benefits from real-world API data and new evals like GraphWalk for multi-hop reasoning. The team explains that most improvements come from post-training techniques, with vision capabilities lifted by a new pre-trained base. Fine-tuning is available day one, and developers are encouraged to opt in to data sharing for future model iterations.

S1: the $6 DeepSeek R1 Competitor (ft. Entropix)

Feb 26, 2025 · 17:43

Tim Kellogg joins hosts Alessio and Swyx to break down his viral blog post on S1, showing how Stanford researchers cloned DeepSeek R1's reasoning for $6 by inserting a 'think token' and fine-tuning on just 1,000 examples. Kellogg argues the real innovation isn't the cost but the technique's simplicity—making advanced inference-time scaling accessible. He explains Entropix, a dynamic sampler that uses model entropy and varentropy to adjust generation on the fly, and notes its creators are starting a company. The conversation contrasts supervised fine-tuning's targeted results with RL's broader reasoning growth, using an analogy: 'RL causes a lot of growth, then SFT trims it.' Kellogg predicts that embedding introspection signals into RL training could solve agent 'doom loops' and unlock higher agency—a key challenge behind products like OpenAI's Deep Research.

smol agents are all you need

Feb 13, 2025 · 22:31

This episode features Aymeric Roucher of Hugging Face discussing smol agents, a lightweight agent framework, and argues that code-based agents outperform JSON-based agents. Roucher explains that agents are defined by LLM control over execution flow, and smol agents simplifies this with a core file under 1000 lines. He describes the origin from transformers.agents and the security-focused custom Python interpreter. The library achieved third place on the GAIA benchmark, using o1 and Sonnet models, and Roucher projects 90% accuracy by 2026 based on a sigmoid trend line. Hugging Face is launching an agent course and plans to build GUI agents for computer use, while also fine-tuning R1 for agentic tasks.

The AI Architect: Bret Taylor

Feb 11, 2025 · 1:35:59

Bret Taylor, CEO of Sierra and chairman of OpenAI, recounts engineering AI's future across three eras — rewriting Google Maps' frontend in a weekend (cutting un-gzipped code to 20K), building Sierra's in-house agent platform for consumer brands like Sonos and SiriusXM, and navigating OpenAI's leadership crisis. He argues that AI agents' core abstractions are still pre-React, compares today's agents to jQuery, and predicts that outcome-based pricing will dominate as agents complete jobs rather than assist humans. He urges software engineers to move up the stack toward domain-specific agents and to embrace Rust-like languages for AI-generated code, emphasizing formal verification and safety. Taylor also shares how OpenAI prioritizes mission over product features, and why the relationship with Microsoft remains OpenAI's most important partnership.

The Agent Reasoning Interface: Claude, ChatGPT Canvas, Tasks, Operator — with Karina Nguyen, OpenAI

Feb 1, 2025 · 1:06:29

OpenAI's Karina Nguyen, who led the creation of ChatGPT Canvas and Tasks, explains how her team trains separate models for new interaction paradigms rather than prompting the core model, enabling rapid iteration on user feedback. She details the 'behavioral design' process that shapes model personality in collaborative contexts like Canvas—deciding when to rewrite vs. edit—and how that extends to defining agents as a progression from one-off actions to fully trustworthy long-horizon delegation. Drawing on her time at Anthropic building Claude.ai from scratch and co-creating Claude 3, she contrasts the product mindsets: OpenAI takes more product risks across consumer features while Anthropic focuses on enterprise. She also shares her vision for ChatGPT evolving into a 'generative OS' where UIs adapt dynamically to user intent, and argues that human creativity—not model capability—is the current bottleneck for rethinking software interfaces.

Beating OpenAI and Anthropic by Looking At Data: the new #1 on SWE-Bench w/ W&B CTO Shawn Lewis

Jan 28, 2025 · 34:31

Shawn Lewis, CTO of Weights & Biases, built a coding agent using OpenAI's o1 model that achieved 64.6% on SWE-bench verified, the top score at the time. He developed a TypeScript agent framework called PhaseShift and an Eval Studio tool backed by Weave, emphasizing that rigorous data inspection and tooling are critical for agent development. Lewis explains his process of running parallel rollouts and a crosscheck method to select the best trajectory, which contributed a 6% improvement. He also discusses the challenges of adapting reasoning models like o1 for agentic tasks and shares plans to open-source PhaseShift. The episode highlights Lewis's dogfooding approach, using Weights & Biases' own tools to build and evaluate the agent.

Outlasting Noam Shazeer, Crowdsourcing Chai AI w/ 1.4m DAU — with William Beauchamp, Chai Research

Jan 26, 2025 · 1:13:33

William Beauchamp, founder-CEO of Chai AI, explains how he pivoted from algorithmic trading to build a character chatbot platform before Character.ai, growing to 1.4M DAU and $22M+ revenue by crowdsourcing model improvements through Chaiverse. Starting with GPT-J in 2021, he found product-market fit with a therapist bot and shifted to user-generated content, letting users define prompts, images, and names. Competing against well-funded rivals like Character AI and Talkie, Chai ships over 100 LLMs weekly via its developer platform, spending $10M on compute in 2024 and tripling it. Beauchamp argues AI follows an S-curve, not scaling laws, and focuses on inference optimization using rejection sampling and reward models to serve better responses. He prioritizes 'insanely great' products over technology-driven features, noting that audio and image features failed to move metrics, while a data flywheel and aggressive user acquisition drove rapid growth.

The Unreasonable Effectiveness of Reasoning Distillation: using DeepSeek R1 to beat OpenAI o1

Jan 24, 2025 · 23:35

Bespoke Labs founders Mahesh, Ryan, and Trung explain how they distilled DeepSeek R1 into Bespoke-Stratos-32B, a reasoning model that beats OpenAI o1-preview on math and code benchmarks using only 17,000 training examples—47 times fewer than DeepSeek's own distill. They achieved this in a 48-hour sprint leveraging their data curation library Curator. Unlike Sky-T1, which required rewriting unreliable QWQ traces, R1's coherent reasoning allowed them to skip re-annotation and simply filter for correctness. The team argues data quality matters more than quantity, and that smaller models (like their 7B variant) can also improve with better teacher data. They see this as evidence that reasoning can emerge purely from supervised fine-tuning on high-quality traces, without complex search algorithms.

OpenAI o1 isn’t a chat model (and that’s the point)

Jan 17, 2025 · 31:59

Ben Hylak and Dan McAteer explain why OpenAI’s o1 is not a chat model and how mastering its prompting turned Ben from a skeptic into a proponent. Ben’s prompt structure—goal, return format, warnings, context dump—emphasizes describing what you want rather than how to think, a shift from earlier models. Dan uses o1 for coding by feeding it full code context, achieving one-shot implementations that reach 100% where previous models maxed out at 95%. Ben notes o1 is the most capable but hardest model to use, with experiments costing $20 per try, and argues model routing will let humans or systems trade off cost, speed, and intelligence. Dan also applies o1 to scientific research, feeding it papers from multiple fields to find novel connections that humans would miss.

Beating Google at Search with Neural PageRank and $5M of H200s — with Will Bryk of Exa.ai

Jan 10, 2025 · 55:54

Will Bryk, CEO of Exa.ai (formerly Metaphor), details building a neural search engine from scratch using link prediction as 'Neural PageRank' — predicting documents rather than keywords. Exa's new product offers near-perfect lists (e.g., 'startups working on hardware in SF') by scaling compute per query, from milliseconds to a day, like o1 for search. Bryk argues LLMs will become the interface to search, while Exa provides the 'super knowledge' that even AGI will need. He contrasts Exa's neural approach with Google's keyword-based system and Perplexity's reliance on Bing. The company recently purchased a $5M H200 cluster and maintains a culture of nap pods and first-principles thinking.

The State of Reasoning — from Nathan Lambert, Interconnects/AI2 [LS Live @ NeurIPS 2024]

Jan 2, 2025 · 16:22

Nathan Lambert argues that language models do perform reasoning, contrary to skeptics, and that embracing chain-of-thought and reinforcement learning (RL) is key to advancing their capabilities. He explains OpenAI's o1 as large-scale RL on verifiable outcomes, noting that post-training flops exceed pre-training, and highlights relatives like DeepSeek and Qwen which are narrower. Lambert details OpenAI's new reinforcement fine-tuning API, which uses the same infrastructure as o1 and requires only dozens of labeled samples, and contrasts it with process reward models or Monte Carlo tree search. He presents his own project using RL on math evaluations (GSM8K, MATH, AFeval) to show gentle RL fine-tuning can boost specific capabilities without degrading general performance. The talk concludes that reasoning is worth pursuing and that new, less-human-like forms of model reasoning are emerging.

2024 Year in Review: The Big Scaling Debate, the Four Wars of AI, Top Themes and the Rise of Agents

Jan 1, 2025 · 1:51:07

In their 100th episode, hosts Alessio and Swyx recap 2024 in AI, arguing that pre-training scaling has hit a wall—backed by Ilya Sutskever and others at NeurIPS—and that inference-time compute (o1, o3) is the new frontier. They dissect the "four wars": data quality (lawsuits vs. synthetic data), GPU haves vs. have-nots (with the middle class dying), multimodality (Sora, Veo 2, Gemini 2.0's native image output), and the LLM OS/agents stack (LangChain, E2B, memory). Market share shifted from OpenAI's 95% to 50-75% as Anthropic and Gemini gained ground; prices dropped ~3 orders of magnitude for same ELO. The episode predicts 2025 as the year agents finally enter production, driven by models like o1 and tools like Devin, and warns that AI will set the skill floor for roles.

0 to over $8M ARR in 2 months as a Claude Wrapper (Bolt.new, Qodo)

Dec 2, 2024 · 1:36:42

Eric Simons of Bolt.new/Stackblitz and Itamar Friedman of Qodo discuss Bolt.new's viral growth from $4M ARR in four weeks to over $8M ARR in two months as a Claude wrapper, driven by its in-browser WebContainer OS that eliminates local dev setup. They contrast Bolt's '0 to 1' creation for non-developers with enterprise-focused agents like Qodo's testing and code review tools, arguing that specialized agents outperform general-purpose ones. Simons details how WebContainer's unified sandbox enables error self-healing loops, while Friedman explains AlphaCodium's task-breaking that boosted o1 preview to 93rd percentile on Codeforces. The episode covers open-source strategy, pricing tiers ($20 to $200/month), and founder advice on ignoring expert doubt to ship fast, illustrated by Simons' Ironman completion while launching Bolt.

Why Compound AI + Open Source will beat Closed AI — with Lin Qiao, CEO of Fireworks AI

Nov 25, 2024 · 55:49

Lin Qiao, CEO of Fireworks AI, argues that compound AI systems combining multiple open-source models will outperform closed monolithic models like OpenAI's. She explains Fireworks' evolution from a PyTorch platform to a full-stack inference and customization engine serving 40+ customers including Cursor. Qiao details their distributed inference engine, Fire Optimizer for quality-latency-cost tradeoffs, and upcoming o1-like model built on open-source foundations. She defends their quantization approach after public criticism from rivals, emphasizes specialization over general intelligence, and invites developers to test their free LoRA adapter hosting.

Agents @ Work: Lindy.ai (with live demo!)

Nov 15, 2024 · 1:08:01

Florent Crivello, founder of Lindy.ai, explains how his no-code agent platform evolved from an overly LLM-dependent design to a deterministic 'on rails' system that puts AI in a box, making agents more reliable and easier to use. The product was rebuilt from scratch as Lindy 2.0, featuring step-by-step workflows with structured outputs and memory management that still requires pruning. Crivello reveals that Claude 3.5 Sonnet ended model bottlenecks but warns against over-engineering cognitive architectures, advocating instead for letting models scale. He discusses building their own eval tools (now considering Braintrust), hiring vertical GMs for go-to-market, and the decision to return to in-person work, arguing remote work hampers creativity. He also shares his contrarian views on Europe vs. US tech, testing the Overton window with spicy tweets, and balancing AI safety concerns (10% p(doom)) with building toward a potential utopia.

Singapore: the AI Engineer Nation — with Minister Josephine Teo

Oct 19, 2024 · 56:40

Singapore Minister Josephine Teo outlines the city-state's refreshed national AI strategy and the Ministry of Digital Development's approach to balancing innovation, safety, and talent development. She details plans to triple AI practitioners from 5,000 to 15,000, the AI Verify testing tools and AI Safety Institute for governance, and internal government AI bots like AI Bot for RAG-based knowledge retrieval. On sovereign AI, she discusses hybrid cloud infrastructure and pragmatic data center expansion. Teo also explains Singapore's new law requiring transparency for AI-generated election content, emphasizing fact-based political discourse.

Building AGI in Real Time (OpenAI Dev Day 2024)

Oct 4, 2024 · 2:09:35

OpenAI's DevDay 2024 unveils the Realtime API with WebSocket-based voice and function calling, Vision Finetuning, Prompt Caching, and Model Distillation. Product leads Olivier Godement, Romain Huet, and Michelle Pokrass detail design choices like zero-code caching and WebSockets for human-level latency. Alistair Pullen of Cosine explains how fine-tuning GPT-4o on custom reasoning traces beat o1 on SWE-bench Verified, while Sam Altman and Kevin Weil discuss AGI levels (o1 as level 2 reasoner), the coming agent era (level 3), safety through iterative deployment, and the importance of building for the frontier. The episode also covers moderation policy relaxation, plans for o1 system prompts by year-end, and the competitive admiration for NotebookLM.

[Paper Club] 🍓 On Reasoning: Q-STaR and Friends!

Sep 18, 2024 · 47:12

This episode of Paper Club surveys three reasoning papers — STaR, Quiet-STaR, and V-STaR — arguing that while STaR is foundational for bootstrapping reasoning via rationales and rationalization, Quiet-STaR's attempt to generate rationales at every token yields only marginal gains (5–10% on GSM8K and CQA), and V-STaR's verifier trained with DPO on both correct and incorrect solutions delivers the most practical improvement, beating majority voting. The host explains STaR's two-loop process of generating rationales and rationalizations from wrong answers, highlights examples like a filtering-straw question where human raters evaluated reasoning quality, and notes that STaR on GPT-J 6B achieved human-like step counts in math problems. V-STaR's verifier selects among candidate solutions, scaling with K candidates, and is compared to process reward models from OpenAI's "let's verify step by step."