o3

🔬How GPT‑5 derived new results in theoretical physics and quantum gravity — Alex Lupsasca, OpenAI

May 5, 2026 · 1:31:51

Alex Lupsasca, a theoretical physicist at OpenAI and recipient of the 2024 New Horizons Breakthrough Prize, details how GPT-5 and subsequent models derived new results in quantum field theory and quantum gravity, solving problems that had stumped experts for over a year. The work focused on 'single minus' gluon tree amplitudes, long believed to be zero, but which humans discovered might be non-zero in a special kinematic region. GPT-5.2 Pro conjectured a simplified formula for these amplitudes, and an internal OpenAI model later proved it, reducing a factorial number of Feynman diagram terms to a linear number. The AI then autonomously extended the result to graviton amplitudes using the gluon paper as a seed, producing a complete paper draft in under an hour. Lupsasca argues this marks a threshold where AI is superhuman on certain physics tasks, accelerating research by acting as a 'scout' that reduces confusion and suggests next questions. He also discusses challenges including AI slop on arXiv and the need for better verification methods.

The End of SWE-Bench Verified — Mia Glaese & Olivia Watkins, OpenAI Frontier Evals

Feb 23, 2026 · 27:10

OpenAI researchers Mia Glaese and Olivia Watkins explain why they are retiring SWE-Bench Verified, arguing the benchmark is saturated and highly contaminated, with models like GPT 5.2 regurgitating ground-truth solutions or task IDs. They detail how a contamination auditor agent found evidence across Claude Opus 4.5 and Gemini Flash, and how over half of unsolved problems had unfair tests (e.g., requiring specific function names not in the spec). The team is pivoting to SWE-Bench Pro from Scale, which features harder, longer tasks (1-4+ hours), more diverse repos and languages, and far less contamination. They advocate for future coding evals to measure open-ended design decisions, code quality, maintainability, and real-world product building, while also calling for more real-world usage metrics to track AI's impact on jobs and productivity.

[State of RL/Reasoning] IMO/IOI Gold, OpenAI o3/GPT-5, and Cursor Composer — Ashvin Nair, Cursor

Dec 30, 2025 · 45:13

Ashvin Nair, now ML lead at Cursor, traces his path from Berkeley robotics and an OpenAI Dota-era internship to OpenAI's reasoning team (which grew from a dozen to 300+ people) and explains why IOI Gold in 2022 felt like solving AI but didn't change the world—because RL doesn't generalize beyond training distribution. He argues most RL research from 2017-2022 overfit to benchmarks, rewarding complex ideas over simple ones that scale. At Cursor, he sees a unique opportunity for continual learning with policy updates every two hours and product-model co-design, keeping engineers in the loop instead of context-switching. His bet is that the next paradigm shift is continual learning with infinite memory: models experience something once and never forget it, storing millions of deployment tokens in weights without overloading capacity.

How Zyphra went all-in on AMD + Why Devs feel faster with AI but are slower — with Quentin Anthony

Nov 3, 2025 · 1:00:02

Quentin Anthony, head of model training at Zyphra and advisor at EleutherAI, explains why his company moved all training to AMD MI300X GPUs, outperforming Nvidia H100s on certain workloads thanks to 192GB VRAM and higher memory bandwidth, and describes his kernel development approach of writing directly in ROCm or GPU assembly rather than using Triton. He also shares his experience in the METR study on AI coding productivity, where he was one of the few developers with measurable speedup, and offers tips: timebox AI use, avoid the slot machine effect, maintain context hygiene, and use direct API over tools like Cursor. Additionally, he argues that open source AI research benefits from siloed focused teams with guaranteed funding over grand collaborations, and notes that kernel datasets alone won't solve GPU programming due to evaluation challenges.

⚡️Traversal: Causal ML and Reinforcement Learning

Oct 5, 2025 · 45:06

Anish and Raaz, co-founders of Traversal, explain how their agentic architecture combines causal machine learning and LLMs to turn incident troubleshooting from a massive search problem into an intelligent, context-driven process. Both have PhDs in causal ML and RL from MIT and Berkeley, and they applied that expertise to root cause analysis in complex enterprise systems with petabytes of fragmented data across logs, metrics, traces, and Slack. Their system dynamically combines statistical tests (for time-series) with semantic understanding (via LLMs) to winnow billions of signals to root cause candidates in under two minutes. They note that reasoning models like o3 are critical for complex incidents, while Claude is better for tool calling and unsticking. Traversal uses a mix of infrastructure size and investigation count for pricing, and sees self-healing as a continuum: 10–20% of issues can be autonomously resolved now, with 30–40% requiring senior engineer sanity-check within 6–12 months. They are hiring in New York and emphasize that building proprietary evals is core IP, making public benchmarks a tension.

Amp: The Emperor Has No Clothes

Sep 25, 2025 · 1:21:59

Quinn Slack (CEO) and Thorsten Ball (Amp Dictator) from Sourcegraph discuss Amp Code, a coding agent that ships 15x/day with no code reviews. They explain the strategic pivot from Cody to Amp, emphasizing that building the best coding agent trumps everything else. The team dogfoods Amp internally, using it to build itself, and maintains both a VS Code extension and CLI, with a 50/50 internal split. They argue that model choice is becoming less important than the harness around the model, and they reject features like prompt enhancers, custom subagents, and MCP servers as not adding value. Common failure modes include outsourcing thinking but not typing and using agents without proper context engineering. They predict async agents running 24/7 will dominate, and they target early adopters willing to adapt their codebases and workflows for maximal agent effectiveness.

Greg Brockman on OpenAI's Road to AGI

Aug 15, 2025 · 1:08:37

Greg Brockman, OpenAI co-founder and president, explains how GPT-5's hybrid reasoning and aggressive pricing advance the road to AGI, while releasing GPT-OSS for open source. He traces reasoning evolution from GPT-4 via reinforcement learning, noting Dota's pure RL success and IMO gold's direct transfer to IOI gold. Brockman details GPT-5's router that hides model choice from users, and says pricing has dropped ~10X per year since GPT-4. He argues compute remains the bottleneck, but connecting AI to real-world domains like healthcare is still wide open. Brockman adds that ARC Institute's DNA models are identical to language models, and that more problems emerge over time—it's never too late to contribute.

The RLVR Revolution — with Nathan Lambert (AI2, Interconnects.ai)

Jul 31, 2025 · 1:19:00

Nathan Lambert from AI2 explains the RLVR (Reinforcement Learning with Verifiable Rewards) revolution, arguing that verifiable rewards for math, code, and instruction-following are scaling more reliably than human feedback, and that open models like Tulu 3 can match frontier labs on core evals with just 10-15 tasks vs. hundreds. He traces overoptimization through three phases—control, RLHF, and RLVR—and warns that models learn to cheat unit tests unless reward design penalizes it. He analyzes o3's search-heavy approach (e.g., 80 websites per query), hybrid reasoning models like Gemini 2.5 and Claude, and predicts that pure reasoning models will become the default as inference costs drop. Lambert introduces a four-skill taxonomy for agent models—skills, calibration, strategy, abstraction—and says planning improvements often come from mundane data fixes. He advocates for fully open models, citing AI2's Olmo 32B as approaching GPT-4 level, but notes that building an 'American DeepSeek' requires massive resources and nonprofit constraints.

AI is Eating Search

Jul 23, 2025 · 56:22

Robert McCloy of Scrunch AI argues that AI is replacing web browsing, not just search, as consumers prefer ChatGPT and similar tools over traditional browsing. He explains that AI search platforms still rely on traditional search under the hood, but re-rank results based on metadata clarity—so clickbait titles fail while descriptive content wins. Prompt injection works currently but is a temporary tactic; the durable strategy is clear, structured content with server-side rendered HTML because AI tools don't execute JavaScript. Scrunch helps brands optimize for platforms like ChatGPT, Perplexity, and Meta AI; customer Clerk saw a 6x traffic increase and 9x conversion lift from AI-referred visitors. Robert predicts the future is about 'agent experience'—optimizing how AI systems interact with your content to serve end users directly.

⚡️ARC-AGI-3: The Interactive Reasoning Benchmark

Jul 18, 2025 · 39:41

Greg Kamradt, president of ARC Prize Foundation, previews ARC-AGI-3, an interactive reasoning benchmark of 100 novel 2D games designed to measure AI's sample-efficient learning and generalization—matching human learning efficiency. The benchmark moves from static grids to interactive environments where agents must explore, plan, and adapt to new game mechanics each level. Five games preview with three public and two private for a 30-day agent competition offering a $10,000 prize pool. Kamradt argues that AGI will be declared via an interactive benchmark, not a static one, and that human-level generalization remains the target: 'when machines can match human learning efficiency and generalization, that is AGI.' He recounts xAI's Grok 4 achieving 16% on ARC-AGI-2 and pitching Elon Musk on V3, which aims for a 36-month durability before being solved. The ARC Prize Foundation, with three FTEs and a team of game dev contractors, plans to release the full 120-game set in Q1 2026.

⚡️Ranking Agentic LLMs — Pratik Bhavsar, Galileo

Jul 14, 2025 · 34:10

Pratik Bhavsar from Galileo presents their Agent Leaderboard, which evaluates LLMs on tool calling across multiple benchmarks, revealing that Gemini models top the charts for performance and cost-efficiency. The leaderboard, built on datasets like BFCL, XLAM, and Tau Bench, uses an LLM-as-judge metric called Tool Selection Quality (TSQ). Surprising findings include reasoning models like o1 struggling with multi-tool outputs, Mistral Small excelling as the best open-source model, and Llama models performing poorly. The episode also previews V2 of the leaderboard, which introduces domain-specific, harder, multi-turn scenarios with a user and tool simulator, and a new Action Completion metric to measure whether all user requests are accomplished.

⚡️Anthropic vs Cognition on Multi-Agents: A Breakdown with Dylan Davis

Jul 5, 2025 · 27:11

Dylan Davis from Gradient Labs breaks down the Anthropic vs Cognition multi-agent debate, arguing both blog posts are right for different use cases. Anthropic's multi-agent architecture for deep research outperforms single-agent by 80% but uses 15x the tokens, relying on a lead orchestrator and parallel sub-agents. Cognition's single-agent approach for coding avoids dependency conflicts by compressing context sequentially, as demonstrated with a Flappy Bird example where multi-agent failed. A decision framework weighs task independence, need for diverse perspectives, and token cost trade-offs. The episode also covers Anthropic's evals using five dimensions (factual accuracy, citation accuracy, completeness, source quality, tool efficiency) judged by a single LLM, and a strategic disregard for current costs to build for future efficiency drops.

Scaling Test Time Compute to Multi-Agent Civilizations — Noam Brown, OpenAI

Jun 19, 2025 · 1:17:47

Noam Brown, OpenAI researcher behind Cicero and reasoning models, joins hosts Alessio and Puix to discuss how test-time compute scaling drives multi-agent AI and the limits of the system 1/2 analogy. He explains that his Diplomacy bot Cicero reached top 10% of humans in 2022, and later he won the 2025 world championship. Brown argues that reasoning models like o3 succeed in unverifiable domains (e.g., Deep Research) and that harnesses and routers will be washed away by scale. He reveals OpenAI's multi-agent team is pursuing a principled approach, not heuristic, and envisions AI civilizations collaborating. He also shares his coding stack (Codex, Windsurf) and predicts test-time compute will hit cost and wall-clock bottlenecks.

⚡️Launching AI Diplomacy: the hardest LLM Game Benchmark yet - Alex Duffy

Jun 11, 2025 · 34:32

Alex Duffy from Every launches AI Diplomacy, a benchmark where LLMs play the board game Diplomacy, revealing distinct model personalities and arguing that benchmarks function as memes that spread and saturate. He and Tyler Marquez built the front end and harness using Open Router, finding that Claude refuses to lie and never wins, o3 schemes in its diary, and DeepSeek is flowery or aggressive. His talk 'Benchmarks are Memes' highlights Simon Wilson's pelican-on-a-bicycle test as an example of an idea that gets adopted and then saturated by models. Duffy also shares his creative writing workflow at Every: dictation, prompts with style guides and editor notes, then heavy editing to reflect on AI output. Future plans include a data viewer, front-end improvements, and a human vs AI Diplomacy tournament to test jailbreaks.

⚡️Multi-Turn RL for Multi-Hour Agents — with Will Brown, Prime Intellect

May 23, 2025 · 38:59

This episode features Will Brown of Prime Intellect discussing Claude 4's emphasis on agentic tool use over pure reasoning, the controversy around Claude's safety stress-testing results (including alleged dark web uranium searches), and his team's paper on multi-turn reinforcement learning for LLM agents. Brown explains how turn-level credit assignment in GRPO can incentivize proper tool use while avoiding reward hacking, and argues that flexible LLM-based reward models will replace brittle deterministic parsers. He also critiques LMArena's funding model and calls for academia to lead in evaluation research.

ChatGPT Codex: The Missing Manual

May 16, 2025 · 53:32

Josh Ma and Alexander Embiricos from OpenAI introduce ChatGPT Codex, a cloud-hosted autonomous software engineer that runs in its own sandbox, arguing it delivers value through single-shot, long-horizon task completion. They explain that Codex evolved from internal experiments (codenamed WHAM) and now offers a 60-tasks-per-hour rate limit with a one-hour hard cutoff, emphasizing it is not meant for interactive use but for delegation—users fire off tasks and go about their day. Best practices include using agents.md, installing linters and formatters, keeping code modular, and having good architecture. The team intentionally kept it a research preview to gather feedback on environment customization and pricing, with plans to integrate seamless transitions between cloud and CLI.

⚡️Open Questions in Agentic RL — Will Brown (Prime Intellect)

May 9, 2025 · 17:13

Will Brown of Prime Intellect maps the frontier of open-source agentic reinforcement learning, arguing that training models like o3 requires multi-turn tool use, intermediate reward verification, and efficient context management. He outlines challenges such as scaling tool calls to hundreds (e.g., Deep Research's 100 calls), assigning credit across long chains, and avoiding context blow-up from websites or images. Key solutions include async RL pipelines to hide compute inefficiencies, offloading sub-tasks to small models as tool calls, and using reasoning reward models for step-level verification. Brown highlights progress in model merging for decentralized skill acquisition, where specialized models trained on code, math, or other tasks can be weight-averaged effectively. The talk concludes that while hard, these problems are solvable via open-source recipes, infrastructure, and community collaboration.

What is an RL environment? w/ Nous Research's Roger Jin

Apr 29, 2025 · 15:27

Roger Jin of Nous Research explains why reinforcement learning (RL) environments are critical for training open-source language models, arguing that RL overcomes supervised learning's limitations—handling non-differentiable rewards, multi-step objectives, and negative feedback—while enabling models to surpass expert labelers. He motivates a standard environment abstraction to scale to millions of environments, mirroring the data-scaling era. Nous's infrastructure separates trainer, inference, and environment manager microservices, with environments as independent microservices running async. The minimal interface combines `get_item` (data loading/curriculum) and `collect_trajectories` (fused inference + scoring) to support multi-turn and multi-agent setups. Environments return raw tokens, allowing custom chat templates, token-level advantage overrides, and extensibility like per-environment attention masking. This design aims to unify open-source RL training and foster a shared pool of environments.

Fullstack-Bench: The Eval for Coding Agents — with Sujay Jayakar, Chief Scientist, Convex

Mar 19, 2025 · 31:13

Sujay Jayakar, Convex chief scientist, presents Fullstack-Bench, a benchmark evaluating how well AI coding agents build full-stack apps on Convex, Supabase, and FastAPI+Redis, finding Convex’s tight feedback loops and strong abstractions let agents autonomously complete tasks within time limits. The benchmark tests three tasks (chat, to-do, files) across three backends with 30-60 minute limits and human hints. Convex succeeded autonomously on chat and to-do tasks, while Supabase agents struggled with RLS policy debugging and FastAPI agents with manual SSE wiring. Sujay discusses how platform design impacts AI code generation: simple procedural code, strong abstractions, and guardrails like end-to-end type safety improve LLM performance. He notes that API choices matter—things too similar to Firebase but not exact cause hallucinations—and that models like Claude 3.7 sometimes regress on platform-specific evals. The episode concludes with Convex’s focus on enabling AI-generated backends and anticipation of new agent workflow patterns.

Why is everyone cloning Deep Research?

Feb 18, 2025 · 1:00:10

In this episode, hosts Alessio and Swyx interview Aarush Selvan and Mukund Sridhar, the lead PM and tech lead for Gemini Deep Research, about how Google created the deep research agent category and the technical decisions behind it. They finetuned a special version of Gemini 1.5 Pro for the feature, which autonomously browses the web for about five minutes to produce a research report. The team built an asynchronous platform to handle long-running tasks and developed an ontology of research behaviors—from broad shallow exploration to deep focus—to evaluate performance. They found that users value the visible browsing process even when it takes minutes, contrary to typical latency norms. The discussion covers iterative planning, trade-offs between model memory and web grounding, and future directions including thinking models, multimodality, and personalization.

The State of Reasoning — from Nathan Lambert, Interconnects/AI2 [LS Live @ NeurIPS 2024]

Jan 2, 2025 · 16:22

Nathan Lambert argues that language models do perform reasoning, contrary to skeptics, and that embracing chain-of-thought and reinforcement learning (RL) is key to advancing their capabilities. He explains OpenAI's o1 as large-scale RL on verifiable outcomes, noting that post-training flops exceed pre-training, and highlights relatives like DeepSeek and Qwen which are narrower. Lambert details OpenAI's new reinforcement fine-tuning API, which uses the same infrastructure as o1 and requires only dozens of labeled samples, and contrasts it with process reward models or Monte Carlo tree search. He presents his own project using RL on math evaluations (GSM8K, MATH, AFeval) to show gentle RL fine-tuning can boost specific capabilities without degrading general performance. The talk concludes that reasoning is worth pursuing and that new, less-human-like forms of model reasoning are emerging.