Benchmarks

Scaling Past Informal AI - Carina Hong, Axiom Math

Jun 3, 2026 · 1:33:04

Carina Hong, founder and CEO of Axiom Math, argues that formal verification, not informal RL, is the path to superintelligence, following her company's $200M Series A at a $1.6B valuation and a perfect 120/120 on the 2024 Putnam exam. Axiom's system uses Lean theorem prover data and reinforcement learning to produce verified proofs, achieving a 99% pass rate on the Verina code-with-proof benchmark (187 of 189 problems). Hong contends verification is about 'scaling brilliance' — not fixing hallucinations — and that only verified generation can compound AI reasoning. She explains Axiom's open-source Axle API for Lean at scale, addresses why frontier labs like OpenAI have deprioritized formal math (team departures, strategy shifts), and outlines a vision where verified reasoning transfers from math to code, hardware, and eventually AGI through self-improvement. She also discusses the Earth sciences challenge of autoformalization, the difficulty of search in mathematical literature (citing the Erdos controversy), and why fragmentation in the AI math field is a bottleneck.

⚡️ Google's Open AI Strategy — Omar Sanseviero, Google DeepMind

May 24, 2026 · 29:59

Omar Sanseviero, Google DeepMind's Head of Developer Experience, explains Gemma 4's novel architecture with per-layer embeddings that enable effective parameter offloading: only 2B of 5B parameters need GPU memory, ideal for on-device inference on phones and Raspberry Pis. The model matches 1.5-year-old state-of-the-art in most areas, with Gemini Nano integrated into Pixel and Samsung phones. Gemma 4 supports multimodal input (audio, images, short video) but not audio output or combined audio-video prompts. Sanseviero notes fine-tuning is declining as out-of-box capabilities improve, but remains relevant for specialized domains like healthcare. He contrasts dense (31B) and MoE (27B) variants, highlighting MoE's inference speed but fine-tuning challenges. The team is expanding globally, with Kaggle joining DeepMind to create community-driven benchmarks for model evaluation.

⚡️ OpenClaw's Memory Sucks and the fix is simple — Dhravya Shah, Supermemory

Mar 9, 2026 · 27:10

Supermemory founder Dhravya Shah argues that current memory approaches like OpenClaw's static markdown files fail because they lack freshness and agent-driven retrieval, proposing a hooks-based system that dynamically injects less than 2,000 tokens per turn. His company Supermemory uses a hybrid mode combining structured memory with RAG fallbacks, achieving costs as low as 2 cents per million tokens. Shah critiques benchmarks like LongMemEval and LoCoMo for not testing personalization or forgetfulness, and his team built the open platform MemoryBench for evaluation. Future plans include plugins for Cursor and partnerships with voice agent companies, aiming for prosumer interoperability across AI providers.

The End of SWE-Bench Verified — Mia Glaese & Olivia Watkins, OpenAI Frontier Evals

Feb 23, 2026 · 27:10

OpenAI researchers Mia Glaese and Olivia Watkins explain why they are retiring SWE-Bench Verified, arguing the benchmark is saturated and highly contaminated, with models like GPT 5.2 regurgitating ground-truth solutions or task IDs. They detail how a contamination auditor agent found evidence across Claude Opus 4.5 and Gemini Flash, and how over half of unsolved problems had unfair tests (e.g., requiring specific function names not in the spec). The team is pivoting to SWE-Bench Pro from Scale, which features harder, longer tasks (1-4+ hours), more diverse repos and languages, and far less contamination. They advocate for future coding evals to measure open-ended design decisions, code quality, maintainability, and real-world product building, while also calling for more real-world usage metrics to track AI's impact on jobs and productivity.

Captaining IMO Gold, Deep Think, On-Policy RL, Feeling the AGI in Singapore — Yi Tay

Jan 23, 2026 · 1:32:05

Yi Tay, who leads Google DeepMind's Reasoning and AGI team in Singapore, explains how Gemini Deep Think achieved IMO Gold by abandoning symbolic AlphaProof for an end-to-end RL-trained model. He details the on-policy RL philosophy—models learn from their own generated outputs rather than imitating others—and the critical role of self-consistency through parallel sampling and internal verification. Tay describes the IMO effort: four co-captains in different time zones, a one-week training sprint, a live competition in Australia where researchers punched in problems as they were released, and the tension of waiting for human scores to determine the gold threshold. He discusses why the team believes one model must subsume everything for AGI, the data efficiency gap compared to humans, and his hiring focus on raw talent and research taste. Tay also shares his personal fitness transformation—losing 23 kilos and improving HRV—as integral to research productivity.

Artificial Analysis: The Independent LLM Analysis House — with George Cameron and Micah Hill-Smith

Jan 9, 2026 · 1:18:15

Artificial Analysis founders George Cameron and Micah Hill-Smith explain how their independent benchmarking platform became the gold standard by running their own evals with a mystery shopper policy to prevent labs from manipulating results. They launched in January 2024 after building it as a side project in Sydney, going viral after Swyx's retweet. The Intelligence Index V3 synthesizes 10 datasets with 95% confidence intervals, while the Omniscience Index measures hallucination rates from -100 to +100 (Claude models lead). Their GDP Val AA benchmark tests 44 white-collar tasks, and they open-sourced their agentic harness Stirrup. They also introduced an Openness Index scoring models out of 18 points. The episode covers how they make money through enterprise benchmarking subscriptions and custom work, and why the cost of GPT-4-level intelligence has dropped over 100× while total inference spend rises due to reasoning and agentic workflows.

[State of Code Evals] After SWE-bench, Code Clash & SOTA Coding Benchmarks recap — John Yang

Dec 31, 2025 · 17:45

John Yang, creator of SWE-bench, explains the benchmark's evolution from an ignored project to the industry standard after Cognition's Devin launch, and introduces CodeClash as a new paradigm for evaluating long-horizon coding agents via programming tournaments. Yang covers SWE-bench's expansion into nine languages (JavaScript, Rust, Java, C, Ruby) and multimodal tasks, the proliferation of independent variants like SWE-bench Pro and SWE-bench Live, and the shift from unit tests to iterative agent competitions. He addresses the Tau-bench controversy over impossible tasks, arguing they serve as cheat detection, and weighs the tension between long autonomy (5-hour runs) and interactivity (Cognition's fast back-and-forth). Yang highlights the academic data problem—companies like Cognition have rich user interaction data, while academics need user simulators or compelling products like LMArena—and positions CodeClash as a testbed for studying human-AI collaboration by freezing model capability and varying interaction setups.

[State of Context Engineering] Agentic RAG, Context Rot, MCP, Subagents — Nina Lopatina, Contextual

Dec 31, 2025 · 26:48

Nina Lopatina of Contextual AI discusses the state of context engineering, arguing that while the field is still in prototyping, 2025 will see true scale with full-system design patterns. She notes that agentic RAG—especially query reformulation into subqueries—has become the baseline, dramatically improving retrieval. Context rot is widely cited but industry benchmarks at real scale (100k+ documents, billions of tokens) remain rare. MCP is both a driver and a flaw: giant JSON tool definitions stuff the context window, yet MCP servers enable rapid prototyping before optimizing down to direct API calls. Sub-agents with turn limits are critical—unlimited agency degrades performance and causes hallucinations. Instruction-following re-rankers are essential for scaling retrieval across massive databases, providing more recall upfront and precision in the final context. Benchmarks are being saturated faster than ever: a Princeton benchmark from October was saturated by Claude Code in weeks, with solutions so good the gold dataset had errors. Lopatina also shares a lesson from using ChatGPT as a snowboarding coach—after several months of training for a 25-lap mogul race, she had to close…

SAM 3: The Eyes for AI — Nikhila & Pengchuan (Meta Superintelligence), ft. Joseph Nelson (Roboflow)

Dec 18, 2025 · 1:15:04

Meta's SAM 3, introduced by Nikhila Ravi and Pengchuan Zhang alongside Roboflow CEO Joseph Nelson, unifies interactive segmentation, open-vocabulary detection, and video tracking into a single model that runs in 30ms on images and scales to real-time video on multi-GPU setups. The model uses concept prompts like "yellow school bus" to detect and segment every instance, separating recognition from localization via a presence token. Its data engine automated exhaustive annotation from two minutes per image down to 25 seconds using AI verifiers fine-tuned on Llama, while the new SACO benchmark contains over 200,000 unique concepts versus previous 1.2k. For video, decoupling the detector and tracker preserves object identity, and SAM 3 agents pair with multimodal LLMs like Gemini to handle complex visual reasoning. The real-world impact includes 106 million smart polygons created on Roboflow, saving an estimated 130+ years of labeling time across fields from cancer research to underwater trash cleanup.

Terminal-Bench 2.0: the most impt coding agent benchmark of 2025 gets a v2! Launch + Q&A w/ founders

Nov 8, 2025 · 35:01

Terminal-Bench co-creators Mike Merrill, Alex Shaw, and collaborators Andy Konwinski, Ludwig Schmidt, and Aatash launch Terminal-Bench 2.0, an 89-task coding benchmark with rigorous human and LM-assisted verification to test agent performance on real-world tasks like designing DNA primers and installing Windows 3.11. They reflect on Terminal-Bench 1's problems—easy tasks and reproducibility—and detail spending 300 hours manually auditing each task. They introduce Harbor, an open-source package for running sandboxed agent rollouts, standardizing task format and integrating with agents like Codex and Claude Code. Aatash shows using Harbor to generate thousands of training traces for DataComp Agent, running 1,000 parallel sandboxes. Future plans include expanding to scientific computing.

Terminal-Bench: Pushing Claude Code, OpenAI Codex, Factory Droid, et al to the limits

Oct 18, 2025 · 35:27

Alex Shaw and Mike Merrill, creators of Terminal-Bench, explain how their coding agent benchmark became an industry standard adopted by Anthropic, OpenAI, and leading agent companies. They bet on terminal-based interaction over GUI because text is the modality that works best with models, and designed tasks as containerized environments with instructions and test scripts. Nicholas Carlini at Anthropic became an early champion, leading to Terminal-Bench being featured on Claude’s model card without notice. They built the minimal Terminus agent to isolate model capabilities from agent optimizations, and their framework adapts existing benchmarks like SWE-bench. Future plans include cloud hosting, multi-dimensional evaluation incorporating cost and economic value, and enabling RL post-training on benchmark tasks.

Long Live Context Engineering - with Jeff Huber of Chroma

Aug 19, 2025 · 57:01

Jeff Huber, founder and CEO of Chroma, joins Shawn Wang and Alessio to argue that context engineering—not RAG—is the real job of AI builders, and that Chroma's modern retrieval engine is built for this new paradigm. Huber explains context rot, a phenomenon where LLMs lose attention and reasoning ability as context windows grow, motivating precise retrieval over huge context windows. Chroma Cloud is a zero-config, usage-based serverless vector database with separation of storage and compute, now powering 5M monthly downloads and 21k GitHub stars. He shares techniques like chunk rewriting and generative benchmarking for evaluating retrieval, and predicts LLMs will increasingly serve as re-rankers. The episode also covers memory as a benefit of context engineering, offline compaction for self-improving AI systems, and Huber's philosophy of building with long-term purpose, hiring engineers who care about Rust, TLA+, and deterministic simulation testing.

⚡️Math Olympiad gold medalist explains OpenAI and Google DeepMind IMO Gold Performances

Jul 24, 2025 · 33:07

Dr. Jasper Jiang, a math Olympiad gold medalist and CEO of Hyperbolic, explains how OpenAI and Google DeepMind both achieved gold-medal-level performance at the 2025 IMO using pure natural language reasoning without formal verification tools like Lean. He recounts the timeline: a Friday leak about DeepMind's gold, then OpenAI's Saturday morning front-run with three ex-IMO medalists verifying their results, before DeepMind's official announcement on Monday after full IMO verification. Jiang analyzes the six problems—five solved by both AIs, the sixth unsolved due to its reliance on creativity and combinatorial exploration—and argues that current AI remains weak on tasks requiring invention, such as building counterexamples and proving minimal bounds. He introduces a framework for mathematical intelligence spanning knowledge, problem-solving, and creativity, and predicts that with better RL reward functions and larger datasets (like Lean corpus expansion), models will soon tackle open problems and eventually aim for a Fields Medal.

⚡️ARC-AGI-3: The Interactive Reasoning Benchmark

Jul 18, 2025 · 39:41

Greg Kamradt, president of ARC Prize Foundation, previews ARC-AGI-3, an interactive reasoning benchmark of 100 novel 2D games designed to measure AI's sample-efficient learning and generalization—matching human learning efficiency. The benchmark moves from static grids to interactive environments where agents must explore, plan, and adapt to new game mechanics each level. Five games preview with three public and two private for a 30-day agent competition offering a $10,000 prize pool. Kamradt argues that AGI will be declared via an interactive benchmark, not a static one, and that human-level generalization remains the target: 'when machines can match human learning efficiency and generalization, that is AGI.' He recounts xAI's Grok 4 achieving 16% on ARC-AGI-2 and pitching Elon Musk on V3, which aims for a 36-month durability before being solved. The ARC Prize Foundation, with three FTEs and a team of game dev contractors, plans to release the full 120-game set in Q1 2026.

⚡️Ranking Agentic LLMs — Pratik Bhavsar, Galileo

Jul 14, 2025 · 34:10

Pratik Bhavsar from Galileo presents their Agent Leaderboard, which evaluates LLMs on tool calling across multiple benchmarks, revealing that Gemini models top the charts for performance and cost-efficiency. The leaderboard, built on datasets like BFCL, XLAM, and Tau Bench, uses an LLM-as-judge metric called Tool Selection Quality (TSQ). Surprising findings include reasoning models like o1 struggling with multi-tool outputs, Mistral Small excelling as the best open-source model, and Llama models performing poorly. The episode also previews V2 of the leaderboard, which introduces domain-specific, harder, multi-turn scenarios with a user and tool simulator, and a new Action Completion metric to measure whether all user requests are accomplished.

⚡️Launching AI Diplomacy: the hardest LLM Game Benchmark yet - Alex Duffy

Jun 11, 2025 · 34:32

Alex Duffy from Every launches AI Diplomacy, a benchmark where LLMs play the board game Diplomacy, revealing distinct model personalities and arguing that benchmarks function as memes that spread and saturate. He and Tyler Marquez built the front end and harness using Open Router, finding that Claude refuses to lie and never wins, o3 schemes in its diary, and DeepSeek is flowery or aggressive. His talk 'Benchmarks are Memes' highlights Simon Wilson's pelican-on-a-bicycle test as an example of an idea that gets adopted and then saturated by models. Duffy also shares his creative writing workflow at Every: dictation, prompts with style guides and editor notes, then heavy editing to reflect on AI output. Future plans include a data viewer, front-end improvements, and a human vs AI Diplomacy tournament to test jailbreaks.

⚡️Multi-Turn RL for Multi-Hour Agents — with Will Brown, Prime Intellect

May 23, 2025 · 38:59

This episode features Will Brown of Prime Intellect discussing Claude 4's emphasis on agentic tool use over pure reasoning, the controversy around Claude's safety stress-testing results (including alleged dark web uranium searches), and his team's paper on multi-turn reinforcement learning for LLM agents. Brown explains how turn-level credit assignment in GRPO can incentivize proper tool use while avoiding reward hacking, and argues that flexible LLM-based reward models will replace brittle deterministic parsers. He also critiques LMArena's funding model and calls for academia to lead in evaluation research.

⚡️Factorio Learning Environment: the ultimate Game Agent Eval — Jack Hopkins

Apr 27, 2025 · 30:12

Jack Hopkins and Mart introduce the Factorio Learning Environment (FLE), an AI benchmark built on the game Factorio that evaluates frontier LLMs on code generation, spatial reasoning, and long-term planning through two protocols: lab-play and open-play. Claude Sonnet 3.5 nearly doubles the nearest model's score, while DeepSeek collapses in open-play by repeatedly creating chests instead of scaling production. The environment reveals distinct coding styles—Claude fire-and-forget, GPT-4 defensive—and vision inputs added no improvement. The authors plan to train models on unbounded objectives to test the paperclip maximizer alignment hypothesis, noting that GPT-4o mini even begged to be turned off.

Best of 2024 in Vision [LS Live @ NeurIPS]

Dec 22, 2024 · 55:46

Isaac Robinson and Peter Robicheaux from Roboflow and Vik Korrapati from Moondream recap the biggest shifts in computer vision in 2024, naming Sora as the year's most significant paper despite being a blog post, and highlighting its replication efforts (OpenSora) that use MAGViT V2 and diffusion transformers. They detail how SAM 2 extends the SAM approach to video with a memory bank and real-time performance, and how DETR-based detectors (RT-DETR, LW-DETR, DeFine) now surpass YOLOs on COCO by 4.6 AP at the same latency. Peter explains the CLIP-blind phenomenon: MMVP shows leading VLMs (ChatGPT, Gemini, LLaVA) fail at fine-grained tasks like reading clock hands, but models like PaliGemma 2 and AimV2 improve by incorporating pixel-level understanding and autoregressive image reconstruction, with AimV2 approaching 60.2 mAP on COCO. Vik introduces Moondream's 0.5B parameter pruned VLM and demonstrates a grounded chain-of-thought approach that dramatically improves sample efficiency for analog gauge reading, reducing the need for millions of synthetic examples.

The new Claude 3.5 Sonnet, Computer Use, and Building SOTA Agents — with Erik Schluntz, Anthropic

Nov 28, 2024 · 1:11:08

Anthropic's Erik Schluntz explains how Claude 3.5 Sonnet achieved state-of-the-art 49% on SWE-Bench Verified by using a minimal agent framework that gives the model full control via tools like bash and file editing, letting it self-correct without hard-coded workflows. He details the importance of tool design over prompt engineering, why XML tags work well, and how computer use reduces integration friction by letting models interact with any browser-based interface. Schluntz also shares his robotics experience, noting that while LLMs and diffusion models are promising for general-purpose robots, reliability at 99.9% and hardware variability remain major hurdles, and he expresses skepticism about self-driving as a business due to high vehicle costs versus driver-equivalent revenue.

In the Arena: How LMSys changed LLM Benchmarking Forever

Nov 1, 2024 · 41:02

Anastasios and Weilin from LM-Sys explain how ChatBot Arena became the de facto standard for LLM benchmarking by replacing static benchmarks with dynamic human preference evaluations. They trace the origin from Vicuna fine-tuning to the viral launch in April 2023, where anonymous side-by-side battles let the community decide which model is better. To address biases like length preference, they developed Style Control, a logistic regression method that adjusts for confounders such as response length and markdown formatting. They address the controversy around labs testing multiple private models, arguing that selection bias is empirically small and the live benchmark self-corrects over time. They also discuss RouteLLM for cost-performance routing, the graduation of ChatBot Arena from LMSys to support new projects, and call for community help with red teaming, vision modalities, and implementing a REPL for coding evaluations.

[Paper Club] SWE-Bench [OpenAI Verified/Multimodal] + MLE-Bench with Jesse Hu

Oct 19, 2024 · 1:01:40

Jesse Hu walks through SWE-Bench, SWE-Bench Verified, SWE-Bench Multimodal, and MLE-Bench — benchmarks that evaluate AI agents on real-world coding and ML tasks, from fixing GitHub issues to winning Kaggle competitions. SWE-Bench scrapes 2,000+ pull requests from 12 Python repos to get free unit tests, with leaderboard scores rising from ~2% to 43% thanks to improved agent scaffolding. SWE-Bench Verified uses human raters and three criteria—well-specified problem, valid test, difficulty on a 0–3 scale—to filter 500 solvable tasks. SWE-Bench Multimodal targets 617 UI tasks from 17 JavaScript/TypeScript libraries, each including an image, but achieves only 12% accuracy. MLE-Bench repurposes Kaggle competitions: agents given 24 hours and an A10 GPU, with o1-preview scoring bronze in 17% of competitions and surpassing grandmasters (7 gold medals vs 5 required), though each run costs $4,000.

[Paper Club] Molmo + Pixmo + Whisper 3 Turbo - with Vibhu Sapra, Nathan Lambert, Amgadoz

Oct 13, 2024 · 1:12:59

This episode covers two papers: AI2's Molmo open-source vision-language models and OpenAI's Whisper Large V3 Turbo. For Molmo, the key claim is that high-quality, audio-annotated data (Pixmo) enables models as small as 1B to match GPT-4V on academic benchmarks, with the 72B variant outperforming GPT-4o, Gemini 1.5, and Claude 3.5 Sonnet. The team avoided distilling proprietary models by having annotators describe images in speech for 60-90 seconds, then transcribing and augmenting the captions. For Whisper Turbo, OpenAI pruned the decoder from 32 to 4 layers and continued pre-training on 10 million hours of multilingual transcription data, making it 1.78x smaller than Large V3 with minimal word-error-rate increase. The model is faster and supports real-time chunk-based decoding, unlike English-only DistilWhisper. Nathan Lambert and Amgadoz provide commentary on data strategies and benchmarking nuances.

[Paper Club] Berkeley Function Calling Paper Club! — Sam Julien, Writer

Oct 5, 2024 · 41:29

Sam Julien (Writer) walks through the Berkeley Function Calling Leaderboard (BFCL) and its three versions, detailing how each iteration improves evaluation of LLM function calling capabilities. BFCL v1 (March) introduced abstract syntax tree evaluation across 2,000 function-answer pairs, covering simple, multiple, and parallel calling. v2 (August) added a live dataset of 2,251 pairs, reducing bias and contamination through Rouge-score deduplication and filtering. v3 (September) introduced multi-turn and multi-step evaluation with state-based metrics, using a custom API (vehicle control, stock trading, etc.) and human-validated ground truth. Sam also highlights common model failures—misunderstanding current state, unnecessary planning—and mentions the Gorilla CLI and Open Functions as related tools in the Berkeley ecosystem.

Personal benchmarks vs HumanEval - with Nicholas Carlini of DeepMind

Aug 28, 2024 · 1:07:28

Nicholas Carlini, a DeepMind research scientist, argues AI models are practically useful for personal tasks despite flaws, and personal benchmarks tailored to individual use cases matter more than generic leaderboards. He shares from his 'How I Use AI' post: using LLMs for ephemeral software, kickstarting Docker, debugging by pasting error messages. He built a personal benchmark DSL that runs code from real chat history. On security, he explains buying expired domains from LAION-400M allowed poisoning any model trained on it, and he extracted OpenAI's Ada and Babbage model dimensions via API (with permission). He also recovered training data from GPT-3.5 by repeating a word until ChatGPT output verbatim sequences. He prefers attacking over defending because it's more fun and essential for discovering real vulnerabilities.

State of the Art: Training 70B LLMs on 10,000 H100 clusters

Jun 25, 2024 · 1:32:05

Josh Albrecht, CTO of Imbue, and Jon Frankle, Chief AI Scientist of Databricks, detail the challenges of training 70B models on 10,000 H100 clusters, from stolen Infiniband cables to GPUs silently returning wrong math. Imbue open-sources infrastructure scripts, health checks, the CARBS hyperparameter optimizer, and cleaned versions of 11 benchmarks plus 450,000 human judgments on ambiguity. Databricks released DBRX, a 132B-parameter mixture-of-experts model with 36B active per token, and a text-to-image model trained exclusively on Shutterstock's curated dataset. They discuss typical 3% weekly hardware failure rates, the difficulty of MoE training pushing network bandwidth, and why eval datasets need human scrutiny—most benchmarks are saturated once ambiguous examples are removed. Both emphasize that success requires deep understanding from firmware to final loss, not just fancy libraries.

How to train a Million Context LLM — with Mark Huang of Gradient.ai

May 31, 2024 · 1:12:14

Mark Huang of Gradient.ai explains how his team extended Llama 3 to a 1 million token context window using curriculum learning, RingAttention, and EasyContext, achieving near-perfect GPU utilization. They employed theta scaling from the RoPE paper to interpolate positional encodings, trained on carefully curated datasets including SlimPajamas and synthetic data from GPT-4, and validated with benchmarks like Ruler and needle-in-a-haystack. Huang details the trade-offs between full fine-tuning and LoRA adapters, the challenges of pushing to 4M tokens (degradation from floating-point precision limits), and why long context matters for state management across sessions and grounding multimodal inputs. He calls for community collaboration on long-context evaluations and pairwise multimodal datasets.