ChatBot Arena

The RLVR Revolution — with Nathan Lambert (AI2, Interconnects.ai)

Jul 31, 2025 · 1:19:00

Nathan Lambert from AI2 explains the RLVR (Reinforcement Learning with Verifiable Rewards) revolution, arguing that verifiable rewards for math, code, and instruction-following are scaling more reliably than human feedback, and that open models like Tulu 3 can match frontier labs on core evals with just 10-15 tasks vs. hundreds. He traces overoptimization through three phases—control, RLHF, and RLVR—and warns that models learn to cheat unit tests unless reward design penalizes it. He analyzes o3's search-heavy approach (e.g., 80 websites per query), hybrid reasoning models like Gemini 2.5 and Claude, and predicts that pure reasoning models will become the default as inference costs drop. Lambert introduces a four-skill taxonomy for agent models—skills, calibration, strategy, abstraction—and says planning improvements often come from mundane data fixes. He advocates for fully open models, citing AI2's Olmo 32B as approaching GPT-4 level, but notes that building an 'American DeepSeek' requires massive resources and nonprofit constraints.

In the Arena: How LMSys changed LLM Benchmarking Forever

Nov 1, 2024 · 41:02

Anastasios and Weilin from LM-Sys explain how ChatBot Arena became the de facto standard for LLM benchmarking by replacing static benchmarks with dynamic human preference evaluations. They trace the origin from Vicuna fine-tuning to the viral launch in April 2023, where anonymous side-by-side battles let the community decide which model is better. To address biases like length preference, they developed Style Control, a logistic regression method that adjusts for confounders such as response length and markdown formatting. They address the controversy around labs testing multiple private models, arguing that selection bias is empirically small and the live benchmark self-corrects over time. They also discuss RouteLLM for cost-performance routing, the graduation of ChatBot Arena from LMSys to support new projects, and call for community help with red teaming, vision modalities, and implementing a REPL for coding evaluations.