MT-Bench

In the Arena: How LMSys changed LLM Benchmarking Forever

Nov 1, 2024 · 41:02

Anastasios and Weilin from LM-Sys explain how ChatBot Arena became the de facto standard for LLM benchmarking by replacing static benchmarks with dynamic human preference evaluations. They trace the origin from Vicuna fine-tuning to the viral launch in April 2023, where anonymous side-by-side battles let the community decide which model is better. To address biases like length preference, they developed Style Control, a logistic regression method that adjusts for confounders such as response length and markdown formatting. They address the controversy around labs testing multiple private models, arguing that selection bias is empirically small and the live benchmark self-corrects over time. They also discuss RouteLLM for cost-performance routing, the graduation of ChatBot Arena from LMSys to support new projects, and call for community help with red teaming, vision modalities, and implementing a REPL for coding evaluations.

The Origin and Future of RLHF: the secret ingredient for ChatGPT - with Nathan Lambert

Jan 11, 2024 · 1:35:27

Dr. Nathan Lambert traces the origin and future of Reinforcement Learning from Human Feedback (RLHF), the secret ingredient behind ChatGPT, explaining how it evolved from robotics and early preference learning to become the core alignment technique for large language models. He details the three-phase process (instruction tuning, preference data collection, RL optimization), noting that RLHF's data costs for LLaMA2 were around $6–8 million, and that synthetic data from GPT-4 is cheaper and often more accurate than human labels. Lambert contrasts DPO with PPO, arguing DPO is simpler but may have lower peak performance. He discusses emerging methods like Constitutional AI, which uses AI-generated critiques based on principles, and highlights the challenge of evaluating RLHF models, noting GPT-4 Turbo's lead over earlier versions. The episode covers open questions about data aggregation, reward model agreement (65–75%), and the need for qualitative model interaction.