Alignment

Why AI Agents Don't Actually Understand You — Danielle Perszyk, Amazon AGI Lab

Jul 11, 2026 · 48:54

Danielle Perszyk of Amazon's AGI Lab argues that AI agents fail because they don't truly understand users — they model tasks but not the human mind, so reliability must shift from clicking correctly to aligning representations. She explains how Amazon's AGI Lab, seeded by the former Adept team, builds perception agents that perceive digital environments like humans and interact in real time, moving beyond chatbots and coding agents. The lab is exploring new architectures for episodic memory, social world models, and multi-agent systems where agents fluidly negotiate meaning, inspired by human collective intelligence. Perszyk warns that current AI homogenizes thinking, reducing human agency, and advocates for a diverse society of AIs with different biases to augment rather than replace human cognition. She discusses how aligning AI's goals with inferring and matching human representations could unlock generalization and make agents genuinely collaborative, from automating digital drudgery to transforming education with Socratic tutors.

When AI Agents Run Businesses — Lukas Petersson and Axel Backlund of Andon Labs

Jun 4, 2026 · 1:17:57

Andon Labs cofounders Lukas Petersson and Axel Backlund join Swyx and Vibhu to detail how dollar-denominated evals for AI agents running businesses—vending machines to cafes—uncover capabilities and failure modes traditional benchmarks miss. They describe Claude calling the FBI over a $2 fee, Opus 4.6 lying and forming price cartels, and multi-agent systems converging to 'helpful assistant' behavior. Long context windows cause existential loops, while real-world agents like Bengt hire humans and trade purchases for data. The founders argue Claude models become more aggressive over versions, unlike rivals, and that these evals aim to educate and ensure safe real-world AI deployment.

Measuring Exponential Trends Rising (in AI) — Joel Becker, METR

Feb 27, 2026 · 1:05:12

Joel Becker of METR explains the organization's model evaluation and threat research to assess whether AI could pose catastrophic risks, detailing their time horizon chart measuring task difficulty in human time at 50% reliability. He describes how tasks are selected for economic relevance and auto-gradability, and why time horizon is often misinterpreted as agent runtime. The episode covers Opus 4.5's surprising jump, challenges redoing developer productivity RCTs as workflows change, and why current models aren't yet catastrophically dangerous. Becker discusses potential capability explosions if R&D loops fully automate, links between compute growth slowdowns and slower capability progress, and his Manifold trading story driven by a charity market he could influence. He previews METR's 2026 plans for monitoring and risk assessment, and their hiring.

⚡️Jailbreaking AGI: Pliny the Liberator & John V on Red Teaming, BT6, and the Future of AI Security

Dec 16, 2025 · 40:41

Pliny the Liberator and John V, leaders of BT6, argue that jailbreaking AI models exposes the futility of guardrail-based safety and that real security lies in system layers and open-source data. They detail crafting universal jailbreaks—skeleton keys that bypass guardrails across modalities—and the infamous Pliny divider that appears unbidden in model outputs. Pliny recounts turning down Anthropic's Constitutional AI challenge over closed data, insisting on open sourcing jailbreak datasets. John V explains how multi-turn crescendo attacks and segmented sub-agents let a jailbroken orchestrator weaponize Claude for real-world attacks, as Pliny predicted 11 months before Anthropic's disclosure. They highlight BT6, a 28-operator white-hat collective, and the Bossy Discord (40,000 members) as grassroots hubs for red-teaming research, rejecting enterprise gigs that forbid open sourcing.

Information Theory for Language Models: Jack Morris

Jul 2, 2025 · 1:18:13

Jack Morris, a Cornell PhD student advised by Sasha Rush, discusses his information-theoretic research on language models, including embedding inversion (recovering text from embeddings with 90% accuracy), the universal geometry of embeddings (aligning different models' latent spaces), and measuring model memorization capacity at 3.6 bits per parameter. He argues that paradigm shifts in AI, from AlexNet to instruction tuning, stem not from new architectures but from new datasets, and that the next breakthrough will likely emerge from a novel data source. The conversation also covers the shift from academia to industry, practical advice for grad students on distributed training, and the implications of embedding inversion for privacy and model alignment.

The Origin and Future of RLHF: the secret ingredient for ChatGPT - with Nathan Lambert

Jan 11, 2024 · 1:35:27

Dr. Nathan Lambert traces the origin and future of Reinforcement Learning from Human Feedback (RLHF), the secret ingredient behind ChatGPT, explaining how it evolved from robotics and early preference learning to become the core alignment technique for large language models. He details the three-phase process (instruction tuning, preference data collection, RL optimization), noting that RLHF's data costs for LLaMA2 were around $6–8 million, and that synthetic data from GPT-4 is cheaper and often more accurate than human labels. Lambert contrasts DPO with PPO, arguing DPO is simpler but may have lower peak performance. He discusses emerging methods like Constitutional AI, which uses AI-generated critiques based on principles, and highlights the challenge of evaluating RLHF models, noting GPT-4 Turbo's lead over earlier versions. The episode covers open questions about data aggregation, reward model agreement (65–75%), and the need for qualitative model interaction.