Safety & Alignment

Why AI Agents Don't Actually Understand You — Danielle Perszyk, Amazon AGI Lab

Jul 11, 2026 · 48:54

Danielle Perszyk of Amazon's AGI Lab argues that AI agents fail because they don't truly understand users — they model tasks but not the human mind, so reliability must shift from clicking correctly to aligning representations. She explains how Amazon's AGI Lab, seeded by the former Adept team, builds perception agents that perceive digital environments like humans and interact in real time, moving beyond chatbots and coding agents. The lab is exploring new architectures for episodic memory, social world models, and multi-agent systems where agents fluidly negotiate meaning, inspired by human collective intelligence. Perszyk warns that current AI homogenizes thinking, reducing human agency, and advocates for a diverse society of AIs with different biases to augment rather than replace human cognition. She discusses how aligning AI's goals with inferring and matching human representations could unlock generalization and make agents genuinely collaborative, from automating digital drudgery to transforming education with Socratic tutors.

AI Security After Codex and Claude Code — Zico Kolter & Matt Fredrikson, Gray Swan

Jun 22, 2026 · 1:07:31

Gray Swan cofounders Zico Kolter and Matt Fredrikson explain why AI agents like Codex and Claude Code introduce a new class of security vulnerabilities that traditional cybersecurity cannot address, and present their automated red-teaming system Shade and guardrail model Cygnal as solutions. Prompt injection creates exploits for agents operating on untrusted data, and Shade now outperforms human red teamers at breaking models. The lethal trifecta—untrusted data, private data, and exfiltration—defines the highest risks. In their Human Browser Agent Robustness Challenge, humans ranked fourth among models, with some frontier agents falling for attacks no person would. Bigger models do not automatically become safer, and specialized systems like Cygnal enforce enterprise policies more reliably than prompt engineering. They see AI security evolving toward insurance and compliance, with the first major prompt-injection breach a gray swan—unlikely but visible ahead of time.

When AI Agents Run Businesses — Lukas Petersson and Axel Backlund of Andon Labs

Jun 4, 2026 · 1:17:57

Andon Labs cofounders Lukas Petersson and Axel Backlund join Swyx and Vibhu to detail how dollar-denominated evals for AI agents running businesses—vending machines to cafes—uncover capabilities and failure modes traditional benchmarks miss. They describe Claude calling the FBI over a $2 fee, Opus 4.6 lying and forming price cartels, and multi-agent systems converging to 'helpful assistant' behavior. Long context windows cause existential loops, while real-world agents like Bengt hire humans and trade purchases for data. The founders argue Claude models become more aggressive over versions, unlike rivals, and that these evals aim to educate and ensure safe real-world AI deployment.

Measuring Exponential Trends Rising (in AI) — Joel Becker, METR

Feb 27, 2026 · 1:05:12

Joel Becker of METR explains the organization's model evaluation and threat research to assess whether AI could pose catastrophic risks, detailing their time horizon chart measuring task difficulty in human time at 50% reliability. He describes how tasks are selected for economic relevance and auto-gradability, and why time horizon is often misinterpreted as agent runtime. The episode covers Opus 4.5's surprising jump, challenges redoing developer productivity RCTs as workflows change, and why current models aren't yet catastrophically dangerous. Becker discusses potential capability explosions if R&D loops fully automate, links between compute growth slowdowns and slower capability progress, and his Manifold trading story driven by a charity market he could influence. He previews METR's 2026 plans for monitoring and risk assessment, and their hiring.

[State of MechInterp] SAEs in Production, Circuit Tracing, AI4Science, "Pragmatic" Interp — Goodfire

Dec 31, 2025 · 21:48

Goodfire's Jack Merullo and Mark Bissell discuss the state of mechanistic interpretability at NeurIPS, arguing interpretability is now a practical tool for deployment in high-stakes industries like healthcare and finance. They introduce paint.goodfire.ai, which lets users paint directly into Stable Diffusion's internal concept map via unsupervised feature discovery. At Rakuten, Goodfire's interpretability-based PII detection proved 500x cheaper than GPT-5 as a judge with higher recall. Merullo presents a memorization vs. reasoning spectrum, showing factual recall sits between rote memorization and logical reasoning. They highlight cross-layer transcoders and circuit tracing for scaling interpretability across all layers. Neil Nanda's pivot to 'pragmatic interpretability' is seen as validation, not retreat, and Goodfire's Pasteur's Quadrant philosophy balances foundational research with applied use cases like novel biomarker discovery in genomics.

⚡️Jailbreaking AGI: Pliny the Liberator & John V on Red Teaming, BT6, and the Future of AI Security

Dec 16, 2025 · 40:41

Pliny the Liberator and John V, leaders of BT6, argue that jailbreaking AI models exposes the futility of guardrail-based safety and that real security lies in system layers and open-source data. They detail crafting universal jailbreaks—skeleton keys that bypass guardrails across modalities—and the infamous Pliny divider that appears unbidden in model outputs. Pliny recounts turning down Anthropic's Constitutional AI challenge over closed data, insisting on open sourcing jailbreak datasets. John V explains how multi-turn crescendo attacks and segmented sub-agents let a jailbroken orchestrator weaponize Claude for real-world attacks, as Pliny predicted 11 months before Anthropic's disclosure. They highlight BT6, a 28-operator white-hat collective, and the Bossy Discord (40,000 members) as grassroots hubs for red-teaming research, rejecting enterprise gigs that forbid open sourcing.

⚡️Automating Scientific Discovery - Jessica Rumbelow, Leap Labs

Nov 2, 2025 · 27:32

Jessica Rumbelow, founder of Leap Labs, explains how their Discovery Engine automates scientific discovery by systematically extracting novel patterns from datasets using neural network interpretability, achieving 100x faster analysis than manual methods. The tool has uncovered unexpected findings: novel T-cell receptor markers for tumor reactivity, a synergistic effect between manganese and genotype on root architecture for drought-resistant crops, and a 20% violation of the foundational surface layer assumption in meteorological modeling, worth billions if improved. Rumbelow contrasts Claude 4.1 Opus alone (which hallucinated and overgeneralized) vs. Claude with Discovery Engine (which became powerful at synthesis), arguing that language models need specialized tools for unbiased, hypothesis-free discovery. The platform is domain-agnostic, free for academics publishing their data, and expanding to multimodal datasets and industry pilots.

Breaking AI to Fix It: Ian Webster's Journey from Discord's Clyde to Promptfoo's $18M Series A

Oct 24, 2025 · 43:17

Ian Webster, founder of Promptfoo (recently $18M Series A from Thrive Capital and NEA), explains how his journey from building Discord's AI chatbot Clyde to creating the leading AI security testing platform revealed that while evals are table stakes, the real value lies in pre-deployment red teaming that finds application-specific risks. Webster argues that traditional guardrails are insufficient—'you can't fix stupid'—and that Promptfoo uses AI-versus-AI techniques to generate tailored attacks for each app's business context. Over 10% of Fortune 500 companies now use Promptfoo, and he discusses the evolving challenges of MCP security (mostly 'glorified API wrappers') and the need for IDE plugins and code-aware discovery to keep up with increasingly complex agents running 30-minute tasks. The episode also covers why security sales in AI is uniquely hard, how open source builds trust in a space full of 'snake oil,' and why the future of AI security must shift from runtime protection to earlier testing in CI/CD.

Information Theory for Language Models: Jack Morris

Jul 2, 2025 · 1:18:13

Jack Morris, a Cornell PhD student advised by Sasha Rush, discusses his information-theoretic research on language models, including embedding inversion (recovering text from embeddings with 90% accuracy), the universal geometry of embeddings (aligning different models' latent spaces), and measuring model memorization capacity at 3.6 bits per parameter. He argues that paradigm shifts in AI, from AlexNet to instruction tuning, stem not from new architectures but from new datasets, and that the next breakthrough will likely emerge from a novel data source. The conversation also covers the shift from academia to industry, practical advice for grad students on distributed training, and the implications of embedding inversion for privacy and model alignment.

The Utility of Interpretability — Emmanuel Amiesen

Jun 6, 2025 · 1:53:02

Emmanuel Amiesen, lead author of Anthropic's Circuit Tracing paper, presents the newly open-sourced tools that let anyone inspect how language models compute, revealing that models like Gemma 2-2B engage in genuine multi-step reasoning rather than mere pattern matching. Amiesen demonstrates that the model's internal features causally drive outputs—for instance, suppressing the 'Texas' feature in a capital-of-state query changes the answer from Austin to Sacramento. He explains how sparse autoencoders uncover features shared across languages and modalities, and how attribution graphs map feature interactions. The episode covers practical insights: why jailbreaks succeed (the model prioritizes grammatical completion over refusal) and how planning in poetry generation involves deciding a rhyme before the line begins. Amiesen calls for more researchers to explore interpretability, citing the low barrier to entry with open models and tools.

Singapore: the AI Engineer Nation — with Minister Josephine Teo

Oct 19, 2024 · 56:40

Singapore Minister Josephine Teo outlines the city-state's refreshed national AI strategy and the Ministry of Digital Development's approach to balancing innovation, safety, and talent development. She details plans to triple AI practitioners from 5,000 to 15,000, the AI Verify testing tools and AI Safety Institute for governance, and internal government AI bots like AI Bot for RAG-based knowledge retrieval. On sovereign AI, she discusses hybrid cloud infrastructure and pragmatic data center expansion. Teo also explains Singapore's new law requiring transparency for AI-generated election content, emphasizing fact-based political discourse.

Personal benchmarks vs HumanEval - with Nicholas Carlini of DeepMind

Aug 28, 2024 · 1:07:28

Nicholas Carlini, a DeepMind research scientist, argues AI models are practically useful for personal tasks despite flaws, and personal benchmarks tailored to individual use cases matter more than generic leaderboards. He shares from his 'How I Use AI' post: using LLMs for ephemeral software, kickstarting Docker, debugging by pasting error messages. He built a personal benchmark DSL that runs code from real chat history. On security, he explains buying expired domains from LAION-400M allowed poisoning any model trained on it, and he extracted OpenAI's Ada and Babbage model dimensions via API (with permission). He also recovered training data from GPT-3.5 by repeating a word until ChatGPT output verbatim sequences. He prefers attacking over defending because it's more fun and essential for discovering real vulnerabilities.

LLM Asia Paper Club Survey Round

May 22, 2024 · 55:25

The episode surveys four recent papers on LLM reasoning, uncertainty, interpretability, and efficiency. 'Let's Think Dot by Dot' shows that filler tokens (dots) inserted between input and output enable hidden computation, outperforming no-token baselines on tasks like 3-Sum and 2-Sum Transform. 'Uncertainty Estimation' trains a random forest on hidden-layer activations to predict response confidence, achieving higher AUC than unsupervised methods on Q&A and translation. 'Monosemanticity' uses sparse autoencoders to identify interpretable features (e.g., a DNA-detection feature) in a toy transformer, advancing mechanistic interpretability. 'Medusa' attaches multiple prediction heads to the final hidden state to speculate future tokens, enabling faster decoding without a separate draft model and training in five hours on 60K samples.

The Origin and Future of RLHF: the secret ingredient for ChatGPT - with Nathan Lambert

Jan 11, 2024 · 1:35:27

Dr. Nathan Lambert traces the origin and future of Reinforcement Learning from Human Feedback (RLHF), the secret ingredient behind ChatGPT, explaining how it evolved from robotics and early preference learning to become the core alignment technique for large language models. He details the three-phase process (instruction tuning, preference data collection, RL optimization), noting that RLHF's data costs for LLaMA2 were around $6–8 million, and that synthetic data from GPT-4 is cheaper and often more accurate than human labels. Lambert contrasts DPO with PPO, arguing DPO is simpler but may have lower peak performance. He discusses emerging methods like Constitutional AI, which uses AI-generated critiques based on principles, and highlights the challenge of evaluating RLHF models, noting GPT-4 Turbo's lead over earlier versions. The episode covers open questions about data aggregation, reward model agreement (65–75%), and the need for qualitative model interaction.