Imbue

The new Claude 3.5 Sonnet, Computer Use, and Building SOTA Agents — with Erik Schluntz, Anthropic

Nov 28, 2024 · 1:11:08

Anthropic's Erik Schluntz explains how Claude 3.5 Sonnet achieved state-of-the-art 49% on SWE-Bench Verified by using a minimal agent framework that gives the model full control via tools like bash and file editing, letting it self-correct without hard-coded workflows. He details the importance of tool design over prompt engineering, why XML tags work well, and how computer use reduces integration friction by letting models interact with any browser-based interface. Schluntz also shares his robotics experience, noting that while LLMs and diffusion models are promising for general-purpose robots, reliability at 99.9% and hardware variability remain major hurdles, and he expresses skepticism about self-driving as a business due to high vehicle costs versus driver-equivalent revenue.

State of the Art: Training 70B LLMs on 10,000 H100 clusters

Jun 25, 2024 · 1:32:05

Josh Albrecht, CTO of Imbue, and Jon Frankle, Chief AI Scientist of Databricks, detail the challenges of training 70B models on 10,000 H100 clusters, from stolen Infiniband cables to GPUs silently returning wrong math. Imbue open-sources infrastructure scripts, health checks, the CARBS hyperparameter optimizer, and cleaned versions of 11 benchmarks plus 450,000 human judgments on ambiguity. Databricks released DBRX, a 132B-parameter mixture-of-experts model with 36B active per token, and a text-to-image model trained exclusively on Shutterstock's curated dataset. They discuss typical 3% weekly hardware failure rates, the difficulty of MoE training pushing network bandwidth, and why eval datasets need human scrutiny—most benchmarks are saturated once ambiguous examples are removed. Both emphasize that success requires deep understanding from firmware to final loss, not just fancy libraries.

Why AI Agents Don't Work (yet) - with Kanjun Qiu of Imbue

Oct 21, 2023 · 1:12:37

Kanjun Qiu, CEO of Imbue (formerly Generally Intelligent), argues that AI agents remain unreliable because they lack robust reasoning and proper abstractions, which Imbue tackles by training foundation models optimized for reasoning over data like code. With a $200M Series B and $1B+ valuation, Imbue builds internal tools for debugging and inspecting agent decision-making, rejecting pure reinforcement learning after their Avalon environment revealed RL cannot handle planning. Instead, they emphasize natural language reasoning for inspectability, use code as a curriculum for reasoning, and design interfaces that let users fork and modify agents mid-execution. Qiu also discusses lessons from earlier startups (Sorceress in recruiting, Embark in VR), the importance of treating team members as creative agents, and her role in co-founding communal living space The Archive to foster scenius.