SWE-Agent

The new Claude 3.5 Sonnet, Computer Use, and Building SOTA Agents — with Erik Schluntz, Anthropic

Nov 28, 2024 · 1:11:08

Anthropic's Erik Schluntz explains how Claude 3.5 Sonnet achieved state-of-the-art 49% on SWE-Bench Verified by using a minimal agent framework that gives the model full control via tools like bash and file editing, letting it self-correct without hard-coded workflows. He details the importance of tool design over prompt engineering, why XML tags work well, and how computer use reduces integration friction by letting models interact with any browser-based interface. Schluntz also shares his robotics experience, noting that while LLMs and diffusion models are promising for general-purpose robots, reliability at 99.9% and hardware variability remain major hurdles, and he expresses skepticism about self-driving as a business due to high vehicle costs versus driver-equivalent revenue.

Language Agents: From Reasoning to Acting — with Shunyu Yao of OpenAI, Harrison Chase of LangGraph

Sep 27, 2024 · 1:26:33

Shunyu Yao of OpenAI and Harrison Chase of LangChain discuss the evolution of language agents, arguing that combining reasoning and acting through techniques like ReAct remains foundational, while tool design and cognitive architectures like CoALA will shape the next wave. Shunyu traces his work from text games to ReAct, showing how thinking as an action improves reliability; Harrison explains LangChain's adoption and the shift to LangGraph for stateful orchestration. They emphasize agent-computer interfaces (ACI) as a neglected area, noting that SWE-Agent's success came from optimizing tools for models, not humans. The conversation covers memory types (semantic, episodic, procedural) and applies CoALA's three dimensions to current systems. They highlight promising applications: customer support, coding agents, and spreadsheet-style UX for batch operations. Shunyu stresses that better benchmarks like SWE-Bench (which solved 30% of GitHub issues) and TauBench (best model 48%) are critical, and that data—not architecture—will drive future model improvements.