Best of 2024 in Agents (from #1 on SWE-Bench Full, Prof. Graham Neubig of OpenHands/AllHands)
Dec 25, 2024 · 51:25
Professor Graham Neubig of CMU and AllHands AI, maintainer of the open-source coding agent framework OpenHands, delivers a talk on the state of agents in 2024, arguing that agents are now capable of automating significant portions of software development but still struggle with information gathering and error recovery. He demonstrates OpenHands solving real coding tasks live, explains key design choices like using Python code execution over granular tool calls to reduce LLM invocations, and shares that Claude currently outperforms other models for agentic tasks due to superior instruction following and error correction. Neubig discusses eight perennial agent problems including agent-computer interface, human-agent interface, model selection, planning, workflows, exploration, search, and evaluation. He predicts every major LLM trainer will focus on agent-oriented models by mid-2025, that agent benchmarks like SWE-Bench and WebArena will saturate and require harder successors, and that the biggest challenge will be designing human-agent interfaces for systems that succeed only 30-40% of the time autonomously but 80-90% with human feedback. The talk also covers workflow memory as a…
