Minecraft

⚡️Factorio Learning Environment: the ultimate Game Agent Eval — Jack Hopkins

Apr 27, 2025 · 30:12

Jack Hopkins and Mart introduce the Factorio Learning Environment (FLE), an AI benchmark built on the game Factorio that evaluates frontier LLMs on code generation, spatial reasoning, and long-term planning through two protocols: lab-play and open-play. Claude Sonnet 3.5 nearly doubles the nearest model's score, while DeepSeek collapses in open-play by repeatedly creating chests instead of scaling production. The environment reveals distinct coding styles—Claude fire-and-forget, GPT-4 defensive—and vision inputs added no improvement. The authors plan to train models on unbounded objectives to test the paperclip maximizer alignment hypothesis, noting that GPT-4o mini even begged to be turned off.

Why AI Agents Don't Work (yet) - with Kanjun Qiu of Imbue

Oct 21, 2023 · 1:12:37

Kanjun Qiu, CEO of Imbue (formerly Generally Intelligent), argues that AI agents remain unreliable because they lack robust reasoning and proper abstractions, which Imbue tackles by training foundation models optimized for reasoning over data like code. With a $200M Series B and $1B+ valuation, Imbue builds internal tools for debugging and inspecting agent decision-making, rejecting pure reinforcement learning after their Avalon environment revealed RL cannot handle planning. Instead, they emphasize natural language reasoning for inspectability, use code as a curriculum for reasoning, and design interfaces that let users fork and modify agents mid-execution. Qiu also discusses lessons from earlier startups (Sorceress in recruiting, Embark in VR), the importance of treating team members as creative agents, and her role in co-founding communal living space The Archive to foster scenius.