Twitch

Claude Plays Pokémon Hackathon: Escape from Mt. Moon!

Apr 5, 2025 · 1:15:00

David Hershey (creator of Claude Plays Pokémon), Andrew (a developer building a Pokémon-playing virtual streamer), and Jesse Han (Morph Labs CEO) detail how to build agents that escape Mt. Moon in Pokémon FireRed using Anthropic's Claude and Morph Cloud's Infinibranch technology. Hershey explains that Claude's vision is often unreliable—it spent 8 hours pressing A on a doormat thinking it was a dialogue box—and that spatial reasoning remains a core bottleneck, best mitigated by touchscreen controls rather than prompt engineering. Andrew reveals he relied on reading raw RAM and A* pathfinding rather than computer vision, calling it “cheating” but pragmatically useful: his biggest breakthrough was a single prompt line telling the agent to try something else if it fails to grab a starter Pokémon. Jesse introduces Morph's EVA agent framework, which uses low-overhead snapshotting for test-time search, and the hackathon's judging criteria—escape Mt. Moon in the fewest agent turns, with a $1,000 prize for the coolest use of Infinibranch branching. The episode serves as both a technical walkthrough and a challenge to build general-purpose agents that can handle complex, open-ended tasks…

How Claude Plays Pokémon was made

Mar 4, 2025 · 37:38

David Hershey of Anthropic explains how he built Claude Plays Pokémon, an agent that uses Claude to play Pokémon Red, revealing both surprising capabilities and stark limitations of AI. He describes the simple tool-using agent loop: Claude sends button presses, receives screenshots and RAM state, and maintains a knowledge base and 30-message conversation history with summarization. Key challenges include Claude's poor spatial awareness—it famously walked up and down outside Oak's lab 12 times—and its tendency to hallucinate game knowledge. Hershey notes that newer models (Sonnet 3.7) required stripping away band-aid prompts, as simpler instructions yielded better performance. The project also showed emergent behaviors: nicknamed Pokémon made Claude more protective, and the model even wrote meta-commentary about its own mistakes in its knowledge base. While the current run is stuck in Mount Moon after 52 hours, the biggest hurdles remain vision and navigation, not prompting tricks.