Fullstack-Bench: The Eval for Coding Agents — with Sujay Jayakar, Chief Scientist, Convex

Mar 19, 2025 · 31:13

Sujay Jayakar, Convex chief scientist, presents Fullstack-Bench, a benchmark evaluating how well AI coding agents build full-stack apps on Convex, Supabase, and FastAPI+Redis, finding Convex’s tight feedback loops and strong abstractions let agents autonomously complete tasks within time limits. The benchmark tests three tasks (chat, to-do, files) across three backends with 30-60 minute limits and human hints. Convex succeeded autonomously on chat and to-do tasks, while Supabase agents struggled with RLS policy debugging and FastAPI agents with manual SSE wiring. Sujay discusses how platform design impacts AI code generation: simple procedural code, strong abstractions, and guardrails like end-to-end type safety improve LLM performance. He notes that API choices matter—things too similar to Firebase but not exact cause hallucinations—and that models like Claude 3.7 sometimes regress on platform-specific evals. The episode concludes with Convex’s focus on enabling AI-generated backends and anticipation of new agent workflow patterns.

3 episodes