# Fullstack-Bench: The Eval for Coding Agents — with Sujay Jayakar, Chief Scientist, Convex Latent Space · 2025-03-19 Sujay Jayakar, Convex chief scientist, presents Fullstack-Bench, a benchmark evaluating how well AI coding agents build full-stack apps on Convex, Supabase, and FastAPI+Redis, finding Convex’s tight feedback loops and strong abstractions let agents autonomously complete tasks within time limits. The benchmark tests three tasks (chat, to-do, files) across three backends with 30-60 minute limits and human hints. Convex succeeded autonomously on chat and to-do tasks, while Supabase agents struggled with RLS policy debugging and FastAPI agents with manual SSE wiring. Sujay discusses how platform design impacts AI code generation: simple procedural code, strong abstractions, and guardrails like end-to-end type safety improve LLM performance. He notes that API choices matter—things too similar to Firebase but not exact cause hallucinations—and that models like Claude 3.7 sometimes regress on platform-specific evals. The episode concludes with Convex’s focus on enabling AI-generated backends and anticipation of new agent workflow patterns. ## Questions this episode answers ### What is the Fullstack-Bench benchmark for coding agents? Fullstack-Bench is a manual eval created by Convex where an AI coding agent (like Cursor Composer) must implement the backend for full-stack apps. It includes three tasks: a chat app, a to-do app, and a files app, with time limits of 30 minutes for the first two and 60 minutes for the files task. The agent receives a pre-built frontend and can choose among Convex, Supabase, or FastAPI+Redis backends, with human hints if stuck. [2:57](https://addtry.com/e993117f-30d2-4916-9c85-1af420e7ff20?t=177000) ### How did Convex compare to Supabase and FastAPI in the Fullstack-Bench results? According to Sujay, with Convex, the AI agent autonomously solved the chat and to-do tasks with few errors, and nearly completed the files task within 60 minutes. In contrast, on Supabase and FastAPI+Redis, the model got lost and made no progress. Sujay attributes Convex's success to tight feedback loops and procedural code, while the other stacks introduced complexities like RLS rules and manual SSE setup that confused the agent. [9:43](https://addtry.com/e993117f-30d2-4916-9c85-1af420e7ff20?t=583000) ### What platform design choices does Convex recommend for AI-friendly coding tools? Sujay highlights several: tight feedback loops, strong guardrails like end-to-end type safety, and using standard procedural code rather than non-procedural DSLs. He noted that models struggled with Supabase's RLS rules (causing infinite recursion) and got confused by subtle API differences (e.g., Convex's null vs undefined). Strong abstractions and avoiding surprising behavior help AI agents generate correct code. [28:58](https://addtry.com/e993117f-30d2-4916-9c85-1af420e7ff20?t=1738000) ### Why did Claude 3.7 perform worse than Claude 3.5 on Convex's own evals? On Convex's internal 'Convex evals'—quiz-like tests of Convex knowledge—Claude 3.7 scored lower than Claude 3.5 with the same prompts. Sujay suspects that because their cursor rules were optimized for Claude 3.5's knowledge gaps, they don't cover Claude 3.7's different hallucinations. New models may hallucinate different things, so rules need updating, and performance isn't guaranteed to improve with newer releases. [17:23](https://addtry.com/e993117f-30d2-4916-9c85-1af420e7ff20?t=1043000) ## Key moments - **[0:00] Intro** - [0:17] Sujay Jayakar: "Co-founder and chief scientist, whatever that means." - **[0:36] Convex Overview** - [0:36] Convex is a reactive database built from first principles as a competitor to Firebase. - [1:16] Convex began rigorous AI coding experiments to make their platform more amenable to autonomous development. - **[1:32] Benchmark Motivation** - **[2:57] Eval Setup** - [3:37] Cursor Composer with Claude 3.5 autonomously coded a full-stack to-do app for over ten minutes, fixing its own bugs. - **[4:08] Design Choices** - **[5:25] Task Matrix** - [7:15] Fullstack-Bench starts from a scaffolded frontend to focus on backend generation and state management. - **[8:01] Results & Hints** - [8:01] Convex backend autonomously solved two tasks; FastAPI and Supabase backends confused the model. - [8:55] Fullstack-Bench uses a time limit, human hints when stuck, and a predetermined rubric for manual grading. - [10:38] Q: How do you give hints to a coding agent that has been running for 40 minutes and you need to catch up? - [11:20] For FastAPI chat task, Sujay gave hint to directly use useEffect after model got confused with TanStack Query and SSE. - **[11:55] Meta Evals** - [12:54] Convex's subtle distinction between null and undefined caused the model to get stuck, requiring explicit expert hint. - **[14:02] Eval Comparisons** - [14:24] Fullstack-Bench is more focused and easier to grade compared to SWE-bench, offering a trade-off between generality and automated evaluation. - [16:59] Claude 3.7 performs worse than Claude 3.5 on Convex evals with the same prompting, suggesting different hallucination patterns. - **[18:16] Future Work** - [18:54] Future Fullstack-Bench work aims to use Jepsen model checker for automatic correctness grading of highly concurrent backend code. - **[20:32] AI Age** - [20:57] Sujay: Making a developer platform a great AI coding target is table stakes in 2025. - [23:12] Many AI coding hallucinations on Convex occur because its API is close but not identical to Firebase. - [24:09] Swyx: "AI calcifies your API. You are locked into whatever pre-existed pre-AI." - **[26:17] Agents** - [26:43] Sujay outlines two AI coding paths: building same apps faster vs creating new AI-native backends; Convex focuses on the former. - **[28:58] Takeaways** - [29:46] AI models struggled debugging Supabase RLS recursion loops, but handled procedural Convex code better. - [30:21] Sujay: Strong libraries with good abstractions are a better bet for AI coding than giving models full flexibility from scratch. ## Speakers - **Swyx** (host) - **Alessio** (guest) - **Sujay Jayakar** (guest) ## Topics Agent Platforms ## Mentioned Convex (company), Supabase (company), 4o (product), AI Town (product), Bolt (product), Brain Trust (product), Claude (product), Cursor Composer (product), FastAPI (product), Firebase (product), Fullstack-Bench (product), Jepsen (product), MySQL (product), Next (product), Postgres (product), Redis (product), SWE-Bench (product), SWE-Lancer (product), TanStack Query (product), o3 (product) ## Transcript ### Intro **Alessio** [0:03] Hey, everyone. Welcome back to a new Latent Space Lightning Pod. This is Alessio, partner and CTO at Decibel, and I'm joined by my co-host, Swyx, founder of Small AI. **Swyx** [0:12] Hey, and today we're very happy to have Sujay, who is co-founder, CTO of Convex. **Sujay Jayakar** [0:17] Co-founder and chief scientist, whatever that means. **Swyx** [0:21] So Convex, you've been on my radar for a long while because, uh, you have a really interesting sort of database and compute platform, which you're gonna talk about, uh, la-- in the second half of this chat. You, uh, Jamie, reached out to me specifically to talk about this new benchmark that you guys are working on. ### Convex Overview **Sujay Jayakar** [0:36] Totally. So yeah, just, you know, for context, Convex is like a competitor to Firebase. It's a reactive database. It's a database that's thought from first principles on what if we didn't have to use, you know, MySQL and Postgres and the same things we've been doing for thirty plus years. What if we built something kind of like Firebase that was purpose-built for full stack app development and made it as powerful and delightful as possible? And, you know, over the journey of building Convex and, you know, talking to users, we've seen, um, in the past, you know, six months, so many more users are writing code for Convex using all the coding tools that have been taking off. And we've kind of like, you know, been supporting people and cheering them on, and it's like in the past few months, we, you know, given how much it's changed coding, we decided to just start to be a little bit more rigorous and start doing some, uh, experimentation and trying to be more quantitative and scientific about it. And the results are, you know, pretty interesting. ### Benchmark Motivation **Swyx** [1:34] Yeah. So I don't know if you wanna kick off your screen share. **Sujay Jayakar** [1:36] Yeah. **Swyx** [1:36] We can probably get that up so people can take a look at it. Yeah, I think so, like, basically what you're say-- what I'm hearing from you is you, you did a bunch of evals on behalf of your user community, right? And also, like it, it's a way for you to benchmark yourself against the competition. **Sujay Jayakar** [1:52] Yeah, totally. And so I think there's like this kind of dream of having fully autonomous full stack app development is something that like a bunch of users came to us a few months ago, and some of them are kinda running software consultancy startups where they... customers come in, and then they u-mo-mostly using AI coding build apps for them. Others are kind of like doing stuff more in the bolt space, where their users are coming up with prompts, and then they're building a whole full stack app. Some of them are web, some of them are mobile. And yeah, so this is trying to make this more rigorous and something that we can run ourselves for the benefit of being able to reshape our platform to make it as amenable to autonomous vibe coding or AI coding, whatever you wanna call it. So this is a post I just put out, uh, yesterday, and I think it's like, you know, kind of one of the moments where we were running these experiments and my like, you know, ears perked up a bit. ### Eval Setup **Sujay Jayakar** [2:57] And we'll like kind of flip through this pretty quickly. But like, you know, with this, uh, particular, um, eval, you know, we have the agent, uh, filling out the backend for a to-do app. So we kind of noticed that, you know, stuff like D0 already generates front-end stuff quite well. So this is like, how can a model flush out the data model, the network transport, the state management? And you know, we're not gonna have, gonna have to watch through all of this, but it's pretty remarkable to just see that when it has the right feedback, Cursor Composer, and this was even on Cloud three/five, it can just autonomously code for, uh, ten plus minutes, and it can fix its own bugs. It can get to the point where it's like pretty much a fully working app with just an initial prompt. And so being able to now reproduce this and see how it changes to the platform or changes to the backend, you know, like we... this framework supports Convex, but we also compare it to Supabase, and we compare it to like a more traditional kind of three-tier stack with, um, Python, FastAI and Redis, FastAPI and Redis. And so that lets us kind of study like what works well and what wor- what doesn't work. ### Design Choices **Alessio** [4:08] One of the things we always highlight about Devin is, you know, you have the code, you have the environment and all that. What are like some of the design decisions that you guys made here on what to expose to the models? Like, you're not bringing people to code somewhere else, you're just letting them do this in their existing ID. Um, did you ever think about building your own kind of like all-in-one environment for people to work in or? **Sujay Jayakar** [4:30] Yeah. I think some of it is, um, you know, for this particular study, it's very manual still. And so I think like I was just most familiar with Cursor Composer. So I like... we have a standardized environment where we create the front-end app in Next, and we set up authentication, and we set up some user tables and stuff like that. And then we, from that starting point, kind of let Cursor Composer go off. But I think, you know, it's like kind of that future working of, um, I would love to just be able to, uh, use like Cloud Code, use, um, or even just make stuff more standardized where it's like running in, you know, it's not just running on my laptop, it's running on some containerized environment I can do at scale. **Swyx** [5:12] Yeah, interesting. So, uh, so you have three... uh, I think if, if I scroll down on your benchmark here, you have three tasks, right? You've, you've got the chat, you got the to-do, you got the files, and I think that-that's a pretty reasonable thing to start off. ### Task Matrix **Swyx** [5:25] Obviously, you could do more than that, but three is a pretty good start. One exercise that, um... there are a couple exercises that I think that come to mind. One is the Seven GUIs projects, uh, which is a, a similar set of tasks that are fully-- supposed to fully circumscribe the common tasks that you do in front end. I think adding, adding back ends to that is, is a pretty interesting task. And then there's also, there's this old project run by Thinkster, uh, who now runs Bolt, that does, uh, sort of real world comparisons. So like I think it's something like build a medium clone, but you swap out any front end, you swap out any back end. **Alessio** [6:08] Is that-- I, I don't know, I don't know if that rings a bell. Well, anyway, th-that's, that's really cool. So you have the three tasks, and then you also have the three backends that you add in, right? So the... You mentioned the API and, um, but you also have Supabase, and you also have Con-Convex. Something that's interesting is like-- So, you know, obviously I, I read that and I'm like, "Okay, so it's like a three-by-three matrix where you just, you know, you, you, you, you, you make, uh, as much coding as possible." Um, but I think what's weird here is also the, the missing part is the scaffold that people use. **Sujay Jayakar** [6:41] Mm-hmm. **Alessio** [6:41] Uh, and different-- obviously, different scaffolds would, will lead to different results. **Sujay Jayakar** [6:44] Yeah. I mean, and I think, like, in both... Like, for, for this study, I think, like, scoping out a lot was, like, I think one of the-- just, you know, in terms of, like, trying to get something, get to the interesting parts. You know, I think, like, same way of, like, kind of starting with the front-end app fully fleshed out. I did kind of... Yeah, to your point, like, I kind of assumed that a lot of these, like, full-stack but, like, autonomous coding tasks are going to start fr-from some form of template, right? Like, you know, there's, um... In that, like, these tasks don't se- uh, they don't test the agent's ability to, you know, initialize a new Next app, wire up auth, and configure everything. And it's a fair kind of point that, like, actually being able to do fully flexible autonomous coding is going to require just, like, grabbing things from all different types of frameworks. But for here, just to, like, kind of focus on the, like, writing the back end and connecting it to the front end and handling state management, those were the types of things I wanted to test with this. But yeah, it's, um, you know, it's, it's focused on that and not entirely general with the full, uh, problem of getting everything wired up, for sure. **Alessio** [7:58] Fair enough. So what, what were your results? What, what's interesting? ### Results & Hints **Sujay Jayakar** [8:01] Yeah. So I think, like, you know, the thing that's... The kind of way that we measure performance here is that, you know, we put a time limit on each, you know, for each element of the three-by-three matrix. Um, for the first two tasks, we said thirty minutes, and then sixty minutes for the files task. And then along the way, um, this is, you know, currently still really manual, 'cause I think it's kind of, you know, it's still... We're not exactly sure how to do evals for this, like, thing where the human needs to be in the loop. Uh, so along the way, I am, you know, giving it hints if needed, if it ge- if the, uh, Cursor composer gets stuck and gets into loops. And then I'm also, uh, if it thinks it's done, I'm going and, like, trying the app and starting grading and then reporting any errors that, um, you know, show up in the browser and aren't immediately visible. So I think-- So then this is the kind of quantitative data. Oh, and then the final step, sorry, is that we... Once either the time limit is done or the model stops making progress and it's, uh... then we grade it. So we say for each particular task, there's like, you know, if you have a chat app, like, does posting a message work? Does posting a message work and live update on the page without a page refresh? You know, if you navigate to another channel, does it show that channel's messages? So we have kind of some predetermined rubrics for each one. And yeah, I mean, I think the interesting results are that, you know, like, there's a few reasons for this that we'll dig into. We did notice that, um, I mean, with Convex, it pretty much autonomously just solves the first two tasks. It has a few, uh, round trips on, like, some errors that are only show up when playing with the front end. And then it's able to complete this files task and, and kind of get to close to a full grade, um, within the sixty minutes, where on the other two, the model got pretty lost and ended up not making any progress. And, you know, other than, like, just kind of validating some hypotheses that we had when we were building Convex, I think there's some pretty interesting observations from the video of, like, what made the model work and what didn't, that should be pretty general for AI coding in general. **Alessio** [10:21] And a, a quick question on the hints. How, how do you do hints? I feel like if the model has been working for, like, forty minutes, how do you catch up to where the model is? This is something I'm running more and more into when I use the Cursor agent. It's like, I just let it go, and then it's like, it doesn't work, but, like, I don't even know what it did this whole time. **Sujay Jayakar** [10:38] Totally. **Alessio** [10:39] Like, what's that flow? **Sujay Jayakar** [10:40] Yeah. It's, you know, re-- this is why, you know, part I'll... Just so you know, in terms of transparency, right? Like, why I cut off the first two tasks at thirty minutes is that it just takes a lot of human time to, like... I can't just let it run in the background to... 'Cause I'm trying to administer this test as fairly as possible, right? And so, like, making sure that I'm giving good hints for all three of the back ends and all three of the tasks. And yeah, like, it is just, you know, noticing that the model's getting into a loop. So even though it might think it's making progress, like noticing that it got back to a state that it was at before, stopping it, and then doing some debugging and very quickly, like, seeing, like, this is the thing that, um, it should do differently. So I actually have some details here on, like, hints I gave. This, like, um, you know, for chat and the fast API, um, it was getting really confused about trying to combine some, um, TanStack Query stuff with the SSE and, like, at a certain point, I just said, like, "Hey, can you... You should just directly use useEffect, and here's what it should do." There's, like, another thing I got confused on where I noticed that it was, like, double stripping the data colon prefix for it, and it just couldn't figure it out. It kept on, like, alternating between the two. So yeah, I think this is, like, part of the, you know, for, like, the meta discussion. ### Meta Evals **Sujay Jayakar** [11:55] I think this type of, like, a mostly autonomous coding is just kind of hard to do great evals for. And I'd be curious if, like, there are other more automated approaches that folks have taken. You know, like, one idea would be to have a Much stronger, like think a reasoning model that has private access to the solution and then have that, you know, administer the hints. But I don't... Yeah, there's not a super satisfying answer quite yet. **Swyx** [12:25] Yeah, I'm not aware of one. Uh, yeah, go ahead. **Alessio** [12:27] I was gonna say, do you have a sense of how people can then take these hints and try to work them back into the model, maybe put it in the cursor rules? Any research you've done on that side? **Sujay Jayakar** [12:38] That's a really good question. I think the main angle I was thinking of is like looking at these hints, and there's some kind of like stuff in the observation section on what things the model got stuck on and what that implies for like what makes something easy to generate. Like one example where Convex didn't do super well is, uh, this hint at forty-eight minutes. Convex had like a pretty subtle distinction between null and undefined, like just similar to JavaScript. And we noticed that like because this is a subtle thing that's unexpected, it was... The model got stuck on it and couldn't even figure it out. And so this is something where I had to explicitly tell it like, "Hey, as a Convex expert, this is the thing that you are running into." So, you know, from a platform perspective, like I would if I, you know, if, if we can do it in a way that's backwards compatible, I would love to tweak Convex to just match the model's expectations here. But another approach would be to, uh, tweak the, like the guidelines that we prompt it with for sure. **Alessio** [13:42] Yeah, I found that that works with, um, our spec in Rails. Like the models always wanna use like the shoulda matchers, but you really should not be doing it. And putting in the cursor rules work, but at some point the model should learn that they shouldn't use it. **Sujay Jayakar** [13:55] Yeah. **Swyx** [13:56] Yeah, um, so you know, I think the... This is interesting. Um, the, the-- How do you compare it versus the other benchmarks that you see out there, like the SWE-bench? And then, you know, more recently, now people are starting to compare s-benchmarks with the, like the, like actual m-money-making projects like SWE-Lancer, something that OpenAI recently released. ### Eval Comparisons **Swyx** [14:20] Yeah, just any thoughts on the meta game of benchmarking? **Sujay Jayakar** [14:24] Yeah. I mean, I think like the... This is, you know, I guess like in terms of experimental design, right? Like the SWE-bench category of like starting from real GitHub data and like working towards that does feel like a really nice like end-to-end test. It's almost kind of like in my mind, you know, feels like the same type of test as like an integration test. This test is definitely a lot more targeted, right? It's like saying for this very specific skill of taking a front-end app and taking a specification of it, can a agent then perform in this way that is easy to grade, right? It's like easy to say that given a front-end app and given a back-end implementation, like just, you know, test the functionality that it works because then also the front end is kept standard, right? So I think like, you know, the loss of generality here is, in my mind, it's the trade-off is that versus like, uh, power to grade, right? Like here we can have a-- without having a gold standard of what the answer should be, we still can, uh, evaluate the solutions quite effectively, right? Um, so I think like exploring different points on that trade-off space of like generality versus like focused and easy to grade is something that we're pretty interested in. We have another suite of evals for, um, we just call them Convex evals, that are more coding, like kind of small units of code focused, and they're just testing Convex knowledge. And that's kind of been another thing that we've observed there, is that we keep the test very focused in a way such that they're extremely easy to automatically test and then so we can get pretty precise information on, you know, does the model understand a particular concept in Convex? And in a way that like if we're kind of doing more like of the LLM as judge, like type approach, I'm, you know... And we will need to do some more of that too in the future, but we started with things being pretty focused. **Swyx** [16:27] Awesome. Um, do you see the, for Convex evals, which i sound, sounds like just human eval for Convex, um- **Sujay Jayakar** [16:33] Yeah, exactly **Swyx** [16:34] ... Yeah. **Sujay Jayakar** [16:38] Yeah. And it's, it's kind of interesting with-- I mean, the numbers are going up as-- 'cause like the kind of the purpose of the Convex evals is that we wanted to optimize cursor rules for Convex. And so the kind of feedback loop we have is that we have, uh, you know, we test on a few models and then we add new, uh, you know, like quiz questions, right? We add new tasks over time, and then we optimize the, um, cursor rules equivalence to get performance to continue to increase. I think, uh, one thing that has been interesting is that we haven't seen model improvements necessarily show up in, um, in like our Convex evals improvement. Like for example, we just tried Claude 3.7 and it performs worse than Claude 3.5 on Convex evals with the same prompting. I think this is like a, you know, we still have to debug exactly what's going on. Like a lot of the point for our cursor rule stuff is to fill knowledge gaps, uh, for, you know, the things that the model would hallucinate. So it's possible that 3.7 just hallucinates different things than 3.5 does. But it hasn't just been like a magic bullet, and that's, I think in itself been pretty interesting. Same holds with o3. You know, o3 does do better than 4o. I mean, we use Brain Trust for tracking all this quantitatively, but I can't remember off the top of my head. But it's not like a slam dunk. **Swyx** [18:07] Any, any, uh, last questions? **Alessio** [18:09] Uh, any, um, goals for the next version of this? Any stuff that, that you wanna add? Stuff that you missed? **Sujay Jayakar** [18:16] Yeah. I mean, I think one of the things I'm kind of excited about would be... And this is, you know- I think this would be more research. So I'm gonna-- You know, in terms of how... I'm not sure if it'll work or not, but, you know, like the-- I have some discussion of this, you know, in future work. ### Future Work **Sujay Jayakar** [18:34] I think this, like, theme of like gradability is something I'm pretty interested in for, like, if we give the model the task of, like, filling out an API description, and this could be like, you know, implement the API, implement the API server for like a key value store. There's, like, a lot of room to do automatic verification of that. I think like, you know, I've wired up a version of this, you know, using L, which is the, like, model checker from JePSen. And, you know, can a model write highly correct, highly concur-- like code that executes under very high concurrency even when there's ... You know, like there's some pretty non-trivial requirements. So like if it's writing a key value store and it's using, say, MySQL under the hood, but then it wants to use Redis on the side for caching, can it still be serializable? And that is something then that we can grade automatically, and we can give a score as feedback, right, using tools like JePSen. And I think the other kind of in the same theme of like turning grading into something that can be quantitative and automatic, I'm pretty interested in doing the same for throughput. Like can-- If we have an open loop benchmark as part of our testing harness, if we have the model implement a simple like key value store, its score can be how much load that can take, how many requests per second it can satisfy on some, uh, read/write ratio before congestion collapse happens. So this is kind of like taking some of the ideas from TPC for evaluating databases and then turning that into this form of grading we can use for models. And that's, that's I think one thing I'm pretty excited about. And I think that like care of having very strong correctness testing and trying to push things to be as fast as possible will like kind of elicit a lot of the difficulties of doing systems engineering. ### AI Age **Swyx** [20:32] Oh, unless you guys have to jump, I'll, I'll, I'll go transition us to part two of the, the chat, which is more just my-- Like, uh, I think we're talking about... And then if you, if you don't have anything to speak, Jerry, we can, we can, uh, stop it as well. But, uh, this is more just about like, uh, I guess running Convex in, in the, in the age of AI, right? Like, like I think this is something that you were, you were saying in pre-chat about building developer tooling in general. **Sujay Jayakar** [20:57] Yeah, totally. So we've been at Convex since twenty twenty-one. And, you know, along the way time we've like, especially because we're kind of in this like thing for first principles, like redesign things that have been the same for a while. We've, you know, thought a lot about ergonomics for humans and like what helps people who may not have a lot of backend experience write really effective Convex code. And I think, you know, in the past year, and certainly in the past few months, it's just been very apparent that, you know, people are writing code really differently. And, you know, if someone's a dev tool startup and they're writing something that is like in some set of APIs or systems that are targets for AI coding, they... You know, it's silly not to strongly consider how AI coders perform on one's platform. So that was like the, you know, motivation for kind of investing in making some of this analysis a bit more rigorous. And then also, yeah, like, you know, users kept on telling us, they're like, "Hey, like Convex works really well for this, but I noticed that there's some knowledge gaps, and there's like this is consistent hallucinations." And I think shaping one's platform to be a great coding target is like, you know, something that's just table stakes in twenty twenty-five. **Swyx** [22:07] Have you made any-- You said shaping one's platform to be a great coding target. I think a lot of people have basically tended to take their perspective on developer experience and just go out and just crafted it onto this new thing that people are calling for agent experience and say like, "Oh, yeah, like, you know, because we emphasize developer experience so much, and obviously Convex does, then that also translates well to LLM's coding." But have-- has anything changed for you? Has anything-- Has it influenced your design of Convex in any way? **Sujay Jayakar** [22:36] We haven't shipped any of these ideas yet. You know, some software it's like we are-- We have some ideas for V2s of our API. And I think one of the cool things about now having the eval set kind of written out is that we can do like synthetic user studies, right? It's like easy to just like make a change and then just test it, right? And see like does it match the LLM's hallucination. I mean, I think one thing that's kind of interesting given these, you know, model's inherent str- internal structure is that we notice that a lot of hallucinations come from parts of our API that are like very close to Firebase, but not exactly Firebase. And I think these are things that like a human wouldn't, you know, be as susceptible to. But then like especially when Convex is not that represented in the pre-training set, like it's, you know. It was one of those things that when our, our customers actually gave us that hypothesis, and it just all kind of clicked. So I think that's where, um, you know, walking in with the assumption that good DX implies good AX or whatever we're calling it, like, you know, the, um, uh, that's-- that can be kind of limiting, and I think it's like, you know, writing a, a great eval set is just a lot of pain. You know, it's a pain in the butt. It's a lot of work. But I think it's just kind of hard to substitute that. **Swyx** [23:53] In some ways, I mean, that, that's annoying about the Firebase thing. Uh, so, uh, is your intention to move further away from Firebase or move closer to? **Sujay Jayakar** [24:01] Yeah. Um, I think we're gonna, we're gonna have to try both, right? And we'll see. I think, yeah. **Swyx** [24:07] Okay. Uh, like one tricky thing I feel like is, is almost like AI ca-calcifies your API. The cost of switching APIs is so high 'cause it's, you don't know what the hell is in those, the, the dataset. You're, you're not gonna reach out to every model lab and like ask them to update the dataset. It's almost like you can't change it. Like, whatever pre-existed pre-AI was like, is you're locked into it, you know? I don't know. **Sujay Jayakar** [24:29] Yeah. I mean, yeah, it is, it is a little depressing, right? I know people talk about this like, kind of, you know, in terms of tr- even in terms of, like, the knowledge cutoffs are really far in the past, right? Yeah. I mean, I think- **Swyx** [24:40] I don't think that's a real concern now because they're updating, like, every few months. The, but, but it's still the, the bulk of the data is still old. **Sujay Jayakar** [24:49] Yep. **Swyx** [24:49] Yeah. **Sujay Jayakar** [24:49] And I do ... I mean, I was pretty ... Depressed is too strong, but I was pretty pessimistic about this before we did the kind of evals work. It has been pretty cool to see, like, how steerable the models are in that, you know, with, like, some ... And we're ... We have to do a little bit more work on, like, token efficiency. Like, how can we get our evals to pass in a high rate with as few, like, tokens in the, uh, critical roles as possible? Um, because we've noticed, like, kind of really long ... As we've, like, if we've made it really long, um, we've noticed model performance getting worse on, like, very complex tasks. I think, and this is, you know, above my pay grade on how it knows to attend to particular parts or not, but that's, like, I think one kind of interesting trade o- like, optimization problem. Yeah. And, you know, I think it's, you know, from the aesthetics of API design, this is like, you know, just typicating, right? Maybe this will ... If we are ... Things that are non-standard and things that don't match whatever in some ... And I think that's maybe not a ... So- **Swyx** [25:55] Yeah. Awesome. I, I have a slightly different a- a question if you, if you don't mind, unless, unless you have anything else to, to pitch in on this design topic. I think the last ... Like, one of the other big impacts that Convex had in the AI engineering world that I saw, that hit my consciousness at least, was AI Town, which, uh, which the a16z guys, you know, super value-add investors, you know, luck with you guys. And I think more generally, there has been a lack of follow-through on that kind of agent work that I've seen. Like, very cool demo. People loved seeing it. Like, then they're now like, "Okay. Yeah. Like, but how do I use this?" ### Agents **Swyx** [26:30] So basically, I think my question is, like, how does Convex see, like, the backend, like being the backend of agents? Do you have everything you need? Is it ... Are you just waiting for the market to catch up? Or like, what ... where are we in, in, in that cycle? **Sujay Jayakar** [26:43] Yeah. I think it's like a ... It's a very interesting, like, strategic question, right? In that as a backend platform, um, you know, I think when we were really starting to dig into, like, you know, this particular round of AI investment from us, like, I think it felt like that there's kind of two paths in the decision tree. There was, like, people using AI coding to write backends that are, like, kind of the same as they were before. There's like, you know ... There's, like, the hypothesis there being that there's all this pent-up demand for apps that maybe aren't so different than apps a year ago or two years ago. And that, like, a much larger set of people can do it at a much smaller cost. And then there's the other side of, like, will backend infrastructure look different with all the changes in AI? So two years ago, that was, like, vector databases, right? Which, like, you know, were used before, but now are just so ubiquitous in a way that weren't. And there's like ... I wish I remembered off the top of my head. There was a really good blog post on, um, AI agent architectures around, like, you know, using temporal, right? And, like, using, you know, like, the usual techniques for making, like, a unreliable service and adding retries and exponential backoffs and all of that. I think, like, yeah, we're very interested ... I mean, we have, like, a workflow system on Convex, like, for durable execution, and I think we're still, like, investing more on that first category of, like, saying, like, how do we take the usual shape of apps on Convex and just make it very easy to AI generate and then kind of getting that in a really good place before investing more on these, like, the new shape of apps. But yeah, like, I imagine, like, that those types of patterns of, like, durable execution, of, like, being able to very ... have very close proximity to all the data sources and being able to query your data points database and, you know, in vector stores and all that. But it, you know, it's something we are excited to invest in. Haven't, haven't gotten around to it yet. **Swyx** [28:38] Yeah. I would say a lot of people are still figuring out the patterns. Like, you know, as, as, as, uh, hyped as the Anthropic blog post is on agent architectures, actually, most of it is, like, explaining what is not an agent. **Sujay Jayakar** [28:49] Yeah. **Swyx** [28:51] So I think we still need to map it out as well. Cool. Any, any other parting thoughts on, uh, just Convex's general AI strategy? ### Takeaways **Sujay Jayakar** [28:58] Yeah. I mean, I think the, um, you know, this is, like, kind of the takeaways from the Fullstack-Bench thing, but I th- you know, I think they're pretty broadly applicable. You know, I think, like, just, like, for human coders, I think, you know, this is a place where DX and AX, you know, line up a bit. It's like, you know, having just really tight feedback loops and having strong guardrails. Like, in Convex that's, like, end-to-end type safety, but that's, like, you know, could take its in many, many different forms, right? Like, making it so code is very cheap and easy to grade. Um, you know, making it so that, um ... You know, one other thing we observed is that just having more represented in code and, like, this is kind of, like, our conversation about stuff being in the pre-training set or not. Like, having things just be standard and procedural code and not doing kind of weird, crazy stuff- **Swyx** [29:44] Yes. Another strong- **Sujay Jayakar** [29:45] ... is just like- **Swyx** [29:46] Yeah. **Sujay Jayakar** [29:47] Totally. And we even noticed, like, you know, reasoning kind of being one also big separation. Like, reasoning about procedural code when debugging is, like, something we noticed models doing pretty well. The particular example was, like, RLS rules and Supabase, where debugging, like, an infinite loop, uh, for infinite recursion for the RLS rules was something that the models just really struggled with in a way that we didn't see for procedural code. So that's another kind of interesting thing. But that shows up in a lot of domains, right? Um, and I think, yeah, the final kind of observation from that article was just that, like, strong abstractions are good, you know? Like, models, you know, when given the full flexibility of FastAPI and doing SSE and wiring everything from scratch, it just couldn't help but mess it up. So having, like, kind of strong lib- like, choosing good libraries that have strong abstractions is, like, kind of always a good bet with models and that's, uh, you know, hopefully ag- also something that's pretty general. **Swyx** [30:42] Yeah. I think that's a good message for tool builders out there anyway. So, like, uh, yeah. Awesome. Um, well, thank you so much for sharing and jumping on and, um, yeah, congrats. I, I look forward to seeing more of Convex's, you know, progress over the, over the, the coming years. **Sujay Jayakar** [30:55] All right. Thanks for having me. **Swyx** [30:56] Thank you. All right. I will stop. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.