The Great Evals Debate — Ankur Goyal & Malte Ubl

Hi, welcome to the Linked Space lightning pod. We are here with for a very special episode because this is not really about a company or some launch that someone's doing. This is actually just a debate that I accidentally set off about coding agents mostly not having evals.

Or they do have evals. It's mostly it's not really like standardized or and entirely it's a lot of it's vibe-coded. And I think a lot of it is based on the fact of this clip that I had from Boris of Claude Code saying that they basically just vibe it.

And that's a 500 probably higher than that $500M business that doesn't have evals. So I wanted to invite Ankur and Malte from Braintrust and Vercel to just talk about like the state of evals here and like the evals debate.

You guys have your own perspective. Ankur, you got a bit in a bit of like a blogging back and forth on evals. What are you guys feeling? Like what's maybe a position statement from each of you?

Yeah, I mean, I can start a little bit. Like I think the like one maybe important background on me is also that I worked on Google Search just before leaving for Vercel. It's almost four years ago, but like I do vaguely remember it.

And obviously that was also before ChatGPT moment and before kind of engineering becoming a mainstream thing. And so I left a world in which I was doing evals every single day and kind of thinking that that would be it,right?

Like I wasn't really seeing that in my future as someone joining an infrastructure company. And so it is both kind of fun to see it coming back and I think it does give me a little bit of extra experience for what like how it helps and how it doesn't help.

One thing I tell people for sure is that like you have to put in the work,right? And even vibe checking like as the eval like is important. Like if you are a product manager in Google Search, you're probably doing like 500,000 searches a day.

And if you are but then you want to do real evals because if you want to do 50,000 searches, you're not going to do that yourself. And I think there's something very similar here where essentially you just see like, okay, I want to know if I'm doing well and how fast can I find out.

And like, yeah, the vibe check can tell you really fast, but if you want to know tomorrow, then the only way to do it is to have someone do the 50,000 searches or the 50,000 coding exercises,right? And if you want to know in, let's say, in three weeks, you can run an A/B test.

And if you don't care, then you know, do neither of those things. But it is like I think like the way I think about evals is essentially like it's the thing that can tell me tomorrow whether my change is good.

And I can operate without that knowledge, but it's but it's super, super helpful.

Yeah, I have very little to add to that. I strongly agree. I think like fundamentally, if you're building an AI product, you're dealing with this like non-deterministic magic. And you don't you can't like type whether it's a model or prompt, whatever, you can't really type something and know what's going to happen.

So you need a feedback loop. And I think that fundamentally, evals are one important factor in helping you build a feedback loop. And offline evals, which I think is what a lot of the debate was about, are both the most challenging to build feedback loop and also the most efficient once built.

And then I think A/B tests are a little bit less challenging to build and a little bit less efficient. And then like pure vibes are basically like zero effort to build, but then also the least efficient. And you have to do a little bit of all of these things.

And I think what I've seen both firsthand from building Loop, which is our own agent in Braintrust, but much more secondhand from working with Malte and a bunch of really great companies building AI products, is that the best people are extremely deliberate about the investment in each of the feedback loops.

And they're constantly questioning whether they've put enough energy into each of the feedback loops and making them efficient. And then I think coding is it's like such an interesting topic for a few reasons. The first is it is the use case that has the most product market fit in AI, I think, other than ChatGPT.

And so it is just an important and interesting use case. The second thing is that to a large extent, it's very verifiable. And so it's like it's almost insane that it's not evaled more. Or at least the fact that it's not evaled more is super interesting because unlike something like, let's say, creative writing or summarization, et cetera, where there's some nuance, there's a lot of objective stuff that you can evaluate.

Now, if you talk to the people that are really deep into it, they'll disagree and say that refactoring a code base is something that's very hard to evaluate. Or the visual quality of an artifact that you produce on Visi is very hard to evaluate.

And I think therein lies the subtlety. But then, you know, the third point, which I've watched Boris's podcast, which I thought was likeultra,ultra insightful. And I think the thing that's very clear from what he said is that based on how people are using Claude Code internally and the enthusiasm in the community, he feels like they have a pretty good feedback loop.

And I think that's awesome. Like it doesn't really matter whether they're running a script that has a for loop that iterates through a bunch of test cases and, you know, whatever, that's their feedback loop. What matters is that they have a good feedback loop.

And I think that's ultimately what you need to do. And however you get there is how you get there. And if you're happy with the feedback loop that you have, that's awesome. But I do think it's important for everyone to think about do they have a good feedback loop?

And if not, how do they make it more efficient?

Yeah, maybe one thing to add on what Ankur was saying. Just yesterday, I think we saw both Cognition, congrats, swyx, and Ankur's ownership RL fine-tunes of unnamed open source models. And the reason why you can do that is because you can implement those RL loops based on what is effectively an eval that you built,right?

That you run on every iteration. So that's the reason why this coding stuff works so well is because you can write these evals,right? And I have this like hot take, and this might be like not fair to Boris, but I think he comes from a from a particular position of extreme unusual privilege, which is that he works at an AI lab where like people in the office next door are like writing the evals and are like training the model like every day in exactly that way.

And so it might be like I'm sure there's some pretty deep hand-in-hand work in making the agent work well and be and the model be particularly well trained to what the agent needs and so forth. Like there's going to be feedback loop like that.

And I think we have to consider that when we make recommendations for the world out there, which is different,right? Like where you don't have this privilege and where you have to kind of make do with what you got from the either big model labs or the open source models, et cetera.

Yeah, I don't know if there was like a particular point to respond on that. I will say like I think all the agent labs, if we can call it that, you know, Vercel has your own benchmarks, Cognition has their own, Vercel has their own, and typically everyone trusts their own benchmarks much more than the public ones.

But then they're not comparable because they're not open source, because probably because they're IP or because they use proprietary code.

Vercel's is open source. Or at least they have a big open source one.

Well, Next.js the Next.js one.

Even now, like yeah, we are open source Next.js. We intend to do it for the other tech that we're building. And that's probably because we come from a slightly different permission. Like our goal is just that all the AI models and agents are just incredibly good at the tech we've put out,right?

So we're incentivized.

Yeah, there's some self-interest in doing it.

Yeah, 100%.

Yeah, yeah, so, so, you know, and then, and then, you know, Cognition, like a lot, it's a lot of banking customers that like would not appreciate some of the, those evals work being publicized. But I think like, yeah, anyway, so the point being like actually we spent a lot of effort on public benchmarks and in fact, like the team like really burned themselves out over doing this.

And in the end, like people look at a chart and then they go like, allright, like it's, it's, and then let's check the vibes,right? So.

Well, yeah, I mean, I think those evals are, they're not a public benchmarks that people publish are not in service of creating a feedback loop, which is not to say that they're stupid or anything like that. It's just that the value proposition of publishing an eval is completelyorthogonal to the value proposition of building evals in service of building a good product.

I think the purpose of publishing benchmarks is marketing. And it's, it's good marketing. Like I come from database land where TPCC and TPCDS, all this, those are standard benchmarks. And then people like ClickHouse make ClickBench or whatever, they make their own benchmarks.

It's, it's very, it's a very sort of like mostly objective way of communicating the value proposition and trade-offs of your technology. But I think it's totally independent from the practice of using evals as a feedback loop mechanism for iterating faster.

Yeah, okay, awesome. I want to go back to a little bit something that Ankur, you touched on. Let's just discuss call these like the primary dimensions of evals. You talk about effort, you talk about what is that effectiveness or efficiency.

I think there's one thing that I don't know how to capture, which I'm curious how you guys think about it, is open-endedness. This is not like, this sounds like a made-up term, but it's a real team at Google that is actually exploring open-endedness.

And I think Malte, to make this relevant to your background, search is open-ended, but it is not as open-ended as code gen. And like, I think the reason people lean on vibes more and spot checks more than traditional evals or offline evals is because they actually cannot predict what was.

It's very easy to kill client things that you've already written evals for,right? It's always the stuff that you don't know that you don't know that is going to catch you off guard. Like some competitor is going to come in and eat your lunch or whatever,right?

So to me, that is the, I think the debate of like why people are moving from offline to online or online to vibe checks is because it costs a lot of effort to do offline and it probably lasts maybe six months if you're lucky.

Well, I think, I think you're conflating two different things. One thing is sitting in a room and enumerating a set of scenarios or tests that you think represent a workload. And then the other thing is choosing to run those tests outside of production.

And both of those things are activities that can happen offline. But I don't think anyone who's legit, who's doing offline evals is doing the former anymore. Like, in fact, when we started Braintrust and we talked to customers, we're sort of very excited about Braintrust helping them create golden datasets.

And now I actually think that term is kind of a dirty term. People don't really want to create golden datasets. It's, I think it's, it's, it's often a wasted effort to the point that you're making. I think the best teams view offline evals as a mechanism of reconciling what they see in production with real users who are using the product with tests and iteration that they can do offline.

Like the best teams that we work with on a daily or even more frequently sometimes basis are discovering use cases from logs online and then pulling them into their environments and then playing with them. And I think the difference between doing offline evals and online evals or offline evals and just pure vibe checks is I think the pure vibe check or pure online version of this is you observe some behavior in your logs.

You're like, oh yeah, let me like play with it. Maybe you tweak the prompt a little bit and then you deploy it. And then to Malte's point, like either statistically unsignificant, insignificantly now or statistically significantly three weeks later, you observe the impact of that change.

And then I think the offline eval way of thinking about it is you discover some interesting behavior from prod and then you have one or two clicks that allow you to capture that behavior and repro it offline. Iterate.

And as you iterate, you not only test that, but you also test a bunch of other scenarios from other users that may also be recent. And you gain a little bit more confidence about the impact of the change.

And then maybe you ship it still like today or something. Like frequently when I'm using V0, I will complain to, there's an engineer at Vercel named Max who's like an incredible engineer. And when I complain to him, like I know that two clicks later, he's taken my complaint and is playing with it, not in prod, but on his laptop.

And I think that's, that is what offline evals are really about. It's not about manufacturing a golden dataset anymore.

Anything to add?

Yeah, I think, I think there's definitely the, the, you know, workflow velocity. There's the, like the understanding of what the eval tells you,right? I think that the common quip that if the vibes and the data disagree, the vibes are probablyright is true,right?

So you have to like be honest with yourself, like do they, do they agree and kind of iterate on them over time? The other thing that like, I don't know, it's just my weird head who thinks about it in these terms, but like to me, often the, I like to say the first derivative controls for methodology.

Like, like if on a, you have your number,right? And it's more important that you know that it's good if it goes up than that you understand what one or two on it means. That's very difficult to understand sometimes.

And so I need to like, if you, if you, if you invested in your evals, if the opposite is true,right? Like, so it's so nice to be able to be aggressive in the changes you make, knowing that your encoding of what you thought yesterday was a good idea for your product hasn't regressed,right?

Even though the world has changed, like that is just like very traditional kind of having testing in software engineering that allows you to move faster because you have higher confidence. And I think this is like, you just have to understand the trade-offs between that and potential extra work, obviously.

One of the things that I've noticed along these lines, I like, I hate writing unit tests personally. I write them, actually Claude writes a lot of them now, but I've always hated them because they're so brittle. Like I change the design of my system and then the unit test slows me down rather than enables me to verify whether my system still works.

I've always been like a hardcore integration test person. And I think the same or maybe an analogous set of design principles and trade-offs apply to thinking about evals. Like if you construct evals in a way that represent the true north star of the problem that you're solving, then they tend not to break as you change the underlying system.

Whereas if you hard code them to a narrow subset of like an implementation detail of your problem, they can be very, very brittle. And so I also, I think you have to, if evals are not working very well for you, obviously some of that is like the cost of evals and maybe the tools that you're using or the framework or whatever, but some of it, you kind of have to zoom out and say like, okay, how do I, how do I reframe how I'm thinking about what I'm evaluating so that this can be productive for me?

Or if you choose not to, maybe your competitor will,right? So maybe use that as a guiding principle.

Yeah, I think evals as a competitive advantage is underrated. I don't know if that's a, that's a spicy to say. Okay, so, and I wanted to maybe ask Malte, like are there any best practices that you guys have have developed on evals, maybe around Braintrust or like principles that you, that you think that maybe Ankur hasn't really covered because he is from his point of view, you're from sort of the user point of view.

Just, I just want to leave it open-ended as to stuff that you've seen.

Yeah, I think from a just pure workflow point of view, and Ankur kind of already like highlighted some of this, but I can maybe go a little bit more concrete,right? Like the, the actual workflow is that with this tool like Braintrust,right?

You essentially are logging and you're collecting facts about your production system. And what's really interesting is then how do I turn the thing that I know people did in production into, into, into something that I can validate in my, my code base?

And so what we, what we do there is that we try to, on some relatively regular level, kind of upgrade failure modes that we observe in production into something that where we then in the future can measure how, how well we do based on, on those previous failure modes,right?

And so you're in this kind of constant loop that is actually, in a way, very vibes driven,right? Because there's the ground truth of like what actually matters, which you can obviously sometimes you envision what it is, but like in practice, what actually happens is that you are driven by, by the failures and the successes of your production users.

And then you'll, you'll lift that into something that's expressible in, in, on the code side. I think the other thing that maybe is worth mentioning is that because we're talking about coding in particular, it is in fact possible to do the essentially like evals like, does this program compile?

Does this program start up? Does this program start up and render the first page without client-side error,right? You can, you can create some of these very trivial, like quote unquote trivial evals, but like what we found is that they're actually, they're extremely valuable,right?

That's actually where, so, so the, like what we then, what we actually literally did is we turned those, those into our RL pipeline,right? And so we, we essentially trained a model to, to fix the errors that the eval was previously detecting.

And that, that is extremely effective. And it's while, you know, basically what we do is we have this like composite model architecture where you run the frontier model and then we run the fine-tuned model after to fix its errors.

That's, doesn't perform better than an agentic loop, but it's always of magnitude faster,right? Because all that, like, you know, like the agent might all the time find out, oh yeah, I made a mistake. And then it totally knows how to fix it,right?

That's not the problem, but it's fixing it using a model that has somewhere between 30 and now maybe the fastest one, 900 tokens per second, which is still snail pace for trivial errors that these models make all the time, like const A equals one, next line A equals two, which doesn't compile, but it's very, very easy to fix.

Okay. So, so you're saying you don't do an agentic loop, you only do frontier draft and, and fine-tuned fix. Is that.

That's not what I'm, that's not what I'm saying. I'm saying we do avoid having to do the agentic loop in more cases so that the total time goes down because you use the agentic loop for fixing difficult things and not for fixing trivial things.

So I'll briefly mention that in the swyx rep work, part of the RL objective was, was time to complete. And so you can just actually constrain the number of turns it takes if you, if you have the agentic driver model, not be incentivized to take forever because you want to prioritize human interaction.

Yeah, I think there's another point. I'll inject a little bit of spice into this discussion that Malte made, which is using evals as a way to get product insight. And I, and I think this may be where evals caught the attention of Lenny.

I don't know for sure, but we've, we've, we've seen like a massive surge of product managers, product managers and designers getting interested in participating in the eval process among our customers. And I think the reason is that in many ways, coming to the table of product building with representative examples and criteria that articulate what good versus bad is for a use case is just a more precise and usable form of product management than writing a spec.

And so what we've seen from some of the best teams is like actually leaning into getting product managers very, very involved in the eval process. And if a product manager who has a very good instinct about what the vibes should be wants to communicate that to an engineer who is maybe less familiar with the vibes for a particular use case and in coding, this doesn't make any sense because the engineer has the best vibes.

But if you think about use cases like finance, for example, or healthcare, where AI is providing like a massive amount of impact, the product manager has a way better, usually has a way better understanding of what the user actually wants.

One thing they could do in like 2015 is write a spec that describes like, hey, you know, the user wants this and therefore blah, blah, blah, blah, blah. Now what they can do is construct evals that help illustrate that point or even construct scoring functions or LLM as a judge, as they're called, that are, they're kind of written like specs.

They have the same structure and they're kind of looking for certain criteria, but they just end up getting utilized differently. And I actually think there's a lot, I think, I think there's a lot there. And it's, it's, it's also somewhat coincidentally interesting that that particular phenomenon is mostly irrelevant to the coding domain, but is like highly relevant to all the other domains in which someone might use AI.

I think that'sright. And it's actually, I think the magic keyword that everyone's saying is rubrics these days for LLM-as-judge.

Really?

Is it? I don't know. That's what we use.

Okay.

So like, yeah, we, we, we built this thing called the junior dev eval. We're, we're upgrading the senior dev eval, working with like McCormick and like other a lot of these like high-end annotator companies to, to, to do all these.

And coming back to the point on product management, yes, I think it'sright. I think it's both the managers love, like, you know, would love to have more of a say in engineering,right? And they write this like 50-page PRD, no one reads it, or they, they have to go engage in this back and forth and say like, well, you didn't exactly get what I meant.

But here they can actually come out with a number and if the number went up, they're happy, everyone's happy,right? And they get to treat the products as a black box. So it's a, I don't even think it's just the number though.

I think it, it, if you, if you just think about vibes, like forget about the term eval and you're working in the finance domain and like I don't know much about finance, but if I, like let's say I build something that, I don't know, processes an invoice or something and swyx, maybe you're like a, you know, finance wizard and you're like, hey, Ankur, the vibes are off.

I'm like, what do you, what do you mean, dude? I have no, I have no idea what you mean. How are you going to communicate that to me?

Domain expertise.

Exactly. Yeah. So, so you could write a spec or you could just give me some examples. And I think the beauty of AI is that give me some examples is actually a very powerful way of communicating product intuition.

And so again, I think like if you just forget about the term evals or, or, and you don't, like I think vibes are actually a form of evals. They're just like a very, very expensive scoring function that is extraordinarily accurate in a lot of use cases, maybe not for compiling code, but certainly for like aesthetic quality, for example.

So if you, if you just generalize and you think of evals as including all these things, I think it's a great way of communicating product intuition.

Yeah, I, I agree with that. I would say like the, the, so what that's one of the responses to the great evals debate, which was, well, vibes are also part of evals. Then I'm like, well, then you're, then just everything is evals,right?

Like you're always doing evals that, you know, because you are an evals person, you obviously see the whole world in, in, in your point of view. But I think we're trying to, to like push towards people towards.

We can stop using the term evals and just call it feedback loop. Because I think that's, I think that's not, there's no feedback loop vendor or something. Like that's not the perspective I'm trying to communicate here.

No, no, no, no, I'm not accusing you of that. Yeah, I think feedback loop isright. I think that's why there's also all these like RL startups that are coming out. We're working with Applied Compute and like there's like 20 different ones.

I just had, I just had a call the other day with another one and it's like, feels like it should be vibe-codable, all these environments. I don't know why they're so hard, but I wonder if you guys have, have interacted with the RL environment startup groups and like, is that a part of the evals movement that is, is happening or is that separate?

Yeah, I think we obviously are familiar and friendly with a bunch of these companies and keep an eye on it. And I think that I like to think of evals as like really like three components, like, you know, data or examples and then something that you're trying to evaluate, like an agent or whatever, and scores, which help you compute whether, you know, your thing works or not.

And I think RL environments, they're, they're like definitely a powerful form of the third thing and not necessarily a powerful form of the first thing, data. And I think that what we have really tried to solve for at Braintrust is helping people solve the data problem.

Like we, we, we, we are really, really about helping you capture data from prod and figure out what's worth looking at and then pull that offline. And I think you can use that to help inform how you use an RL environment down, down.

I would say practically speaking, outside of companies like yourself, swyx, that are at the very frontier and doing like training your own models and doing like really crazy stuff, we haven't seen RL trickle into the average company that's trying to figure out how to just use a few prompts and maybe build like a basic agentic loop and get it working.

I'd be quite excited for that to happen because I think that the RL paradigm is more programmable than manual. And I think that's quite cool. But I think what I've heard and experienced so far is that you have to have a pretty specific set of expertise, which may be what you're alluding to with the vibe coding thing.

You need to have like a pretty specific expertise to design the RL environment in a way that's not vulnerable to reward hacking. And I think that either you'll end up with some fixed number of very well-engineered RL environments, or you need to somehow employ this expertise to apply it to the N+1 problem.

And I don't, I don't see any evidence yet that the average company that's trying to build an AI application will be able to recruit that expertise. It's kind of an open question to me how, how far it will propagate outside of the companies like yourself who are at the very frontier.

I'll just briefly interject in case Malte has something to follow on. Actually, I see a lot of it in computer use in terms of the RL environment. And I think you're coming from a very human-centric point of view of data.

Like humans are the source, humans and domain experts are the source of truth and they're effectively distilling their knowledge into the environment using as the dataset. I think that the, what I see the potential RL environments is, is that these are a lot of encoded opinions in a realistic environment, which would not be conveyable at human scale.

So you're trying to scale up individual humans by just having them construct realistic environments that embed a lot of like normal assumptions that an RL agent can go in there and just work on. Yeah. So I think the difference is kind of the trying to decouple from the humans because humans are expensive and slow into an environment which we can parallelize control and play back and forward.

Yeah, that would be my twist.

One comment I wanted to kind of dig in a little bit and it's a little bit not directly related to what we were discussing, but I think there's a, there's an interesting link is that swyx, I think you were saying earlier in the conversation that there is basically, obviously people have proprietary evals.

People publish some notion of leaderboards publicly, which are marketing and they're like, you know, people find that interesting from a rarely, but they don't have a, you know, particular purpose. And I think there is a, like the next year's eval that we release that you already mentioned and which we, you know, we're planning to do more in that space, I think points to an interesting future where the question is not so much like, you know, is there going to be an RL process somewhere or is there going to be a better model?

But like, can there be kind of an inversion of control of like who's writing the evals versus who is building the agent and who's building the model? Likeright now it's always the same entity. And I could see in the future that this actually changes,right?

Like if, if, if I'm a business and I want an agent to be incredibly good at like automating my business because my business model isn't that someone clicks around my UI, my business model is that something else happens on the backend, then I'm incredibly incentivized to say, hey, you know, model lab, hey, agent lab, how can we work together so that you have an easy time getting really good at the tasks that are relevant for my business?

And so I think that that is a, there's a new way of, of, of thinking about whose responsibility is making the evals. It might not be the same entity that then takes those into production in terms of making a better model or a better agent.

Yeah. I think that's why like companies like Artificial Analysis, Ella Marina, these are, these are like the independent, like the new gardeners of like independent reading agencies for various companies. Yeah, that makes sense.

Yeah, but I think that's a, that's a bit, it's a bit of a different take than from what I, what I meant because they are independent,right?

Yes.

But I don't, I don't, I exactly don't mean that. I'm like, I'm not independent. Like because I'm, I'm Vercel and I publish an eval that I want OpenAI and Anthropic to use to make sure when they ship the next model, that they're better at the stuff that I care about.

And that's, I think, somewhat different from, now obviously like you could have a middle person like Ella Marina in there to, to facilitate that process, but then my main point, my main point was like the, the, the creator of the eval is an entity that has, that wants to achieve a goal.

It's just that the goal isn't that their own models get better, it's the goal that someone else's model get better.

Yeah, this, this idea has been around forever. Like OpenAI, the evals repo and the sort of expressed, not, they weren't like shy about it. The expressed intent of the repo was for people to contribute evals that they could use to make their models better at.

And, you know, the trade-off is, hey, we're OpenAI, we'd appreciate having more evals, excuse me, to, to test with. And then in exchange, if you contribute evals that you care about, you'd get, you know, perform better on those evals with future versions of our model.

The interesting thing that, that we saw, and we've talked to all the labs about this because we work with basically all the commercial customers that they would want to optimize for. The interesting thing is that most customers, or actually I'd say a stronger statement, like all customers are quite afraid and reticent to just hand over the data that they used to do evals on to, to labs.

I wouldn't say I fully understand it, that position myself. And I think that in some utopian sort of construct, it, it could be very valuable for everyone to exchange data this way, but it just hasn't happened. And maybe RL environments are kind of a cool solution to that.

I definitely see, I'm like, maybe to illustrate Malte's point, let's say that I'm starting like a healthcare startup and I'm, I'm really, you know, my job is to figure out like based on your, like a picture of your tongue, what diseases you might have or something like that.

And, you know, GPT-5 is like okay at this, but I want it to be like really great at this problem. How do I do that? Well, I could like train my own model. That's going to be really hard.

I could get a bunch of evals together, like take a bunch of my friends who have diseases, you know, pictures of their tongues and then upload them to OpenAI's evals repo. Or I could create an RL environment that I publish on some marketplace or something that people can use to actually test.

Maybe I have a way of generating photos of tongues that have illustrate various diseases. Maybe I could somehow publish that and then that's a very efficient way for people to get better at that. And this is kind of a white space thingright now.

Like if I'm starting this tongue disease business, like over the next several, you know, hours, days, weeks, months that this mechanism should exist. Like it's from a capitalist standpoint, it like makes sense for this mechanism to exist. That's, that's kind of what you're saying,right, Malte?

Yeah, exactly. I think it's always important to track the incentives,right? Like I don't do that if I'm the medical agent startup because then it's, then it's for me privately. But if I am something that I want the world to use through AI, then I'm incentivized to make the AI extremely good at this.

So, and that always happens when there's a bit of a detachment in the business model from the, from that task itself.

Yeah, I guess Next.js is kind of like a good subtle example of that,right? Like you don't, you may not want your competitors to be able to generate as aesthetically pleasing designs as V0, but you certainly want every model to be able to generate Next.js with extreme precision.

Exactly. Like that's our explicit goal. And we also want to be able to ship new versions of the framework and then have quick updates in the base models to be able to like, to essentially learn the new skills.

Have you tried building an RL environment for Next.js?

Well, we just shipped the evals last week, but I think it's a, it's a very interesting idea.

Well, I think like RL environment for Next.js is just npm install Next.js.

Is it? No, no, is, is it or can you, can you actually design something that is better? I don't, I don't know. I mean, I think it's, I'm not enough of an expert to, to, to know the answer to that question, but I'd imagine that you can probably apply some creativity and, and create something that's better.

You know what? The Next.js templates page is the set of RL environments because Next.js itself is too horizontal to be meaningful, but actually focusing on the top like five, ten use cases of Next.js, yeah, I would say so.

Okay. I don't want to, I'm mindful of your time. You guys are both very busy, important people and like, but I also, I feel like we come to a good spot in terms of like, you know, if you want to control your AI engineers, you want to control your labs, you should build evals so that it's like a tool of communication and also.

Control might be a little bit of an overstatement.

I just, I'm being spicy. I'm being spicy. No, you know, like he who writes the evals, you know, rules the world, whatever,right? Like something, something of that nature. But no, I think being a very good place to end, I think very educational for people who are like, haven't really thought deeply about this.

Obviously, you know, people can reach out to both of you who are very online and very generous with your time. And yeah, I mean, I think this, this really is a very deep debate. It's for people who are serious about building good AI systems.

And I mean, I fully agree. There's more to have with, with this debate about like how do you like push the state of evals? It's something I'm interested in exploring at Cognition. I don't have the time yet or the material.

I'm just saying like, you know, if we want to define what's next for coding agents, we have to build the evals and then we have to kill climb the evals. And that's like basically what my job is. So I'm excited for that.

Anything to plug for both of you? Anything coming up or any calls to action?

Well, we're shipping crazy stuff every week. So I don't, I'm not sure I have anything specific to plug, but come visit braintrust.dev and join, join the party.

My plug is to listen to the other podcast with you that I recorded this week, which does plug everything we're doing.

While we're all here, I will plug the fact that you can now, with one click, send AI traces from a Vercel application to Braintrust. We are very proud to be part of the Vercel marketplace. It's, it's, it's super exciting.

Yeah, how did that sleep on Mike? These are literally the best two guys to talk, talk about that on. Okay, go test it out. We'll put some links in the initial notes if you can find it. Thank you guys.