[Paper Club] SWE-Bench [OpenAI Verified/Multimodal] + MLE-Bench with Jesse Hu

All right. How, how is this? Is it, is it giving the infinite thing?

Uh, I'm only seeing your SWE-Bench screen.

Uh, yeah. Is, is it, uh...

Can you, can you see me change windows?

Yes.

Okay, cool.

Yes.

All, all right. So, um, I'm sure a lot of folks have, uh, either seen or tried to, uh, get through SWE-Bench or even compete on SWE-Bench. Um, it's basically trying to solve the problem of how do we actually evaluate real coding tasks.

So there's been these benchmarks that are out there before, um, such as HumanEval, and the problem with them is that they basically, um... Well, in, in a few words, they're, they're too easy now, that, um, a lot of models can just spit out answers if it's just a, a direct, um, problem, um, if it only requires you to edit one file, uh, if it's just a riddle.

Um, so the problem is: How do you actually evaluate real coding tasks where you might need to understand a code base, crawl through it, understand how files play between each other, uh, figure out if you need to edit multiple files, make those edits, run tests and things like that.

Um, and this plot down here is showing, you know, I, I think through multiple techniques and through the, the most recent models, you can actually basically cap out on the performance on HumanEval. So there's definitely need for benchmarks that are a lot harder and are a lot closer to, I guess, what we human engineers do on a day-to-day basis, uh, in the hope that, um, you know, you can make it something you can climb towards and evaluate, uh, I guess, replacing, uh, software engineering.

Um, so let me switch over to the paper. Um, and I'll skip a bunch of it because, um, some of it is no longer relevant. But basically, the idea is: How do we get representative coding tasks basically for free?

Um, and the way they do that is they pull a bunch of these GitHub issues and PRs from 12 Python repos. Uh, and the repos like Django, uh, I can't remember the rest of them. A bunch of numeric computing, uh, repos, uh, things that are really well-maintained, have a big community, have a lot of unit tests.

Um, and, um, I'll scroll down a bit here. So what they do is you want to scrape those PRs-

Mm. Mm

... um, which means, uh, you also have the commit at which that, uh, issue and PR was initiated at. And I'll go into this a l- a little bit later because it's a little bit tricky to make sure that you're not cheating in time.

Um, then they do some filtering to make sure that there's actually tests that are modified in that PR, um, because what that gets you is that if you can get tests that were added in the PR, you get free tests to see if you can pass, uh, the issue as before.

Mm.

Um, so basically free tests. Um,

yeah. I mean, a- and I, I guess, it, it seems like it's a straightforward task. Let me find PRs that have tests on them such that I can reuse them to determine if the agent can pass the same test.

But it's really hard to mine for these, and it's really hard to clean these up. So I, I think somewhere in here they actually do a lot of manual work, uh, from the authors themselves, make sure that these are actually, um, legitimate, uh, legitimate PRs.

And then we'll go into Verified, um, to see, like, there's actually more work to be done in filtering these down.

Um,

so here's the breakdown of all the repos that we use. So it's Flask, Django, Matplotlib, PyLint, PyTest, Requests, Scikit-learn, Seaborn, Sphinx, NumPy, Xarray, and Astropy. And the positives here-- Oh, Eugene, do you have a question?

Yeah. I just wanted to understand better what do you mean by the whole test and PR part. Is it a case of, uh, you-- since you already have the PR with the unit test, you can essentially yank up the change and get the model to train or, or evaluate that they are able to pass the, the new test.

Is, is that what you're trying to aim for in simplicity? So that means they may come up with a different solution-

Mm-hmm

... different from the actual solution that was given, but as long as they pass the test, it, it, it meets the criteria.

Yep, exactly. So, um, one, one approach would be let's write issues and let's write tests manually to verify whether that issue is solved. But the idea is like how can we just pull this data off the internet for free?

So if you look in the actual dataset, you'll get the instance ID, which I think has the PR ID in it. Um, you'll get something called a patch, which was the sort of like ground truth solution, um, which for the most part you can ignore.

Uh, you'll get the problem statement, which is a scrape of the issue, um, that was created. And then here's the important part, and this is actually the most confusing part. It took me a few s- days to get.

There's these tests. Uh, one of them is called fail to pass, and the other one is called pass to pass. Um, but in general, these are the unit tests that you need to pass to make sure that you're actually solving the issue.

Uh, 'cause there's, there's way too many tests to, to, to run the, the whole thing. But the, the pass to pass ones are the ones that you should not regress on. And then the fail to pass ones are ones that failed before solving the issue- And then now pass after solving issue.

So these are the, the tests that were added in for the PR.

Do- does that make it more concrete, Eugene?

Yep. Makes sense to me.

Um,

yeah. So they basically pull those, um, for each of these repos. It's dominated, uh, I think mostly by Django, if I remember correctly. And I'll, I'll say the, the critique here is that these are, like, really robust, uh, you know-

Hm

... repos that are-- have a lot of history, have a lot of people. But they don't really represent, like, what it is to do day-to-day work for us that are building, like, apps and client things. Um, so there, there's a little bit of a, a diff there.

Um, and then they sort of go into how they approached it at the time. Um, but if you take a look at the results, the results are, are quite poor to what people are currently putting up on the leaderboard today.

Uh, so they, um, based on the technique, they either get, like, somewhere around, like, 2% or up to, you know, 50... Well, I, I think this is actually recall, so they, I think hit, like, uh, I think somewhere around 2% based on Claude.

Um, but now people are hitting things like 50%. So I just wanna switch over here to their website, which is a lot clearer than the paper. Um, so if you go to the website, you can actually see the current state of the leaderboard.

And this is just, like, amazing work. They, like, facilitate the community. They review all the submissions. They update the leaderboard, uh, on, on, like, some sort of, like, biweekly basis. And you can see here there's been a ton of submissions where they started out down here on, like, if we just use GPT-4 plus RAG, what do we get?

It's, like, a measly 3%. Um, and then up to the most recent submissions where they get up to 43%. Um,

and any questions around this? 'Cause th- this also gets a little bit confusing with all the different splits. But, um, happy to touch upon, like, you know, what was going on with, like, Devin or, uh, any of the, the other submissions.

Uh, I'll gladly read out some of the things from the comments. So, so one of the things was that, uh, from Sean, um, uh, he wonders what fixes that, uh, verified, verified applied. Um, um, yeah. Uh, basically for the, for the, uh, SWE-Bench verified, like, like, um, as a...

Also, SWE-Bench is very different, um, compared to the HumanEval. There's, if... There's still demand for better HumanEval, where, uh, do you, do you still see there's a demand for it? Or, like, is this meant to replace or is this a separate thing?

Also, another interesting one that came out is that somehow this still remains so far despite o1 coming out after, after it came out. So how, how does this-- what-- how does the work that you have done apply to chain-of-thought usage in o1, et cetera?

I think those, those were the questions so far.

Okay. Um, that's quite a few questions, and I don't know if I have answers for any of them. Uh-

Yeah.

Let me see if I can pick off.

Okay. Why don't we start. Do you think there's, uh... Is there still, uh, space for a new version of HumanEval, uh, since this is actually quite dramatically different?

Well, the, the way I would like to see this broken down is actually sort of like a multi-step thing, where people have sort of broken this down into, um, tasks of, like, file localization versus, like, file repair generation.

And I think you could take something like this and call it representative tasks-

Mm-hmm

... or repre-- yeah, sorry. Representative tasks, and then break it down into steps like retrieval and prepare, such that you indep- independently attack those. And then the second on these tasks would be more of like a more representative HumanEval.

So I, I guess that's my wish list for someone that was making a benchmark today.

Okay. Then I think the, the other question was regarding comparison to o1. Uh, since you're doing multi-step approach, you are e- in a way, you are still do- you are already doing chain-of-thought. Uh, and yet you are, and you are able to outperform o1.

And on the flip side, the, the, the team at OpenAI insists that you-- just replicating chain-of-thought is insufficient to replicate o1, so. But you are surpassing that.

Sorry, I... Yeah, I, I don't really understand the question.

Uh, basically, uh, uh, y- currently the models are performing o1s, the GPT o1. What's your thoughts on that?

Oh, okay. So I, I think your question is if you go to the model card, is, is, is that-- is, is that what you're referencing?

So-

Yep, I believe so

... if you go to the... So, so here, here's what's really important here. And what-- we can go to the, the Devin, uh, in parallel with this. So what's really important here is that this benchmark uniquely shows off sort of like agentness or, or s- or what they're calling scaffolding now.

Like, all of the delta between getting from 2% up into... Or, you know, what they call 2% for GPT-4, which is not really legit, right? Like, it's like, can we just solve this with just the model? Like, no, not really.

You have to have RAG. You have to be able to go around and execute things. Uh, the diff between that and something like Devin is all in, like, sort of, like, the agent scaffold or the agent code, right?

So that's what's really exciting about this stuff. It shows off what you can do just from prompting and just from adding tools. Um, and, uh, and yeah, I think that's what we're seeing here, is that they've tried different open source scaffolds in the o1 model card in evaluation.

Um, but all of these are, um, open source scaffolds that are not at the top of the leaderboard. So in the o1 report, they used AgentList. Um, and AgentList was scoring... Let me see. On Verified, they were scoring somewhere...

I can't find it.

Okay. Did they--

They might have blown up the

Well, at least on Lite they were scoring... Okay, well this is not super comparable, but I, I would bet that on, on Verified they were scoring less than this. So the delta here is comparing AgentList with 4o versus o1 and, and not like, can we beat the top of the leaderboard, which is all closed source scaffolds.

Does, does that make a little bit of sense?

Yes, that makes sense. Awesome.

Yeah. So what... Yeah, so what's really cool is, like, everyone is using the same base models here, or the, the same model APIs period. I, I don't know if anyone except for Cosine was, was even training, but, like, the diff here is all in prompting.

Um, okay. Um, yeah, let me move on so we can get to Verified as well. Um, so the problem is how do we evaluate real coding tasks? Well, we're going to pull a bunch of PRs to get the, the tests for free, to get the issues for free, uh, to get the scale for free, right?

Because they started with 2,000 tasks. Um, and so that's really cool work, and like pushing coding things forward. Um, some of the critiques I had was that in practice, um, you have to do a lot of fancy Docker trickery, um, just because you have to set up the entire environment for that repo.

If you want to execute anything, you have to install that repo itself, all the dependencies. Um, and then the other tricky thing is since you can cheat by basically coming from the future because, you know, a PR in 2019, uh, if you have all the information of everything that came before it, you're gonna be able to beat it.

Um, you have to set up that PR in 2019, right? So you have to set up the Docker repo for every commit, not just for every repo. Um, and the second critique is there's, there's way too many ways to cheat.

Um, so as opposed to s- the other ways you could set up a, a test benchmark, um, the entire test split here is public. Um, so you can do things like just overfit to the patches in the test set.

You can do things like m- like let me at, uh, at runtime pull the PR and just get the answer and just use it, right? And so I, I think what they're really struggling with is that there are thousands of ways to cheat on here, and as a result, um, the SWE-Bench team, you know, like, uh,

there's a, there's a separate repo called Experiments where you, you actually submit your, uh, run result. Um, they actually require you to submit these things called trajectories that are proving, um, what you did and that you're not cheating.

And a lot of this, a lot of this ends in controversy. A lot of this ends in people saying, "Well, we don't want to submit at all because if we review what we're doing as intermediates, then we review the entire-- we reveal the entire system.

And, like, we want to make money off of this and we can't do it." So that's, that, that sort of explains a little bit of why, like, you see some people submitting or not submitting or pulling down their entire submission, um, and why there's so much controversy around this, because you can really easily cheat it.

So to verify, you have to reveal something about your system. Some people don't want to reveal it, um, and s- and some people are cheating.

Yeah. And if anyone knows where Honeycomb went, um, w- we also want to find out because th- those guys were super smart. Um, and, uh, yeah, it looks like they, they pulled down their page as well. But I, I don't think that has anything to do with the leaderboard.

Cool. Um, any questions there?

All right. Cool. So now I'll move on to Verified. Um, and before I cover Verified, I just want to, uh, cover what, um, Swix was saying, which is that, like, one of the issues with the re- initially release was that the full dataset has, I think, over 2,000 instances.

And I think at the time, depending on what you do for your agent, it could cost anywhere from, like, a few cents all the way up to four or five, maybe, like, $8 per run. Um, and so, like, the math there is not really great because you're saying every time you run an experiment, I have to blow, like, $16,000 in OpenAI credits.

Uh, and I think that was the motivation for creating Lite, uh, which is just a subset of, um, a full that's, like, 300, so you don't have to spend all that money. You can run it faster. Um, but I don't think they did a great job in selecting what goes into Lite.

So I think that's the, the sort, sort of a motivation for, for this Verified work. Um, so this Verified work actually came from OpenAI's, uh, I think they have, like, a coding research team. Um, and the problem is, like- Is SWE-Bench actually solvable?

Because one of the, I guess, depending, depending on what you think, one, one of the requirements for a benchmark is you should be able to hit, uh, somewhere around, like, ninety to a hundred percent. I think if you look at some of the examples by hand, and so the way you would do that is you would pull this ID, go to Django, Django.

Uh,

go to issues, and drop in that issue there. Sometimes it feels like you try to read the issue

and you're just like, okay, even if I was, like, an oracle or, like, some sort of god coder, I couldn't-- I would not be able to solve this because there, there's some context that was left out here. There's something that, like, this is, is not complete about this.

Um, so, so as far as I can tell, that, that's sort of the motivation here is, like, are these actually solvable problems? Um, so what they ended up doing is they created a three criteria survey that they sent out to human raters.

And they, they took this, um, themselves first to, to get calibrated. Um, but the three cri-criteria are: Is the problem well-specified? And then they have, like, a, a ranking of the difficulty. Uh, or sorry, the how well-specified it is.

Um, how valid is the test or the criteria? And then how difficult are the tests? Um, and here they break it down into the amount of time that you expect to, to spend fixing it. And, uh, I think in-- somewhere in here, they basically said they had paid all these raters and they had, uh, three raters, um, each take a pass at-- Uh, for, for each task, they had three raters go at it, and so they could sort of corroborate between those three raters the difficulty and whether it's well-specified.

And then finally, they did a filtering. They just said, "We filter out anything where, um,

something has a score above two, uh, and we filter out anything that is really hard and long. Um, and then we are probably about the five hundred how to handle verify. So it's-- I think they, they came up with the number first, and then they came up with the criteria.

But basically the idea is like, let's just stick to the ones that are, um, actually feasible. Um, and in that way, I think we should be able to hit a hundred percent eventually.

Oh, that's interesting. Yeah. This, this is actually, um, like a really good guideline on how to set up your annotations.

Yeah, I can-- maybe I can chime in on this, right?

Yeah, go for it.

I mean, I think, like, you can see on the fourth paragraph, a team of engineers first labeled fifty sample trials. So essentially these fifty samples, you can consider them as the golden, golden dataset. And then when you hire new annotators, hire or train on new annotators, these new annotators have to go through these fifty samples, or maybe initially it'd be twenty, and then if they fail, then you give them another chance on another twenty or twenty-five.

Um, and then this is how you eventually graduate your annotators, right? Uh, because it's not like anyone from Mechanical Turk or anyone from Scale can, can actually just do this. You do need an evaluation set to evaluate your human annotators in the first place.

Then after that, they shared a few things about their annotation criteria, having different labels. Um, personally, I'm more inclined towards a binary label, uh, which makes it easier. But in this case, because they wanted such fine-grained granular data, I think it makes sense that they have a four, uh, Likert scale of four.

Uh, zero, zero, one, two, three. And even then, I think there's genius in that in the sense that people cannot choose a tie. You cannot choose a middle, a middle label. They force you to either say incline or, or decline, whi-which I think is brilliant.

Um, that's it. A-as they're collecting this, right? Collecting data is extremely expensive. Uh, they don't just collect the label. They actually then get data on how to weigh their interrater, uh, rela-- Uh, no, sorry. You collect data on how to rate the difficulty, right?

And towards the end, they actually measured, uh, they had it labeled three times. And then even that last sentence, uh, we conservative, conservatively ensemble on annotations by taking the highest severity label, right? There's, there's so many ways to do this.

You could take majority vote, you could take highest severity level, you could take average. Um, and, and the, the different ways to do this is actually very nuanced. Uh, and they chose the approach that-- And in, like, some later below, they actually mentioned that this leads to the high, uh, a lot of false positives, whereby they actually throw out a lot of good, uh, a lot of good test sets because just because one, uh, one annotator said that it was not good.

In comparison, if you just choose two, if, if you choose majority vote, you might have fewer false positives, but they just wanted a very high-quality dataset. So in, in that section over here, you can see that there's a lot of thought being put into how they actually collected this initial annotation data.

Um, and it sounds like you've been put through human rating hell before.

I mean, I've had to rewrite some of my SOPs a handful of times just to get better coherence, just to get better inter-rater reliability. So, uh, I, I, I thought their, their work was, was, uh, I, I really enjoyed it.

Yeah. Yeah. It's definitely some of the harder, less enjoyable, less rewarding work out there.

Um, I will say that I had a question which is like, um, if, if they were going to have all these humans paid and assuming, you know, they, they have those billions and upon billions of dollars, why didn't they just have the humans solve the problems and then collect the demonstrations?

Um, because it, it's, it's not too dissimilar from like, you know, collecting, I guess, like, chain of thought or reasoning data. Um, and the reason why I ask that is 'cause, like, if I were to do this and I were to take this survey and say, like, you know, "How difficult is this?"

I, I might, you know, be having a good day and say, "Everything's so easy. I can definitely solve this problem." Or I might just say like, you know, "It's too hard," even though I haven't actually solved it.

Yeah. That's a good question. I would imagine it's because it's cost- costly to try to get them to solve the problem. Uh, I, I can imagine if I were them, and if I were to do something like this, I would get an LLM to solve the problem, and have a human rate it.

And that's what they did in CriticGPT. Have the LLM solve the problem, and then have the human rate it.

Which is also by OpenAI. CriticGPT.

Cr- Critic?

CriticGPT.

Okay. Cool. Good to know.

Yeah. So, so that, that was the exact approach they did. I think that was the approach they did. They did this approach whereby get an LLM solve the problem, uh, insert bugs, human will insert bugs, and human would...

Hu- human is really just a preference tracker, which is just a lot faster.

Awesome. Um, anything else to cover here? Um, 'cause if not, I'll, I'll go through a couple of the, the leading people on the leaderboard.

Oh, I also wanted to point out, um, in the paper, I think there's a section on, um, difficulty correlating with context length. Um-

Oh.

I don't-

That's true.

Yeah. I thought that was interesting. I don't know how relevant it is now. I think it's still relevant, but, um, folks who disagree, I would love to hear your views. Um, how relevant-- I'm not sure how relevant it is now, but essentially the point that we're making is that when they increase context length, performance drops.

Yeah, exactly. Exactly as you see there. And it's not just perform- uh, it's very interesting that they actually have two performance measures. One is resolve and one is applied. Um, so at a, actually that, uh, went into... Applied means the patch can be applied successfully.

And the patch is actually not a code, it's actually a git diff or a git patch applied. So the thing is, even as the context length increases, um, on the left side of the graph, you can see the red bars gradually getting shorter.

It means that the LLM is just fi- finding it harder to actually create a valid patch. And of course, the res- resolution rate actually drops. Um, so what this means is that maybe the inclination here, the, the, the suggestion here, the-- what we can draw from this is that RAG still matters, good retrieval still matters.

But that said, this paper has been a while. I'm not sure if with the li- longer context length, stronger models if we will still see such a, such a pattern.

Yeah. That's a really good question. Um, my, my guess would be that, like, long context works but is sort of a lie as far as accuracy. And that RAG matters no matter what, because even in the longest context windows, you can't fit the whole code base.

Um, but yeah, I'd, I'd love to see this updated.

Yeah. I think it's also related to your previous point about, uh, the question not being clear enough. Like, like, especially if the model is-- I mean, the model, you can just treat it as a new human who just joined the project.

Um, if you are just looking at the key files and the issue, maybe you don't have enough context. And, and then as subsequently when you-- like, we see a lot of people via Continue Dev and using Continue Dev and et cetera, has been dumping in their entire code base, uh, to provide that context.

It may not be efficient, but some folks do that, and it kind of works. Um, but I think, I think the point is, even though it works, does it degrade in the performance?

Yeah. I guess, uh, only the evals will tell.

Um, so while we're on that, I'll go ahead and give you some alpha here, which is that, um, you can go ahead and for each of the leaders on the leaderboard, um, you can kind of like infer what techniques they're using by looking at their trajectories.

Um, and you can see that people are all over the map. Um, so from memory, what I remember is like, um, AgentList is doing sort of like a, a full file search and, um, what they call, um, file level localization, where I think they, they take files and then they pass them to the LLM to decide if it's relevant.

And then they do function level localization, and they do line level. Um, then they take all of these proposals for what line to edit, then they repair it, um, and they sample multiple times from OpenAI, and then they re-rank.

Um, so it's like a very linear multi-step process. Um, and then there's stuff that's pretty wild, um, like, uh, like these guys Groo, where as far as I can tell, they only do sort of a symbol search, which is like, let me just search for like a variable or a function name, and that's how they get the files.

Um, 'cause I, I guess naively you would think like, let, let me just do like, um, some sort of, you know, chunking strategy or something. But people take all sorts of strategies to get to which file or, or which function to edit, and then people do all sorts of things after this.

Um, so if you want to explore for your own, what you can do is you can, uh, like find your favorite entrant. You can clone the experiments repo And then you can actually see for any given task, like what happened.

So this is really hard to follow, but I'll, I'll scroll down really fast. Um, they'll say like, "Oh, uh, your task is to solve this issue." Um, there's some obscure search that gets you a bunch of files. And then, um, instead of some other frameworks where it's only three steps, this is like, I think, like 20 something steps.

So it's very agentic, and the agent can do things such as search assemble. It can do things like read file. It can do things like run Bash.

Um, which is quite flexible. And then you can see all the tricks that they use as well. So they have pagination in their read file tool. They have line numbers next to each of the lines. Um, and so, um, that one's called Groo.

And then there's another one called Honeycomb, which also scored really well, and I think is number one on the full set. Um, you can see here what they do is something really interesting, where they're going to try to reproduce the bug first.

It's like pretty cool, very intuitive, um, but it takes a ton of steps, and they try to run Bash and fail, and try to run Bash and fail, and try to run Bash and fail. And it's very, very, very, very execution-focused.

Um, so yeah, there's just like a, a, a really large diversity of, of, of, uh, of techniques out there that people are using.

Um, yeah, any questions there?

Um, is anyone competing on this or trying to compete on this?

Maybe is anyone in this room fine-tuning a model for, to compete on this benchmark?

Nice. Yeah, I think it's still pretty wide open, um, 'cause all, all the delta here is from all these tricks and, and prompt hacks and, you know, you can, I guess, pretty easily beat, uh, Amazon Q or whatever their previous version is.

Um, but I'd be curious to see if the large labs have their own entrance coming out soon.

All right. So, um, now if we have time, um, there's two newer papers. Um, one is from the original SWE-Bench team, uh, and this is called SWE-Bench Multimodal. Um, the idea here is like, you know, screw all this Python stuff, uh, scientific computing, no one cares.

Like, let's get onto some real hard stuff, which is UIs. Uh, so UIs for diagramming, uh, for mapping, uh, a lot of web stuff, and then I think they have some games in there too. Um, so the question is like, how do we take sort of the pipeline that we've created for SWE-Bench, and how do we apply it for, um, a lot of JavaScript?

Um, so now you have things like Prism in there, Highlight.js. I don't know what this is. Uh, it's a bunch of JavaScript libraries, I think. And, um, this ends up being a lot trickier than the Python side. So they want to target front-end game development DevOps.

Um, and they're only able to pull six hundred and seventeen tasks from 17 libraries this time. Um, and the most important here-- uh, most important thing here for them is that every task has to contain an image either in its problem statement or the unit tests.

It's very w-weird, but it has to create-- it has to contain at least one image somewhere. Um, and then most importantly, they want to show off that this is, you know, guys, this is really hard. It's a really hard benchmark, so we can only get 12% on it today, um, according to, you know, their, their implementation.

Um, so if we scroll down a bit, um, I think they take basically the same approach, which is that they look for the repos with-- that are really popular. Um, this time they, they want a visual aspect, um, and then scrape the PRs.

Uh, this time they want to make sure that it's JavaScript or TypeScript rather than Python. And then they inspect the issue and test patch for working hyperlinks that point to an image. Um, so this is tricky because, like, nobody is very good at this.

Like, you know, you can-- I can-- you can draw assets in GitHub. There's like lots of third-party assets. So I don't know. It's like not very well specified to how to add images to your PRs in the first place.

Um, but they manage to do that, and then they filter out the less than 1% of them that actually have good images.

Um, but yeah, I think in general, they're just using the exact same, uh, pipeline that they had set up before. It's a lot of human work. It's a lot of filtering, um, just to get these, um, just to get these, these tests curated.

Um, but I will say, I got confused halfway down this paper because I, I took a look at the, um, actual data set, and as opposed to when we had this, um, test set, uh, test patch and failed tests and et cetera stuff before, um- This new one is actually pretty bare bones, so I'm a little bit confused about whether it's ready and whether, um, and how the tests actually work.

Uh, because if you think about it, it is like even if you have an image for the test, there are so many ways you can do it. Um, there's like exact pixel match. There might be some sort of semantic thing.

Um, you know, zooming out if, if you're to, to try to judge whether a UI is correct, it's like extremely subjective. It might be iterative. It might be, you know, I have to interact with it first to get it right.

So I'm super curious to see how they actually do the judging criteria, because it's not clear at all from the dataset, um, and it's not clear at all from the paper.

Let's just pop one of these out and see what it looks like.

Oh, God. When I place the mouse outside the legend, I expect the on leave event to be called all the time.

It's like the stuff of nightmares for me.

Um, okay, so luckily they have a test here. So this, this should probably work in the existing framework.

Let's see.

Yeah, I think this is pretty straightforward extension, uh, to me. I think

it's a little unclear because the code isn't out yet on what exactly is going on, on how they verify, uh, that the tests are passing.

And, um, yeah, like I said before, it's just, um, a little... Yeah, there's just... I, I guess we're still waiting on the code is all.

Oops. Yeah, UI, UI is even harder than the Python stuff. Um, does anyone have other ideas of how to evaluate UIs? I know there, there's some more recent stuff where people are using like agents to do QA, UIT type work.

Um, there's some ideas about how to do it if it's, uh, not exact pixel match, but maybe like semantically the same. Um, but yeah, in general, I think UI is just, it's much, much more difficult to evaluate than just having some functional, uh, unit tests.

I think the biggest issue is subjectivity. And for context, I ran a UI testing company.

And what'd you find out?

Humans are unreliable in subjectivity. That, that's just the root of everything.

Yeah. So I, I guess if we had infinite money and time, and we had very, I guess, deterministic humans, we could throw humans at it. But probably isn't, isn't a feasible approach.

I think the most, the most realistic one would be the subset, uh, instead of what users would want. More specifically, let's say the step where you have a designer create something in PSD and then, uh, and then does the AI translate that into code.

That might be the most, uh, objective one. But then I can... Then here's the next curveball. Normally people design PSDs, they only design for a specific screen size. What happens when it's not that specific screen size? Welcome to UI help.

Yeah. Yeah, I think the broader thing is that, um, the more and more realistic you get, the more you run into sort of like multi-turn or iterative things. Where now the task of AI isn't just to just, you know, extract from your brain what the problem is and, and kind of directly solve it.

The task is like, how can it ask you clarifying questions and extract more knowledge from your brain and iterate? Just really, really, really hard to evaluate.

Um, and there are some people that have started down that path actually, uh, in, in web agents, but it's, it's, um,

it's really a doozy to even try to understand what's going on, to be honest. So I don't really want to go into it.

All right. Um, any questions there before we go on to the final paper?

Thanks.

Big market.

Does anyone have an answer to what the best web agents out there are? I have no idea.

Alex wants to know, maybe the question is, what is the first measurable consequence or result of the UI?

Do you want to elaborate on that, Alex?

All right, maybe we'll come back to it. I think it's a good question, though. Um, so the final paper, and this came out really recently as MLE-Bench. And so this is, I think, coming from sort of like the insane OpenAI thesis about how AI then creates AI, which accelerates the whole thing to AGI, ASI, um, but in a very constrained, very, um, you know, toy sandbox version.

The question is, instead of solving regular Python, instead of solving UIs, can we build models, uh, from agents? And what they do is pretty smart. They just basically scrape Kaggle and sit on top of all the existing Kaggle infrastructure.

So they pull down a Kaggle competition, they pull down the dataset, um, and then they use, uh, the Kaggle APIs to just create the submission and that's, that's it. Um, it's, it's a lot of orchestration work. It's a lot of, uh, figuring out, like, how to do meta program- meta programming on top of it.

Um, but luckily Kaggle exists, and it's, like, pretty well-defined. Um, so you give the agent the dataset description, uh, the leaderboard for some reason. I'm not really sure why. And then the agent, uh, is able to or should be able to train the model from scratch, test it, iterate, create the final submission, and then, uh, submit to Kaggle.

And in testing their models, 'cause, um, one of the recurring themes is that they don't do any of the scaffolding or, you know, prompting or agent code. They usually use the open source ones and advertise their models. Um, they show off, you know, what does a step look like?

Well, a step could look like listing the files out, or it could, um, be calling some arbitrary bash. Or, um, I have no idea what's going on here, but, uh, doing some really hardcore, uh, data science work.

Um, and what they are able to find out is a, a few things. Um, one-- Uh, sorry, this is jumping kind of all over the place, but they, they got some pretty decent results. Let me pull up the paper.

Um, their final results with o1 preview and this, uh, aid scaffolding from a different company was that they got a bronze medal. I, I don't think I've ever achieved, uh, once, uh, but I haven't competed that much, uh, in seventeen percent of competitions.

Um, and they have some comparisons to grandmasters and some other things over, yeah, somewhere below here, but seems pretty good to me.

Um, and they spent a lot of the paper honestly trying to sort of like strawman themselves. They, they asked like, "Oh, is this a real, is this the real deal? Can we cheat on this? Um, is this, is this a legitimate benchmark?"

Um, so what they make sure is that developers have to mention the models and scaffolding. They have to mention whether you had internet access. You have to mention the hardware, the runtime, and then you have to mention whether you already had existing solutions in the Kaggle prompts, because that's also cheating.

Um, and then this section is what I titled How to Cheat, because basically, you can write the predictions directly to the submission panel. If it's already in the model weights, uh, you can view the solutions online. You can, uh, yeah, you can just directly write the predictions, uh, 'cause maybe, maybe the, maybe the agent or the AI is good enough so that you don't need to write a model.

You j- you just are the-- you, you are the model, so you just directly spit out the answer. Um,

and then, yeah. And then if there's plagiarism, they can sort of detect it there.

Um, and then in the setup, this is where it gets kind of interesting. They give you, or they have you run thirty-six VCPUs with a NVIDIA A10 GPU, um, and have a, a twenty-four-hour, uh, time to solve, uh, all the problems.

So this is really sort of this like, oh, can you as an agent sit down in a room, uh, no sunlight, no food, solve Kaggle competitions, uh, with this, with this GPU, uh, given, given twenty-four hours and just the spec that everyone else gets?

Um, and then, you know, of course, they say o1 preview, uh, significantly outperforms the other models. Um, and, uh, since a grandmaster requires five gold medals and o1 preview gets an average of eight-- or sorry, seven gold medals, um, they're outcompeting even Kaggle grandmasters.

So, yep. Um, and then finally, another thing that they, they say as sort of, um, a critique of their own setup is that, um, it might be easier to solve these now than it was before when the Ka- Kaggle competitions came out because we have more-- we have, you know, better approaches, we have better technology.

We, we might have better hardware.

Um, and then this is some really good information on how to make these things better. Um, a lot of these agents run early or terminate early instead of using the full twenty-four hours. And all three agents didn't factor in the compute and time limitations in their strategies.

So, like, as humans, we would say like, you know, maybe time boxes to four hours, and then in the last four hours, you're gonna yellow it and try everything. What's interesting is like, and I, I've seen this in practice, it's like it's hard to get the agent to say, to, to think in numbers of steps, and especially in time, because it doesn't know time.

So if you, if you tell it like, "Please complete under 50 steps," it won't, it won't do it. Um, and maybe that's just like not represented in, in the training data. Um, but it's, it's one of those gaps between what a human would do and, and what the agent would do.

Yeah, so the-- does o1 preview run on an A10? The, the, uh, the A10 is for the agent powered by o1 to run its own model. Uh, maybe it's just like, uh, as basic as linear regression or, you know, uh, boosted trees or whatever, uh, to solve the Kaggle task.

And actually, down here they say, um, they compared a CPU-only setup to a GPU setup to a multi-GPU setup, and it kind of made no difference really. So, um, if you were to, to run this yourself, you might just want to run it on a CPU setup, and you could probably solve, uh, a decent amount of problems.

It's really interesting.

Yeah, and then finally I did some back of the napkin math, and they did twenty-four hours times seventy-five competitions, which is eighteen hundred GPU hours, and then they used a ton of tokens on o1. So what they spent, and this is per, uh, seed, what they call it.

So they, they run, um, these things over and over again with multiple seeds to get like a, a pass at K computation. Just for one seed, for one run of, uh, these things costs, uh, four thousand dollars all in with the GPU plus the tokens.

Um, and a bulk of the cost is actually the token. So even if you cut the GPU out, it, it'll still cost you three grand to run an o1 preview.

Um, and then finally, if you do want to run this yourself, it's extremely painful, um, because this apparently costs you two days to even just prepare and download the data. So, uh, so good luck to anyone that's, uh, going to compete on this.

Um, yeah. So, so real quick, my critiques are, uh, four K per run per seed. Hopefully, that'll go down. Hopefully, you don't need the GPU. Um, it's unclear how this actually maps to real-world ML stuff because Kaggle is pretty nicely defined.

You get the dataset, it's cleaned for you. The criteria is defined for you. There's a company that's sponsoring it. They, you know, have a real-world use case. Uh, but in real practice, you're kind of just like going back and forth on data.

You're cleaning data. You're figuring out like, what the heck am I actually going to produce? Like, how does this even like move the needle on the business? Um, so a lot of questions there on how like if-- whether solving this actually solves anything in the real world.

And then, um, as opposed to the SWE-Bench folks, I don't think they've put out anything as far as a leaderboard or any real community. Um, but maybe, maybe they're working on that or will combine forces soon.

Uh, Eugene, uh, the real-- the question I have for Kaggle is most of it has ended up on GitHub as training data. So I don't know if they addressed that one specifically, but they do a lot of, uh, checks against, um, overfitting on the Kaggle tasks themselves.

And so they do something where they obfuscate some of the details of the, um, of the competitions, and then they rerun it. And, um, I guess if they were overfitting on the competitions themselves, this should go down a lot.

Uh, but according to them, it does not overfit to the competitions.

I think I can expand the question on that. I think there's two parts to it. One step is understanding the composit- uh, the task. The second step is implementing the solution for the task. If, um, my question is, it-- the-- I guess if you mask the question or change the question, you are fixing-- you are preventing the overfitting for the first half.

But if the model understood the task, it already has it memorized might be the issue.

Hence why, why it's scoring grandmasters.

So you, you're saying one part of it is like mapping from the description to the task, and then the second part is whether it already knows like the dataset?

Yeah. Because a lot of the Kaggle, uh, solutions are on GitHub. So even if-

Mm.

I do not put the-

Oh, I see

... the competition task in the GitHub, if you have memorized the code, you kind of already have the task to code map up.

Yeah. Okay, keep going. Oh, I mean, yeah, Eugene has a good answer. There's, there's always-- We can test this because there's always new Kaggle competitions. Yeah. Um, I guess the, the test is just Kaggle.

So I guess, like, I guess we'll wait to see for the next one whether o1 ends up on the leaderboard.

Test of time.

Yeah.

Awesome. Um, yeah, that's about it. Um, any other-- any other questions?

By the way, for the rest of folks, uh, I think since we are reaching the end, feel free to unmute and ask questions directly.

Did, did they have any examples of the code that was generated for this?

On MLE-Bench?

Yeah. Just be interesting to look at.

I actually haven't taken a deep look at the repo yet, but I remember there's the examples directory. No, I, I don't think they have any directories or anything like that yet.

Okay.

So then for SWE-Bench, have you looked at any of the generated code just to,

you know, does it smell good or?

Um, I, I think they even have a viewer in here. You can take a look at the submissions. I could be wrong. Um, let's see. You mean from the, from the folks that are, like, sort of ranking on the leaderboard?

Yeah.

Yeah. I mean, it, it's real-it's really hard to tell. Um, I, I put this thing together, uh, for my own testing. Um, it, it's, it's really hard to distinguish, like, if it's, like, um, AI generated or not, but there are some smells that come, like, purely from AI-generated code where it'll sort of like miss a line edit.

It's really hard to describe, but there, there's a lot of times where, like, code will be duplicated accidentally, and you'd be like, "Oh, on-only an AI would do that." Is that-

Mm-hmm.

Is that sort of what you're asking about?

Yeah. Just like, you know, when it writes a function, does it do it in a straightforward way, or did it have to iterate a bunch of times and ends up with like some convoluted thing that does the same thing but is harder to read?

Oh. Uh, well, I'll say from my experience, um, like when I run things, um, I-- Yeah, sometimes I-- it, it, it does keep adding and adding and adding and adding code.

Mm-hmm.

Uh, but I mean, that, that's just sort of like me being bad at prompting, I think. Not really.

Yeah, I, I guess what I'm, what... Yeah, uh, there, there's something nuanced about this data set in particular, where all the edits are super short.

Mm-hmm.

And it's like a prior that you can put into your code. But, um, if you don't have that, then yeah, like, agents tend to just keep making obnoxiously long edits.

But there, there's nothing that, like, I've seen from simple edits where I can look at it and say like, "Oh, that's definitely like LLM-generated code." Has anyone else seen anything like that?

All right, cool. Um, I'll hand it back to someone else.

I feel like I had an answer to the question, but I w-- I don't remember what the question was. Was it, uh, have we seen any, uh, generated ML, uh, code from agents? Or was that not the question?

I only heard half of it.

It was like when, um, the agent solves SWE-Bench, do the final results look like something a human would have written, or does it look like a convoluted thing that went through a bunch of iterations and eventually solved the problem or-- but would be very hard to maintain?

Oh, I see. Yeah. Unfortunately, I don't have any further info on that. I must have misheard. All good.

All right. I'll drop my email on here in case anyone wants to chat more about this stuff, which is like one of my favorite topics.

Also, he's on the maintenance space Discord . Feel, feel free to, to ping him on it. Yeah, great work, Jess. Uh, so for next week, right, for next week, we will be going, uh, s- uh, we have, let me see, Ethan from NVIDIA who's going through the Megatron, uh, Megatron distillation or MOE.

Um, I'm not so sure which one. Um, and yeah, look forward to it. And also, if anyone has any papers that they want to cover, feel free to, to suggest and als- and contribute as well. And, and we may see y'all in the next, next week as well.

Oh, okay. So it's Megatron. All right. Yep. So it's Megatron. Yep. Okay. Then, yeah, take care, everyone. So, uh, same bad time, same bad place. See you around.

Eugene, is it the Megatron paper or is there a paper specifically on Megatron distillation?

Uh, I'm a bit confused because you said MOE and then Swig said distillation, and I'm like, okay, so which one is it? Or is it both? But-

Mm-hmm.

But anyway, it's, it's someone from NVIDIA that's covering it. So, so I think we can confirm later.

Okay. So at least I just need to make sure I read the paper real quick.

Okay. Thank you all.

Yeah, I know.

Yep. No worries. I think we can confirm in the Discord.

All right.

Well, thank you.