# [State of Code Evals] After SWE-bench, Code Clash & SOTA Coding Benchmarks recap — John Yang

Latent Space · 2025-12-31

<https://addtry.com/30cecdc2-8912-4202-8ef9-415ae749b31e>

John Yang, creator of SWE-bench, explains the benchmark's evolution from an ignored project to the industry standard after Cognition's Devin launch, and introduces CodeClash as a new paradigm for evaluating long-horizon coding agents via programming tournaments. Yang covers SWE-bench's expansion into nine languages (JavaScript, Rust, Java, C, Ruby) and multimodal tasks, the proliferation of independent variants like SWE-bench Pro and SWE-bench Live, and the shift from unit tests to iterative agent competitions. He addresses the Tau-bench controversy over impossible tasks, arguing they serve as cheat detection, and weighs the tension between long autonomy (5-hour runs) and interactivity (Cognition's fast back-and-forth). Yang highlights the academic data problem—companies like Cognition have rich user interaction data, while academics need user simulators or compelling products like LMArena—and positions CodeClash as a testbed for studying human-AI collaboration by freezing model capability and varying interaction setups.

## Questions this episode answers

### What is CodeClash and how does it improve upon SWE-bench for evaluating coding agents?

John Yang describes CodeClash as a tournament where AI models maintain their own codebases, iteratively improving them and then competing in arenas to determine which is better, judged by an LLM. Unlike SWE-bench’s independent tasks with unit test verification, CodeClash evaluates long-horizon, consequential development, capturing self-determined coding and adaptation over multiple rounds.

[3:11](https://addtry.com/30cecdc2-8912-4202-8ef9-415ae749b31e?t=191000)

### How did SWE-bench become the industry standard for evaluating AI coding agents?

SWE-bench was released in October 2023 but initially ignored. It gained traction after Cognition’s Devin launch. John Yang recalls receiving an email from Walden about two weeks before the release, saying they had a good score. The Devin demo was mind-blowing and triggered an arms race among AI labs, with SWE-bench becoming the de facto benchmark.

[0:37](https://addtry.com/30cecdc2-8912-4202-8ef9-415ae749b31e?t=37000)

### What is the controversy around Tau-bench, and what is John Yang’s view on impossible tasks in benchmarks?

Some argue Tau-bench tasks are underspecified or impossible, making high scores unachievable. John Yang acknowledges the concern but appreciates the idea of intentionally including impossible tasks to catch cheating. He notes that Karthik defended the benchmark, and mentions ImpossibleBench, which modified SWE-bench issues to be impossible, revealing that models often claim success instead of refusing.

[9:13](https://addtry.com/30cecdc2-8912-4202-8ef9-415ae749b31e?t=553000)

## Key moments

- **[0:00] Welcome**
- **[0:20] SWE-bench Origins**
  - [0:51] Cognition's Walden emailed John Yang two weeks before Devin's launch with a strong SWE-bench score, and the release was mind-blowing.
- **[1:11] SWE-bench Variants**
  - [1:49] SWE-bench Multilingual covers nine languages across 40 repos, including JavaScript, Rust, Java, C, and Ruby.
- **[3:08] CodeClash**
  - [3:11] “I don't like unit tests as a form of verification”
  - [5:22] CodeClash is shifting from programming games to building economically valuable arenas, aiming for real-world utility.
- **[5:58] Group Benchmarks**
- **[7:20] 2025 Trends**
- **[9:12] Tau-bench**
  - [9:14] Tau-bench's impossible tasks: critics say they prevent high scores, others argue they detect cheating.
  - [10:39] ImpossibleBench altered SWE-bench issues to be unsolvable; models still claimed success, revealing overconfidence.
- **[11:03] Future Evals**
  - [11:05] By 2026, coding evals will proliferate with more SWE-benches and creative environments like Terminal-bench 3.0 and 4.0.
  - [12:11] John Yang envisions 5-hour autonomous coding runs vs Cognition's emphasis on fast back-and-forth interactivity.
- **[14:31] Call to Action**
  - [14:31] Academics need user interaction data from companies or better user simulators to advance coding evals.
  - [15:30] CodeClash can test human-AI collaboration by varying setups: solo agent, multi-agent, human+agent, across multiple arenas.
- **[16:21] Cognition's Push**

## Speakers

- **John Yang** (guest)

## Topics

Benchmarks, Coding Agents

## Mentioned

Cognition (company), Meter (company), AlgoTune (product), CodeClash (product), Critical Point (product), Devin (product), GDPeval (product), Halite (product), SWE-Bench (product), SWE-Bench Pro (product), SWE-Bench Verified (product), SWE-bench Live (product), SWE-bench Multilingual (product), SWE-bench Multimodal (product), SWE-efficiency (product), SciCode (product), SecBench (product), SuiteLancer (product), Tau-bench (product), Terminal-Bench (product)

## Transcript

### Welcome

**Host** [0:03]
Light in space like 2025.

Break ups. Light in space like 2025. We're here at NeurIPS with John Yang of SWE-bench and many other things, but welcome.

**John Yang** [0:18]
Thanks so much for having me. Yeah, really happy to be here.

### SWE-bench Origins

**Host** [0:20]
Uh, last year I talked to Ophir and, uh, I think Carlos as well, one, one of your co-authors.

**John Yang** [0:26]
Yeah.

**Host** [0:26]
How's SWE-bench doing? Like, just, just generally the project is, like, one and a half years old?

**John Yang** [0:32]
Yeah, yeah.

**Host** [0:33]
Or so.

**John Yang** [0:33]
I think one and a half years old in terms of when it was actually useful.

**Host** [0:36]
Yeah.

**John Yang** [0:37]
You know, we put it out October 22, 23, and then people didn't really touch it too much, and then of course, like, Cognition came on the scene and Devin was an amazing release. And I think after that it kind of kicked off the arms race of-

**Host** [0:49]
Did they tell you beforehand or they just showed up?

**John Yang** [0:51]
It was... You know, I got an email about, like, two weeks ago. I think it was from, I think it was from Walden. It was like, "Hey, you know, we have a good number on it." I was like, "Wow, congrats, you know, thanks for using it."

And then the release was, like, mind-blowing. I was like, "Wow, these guys did an excellent job."

**Host** [1:06]
Amazing. And then SWE-bench Verified was, like, maybe last year.

**John Yang** [1:10]
Yes, that's right. Yeah.

**Host** [1:11]
Uh-

**John Yang** [1:11]
Like-

### SWE-bench Variants

**Host** [1:11]
Catch us up this year. Like, you have, uh, other languages. You've, uh, there, there's, like, a whole bunch of varieties of SWE-bench now.

**John Yang** [1:18]
Yeah.

**Host** [1:18]
So what should people know?

**John Yang** [1:19]
Yeah, for sure. Um, I think there's a couple extensions that have happened. One is, like, more SWE-benches, SWE-bench Pro, SWE-bench Live, um-

**Host** [1:27]
Oh, SWE-bench Pro, was that with you guys? Because it looks independent. It's, like, different authors.

**John Yang** [1:31]
It's completely independent, yeah.

**Host** [1:33]
So they just called themselves SWE-bench Pro without your blessing?

**John Yang** [1:35]
Yeah. I think, uh, I think we're, we're, we're, we're okay with it. Uh, when we came out we were like, "Oh, cool. Interesting." Would've been, you know, fun to be part of it, but you know, I mean, congrats to them.

It's a great benchmark, yeah.

**Host** [1:46]
All right.

**John Yang** [1:47]
Yeah, yeah.

**Host** [1:47]
Uh, but yeah, uh, multimodal.

**John Yang** [1:49]
Yeah.

**Host** [1:49]
The big one.

**John Yang** [1:49]
We did multimodal and multilingual, um, and I think, like, those have- multilingual seems to be the-

**Host** [1:55]
Is, is that, is it, like, JavaScripts? What else?

**John Yang** [1:57]
Yeah, yeah.

**Host** [1:57]
Ruby.

**John Yang** [1:57]
Yeah, multilingual is, like, it's, like, nine languages-

**Host** [2:00]
Yeah

**John Yang** [2:00]
... across, like, 40 repos. But yeah, you got them. Like, JavaScript, Rust, Java, C, you know, Ruby. Yeah, yeah, you got them. Yeah.

**Host** [2:08]
And then core SWE-bench itself, a lot of people, like, they, they talk about the, the Django focus.

**John Yang** [2:13]
Yes. Django.

**Host** [2:14]
Uh, is there- Is, is there, is there, like... I don't know. How do we, how do we move past Django?

**John Yang** [2:19]
Yeah, for sure. I mean, it, it's cool to see, um, a lot of the newer benchmarks, like, really try to di- diversify the repos. Like, in the two follow-ups we did with multimodal and multilingual, we made it a point to do that.

So I think-

**Host** [2:31]
But you can also just put out SWE-bench 2025 and just-

**John Yang** [2:34]
That is true, and do a new distribution. Yeah. Yeah, so it's been cool to see the follow-ups. I think quietly, and, and it's an open question for me, I'm excited to see how people curate the next sets. Like, it's kind of interesting to see in the literature or in their blog posts, like, how they're justifying why they're creating their separate split.

The easier ones were like, "Oh, more languages, more repos." And then I think now people are like, "Well, ours is more difficult because of this curation technique." And I'm, yeah, I, I'm excited to see how, how long that lasts and, you know, where we're gonna, like, guide the evaluations towards.

### CodeClash

**Host** [3:08]
Yeah. And more recently you're working on CodeClash.

**John Yang** [3:11]
Yes, that's right.

**Host** [3:11]
Uh, so let's give people... You, you've already done other episode, uh, other podcasts about it.

**John Yang** [3:15]
Yeah, for sure.

**Host** [3:15]
We'll refer people to, to that with, uh, your chat with Andy.

**John Yang** [3:18]
Yeah.

**Host** [3:18]
Uh, but just give, like, a people, like, a one, two sentence-

**John Yang** [3:21]
Yeah, no, happy to do it, especially on your podcast. It's an honor. Um, yeah, so basically the idea is I don't like unit tests as a form of verification, and I also think there's an issue with SWE-bench where all of the task instances are independent of each other.

So the moment you have the model kind of submit it, oh, it's done, you know, and, a- and that's the end of the story, end of the episode, you know? So with CodeClash, what we're thinking is let's try to really evaluate, like, long horizon development and, uh, development on a code base that is consequential and conditional upon what a model did, you know, before to that code base.

And so the general idea is you have two or more language models and they play a programming tournament, and what that means is each model maintains their own code base. In each round of the tournament, first they get to, like, edit and improve their code base however they see fit, very self-determined.

And then in the competition phase, those two code bases are, are pitted against each other. So the code bases are run and there's generally an arena. You know, we have a lot of diverse arenas, but the arenas determine, like, code base A is better than code base B, and then you kind of repeat that across multiple.

**Host** [4:33]
A- as determined by an LLM judge.

**John Yang** [4:34]
Yeah. Yeah, so LLM judge is definitely one of the mechanisms. Uh, we started with some pretty, like, simple programming games, so one of the cooler ones is, like, Halite, which, uh, Michael Trulкости-

**Host** [4:45]
Oh, yeah, I played it, uh, for Jane Street.

**John Yang** [4:47]
Yes. That's right.

**Host** [4:48]
Yeah.

**John Yang** [4:48]
That's right. You know, that's awesome. Yeah. Halite 1, 2, 3, like, Michael Trul of Cursor wrote this, uh, game.

**Host** [4:55]
Two Sigma or Jane Street?

**John Yang** [4:56]
Yes. Yes. Oh, oh, Two Sigma.

**Host** [4:57]
Two Sigma.

**John Yang** [4:57]
Two Sig- It's Two Sigma.

**Host** [4:58]
Yeah, I worked at Two Sigma. I'm like-

**John Yang** [5:00]
Oh, there you go. Yeah.

**Host** [5:01]
This is too long ago.

**John Yang** [5:02]
There you go, yeah, 2016 at this point, but we're bringing it back, you know.

**Host** [5:06]
Halite is fun. I, I, I would say if you've never done a programmatic competition where you have to control fleets of, uh, ships and attack things and defend things-

**John Yang** [5:14]
Yeah

**Host** [5:14]
... and collect resources-

**John Yang** [5:16]
Yeah

**Host** [5:16]
... it's like y- play StarCraft, but you can code, right?

**John Yang** [5:18]
Yeah, exactly.

**Host** [5:19]
Right.

**John Yang** [5:19]
Exactly. Yeah, yeah.

**Host** [5:21]
A lot of games.

**John Yang** [5:22]
Yeah.

**Host** [5:22]
Is there... Are there non-games or are you focusing games?

**John Yang** [5:25]
No, I think that's an excellent point. So for kind of the initial release for scientific purposes, we kind of used existing programming games. Uh, the current ongoing effort is, you know, to build economically valuable arenas. That's, you know, the popular word these days, so this-

**Host** [5:40]
Yeah, SuiteLancer is a big one this year.

**John Yang** [5:42]
Yeah. GD- GDPeval-

**Host** [5:43]
GDPeval

**John Yang** [5:43]
... is awesome. Yeah, just, uh, I mean, I think the big selling point of Terminal-bench and SWE-bench and these evals is that it was really close to real world utility, and so I think it's resolvable for CodeClash, and that's what we're working on.

Yeah.

**Host** [5:57]
Okay.

**John Yang** [5:57]
Yeah.

**Host** [5:58]
Um, so you're part of, uh, Ophir's group.

### Group Benchmarks

**John Yang** [6:00]
Yes.

**Host** [6:01]
Um, the other students have also been putting out a lot of other stuff. Uh, what would you highlight?

**John Yang** [6:05]
Yeah, no, I mean, Ophir is such a prolific mentor when it comes to benchmarking. SWE-efficiency I really like in the line of-

**Host** [6:11]
Yeah

**John Yang** [6:12]
... performance optimization.

**Host** [6:12]
What's the TLDR on that one?

**John Yang** [6:14]
Yeah, for sure. Um, so SWE-efficiency was wrote by this PhD student called Jeffrey Ma, who happened to be my high school classmate, and the idea there was, like, you take a code base and you just want to, you know- Do modifications that will literally make the code run faster.

So I think this is like paralyzation, SIMD operations, stuff like that. Yeah.

**Host** [6:32]
So, so no, no behavior change, just faster.

**John Yang** [6:35]
Exactly.

**Host** [6:36]
Okay.

**John Yang** [6:36]
Keep the unit test passing, but I want better runtime.

**Host** [6:39]
Okay.

**John Yang** [6:39]
Yeah, yeah. And then, and then there, there's AlgoTune that is kind of in line with that, and then there's also kind of pushing along like the scientific coding domain, uh-

**Host** [6:49]
SciCode.

**John Yang** [6:50]
Yeah, exactly.

**Host** [6:50]
Which is like-

**John Yang** [6:51]
SciCode 2 is awesome. They did like a quick-

**Host** [6:54]
And, and for, for people, SciCode is... My, my- The way I explain SciCode is it's human eval but better.

**John Yang** [6:59]
Yes, exactly. Exactly. I think, you know, there's a lot of good stuff that these days where, yeah, that's, that's the way to go.

**Host** [7:05]
Which is like SWE-bench is expensive to run. Any agentic benchmark is expensive to run.

**John Yang** [7:09]
Yeah.

**Host** [7:10]
Actually, you do need some completions benchmarks-

**John Yang** [7:12]
Yeah

**Host** [7:12]
... that just, just, just complete.

**John Yang** [7:13]
Exactly. Like, you know, you can do well on those first and then sort of graduate to the multi-turn expensive stuff. Yeah, yeah.

### 2025 Trends

**Host** [7:20]
Uh, okay. Other than that, just like broadly other work in the field in 2025-

**John Yang** [7:25]
Yeah

**Host** [7:25]
... in terms of coding evals, um, obviously we shot up Meter. They use SWE-bench, and they have a very interesting like, I guess human hours worked number.

**John Yang** [7:34]
Yeah. They like the X-axis being sort of the runtime and the- or yeah, Y-axis being the completion. You know, like we can do more long-running SWE-agent tasks. Yeah, I think the projections are, are quite interesting and I definitely appreciate them kind of using SWE-bench Verified to, to sort of proxy a lot of these things, but yeah, they're great.

Yeah.

**Host** [7:53]
Okay.

**John Yang** [7:53]
Yeah.

**Host** [7:53]
Any other work that like caught your eye this year?

**John Yang** [7:55]
That's kind of... Yeah, I mean, I, I think within the co- okay, Terminal-bench, SWE-bench, uh, yeah, Critical Point was kind of cool. Um-

**Host** [8:02]
Critical Point?

**John Yang** [8:03]
Yeah. The, it's like a very new benchmark that, uh, Ophir did. Um, and I think it's kind of related to physics. Um, there's this one called SecBench, kind of related to cyber-

**Host** [8:12]
Security

**John Yang** [8:12]
... security. Yeah, exactly. SRE-bench, which I, I think is affiliated with LOD. Like it's just cool to kind of see people really dive into different coding domains. And then stepping a little bit outside of coding, um, I personally think it's quite interesting to think about the user simulator stuff.

So like Tau-bench-

**Host** [8:29]
Vending Bench

**John Yang** [8:30]
... Tau-bench too. Yeah, and Vending Bench and-

**Host** [8:32]
I, I got mixed feelings. Uh-

**John Yang** [8:34]
Yeah. No, I'm interested.

**Host** [8:35]
Well, I mean, it's, it's like, it's like you're sampling one path. I, I don't know how realistic it is, to be honest.

**John Yang** [8:40]
Yeah.

**Host** [8:40]
It's, it's just the reality is, but it is cool.

**John Yang** [8:42]
No, for sure. Yeah, I agree. I, I think it's a good initial effort. Um, to me, I think it's super cool to see companies like, you know, I'm sure Mercora and stuff are focusing on building environments like for code, beyond code, and so I think it, it might be interesting to have like work gym style stuff.

This is stuff that my advisor, Di Yang at Stanford thinks about a lot, so yeah.

**Host** [9:03]
Yeah.

**John Yang** [9:03]
Yeah.

**Host** [9:04]
I just realized we ha- we're talking about Terminal-bench-

**John Yang** [9:06]
Yes

**Host** [9:06]
... in front of the author.

**John Yang** [9:07]
We have the author.

**Host** [9:07]
Folks.

**John Yang** [9:08]
Yeah, yeah.

**Host** [9:09]
Uh, you know, really, really, really good work, uh, just overall. Um-

**John Yang** [9:12]
Thank you.

**Host** [9:12]
Yeah.

### Tau-bench

**John Yang** [9:12]
Yeah.

**Host** [9:12]
Let's talk about Tau-bench-

**John Yang** [9:14]
Yeah

**Host** [9:14]
... uh, because you mentioned Tau-bench.

**John Yang** [9:15]
Yes, yes.

**Host** [9:16]
Uh, there's some discussion or some people are saying that Tau-bench is, uh, impossible to get a high score on-

**John Yang** [9:24]
Mm

**Host** [9:24]
... because some of the tasks are under-specified or just impossible.

**John Yang** [9:28]
Yeah.

**Host** [9:29]
I don't know if you're up to speed on that.

**John Yang** [9:30]
I'm a little bit-

**Host** [9:31]
It's a little spicy

**John Yang** [9:32]
... feed. Yeah, it's a bit spicy. I think I saw, so I, you know, for co- like I worked with Xunyu and Karthik back in Princeton very closely, and I think Karthik I just saw posted a tweet kind of, um-

**Host** [9:42]
Defending it

**John Yang** [9:43]
... yeah, like rebutting some of these claims. Um, yeah, I mean, it, it's-- I, I think I get the concern. Um, but yeah, I think it, it also brings up just maybe like interesting research problems to solve of like, okay, like why is it impossible?

Is it the ambiguity? Is it kind of the user simulator that has issues? And I think generally we all agree that, you know, we'll improve on these things over time for evals.

**Host** [10:05]
So I actually really like benchmarks that intentionally, uh, I think we should intentionally include impossible tasks-

**John Yang** [10:10]
Mm-hmm

**Host** [10:10]
... as a flag-

**John Yang** [10:11]
Yeah

**Host** [10:11]
... of like, "Hey, you're cheating."

**John Yang** [10:13]
Yes.

**Host** [10:13]
It's kind of sad that like Karthik actually is defending it because the master move would be like, "Oh yeah, you caught us." Like that, that, that was, uh, you know, like everyone reporting above 75 on Tau-bench Retail, uh, you've been cheating.

**John Yang** [10:24]
Yeah. Oh, interesting. That would be, that'd be cool. Yeah. I mean, yeah, you'll have to ask the Tau-bench authors, but yeah, no, that, that's, that's fun. Um, yeah, I, I think there was, uh, ImpossibleBench was a recent benchmark, uh, maybe from...

Was it from Anthropic? I don't know. But they basically took SWE-bench Verified and they changed the issues to make them impossible and they checked like how often the models would be like, "I actually just can't do this. I don't know what's going on."

**Host** [10:50]
Oh, like for refusals?

**John Yang** [10:51]
Yes, yes, yes. So-

**Host** [10:53]
Oh, how did they do?

**John Yang** [10:53]
I thought that was interesting. I think they're all, the models are all kind of attempting and saying like, "Oh, I did it," you know? So maybe not great on that front.

**Host** [11:00]
That's cool. But no, that, that, that's a, that's an important one.

**John Yang** [11:02]
Yeah.

**Host** [11:03]
Um, how does coding evals evolve next year?

### Future Evals

**John Yang** [11:05]
Wow, that's a great question. I mean, honestly, I think, I think it's people will make more SWE-benches. Um, I think Terminal-bench has really got something going where you, you, you ask people to... You know, with SWE-bench you're, you're confined in some sense to the domain of issues and PRs that already exist, um, wh-which I think has its benefits of being close to reality and natural.

But I think with Terminal-bench there's a lot of creativity that you can infuse into that. So I would personally be really excited, like the 2.0 job was really excellent and I'd be super excited to see, you know, 3.0, 4.0.

**Host** [11:36]
Be-because of like the environments-

**John Yang** [11:38]
Yeah

**Host** [11:38]
... angle?

**John Yang** [11:38]
I mean, the environments, you know, bringing more people into the fold. You know, I think, correct me if I'm wrong, Mike, but early on you had PhD students, very smart CS people who are adding tasks and, you know, what does that look like when you fold more coding environments for non-coding tasks, non-coding environments in general, and ask people to make stuff there.

So that's pretty cool. And then of course for myself, I think just like this long-running SWE-agent kind of thing just feels very compelling. I think the vision of like, "Hey, I tell it a goal. I don't have to be super specific about my task.

I have like a decent verifier that proxies what I want." Something literally like a code base that makes the most money in this like setting, you know? Like that's my verifier, you know? And I walk away for five hours.

The thing is just running. I'm hanging out with you, talking to my friends. I come back and it gives me like literally a soda code base on, on, on that, you know, task. I think that would be super cool.

**Host** [12:34]
Okay, I'll push back. We're part-time Incognition.

**John Yang** [12:36]
Yes.

**Host** [12:36]
And we are, uh- Emphasizing a lot of interactivity.

**John Yang** [12:40]
Mm.

**Host** [12:41]
Because the, the point is that you're going to underspecify, right?

**John Yang** [12:45]
Right.

**Host** [12:45]
And actually what people want is back and forth, back and forth and on like a really-

**John Yang** [12:49]
Mm

**Host** [12:49]
... fast timeframe, which is terrible for a benchmark author. Right?

**John Yang** [12:53]
Yes.

**Host** [12:53]
'Cause how are you gonna do that?

**John Yang** [12:54]
Yeah, yeah.

**Host** [12:55]
Uh, but, but realistic.

**John Yang** [12:57]
Yeah.

**Host** [12:57]
So, um, I, I think, like, the, this, uh, this, this is where I, I'm a little bit an- anxious or cautious about this push for long autonomy.

**John Yang** [13:05]
Right.

**Host** [13:05]
We're gon- I mean, you know, let's say this time next year we'll have... Five hours is, is pessimistic. Like-

**John Yang** [13:11]
Yeah

**Host** [13:11]
... it'll be, it'll be 24.

**John Yang** [13:12]
It'll be long. Yeah, right. Days. Yeah.

**Host** [13:13]
Um, but I don't know if that actually materially changes the industry.

**John Yang** [13:18]
Mm.

**Host** [13:18]
So we approach it, like, as in evals, you know, we have the people, people who make evals here.

**John Yang** [13:23]
Yeah.

**Host** [13:23]
We push the industry in ways that we want it to push, but I don't know if we... Like, that's a productive way because that's more of like a, a stunt that, that like, yeah, it's a proof of concept that prove- existence proof.

It can be done.

**John Yang** [13:35]
Yeah.

**Host** [13:36]
But will you use it-

**John Yang** [13:37]
At, in practice

**Host** [13:38]
... for real life?

**John Yang** [13:39]
Yeah. Yeah. I mean, honestly, um, to me, I think there's, there's potentially room for growth, so I, I would actually agree with y- your take here. Um, I mean, uh, with my lab at Stanford with D, like, there's a...

You know, her emphasis is on human AI collaboration. And so I, I definitely don't believe in this idea of just kind of getting rid of the human. Um, but yeah, maybe just, like, finding the balance of, like, you know, just because the developer ecosystem is so diverse and there's so many participants in it who want different things out of it, like just enabling different levels of abstraction.

Um, and you know, it depends on the task. Like, there's settings where you wanna be, you know, more involved and more sort of hands-on, and so you wanna use Windsurf for that. But then maybe there's kind of this general data processing thing.

It's just a lot of JSON parsing you don't really care about, and that's the one I kind of wanna walk away from and just let it figure it out.

**Host** [14:29]
Yeah.

**John Yang** [14:29]
Um, so yeah, I, I would agree with you generally.

### Call to Action

**Host** [14:31]
Yeah. Amazing. Any call to action? What, what do you want help on? How can people, uh, I guess, like, find more of your work?

**John Yang** [14:38]
Definitely. For the call to action, super jealous of all the great data that Cognition and, you know, Cursor would get. Like, that user-entered action data is, like, really fascinating. From an academic standpoint, it feels like there's two difficult approaches to resolving that.

Either you build, like, a really compelling product like LMArena that people have people use consistently, whi- which is, I mean, really tricky in and of itself, or you build, like, really good user simulators that try to mimic sort of these settings.

Uh, but that is also, like, non-trivial. I don't think it's as simple as, "Hey, ChatGPT, act like a human," right?

**Host** [15:13]
Yeah.

**John Yang** [15:13]
So it would be really cool to sort of get inspiration of like what exactly does that data look like or, or between the two, like what's the best way to scale up sort of evaluating human AI interaction. And then I think for visibility for my own work, pushing more arenas.

Like I think for, for CodeClash, what I'm excited about is the current framing is really long-running SWE agents, but you know, you could have multi agents, like two agents work together on the code base-

**Host** [15:41]
Mm

**John Yang** [15:41]
... and what happens. You have a human and an agent work on the code base versus just AIs. What happens there? You know, like when the models improve and hopefully they hill climb and they become better at digesting logs and iterating on analysis, you know, how does, how does human AI interaction like change with model capability?

Um, and so I'm kind of hoping, you know, I'm trying to inspire and, and, and convince people that the... it's a very cool test bed where you can do a lot of different sort of combinations of like human AI on different arenas, playing one arena at a time, N arenas at a time, you know.

I just, you know.

**Host** [16:17]
Yeah. I think very interested to work with you on, on the interaction stuff.

**John Yang** [16:19]
Oh, that would be awesome. Yeah.

**Host** [16:21]
Um, and then I, I think, uh, one, one more thing I'll add is for Cognition, uh, is we're gonna be pushing a lot of code-based understanding, which is kind of-

### Cognition's Push

**John Yang** [16:28]
Yeah

**Host** [16:28]
... code-based retrieval plus plus.

**John Yang** [16:29]
Yes.

**Host** [16:30]
And mostly it is helping, um, humans understand their own code bases better to enable humans-

**John Yang** [16:36]
Yeah

**Host** [16:37]
... or to, to sort of mind meld the human with the machine, uh, to do the highest possible task that LLMs cannot do alone, humans couldn't, couldn't do alone. And then the other thing is also au- like basically automatic context engineering for an LLM.

So that, that, that is like sort of like a research sub-agent, uh, that we're, that we're working on.

**John Yang** [16:55]
That's so awesome.

**Host** [16:56]
Um-

**John Yang** [16:56]
Yeah

**Host** [16:56]
... so I, I don't know what the benchmark would be 'cause like how do you, how do you benchmark understanding?

**John Yang** [17:00]
That is true.

**Host** [17:02]
Uh, apart from... I think like just, yeah, it's als- mostly like you freeze a repo, um, have some manually curated answers and then, you know, pose trivia questions. That's very easy to saturate, so I don't know how else to-

**John Yang** [17:13]
Yeah. I think, um, I, I think Silas tweeted a while ago, like sort of like the, the wiki, the code wiki stuff-

**Host** [17:18]
Yes

**John Yang** [17:18]
... set up. That's incredible. I mean, I, I use that on a daily basis.

**Host** [17:21]
Yeah. With, with, uh, Google actually just, uh, came out with their own version.

**John Yang** [17:23]
Oh.

**Host** [17:24]
Yeah.

**John Yang** [17:24]
Yeah, with the, the anti-gravity people that's-

**Host** [17:26]
Uh, no, no, no. This is like a separate-

**John Yang** [17:28]
Separate team.

**Host** [17:28]
Yeah, yeah.

**John Yang** [17:29]
Gotcha. Gotcha.

**Host** [17:29]
Um-

**John Yang** [17:29]
Yeah

**Host** [17:30]
... but cool. That's the State of Code.

**John Yang** [17:32]
Yep.

---

This library is powered by PodHood (https://podhood.com), the podcast website platform.