# [State of Evals] LMArena's $1.7B Vision — Anastasios Angelopoulos, LMArena

Latent Space · 2025-12-31

<https://addtry.com/37c6f092-ea0e-4653-acec-b0eff075212f>

Anastasios Angelopoulos, founder of LMArena (now Arena), discusses the platform's $100M raise at a $1.7B valuation, its spin-out from Berkeley incubation by a16z's Anjney Midha, and its mission to be the industry's north star for real-world AI evaluation. He defends against the 'leaderboard illusion' paper, citing factual errors and reaffirming that models cannot pay to be on or off the public leaderboard. Arena funds inference costs for millions of monthly users, with 25% of its 5M+ users in software, and is expanding into occupational verticals (medicine, legal, creative) and multimodal video arenas. Key challenges include consumer retention, which improved with sign-in and persistent history, and moving off Gradio to React for better development. Angelopoulos calls for top talent in ML, product, and go-to-market to join the high-performance team.

## Questions this episode answers

### How did LMArena begin, and what made the team decide to turn it from an academic project into a company?

Anastasios Angelopoulos explains that LMArena started as an academic project at UC Berkeley, incubated by Anjney Midha of a16z. Midha provided grants and formed an entity, giving them the option to walk away. However, the team realized that the only way to scale the platform and meet industry needs was to build a company, as academic or nonprofit structures wouldn’t provide sufficient resources.

[1:37](https://addtry.com/37c6f092-ea0e-4653-acec-b0eff075212f?t=97000)

### What was the 'leaderboard illusion' criticism of LMArena, and how did the platform respond?

Anastasios Angelopoulos describes a paper called 'Leaderboard Illusion' that accused LMArena of undisclosed private pre-release testing, claiming it created bias and favored closed-source models. He says their response pointed out factual errors—for instance, the paper alleged only 9% open-source sampling when it was actually 60/40—and noted that pre-release testing, with code names like NanoBanana, was transparent and beloved by the community.

[10:23](https://addtry.com/37c6f092-ea0e-4653-acec-b0eff075212f?t=623000)

### How does LMArena ensure its leaderboard remains fair and not influenced by model providers' payments?

Anastasios Angelopoulos states that the public leaderboard is a 'charity' and a loss leader; it will never be pay-to-play. Model providers cannot pay to be added or removed, and rankings are statistically sound, reflecting millions of real user votes. He emphasizes that platform integrity is paramount, and the leaderboard’s openness means even poorly performing models remain listed, with no option to buy a better score.

[17:11](https://addtry.com/37c6f092-ea0e-4653-acec-b0eff075212f?t=1031000)

### What new evaluation categories is LMArena expanding into?

Anastasios Angelopoulos shares that LMArena now surfaces occupational and expert categories, leveraging its user base—with single-digit percentages in fields like medicine, legal, finance, and creative marketing—to show model performance on these verticals. They are also working on multimodal arenas such as video, set to launch later this year or early next, and have already launched Code Arena for agent evaluation.

[18:36](https://addtry.com/37c6f092-ea0e-4653-acec-b0eff075212f?t=1116000)

## Key moments

- **[0:00] Intro**
- **[1:17] Spinning Out**
  - [1:21] Anjney Midha at a16z incubated the LMArena team in a Berkeley basement with grants and formed an entity before the founders committed to a startup.
  - [2:49] Anastasios Angelopoulos decided to spin Arena out as a company because academic and nonprofit paths couldn't provide the resources to scale the platform.
- **[3:36] $100M Raise**
  - [3:43] Arena's $100M raise primarily covers inference costs for millions of monthly conversations and hiring world-class talent in ML, product, and go-to-market.
  - [4:50] LMArena now has over 250 million conversations and 25% of its users work in software, making it one of the largest diverse AI evaluation platforms.
- **[6:01] Competitive Landscape**
  - [6:24] Artificial Analysis uses public benchmark aggregation, while Arena's real-user organic prompts provide a more realistic evaluation, says Angelopoulos.
- **[8:37] Leaving Gradio**
  - [8:37] Arena migrated from Gradio to React to build custom features like video loading icons and to hire from a larger developer pool.
- **[10:07] Cohere Kerfuffle**
  - [10:19] Cohere vs Arena: Arena's response highlights factual errors in the 'Leaderboard Illusion' paper's claims about pre-release testing and open-source sampling.
  - [12:17] Google's 'NanoBanana' image model debuted on LMArena as a secret code name and became a global sensation, boosting Google's market share.
- **[15:51] Platform North Star**
  - [15:52] Anastasios Angelopoulos calls LMArena's public leaderboard a charity; models cannot pay to get on or off, ensuring transparent scores from millions of votes.
  - [18:36] Arena plans to launch video evaluation later this year, and its Expert Arena already shows scores by occupation like medicine, legal, and finance.
- **[19:45] Community Secrets**
  - [21:33] Sign-in and persistent chat history were the biggest drivers of user retention on Arena, earning user loyalty every day.
- **[21:42] Join the Team**
  - [21:43] Anastasios Angelopoulos invites top talent to join Arena and suggests evaluating Cognition's Devin on Code Arena to demonstrate its agent capabilities.

## Speakers

- **Anastasios Angelopoulos** (guest)

## Topics

Evals, Startups

## Mentioned

Artificial Analysis (company), Cognition (company), Cohere (company), DeepSeek (company), Hugging Face (company), Meta (company), Sequoia (company), a16z (company), Arena (product), BFL (product), ChatGPT (product), Code Arena (product), Devin (product), Expert Arena (product), Gradio (product), Llama (product), NanoBanana (product), Next.js (product), React (product), Reeve Image (product)

## Transcript

### Intro

**Host** [0:03]
Light in space like 2025. Recaps. Light in space like 2025. All right, we're here with Anastasios from Arena. I actually don't actually know your... Stars even aids, like very-

**Anastasios Angelopoulos** [0:18]
Angelopoulos.

**Host** [0:18]
Yeah. Every-

**Anastasios Angelopoulos** [0:19]
Yeah. There you go.

**Host** [0:20]
Um, congrats on all the success. You got the Arena handle.

**Anastasios Angelopoulos** [0:23]
Yeah, we did. We got the Arena handle. Thank you. Big branding moment.

**Host** [0:27]
Uh-

**Anastasios Angelopoulos** [0:27]
Really

**Host** [0:27]
... I mean, I, I think X is, like, being more commercial, so obviously you bought it.

**Anastasios Angelopoulos** [0:31]
Yes.

**Host** [0:31]
But, like, at least you, like-

**Anastasios Angelopoulos** [0:32]
Yes

**Host** [0:32]
... have a place to go to where you can be like, "Hey," like, "we really like this." But I do think, like, dropping LM-

**Anastasios Angelopoulos** [0:39]
Yeah

**Host** [0:40]
... has changed the, the, the, the, the feel of it. I don't-

**Anastasios Angelopoulos** [0:43]
Interesting

**Host** [0:43]
... know how you feel. Yeah.

**Anastasios Angelopoulos** [0:44]
Yeah. I don't know. I mean, the reason we kept the LM at the beginning is because we were, like, we started as LMSIST, right? Out of the LMSIST sort of con- you know, uh, conglomerate at Berkeley. So we decided to-

**Host** [0:55]
You know, those language models, yeah.

**Anastasios Angelopoulos** [0:56]
Exactly.

**Host** [0:57]
So.

**Anastasios Angelopoulos** [0:57]
So we wanted to maybe broaden a little bit. Yeah.

**Host** [0:59]
Yeah.

**Anastasios Angelopoulos** [0:59]
And, and we were the first Arena, so we feel like let's kinda try to own that.

**Host** [1:03]
Yeah.

**Anastasios Angelopoulos** [1:03]
So.

**Host** [1:04]
Last time you, we had you guys on, w- you hadn't really spun out yet, and we-

**Anastasios Angelopoulos** [1:08]
Right

**Host** [1:08]
... I had a call with Alessio, and I was like, "These guys are gonna start a company." And I didn't know, I think you actually were already started at the time.

**Anastasios Angelopoulos** [1:15]
I don't, I don't remember.

**Host** [1:17]
Uh-

**Anastasios Angelopoulos** [1:17]
Maybe

### Spinning Out

**Host** [1:17]
... 'cause Anj, I had a, I, I chatted with Anj.

**Anastasios Angelopoulos** [1:21]
Mm-hmm.

**Host** [1:21]
And he said he was your founding CEO.

**Anastasios Angelopoulos** [1:24]
He was indeed.

**Host** [1:25]
Uh-

**Anastasios Angelopoulos** [1:25]
Yeah

**Host** [1:25]
... which, like, people don't know.

**Anastasios Angelopoulos** [1:27]
Yeah.

**Host** [1:27]
The, the- Anj- Anj is a very interesting character. We, we have a podcast scheduled with him.

**Anastasios Angelopoulos** [1:31]
Yeah, yeah.

**Host** [1:31]
He does a lot more than normal VCs.

**Anastasios Angelopoulos** [1:33]
He does. He's been incredible to us.

**Host** [1:35]
You, you wanna shout out some stuff that he did?

**Anastasios Angelopoulos** [1:37]
Yeah, absolutely. So you know, the way the company started was as an incubation by Anj.

**Host** [1:42]
Yeah.

**Anastasios Angelopoulos** [1:42]
So what he did is he kinda, like, found us at Berkeley and picked us out of the basement and was like, "Hey, these guys seem like they're onto something," and started working with us really early, gave u- you know, gave us some grants.

He was not, you know, a16 was not the only one to do this. We also had a great grant from Sequoia. But, um, Anj was in particular quite, quite supportive of us and, you know, gave us some resources in order to continue building out Arena before we even were committed to s- to starting a business.

And in that capacity, he sort of like, you know, formed an entity for us and this and that. And you know, he was like, "Hey, you guys can walk away at any time if you guys don't wanna start a business."

I mean, it was really incredible, very aggressive investment move by him. Right? Because, uh, of course any money that he spent-

**Host** [2:24]
This is his business, yeah

**Anastasios Angelopoulos** [2:24]
... at the end of the day, like, we could walk away and give, leave him with nothing. But I think he knew wisely that the right thing for Arena was to start a company out of it. It was the only way that we could scale and that, you know, Weiling Yan and I would, would ultimately see that and be excited about doing it ourselves, which is, which, which ended up being the case.

**Host** [2:42]
Was there a moment for you where you were, uh... I'm sure you were debating it yourself.

**Anastasios Angelopoulos** [2:46]
Mm-hmm.

**Host** [2:46]
You had other opportunities. What was the deciding factor for you?

**Anastasios Angelopoulos** [2:49]
It became clear that the only way to scale what we were building was to build a company out of it, that the world really needed something like Arena, Arena being really a place to sort of measure, understand, and, and advance the frontier AI capabilities in, on real-world users, on real-world usage based on organic feedback.

And that in order to achieve the, the scale and, you know, distribution necessary and the quality, of course, of the platform necessary to, to do this effectively, we would need to start a company out of it. You know, we considered other options.

Are we gonna keep doing this as an academic project? Are we gonna do it as a nonprofit? Blah, blah, blah. But ultimately, under those c- constructs, we didn't feel like we'd have the resources necessary to accomplish our mission.

### $100M Raise

**Host** [3:36]
So you raised 100 million or-

**Anastasios Angelopoulos** [3:38]
Yep

**Host** [3:39]
... 80? 100.

**Anastasios Angelopoulos** [3:40]
100.

**Host** [3:40]
$100 million. That's a lot of resources.

**Anastasios Angelopoulos** [3:42]
It's great.

**Host** [3:43]
What-

**Anastasios Angelopoulos** [3:43]
Yeah

**Host** [3:43]
... what's it for?

**Anastasios Angelopoulos** [3:44]
Well, you know, obviously-

**Host** [3:45]
Asking on behalf of, like, everyone-

**Anastasios Angelopoulos** [3:46]
Of course

**Host** [3:47]
... who are like-

**Anastasios Angelopoulos** [3:47]
Yeah, everybody

**Host** [3:47]
... "Dude is in Arena."

**Anastasios Angelopoulos** [3:48]
How are you gonna spend your money?

**Host** [3:50]
Yeah.

**Anastasios Angelopoulos** [3:50]
Yeah. It's great.

**Host** [3:51]
Yeah.

**Anastasios Angelopoulos** [3:51]
Yeah. I-

**Host** [3:52]
Please tell.

**Anastasios Angelopoulos** [3:52]
First of all, we don't necessarily need to spend all that money, right? The, the purpose of money at a company is to give you cards to flip. It's to say, "Hey, you, you have, uh, enough resources necessary" and it's so that if your first bet fails, you can make another bet and another bet, of course.

So that's not to say that we're gonna spend all of it. Of course, you wanna spend things responsibly. Having said that, the platform's actually quite expensive to run. We fund all of the inference on the platform. You know, the way, the way that it works is the platform-

**Host** [4:17]
Like, you pay market rates? They don't give you discounts?

**Anastasios Angelopoulos** [4:19]
No, no, we get discounts, but they're, but they are, um, standard enterprise discounts.

**Host** [4:25]
Okay.

**Anastasios Angelopoulos** [4:25]
The same that would be given to any other customer of the LLMs.

**Host** [4:28]
And if you disclose any, uh, I don't know any numbers. I, I'll just, I see numbers of votes.

**Anastasios Angelopoulos** [4:32]
Uh-huh.

**Host** [4:33]
But, like, what's that in, like, monthly tokens or... I don't know.

**Anastasios Angelopoulos** [4:36]
I don't know about tokens.

**Host** [4:37]
Whatever you see.

**Anastasios Angelopoulos** [4:37]
You know, I'd have to back, back that out. But we have, like, you know, let's say, again, this is off the cuff, but I can safely say we have more than 5 million now, 5, 6 million. We have probably 250 million conversations that happen over the course of the platform.

We're on the order of, you know, mid 10s of millions of conversations every month that are happening on the platform. Um, it's actually quite a large, you know, it's one of the largest consumer platforms for LLMs. Of course, nothing is really comparing to ChatGPT, but- You know?

**Host** [5:05]
But no, still, I mean-

**Anastasios Angelopoulos** [5:06]
I think-

**Host** [5:06]
... the, the largest scaled ones. I would-

**Anastasios Angelopoulos** [5:08]
The benefit of this is that, like, it's actually quite a diverse population, so 25% of the people on our platform, for example, do software for a living still at this scale.

**Host** [5:16]
How do you know?

**Anastasios Angelopoulos** [5:17]
Because we, like, do all sorts of th- inst- we, we either survey them or we'll, like, analyze the prompt distribution that's coming to the platform. I'd be happy also to share more. We've done something called Expert Arena, which is, like, trying to understand the distribution of experts that are coming to the platform.

**Host** [5:31]
A lot of that can be, like, unauthenticated or whatever usage.

**Anastasios Angelopoulos** [5:34]
It is, but about half of our users now are logged in.

**Host** [5:37]
Yeah.

**Anastasios Angelopoulos** [5:38]
And so we have some ability to understand them, and, and we also have surveys that we run on the platform that tell us a bit more about who are the actual users.

**Host** [5:47]
Yeah.

**Anastasios Angelopoulos** [5:47]
Of course, there's always, like, response bias in surveys, so you have to take it with a grain of salt, but nonetheless.

**Host** [5:52]
And if you don't know Anastasios' background, like he's, he's the guy to correct for response bias.

**Anastasios Angelopoulos** [5:56]
Yeah. Well, okay.

**Host** [5:57]
Yeah.

**Anastasios Angelopoulos** [5:57]
There's a lot of, there's a lot of guys like that, and girls, so.

**Host** [6:01]
Guys and girls. Um, you're not the only player. Uh, there, there are this Artificial Analysis started-

### Competitive Landscape

**Anastasios Angelopoulos** [6:05]
Yep

**Host** [6:05]
... Arena. Yup AI.

**Anastasios Angelopoulos** [6:07]
Yep.

**Host** [6:07]
It's, like, some crypto people that, uh, started this.

**Anastasios Angelopoulos** [6:10]
Yep. Yep. Yep.

**Host** [6:10]
I don't know if you have a conversation with them on like, "Well, hey, this is our thing," or like, "Let's work together on something." I, I, I don't know if you've, uh, what the initial-

**Anastasios Angelopoulos** [6:17]
No

**Host** [6:17]
... was like.

**Anastasios Angelopoulos** [6:17]
You know, so I've talked to the Artificial... I've actually talked to both groups.

**Host** [6:21]
Yeah.

**Anastasios Angelopoulos** [6:21]
Both seem, you know, great people.

**Host** [6:22]
Am, am I missing any major players? It, it's just those two, right?

**Anastasios Angelopoulos** [6:24]
No.

**Host** [6:24]
Okay.

**Anastasios Angelopoulos** [6:24]
I think those, I don't know Yeah, I don't think so.

**Host** [6:27]
Okay.

**Anastasios Angelopoulos** [6:28]
I think those are, like, some of course large, depending on how you define the term.

**Host** [6:32]
Yeah.

**Anastasios Angelopoulos** [6:33]
Artificial analysis obviously has, like, huge, like, market mind share-

**Host** [6:37]
Yeah

**Anastasios Angelopoulos** [6:37]
... um, around the analysis of, of different AI systems.

**Host** [6:42]
Yeah, they told me they were just like, "We, we are gonna be Gartner of AI."

**Anastasios Angelopoulos** [6:44]
Yeah. That's, that's kind of their goal.

**Host** [6:46]
Right.

**Anastasios Angelopoulos** [6:46]
And I think they're going after that consulting market and so on.

**Host** [6:48]
Yeah.

**Anastasios Angelopoulos** [6:49]
But, uh, Artificial Analysis from what I understand is, like, a team of consultants that are doing this.

**Host** [6:51]
Yeah.

**Anastasios Angelopoulos** [6:52]
Seem like really, really nice guys, um, going after that, that particular market. It's different from our platform in the sense that their analysis is based on sort of, like, aggregating public benchmarks and turning those into analytics.

**Host** [7:08]
I- independently rerunning.

**Anastasios Angelopoulos** [7:10]
And independently rerunning.

**Host** [7:11]
Which matters.

**Anastasios Angelopoulos** [7:11]
Yeah, yeah, yeah. Which matters, yes. Um, and, and, and, and using those in order to compile sort of reports and so on that, that educate the field on the performance of all these different models.

**Host** [7:22]
But they also have arenas.

**Anastasios Angelopoulos** [7:24]
They have arenas, but the arenas are not based on organic usage. Like, the, the thing that distinguishes our platform versus theirs is that the users are actually inputting their own use case. They're actually asking their own question, and that gives a, a level of realism-

**Host** [7:40]
Yeah

**Anastasios Angelopoulos** [7:40]
... that platform doesn't have.

**Host** [7:41]
Yeah.

**Anastasios Angelopoulos** [7:41]
Of course, they ha- they specialize in a slightly different thing, but I see those platforms kind of diverging in that sense.

**Host** [7:46]
Yeah.

**Anastasios Angelopoulos** [7:47]
Um-

**Host** [7:47]
Yeah. A- and sometimes it's like, it's the only way to do this. So like for example, for AA, their video arena is pre-generated videos. You can't enter in your own video prompts.

**Anastasios Angelopoulos** [7:56]
That's correct, but we're doing it organically.

**Host** [7:57]
Yeah.

**Anastasios Angelopoulos** [7:58]
Like-

**Host** [7:58]
Exactly

**Anastasios Angelopoulos** [7:58]
... and so I think-

**Host** [7:59]
But, uh, as a voter, it does help in terms of I don't have to wait.

**Anastasios Angelopoulos** [8:04]
It does-

**Host** [8:04]
You know

**Anastasios Angelopoulos** [8:04]
... but also, why would you go? Do you actually care about, like, other people's videos?

**Host** [8:09]
To, to inform your own intuition.

**Anastasios Angelopoulos** [8:11]
Maybe, yeah.

**Host** [8:11]
Yes.

**Anastasios Angelopoulos** [8:11]
Maybe you're, like, interested in comparing-

**Host** [8:13]
Like, I'm a shitty prompter, right?

**Anastasios Angelopoulos** [8:15]
Are you?

**Host** [8:15]
So-

**Anastasios Angelopoulos** [8:15]
I don't believe that.

**Host** [8:16]
I'm a, a terrible prompter.

**Anastasios Angelopoulos** [8:17]
No, no, no.

**Host** [8:17]
I learn by example.

**Anastasios Angelopoulos** [8:18]
Don't denigrate yourself. Don't denigrate yourself.

**Host** [8:20]
Um- There, there, there are many prompters much better than me. Let's say that, right? That's, that's, that's a fact.

**Anastasios Angelopoulos** [8:25]
People have all sorts of cool ways of prompting LLMs.

**Host** [8:27]
So, yeah, yeah.

**Anastasios Angelopoulos** [8:27]
Yeah, it's educational to see.

**Host** [8:28]
The, the only way to learn is by, like, well, look at other people's prompts and see the results and go like, "Oh, I didn't know you could do that," and that's how I-

**Anastasios Angelopoulos** [8:35]
Totally.

**Host** [8:36]
Yeah.

**Anastasios Angelopoulos** [8:36]
Yeah, yeah, yeah.

**Host** [8:37]
Okay. So let's come back to arena. Oh, one thing I do wanna say is the number one use of funds is getting off Gradio. Uh-

### Leaving Gradio

**Anastasios Angelopoulos** [8:43]
Oh, yeah. Yes. We... Well, listen. Gradio, incredible platform. Gradio scaled us to a million MAL.

**Host** [8:50]
Yeah.

**Anastasios Angelopoulos** [8:50]
That's incredible. And of course-

**Host** [8:52]
Did you tell the Hugging Face folks that?

**Anastasios Angelopoulos** [8:53]
Of course.

**Host** [8:54]
Yeah.

**Anastasios Angelopoulos** [8:54]
Yeah. We're really, really grateful to Gradio for taking us so far. Eventually, you know, it became time for us to move off of that and-

**Host** [9:01]
I mean, it's-

**Anastasios Angelopoulos** [9:01]
... be able to react sort of-

**Host** [9:02]
I, I'm sure Hugging Face would've loved you to stay on.

**Anastasios Angelopoulos** [9:04]
They would have, I'm sure.

**Host** [9:05]
Um-

**Anastasios Angelopoulos** [9:06]
Yeah, yeah.

**Host** [9:06]
Was there a technical, like, reason you, you, you just couldn't get the performance?

**Anastasios Angelopoulos** [9:09]
Yeah. It just became hard to develop, and there were all these tools that we wanted in React and, you know, to do all the fancy things that you can do in React became kind of like 100 times-

**Host** [9:19]
Give us one example of that. I don't know. What, what's a feature that you really wanted?

**Anastasios Angelopoulos** [9:22]
Let's say we wanted to create, like, our own custom, like, loading icons for video with notifications.

**Host** [9:27]
Okay.

**Anastasios Angelopoulos** [9:28]
How are we gonna do that in, in React? It's hard.

**Host** [9:30]
Uh, yeah. I mean, we'll make a-

**Anastasios Angelopoulos** [9:32]
I'm sure the-

**Host** [9:32]
... a custom component in React

**Anastasios Angelopoulos** [9:33]
... I'm sure the Hugging Face guys are gonna come in and say like, "Hey, you can do that in Gradio"-

**Host** [9:37]
Yeah

**Anastasios Angelopoulos** [9:37]
... which maybe you can but-

**Host** [9:38]
Of course they're gonna say that

**Anastasios Angelopoulos** [9:39]
... also fewer developers know. How are we gonna hire for that? We have to reskill them. The people are less familiar with that stack. It would-

**Host** [9:44]
Yeah.

**Anastasios Angelopoulos** [9:45]
You know. Anyway.

**Host** [9:46]
So it's full React Next.js, all this, all this-

**Anastasios Angelopoulos** [9:48]
Yes. Yeah, yeah, all that.

**Host** [9:49]
Okay, cool. Are there use of funds that are, that might, that might be interesting? Like, uh, you know, basically-

**Anastasios Angelopoulos** [9:53]
No, that's basically it

**Host** [9:54]
... how you deploy the resources. Okay.

**Anastasios Angelopoulos** [9:55]
Yeah. It's, it's, it's on primarily inference that funds the free usage of the platform and, and then also, uh, uh, hiring of course, headcount.

**Host** [10:05]
Yeah.

**Anastasios Angelopoulos** [10:05]
We have an office, you know.

**Host** [10:07]
That's us.

### Cohere Kerfuffle

**Anastasios Angelopoulos** [10:08]
It's in SF.

**Host** [10:09]
I'll tackle one of the major things this year, which I'm sure you're tired of thinking about, but I-- for people who are not in the loop, this is, uh, gonna be news to them, uh, the leaderboard illusion, the whole thing with Cohere.

Let's summarize the, what they said and then your response.

**Anastasios Angelopoulos** [10:23]
Mm-hmm. So leaderboard illusion's a paper that critiques LMArena, and the main-

**Host** [10:28]
Pretty, like-

**Anastasios Angelopoulos** [10:29]
Yeah

**Host** [10:29]
... brutally .

**Anastasios Angelopoulos** [10:30]
Well, you know, I would say unscientifically. Um-

**Host** [10:34]
And let's be clear, Cohere wasn't doing that well on the leaderboard.

**Anastasios Angelopoulos** [10:36]
I think, uh, Cohere was, like, 74. It's all good. You know, it's actually not... It's a, it's a respectable place that they had on the leaderboard. I don't even think it was really Cohere people, like the Cohere model developers doing this.

It was more their research side. But in any case, what does the, what does the leaderboard illusion say? It says that LMArena was what... Their-- the claim is that LMArena was doing this undisclosed, quote-unquote, private testing on our platform, that model providers will send us pre-release models, and we'll expose them and so on and so forth, and that this creates so-called inequities in the leaderboard, you know, due to that pre-release testing.

For example, they will s- they cited that Meta at some point tested-

**Host** [11:15]
Llama 4

**Anastasios Angelopoulos** [11:15]
... some amount of models with us.

**Host** [11:17]
Yeah.

**Anastasios Angelopoulos** [11:17]
Of course, we can't disclose all of the details of how all that was done, but that is, that is the main claim of the paper. Now, our response to that paper, it's online. You can find it on Response to Leaderboard Illusion, and our response to that paper is essentially pointing out a series of factual mistakes in the paper that, that question the validity of the claims.

So you can go look at the first version of the paper on Archive yourself, and you'll see the claims. I think most scientists would view that as, um-

**Host** [11:43]
Oh, they've corrected it.

**Anastasios Angelopoulos** [11:44]
They've cor-... Of course-

**Host** [11:45]
Oh

**Anastasios Angelopoulos** [11:45]
... because we... I mean, but they didn't correct everything. They just corrected-

**Host** [11:48]
Okay

**Anastasios Angelopoulos** [11:48]
... some aspects that were just blatantly unscientific and false. But you know, they, for example, um, said that we were, that we only sampled, like, 9% open source models and, like, you know, 60%, like, closed source models, and this created an gap between open and closed source.

But rea- in reality, we're actually really supportive of open source models, and, uh, it was more like 60/40. And, uh, so that was, you know, one of the examples of, of a, of an er- a c- an error in the claims.

Another example is that they were claiming that there was some sort of bias introduced by this pre-release testing, and that it was undisclosed. In reality, as you probably know, we've been doing this pre-release testing for a long time.

Our community loves it. They loved basically getting like-

**Host** [12:28]
The secret code names

**Anastasios Angelopoulos** [12:29]
... yeah, the secret code-

**Host** [12:30]
And it's like-

**Anastasios Angelopoulos** [12:30]
Like NanoBanana.

**Host** [12:31]
Yeah.

**Anastasios Angelopoulos** [12:31]
All that. So NanoBanana, by the way-

**Host** [12:33]
It started on you

**Anastasios Angelopoulos** [12:34]
... started on us.

**Host** [12:34]
Yeah, yeah.

**Anastasios Angelopoulos** [12:35]
Right? And people loved it. Went like global sensation.

**Host** [12:37]
Yeah.

**Anastasios Angelopoulos** [12:37]
Like non- non-zero fraction of the global population using NanoBanana.

**Host** [12:39]
Did you talk to Laina about naming it Banana or was that-

**Anastasios Angelopoulos** [12:42]
No

**Host** [12:43]
... her decision?

**Anastasios Angelopoulos** [12:43]
It was, it was their decision, I believe.

**Host** [12:45]
Okay.

**Anastasios Angelopoulos** [12:45]
But it was sort of this randomly generated thing.

**Host** [12:47]
Ah.

**Anastasios Angelopoulos** [12:47]
And it just went-

**Host** [12:49]
No, no, so apparently, uh, Naina, who's a PM-

**Anastasios Angelopoulos** [12:51]
Yeah, yeah

**Host** [12:52]
... is named after her because her nickname's Naina.

**Anastasios Angelopoulos** [12:54]
Oh. Oh, that's sweet.

**Host** [12:55]
Yeah.

**Anastasios Angelopoulos** [12:55]
I didn't know that.

**Host** [12:55]
It was Naina-

**Anastasios Angelopoulos** [12:56]
Yeah, yeah

**Host** [12:56]
... put Banana in it, yeah.

**Anastasios Angelopoulos** [12:57]
Oh, that's sweet.

**Host** [12:58]
And that's the whole thing.

**Anastasios Angelopoulos** [12:58]
Yeah, I didn't know that, didn't know the origin story. Yeah, I mean, to us it just looked like sort of a random thing. And then it went-

**Host** [13:03]
But it went like clearly heads and shoulders above-

**Anastasios Angelopoulos** [13:06]
It's huge

**Host** [13:06]
... which, like, before that, there was Reeve Image, remember?

**Anastasios Angelopoulos** [13:09]
Yeah, I do. Of course, yeah.

**Host** [13:11]
And also, uh, BFL and, uh, all those AI models.

**Anastasios Angelopoulos** [13:13]
Yeah, I mean, all those models are also great.

**Host** [13:15]
Yeah.

**Anastasios Angelopoulos** [13:15]
Um, and I think those teams are also improving quite quickly.

**Host** [13:18]
Yeah, yeah.

**Anastasios Angelopoulos** [13:18]
But NanoBanana was a sensation.

**Host** [13:20]
Yeah.

**Anastasios Angelopoulos** [13:20]
I mean, m- that moment alone changed Google's, like-

**Host** [13:24]
Roadmap

**Anastasios Angelopoulos** [13:25]
... yeah, market share.

**Host** [13:26]
Yeah.

**Anastasios Angelopoulos** [13:27]
Seriously.

**Host** [13:28]
Yeah, yeah.

**Anastasios Angelopoulos** [13:28]
I mean, Google's stock, billions of dollars-

**Host** [13:31]
Yeah

**Anastasios Angelopoulos** [13:31]
... are moving because of NanoBanana.

**Host** [13:32]
And now there's like an OpenAI Code Red and everything.

**Anastasios Angelopoulos** [13:35]
I, I-

**Host** [13:35]
Yeah

**Anastasios Angelopoulos** [13:35]
... I don't, I don't know about that, but yes-

**Host** [13:37]
Yeah

**Anastasios Angelopoulos** [13:37]
... the Information reported this.

**Host** [13:38]
I, I would say, like, the, uh, image generation, I would say, has been like this weird part of AI in- overall, 'cause it's not strictly AGI-critical. Like, it's not reasoning. It's, it's, it's, it's, it's not like feeding more context into the model.

It is the model generating a visual, uh, representation. And so it's basically like, uh, I, I always think like, well, you know, Gemini used to get a lot of complaints for generating like racist, racist images or whatever.

**Anastasios Angelopoulos** [14:06]
Mm-hmm.

**Host** [14:07]
Um-

**Anastasios Angelopoulos** [14:07]
Yeah, that was a hilarious moment

**Host** [14:08]
... and, and ChatGPT also had, had it i- in the past. And I'm like, "Well, can we just get rid of this? Like, do we have to do image generation?"

**Anastasios Angelopoulos** [14:15]
Mm-hmm.

**Host** [14:15]
Because let's just focus the, the positive reputation of AI in general on language models and coding and, you know, the other stuff.

**Anastasios Angelopoulos** [14:22]
Yeah.

**Host** [14:23]
Um, but I, I, I, I'm wrong. I'm- I'm wrong. I'm such a huge NanoBanana Pro show.

**Anastasios Angelopoulos** [14:28]
Yeah, I-

**Host** [14:29]
Uh

**Anastasios Angelopoulos** [14:29]
... totally agree.

**Host** [14:29]
It's good.

**Anastasios Angelopoulos** [14:30]
I was also kinda wrong about this. I didn't see the positive benefits. But actually I think that these, like, multimodal models are gonna become some of the most economically valuable aspects of AI.

**Host** [14:39]
Yeah. AC.

**Anastasios Angelopoulos** [14:39]
Both in, both in consumer and also in enterprise, because one of the fastest-growing segments, market segments in AI adoption is marketing and in, in market- marketing and design.

**Host** [14:50]
Yeah, ads and-

**Anastasios Angelopoulos** [14:51]
Yeah.

**Host** [14:51]
And so, so, uh, I'm a content creator, right?

**Anastasios Angelopoulos** [14:53]
Yeah, of course.

**Host** [14:53]
I need-

**Anastasios Angelopoulos** [14:54]
I'm sure you're using it all the time

**Host** [14:55]
... infinite supplies of diagrams and explainers and-

**Anastasios Angelopoulos** [14:57]
Totally

**Host** [14:58]
... infographics.

**Anastasios Angelopoulos** [14:59]
Yeah. Soon we're not gonna be even making the product papers.

**Host** [15:01]
YouTube thumbnails, yeah.

**Anastasios Angelopoulos** [15:02]
We're- they're just gonna be our paper figures are gonna be made by AI.

**Host** [15:04]
Actually, yes, yes. I, I do think that it actually one-shots. So, so I, uh, DeepSeek came out with, uh, V3.2 recently. I took their explanations, which are very wordy. Uh, they like very concise papers. It's 23 pages long, but it's very dense.

**Anastasios Angelopoulos** [15:16]
Yeah.

**Host** [15:17]
And so I just took their explanations of the RL environment stuff, and I fed it into NanoBanana Pro, and it fe- spit an image that I he- I used to understand the paper better. And the fact that I can just casually generate like a paper-quality diagram that would usually take a PhD student like a month to, in like in Photoshop or something to do is, is incredible.

**Anastasios Angelopoulos** [15:36]
Yeah.

**Host** [15:36]
Yeah.

**Anastasios Angelopoulos** [15:36]
It's incredible. It is amazing.

**Host** [15:38]
I wanna ask about your principles running Arena. I think you, you manage a giant community, 5 million mile. What have you decided are the core principles, I guess, before becoming a company and now that you're a company? I don't know if there's anything that's changed for you.

### Platform North Star

**Anastasios Angelopoulos** [15:52]
I don't think anything has really changed. We want to provide the North Star of the industry and center the use cases of real users, foreground those, so that people know what to target. The goal is to create a benchmark that is constantly fresh, that does not suffer over-fitting because of the fact that we constantly have new data points coming in, that tracks the, you know, all the different new models, um, all the different new use cases of AI, and, um, gives the whole world sort of ground truth, uh, for how real users are using these models and how, how good they are on those use cases.

We continue to do quite a few open source data releases. We've probably released more data than y- basically anybody on the real-world use cases of AI, millions and millions of conversations, real-world conversations from real users that the community's using to study and, and improve on.

**Host** [16:47]
Yeah. And then I, I think, like, in terms of what you will build versus what we'll not build-

**Anastasios Angelopoulos** [16:53]
Mm-hmm

**Host** [16:54]
... I guess I, I'm not necessarily caught up on e- everything that you've launched. I, I know recently you've done the, the Dev or Code Arena.

**Anastasios Angelopoulos** [17:00]
Yeah, Code Arena.

**Host** [17:00]
That's, that's the most recent one, right?

**Anastasios Angelopoulos** [17:02]
Code Arena, Expert Arena.

**Host** [17:02]
Yeah, Expert Arena. So basically like what, what is in the critical path for you, let's say i- in- for next year, and what, what have you decided you'll never do?

**Anastasios Angelopoulos** [17:11]
So let me first talk about things that I'll never do. The platform, integrity comes first to the platform. The, basically the public leaderboard that we show on LMArena, I think of as a charity. It's a loss leader for us.

We don't really make money on the public leaderboard. You can't pay to get on the public leaderboard. It's not like a Gartner in that sense.

**Host** [17:31]
Mm.

**Anastasios Angelopoulos** [17:31]
It's not like any of these like, uh, you know-

**Host** [17:33]
Pay-to-play

**Anastasios Angelopoulos** [17:34]
... pay-to-play systems. Never going to be like that. Models are gonna be listed on the leaderboard whether or not the providers pay and whether or not they're getting a good score. They can't pay to take it off either.

And so what that means-

**Host** [17:47]
That's very important

**Anastasios Angelopoulos** [17:47]
... that- that's very important. And so what that means is that the leaderboard ha- has a certain integrity that will never be compromised, of course.

**Host** [17:53]
Yeah, but, but not all preview models will make it onto the-

**Anastasios Angelopoulos** [17:56]
No, but that's okay

**Host** [17:56]
... that's right

**Anastasios Angelopoulos** [17:56]
... those preview models have never been released.

**Host** [17:58]
Yeah, yeah.

**Anastasios Angelopoulos** [17:59]
Right? Who, who cares about putting unreleased models on a, on the leaderboard? The point is that for every released model, the score that you see on the leaderboard is statistically sound. It reflects the real-world capabilities of the model.

**Host** [18:11]
Yeah.

**Anastasios Angelopoulos** [18:11]
Why? Because millions of people from around the world have voted for it, and that's where that, where that number comes from. All we do to con- compute that number is millions of people are voting, we take those votes, we turn them into a number.

That's always going to remain, you know, a transparent and fair reflection of model performance. Where are we going? Lots of different new categories. I don't know if you recently saw, we expressed- we, um, we exposed, uh, occupational and expert categories.

So now, single-digit percentage of our user base, we're millions, millions to tens of millions of users, right? So single digit percentages means a lot. Single-digit percentage of our user base are in medicine, in legal, in business, you know, finance, accounting, creative, marketing, stuff like this.

And we're able to show the performance of these models on all these different verticals because we have all these users in our, in our user base. Uh, and we're, we're working more towards multimodal, you know, video we're soon to launch on the site at some point, you know, later this year or early next.

So lots of things in the pipeline.

**Host** [19:08]
Amazing. Would you expose an API?

**Anastasios Angelopoulos** [19:12]
We've thought about it, yeah. I think it's a, it's a possibility.

**Host** [19:15]
I hope so.

**Anastasios Angelopoulos** [19:15]
Yeah.

**Host** [19:15]
What, what, what are the counter arguments? Like, why, why not?

**Anastasios Angelopoulos** [19:18]
Well, there's obviously a need for an API. The question is more of focus of our company. Just because we're a startup, and so we really should be doing one thing well.

**Host** [19:31]
Arenas.

**Anastasios Angelopoulos** [19:32]
Yeah, arenas.

**Host** [19:33]
Yeah.

**Anastasios Angelopoulos** [19:33]
So I'm not, I'm not sure that we- how far we wanna sort of splay out and on what timeline we'd wanna do that.

**Host** [19:38]
Yeah. Any other sort of like, um, community management tips? You know, m- more broadly, like every AI company like really wants to grow their community. You're obviously-

### Community Secrets

**Anastasios Angelopoulos** [19:47]
Mm-hmm

**Host** [19:47]
... one of the strongest in the world. What's really wo- really worked?

**Anastasios Angelopoulos** [19:51]
Well, so first of all, I want to give a shout-out to our community manager, Greg, who is doing an awesome job managing our community, whether that's on Discord, um, or on LMArena. He's really incredible. So I would say hire Greg, but don't-

**Host** [20:04]
Don't hire Greg

**Anastasios Angelopoulos** [20:04]
... hire Greg.

**Host** [20:04]
Don't hire Greg.

**Anastasios Angelopoulos** [20:05]
Don't hire Greg.

**Host** [20:06]
Find a Greg.

**Anastasios Angelopoulos** [20:06]
He's ours.

**Host** [20:07]
Find...

**Anastasios Angelopoulos** [20:07]
Find a Greg.

**Host** [20:07]
Find a Greg.

**Anastasios Angelopoulos** [20:08]
But in general, you know, the question of how do you get to so many users, that is a tough question.

**Host** [20:13]
And keep. And keep.

**Anastasios Angelopoulos** [20:14]
And keep and retain them. That is a tough question because consumer is one of the hardest markets in the world. There's a lot of websites in the world that people can go to, you know. Why should they go to yours?

And the reality is if you want to create a really dominant product, you have to provide people value. And to be frank, I don't think we're all the way there yet. It's not like I have the solution and answer for how to build a, a great consumer product.

If I did, we wouldn't be at 10- tens of millions of users, we'd be at hundreds or, you know, uh, we'd be at a billion users.

**Host** [20:42]
Yeah.

**Anastasios Angelopoulos** [20:42]
We'd be like-

**Host** [20:43]
Is there a world you, like, are bigger than ChatGPT or...

**Anastasios Angelopoulos** [20:46]
I don't know. I don't, I don't know that we need to be.

**Host** [20:49]
Yeah.

**Anastasios Angelopoulos** [20:50]
And I don't know that we ever will be-

**Host** [20:51]
Yeah

**Anastasios Angelopoulos** [20:51]
... because that's a, that's an extraordinary generational product that they built, right? It took a lot of time and, and to some extent, it also involved luck. There's a lot of lightning in the bottle moments, like NanoBruno was for us, where our user base just like goes up by a lot.

But when those users come, they can just as easily leave. So the way I think about it is every user is earned. You have to earn them every single day. They can leave at any moment. They're fickle. And so all the time you have to be thinking about, how do I provide this person value?

Learning, how are they using my website? What more could I give them? And how do I build in all the retention mechanisms so that they stay, and then they're also bringing their friends.

**Host** [21:29]
Is there one that's working in terms of retention? Like you said, a lot of people are signing in now, and that's new.

**Anastasios Angelopoulos** [21:33]
Yeah, sign-in was a big driver of retention.

**Host** [21:35]
No, no, but what, what did you give them in order to encourage them to sign in?

**Anastasios Angelopoulos** [21:38]
Oh, like history. Persistent history.

**Host** [21:39]
That's, that's it. That's enough.

**Anastasios Angelopoulos** [21:40]
Yeah. That's, that's one thing that has had a big impact.

### Join the Team

**Host** [21:43]
Okay. Yeah, cool. What do you want from people? What, uh, what are you looking for help on, like any call to action?

**Anastasios Angelopoulos** [21:49]
Yeah, I, uh, we are always looking for people to come and join us. If you are one of the best people in the world in your area, whether that's consumer product, whether that is machine learning, whether that is, you know, B2B, go to market, marketing, all these things, we need you at Arena.

We're building like a high-performance team of real experts in everything that they do and, you know, I'm always looking for excellent people to work with.

**Host** [22:12]
Do you need like... What about partnerships, right? Like, let's say I'm in Cognition, I wanna partner with LMArena or just Arena. What works for you? What existing partnerships do you already have that, that's really fruitful?

**Anastasios Angelopoulos** [22:25]
Yeah. So I mean, we of course, partner with all of the major model labs.

**Host** [22:29]
Yeah.

**Anastasios Angelopoulos** [22:30]
Um-

**Host** [22:30]
And that's, that's just straightforward, like, "Hey, we have a new model here. Here you go."

**Anastasios Angelopoulos** [22:32]
Yep, exactly. So I think the, the most straightforward thing would be for someone like Cognition, it's like let's evaluate Devin.

**Host** [22:38]
But that's, that's an agent.

**Anastasios Angelopoulos** [22:39]
Yeah. But we should be continuing to shape-

**Host** [22:42]
You know-

**Anastasios Angelopoulos** [22:42]
... our... Well, Code Arena's an agent evaluation.

**Host** [22:44]
That's true. That's true. That's true.

**Anastasios Angelopoulos** [22:44]
And in fact, I think-

**Host** [22:45]
But it's more focused on... Like, all these a- arenas tend to be focused on the model rather than the harness, so.

**Anastasios Angelopoulos** [22:51]
But that maybe should change.

**Host** [22:53]
Maybe it should change, yeah.

**Anastasios Angelopoulos** [22:53]
Maybe we should be evolving towards that direction, and I think the Code Arena is a good example of an arena that would support a full-featured harness like a Devin.

**Host** [23:01]
Yeah.

**Anastasios Angelopoulos** [23:02]
And so in my view, I'm-- if, if I'm talking to Cognition, I'm saying, "Hey, let's get Devin on the arena and figure out how to l- you know, loop together the Devin harness-

**Host** [23:11]
Yeah, I think that's-

**Anastasios Angelopoulos** [23:11]
... so that we can..."

**Host** [23:12]
Yeah.

**Anastasios Angelopoulos** [23:12]
I'm sure, I'm sure that there's something that could be really valuable there, especially given Devin. Last week, people talking about Devin dead. Did you see that?

**Host** [23:19]
Yeah.

**Anastasios Angelopoulos** [23:19]
People were saying, "Devin's gone." Devin's not gone.

**Host** [23:21]
It's not gone.

**Anastasios Angelopoulos** [23:21]
Devin's everywhere.

**Host** [23:22]
It's doing very, it's doing very well. Uh-huh.

**Anastasios Angelopoulos** [23:23]
But people... So can we highlight that for people and show them, hey, Devin is actually the best or one of the best in the world at doing what it does. LMArena can actually do that.

**Host** [23:32]
Yeah.

**Anastasios Angelopoulos** [23:32]
And our, our, our place as a central evaluation platform allows, allows that to happen.

**Host** [23:36]
Yep. Love it. All right. Thank you for owning the State of Evals.

**Anastasios Angelopoulos** [23:40]
Thanks so much.

**Host** [23:41]
And for congrats on a wonderful year.

**Anastasios Angelopoulos** [23:42]
Appreciate it. Congrats to you too.

**Host** [23:43]
Thank you.

**Anastasios Angelopoulos** [23:43]
Congrats on all the growing, you know-

**Host** [23:45]
Yeah

**Anastasios Angelopoulos** [23:45]
... momentum in your podcast and in your career.

**Host** [23:48]
Thank you.

**Anastasios Angelopoulos** [23:48]
It's really impressive to see.

---

This library is powered by PodHood (https://podhood.com), the podcast website platform.