[Paper Club] Berkeley Function Calling Paper Club! — Sam Julien, Writer

All right. Hello, everybody. Today, we're going to be talking about the Berkeley Function Calling Leading- Leaderboard, or BFCL. I'm Sam Julien. If you don't know me, I'm in the Discord, I'm around. In my day job, I lead developer relations for Writer, which is an enterprise AI company.

Then I also write at samjulien.com and have a, a newsletter and, and things like that. So, uh, BFCL, it is the Berkeley Function Calling Leaderboard that, uh, we've come to know and love recently. And, uh, it basically ranks models according to how well they do function calling, a- also known as tool calling, depending on who you are.

Uh, and they evaluate on, on different things and post these scores, and, uh, and it's great. And so, um, so the main thing is the leaderboard, but behind the leaderboard, there's also these three, um, releases, three blog articles that they've come out with.

Um, and I can't ... I'm gonna open the chat to the side here so I can see. Um, and w- when I was preparing for this, I didn't realize how quick all of these ... Like, this is all super recent.

The first blog article was in March. Uh, second version was in August, and third version was in September, which, uh, was a little mind-blowing to me. Uh, they've done a lot in a very short amount of time. Uh, and then I listed out the folks who are on the team.

Some of them, uh, you'll notice from other projects. I think Shishir has been on a, in a lot of different, uh, avenues. And then they also have a Discord server if you're super interested in chatting with these folks.

They have a, an LLM, Gorilla LLM Discord. So first up, we're just gonna walk through sort of the three, um, blog articles and just kind of go over what's in each of them. Uh, the third one we'll have a little bit more time, uh, since it's the most recent and also the most, um, uh, kind of the, the biggest evolution as they've been doing it.

So the first one that came out, like I said, in March, um, was really the first of its kind to, um, uh, make a function calling leaderboard. So it's, it's ... The pur- the purpose of it is to evaluate the LLM's ability to call functions and provide a, uh, comprehensive benchmark for function calling capabilities.

Oh, and I see that I need to make Vibhu co-host, so let me see if I can do that really quick. If I can look at the participants,

and

make Vibhu ... Oops. Person.

There we go.

Okay. Awesome. Done and done. Okay. So v1, uh, basically was just the starting point, uh, but a pretty solid starting point. They ha- they had, um, uh, a, a pretty diverse, uh, data set, 2,000 question function answer pairs across multiple languages and domains, uh, and includes simple, multiple parallel, and parallel multiple function calling scenarios.

And so the first version of the leaderboard looked like this. Um, not terribly different than what we see today, but there are some, some new things that you'll see as we evolve. The initial data set composition's really interesting because it's very, um, code oriented.

A lot of the, um, this initial set of function calling was very focused on specific, like, language tasks, um, across Python and other languages. And then the main, like, I, I think sort of the main innovation of this, um, first version of the leaderboard was that it used this, it introduced this, like, abstr- abstract syntax tree evaluation where it would, um, look at the executable functions and evaluate them.

And they have this nice flowchart in here. Uh, it's pretty interesting how they created this, where it will evaluate, um, the f- the, the functions and the parameters and see if it matches, uh, what was expected and then, uh, kind of run it through these different scenarios.

And they go into more detail, uh, in the actual, um, in the actual blog article of how they actually parse through the functions with, with abstract syntax tree navigation and match the parameters and that kind of thing. So if you wanna dig deeper into that, you can go through there.

So that was sort of the first version. Like I said, it was only s- six or seven or eight months ago, so not that long ago. And then with the second version they were really focused on, uh, improving the, the data set.

So they came out with this second version titled Live Dataset, and they were really focused on getting more real world scenarios, like, uh, with live user contributed data, having more rare use cases, like having lots and lots of nested parameters.

And then, uh, fixing some of the issues from the first version, just figuring out some areas of, like, data contamination and bias and, and things like that. So they, they spent a lot of time ... Like, most of the second, uh, article is just about all of the different issues that they've, that they've worked on with the data pre-processing, filtering, quality improvement, that kind of thing.

And I'll share these slides in the, in the Discord as well, so, uh, you don't have to worry about taking notes. Uh, and they ended up with a data set of 2,251 question function answer pairs. Uh, so that they could address some of these issues.

So they did a lot of like de-duplication using Rouge scores. Uh, they did a bunch of filtering, uh, and just like standardized the function documents to better fit their format and enhance their user prompting.

Like somebody needs to mute. Um, so then if you look at the, the, the, the way they display the dataset composition in the second version, you can see that it's a lot less focused on specific coding languages and things like that, and more about what they were trying to accomplish with it.

So like relevance detection, irrelevance detection, um, a-and ca-- maintaining some of these things with the, the abstract syntax tree and parallel multiple function calling and things like that. So a lot of the sort of like underlying methodology of how they did the benchmarking didn't really change in the second version so much as the, um, the data process.

So in that second version, they have this nice, uh, another nice visual. I really like how m- how many visuals they include in these, in these blog articles. So they have this one for how they actually did, um, the kind of the flow for all of the, uh, live data that they were getting.

So they took the, the user queries and did this data pre-processing with them, and then did some filtering and some standardization and things like that to, um, end up with this dataset. Um, so that's the second version. And then the third version was a, a pretty big leap forward.

Oops. Um, m-mm, to, to go into multi-turn and multi-step function calling. So previously, um, it was single-turn or, um, some parallel function calling, but this is more-- this is really digging into multi-turn and multi-step function calling, which I'll show a, a diagram of in a second.

Um, but really to get to where we're testing for like agentic, um, mimicc-- you know, mimicking agentic behaviors where we're kind of all trying to get to by using function calling. Um, so o-what-- one of the sort of like main innovations of this that we'll talk about in a second, um, is the, um, what, what they called state-based evaluation.

So they really, um, updated their evaluation from the abstract syntax tree to something called state-based evaluation, which we'll, we'll talk about in just a second. So they define the terms in the, in the third version of single-turn, multi-step, and multi-turn, and that's basically the difference in being able to say something like just, you know, "Find me flights from New York to Tokyo tomorrow," versus like, "I wanna plan a flight," or, uh, "I wanna f-find multiple flights," or, um, or multi-turn being sort of like maintaining a context and, and asking follow-up questions.

And that's where like, uh, having to evaluate state comes into play because the model's gonna have to remember that it's got these different responses kind of rolling of like, "I wanna book this one." "Okay, book me the ten AM version."

"Okay, like, what's the confirmation ID?" That kind of thing. Um, so really interesting how they set this up.

So, um, the dataset composition, they also made some pretty interesting improvements on how they did the data, um, composition. Uh, they started categorizing them according to like multi-turn, uh, and, um, augmented. I see a question in the chat.

Go full screen or slideshow. They, um, I can. Um, I was leaving it like this so that we could bounce back and forth to the actual blog articles, but for now I could do... Oh, actually, then I'd have to stop sharing and re-sharing.

So I'm gonna leave this for now, unless it's like really, really hard for people to see. I can also hide the navigation and zoom in a little bit probably.

Okay. Oh, that's gonna be too much. Let's go back to fit. Um, I'll also, uh, share these slides. Let's see if I can just...

Let me just dump the link in here as well

so that if you wanna look at it close up, you can.

Okay. So, um, so they had to change their approach to actually curating the data, which was really interesting because they, um, they wanted to be able to capture the, the multi-turn and the long context multi-turn and, and that kind of thing.

Uh, and so they did this really interesting approach where they basically created their own API for the sake of this, uh, testing, and then did this like mapping to create a graph edge construction and like generate tasks through that.

Um, and so here, I'll just show really quick.

I'm jumping ahead a little bit, but what was cool is they built out this like whole API code base where they had like vehic-vehicle control and stock trading and travel booking and then like a file system and then also like a couple of different like cross-functional APIs, like for messaging and simulating Twitter and creating tickets and doing math and that kind of thing.

Um, and kind of like built out this, this system where they would, uh, build out these APIs and generate, uh, a graph from it and then use that to derive a bunch of, um, example queries and function lists and that kind of thing, and then, uh, generate this dataset that way, which I thought was like pretty ingenious.

Um, so you can now see the, the third version, the dataset composition looks really, really different than it did in the first couple, and that's because they're really honing in on like what they're trying to test for there.

And so- Um, and they actually go into, in, in the, in the, uh, article, they go, they go deeper into like what all of these mean and, and what they do. Um, but you can see that they're doing things like having a s- a, a specific set of questions for testing for long context and, or testing for missing params, that kind of thing.

And so yeah, like I said, they, they, uh, they turn that into, uh, this nice system with, with an API that they created that they could use to evaluate. Um, and then they have this interesting validation process where they, they ch- they check for the questions, and they use this, um, human ground truth as a way to, uh, check for these things.

Um, and they go into a little bit more on that, uh, validation side. Um, yeah, so they actually like have humans label like what, what would be the correct answer for these different scenarios, and then that way they could check to see how close the model gets to aligning, uh, with the human response.

A- and so then that gets us into state-based evaluation. Uh, and so they use this as the primary metric to assess the performance of the models. Uh, and this is really interesting 'cause the way they, they, they basically compare the, the final state after all the function calls are executed to see whether the system's internal changes after each step like align with what you're expecting, which is better for reflecting worl- real-world performance.

'Cause like in multi-turn interactions, you're trying to update a system state, like whether it's like with a vehicle or a file system or something like that. Like, you wanna be able to check to see whether, um, you know, you've successfully deleted a file or something like that.

It's not, it's not as straightforward as just like answer a math problem or, uh, call an API or something like that. So, uh, they compare the attributes of the system state after every turn against the expected state. So for example, if there, if you had a, a series of function calls of asking the LLM to like create a file, write data to the file, and close it, it would basically check after each turn whether the file exists, whether the correct data was written, whether the file was properly closed.

Um, and if they're present and correct, then the evaluation succeeds.

And so then they also have a, a section at the end, um, on the results and error ana- error analysis. I thought this was like a super interesting section of w- what the models struggled with, and they gave a few examples.

So the first one was failure to perform implicit actions. So they had this example here of like, um, asking the model to like re... to f- fill the, a fuel tank. Um, and so, uh, the model thought that the tank was, um, already full or nearly full and didn't like understand that they didn't actually need to, to, to, to, to do the filling.

Um, so that was, that was really interesting. And then the second one they gave was understanding, failure to understand the current state before performing the action. So if, for example, um, you, you're already in, the present working directory is already Alex, and you tell it to go into the directory named Alex, it didn't know that it could just like check to see whether you were already in the present working directory named Alex.

It would just create it anyway. Um, so pretty interesting. And then the last example that they give is, um, LLMs incur unnecessary planning and thinking, which I thought was like a really funny way of putting that. Um, but, um, so they give the example of like you're already authenticated in the Twitter API, and you ask the model to do something with Twitter, and it, um, or it like goes, goes ahead and tries to authenticate you again.

It didn't know that you were already authenticated with Twitter. Um, and so it sort of like unnecessarily added a step. And so it's interesting because all three of these sort of tie into like knowing the current state, uh, or just like current context that the question is asked in, and, and that turns out to be like more challenging than you would think, uh, for these models.

Um, and so that's like, that's all I had, like for the actual overview, and then I figured if anybody wanted to just like chime in and talk about any of these things, we could. They also posted the, the, the data set on Hugging Face, which is pretty interesting.

It doesn't quite work. Um, there's some sort of error where you can't like see all of it, but you can see some of it. A lot of them are the math problems, but there's also like, um, some travel questions and, and other like planning questions and things like that, which are kind of just like interesting inspiration for function calling in general.

Um, but yeah, that's all I had for the actual, like, overview of everything.

Since n- uh, nobody's jumping in here, I, I found what you mentioned about using the syntax tree to generate, uh, sort of like I, I think I, I don't know if I understood this correctly, but, um, uh, generating, you know, sort of test cases that are like a sequence of API calls basically, and then from that generate, generating a s- like a, a scenario that sort of like implements that so that you can check that it has an absolute correct answer but-- and then like a scenario that you can have the LLM, the function calling LLM follow through.

Is that, did I understand that correctly?

I, I think so. You know, caveat that I'm, I'm still wrapping my head around it too, but yeah, I think that's basically what they did, where they- Um, they kind of had these different categories and that they wanted to accomplish, and then they used the, um,

the co- like the source code and everything to, to create the graph and like generate the tasks to do it. But then they-

I still think J-J... How did... Did you grok how they graph... how they did the graph edge construction?

Yeah. Let's see. We talk about that a little bit more down here. Yeah. Yeah, so there's this section here where they talk about, um, each function represents a node, and then they manually map out direct edges, meaning a function's output is an input of the downstream function.

So I guess they've got this example here, um, of placing order. Like getting the available stock, uh, it is necessary in order to place the order for the stock. Um, whenever we-

Yeah

... need a data set, we sample a node on the graph and randomly traverse the graph to generate an execution path.

I see. Okay.

Through the execution path, we're able to extrapolate. Yeah.

All right. Okay. So they, they basically go something like, okay, manually create the graph for the API, and then sample a node in the graph, and then traverse the graph according to the edges. And then from that, then generate...

So you have function calls, a list of function calls, and probably parameters, and then generate a scenario based on that, that sort of matches with that. Is that kind of what I-

Yeah, I think so, and I forgot to mention this part, which is really interesting. They, they use this like data set from PersonaHub to like take, uh, a few examples of different personas and then like generate the, the different queries based on those personas basically.

So like stock trading by an elderly hermit, for example.

Yeah. Yeah. I saw that.

Okay.

Yeah.

Okay.

But, but then I think after, like once they had that... Oh yeah, so they've got a, a triplet of question function list and initial config, and then-

Yeah

... they had humans actually like-

Okay. Yeah

... look at it and verify everything, and like validate that these are like good questions that actually make sense.

Right. Okay. Yeah.

Yeah.

That's a good... I agree. That is a, a pretty cool way to generate a data set like this where you kinda get a good covering set. Um, but, and, and sort of like generate the language around it, but then, um, yeah.

And then I guess it's small enough that you can label the whole thing, or so... Did they label the whole thing?

I think so because there were only like 2,200 questions.

All right. Yeah, so you can probably

So it seemed like they labeled everything.

Yeah.

Yeah.

Okay.

And they did, they did a series of like validations for each one. Oh, and also built tests around unit tests and error handling tests and things like that.

All right.

Um.

Got it.

Yeah.

Yeah, 'cause I mean, I guess at, at the end of the day, the whole idea here, if I understand correctly, is that I can, because I'm tracking my state, I can know that the f- the, the series of function calls that it made ended, I ended up in the correct state because of them or I didn't.

And that, so you get-

Yeah

... like a hard, hard yes/no on that.

Yeah. I think that's exactly what they were designing around because of the state-based evaluation.

Yeah. Yeah, that's pretty cool.

Yeah. There was also a whole section I didn't really dive into of like, um,

a- how they, they go through like a lengthy dis- uh, discussion on like response-based evaluation and why they don't use, um, like the react technique and things like that that I didn't really dig into. Uh, but other things that are k- just interesting, um, around like the limitations for evaluating with multi-turn scenarios.

Um.

Okay. Yep.

Yeah. Yeah.

Cool.

Well, yeah. That's-

Yeah, thank you.

Yeah. That's all I got. This, these aren't like super long, and so it's not like a, not like a crazy amount of stuff to, to read through, but they're... it's definitely really interesting, and then helps you kind of understand things a little bit better.

The only thing I didn't, I didn't see a lot of information on was how they derived this hallucin- hallucination measurement. I don't know if anybody else read through it and, and caught that, but that was the only thing that I didn't see like a lot of information on how they did the hallucination measurement.

Uh.

Well, I believe, and I, I, I could, uh, this is, uh, not from the paper, but just like, uh, a thing that I think it would be is 'cause they have a, um, they have a data set or something of a data set called, uh, like API Zoo, uh, where basically it's sort of the, uh, open source repository of like the correct documentation of all the API or of a large amount of, of AB- uh, APIs.

So presumably, um, that could be like a ground truth to benchmark against and then like how far, you know, you are, I don't know, probably semantically or something from that, um, or state-based wise. I don't know.

Oh, interesting.

Be my guess.

That's interesting. I don't suppose anybody tried to run this locally, did they? Uh, 'cause they link out to the code. You can actually like run it locally. I did not try to do it, but, um, just so, just so people know, you actually can.

You can just clone this and try it yourself, although you need like 1,000 API keys like any other, any other leaderboard thing. Um, but yeah.

I've not tried-

Cool

... the leaderboard specifically, but there's a, there are a few things in that repo that are, that are quite useful. Um-

Oh, really?

... open functions, Gorilla CLI, et cetera, et cetera. Check them out

Oh, cool

Worth looking into.

Sam, is there, is there any, um, description on how they, how they provide the, the state of the model? Like, what, what API queries it has already executed or what the original request is? How do they go about that?

Yeah, I think there was in this section. I mean, they don't go into a ton of detail, but

basically, let's see. Uh, so yeah, it says, "As the model interacts with the API, the backend tracks how the state evolves. Each function call alters the state, such as creating files. It... E- at every turn we compare the instance's current state from executing the model function calls with the expected ground truth."

So they don't go into, like, a, any technical detail. I bet, I bet it's in the code somewhere. Um-

Mm-hmm

... but it sounds like it's done in the API itself.

Hmm.

I'd be curious to know more detail about, like, what, like how that actually happens and how they read it out.

It seems like that would need to be specific to the API, right? 'Cause each API is gonna have its own set of state variables that it needs to track, right? Like a file system is different-

Oh, yeah

... to a Twitter post or whatever.

Cool.

So it seems like they, they, the, their mock APIs probably implement some sort of s- state. That's my guess.

Hmm. That might be a good ... I think, uh, Swix said they're gonna have somebody from BFCL on the podcast soon. That would probably be a good question for them. I'd be super curious about that.

So do they have ... Do, do, do they have, like, an API that orchestrates the, the, the request to, to other APIs, or is this just, like, referring to any API that you hit?

I think it's ... Yeah, they must have had,

they must have had sort of one collective in API that was a collection of the, of the, like, three or four ... Or no, it was, like, four, uh, single things and, like, four, uh, cross-functional APIs. So they must have had something that was orchestrating those different, like-

Well, yeah, that, I mean, uh, if you're not familiar with Gorilla itself, that's what it does, is it, it-

Oh

... it is, it's only ... It's a model that specifically generates API calls and nothing else.

Okay.

Wait, but we're talking about a model, and then we're talking about, like, an API that orchestrates requests to others, other, towards other APIs. So you can make an API call that calls this model, but that, that's, that's where I'm like confused.

Like, what is the model? Um, a model calling API calls that it was trained on. Like, I, I, I showed up to this meeting late, so you might not know this, but I don't know this. I'm not really sure.

Can't picture this.

Yeah, that's a good question. I'm not, I'm not sure how they, how they did that, if they had each API just s- separate, uh, and isolated so that the model would just call them dir- uh, like ex- call, call for the, that API directly, or if they ...

Probably. I mean, that would be my guess, is that they kept them separate because then they would know whether the model was actually calling the right API and not just, like, a gateway. I don't know. That's a good question.

Mm-hmm. So Yikes, you're having somebody from the Gorilla team come speak?

Um, uh, I think that is, uh ... I think that was ... I saw Swix mentioning something about that. Um, I-

I think he's gonna have somebody come on the podcast-

Mm

... was, uh, what I think he said. I don't remember who.

They-

Yeah, there's something in Discord about it. Uh, I, I am not organizing anything at the moment, but, uh, you know, if, if you have questions, they, they tend to be, like, super responsive in Discord, so just go jump in the Gorilla LLM and pick some brains.

Yeah. Yeah, it looks like a pretty active server.

It, it would be so much fun if he hosted the podcast with us as an audience.

I would love that.

I could dig it. That could be fun.

I, I would have loved to have spoken to, like, the, um, the person behind structured generation at OpenAI and then a lot of other people.

Maybe just like a q- a qui- uh, at least, like, a, a questions to ask drop-off box is worth doing, thinking about or something.

Yeah.

Yeah. Or, or, or something we could do is, like, we could do, like, an AI in action where we actually, like, you know, deploy this. Um, I don't know. Sam, would you be, like, open to doing that?

Uh, I, I'm not in charge here. I don't I don't know. I'm just doing this one.

When you say deploy this, what do you mean? Like deploy this, this leaderboard?

No, no, no. It's not deploy. Just, just get it running, um, just to, like, clear up any doubts.

Like the leaderboard, you mean, itself?

Gorilla.

Uh, I'm talking about this, this system that they're describing here.

Oh.

Well, uh, I'm ... Okay. So yeah, so that's the leaderboard, I think. Um, I mean, uh, I mean, yeah, like, if that's something that you are interested in trying to, to make happen, I presume that we would have an open slot on AI in Action at some point, and that, like, sounds interesting.

Oh.

Just, like, try to get the thing working. Um-

I'm, I'm not committing to anything.

Not committing.

I just was like-

I mean, that's, that's fair. That is a, an interesting idea of, like, we could do an AI in Action rather than, of a, like, let's try and get the thing working as opposed to, like, here is how the working thing that I use works.

Um-

No, no, no, no

... pretty interesting. Or-

Like, already, already have it working. No. Get it, get ... Trying to get it working in a span of, like, one hour is not-

Mm

... feasible for me at least.

Hmm. Yeah, I mean, you know, a, a cool idea, I think. Um, we can talk about it-

I, I'm just throwing it out there. If somebody wants to do that, I'm, I'ma be on there.

For sure.

I'm gonna be an audience.

Give it a shot.

Yeah.

Yeah, there's... I've, I've had Gorilla come up very often- Mm ... partially from Yikes and other people. I'm a huge- And I've never actually done- I'm a huge Gorilla shill and kind of a Shashir fan, uh, you know.

Um, but that's, uh, the Gorilla CLI is really helpful, um, 'cause I, I was using GitHub CL- or the GitHub Copilot CLI for quite some time. Um, but Gorilla CLI is, like, open, and I don't have to worry about GitHub connections.

I can just, like, grab it, and it does the same thing basically, so. What does the Gorilla CLI do? I don't, I'm not familiar. Um, it's, you just, you put in a natural language query of the CLI command that you want to run.

Um, and it'll, it'll give you the, the CLI command to do it. It's like, uh, if you used Copilot CLI, it's the same thing basically. I see. Okay. I haven't used that. That's, that's a cool idea. I mean, I've, I've, I've wanted to, to implement it myself, but it sounds like, you know, Gorilla already has the...

They already tied up their, their, their model with, with a little front end. That's cool. Yeah, no. There's, uh... The o- the other interesting thing, I guess, to go over the cool stuff in the repo, there's a thing called, uh, uh, Gorilla Open Functions, where you can sort of more or less, like, staple a Gorilla to whatever model that you're playing with.

So if it doesn't have function calling, you can just implement open functions and it, it'll give it function calling basically. I see.

Oh, cool.

That's...

And then you end up with no files in your file system.

Yeah. Just removes everything.

Well, you know, it's, it does hallucinate from time to time, so.

Yeah, we didn't actually... I, uh, almost wonder if that'd be another either AI in action or Paper Club, just like the Gorilla stuff in general. Like, we didn't-

Mm-hmm.

There's a paper for Gorilla and everything, but-

Yeah

... like, it was gonna be too much to try to cram into this. That's really interesting.

Yeah, I do feel like I, I would like to see a demo of, of the, the, the, like, that, the CLI, CLI, total CLI toolchain. Like, I know that, um, uh, the, the, I've, I've seen a bunch of, like, sort of these things in use and...

But I, you know, I haven't gotten started with them myself, and I've wanted to and never had the time to get started and know what's actually, um, useful. So yeah.

Yeah.

That would be a cool AI in action.

It would.

Cool.

Yeah, um, it's...

Anything else? Any, any other, other discussion topics?

So I actually... This, I, I have been, I've had an idea in my head for a while, and I wanted to share it with people if there's no other... It's relevant to this topic. I, uh, I wanted to share it with people in case if there's no other topics.

So, um, but the idea is basically to build a leaderboard of the quality of predictions of a, of a, um, of an LLM or LLMs using, um, like, and allow y- like, uh... So the idea would be to have questions that are, like, that, that are around the, some future event, like who will be elected president, right?

Um, and, and, and of c- you have a list of them, and then you pred- and then you ask LLMs to predict based on some sort of canonical data that you also bring in for the LLM to use, plus any other data that it wants to.

Like, if it, if it's an agent system, then it can go and search the web and do whatever it wants, and brings in, like, whatever context it wants, and then makes prediction on a given day about that event.

And then those, a series of questions, like, sort of evolves every day, and you basically, the LLM's placing bets basically on that. And then over time, you track the quality of prediction of those LLMs or agent systems. And the reason why I think this is interesting is because, um, judging a, a, an AI or a person's prediction quality is actually one of the only ways that you can actually assess, like, quality of information on the, on the internet, right?

So if, if you have a, you have a person or an AI that can accurately predict the future, then that... And, and do it better in sort of, in, like, sort of stacked rank, right? 'Cause I can always pred- I can very accurately predict that the sun will come up tomorrow.

So it's not, it has, you have to be, it has to be a relative ranking. And if you have a, a, an AI or a person that can predict better than, you know, sort of the, than the crowd, then that person likely has a good model of the world, right?

And that mo- that good model of the world represents, you know, sort of like at least, um, y- somebody who you probably want to listen to more than the crowd, right? So I, I feel like this would be a really cool, um, start to developing a way to judge both people.

Like, you could imagine now look, like, crawling the n- news sites and look at who, people who are making predictions and then sort of give e- everybody, like, track scores for different people, um, and their predictions in addition to LLM.

So I think, I feel like this would be a really cool way to get a project started. I wonder if people have thoughts about this, and if anyone would be interested in working together on it.

Deafening silence. Uh, w- uh, s- I was, uh, I was, I was busy trying to get my Gorilla CLI, uh, working. Um, but I, uh, so I was distracted. What is the thing that you want, uh, help on that you were doing?

Uh, well-

In the Discord server

... okay, summary leaderboard of the quality of predictions that AIs, and agents, and people make. So, like-

Interesting

... like, you're gonna start with n- news items or whatever, like certain current events, and then you predict, and that becomes a bet, kinda, sorta. And I can explain the betting mechanism that is fair. And then, um, and then-

Oh, like a, just a general prediction market?

Well, it's like a, it is like a prediction market, but it, it probably looks a little different, um, because I don't think it's useful to have... It's particularly useful to have it, like, like maintaining an account and the abil- like, that's a separate ability that I think is not necessary to track.

So it, but something like a prediction market, but for a s- um, a set of things that, um, you know, you sort of like, a, a limited set of things that you track over time, and th- whichever LLM and/or person does the best kind of filters to the top.

That sounds interesting and probably related to things that I'm interested in. So if you like, I don't know, if you start cranking on it or throw up a repo and shoot me a link or whatever, I'll probably take a look and, and play around with it.

Yeah. Yeah.

Um, if you're not familiar at, and I don't know if they have, like, an SDK or something, so you can rig up your own, but, uh, you can check out Polymarket. It's, like, one of the biggest, uh, uh-

Yeah

... yeah, ones.

So, I, I mean, I don't, I don't want to place real bets. Um, and I'm not sure that the, well, may- maybe there, I'm sure there are, like, uh-

I'm thinking, like, just throw up a Polymarket instance on, like, test net or something, and just use Monopoly money or whatever. You know what I mean?

Yeah, yeah, Monopoly. Okay. Is, wait, so Polymarket, well, uh, okay, uh, maybe I'm not familiar. I thought I was familiar with Polymarket, but maybe I'm thinking of a different one. That, is that the one that Nate Silver is in?

Monopoly card, uh, Discord thread about this?

Yeah, sure, yeah, totally.

Yeah, yeah.

Take it there and see who else is interested. Uh-

Yeah

... ba- back on Paper Club, do we have any last questions for Sam on Gorilla? I think some stuff po- popped up in the chat, but it looks like you kind of answered it in chat, too. Um, basically that, and then for next week's paper, if anyone wants to volunteer.

Okay. Has anyone, uh, have you guys seen MoML? It's the open weight, open data, multimodal state-of-the-art model. So it's from AI21, I think. I don't know how good the paper is-

I saw it

... but the, the model is pretty good. It's a pretty decent technical report. I'm hoping the paper is good, 'cause I'm down to cover it. They do go a lot into data, how they, um, yeah, how they get the data.

I think they pre-train it from Clip. But it's, it's a pretty good recent multimodal foundation model. Um, I can cover that unless someone else wants to volunteer.

That sounds super interesting, yeah. Um, I was gonna say I can be a backup if I pick a Monte Carlo paper and it actually, and everything works out. Um, but I would actually be pretty interested to, to get a, get a download on, on MoML too, so.

Okay. I'll do, I'll do MoML next week. Um, here's a link. I'll, I'll throw it in Paper Club. But thanks for, um, sharing, Sam. It's always fun.

Real quick, I did wanna check in. Is there anybody that's, like, new to Paper Club that has not presented or is, like, kinda maybe on board with it, maybe not? Um, if so, uh, freaking, uh, raise your hand, I, I guess.

And, uh, you know, if you need, uh, like, a little bit of help or support or whatever, trying to, like, get something s- 'cause I want, trying to facilitate new people that haven't done it before, 'cause I know you and I have, have done it a bunch of times.

Um, but yeah, feel free to, like, ping me in Discord or whatever if you have questions or something, or if you wanna do it, um, feel free to, yeah, let's let one of us know, and we would, we would happily hand over the reins.

Yeah. I'll make an announcement that this is the paper. If anyone else wants to sub in, we can always take it. But yeah.

Cool. Thanks, uh, Sam, for covering. Any last thoughts on this? I know, I know some questions have popped up you're answering.

No, yeah. This has been great. Thanks for everybody for listen- listening and, and joining.

Awesome. Thanks, guys. Take care.

Cheers.

Bye, everyone.

Thanks.