⚡️Ranking Agentic LLMs — Pratik Bhavsar, Galileo

Hey, everyone. Welcome back to another Latent Space Lightning Pod. This is Alessio, partner and CTO at Decibel, and I'm joined by my co-host, Wix, founder of Smol.ai.

Hello, hello. And, uh, today we are excited to talk about agent evals and leaderboards. Uh, Pratik, uh, y- I, I think I'm messing up the intro, but Pratik Bhavsar, uh, you're, uh, from Galileo Labs, and you worked on Agent Leaderboards this year.

Hey, folks. Uh, s- it's so cool to be here, and I'm excited to chat with you on everything about agents today.

Yeah. Awesome. Um, yeah, so I think, you know, I, I first got, uh, notice of this thing w- in our Discord when Erin Staples, uh, who I think works with you guys, uh, posted it-

Yep

... and said, said that she was also contributing to it. And I think the broad thing is, uh, there's a big shift in evals this year, I think, from just model evals maybe, which quickly get saturated. I think LM Arena also had some controversies, and I think more generally people want agent evals anyway, where they are evaluating the ability of, uh, you know, models to do real tasks.

And, uh, instead of, like, testing knowledge, they're actually, you know, doing, doing tool calling and, um, utilizing context very well, that kind of stuff. So, um, yeah, I think one of those things where... This is one of those things where I think there's a lot of fresh, uh, area and potential for progress if you can do a credible eval.

And the other thing I like to see is an eval that has a cost awareness component, uh, because I think, uh, you know, people don't really, uh, stay very honest with, like, oh, yeah, like we spent a lot more tokens, or we spent a lot more money on this agents and we did better, but it's not really clear, like, if the other comparison could also just do the same thing, like just think harder and also have the same performance.

Hundred percent. Yeah. So we have been hearing from our customers that they are now moving towards agents, and it become very clear to us that last year we did RAG evaluation because that's what we are discussing more with the customers.

But, uh, we quickly understood that this is going to be a year of agents. And what we have understood from our previous experiments is that it's not necessary that what you see as top models in, let's say, LM Arena or other specific evaluation or general evaluations, they might not be also the same ranking for other tasks like agentic tasks.

So that was our goal, uh, very fundamentally speaking, that we want to get very specific insights apart from just performance. And cost was the other thing that comes up a lot with enterprise customers, and that was the second focus essentially.

And then there is SLM versus LLM kind of debate. Uh, are the small models also catching up? And there is the big one, open source versus private. So those were the dimensions we were interested in.

Um, and in the eval itself, um, the big thing that you highlight is obviously tool selection, uh, and tool use, and then single-

Yeah

... versus multi-turn interaction. Maybe, yeah, for, for people that are just learning about the leaderboard now, just give a high level overview of what are, like, the key things that you test for.

Sure. So, um, Agent Leaderboard is kind of a eval harness specifically created to test LLMs on hundreds of domains. Uh, we know that there are certain research papers from top companies and research institutes, uh, from Berkeley, from Salesforce, from other companies which are targeting certain domains for evaluation.

But what our goal was that let's take all of these benchmarks, because it's possible that certain models are good at certain datasets. There might be issues of leakage also. So we wanted to create a as much possible holistic benchmark by taking on datasets, slices on single turn, multi-turn, and also on like different kind of edge cases.

Like there is some information missing for the parameter, the tool itself is missing, and long context and all those problems all at once called composite scenario. All those things we wanted to take from these different datasets. We did filtering on it for as hard as possible.

Um, and then we created 14 small datasets, uh, and then evaluated LLM specifically for tool calling. Uh, and we used a, used a Galeos metric called Tool Selection Quality, which is an LLM judge based metric, uh, powered by GPT-4o, and it has multiple judges so that we get higher performance.

And that's how we kind of get the score for each sample, and we aggregate for a dataset, and then we do average over the dataset. And that's how we get the overall score for the ranking. But yes, customers-- Uh, sorry, uh, readers can also go through particular performances or certain datasets as well.

Um, apart from that, uh, we were pragmatic about, uh, selection in a way that we were very interested in how the small models perform as well as open source model performs, because we know that there can be interesting insights.

Historically, what we have seen from our previous initiatives is that, okay, maybe the best GPT or best Claude is the top model, but there might be very small gap with the model just below it, and then it becomes a cost performance trade-off so that users can kind of select a model which is a slightly worse, but also gives them a similar performance.

And there can be some hidden models that nobody's talking about because it's all about the latest top releases and not about like the mid-tier or slightly lower tier, uh, models, uh, from other providers. And sometimes it's just They haven't got enough traction on social, so they might be just missing from the discussions.

So we try to find this kind of models, uh, and get insights as much as possible. So that, that was the whole initiative. Uh, we launched in February this year. Uh, previously we had done RAG-specific, uh, evaluations. Now, a small update on the agent leaderboard also, like we are going to do a V2 of it because we find certain issues with this evaluation.

And let's talk about this more at the end of the podcast, like what's next coming. Uh, we can dive more into that.

Um, what are some of the surprising results? For me, it was Mistral Small being the best open source model above 4o and DeepSeek V3. Um, anything else-

Yes

... that, that jumped out to you?

Oh, yes. So first and foremost, I would say that I was expecting GPTs to perform the best because they kind of pioneered the tool calling, function calling support, and they have been working on that for so long. Um, there was some criticism around Gemini models, although they were catching up.

And I was very surprised to see when we released this leaderboard, of course, there has been some shaking of the leaderboard also later on, but when we released, we found that Gemini models are performing really good, and they were extremely cost efficient at the time compared to the second and th-- second model like GPT-4 at the time.

And so it became kind of a no-brainer that if anybody wants top performance, they can just directly go, uh, go with the Gemini models. And they just not only released like the Pro model, they also released the Flash version later on and Flash Lite.

But the Flash Lite is a very interesting model. Uh, in, in the blog, we have the models when we released this leaderboard, but in the, on the original leaderboard, we also have Flash Lite, which i-- nobody talks about, I think even now, but I think it's got a very decent score, which is like in the top five models.

It's extremely cheap model. Like, it's so dirt cheap that it makes a lot of sense. Only model that is cheaper than it is, and performant is from Amazon, uh, which also nobody talks about. So we found these mo- interesting models, uh, that worked surprisingly well.

Then another surprise for me was that the reasoning models were not performing well enough. Uh, they had certain kind of limitation. When we probed into it, like why are they scoring less overall? They were like the o1, the o4, o3, they were not performing well specifically for multiple tool call outputs that you have this request, and it requires either the same tool being called multiple time or different tools being called all at once, and they were outputting only single tool call.

Um, and because our evaluation required that, okay, all the tools should be outputted at once, that's the instruction as well in the prompt. Uh, it was not following and it was getting low score. So because of certain inherent characteristic of specifically O series model, uh, we did not find them well to be performing on this kind of data set.

We had like multiple data set on the multiple tool output, uh, and that brought down the score for this model. And of course, they are extremely costly, right? So then it became, uh, it, it becomes straightforward that you should not be using this for tool calling and instead go for the regular models probably.

Um,

apart from that, on the open source side, I would say that I was very impressed by Mistral's entry into the ecosystem. Um, 3.1 just straight away went up, and it also very cost efficient. So I think peop-- not people might have known at the time.

And in their launch blog also, there was not much discussion on tool calling, but the model performed very well on our benchmark. Um-

You mean Meta, right? You, you said Mistral, but you meant Meta?

No, I meant Mistral only. Yes. Mistral 3.1-

Oh

... performing really well, uh, on our benchmark. Another-- O- of course, the other surprise was that all the Llama models were not performing well on our benchmark. Uh, 3.3 and even the Llama For All were really performing extremely poor.

So they are like-

Oh, no.

And I, I-- that was unbelievable. Like, it felt like they were not tuned for tool calling at all in some way. Um, so I mean, historically, it's all historical, right? Then we have certain biases, and we are like Meta 1, the Llama 1 was great, Llama 2 was great, 3 also was great, and then suddenly something happened, and it went downhill.

So these are like surprising results for us, I would say. When we started the leaderboard, Claude 3.5 was somewhere in the middle. Um, it was really good at certain other things, but for tool calling, it was not a-awesome.

But then there's, uh, when we released the leaderboard and just in a week, they launched 3.7, and that went straight up, and nobody has beaten it so far. We are not e-- We are at the latest one, um, like the Qwen models, uh, al-ni- also the latest Mistral models and not the o3 Pro models which just came out in the last few days.

We are not evaluated, but it, it held the fort for many, many weeks. So, uh, that was very impressive release, I would say.

Awesome. Um, you know, I think, uh, that's a very good overview, uh, very good findings. I think people can use this leaderboard to pick out what base model that they wanna use for their agents. Um, and that, I think that's a really good, uh, starting point.

Uh, I wanted to dive in a little bit on a little bit, a few of the metrics that you chose. Uh, so first of all-

Mm-hmm

... did you just, did you just rerun, um, all these evals, basically the BFCL, the XLAM? Or do you still, um, do you take it off of, um, whatever they publicly report?

No, we actually took the data slices from the original data set-

Mm

... and then ran it ourselves. Uh, we-

Mm

... got the outputs, uh, and then we validated with our metrics. So it's-

Okay

... fresh. Everything is fresh.

Okay, everything's fresh. Yeah. A-a-and as long as it's like, you know, comparable and, um, it ba- basically, like, do you th-think that these are... Do you have any comments on just, like, BFCL versus XLAM versus, you know, your, uh, your own TSQ thing?

Um, how do they, how do they differ in methodology? Like, what should people know about their pros and cons?

Uh, so I feel like they, they're very similar in nature. It's just that, uh, the data set is from different domains, and-

Yeah

... certain data sets have multi-turn conversions available, while certain do, do not have it. I do not remember exactly which one has it and which one doesn't, but there were these discrepancies. And we really enjoyed, like, I really loved the Tau Benchmark also, 'cause it had-

Ah

... particularly hard evaluation data sets, and a lot of models had some struggle over there. And-

I think Clay, uh, Clay from Sierra actually just released an update to Tau Bench for some reason. Um-

Oh, really? Is it?

Yeah. Uh, Alessio's best friend now.

Ah.

Um.

Yeah. Superb research. Yeah.

So he, he, he released, uh, Tau Squared.

Tau Squared.

As a-

Wow. I was not aware

... notion of dual con... It was yesterday. I just, I happened- ... to see it.

Okay.

Um,

to coordinate, guide, and assist the user.

Yeah.

Wonderful.

So, so Tau Bench, uh, I actually didn't know it was from Sierra. Uh, but it's, it's actually very, very, uh, influential.

Indeed.

Can, can you... Y-yeah, can you talk more about Tau Bench?

Oh, yes. So essentially Tau Bench, when it was released, uh, I mean, yes, the Tau Bench is for two, uh, two domains. It tests agentic capabilities for two domains, which is airline and retail. And-

Mm

... their setup is that they have very long prompts in the beginning with lot of request, and then you have these tools defined, and the tools can interact with certain databases for read and write operations. And it is predefined that once the whole agentic task is done, then they will validate on the final, uh, edits made on-- and the-they will check for the expected value in the data sets, uh, in the database, which are basically CSV style of files.

Uh, and in that fashion, they will validate whether the task was done or not. So it's, it's, it's pretty well defined such that you get a deterministic evaluation all the times. That's the beauty of this, and it's, it's domain specific, which I think is missing from other evaluation data sets.

Uh, and that is also one of the themes for our V2, um, because I think people don't just come from a general mindset, right? Pe-people in the industry come from, let's say, I'm from healthcare, I want to see if my-- if these models work for healthcare or not.

People are coming from investment, finance, insurance industry, and they want to know- ... that is, are the models ready for my domain or not? Because we all know that things work on the general domain side, and when you just put it on the internal data, things start to blow up with the internal terminologies and the cases that happen.

So I think that was the beauty, that they specifically targeted domains. I don't know if they might have expanded domains in this because two was, I felt like less, but it was a decent start. It was more about the methodology rather than the domains I felt with the Tau.

It was very hard-

Yeah, it d-

... as a-

Yeah

... model.

So Tau Squared, you know, just kinda looking at it now, Tau Squared doesn't seem like it expanded domains. Uh, it, it just, they, they expanded the way that they interact with the user.

Mm-hmm.

So some kind of, uh... Oh, they, they added a telecom domain. Okay, there we go.

Okay. I see.

Um, but I, I, I f- it, it seems like the emphasis was on more sort of, um, co-piloting, uh, rather than, uh, two-call, just, you know, th-they call it dual control. Yeah. We'll, we'll see, we'll see what this is.

I, I, it's, it's obviously very new, so I just, I just happened to see it the other day. Um, okay.

Then that's worth solid.

Yeah, yeah. Definitely put that in your, uh, V2.

Absolutely.

Um, and it-- Yeah, and I, I also wanted to go back to, um, TSQ. I think that's the other sort of maybe, uh, novel part. Um, I think a lot of people have queries about LLM-as-Judge.

Mm-hmm.

And, um, yeah, just tell us more, like, you know, do you have any calibration errors? Um, how, how, uh, do, do-- should more people adopt this kind of approach?

Indeed, I agree. LLMs have been notorious with their own biases, right? We have seen historically they have issues with long context, and they have issues with certain kind of outputs. They prefer, let's say, lengthy outputs and stuff. Um, what we-- So this, our philosophy in the company is that any metric that comes out, which is a Galileo metric, we do extensive research on it.

Uh, we create very hard data sets internally, um, and we evaluate it, evaluate them, we annotate them, and check for correlations. And only once we feel comfortable, then only we release it, and which is usually based on the correlation scores, like AU-ROC scores that we find.

If they are high enough, only then we release it. Otherwise, it goes through a rigorous process of prompt optimization. So we allow people to use any LLM with the metric. Uh, and so because LLMs keep on improving, the metric also keeps on improving.

And then the prompts is what we kind of Uh, the prompt is where all the research goes through, and we have seen tremendous value in that. And yes, people would definitely have skepticism. A lot of people, a lot of our customers also have that skep-skepticism.

Some are still stuck on BLEU and ROUGE scores, so which is surprising because they think that LLMs are not there yet. But we-- what we try to do from our side is essentially we try to follow the best practice of prompting.

Uh, we try to exploit the reasoning as much as possible from these LLMs, and then we use basically self-consistency by using multiple outputs and then taking a majority voting over there. People can-- people have flexibility on these things.

They can select the LLMs they feel they want to use. If they want to use much more powerful LLM, go ahead, you can customize it. If you want to change the prompts, go ahead, customize it. If you want to increase the number of judges, go ahead and customize it.

So we allow full flexibility to people to modify this LLM-as-Judges so that they can get the best performance possible. Uh, of course, people should always validate with some small annotated data set before they go ahead and just make it their default evaluation.

So that is always a part of our POCs, and we-- they get clarity and confidence, only then they will be-- they are using it. So I feel I have-- basically, I've been a full-stack ML engineer previously, um, and I've worked a lot with annotators, and I have seen a lot of biases and issues with annotators as well.

Um, it's just so time-consuming that you can't afford to kind of wait for annotation to get completed. So it's just much, much better to add more instructions, do some initial validation, and get the clarity that, okay, this is where it works.

If it doesn't work, yes, go back to annotation. But I feel like it has started working a lot one year back because we-- I have-- I was running these experiments, like we were working with hallucination index two years back, uh, where we were using our own similar LLM-as-a-Judge methodology, and we had certain issues, and there were some correlation issues at the time.

But last year it got w-pretty strong, and this year I feel it's so strong. I have LLM-as-Judges is so strong that you can use it. I've gone through like hundreds of explanations. Like, our LLMs also generate explanations that users can read to validate whether it actually went through this correct reasoning or not.

And I have gone through many explanations in the-- with-- during this V2 experiment that I'm doing right now and also the V1, and I did not find any issues, so, uh, I feel like when we-- so that's why we released all the data set.

People can go and run the experiment. With the V2 also, we are going to release the data sets. People can just take them and run it on our, uh, cluster. It's free. Our, our product is free, so anybody can just sign up and start using it.

They can just put the OpenAI key, and they will get a clarity, okay, it's working or not. We want to be as open as possible while still being a closed source company. So I feel like it's working, and people should definitely pay attention to it.

What, what's the gap between the best prompts and, like, the worst prompts? Like when you run-- as, as you iterated on, on the prompting of the LLM-as-Judge, did you see models change in, like, position in the leaderboard, or were the results mostly the same, just like different scores?

Uh, so I was not working on fine-tuning the LLM-as-a-Judge. It is done by a separate team member, and once the metric is ready, only then I start using it. But, um, that was-- they went through like a months of rigorous process of prompt tuning and, uh, it's a very structured prompt.

It's like an art in itself, as you know, where we have certain sections for certain things on certain instruction. And what we do is like it's this very simple philosophy. We start with a base prompt with what we think is correct, and then we see where it's failing, and then we try to add those instructions again, uh, defining the failure cases in a general way to the prompt of the LLM-as-Judge and see if it's following it or not.

And that iteration goes on for multiple times till we are sure that we are getting the performance that we need. So it's, it's very common sense-oriented approach, I would say, and it's just plain English writing with just following the right way of writing prompts.

Um, and of course, the leaderboard, if you are using different judges with different models, it can-- there can be heavy shake-up of the leaderboard also. So that's why we use the best LLM available at that when we are run-- when we are starting or near completion of the experiments.

We try to use the best LLM available so that, uh, it should not be a situation where we're trying to evaluate, uh, 3.7 with GPT-4o Mini, right? It will be really, uh, risky proposition because GPT-4o Mini can't find, uh, cannot evaluate the hard outputs that 3.7 might be doing correctly, right?

Um, so and so for this V2, I'm using the 3.7 because it is one of the top-rating models right now, and I've seen really good performance from it. So we essentially swap these models as much as possible when we are running the experiment.

Otherwise, it looks way-- We have seen situation previously where we were using probably GPT-4 Turbo or something, or even GPT 3.5 long back. I'm talking like two years back. Um, and we had to use GPT-4 Turbo at the time because with the 3.5, it was something else entirely because it can't find the errors.

Mm-hmm.

Okay. Um, awesome. Uh, maybe give a little preview of the thinking for V2 and, uh, maybe that, that'll be, uh, where we end up. But, uh, yeah, I'm just kinda curious.

Sure. Uh, what we found issues with V1 is, uh, it has a lot of domains, but it does not have a domain vertical in itself that, okay, if somebody is interested on how this model perform in, let's say, telecommunication, they can just-

Yeah

... select, filter, and then see which-- what's the performance from each model, what are the gaps, what's the cost related to them. Um, so w-- first thing first, we are doing domain specific, um, which means that these are like new datasets from scratch.

And why are we doing this? Uh, is of course, the first problem is that they are not available easily. Secondly, it's because, which is, I would say the main reason, is that we don't want leakage at all to be possible in any of the LLM so far.

So we want that this are new datasets, so we guarantee that it's going to be performing as unbiased as possible. Then we saw with V1 that the scores are saturating. We see that the best score has already reached point nine five, and so if we are going to evaluate even better models that are coming out, they'll be definitely in the range of point nine to one, which I feel is not very informative for anybody to take a call because there can be some noise also in this evaluation.

So we want the benchmarks to be harder, uh, s- and this tool calling to be more complicated and realistic at the same time. Uh, this BFCL, if you just open and check it, uh, you would be like, okay, this is okay, but you don't want to use it for realistic evaluation because it's like some math problem statement is given, some simple tool is given, just like let's say some exponential tool, logarithmic tool, multiplication and stuff, and you are just like evaluating on that.

So I was not particularly happy with that, but that's the best we could got-- get at that time. Um, and that's why we mixed up all the datasets available at that time to kind of average out the things, uh, because it's very specific, right?

If some model is good at math, then it might perform really w- great for BFCL, but if some model doesn't understand the terminologies of industry, uh, then it will not perform well for Tau. So we kind of mixed up the dataset during that time.

Um, so two things, uh, domain and one is increasing the hardness. Third thing is multi-turn evaluation. So this whole evaluation is going to be, let's say, a support bots. The whole theme is evaluating support agents with lot of tools, like we are having twenty tools.

We are going to have five domains, uh, some like banking, finance, telecommunication, insurance, healthcare. Um, and we'll be having certain categories also, like coordinate the tools and then kind of adversarial scenarios, customers manipulating, uh, the bot to kind of do certain things.

And, uh, yeah, multi-turn. Coming to multi-turn, the m-- in the multi-turn, what we are doing is we are-- this time-- Last time it was more like there is an input with one message or multiple messages and then give one output.

That was how we were evaluating. This time it's like a realistic scenario that we start with a scenario. A scenario has a first message and, uh, there's like goals to be accomplished. This scenario is given to a user simulator.

Now, user simulator is basically a prompt, uh, which gets the whole history of the chat, and it's supposed to generate a new message, and that simulation is ran by let-- a GPT-4.1 mini model. Now, there is another simulator which I call the tool simulator.

Why we selected tool simulator? E-essentially because of ease of running the simulation. Uh, as we saw with Tau, Tau has like tool definitions and, uh, databases with CSV file kind of thing with read/write operations going on. Now, I wanted to have like, uh, many tools created.

We have like twenty tool per domain, and we have like five domains, so it's like hundred tools. I did not have-- do not have the time to kind of write this, uh, tool with all these databases. So I kind of created a tool simulator, again with the GPT-4.1 mini model, uh, which takes into the input as a tool call.

So it gets the tool call, it gets the response of the tool, tool call schema, and then it is supposed to give an output. So rather than like interacting with an API or calling certain databases, it is going to generate an output based on the history of the conversation and the tool call information that is given.

And the main one that we are evaluating is basically the agent. Uh, the agent, we are swap-swapping all the LLMs to evaluate, and this agent basically is accomplishing the task given by the user. And then user gives a task, the agent either gives a response or gives the tool call.

If it's a tool call, it's given to the tool simulator, and the tool simulator will give an output, and then that output will be used by the agent to give a response to the user. So what we do in this is we-- for every scenario, we have like at least like five to seven goals to be accomplished.

Uh, they are pretty complex like Tau, and they are all-- they all need to be accomplished. Otherwise, it's-- the task is not accomplished. So this time you have to get done many things at once, which makes the thing hard, and it's possible that you might miss out on, on certain things.

And we see that the best models are good at it, but certain models are really struggling on it, and they can go into iterations for no reason. Um, and so the scores are coming out to be lower, much lower, like the pro-- the mean problem point nine five Is like solved that, okay, the scores are lower.

We are getting domain-specific insights. Uh, and then it's a very realistic scenario where you are having multiple turn, and then only finally when everything is done, then the task is accomplished. So what's-- how do we evaluate the performance in this case?

And we use two metrics. Uh, we use the older one, the tool selection quality, which is just telling us whether the tool selected was correct or not. But other thing which is super important from a business point of view and practical point of view is that whether the agent completed all the task or not, right?

Uh, so for that, we have the action completion metric, uh, which will check that whether all the requests that were given by the user, were all of them accomplished and were they accomplished correctly or not. So for every sample, we'll get, okay, this is how good it is at tool selection.

This is how great the model was at co-accomplishing the task. So in that fashion, we'll get the scores for every scenario, and then we'll aggregate the s-scores for all the dataset to get the final score. So this time we are kind of focusing more on action completion metric, I would say, which is telling us that the user asked for these five things and whether the agent completed all the five things or not.

So that's how we are going to evaluate it. We'll also have the tool selection quality metric, but that doesn't tell us the whole story. It just tells a very granular story. It's a debugging metric, but this is like an overall performance metric for the user.

And then so essentially, user workflow will be like in the realistic production workflow is that, uh, they have this agent working, and in certain scenarios, the action completion metric is lower. So they'll go there and then check what went wrong, and then also they might see some tool selection issues, and they will debug them.

Got it. Um, perfect. Uh, I, I think that's a, that's a very comprehensive recap. Um, I like the, I like the new direction for, for V2. It sounds like a lot of work.

It's difficult.

So good luck with that.

It's difficult to-

Yeah

... tune prompts, uh, to get the right direction of what you want to achieve because-

Yeah

... you want to be realistic, and we-- The, the hard part for me that I've spent a lot of time was generating hard scenarios, um, because- ... you take a hard mo-- You take the best model to generate hard scenarios, and you then see if the best model can accomplish those tasks or not.

Like let's say I generate by 3.7 and then check on 4.1, and then it's able to accomplish. So I had to kind of do a lot of iterations on like finding where the model fails and how to trigger this kind of scenarios, uh, and that took a lot of time.

So, but I think we have go-- we have reached a certain stage where we feel confident that, yes, they're realistic and hard scenarios and the model is not able to accomplish all the time. So this is going to like reset-- This is going to calibrate the expectation of the users, uh, the, of these LLMs because our last leaderboard is saying that models are really great at tool calling, so it's like safe, but they're actually not, right?

They're making mistakes, and this is going to recalibrate the expectation of the users that be careful because they're still not perfect in real scenarios.

Awesome. Okay. I think that's about, uh, all we have time for. Uh, anything else, Alessio? Any, anything else, Pratik?

No. Thanks for joining us, Pratik.

My pleasure. This was awesome.