smol agents are all you need

Hey everyone, welcome to another Lightning Pod. Uh, I'm Alessio, partner and CTO at Decibel, and I'm joined by Marcus Swicks, founder of Smol.ai.

Hey, and today it's a very beautiful smol episode because we have the creator of smol agents, Aymeric from Hugging Face. Welcome.

Hi everyone. Thanks for having me.

Uh, Aymeric, what is an agent?

So that's, that's a really good question. Like, everyone's talking about it, but, uh, I feel that no one has a clear definition. Uh, I think really good, um, essay in the direction of a definition is given by what you have your screen on currently.

So this essay, "What Is An AI Agent?" written by Harrison Chase, uh, from LangChain. So I think Harrison is one of the people who have thought the most about it for, for the longest time, so he has quite a good definition.

Uh, he says it's not perfect, but here it goes. So basically, um, you, you struggle when you have a, an LLM, uh, because the knowledge it has is basically what it has seen during its training, but highly compressed.

So if you want to perf-- uh, if you want it to perform better, uh, on any task, you need to give it access to the outside world. So what you will do is build an agent, and basically agent, uh, an agent is an application where an LLM has some level of control over the execution flow of your app.

So, uh, that could be a very low level. For instance, if you have an if/else where the whichever, uh, part of the switch you go to is determined in some parts by an LLM output, this is a router.

This is some level of agency. Um, if you have a while loop of which the continuation depends on an LLM output, this is multi-step agent. If you execute a function based on an LLM output, this is function calling.

So there's basically different layers of agency, and so that's where we see that a-- being agentic is not a one zero definition. It's more basically 50 shades. Uh, and so yeah, he draws this nice, um, uh, continuum of agency.

So basically, when you have a program and you have an em- an LLM embedded with that that has some control over the, the, the execution flow of the app, uh, that's what I would say an agent is, if you want a rigorous definition.

Yeah. So you don't even necessarily need tool use, I guess. Um, but I guess tool use comes in, in, in part of it. Uh.

Yeah. Tool use is so useful that, yeah, often comes.

Yeah. Often I use, uh, Lillian Wang's, um, agent post, um, where she has this sort of equation, uh, where she has-- it talks about memory, planning, action, tool use, m- you know, all, all the other stuff. Um, okay.

But let maybe we'll go s-strict, uh, quickly into, uh, I guess you, you worked on smol agents. You announced it at the end of last year, December 31st. Uh, why were you working during Christmas? And give us the origin story of smol agents.

Yeah. I spent the, the whole Christmas locked in working over, over that 'cause I felt this was a big thing. So basically, I started, uh, with the premise that whenever I was trying to work with an agent library, I found that often too complicated to work with.

Um, so that was one of the pain points, and thus the small in smol agents, we wanted to build something really simple. Um, and second aspect of it was that a few people write, um, code to do code agents.

So basically code agents, uh, or agents that write their actions in code. If, if you want, we can detail that later on. Uh, as opposed to what everyone else does, which is mostly JSON agents. So the combination of the two, doing very simple agents and doing agents, uh, that can act in code, uh, was that what drew me towards building smol agents.

And really the, the, the driving feature of it is to make it very, very short and simple. So the, the main file in it, the agents.py, uh, file that we have at the core of the library is under 1,000 lines of code to be, um, yeah, very easily understandable.

So yeah, if you want to, to take a look at it, it's really... And, and-

Yeah

... the goal is also to make the logic apparent in it, like you can read through the code and see how it works easily. So yeah, agents.py.

Oh, boy.

Exactly.

And what was the, uh, what did you see missing when you started building this? There's a lot of people trying to build agent frameworks. What were some of the, the design opinions, uh, that you had going in?

Um, yeah. I think-- So if you, if you check in this file, you have this CodeAgent class, uh, which is really our go-to class, and you have the ToolCalling class. So the ToolCalling class is more the classic way of doing agents.

It's basically if you plug into, um, OpenAI, uh, tool calling formats, um, what-- the, the way it works is that you send it a request, you tell it you have these tools that you can use, and it will spit you out, uh, basically a dictionary with one key, which is the tool name, and the other key, which is the tool arguments, and then you do this call.

But, um, I found this is highly suboptimal, and that's why I've designed the other code agents, which is really like the, the core, um, opinion in the library is that code agents work better. So that's actually a very interesting paper called, uh, Code Actions: Uh, Do They Yield Better LLM Agents?

CodeAct. Yep.

Yeah, exactly.

We covered this with, um, uh, Graham Neubig, who also runs Open Hands and, uh, Open Devin.

Yeah. And I, I really love this paper. They, they have this great example, uh, of code agents versus JSON. So maybe you've gone through that already, but, um, yeah, this figure here, um, compares, uh, a workflow for a code agent where it can just run a for loop.

In the for loop, it can do four different tool calls. And so in code, you can, um, the LLM can control very precisely how the parallel loops are run and in which sequence, uh, inside the, the-- each, each part of the loop everything's run.

Versus if you had to do this in JSON actions, it would be like 20 different actions, and it would be really hard to parallelize. Plus, code has, uh, other advantages on top of that. For instance, if you want to define a variable with a JSON agent, good luck.

Uh, with code, it's really easy. Well, you just assign, uh, that to a name.

Yeah, yeah. We actually had this, um, in, I think our, uh, 2024 agents, um, roundup, where we actually talked about CodeActs, and, uh, Graham, Graham actually talked about, uh, this as like the number one paper that he re-recommends for people.

But also-

Love

... he has quite a few others that, that might be interesting for comparison between, uh, smol agents and, uh, O-All Hands or Open Hands.

Mm-hmm. Yeah, really interesting. Yeah, I, I put a big plus one on his first paper. I have to look at the others and then

Yeah, we did an episode on this, uh, maybe like almost two months ago now. Okay. Um, yeah, anything else about smol agents? You know, I think, um, the, the, the thing that I got from-- Oh, you also have like this, this thing.

Yeah. Uh, the thing that I got from, uh, smol agents was like, this was like an experiments. Um, you know, I, I never-- I didn't really think, uh, think much about it. I know, uh, Hugging Face has been working on small LM as well.

Um, but smol agents, like is this part of a broader effort? Is this-- Is there a team behind this? Um, you know, what, what's the context?

Um, yeah, the, the idea was that we had transformers.agents, uh, for some time. Uh, it was basically the experimental version of smol agents, uh, that, um, this had started being built, uh, quite a while ago, actually by Lissandre, uh, which is our, our head of, um, open source, and Sylvain, uh, who's n-not here anymore.

They, they had started this thing where you could basically have a code agent that could perform one step. And so I thought that this had potential because they had built from the ground up a new Python interpreter. Uh, basically making code agents, um, has big security, um, challenges.

So they had built a Python interpreter, a custom one from the ground up, and I thought, "Okay, let's reuse that one, uh, make it multi-step, uh, make the state persisting between steps, and we'll have like basically a banger agent, uh, right there."

So I, I, I started transformers.agents on that premise. I thought that it got really good performance, for instance, on the GAIA benchmark, um, which I, I tried, uh, to improve, uh, to improve on, uh, a lot. And then, uh, that was the, the groundwork for starting smol agents.

And now we're a team of two already. Uh, we're recruiting two more, so yeah, it's growing really fast.

Yeah. Do you have any other-- I mean, you know, obviously the Python security issues are well known. Do you have any sandboxing tools that you experimented with and you liked?

Um, yeah. So basically we're going with E2B for now. Um, we are-

Ooh

... building Docker s-support as well. Yeah.

I knew that. I knew that. Um, yeah. Uh, but like, you know, uh, is there, is there, is there any, any reason E2B over like other, other tools? You know, just, just for my info.

Um, basically we discussed a bit with the E2B guys, and it was like the, the start is really quick. It seems well supported. S-it seems like a robust solution, so let's go. Basically we, we don't think very much when we, when we have a new feature to-- We, we take the simplest route that will give us some sort of thing and we, we just, yeah, lock in and, and build that.

Yeah. I m- I mean, I think it's easier to switch, right? Like if you, if you need, uh, some other execution environment and, you know, fair-- full disclosure, me and Alessio are also investors in E2B. Uh, Alessio is slightly bigger than me.

Uh Uh, okay. So we, we talked about this. Um, you also worked on a course, um, right?

Yep.

Um, yeah.

So this is, um, um, another team at Hugging Face, so, uh, I'm working on it with them, but I'm, I'm more supporting on this. So they have launched the agent course, uh, which is like it hasn't gone crazy big like past-- The courses we did in the past was like 10K subscribers, which is already nice, but this one completely broke the-- At, at some point it broke the website that was used to do the, to, to, to-- for people to sign up.

So-

What, what is this? I, I couldn't see the website. What, what-- People say 10K. I was like, "Is it 10K stars?" And then it's not 10K stars, so I don't know what, what, what-

Um, where is that? No, it's not Smolcourse. Uh, it's, uh, Agentcourse. I think the, the link to sign in is on the website. Um-

I see. Anyway, I, I can look it up later, but I, I was just like very surprised. This one, I guess. Um-

Yeah, exactly.

I see, I see. Okay.

And yeah, so-

People really want the courses.

Now they have like-

Oh, it starts today

... sign-ups.

Oh.

And yeah.

Okay. It starts today. Yeah. Okay, amazing. Okay, cool.

And I think it's going to be really interesting because they start from the basics, like really what is an agent, how to code that from, from scratch, and then see a few f-frameworks, but after that. 'Cause I feel like people often when trying to learn about agents, they often start with a predefined framework, and they go into the, the weeds, the specifics of that framework.

Whereas if you want to learn that, it's always good to try to code a, a bit from scratch yourself, I think.

Okay. This-- Is this-- This is all Hugging Face people.

Yeah, exactly.

Wow. You guys spend a lot of time on this. It's crazy. That's r- That's very impressive.

Yeah, yeah.

Um, okay. Um, so people can sign up for the course and, and go through the, the material. Um, anything else to, to note before we, we move on to, you know, the other two stuff?

On the course, uh, what's in there? Yeah, there's a certification for people who want to, to show something on their resume after. And what else? Yeah, we're, we're doing really nice modules on this, uh, with, uh, with very varied use cases, so yeah, I think it's, it's going to be really good.

As you see, yeah, there's quite a lot of, uh, effort. At Hugging Face scale, we tend to do things with very small teams, so this is a big effort.

This is big for Hugging Face, yes. I agree with this. Uh, okay. So one of the things that people are talking about in surprisingly o3 adopted, or actually not even o3, Deep Research adopted was GAIA. Um, oh, actually you guys also have open Deep Research, right?

That's just a different team. Um-

Yeah. Exactly

... anyway, I think, you know, the, the French Hugging Face mafia, uh, is, is running GAIA, and we've covered GAIA briefly before, but I'm sure most people don't know-know what GAIA is. Uh, maybe you want to introduce GAIA.

Yeah. So GAIA, so the, the name is, uh, General AI Assistance, um, Benchmark, so that would be GAIB. But, uh, yeah, I don't know where the A comes from at the end. Um, so basically it's, uh, evaluating agents that are as general as possible in helping you.

Uh, so the idea is to have, uh, an open world environment, so basically, uh, agents that can browse the internet. There's no sandboxed environment as with many benchmarks. Um, and the questions are very general in nature, and it's aimed at tasks that would take a human ten minutes on average to solve.

So it's like-- Actually, it's a bit more than that. It's 10 to 60 minutes. And yeah, this is a good example of, uh, of question. So basically you can show a picture, um, and you need an agent that's able to look at the picture, find a number, uh, do a web search, do rigorous calculations over several steps.

So basically most questions take something like fif- five to 10 steps, so the agent must be really rigorous at that. Uh, there's one true answer. Uh, and also a good thing is that the test sets, the answers are hidden.

Like, no one has access to them except the team, so not too much con- contamination hopefully.

Yeah.

And maybe that's why OpenAI Deep Research didn't disclose the, the results on the test set.

Uh, I, I, I don't know. Uh, did they, did they even try? Did they submit it to you guys?

No, no, no, they didn't submit anything. They just submitted-

Yeah

... on their own.

They just, um... Yeah, Clementine was, was very, uh, pointedly saying that, "You know, you guys only submitted on the val set." Like

Yeah.

Um, okay.

But I'm afraid she will hurt me if I don't submit the, the results for OpenDeepS- uh, uh, Open, uh, Deep Research on the, on the test set as well.

Yes.

I'm running them right now at the moment.

I see. I see. So, so you are doing OpenDeepResearch with smol agents, and you don't have results yet?

Yeah, exactly. Like, we have result- uh, results on the validation set-

Okay

... and I'm running them on the test set at the moment.

Okay. How are they looking on the validation set?

Uh, validation set, we are third. Um, yeah, that, that's the, on the other tab, the, the result for, uh, validation.

Oh, sorry. Yeah. Okay. Yeah.

Exactly.

Okay. Here it is.

Oh, Meta's just submitted. How are they on top of, of us? They submitted after. It's unfair, and they have the same score. Man. Okay. Okay.

So, so, so you basically you take the average of level one, two, and three, right? Okay.

Yeah, exactly. That's, uh, the, the-

But your level three is so much higher than theirs. Come on.

Yeah. Man, should be counted for more.

Okay, great.

Man.

It's interesting that you got the exact same level two. Yeah, go ahead.

Yeah, I was gonna say, I'm sure people are gonna wonder, you have the exact same level two score, but then there's such a big drop-off at level three. Do you have any intuition as to why? Is it just because of like o1 is better than Sonnet or, um...

Yeah.

I was impressed by o1, uh, honestly on, on, uh, agent tasks. So GPT-4 performed well with smol agents already, and when I tried o1, I was, um, yeah, kind of baffled by how good the model is. I tried R1, but R1 is a bit under, uh, o1, uh, with smol agents, and I think this is also a matter of formatting.

Like, sometimes the model struggles to just output the, the code snippets in the correct way that we expect. So if we fine-tune it, it, it should probably get better. But yeah, maybe so higher scores on level three might be linked to us using, uh, o1.

Okay.

And also some other-

I don't, I don't know what Meta uses. Sorry?

Yeah. And also an important thing here is that, uh, many submissions use majority voting, um, which is, well, uh, well, it, it does improve-

Cheating

... scores, but at the... Yeah. It's at the price of higher cost.

Yeah. Uh, okay, yeah. So yeah, it looks like they use, uh, Claude 3.5, uh, and o1. Maybe if they just upgraded to o1, they would do better there as well. Um, okay. Then you wanted to also cover, uh, code execution.

I think, I think you, I think we covered this, right? Uh, mostly, mostly done, uh, done fine. Uh, you wanted to talk, talk about computer use.

Yeah. Computer use, um, actually it's a big step as well. So if you look at the, at the GAIA scores that we have, um, I think one of the big differences that we have with others... So, uh, first, why do I talk always about GAIA?

It's because I think it's a really good benchmark for general assistance. Like, if you look at the questions, it's really... Well, it's a kind of, um, strange way to ask questions, but it's really questions that you could come across, uh, in your daily life.

Um, this kind of difficulty of questions is what you could s- stumble on. So basically if you have an agent on GAIA that works, uh, and that solves these tasks, that means you have an agent that can basically solve most of the 10 minutes tasks that you have to do on your computer.

And that means if you have one agent like this, you can, um, basically submit tasks to two agents in parallel, and you can mostly double your productivity compared to the human who also spends, yeah, 10 minutes doing these tasks.

So I think if we solve GAIA, that's like 90% score, uh, that means mostly we double productivity of every tasks done in, in front of a computer.

Mm.

And if you, if you take the trend line of the scores so far, uh, this should be crossed in 2026 or something.

Okay. How, how do you see this trend line? You just, you have some chart somewhere?

Basically, uh, no, I didn't. I have to share it on Twitter. Um, I did a sigmoid basically- ... fitted on all the scores through time.

Amazing.

And, and you should find that.

You should share that. Yeah.

Yeah. Yeah, I have to tweet it.

Okay. So, so right now we're at 65 is SOTA for test and 70 for validation. Uh, and you say we'll, we'll, we'll, we'll hit 90 by, uh, by 2026. Okay. I mean, that's fair. I, I, I will-

Yes

... I'll probably say it's probably gonna happen earlier than that. Uh, did, did Operator, the OpenAI Operator mention anything on, on this? No, right? Uh.

Um, well, so what's interesting with them s- so they, yeah, they didn't submit for test, but they have these good validation scores, and I think this is mainly due to having a very good web browser because they have Operator, which, um, which, yeah, seems really good.

I haven't benchmarked it thoroughly yet, but it, it seems really amazing. Um, and I think for most of these tasks, well, any real-life task, you need a good web browsing at some point. Web browsing is designed for humans, so that means it's really visual.

Uh, and so a web browsing agent, a good one should use, in my opinion, um, a vision model and perform actions with point-and-click and keyboard, basically. So yeah, I think the way they went with this is really the way forward, and that's also why that's the next step for us at Hugging Face.

We're going to, to push really hard on building GUI agents, so basically agent that can use any GUI.

What, um, browser are you gonna use? What, what technical choices do you have to make?

Um, so until now, there's lots of people who, um, constrain the GUI space to only use browser because that's easier, right? If you have to access to the DOM, to the, to the code, to the source code of the web page, you can basically, um, do RAG on that.

You can... Like, it can help you. But I think what's even more interesting is the more general problem of using any GUI, so not depending on any browser, uh, and just, yeah, let the agency rec- uh, recording on your screen and act on that.

By the way, I think, I think there's-- they call- trying to call it CUA, not GUI in, in terms of like the, the terminology. Um-

Oh, okay

... I think, I think just borrowing from Anthropic calling it computer use, uh, OpenAI internally. When, when I talked to Karina, she called it CUA as well. Uh, it seems to be one of the terms that is being adopted.

Um-

Yeah

... Alessio, any, any, uh, any other thoughts?

Uh, I think it depends also if you-- yeah, if you want to restrict yourself to computer or, um, or basically any graphical u- uh, user interface, uh, which might be more general, but I don't know if they're restricting themselves to computer only.

Huh. Yeah.

I see.

Usually it's a virtual machine, you know? Like a-

Yeah

... like a Linux instance or something. Um.

Yeah.

I saw, um, you know, I, I saw Morph Labs. Um, they have, they have this, uh, time travel thing. Um, uh, I forget what they, what they called it, but it was called like Morph Branching or something, um, where they, they offered an API for computer use.

Um, so it's-- oh, they call it Infinite Branch. Um.

Yeah, that's really cool.

So, so, uh, so this is all-- each of these are individual computers that have been spun up inde-independently. Um, and then they were able to show some kind of, um... I think if I click in here. Um, yeah, they were, they're able to show some kind of, um, uh, time travel on, on this, on, on, on the, on the VMs that they, that they created.

That's really nice.

Yeah.

'Cause-

Yeah, here it is

... I, I think one of the key bottlenecks is the data on this. Like, how do you collect many, um, user interaction traces? And if you can spawn thousands of parallel agents that just try stuff, I think it might be really good to generate synthetic data.

Yeah. Well, um, you know, that, that's, that's, that's, that's what they're promising. I, I, I don't feel like it's got much traction yet, but it, it probably is a, a little bit early for this, this kind of stuff.

Mm.

But, uh, yeah, you know, people basically want to-- just like an operator, people want to run the machine, but then they also wanna take human control. Uh, it seems like they've, they've enabled this somehow.

Yeah.

Um, yeah. Cool.

Yeah.

Well, uh-

I think it's cool.

Yeah.

Um, yeah, I, I think it's also an interesting problem in general. Uh, how do you think-- 'cause like for now we have multi-step agents, but I feel like there's a breakthrough to, to find, uh, about doing more than this.

Like, how do you have an agent that's not only you give it a task and it performs the thing, then it returns the task, but how do you have an agent, for instance, that thinks continuously about what it could do throughout the day, uh, and interacts with you optimally to, to really help you?

I feel like, yeah, we have more to find.

Do you think-- you know, everybody says 2025 is the year of the agents. Do you think that's mostly a UX problem, or do you think, uh, there's still, like, fundamental limitations, like in the models?

I think there's also-- yeah, you're right. I think there's still fundamental limitations because, like, for instance, if you take a VLM at present, it's quite dumb. Like, if you take the same webpage and you g- you feed the, the text as marked down to, uh, LLM versus you feed the page screenshot to a VLM, the VLM just acts dumber.

Like, it, it says-- like, it makes dumb, dumber mistakes. So I feel like there's some progress to do. But as with text agents, we've really found that we made a jump on the S-curve with reasoning models. I think it's going, gonna be the same with the, the next, uh, visual models.

Basically, better base models just allow you to, to jump over this S-curve, and probably I think it's gonna be this year.

Everything's super exciting. January was, was already super exciting with, uh, DeepSeek, um, blowing it wide open. Yeah. Well, we were talking to someone else on S1 later as well. All the R1 derivatives I think are, are happening as well.

So, um, yeah. Well, thanks for coming on. Uh, any other parting thoughts, uh, shameless plugs, call to action?

What do I have? Yeah. Well, if anyone wants to come and contribute to smol agents, we're, we're welcoming everyone and, uh, all forces, uh, to, to join together and to, to make open agents, uh, a, a great thing.

Looking forward to this. And for instance, uh, we're going to fine-tune R1 on agentic stuff, so we'll, we'll get really good powerhouse models soon. So looking forward to, to this.

Yeah, looking forward too. Okay. Well, cool.

Thanks.

Thanks so much, Aymeric.

Yeah, thanks.