# smol agents are all you need Latent Space · 2025-02-13 This episode features Aymeric Roucher of Hugging Face discussing smol agents, a lightweight agent framework, and argues that code-based agents outperform JSON-based agents. Roucher explains that agents are defined by LLM control over execution flow, and smol agents simplifies this with a core file under 1000 lines. He describes the origin from transformers.agents and the security-focused custom Python interpreter. The library achieved third place on the GAIA benchmark, using o1 and Sonnet models, and Roucher projects 90% accuracy by 2026 based on a sigmoid trend line. Hugging Face is launching an agent course and plans to build GUI agents for computer use, while also fine-tuning R1 for agentic tasks. ## Questions this episode answers ### What is smol agents and why does it prefer code agents over JSON agents? Smol agents is a minimalist HuggingFace agent library with its core under 1,000 lines. Aymeric Roucher explains its key design choice: code agents write actions in code rather than JSON. This gives precise control over loops and variables, inspired by the CodeAct paper. For example, a for loop can run multiple tool calls in one step instead of many JSON actions. [2:46](https://addtry.com/c4d67891-a079-4dc1-b86d-176ff5872bda?t=166000) ### What is the GAIA benchmark and what results has smol agents achieved? GAIA (General AI Assistance) is a benchmark for agents solving real-world tasks of 10–60 minutes. Aymeric's team placed third on the validation leaderboard with o1, showing strong level-three performance. He notes others use costly majority voting. He estimates a 90% GAIA score would double human productivity for computer tasks, and the trend line points to this by 2026. [11:34](https://addtry.com/c4d67891-a079-4dc1-b86d-176ff5872bda?t=694000) ### What is the Hugging Face agent course and how popular has it been? The Hugging Face agent course is a free educational program that teaches agent fundamentals by building from scratch before introducing frameworks. Aymeric Roucher, who is supporting the course, says it offers certification and covers diverse use cases. The course generated enormous interest: at one point the sign-up website broke, and it attracted far more than the typical 10,000 subscribers seen by past Hugging Face courses. It started on the day of this recording. [9:08](https://addtry.com/c4d67891-a079-4dc1-b86d-176ff5872bda?t=548000) ## Key moments - **[0:00] Intro** - **[0:21] Defining Agents** - [0:21] Q: What is an AI agent, and how does it differ from a regular LLM application? - **[2:20] smol agents** - [2:46] Aymeric Roucher spent his Christmas break building smol agents out of frustration with complex agent libraries. - **[4:00] Code Agents** - [5:01] The CodeAct paper shows code agents can run parallel for-loops efficiently, avoiding the 20-step JSON actions needed for the same task. - **[6:55] Team History** - [6:55] smol agents evolved from transformers.agents, reusing a custom Python interpreter for secure multi-step code execution. - **[9:08] Agent Course** - [9:08] Hugging Face's free AI agents course attracted so many sign-ups it broke the registration website. - **[11:34] GAIA Benchmark** - [11:50] GAIA benchmark evaluates general AI assistants on open-world tasks that take humans 10-60 minutes to complete. - **[13:12] Validation Scores** - [13:34] smol agents ties Meta on GAIA validation at 70% but outperforms it significantly on the most difficult level 3 tasks. - **[15:20] Computer Use** - [15:59] Prediction: Solving GAIA at 90% by 2026 will double productivity for all 10-minute computer tasks, says Aymeric Roucher. - [17:04] OpenAI Operator's high GAIA scores come from its superior visual web browsing using a vision model, not just DOM access. - [19:07] Morph Labs' Infinite Branch spawns parallel VMs to collect synthetic computer-use traces, tackling the data bottleneck. - **[20:48] Year of Agents?** - [20:59] 'If you take a VLM at present, it's quite dumb… the VLM just acts dumber than an LLM on the same page.' — Aymeric Roucher - [21:30] Prediction: Reasoning models will trigger a jump on the S-curve for visual agents, similar to what they did for text agents. - **[21:53] Outro** - [21:53] smol agents welcomes open-source contributors and will fine-tune DeepSeek R1 for agentic tasks. ## Speakers - **Alessio** (host) - **Marcus Swicks** (host) - **Aymeric Roucher** (guest) ## Topics Agent Platforms ## Mentioned Anthropic (company), DeepSeek (company), Hugging Face (company), Meta (company), OpenAI (company), Agent course (product), Claude (product), Deep Research (product), E2B (product), Gaia (product), OpenDeepResearch (product), Operator (product), o1 (product), r1 (product), smol agents (product) ## Transcript ### Intro **Alessio** [0:03] Hey everyone, welcome to another Lightning Pod. Uh, I'm Alessio, partner and CTO at Decibel, and I'm joined by Marcus Swicks, founder of Smol.ai. **Marcus Swicks** [0:11] Hey, and today it's a very beautiful smol episode because we have the creator of smol agents, Aymeric from Hugging Face. Welcome. **Aymeric Roucher** [0:18] Hi everyone. Thanks for having me. **Marcus Swicks** [0:20] Uh, Aymeric, what is an agent? **Aymeric Roucher** [0:21] So that's, that's a really good question. Like, everyone's talking about it, but, uh, I feel that no one has a clear definition. Uh, I think really good, um, essay in the direction of a definition is given by what you have your screen on currently. ### Defining Agents **Aymeric Roucher** [0:35] So this essay, "What Is An AI Agent?" written by Harrison Chase, uh, from LangChain. So I think Harrison is one of the people who have thought the most about it for, for the longest time, so he has quite a good definition. Uh, he says it's not perfect, but here it goes. So basically, um, you, you struggle when you have a, an LLM, uh, because the knowledge it has is basically what it has seen during its training, but highly compressed. So if you want to perf-- uh, if you want it to perform better, uh, on any task, you need to give it access to the outside world. So what you will do is build an agent, and basically agent, uh, an agent is an application where an LLM has some level of control over the execution flow of your app. So, uh, that could be a very low level. For instance, if you have an if/else where the whichever, uh, part of the switch you go to is determined in some parts by an LLM output, this is a router. This is some level of agency. Um, if you have a while loop of which the continuation depends on an LLM output, this is multi-step agent. If you execute a function based on an LLM output, this is function calling. So there's basically different layers of agency, and so that's where we see that a-- being agentic is not a one zero definition. It's more basically 50 shades. Uh, and so yeah, he draws this nice, um, uh, continuum of agency. So basically, when you have a program and you have an em- an LLM embedded with that that has some control over the, the, the execution flow of the app, uh, that's what I would say an agent is, if you want a rigorous definition. **Marcus Swicks** [2:10] Yeah. So you don't even necessarily need tool use, I guess. Um, but I guess tool use comes in, in, in part of it. Uh. **Aymeric Roucher** [2:16] Yeah. Tool use is so useful that, yeah, often comes. ### smol agents **Marcus Swicks** [2:20] Yeah. Often I use, uh, Lillian Wang's, um, agent post, um, where she has this sort of equation, uh, where she has-- it talks about memory, planning, action, tool use, m- you know, all, all the other stuff. Um, okay. But let maybe we'll go s-strict, uh, quickly into, uh, I guess you, you worked on smol agents. You announced it at the end of last year, December 31st. Uh, why were you working during Christmas? And give us the origin story of smol agents. **Aymeric Roucher** [2:46] Yeah. I spent the, the whole Christmas locked in working over, over that 'cause I felt this was a big thing. So basically, I started, uh, with the premise that whenever I was trying to work with an agent library, I found that often too complicated to work with. Um, so that was one of the pain points, and thus the small in smol agents, we wanted to build something really simple. Um, and second aspect of it was that a few people write, um, code to do code agents. So basically code agents, uh, or agents that write their actions in code. If, if you want, we can detail that later on. Uh, as opposed to what everyone else does, which is mostly JSON agents. So the combination of the two, doing very simple agents and doing agents, uh, that can act in code, uh, was that what drew me towards building smol agents. And really the, the, the driving feature of it is to make it very, very short and simple. So the, the main file in it, the agents.py, uh, file that we have at the core of the library is under 1,000 lines of code to be, um, yeah, very easily understandable. So yeah, if you want to, to take a look at it, it's really... And, and- **Marcus Swicks** [3:52] Yeah **Aymeric Roucher** [3:52] ... the goal is also to make the logic apparent in it, like you can read through the code and see how it works easily. So yeah, agents.py. **Marcus Swicks** [3:59] Oh, boy. ### Code Agents **Aymeric Roucher** [4:00] Exactly. **Alessio** [4:00] And what was the, uh, what did you see missing when you started building this? There's a lot of people trying to build agent frameworks. What were some of the, the design opinions, uh, that you had going in? **Aymeric Roucher** [4:11] Um, yeah. I think-- So if you, if you check in this file, you have this CodeAgent class, uh, which is really our go-to class, and you have the ToolCalling class. So the ToolCalling class is more the classic way of doing agents. It's basically if you plug into, um, OpenAI, uh, tool calling formats, um, what-- the, the way it works is that you send it a request, you tell it you have these tools that you can use, and it will spit you out, uh, basically a dictionary with one key, which is the tool name, and the other key, which is the tool arguments, and then you do this call. But, um, I found this is highly suboptimal, and that's why I've designed the other code agents, which is really like the, the core, um, opinion in the library is that code agents work better. So that's actually a very interesting paper called, uh, Code Actions: Uh, Do They Yield Better LLM Agents? **Marcus Swicks** [4:59] CodeAct. Yep. **Aymeric Roucher** [5:01] Yeah, exactly. **Marcus Swicks** [5:01] We covered this with, um, uh, Graham Neubig, who also runs Open Hands and, uh, Open Devin. **Aymeric Roucher** [5:09] Yeah. And I, I really love this paper. They, they have this great example, uh, of code agents versus JSON. So maybe you've gone through that already, but, um, yeah, this figure here, um, compares, uh, a workflow for a code agent where it can just run a for loop. In the for loop, it can do four different tool calls. And so in code, you can, um, the LLM can control very precisely how the parallel loops are run and in which sequence, uh, inside the, the-- each, each part of the loop everything's run. Versus if you had to do this in JSON actions, it would be like 20 different actions, and it would be really hard to parallelize. Plus, code has, uh, other advantages on top of that. For instance, if you want to define a variable with a JSON agent, good luck. Uh, with code, it's really easy. Well, you just assign, uh, that to a name. **Marcus Swicks** [5:54] Yeah, yeah. We actually had this, um, in, I think our, uh, 2024 agents, um, roundup, where we actually talked about CodeActs, and, uh, Graham, Graham actually talked about, uh, this as like the number one paper that he re-recommends for people. But also- **Aymeric Roucher** [6:10] Love **Marcus Swicks** [6:10] ... he has quite a few others that, that might be interesting for comparison between, uh, smol agents and, uh, O-All Hands or Open Hands. **Aymeric Roucher** [6:18] Mm-hmm. Yeah, really interesting. Yeah, I, I put a big plus one on his first paper. I have to look at the others and then **Marcus Swicks** [6:25] Yeah, we did an episode on this, uh, maybe like almost two months ago now. Okay. Um, yeah, anything else about smol agents? You know, I think, um, the, the, the thing that I got from-- Oh, you also have like this, this thing. Yeah. Uh, the thing that I got from, uh, smol agents was like, this was like an experiments. Um, you know, I, I never-- I didn't really think, uh, think much about it. I know, uh, Hugging Face has been working on small LM as well. Um, but smol agents, like is this part of a broader effort? Is this-- Is there a team behind this? Um, you know, what, what's the context? ### Team History **Aymeric Roucher** [6:55] Um, yeah, the, the idea was that we had transformers.agents, uh, for some time. Uh, it was basically the experimental version of smol agents, uh, that, um, this had started being built, uh, quite a while ago, actually by Lissandre, uh, which is our, our head of, um, open source, and Sylvain, uh, who's n-not here anymore. They, they had started this thing where you could basically have a code agent that could perform one step. And so I thought that this had potential because they had built from the ground up a new Python interpreter. Uh, basically making code agents, um, has big security, um, challenges. So they had built a Python interpreter, a custom one from the ground up, and I thought, "Okay, let's reuse that one, uh, make it multi-step, uh, make the state persisting between steps, and we'll have like basically a banger agent, uh, right there." So I, I, I started transformers.agents on that premise. I thought that it got really good performance, for instance, on the GAIA benchmark, um, which I, I tried, uh, to improve, uh, to improve on, uh, a lot. And then, uh, that was the, the groundwork for starting smol agents. And now we're a team of two already. Uh, we're recruiting two more, so yeah, it's growing really fast. **Marcus Swicks** [8:06] Yeah. Do you have any other-- I mean, you know, obviously the Python security issues are well known. Do you have any sandboxing tools that you experimented with and you liked? **Aymeric Roucher** [8:16] Um, yeah. So basically we're going with E2B for now. Um, we are- **Marcus Swicks** [8:20] Ooh **Aymeric Roucher** [8:20] ... building Docker s-support as well. Yeah. **Marcus Swicks** [8:23] I knew that. I knew that. Um, yeah. Uh, but like, you know, uh, is there, is there, is there any, any reason E2B over like other, other tools? You know, just, just for my info. **Aymeric Roucher** [8:32] Um, basically we discussed a bit with the E2B guys, and it was like the, the start is really quick. It seems well supported. S-it seems like a robust solution, so let's go. Basically we, we don't think very much when we, when we have a new feature to-- We, we take the simplest route that will give us some sort of thing and we, we just, yeah, lock in and, and build that. **Marcus Swicks** [8:50] Yeah. I m- I mean, I think it's easier to switch, right? Like if you, if you need, uh, some other execution environment and, you know, fair-- full disclosure, me and Alessio are also investors in E2B. Uh, Alessio is slightly bigger than me. Uh Uh, okay. So we, we talked about this. Um, you also worked on a course, um, right? **Aymeric Roucher** [9:08] Yep. ### Agent Course **Marcus Swicks** [9:08] Um, yeah. **Aymeric Roucher** [9:10] So this is, um, um, another team at Hugging Face, so, uh, I'm working on it with them, but I'm, I'm more supporting on this. So they have launched the agent course, uh, which is like it hasn't gone crazy big like past-- The courses we did in the past was like 10K subscribers, which is already nice, but this one completely broke the-- At, at some point it broke the website that was used to do the, to, to, to-- for people to sign up. So- **Marcus Swicks** [9:34] What, what is this? I, I couldn't see the website. What, what-- People say 10K. I was like, "Is it 10K stars?" And then it's not 10K stars, so I don't know what, what, what- **Aymeric Roucher** [9:41] Um, where is that? No, it's not Smolcourse. Uh, it's, uh, Agentcourse. I think the, the link to sign in is on the website. Um- **Marcus Swicks** [9:51] I see. Anyway, I, I can look it up later, but I, I was just like very surprised. This one, I guess. Um- **Aymeric Roucher** [9:58] Yeah, exactly. **Marcus Swicks** [9:59] I see, I see. Okay. **Aymeric Roucher** [9:59] And yeah, so- **Marcus Swicks** [10:01] People really want the courses. **Aymeric Roucher** [10:02] Now they have like- **Marcus Swicks** [10:02] Oh, it starts today **Aymeric Roucher** [10:03] ... sign-ups. **Marcus Swicks** [10:04] Oh. **Aymeric Roucher** [10:04] And yeah. **Marcus Swicks** [10:04] Okay. It starts today. Yeah. Okay, amazing. Okay, cool. **Aymeric Roucher** [10:06] And I think it's going to be really interesting because they start from the basics, like really what is an agent, how to code that from, from scratch, and then see a few f-frameworks, but after that. 'Cause I feel like people often when trying to learn about agents, they often start with a predefined framework, and they go into the, the weeds, the specifics of that framework. Whereas if you want to learn that, it's always good to try to code a, a bit from scratch yourself, I think. **Marcus Swicks** [10:28] Okay. This-- Is this-- This is all Hugging Face people. **Aymeric Roucher** [10:31] Yeah, exactly. **Marcus Swicks** [10:33] Wow. You guys spend a lot of time on this. It's crazy. That's r- That's very impressive. **Aymeric Roucher** [10:36] Yeah, yeah. **Marcus Swicks** [10:36] Um, okay. Um, so people can sign up for the course and, and go through the, the material. Um, anything else to, to note before we, we move on to, you know, the other two stuff? **Aymeric Roucher** [10:45] On the course, uh, what's in there? Yeah, there's a certification for people who want to, to show something on their resume after. And what else? Yeah, we're, we're doing really nice modules on this, uh, with, uh, with very varied use cases, so yeah, I think it's, it's going to be really good. As you see, yeah, there's quite a lot of, uh, effort. At Hugging Face scale, we tend to do things with very small teams, so this is a big effort. **Marcus Swicks** [11:05] This is big for Hugging Face, yes. I agree with this. Uh, okay. So one of the things that people are talking about in surprisingly o3 adopted, or actually not even o3, Deep Research adopted was GAIA. Um, oh, actually you guys also have open Deep Research, right? That's just a different team. Um- **Aymeric Roucher** [11:22] Yeah. Exactly **Marcus Swicks** [11:22] ... anyway, I think, you know, the, the French Hugging Face mafia, uh, is, is running GAIA, and we've covered GAIA briefly before, but I'm sure most people don't know-know what GAIA is. Uh, maybe you want to introduce GAIA. **Aymeric Roucher** [11:34] Yeah. So GAIA, so the, the name is, uh, General AI Assistance, um, Benchmark, so that would be GAIB. But, uh, yeah, I don't know where the A comes from at the end. Um, so basically it's, uh, evaluating agents that are as general as possible in helping you. ### GAIA Benchmark **Aymeric Roucher** [11:50] Uh, so the idea is to have, uh, an open world environment, so basically, uh, agents that can browse the internet. There's no sandboxed environment as with many benchmarks. Um, and the questions are very general in nature, and it's aimed at tasks that would take a human ten minutes on average to solve. So it's like-- Actually, it's a bit more than that. It's 10 to 60 minutes. And yeah, this is a good example of, uh, of question. So basically you can show a picture, um, and you need an agent that's able to look at the picture, find a number, uh, do a web search, do rigorous calculations over several steps. So basically most questions take something like fif- five to 10 steps, so the agent must be really rigorous at that. Uh, there's one true answer. Uh, and also a good thing is that the test sets, the answers are hidden. Like, no one has access to them except the team, so not too much con- contamination hopefully. **Marcus Swicks** [12:42] Yeah. **Aymeric Roucher** [12:42] And maybe that's why OpenAI Deep Research didn't disclose the, the results on the test set. **Marcus Swicks** [12:47] Uh, I, I, I don't know. Uh, did they, did they even try? Did they submit it to you guys? **Aymeric Roucher** [12:51] No, no, no, they didn't submit anything. They just submitted- **Marcus Swicks** [12:53] Yeah **Aymeric Roucher** [12:54] ... on their own. **Marcus Swicks** [12:54] They just, um... Yeah, Clementine was, was very, uh, pointedly saying that, "You know, you guys only submitted on the val set." Like **Aymeric Roucher** [13:02] Yeah. **Marcus Swicks** [13:03] Um, okay. **Aymeric Roucher** [13:04] But I'm afraid she will hurt me if I don't submit the, the results for OpenDeepS- uh, uh, Open, uh, Deep Research on the, on the test set as well. **Marcus Swicks** [13:11] Yes. **Aymeric Roucher** [13:11] I'm running them right now at the moment. **Marcus Swicks** [13:12] I see. I see. So, so you are doing OpenDeepResearch with smol agents, and you don't have results yet? ### Validation Scores **Aymeric Roucher** [13:18] Yeah, exactly. Like, we have result- uh, results on the validation set- **Marcus Swicks** [13:21] Okay **Aymeric Roucher** [13:21] ... and I'm running them on the test set at the moment. **Marcus Swicks** [13:23] Okay. How are they looking on the validation set? **Aymeric Roucher** [13:25] Uh, validation set, we are third. Um, yeah, that, that's the, on the other tab, the, the result for, uh, validation. **Marcus Swicks** [13:32] Oh, sorry. Yeah. Okay. Yeah. **Aymeric Roucher** [13:34] Exactly. **Marcus Swicks** [13:35] Okay. Here it is. **Aymeric Roucher** [13:36] Oh, Meta's just submitted. How are they on top of, of us? They submitted after. It's unfair, and they have the same score. Man. Okay. Okay. **Marcus Swicks** [13:46] So, so, so you basically you take the average of level one, two, and three, right? Okay. **Aymeric Roucher** [13:50] Yeah, exactly. That's, uh, the, the- **Marcus Swicks** [13:51] But your level three is so much higher than theirs. Come on. **Aymeric Roucher** [13:54] Yeah. Man, should be counted for more. **Marcus Swicks** [13:56] Okay, great. **Aymeric Roucher** [13:57] Man. **Marcus Swicks** [13:58] It's interesting that you got the exact same level two. Yeah, go ahead. **Alessio** [14:01] Yeah, I was gonna say, I'm sure people are gonna wonder, you have the exact same level two score, but then there's such a big drop-off at level three. Do you have any intuition as to why? Is it just because of like o1 is better than Sonnet or, um... Yeah. **Aymeric Roucher** [14:14] I was impressed by o1, uh, honestly on, on, uh, agent tasks. So GPT-4 performed well with smol agents already, and when I tried o1, I was, um, yeah, kind of baffled by how good the model is. I tried R1, but R1 is a bit under, uh, o1, uh, with smol agents, and I think this is also a matter of formatting. Like, sometimes the model struggles to just output the, the code snippets in the correct way that we expect. So if we fine-tune it, it, it should probably get better. But yeah, maybe so higher scores on level three might be linked to us using, uh, o1. **Marcus Swicks** [14:44] Okay. **Aymeric Roucher** [14:45] And also some other- **Marcus Swicks** [14:45] I don't, I don't know what Meta uses. Sorry? **Aymeric Roucher** [14:48] Yeah. And also an important thing here is that, uh, many submissions use majority voting, um, which is, well, uh, well, it, it does improve- **Marcus Swicks** [14:57] Cheating **Aymeric Roucher** [14:57] ... scores, but at the... Yeah. It's at the price of higher cost. **Marcus Swicks** [15:01] Yeah. Uh, okay, yeah. So yeah, it looks like they use, uh, Claude 3.5, uh, and o1. Maybe if they just upgraded to o1, they would do better there as well. Um, okay. Then you wanted to also cover, uh, code execution. I think, I think you, I think we covered this, right? Uh, mostly, mostly done, uh, done fine. Uh, you wanted to talk, talk about computer use. **Aymeric Roucher** [15:20] Yeah. Computer use, um, actually it's a big step as well. So if you look at the, at the GAIA scores that we have, um, I think one of the big differences that we have with others... So, uh, first, why do I talk always about GAIA? ### Computer Use **Aymeric Roucher** [15:33] It's because I think it's a really good benchmark for general assistance. Like, if you look at the questions, it's really... Well, it's a kind of, um, strange way to ask questions, but it's really questions that you could come across, uh, in your daily life. Um, this kind of difficulty of questions is what you could s- stumble on. So basically if you have an agent on GAIA that works, uh, and that solves these tasks, that means you have an agent that can basically solve most of the 10 minutes tasks that you have to do on your computer. And that means if you have one agent like this, you can, um, basically submit tasks to two agents in parallel, and you can mostly double your productivity compared to the human who also spends, yeah, 10 minutes doing these tasks. So I think if we solve GAIA, that's like 90% score, uh, that means mostly we double productivity of every tasks done in, in front of a computer. **Marcus Swicks** [16:20] Mm. **Aymeric Roucher** [16:20] And if you, if you take the trend line of the scores so far, uh, this should be crossed in 2026 or something. **Marcus Swicks** [16:27] Okay. How, how do you see this trend line? You just, you have some chart somewhere? **Aymeric Roucher** [16:31] Basically, uh, no, I didn't. I have to share it on Twitter. Um, I did a sigmoid basically- ... fitted on all the scores through time. **Marcus Swicks** [16:39] Amazing. **Aymeric Roucher** [16:39] And, and you should find that. **Marcus Swicks** [16:40] You should share that. Yeah. **Aymeric Roucher** [16:41] Yeah. Yeah, I have to tweet it. **Marcus Swicks** [16:44] Okay. So, so right now we're at 65 is SOTA for test and 70 for validation. Uh, and you say we'll, we'll, we'll, we'll hit 90 by, uh, by 2026. Okay. I mean, that's fair. I, I, I will- **Aymeric Roucher** [16:56] Yes **Marcus Swicks** [16:56] ... I'll probably say it's probably gonna happen earlier than that. Uh, did, did Operator, the OpenAI Operator mention anything on, on this? No, right? Uh. **Aymeric Roucher** [17:04] Um, well, so what's interesting with them s- so they, yeah, they didn't submit for test, but they have these good validation scores, and I think this is mainly due to having a very good web browser because they have Operator, which, um, which, yeah, seems really good. I haven't benchmarked it thoroughly yet, but it, it seems really amazing. Um, and I think for most of these tasks, well, any real-life task, you need a good web browsing at some point. Web browsing is designed for humans, so that means it's really visual. Uh, and so a web browsing agent, a good one should use, in my opinion, um, a vision model and perform actions with point-and-click and keyboard, basically. So yeah, I think the way they went with this is really the way forward, and that's also why that's the next step for us at Hugging Face. We're going to, to push really hard on building GUI agents, so basically agent that can use any GUI. **Marcus Swicks** [17:54] What, um, browser are you gonna use? What, what technical choices do you have to make? **Aymeric Roucher** [17:59] Um, so until now, there's lots of people who, um, constrain the GUI space to only use browser because that's easier, right? If you have to access to the DOM, to the, to the code, to the source code of the web page, you can basically, um, do RAG on that. You can... Like, it can help you. But I think what's even more interesting is the more general problem of using any GUI, so not depending on any browser, uh, and just, yeah, let the agency rec- uh, recording on your screen and act on that. **Marcus Swicks** [18:26] By the way, I think, I think there's-- they call- trying to call it CUA, not GUI in, in terms of like the, the terminology. Um- **Aymeric Roucher** [18:32] Oh, okay **Marcus Swicks** [18:33] ... I think, I think just borrowing from Anthropic calling it computer use, uh, OpenAI internally. When, when I talked to Karina, she called it CUA as well. Uh, it seems to be one of the terms that is being adopted. Um- **Aymeric Roucher** [18:45] Yeah **Marcus Swicks** [18:45] ... Alessio, any, any, uh, any other thoughts? **Aymeric Roucher** [18:47] Uh, I think it depends also if you-- yeah, if you want to restrict yourself to computer or, um, or basically any graphical u- uh, user interface, uh, which might be more general, but I don't know if they're restricting themselves to computer only. **Marcus Swicks** [19:00] Huh. Yeah. **Alessio** [19:01] I see. **Marcus Swicks** [19:01] Usually it's a virtual machine, you know? Like a- **Alessio** [19:03] Yeah **Marcus Swicks** [19:04] ... like a Linux instance or something. Um. **Aymeric Roucher** [19:06] Yeah. **Marcus Swicks** [19:07] I saw, um, you know, I, I saw Morph Labs. Um, they have, they have this, uh, time travel thing. Um, uh, I forget what they, what they called it, but it was called like Morph Branching or something, um, where they, they offered an API for computer use. Um, so it's-- oh, they call it Infinite Branch. Um. **Aymeric Roucher** [19:25] Yeah, that's really cool. **Marcus Swicks** [19:27] So, so, uh, so this is all-- each of these are individual computers that have been spun up inde-independently. Um, and then they were able to show some kind of, um... I think if I click in here. Um, yeah, they were, they're able to show some kind of, um, uh, time travel on, on this, on, on, on the, on the VMs that they, that they created. **Aymeric Roucher** [19:47] That's really nice. **Marcus Swicks** [19:48] Yeah. **Aymeric Roucher** [19:48] 'Cause- **Marcus Swicks** [19:48] Yeah, here it is **Aymeric Roucher** [19:49] ... I, I think one of the key bottlenecks is the data on this. Like, how do you collect many, um, user interaction traces? And if you can spawn thousands of parallel agents that just try stuff, I think it might be really good to generate synthetic data. **Marcus Swicks** [20:01] Yeah. Well, um, you know, that, that's, that's, that's, that's what they're promising. I, I, I don't feel like it's got much traction yet, but it, it probably is a, a little bit early for this, this kind of stuff. **Aymeric Roucher** [20:11] Mm. **Marcus Swicks** [20:11] But, uh, yeah, you know, people basically want to-- just like an operator, people want to run the machine, but then they also wanna take human control. Uh, it seems like they've, they've enabled this somehow. **Aymeric Roucher** [20:19] Yeah. **Marcus Swicks** [20:20] Um, yeah. Cool. **Aymeric Roucher** [20:21] Yeah. **Marcus Swicks** [20:21] Well, uh- **Aymeric Roucher** [20:23] I think it's cool. **Marcus Swicks** [20:24] Yeah. **Aymeric Roucher** [20:24] Um, yeah, I, I think it's also an interesting problem in general. Uh, how do you think-- 'cause like for now we have multi-step agents, but I feel like there's a breakthrough to, to find, uh, about doing more than this. Like, how do you have an agent that's not only you give it a task and it performs the thing, then it returns the task, but how do you have an agent, for instance, that thinks continuously about what it could do throughout the day, uh, and interacts with you optimally to, to really help you? I feel like, yeah, we have more to find. ### Year of Agents? **Alessio** [20:48] Do you think-- you know, everybody says 2025 is the year of the agents. Do you think that's mostly a UX problem, or do you think, uh, there's still, like, fundamental limitations, like in the models? **Aymeric Roucher** [20:59] I think there's also-- yeah, you're right. I think there's still fundamental limitations because, like, for instance, if you take a VLM at present, it's quite dumb. Like, if you take the same webpage and you g- you feed the, the text as marked down to, uh, LLM versus you feed the page screenshot to a VLM, the VLM just acts dumber. Like, it, it says-- like, it makes dumb, dumber mistakes. So I feel like there's some progress to do. But as with text agents, we've really found that we made a jump on the S-curve with reasoning models. I think it's going, gonna be the same with the, the next, uh, visual models. Basically, better base models just allow you to, to jump over this S-curve, and probably I think it's gonna be this year. **Marcus Swicks** [21:38] Everything's super exciting. January was, was already super exciting with, uh, DeepSeek, um, blowing it wide open. Yeah. Well, we were talking to someone else on S1 later as well. All the R1 derivatives I think are, are happening as well. So, um, yeah. Well, thanks for coming on. Uh, any other parting thoughts, uh, shameless plugs, call to action? ### Outro **Aymeric Roucher** [21:53] What do I have? Yeah. Well, if anyone wants to come and contribute to smol agents, we're, we're welcoming everyone and, uh, all forces, uh, to, to join together and to, to make open agents, uh, a, a great thing. Looking forward to this. And for instance, uh, we're going to fine-tune R1 on agentic stuff, so we'll, we'll get really good powerhouse models soon. So looking forward to, to this. **Marcus Swicks** [22:14] Yeah, looking forward too. Okay. Well, cool. **Aymeric Roucher** [22:15] Thanks. **Marcus Swicks** [22:16] Thanks so much, Aymeric. **Aymeric Roucher** [22:16] Yeah, thanks. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.