# ⚡ Open Model Pretraining Masterclass — Elie Bakouch, HuggingFace SmolLM 3, FineWeb, FinePDF Latent Space · 2025-10-20 Elie Bakouch, Hugging Face's pre-training lead and architect behind SmolLM, presents a five-pillar framework for model training spanning data quality, architecture, information extraction, gradient quality, and stability, while breaking down recent innovations in optimizers, Mixture of Experts, and data rephrasing. He explains that the field of optimizers is moving beyond AdamW with Muon and Shampoo, noting that speed-ups are often exaggerated due to undertuned baselines. He deep-dives into MoE architecture, showing that expert specialization requires load balancing at the global batch level, as demonstrated by Qwen's findings. He discusses the rephrasing revolution, where converting low-quality web data into QA format yields non-random MMLU performance even for small models. Elie also highlights Hugging Face's open science contributions: FinePDF (new PDF dataset), FineWeb-Edu2, and tools like Nanotron, Datahub, and Liteval. ## Questions this episode answers ### Why did SmolLM2 achieve only random accuracy on MMLU even after trillions of tokens? Elie Bakouch explains that SmolLM2's pre-training data lacked question-answer formatted examples, so the model couldn't learn the required output style. Performance on MMLU remained random up to 6.5 trillion tokens. Once they added conversational QA-style data, accuracy improved, demonstrating that simple format matching—not just data quality—is critical for benchmark performance. [43:23](https://addtry.com/cf9012bf-8240-48ff-97a2-25fcbf4bd1f8?t=2603000) ### How can I force experts in a Mixture of Experts model to specialize by domain like code or math? Elie Bakouch notes that Qwen found computing load-balancing statistics at the global batch level—aggregating across all GPUs—rather than at the local micro-batch level leads to clear expert specialization. When done globally, experts activate for distinct domains (code, math, literature). Without this, experts remain undifferentiated, harming the model's ability to learn domain-specific features. [23:15](https://addtry.com/cf9012bf-8240-48ff-97a2-25fcbf4bd1f8?t=1395000) ### Is Muon really a faster optimizer than Adam for LLM training? Elie Bakouch explains Muon uses Newton-Schulz iterations to orthonormalize weight matrices, improving stability and learning. But a recent paper found speedups over Adam are exaggerated when Adam is properly tuned. He notes DeepSeek V3 still uses Llama 2’s Adam hyperparameters, indicating optimizer innovation is far from settled. [10:17](https://addtry.com/cf9012bf-8240-48ff-97a2-25fcbf4bd1f8?t=617000) ## Key moments - **[0:00] Intro** - [0:46] Swyx calls SmolLM 3 "one of the single best open source AI research papers I've read" - **[1:10] Research Team** - [1:11] Elie Bakouch leads pre-training efforts at Hugging Face and is the key architect behind SmolLM - [2:13] Hugging Face’s FinePDF dataset exploits underexplored PDF data source, boosting mixed-dataset training performance - **[4:20] Unified View** - [4:20] Elie Bakouch outlines five pillars of model training: data quality, model architecture, information extraction, gradient quality, and training stability - [5:09] "Garbage in, garbage out" is crucial for model training, says Elie Bakouch - [6:19] Multi-token prediction is very good for coding, as it maximizes information extraction per step - [8:54] DeepSeek V3 still uses the same Adam optimizer hyperparameters as Llama 2, indicating optimizer underexploration - **[10:17] Optimizer Innovation** - [10:53] Kimi K2 and recent papers are moving beyond AdamW, exploring new optimizers for large-scale training - [12:24] Muon optimizer uses Newton-Schultz method to orthogonalize the weight matrix, enabling more stable and exploratory learning - [16:27] A March 2025 paper finds claimed optimizer speed-ups are often exaggerated due to undertuned Adam baselines - [20:26] Hyperparameter sweeps should jointly optimize learning rate and weight decay to find optimal values - **[21:15] MoE Architecture** - [21:33] MoE models achieve 3x performance per FLOP but only 2x training speedup due to code overhead - [22:06] Qwen’s research: load balancing computed per micro-batch prevents MoE experts from specializing to domains like code or math - [24:33] Computing MoE load balancing statistics over the global batch yields clear domain-specific expert specialization - [27:17] LongCat Flash’s dual communication expert allocates fewer FLOPs to easy tokens and more to hard ones, a first in large models - [29:02] Qwen3 Next reverses Qwen3’s decision by adding a shared expert, indicating shared experts may help MoE specialization - **[29:26] Expert Specialization** - [30:55] Apple’s downloadable LoRAs and Gemma Nano’s offloadable weights signal a trend toward semantic experts - [32:13] OpenAI’s GPT-5 routing varies active expert count per query, akin to MoE, for efficient inference - [33:06] New MoE models like Qwen Next push sparsity much higher, with active parameters a fraction of total - **[33:09] Sparsity Trends** - [34:55] Granular MoE sparsity faces communication overhead and requires optimized training code bases to be feasible - [36:41] Frontier labs distill dense models into MoE for production rather than training MoEs from scratch - [38:20] DeepSeek’s NSA architecture introduces routing inside self-attention, selecting which blocks to attend per token - **[40:20] Data Quality** - [42:09] Rephrasing only low-quality web data can nearly match the performance gains of rephrasing all data - **[43:20] Small LM Insights** - [43:43] SmolLM 2’s MMLU accuracy was random for 6.5 trillion tokens due to missing Q&A-style training data - [46:46] Q: Could rephrasing make training data too clean for real-world typos? A: Possibly, include some original messy data - [50:41] Elie Bakouch: text rephrasing is analogous to image augmentation, enabling more data repetitions without overfitting - [52:15] GLM’s mid-training phase uses 8 trillion tokens of code and math, blurring the line with pre-training - [53:24] SmolLM 3 releases intermediate checkpoints every 50 billion tokens for full training transparency - **[53:56] Open Source Tools** - [54:00] Hugging Face open-sources Nanotron for large-scale training, Datahub for data filtering, and Liteval for evaluation - [56:01] A code bug caused a training run to spike overnight, a memorable debugging story from SmolLM 3 development - **[57:55] Future Directions** - [57:55] Elie Bakouch sees on-device translation and Apple’s AI push as key use cases for small language models - [58:48] The SmolLM team plans to train an MoE variant and explore distributed training across datacenters - [1:00:06] Alessio: lack of economical token-based APIs for sub-1B models hinders adoption of small language models - [1:01:56] Elie Bakouch: pre-training researchers avoid changing core frameworks, as iteration relies on stable code bases ## Speakers - **Alessio** (host) - **Swyx** (host) - **Elie Bakouch** (guest) ## Topics Language Models, Open Source Tools, Data Tools ## Mentioned DeepSeek (company), Google (company), Hugging Face (company), OpenAI (company), Datahub (product), FinePDF (product), FineWeb (product), FineWeb-Edu (product), GPT (product), Gemini (product), Gemma (product), Kimi (product), Liteval (product), Llama (product), Megatron (product), Muon (product), Nanotron (product), Qwen (product), SmolLM (product), Transformers (product) ## Transcript ### Intro **Alessio** [0:05] Hey, everyone. Welcome to the Latent Space Podcast. This is Alessio, founder of Kernel Labs, and I'm joined by Swyx from Cognition. **Swyx** [0:11] Yeah. **Alessio** [0:11] First time I introduce you at the new role. **Swyx** [0:13] We, we haven't, uh, figured out the official title yet. But yes, uh, uh, they-- we're recording this the day after we announced the Series C, and, uh, it's, uh, it's very good. Uh, you know, Alessio is also part of the Cognition family now. We all, you know, had a, had a fun party at the house. But isn't-- today is not about us. Today we're talking about research, SmolLM, model training, all that stuffs with Elie from Hugging Face. Welcome. **Elie Bakouch** [0:38] Yeah. Thanks, uh, thanks a lot for the invitation, and, uh, I'm very glad to, to be here. And congrats again for the, for this new term. **Swyx** [0:46] Uh, so you, you, uh, obviously, I think, uh, I actually first reached out about SmolLM. I think that, you know, obviously SmolLM 3, one of the single best open source AI research papers I've read, I think probably in, like, a year, maybe two years. Uh, I don't know. Like, yeah, it's very, very good. Uh, but, you know, I think, like, maybe let's, let's just get a high level, like, who are you- **Elie Bakouch** [1:08] Yeah **Swyx** [1:08] ... what do you do at Hugging Face, uh, all that stuff. ### Research Team **Elie Bakouch** [1:11] Yeah. So I think, like, most of the people know Hugging Face for, like, the, the hub or, like, the transformer library. And we also have, like, this research team where we, we basically try to explore in the open all the, the, the science subjects. It could be, uh, we have a team working on data, obviously, uh, with the FineWeb, uh, the recently released FinePDF, and we also have this team working on, uh, pre-training and training models such as SmolLM. Uh, and we are, we are basically a very small team of, uh, thirty people in average. And, uh, we try to, to do some cool stuff that the community will love and will use to build on top of it and, and to contribute to, to the open science basically. **Swyx** [1:52] Yeah. While, while you mentioned, uh, s- uh, the FinePDF, uh, why don't you, you plug FinePDF a little bit. I think a lot of people know FineWeb. **Elie Bakouch** [1:59] Yeah. **Swyx** [1:59] But, uh, what's, what's good about FinePDF? **Elie Bakouch** [2:02] Yeah. I think I have, uh, some- **Swyx** [2:05] Oh, okay. It's already in your slides. Okay. **Elie Bakouch** [2:07] Yeah, it's already, uh, somewhere. I think, yeah, it's yeah. **Swyx** [2:10] Oh, wow. **Elie Bakouch** [2:10] And, uh- **Swyx** [2:11] That's very recent. Okay. **Elie Bakouch** [2:13] Yeah. That's, uh... I mean, I updated it today, so, so... And, uh, so it's basically cool because I think that we, we have a lot of high quality dataset, and the field of dataset people have been building some, some awesome stuff. NVIDIA, uh, uh, Datalogy, they didn't release it yet, but this is pretty cool as well. Um, and one thing that was missing, uh, A2 also, of course, and one thing that was missing is, like, a new source of data, and PDF was, like, greatly underexplored. So that's what, uh, like, uh, our team did. And, uh, you can see, like, the performance is pretty impressive. Like, if you take this curve, uh, like when mixed with, uh, with like, uh, other, other web datasets, not PDF one, it basically have a very, very, very good performance compared to this like Nemo-Trans CV2, which is, uh, very recent release from NVIDIA. So it's pretty cool. And there is all the pipeline. So I didn't work on this. It's Eineck who mainly did a lot of great job on this. But there is all this pipeline where he benchmark a lot of, uh, PDF, uh, extractor, and, uh, we are cooking like a blog post where we'll explain basically, uh, all the, the step of the, this, uh, pipeline. **Swyx** [3:31] Yeah. Yeah. What, what is this refetch from internet step? What is that? Is that, like, a very common thing? **Elie Bakouch** [3:36] Uh, yeah. I think this is the-- so, like, the way- **Swyx** [3:41] To update it **Elie Bakouch** [3:41] ... it work is that- Yeah. So sometime, like, I think the PDF from Common Crawl are not really well extracted, basically. So I think what, like, he did, uh, here is that he refetch it from, uh, internet to maybe have like a better version of it, if it's like more recent or, I don't know. **Swyx** [3:59] Yeah. Yeah. Okay. **Elie Bakouch** [4:01] And there's a lot of it, uh- **Swyx** [4:03] Yeah. Uh, so obviously the, you know, you, you, you get datasets. Uh, I actually previously talked with, uh, Hugo Larochelle of the s- the vision team. **Elie Bakouch** [4:14] Yeah. **Swyx** [4:14] And, uh, Ida Fixen, Obelix was really, was really cool, uh, for, for vision as well. You are on the training side. Uh, what is, what is this- ### Unified View **Elie Bakouch** [4:20] Yeah **Swyx** [4:20] ... unified view of model training? **Elie Bakouch** [4:23] Yeah. So my-- Yeah, I didn't say that, but it's my job at, uh, Hugging Face is to be the guy that's, like, keep in touch with what's, like, the, the pre-training, uh, innovation are. And, like, my view of model training is basically you have, like, those different objectives that you-- that are somewhat orthogonal to each other, and you can, like, explore in parallel. The first one is, like, the-- is basically maximize the, the relevance of the, the raw information, meaning, like, maximize the, the, the quality and the diversity of the data. This is probably, like, the most important one. It's not really the, the one that people are often the, the most excited about, but I think it's pretty well known now that, like, there is this garbage in, garbage out, uh- **Swyx** [5:08] Yeah **Elie Bakouch** [5:09] ... thing that's-- it's basically super important to have very good data and, like, a very good diversity. But you can also optimize for other thing, and this is quite orthogonal to the dataset. So y- you need to have a team that build a very good dataset. You also need to have a team that do research on model archati- architecture to basically have the, like, uh, construct the best model possible under certain constraints, and the constraint are, like, the, the efficiency. So you want a model, uh, that will be, uh, good at, at inference, uh, that will, like, fit on a certain number of GPU, that will have a certain, like, KG cache, uh, consumption and thing like that. So this is like the, the, the model architecture part. And when you are-- it clicks like this, uh, dataset and model arch, you can also maximize the, like, the information you get from the data. So at each step, you basically want to maximize, like, how much you extract from the data. One way to do that is to do this locate distillation. Uh, there is other way to do that. It's also related to, to the, to the architecture part, but there is this multi-token prediction which is very good, for example, for coding. And yeah. And basically there is also other steps such as like maximize the quality of the gradient you get, meaning like when you, you do your update step, you need to, to maximize this, this quality of the, the, the dates. So this is like different optimizer, uh, thing that will approximate the, the action matrix, for example. Or just like, uh, project into data space for learning. And when you have all of this, you, you, you want to have basically the, the right hyperparameters to make sure that your, uh, your, like, your training is not exploding because it happened quite a lot, especially when you scale it just like your gradient or your activation are doing, like, completely diverging. There is all this field where you want it to be stable. You want to understand, uh, how to scale from a, a smaller model to a larger one and do this thing called feature learning, where basically you, you are at each step learning as much as possible from this, uh, gradient. So this is kind of my view of, uh, model training. There is of course, some, some connection between all of them, but, uh- **Alessio** [7:20] And, and for these five steps, how do you think about which ones have the highest variability? I feel like some of them there's kinda like a meta of this is how you get the best results. What are like- **Elie Bakouch** [7:31] Yeah **Alessio** [7:31] ... the parts that are like still very not figured out where, like, you feel like the most interesting research is being done? **Elie Bakouch** [7:36] So I think datasets is like the, the data is like-- is a bit unexplored because, uh, I think people don't find it very interesting. I mean, for, for a bit of time, like people were a bit underexploring like the dataset, especially since it's super important. Uh, but we, we've seen like recent paper and, uh, recent trend to build better and better dataset, and so I'm, I'm confident that like, uh, hopefully people in the open will somewhat catch up with like the, the, the big models. The rest of it, I think the model architecture, we, with like the Qwen, DeepSeek, uh, Kimi architecture, we're already at a point where this is already not optimal, but this is like pretty, pretty advanced. And there is all this, all this thing where basically you want to, to maximize, uh, like to, to, to do some new optimizer and to, to really, uh, have some good stability and scaling feature. This is a bit unex-underexplored and especially like since Qwen and DeepSeek paper doesn't really detail all, all, all they do the stuff. I mean, we've left with a very few, uh, model provider, espe- even in the open that talk about all of this. And for example, a good, uh, a good way to view that is that, uh, DeepSeek V3 is still using the same Adam parameter than, uh, Llama 2. I mean, there is something wrong with it. I don't know if it's just me, but I feel that like the, this, the, the hyperparameters for Llama 2 8B, for example, sh-shouldn't be the optimal one for, uh, this, uh, mega DeepSeek model with, uh, with it a lot of, uh, of parameter. **Alessio** [9:22] Yeah. **Swyx** [9:22] Yeah. I, I, I think like a lot of, uh-- it's almost weird, like I wonder if we've hit the peak big model, uh, craze, uh, because now I, do you expect, you know, ten trillion model releases, you know, one hundred trillion- **Alessio** [9:38] Yeah **Swyx** [9:38] ... model releases? Probably not. I think we might have peaked at two. Um- **Elie Bakouch** [9:42] Yeah. I mean, I'm always curious to know like Claude, uh, Claude Opus, like GPT-5, like what's the scale of those very big model? I'm not sure it's like that crazy compared to the Kimi 1 or Qwen that will probably release soon. **Swyx** [9:57] Yeah, both are-- Yeah, both are one trillion models. Uh, yeah. I mean- **Elie Bakouch** [10:00] Yeah **Swyx** [10:01] ... uh, all, all I will say is I, I have, I have no insider information, but I have a lot of rumors thrown at me, and it, it is in the range of like single digit trillions. Um- **Elie Bakouch** [10:11] Yeah. **Swyx** [10:11] So cool. Uh, I mean, and then the other, the other thing, uh, you know, I wanna-- we probably won't spend that much time on optimizer, but I think like optimizer- ### Optimizer Innovation **Elie Bakouch** [10:18] Yeah **Swyx** [10:18] ... is actually like one of the low-key more interesting trends this year. Is everyone moving to Muon? Uh, I don't know if, I don't know if you have a strong view on this. Like, uh, basically like a-all, all of these five- **Elie Bakouch** [10:27] Yeah **Swyx** [10:28] ... I don't know where you wanna spend the most time. **Elie Bakouch** [10:30] Uh, I really like all the optimizer parts. Uh, but that's- **Swyx** [10:34] Okay **Elie Bakouch** [10:34] ... like more like a, a personal, uh-- I mean, I will not prob-probably talk about this, uh, a lot, but that's one of the, the thing that I find cool because it's very recent basically, and it's like using, uh, uh, some-- I mean, it's very recently that people start, uh, using something different than Adam. Uh, we know for example, that like, uh, Google is using, uh-- I mean, we don't know, but like on like old paper, uh, Google are using like Adafactor, but recently before this, uh, Kimi K2, everyone was using, uh, Adam- **Swyx** [11:08] Yeah **Elie Bakouch** [11:08] ... AdamW. And uh, recently there was the GLM pap- uh, paper. There is this, um, Kimi K2, of course, that is trying to do some new stuff with optimizer, and I find it pretty cool because it's like very deeply influence the learning process of the, the model. Uh, this is actually quite a big of a change, I think. **Swyx** [11:27] Can you give an intuition on what's going on? Like, I think the basic insight of Adam, AdamW was mostly like momentum, right? Momentum care, uh- **Elie Bakouch** [11:34] Yeah **Swyx** [11:34] ... matters. You trade off, you use more memory and like, uh, but in exchange you get, you get much better learning. Like, okay, fine. What is the core insight that, uh, that everyone's sort of like exploring? **Elie Bakouch** [11:43] I think for, uh-- So I actually have, uh, some stuff about it somewhere. Somewhere along, uh- **Swyx** [11:51] Somewhere along. **Elie Bakouch** [11:53] Yeah. **Swyx** [11:53] I love it. **Elie Bakouch** [11:53] Yeah. **Swyx** [11:53] This is like just your, your, your notes of all the papers. **Elie Bakouch** [11:57] Yeah. This is exactly my notes. It's very-- I didn't touch them, to be honest, so. **Swyx** [12:02] It's great **Elie Bakouch** [12:02] And this is like the, the, the new algorithm. So which is very simple, to be honest, and like the, the key part is this like Newton-Schultz method, which is basically taking this, like this matrix and making it better in a sense that it will be, uh, somewhat orthogonal, orthonormal, to be precise, I think. And this will have, uh, the property of like basically spreading, uh, the learning in, in more dimension than with, uh, with this, uh, Adam, uh, Adam, uh, optimizer. And I think, uh, this is why, um, it's what they say. **Swyx** [12:41] It's so stable, I think. Yeah. **Elie Bakouch** [12:42] Yeah. It's, it's more stable. Uh, and also it encourage more to learn and explore new things, and, uh, this is pretty cool. And they, they did this Muon clip optimizer, which is pretty cool. And actually, it's quite simple. They just track the-- this quantity, which is the max logit per head. And if it's-- this quantity is too large, they are basically saying, "Okay, that's, that's not good. I rescale after the backward my, my parameters, uh, by dissonance, uh, to, to make sure that they, they are-- they scale down, and they don't explode anymore." And this is actually... it only have an impact in the, in the early step, and after it stabilize and basically don't do-- like it doesn't have any impact. At this point, you can, uh, deactivate the, the QK clip, and this will just, uh, work as well. But this is pretty cool. And they define it by just looking at the-- their log, and they see this, this crazy maximal, uh, attention logits. And they are like, "Okay, there is something wrong, and I need to, to contain those, uh, attention logits." And the method they choose is this QK clip stuff. There is actually different stuff to, to do that. There is like this, the Iden Z loss. There is the, the QK norm. There's a lot of stuff that kind of, uh, want to patch the same thing, but this one is quite elegant, to be honest, because it's very simple. **Swyx** [14:04] Yeah. **Elie Bakouch** [14:04] Yeah. **Swyx** [14:05] Yeah. Uh, cool. Uh, do, do you-- uh, just a side follow-up. Uh, did you actually follow the sort of, uh, schedule-free optimizer stuff from last year with Aaron Defazio? Anything came out of that? **Elie Bakouch** [14:15] I don't have a strong opinion. I tried it- **Swyx** [14:17] Yeah **Elie Bakouch** [14:18] ... uh, like a few days after it, it was released because I was, uh-- we were writing a paper about the, the WSD schedule, and so this was kind of an alternative to it. And, uh, it didn't perform as well as like a simple Adam W. **Swyx** [14:34] Yeah. **Elie Bakouch** [14:34] And it was, uh, more sensible to some-- to the beta one and beta two. But I think like with this kind of stuff, you, you really need to have this, uh, this knowledge on how to optimize, uh, those hyper parameters for each optimizer to really get the best performance of each one. And, uh, so I don't want to, to say something bad about it because basically, at nineteen percent-- uh, there is a ninety percent chance that it's just me that used the, the wrong hyper parameters. But yeah. **Swyx** [15:03] Yeah. You're holding it wrong. Cool. Uh, I mean, I'm happy to zoom out. Uh, sorry, I, I feel like I, I sort of zoomed in on optimizers just 'cause you're, you're one of the few people that we have- **Elie Bakouch** [15:11] Yeah **Swyx** [15:11] ... on the podcast that can talk about it. Um. **Elie Bakouch** [15:13] Yeah, I mean, and there is a lot more to that. Uh, there is Muon, but there is also Shampoo, Soap. There is bunch of series of optimizer that are important. **Swyx** [15:21] If you- **Elie Bakouch** [15:21] Yeah **Swyx** [15:21] ... do you want to keep going? Yeah, I, I, I was, uh- **Elie Bakouch** [15:24] I don't-- I, I'm not sure I have theee, the slide for Shampoo and Soap. I, I like this slide. This is from my previous set of slide. Basically, I was like, uh, looking at, uh, what the lab are like using, and I, I, I basically saw that every open model are-- were using Adam. **Swyx** [15:44] Yeah. **Elie Bakouch** [15:44] It was kind of crazy. And, uh- **Swyx** [15:46] It's pretty validated at this point **Elie Bakouch** [15:48] ... and since then- **Swyx** [15:48] Yeah. **Elie Bakouch** [15:48] Yeah. And since then, there is this new Muon trend. There is this recent paper talking about the, the different optimizer. And, uh- **Swyx** [15:55] Wait, this is this week **Elie Bakouch** [15:56] ... this basically... Yeah, this is this week. **Swyx** [15:59] Wow, these slides are super live. Oh, shit. **Elie Bakouch** [16:02] Yeah. **Swyx** [16:03] Okay. **Elie Bakouch** [16:05] Yeah. This is this week, and this is basically saying that... So there was a lot of papers that claim, claim some X, uh, spi- speed up and things like that. And what they find, uh, is that the speed up is greatly, greatly exaggerated, and mostly because, uh, often people v-- like undertune the Adam W baseline. And this is like something which is pretty, pretty, uh... Like we-- There is always this problem where people have a new architecture. For example, they are, they are under, under, like the, the baseline of the transformer is often under optimized. And so they, they basically say that when you increase the model size, the speed up is decreasing. But to be honest, uh, even the author, I think, say that this point, they didn't really optimize the hyperparameters. They just like do some, um, let's say that, do some extrapolation, and like extrapolation can be a bit tricky. So this, maybe we can forget this one, but I think the, the, the speed up between the one point two and one point three is pretty fair. And yeah, there, there is a lot of insights, such as like if you don't do weight decay, you can get, uh, those, those kind of curve. And if you, for example, take a look at like, uh, step, uh, uh, ten, uh, ten thousand, you, you can conclude that, uh, the, the one with, uh, weight decay equal zero is the best. But in fact, at the end, it's, it's not at all. So early curve can be, can be a lot-- like can mislead- **Swyx** [17:32] Wow **Elie Bakouch** [17:33] ... a lot the, the, the interpretation. So it's... Yeah. And also like y-you see here, if you, you don't tune, uh, enough your pa- hyperparameters, for example, uh, this is the true value they, they take for Soap. And in one case, it's very bad. It even had like this spike, which is not great at this scale. And, uh, in the other scale, it's like beating the, the large optimizer. So, yeah, it's pretty, it's pretty complicated to do a lot of that, but yeah. **Alessio** [17:58] Is there any value in the early curve? Like, even if it's like, you know, not predictive of like the whole training- **Elie Bakouch** [18:05] Yeah **Alessio** [18:05] ... like seeing kind of like the- **Elie Bakouch** [18:06] So- **Alessio** [18:06] ... a faster slope in the beginning **Elie Bakouch** [18:09] Yeah. So what I look, uh, when I, when I look at the early curve is mostly the spike. If there is some spike early on, it never a good sign for the, for the, for the rest of your training. There is some, um... For example, if you compare different datasets, the early curve can be, can have-- can give some information. It also depends on the, the schedule you are-- the learning schedule you are doing. Uh, we, we often use this WSD schedule, which means that basically you, you, you do this, like, as at our grow, there is this stable phase. And why it's great is because you can basically decay, uh, whenever you want. And so you do one run, uh, one big run with, for example, a fixed set up, and then you decay, uh, at different points, and then you can get, uh, this performance without doing like a different cosine schedule for, uh, for all of them. **Swyx** [19:01] Oh, yeah. **Elie Bakouch** [19:02] But, uh, overall, uh, it's like especially when doing, um, optimizer ablation, you, you never want to, to, to look at early curve and you want to like wait for, uh, the long rate to, to have fully decay and like the learning to be, to be to zero or whatever value. **Swyx** [19:22] Wow. This actually is a, is a really good paper. I, I... Like, there's a lot of insights and experiments in there. **Elie Bakouch** [19:25] Yeah, it's, it's a great one. And, uh, the people who are doing it is like the Mind Project at Stanford. They're really great, and they are doing some amazing open source stuff, so I really encourage people to, to check out, uh, what they do. **Swyx** [19:40] Awesome. Um- **Elie Bakouch** [19:41] Yeah. **Swyx** [19:42] Okay. Uh, anything else on the opt-optimizer world that's going on that's like, uh, interesting to you? **Elie Bakouch** [19:48] Uh, yeah. There is this, uh, this plot from the, the Asian Solar AI paper about Muon. And, uh, the, the idea of this plot is basically saying that you can use Muon P to... I mean, you can use some technique to, to do hyperparameter sweep, and it's working with Muon, for instance. And there is this idea where, uh, you... So in this, in this paper, they only did, uh, one D, uh, hyperparameter sweep. So they fixed everything, and they just changed the learning rate, for example. But, uh, for instance, there is some connection between the learning rate and the weight decay or learning rate and batch size. But you always want to sweep together, uh, some of the hyperparameters, and this is what they, they, they do here, for example. They, they, they sweep both the, the learning rates and the weight decay together to, to find the, the optimal one for, for their training. So- **Swyx** [20:41] Yeah **Elie Bakouch** [20:42] ... and I think it's all optimized now. Yeah. **Swyx** [20:44] I just realized this is a four-dimensional chart. **Elie Bakouch** [20:47] No, a three-dimensional one and that- **Swyx** [20:48] It's 3D, but then- **Elie Bakouch** [20:50] And then there's this one. Yeah **Swyx** [20:51] ... you see, you see this? That's the fourth dimension. Yes, correct. **Elie Bakouch** [20:53] Yeah. **Swyx** [20:54] Uh, very interesting. **Elie Bakouch** [20:56] Yeah. **Alessio** [20:57] What else comes to mind for you? I, I think like, you know, we're kind of on the model consumer's side, and so there's some of these things like, you know, that get to the engineer side and we think are really interesting. **Swyx** [21:09] Yeah. Like, like, like sparsity matters to us, like, uh, efficiency- **Elie Bakouch** [21:12] Yeah **Swyx** [21:13] ... uh, you know, speed. ### MoE Architecture **Elie Bakouch** [21:15] I think the, the MoE is definitely something that is interesting to understand, and it's definitely i-interesting to understand how powerful it is. So for instance, like the, the first, uh, graph are basically saying that, uh, you get better performance for the, the same number of ops. But there is this problem and, uh, it's basically that, for example, here you, you get like, uh, a, a 3x, uh, performance boost. But, yeah, you... If you, if you look at the training times, you only get like a 2x, uh, performance boost. And this is mainly because, uh, MoE are faster to train, but this is depending a lot on your code base. And there is actually some very small detail, uh, that can, uh, completely make your MoE, MoE very bad. I think one is this, um... Is this one, is like the... This is from Qwen, and they are basically saying that... So I don't know if I, I, I need to explain, but like the, the load balancing is basically saying like you, you have your different experts, and you want all of the expert to get approximately the, the same number of token at each batch or a-at the end. Because if you only have like two expert that, uh, activates, uh, every time, it basically like the, the others parts, uh, will, will basically be, be useless. Uh, so one of the, the very important part is this load balancing. And what they find in Qwen, uh... So there is different way to do load balancing. Uh, I can talk about this later. But, uh, what, what they find is basically you need to compute, uh, those two statistic, which is, uh, at each step you look at how many token act-actually selected by, uh, each expert, and you look at the probability that, uh, is affected by, um, by, for-- by the router, basically. And when you compute this, um, this statistic, you, you are like doing that at the micro batch size, uh, level. So this means that, for instance, when you do training on multiple GPUs, uh, let's say you just replicate the m-the model on all the GPUs. You have this global batch, um, you have this global batch, so it means that you, for example, you, you give a hundred document to, to your, to your model. And if you have t-five GPU, you, you give like, uh, twenty to each of them because you want to parallelize, uh, this process. And then after you, you gather all the result together and, and you do the backprop pass. And the fact that there is als- the, the, the statistic is compute at the, the mini batch size is like pretty, pretty important for the model. You can s-- I think the, the most, uh, important plot is this one. So basically, this is saying that- The model is not really activated for code. Like, there is no expert that are specifically activated for code. Uh, same for math, same for literature. So this is like, uh, um, all of this is like the, the local batch. And if you do it at the global batch, so meaning that you give, uh, at, at, at each step, you, you give like enough token basically, so that the statistic is meaningful. You get this, uh, very good and, uh, uh, like this is basically a sign of, uh, of good health for them. We need to have all those, uh, activated, like specific activate-- a-activation for each domain. And they find that, for example, in the, the Megatron, uh, code base, uh, this wasn't the case. This was done as the local batch, um, I think. So it basically means that everyone that is using uh, Megatron, uh, at the time of the paper, uh, I, I mean, it wasn't even a paper, it was just a blog. But they, they had this... They, they were in this regime, basically, where, uh, where the, the MoE were not, like very specialized. And this is super important for end, uh, end usage because, uh, you, you want to have, like a specialized expert so that they really learn something different from each other. Even after, for example, you can do some pruning, and if you want to model only on code, for example, you can just, uh, prune out some, some of the English literature experts and just keep the one that are activated for code. And then you can initialize your MoE with those and like bootstrap them. So there is like, uh, endless possibility here. And, uh, and yeah. But this is like a very small detail, and this will have a huge impact. **Swyx** [25:45] Pretty crazy. **Alessio** [25:45] You mentioned is something that, you know, personally, I never trained a model. How easy- **Elie Bakouch** [25:50] Yeah **Alessio** [25:50] -is it to divide the skills? So when you say, you know, literature, math, and code, what-- Can you maybe run people through the process of like, how do you kinda like funnel these things and different experts? And then is it just as easy as you said to like, "I'll just delete the liter-literature part," or is there some sort of like codependence- **Elie Bakouch** [26:09] Well- **Alessio** [26:09] -between them? **Elie Bakouch** [26:10] Yeah, there is some codependence, definitely. So there is other way to basically limit this codependence, but there is definitely some codependence. And when you do pruning, I, I don't know if people are doing that. I think that's interesting to explore. But, uh, like at least in the open, people are not really like pruning, for example, the, the English literature. That was just some, some research idea that could be called in the future. But, uh, but yeah. But the, the specialization is actually done in the, the outer layer of your MoE, and this is pretty important basically because this is like the, the, the step where you will learn, like each token will learn to which, uh, expert to go depending on like the, the, the meaning of each expert. And, uh, and you can even have some fancy stuff. So this is like, uh, from t-t-two weeks ago, this long cat flash from the, the DoorDash of China, uh, paper that released-- uh, the company that basically released like a state-of-the-art model, which is pretty crazy. **Swyx** [27:10] This is, this is the most recent presentation I've ever seen. **Elie Bakouch** [27:13] Yeah. **Swyx** [27:15] This is all like happened in the last two weeks. **Elie Bakouch** [27:17] So, yeah, I mean, uh, I'm trying to, to, to stay in, uh, in touch with, uh, what's happening. And they, that is this dual communication expert, which I find pretty cool. I think it's the first time I see it in like a big model. I mean, I'm pretty sure it's the first time. And like it allow the, the model to basically say, "This token doesn't, uh, um, like this token is like, uh, for example, very easy to predict, so I should not, uh, have, uh, like allocate a lot of flops to it. But this, uh, this token, which is, for example, for, uh, to, to, to answer a MMLU question is very hard, so I will activate more of the, of the real experts." But those are like just residual connection, basically. And, uh, it's the first time I've seen that. I mean, it's great, but I mean, they, they have a very strong team because my guess is that it's pretty hard to, to like balance it, uh, and to make the, the training stable. And they also have this change also, which is... I, I mean, I'm not sure I understand why they do that. I think I understand, but it's, it's pretty novel as well. It's basically replacing the shared expert with some, some double attention layer. Yeah, I mean, it's, it's pretty crazy. And there is a lot of question for MoE, basically. Do you want to, to have shared expert or not? A GP2SS Qwen3 doesn't have shared expert, but like the, the Qwen3 next, which is like a, which is like a... I mean, they didn't release it yet, but they submit the, the Pierre to Transformer. And they- **Swyx** [28:53] This is literally last night. **Elie Bakouch** [28:56] This is... This is not last night. This is like, uh, four hours ago, I think. **Swyx** [28:59] Oh my- **Elie Bakouch** [29:00] But, uh- **Swyx** [29:01] Oh my God **Elie Bakouch** [29:02] ... but, uh, and, and they basically have shared expert this time. So this is interesting. I think shared expert are pretty important to... Uh, they basically act like as, like the, the knowledge that you don't want. Like you, you want your expert to be very specialized, and you want the shared expert to be just like the basic stuff and anything like that. That's the intuition behind it, and I think DeepSeek is using that as well. ### Expert Specialization **Alessio** [29:27] What do you see as like the even more granular MoE future? You know, I, I think it makes sense for people to say, "Hey, it's like code and math and literature." Do you see a future- **Elie Bakouch** [29:37] Yeah **Alessio** [29:37] -in which the MoE model is like, you know, front-end expert? Like, uh, you know, if it's like a TypeScript code base, it's like, oh, this, uh, expert, like the routes is an expert at, you know, the component, is an expert at the test writing. Like, is that something that makes sense? Are we bottlenecked by the data, or what are people doing there? **Elie Bakouch** [29:58] So my intuition, uh, uh, I don't really know, but my intuition is that, for instance, between math and, I don't know, all the history, uh, like the Greek history, for instance. No, not really the Greek because it's related to math. But let's say code and Greek history. There is not a lot in common, except, uh, just the, the fact that the, the language is the, the same. But between like, for example, I don't know, TypeScript and JavaScript or Python and, and I don't know what language, there is more similarity. So I'm not sure, uh, if it's- Like there is, there is maybe more transfer learning, meaning that, uh, a model trained on like Python and JavaScript will benefit from the JavaScript, uh, data as well. I'm not sure it's the case for even, but for Python great history. **Swyx** [30:47] Yeah. I think like the, the question of semantic experts is always, uh, something that people are wanting to have intuitively, but the machine in- **Elie Bakouch** [30:55] Yeah **Swyx** [30:55] ... machine learning terms, it never works out that way. Uh, I actually feel like people are moving towards more semantic experts. It's, it's like somewhat started by Apple, uh, because they want like sort of downloadable lores and stuff. Uh, I remember-- Oh, Gemini three nano, 3N, uh, also had this, where they had- **Elie Bakouch** [31:15] Oh **Swyx** [31:15] ... sort of like dow-downloadable, offloadable weights. This is like a really severe- **Elie Bakouch** [31:18] Oh, you mean the Gemini 3N? **Swyx** [31:20] Gemma, Gemma, Gemma. Yeah, yeah, yeah. **Elie Bakouch** [31:21] Yeah. Yeah. **Swyx** [31:23] I see, I see some moves. I, I don't, I don't know how serious this is, this thing is, but like if you're, if you're going- **Elie Bakouch** [31:28] Yeah **Swyx** [31:28] ... to the point of having a router for different models, like semantic experts at some point makes sense. **Elie Bakouch** [31:34] Yeah. Yeah, and what I like is I think, uh, there is this recent trend where OpenAI like basically saying that, uh, they, they want like a router for a user to be, to, to, to have like some... And I think it's great for- **Swyx** [31:48] Yeah **Elie Bakouch** [31:49] ... you might as well. Uh, yeah, and, and this is kind of the same principle but in-baked inside the model. So I find it pretty cool because it's basically saying, uh, if my m-- if the request that I have is easy, I will just not allocate and I will basically in-- do inference with, uh, fewer expert, which is the equivalent of GPT-5 Mini, for example. And, uh, if I have like a very strong question, I will basically activate all the experts. But they, they do it at a scale where it's basically varying between, uh, I think it's like there is the, the variance is like four billion parameters, so it, it's not a lot. And GPT-5 Mini and, and the, the big ones are- **Swyx** [32:31] Yeah **Elie Bakouch** [32:31] ... don't have the, the same number of parameter at all. But this is, this is cool to see those behavior cooked and baked into the, into, into the model. **Swyx** [32:41] Yeah. Amazing. **Elie Bakouch** [32:42] Yeah. I don't know. I can talk about, uh, MoE, uh, or other stuff. What do you want? **Swyx** [32:49] Uh, yeah. Guide, guide it however you like. Uh, you know, I think the other, the other trend, uh, what, what definitely was more granular, um, um- **Elie Bakouch** [32:57] Yeah **Swyx** [32:57] ... uh, experts, uh, it was going up to like a hundred and twenty or something. I don't know if you have a chart here. Okay, it looks like some kind of efficiency. **Elie Bakouch** [33:04] Yeah. **Swyx** [33:05] Yeah, go ahead. **Elie Bakouch** [33:06] Exactly. And this is like basic... I think Que... So this is like the actual number of parameter here and the total number of parameter here. ### Sparsity Trends **Swyx** [33:14] I see. I see total. Yeah, yeah. **Elie Bakouch** [33:15] So this is basically the, the MoE sparsity. And you see that so those are like the kind of old model and they are like not really great in terms of sparsity. But if you look at the new one, uh, they are even the, like this communicator is pretty, pretty insane as well. I think we keep like the, the, the new FineWeb they will release will be somewhere here, I think, if I'm not wrong, if I read it well. So this will be Qwen, uh- **Swyx** [33:42] Qwen, Qwen max or Qwen next. Yeah. **Elie Bakouch** [33:44] Qwen3. Qwen next? **Swyx** [33:47] Yeah. **Elie Bakouch** [33:47] Yeah. I think this could beee- **Swyx** [33:48] Uh, yeah, I w- I would say from what I've heard, Gemini Flash is also pretty, pretty strong around- **Elie Bakouch** [33:53] Yeah **Swyx** [33:53] ... around this. Yeah. **Elie Bakouch** [33:54] Yeah. I mean, MoE like, uh, we are this kind of pioneer in this MoE field. I think MoE is, is from them as well, so, so they- **Swyx** [34:03] Of course. Yeah **Elie Bakouch** [34:05] ... everything is from him, so. **Swyx** [34:07] Everything is from him. I, I think like what, what was the difficulty for being this sparse before, right? **Elie Bakouch** [34:14] Yeah. **Swyx** [34:14] Like, is it just stability or training? **Elie Bakouch** [34:15] So this is definitely one thing. Also, I think like training code base, uh, you, you really need to have an optimized, uh, infrastructure and optimized training code base to, to have this, the good sparsity. Because the way it work is that you basically are distributing, uh, the expert on different GPUs. And in the MoE layer, they basically need to sync, uh, before because like the, the router is dispatching the weight to, uh, the, the token to, to the different experts. And basically, the more granular it is-- I mean, the more granular it is, the more this, this step can be, can be costly in term of communication. So this is one of the reason why it wasn't really good. For example, we tried, uh, we are training, uh, MoE currently at Hugging Face, and we'll train soon. We start the training soon. **Swyx** [35:06] Yeah. **Elie Bakouch** [35:06] Uh, and, uh, we tried with Megatron and we, we benchmark here like for example, the Mistral architecture with the, the, the Qwen3, uh, this one. And at the start before any optimization, like the Mistral one was very fast and have very good throughput, and the, the Qwen3 was, uh, very slow to train. But we basically with the new, uh, the new kernel, the, the new all this new stuff, uh, it like became feasible to train. And there is also this, uh, sparsity, uh, this like it can basically make the training sometimes a bit less stable and it's kind of a big thing. I, I don't know if, uh... For example, this OlMoE paper, they basically benchmark, uh, this number of experts, so sixty-four and thirty-two, and they didn't find like much difference between them. But if you look at like a, a more recent sparsity scaling law, you'll find that there is like this is basically, uh, uh, the, the different, uh, sparsity coefficients, and you find that there is actually quite some improvement. And of course, there is like diminishing return, but, uh, you, you can increase the sparsity quite well. And it's very good at inference. I think that's the main arguments. That that inference is very good because you can have very fast inference because there is only a very small subset of the parameter that is activated. **Swyx** [36:31] Yeah. I, I think o-one of those things that's kind of interesting as a pattern is like MoEs as a, as a production artifact, but not as a training- **Elie Bakouch** [36:38] Yeah **Swyx** [36:38] ... target. So the training target- **Elie Bakouch** [36:41] Yeah **Swyx** [36:41] ... you, you just have the d- the dense model, and then you distill to the MoE model. Yeah, I think, like, I, I- **Elie Bakouch** [36:46] Yeah **Swyx** [36:46] ... think this is the pattern for every frontier lab now, and I'm just not sure, like, you know, obviously, uh, everyone's-- why, why don't you just train the, the MoE directly, uh, is because of stuff like this. **Elie Bakouch** [36:57] Yeah. Yeah. I mean, yeah. I think they also train MoE, uh, from the start, but maybe less sparse, you know. They basically do some more sparsity on the production model than for the, the one that it use, uh, to distill, uh, from. Actually, distillation is, like, quite a big thing that people in the open are not really, really doing yet, I think. But, uh, but yeah. I mean, I think that's it for, uh, firmly. **Swyx** [37:27] Nice, nice check. Uh, I think the, the, the other thing, you know, uh, w-- especially you mentioned the GPT-5 router, is this- **Elie Bakouch** [37:34] Yeah **Swyx** [37:34] ... difference because, like, MoEs have routers, then we're putting a router on top of another router. Like, it's- **Elie Bakouch** [37:40] Yeah **Swyx** [37:40] ... ridiculous, right? Like- ... okay, what is- **Elie Bakouch** [37:43] And- **Swyx** [37:43] ... the future of routers, right? Like, is there a fat router that is basically its own LLM? What does that look like? If you, if you just extrapolate, like what, what does it look like in the future? **Elie Bakouch** [37:52] Like the future of the router? **Swyx** [37:54] I-if we're, if we're going to the point of, like, putting a router on top of a router- **Elie Bakouch** [37:57] Yeah **Swyx** [37:57] ... isn't that the same job? **Elie Bakouch** [38:00] Yeah, and, uh, I'm actually-- I don't have it in, in here yet, but let me screenshot something quickly from a local copy. Yeah, if I go back, yeah. There is this, um, this NSA, which is like the, the new DeepSeek attention. And- **Swyx** [38:16] Mm-hmm **Elie Bakouch** [38:17] ... it's basically a router inside attention. So- **Swyx** [38:20] Whoa **Elie Bakouch** [38:20] ... you can see it as like a, a router inside attention, basically. Because so there is this compression part, there is this sliding window attention part, so yeah, let's say. And there is this, this part where you are basically selecting which attention block is interesting for the specific token. So you are routing, uh, the attention of the token to some specific, uh, block of the, uh, previous one. Uh, you have also some routing inside attention happening with those new sparse, uh, architecture. There is also many SPM that have a, uh, very similar, uh, similar attention mechanism. But yeah, I think it's interesting because basically, like, like the more router you have, the more parameter efficient your model is. I think that's the intuition, yeah, at-- especially at inference, because you only activate a, a subset of it. And that's like the same logic for, uh, the, the model router basically, because you, you are selecting the model with like fewer a-activity slots, for example, GPT-5 mini compared to the big one. For MoE, you are basically saying, uh, "I, I'm only selecting a fewer token, or I'm only selecting a subset of it." And, and you also have this same logic into attention also now. **Swyx** [39:36] Yeah. **Elie Bakouch** [39:36] But yeah, this is pretty cool. **Swyx** [39:38] Yeah. **Elie Bakouch** [39:38] I mean, uh- Yeah, I mean, yeah, I think to, to answer also on the question, I think like the, the goal of, of like, uh, ML researcher is basically to, to cook the, the best model possible so that the user don't have to do much with it. ### Data Quality **Elie Bakouch** [40:35] Uh- **Swyx** [40:36] Yeah, out of the box, zero shot. **Elie Bakouch** [40:38] Yeah. That, that's I think pretty cool. And obviously, a big part of it is like the dataset. Uh, there is this recent paper, like the Beyond Web from the- **Swyx** [40:48] Yeah **Elie Bakouch** [40:48] ... the Datology, uh- **Swyx** [40:51] Yeah, we just had the Datology episode, I think last week. **Elie Bakouch** [40:53] Yeah, I saw this. **Swyx** [40:54] And, uh- **Elie Bakouch** [40:54] But it's pretty cool. Yeah. **Swyx** [40:56] So this is super recent again. I'm not used to talks being this recent. **Elie Bakouch** [40:59] Yeah. Yeah. So I, I mean, I, I've read it-- I mean, I've read it, uh, fully, but I didn't spend that much time on it. But if, if I had to, to recap a bit the, the paper, the, the interesting plot are, are this one. And so this is like I think the, the, the interesting one is that you basically want to rephrase, um-- so this is about rephrasing. So basically, you have-- So when you-- Like there is the, the web dataset is, is shit, to be honest, like there is a lot of very bad content in it. And there is one approach, and what we do in, uh, FineWeb-Edu, for example, that is basically filtering some, some educational content or some subset of the internet, which is, uh, often higher quality. And the question is like, what do you do with the lower quality data? I think that's how I see it. That's not exactly what they say in the paper. That's, uh, more my thought. But I think, uh, an interesting result is this, uh, is this, um, is this graph. And basically, this is like the curve between low-quality web and high-quality web. So this is the web, basically. And if you only, uh, rephrase the low-quality parts, you get this curve, uh, the light blue one, which is kind of close to, to the one where you also rephrase. I know it's not that, but it's like it gives you a, a big boost. And I think I'm not sure if they do the ablation here, but I'm not-- Yeah. Okay. So that's what they do. They, they also try to, to rephrase, like in the, the high-quality subsets, and they also repeat, uh, the dataset they rephrase on. And there is only a small difference. Then you could think it's because of the repeat. I'm not even sure it's because of that, to be honest. And the, the, the, the thing that you, you need to understand from it is like, uh, it's very good to rephrase the low-quality data, uh, to have like just better, uh, better benchmark and better performance. And, and yeah. And rephrasing is, is a bit, uh, is a bit tricky to do. Uh, I think a lot of it is making basically the right prompt. **Swyx** [43:11] Mm. **Elie Bakouch** [43:11] And one of the, the good prompt-- I mean, the good style that the, the model learn, the model like, especially because the benchmark are constructed, uh, this way, is the conversational, uh, data. So the Q&A, for example, uh, answer. ### Small LM Insights **Elie Bakouch** [43:25] Uh, for example, MMLU. So this is from SmolLM two, uh, uh, training, so which is, uh, one point seven B, I think. **Swyx** [43:36] Yeah. I know. **Elie Bakouch** [43:38] I don't think. **Swyx** [43:40] Very small. **Elie Bakouch** [43:40] And, uh, yeah, very small. And this is basically until, like six point five trillion of tokens, which is a lot, to be honest. Until this amount of token, the MMLU in the QA formats, meaning that the, the, the model have to select, uh, which answer is... The model have to output, for example, the, the right answer is A or B or C or D. And this is random. Like, for six trillion tokens, it's random. So this is a big deal. And it only learn after like six trillion token to, to, to basically answer question in a QA format. And this is because we, we didn't do at the time a lot of, uh... We didn't include a lot of QA, uh, Q&A, uh, data in our training set. And this is also from, uh, a blog post from Datology. And you see that their training sets, uh, like the, the, the data set is, is very good at MMLU. Uh, and this is, this is like mainly because of the formats. Like all of the data set are random because they don't have like QA formats, and I don't-- uh, they don't disclose it, I think, but, uh, maybe they do, to be honest. But this like the, the fact that they gain accuracy on MMLU and it's not random is because there is some Q&A, uh, formats, uh, inside the data set. So you see like if you just look at the benchmark, uh, you could say, uh, for example, like, uh, this CLM is very bad or it's very bad quality data or I don't know. But with a very light rephrasing and, uh, uh, let's say a style rephrasing, so you take the document from whatever data set and you, you put it in QA format, you would have like non-random performance on MMLU. **Swyx** [45:19] Right. **Elie Bakouch** [45:19] This is, this is very- **Swyx** [45:22] You're very, very sure it's not training for test, right? Like, uh- **Elie Bakouch** [45:25] Yeah. **Swyx** [45:26] This is obviously the, the, the problem. **Elie Bakouch** [45:29] Yeah. There is obviously some contamination that's, uh... I mean, thirty percent is not that much, to be honest. I mean- **Swyx** [45:37] Yeah **Elie Bakouch** [45:37] ... it's, it's great, but, uh, it's not like hundred percent or something like that. But I'm not sure it's contamination. It's more like QA, Q&A formats and, uh, and that's what they, they also find here is that you basically have better performance where you, you put some conversational data into the, into the training, and it's basically plateau at twenty percent, which is, uh, still good to know. **Swyx** [46:02] Yeah. Okay. And, uh, you know, one of the reasons I, I obviously I think we've been following the rephrasing the web research for a while. **Elie Bakouch** [46:09] Yeah. **Swyx** [46:09] I think one of the reasons that like I was a little bit anxious about this is that now the data set is too clean. It's too perfect, right? **Elie Bakouch** [46:18] Yeah. **Swyx** [46:18] I paste in a lot of shit in my data set, in my prompts. Right? **Elie Bakouch** [46:22] Yeah. **Swyx** [46:22] So can it recognize my typos? Because if it's never seen a typo- **Elie Bakouch** [46:26] Yeah **Swyx** [46:26] ... in its life, it's not gonna be u-understanding typos, right? Typos are just one simple form of mistake. There's grammatical mistakes, but then there's also- **Elie Bakouch** [46:33] Yeah **Swyx** [46:34] ... mild conceptual mistakes, all of which gets cleaned up with, with, uh, rephrasing, and- **Elie Bakouch** [46:39] Yeah **Swyx** [46:39] ... sometimes that's bad. **Elie Bakouch** [46:40] Yeah, I totally agree with you, and especially since like all the benchmark we have, uh, we don't have like typos, uh, benchmark, I think. There is no, no such things. Uh, I mean, there, there probably is, to, to be honest, but I don't... Like people don't usually benchmark a-against that. Um, obviously, the, the, the data set innovation and the, the model innovation like always are biased towards like maximiz-maximizing performance on some specific set of benchmark. And you, you kind of miss those use case, as you said, which is, uh, real case, real like user, um, like the, the thing that user care about, to be honest. And, uh... But I'm not that worried because I think the, the solution is just to include some bad data as well. And the model will just learn, just learn on it. And you can probably include- **Swyx** [47:31] The original data. Yeah **Elie Bakouch** [47:31] ... there is probably some, some scientific, uh, ablation to make on that. But you train on so much trillion of token data that, to be honest, I think the model have time to learn, uh, that sometime a user are typing, uh, bad stuff into, into their prompts. **Swyx** [47:47] Yeah. Uh, you know- **Elie Bakouch** [47:48] Yeah **Swyx** [47:48] ... I, I still see-- I, I feel like there, there will be some trade-off somewhere because like, you know, obviously you're-- you have much, much more... much, much fewer typos. I'm not just talking about typos, right? I'm talking about like- **Elie Bakouch** [47:58] Yeah **Swyx** [47:58] ... I'll copy and paste like website data and, and I just throw shit in there. Like because the pre-training data is bad, I actually have confidence that- **Elie Bakouch** [48:06] Yeah **Swyx** [48:06] ... you can handle bad data because I'm gonna paste in that bad data. **Elie Bakouch** [48:10] Yeah. **Swyx** [48:10] Um, anyway, so, so separately, uh, I also wanted to mention, uh, do you know... Do you cover the ten cent, uh, trillion persona paper? Because like persona rephrasing- **Elie Bakouch** [48:19] Yeah **Swyx** [48:19] ... billion persona, trillion persona. Basically, persona rephrasing is a very interesting way of doing multiple epochs. **Elie Bakouch** [48:25] Yeah. **Swyx** [48:25] And you never repeat yourself, but it's actually kind of the same content. It's just you're rephrasing. **Elie Bakouch** [48:29] So I think that's kind of what they did in, uh, Qwen-K2 ablation. **Swyx** [48:34] They did, yeah. **Elie Bakouch** [48:35] Yeah. Uh, I'm not sure. I didn't read the, the paper. I will read, uh, probably after this. **Swyx** [48:39] It's, it's basically role play, right? Like, uh, you're a doctor- **Elie Bakouch** [48:42] Yeah **Swyx** [48:43] ... blah, blah, blah. You're a physician, blah, blah. And then like it's slightly- **Elie Bakouch** [48:46] Yeah **Swyx** [48:46] ... different. **Elie Bakouch** [48:47] Yeah, I totally agree with you. I think that's pretty much... that's pretty important. Um- And, you know, there is this paper from, um, actually from people at Hugging Face at the time that is saying that you, you basically can repeat your data, uh, like up to, to three epoch before seeing diminishing gain, uh, uh, before seeing diminishing return. **Swyx** [49:10] Four is what I remember, but yeah. **Elie Bakouch** [49:11] Four. Yeah. I, I don't remember exactly the number, but I don't think they do, uh, ablation on like the different dataset. So with, uh, with Luma, uh, which is Luma Benal, uh, at Hugging Face- **Swyx** [49:24] Yeah. She spoke at my, uh, conference last year. Yeah. **Elie Bakouch** [49:27] Yeah, yeah. I was there, by the way. **Swyx** [49:29] Oh, yeah. Nice. **Elie Bakouch** [49:29] Was in the public. **Swyx** [49:31] We're, we're gonna do that again at NeurIPS, San Diego. **Elie Bakouch** [49:33] Oh, so at NeurIPS? **Swyx** [49:35] Yeah. Are you, are you coming? San Diego? **Elie Bakouch** [49:36] Uh, probably yes. I, I'm not sure. I didn't book my flight, but, uh, but, uh, I think we book it. And yeah, and I think this is like heavily dependent on, uh, on the dataset, and this is very underexplored. For example, uh, we have like the, the FineMath dataset, which is very high quality math datasets. We also have like the, the, the StackEdu one. And I'm pretty sure that if we repeat more time the, the FineMath datasets, this number of, uh, three X, we probably-- we can probably repeat more of high quality dataset because it contains like more information. And what you said is like with rephrasing, you can pretty much get probably, uh, like, uh, something-- I don't know. I didn't do ablation, but you can probably do like this, this ten X factor when you can repeat your data by rephrasing it a bit, uh, a lot. Uh, like you can repeat, uh, repeat it quite a lot. And, uh, I think this is quite underexplored and I would love to, to see some papers or, or maybe I would do it, uh, myself. I don't know. But, uh, but this is pretty cool and this is kind of good old, uh, data augmentation that we-- people were doing like during the time of computer vision, uh, with basically, uh, I don't know what they were doing, but they are like shifting the image or zooming out, zooming in, uh, adding some bit of noise, stuff like that. And they, they, they had better generalization, meaning that the training test loss were a bit better. So yeah, I think this is still an open question, at least in the open, uh, at least from my knowledge, and this is pretty, a pretty important one. **Swyx** [51:16] I think the, the last dataset piece question I have, then I'm happy to change topics, is, uh, one of the lessons that we had from the Datology episode was that effectively what we're doing was removing the post-training data into the pre-trained data, right? **Elie Bakouch** [51:29] Yeah. **Swyx** [51:29] Uh, that's-- You have the-- You talked about this conversational rephrasing. That's instruction tuning going in. **Elie Bakouch** [51:35] Yeah. I mean, there is all this mid-training stuff that's-- I mean, you, you can, uh, inject the data during post-training, mid-training, uh, pre-training, whatever. I think the, the-- what matter is that you-- I think the what matter is like the model see your data and post-training should be very to, to very like, to, to basically sharpen the distribution of your model to- **Swyx** [52:01] Yeah **Elie Bakouch** [52:01] ... to very specific topic. And so it's, for me, it's good and that's what people do nowadays. Like GLM, uh, I think they train on-- I don't, I don't remember exactly, but I think it's like eight trillion, uh, token at the mid-training state. Can you, can you call it mid-training? I'm not sure, to be honest. Uh- **Swyx** [52:20] You can call it whatever you want. **Elie Bakouch** [52:21] It's pretty crazy. Yeah. Eight trillion token is like... I mean, and they, they train only on code and math, and they do some rephrasing as well. And, uh, and yeah, I think, uh, I think that's basically what a lot of people are doing now. And we are doing this as well. We are, we are basically introducing higher quality data like, um, I don't know where it is. But if you look at the, uh, our different phase, we are basically adding more code and math, uh, through the, the training and less web data. **Swyx** [52:56] Yeah. **Elie Bakouch** [52:56] But we're only doing it like one point four trillion token, which is maybe- **Swyx** [53:02] Yeah. By, by the way, oh yeah, this, this chart, we haven't pulled it up yet, but like this chart I was- **Elie Bakouch** [53:06] Yeah **Swyx** [53:06] ... uh, was, was what wowed me on the SmolLM 3. I was like, "This might be the most open paper I've ever seen." **Elie Bakouch** [53:12] That, that's what we try at least to be, to be as open as possible. We also release like all the checkpoints, the intermediate checkpoints we have. I mean, we didn't release it all, but every ten billion token I think or- **Swyx** [53:24] Mm-hmm **Elie Bakouch** [53:24] ... no, I think fifty billion. Uh, I don't remember exactly what I pushed to run. But every like fifty billion token, uh, I think there is some checkpoints. And, uh, yesterday I was talking about, uh, some stuff about pre-training and someone was saying, "I hope like it would be cool if you release the, the, the intermediate checkpoint because I need to do some analysis about the, the pre-training, uh, dyn-dynamics into SmolLM 3." And I, I mean, it's here, you can use it. But this, this is pretty cool, and I think, uh, that's what we try to do. We also open source all the code base. So there is the Nanotron which is the-- doing the large scale training. ### Open Source Tools **Elie Bakouch** [54:00] There is the Datahub which is doing the, the data filtering. I think this one is very underrated. Um, and it's- **Swyx** [54:05] Okay **Elie Bakouch** [54:06] ... it's very good. That's, that's what we use to, to do FineWeb and, uh, FinePDF, yeah. **Swyx** [54:11] Uh, should we talk about that? Let's, let's, uh, let's open it up. I, I've never seen this, never heard of it. **Elie Bakouch** [54:14] Yeah. Datahub? I mean, I mean, it's just a library that's, that do data filtering and tokenization at scale. So you basically-- It's very efficient to, to do this on-- We do it on CPU mainly. And there is also no-- I think we are doing, uh, some pipeline for doing rephrasing also inside the, the Datahub library. We also have the Liteval one to do evaluation. **Swyx** [54:36] Mm-hmm. **Elie Bakouch** [54:36] And, uh, we are, we have a lot of stuff there. **Swyx** [54:38] Yeah, Liteval, I mean, I think a lot of people are like trying to move from the Hugging Face harness and, uh- **Elie Bakouch** [54:44] Yeah. The, the ending of our harness, yeah. **Swyx** [54:48] Yeah, yeah, yeah. Um, is Liteval the, the new standard now? Is-- Should I-- Should everyone be running that? Like- **Elie Bakouch** [54:52] I mean, the... Yeah, I think it's pretty good. The Datology, the Beyond web paper, they use Liteval, I think. **Swyx** [54:59] Yeah. **Alessio** [55:00] I, I think the community should have one. Uh, it's just that everyone is like-- Even, even the Hugging-- Even the OpenAI evals framework was so heavy that they, they had to op-- they had to release, like, "Okay, here, our, our main framework is too heavy. Here's the simple evals." And then, and then they don't even use that. **Elie Bakouch** [55:13] Yeah. **Alessio** [55:13] So I'm like, "I don't know what to use, man." **Elie Bakouch** [55:15] There is the eval framework, uh, every day . **Alessio** [55:19] Yeah. **Elie Bakouch** [55:19] Yeah, I think there is a lot of, uh, of stuff. But, uh, but I'd like to think that, uh, the Strike ML one is, is great. And I think, uh, one thing which is cool at working at Hugging Face is that we, we, we are this multimodal team which is doing like the SmolVLM. **Alessio** [55:33] Mm-hmm. **Elie Bakouch** [55:33] We have the robotic team. To be honest, I'm not really involved in that, but the, the-- they are doing the, the SmolVLM, uh, model. Uh, SmolVLA, sorry, which is like the, the vision language, uh, action stuff. **Alessio** [55:44] Action. Yeah. **Elie Bakouch** [55:44] But it's pretty cool to, to have like all those people working with the... We are like our SmolLM model as a backbone and then building some, some stuff around it, even inside, uh, uh, our team at Hugging Face, so this is pretty cool. **Alessio** [55:58] Yeah. So many memes. **Elie Bakouch** [56:01] Yeah. I mean, I did not make all the meme yet, but, uh, I have so much. Like, this one was like-- So this was for SmolM3, and I was doing, uh, ablation for, uh, on Optimizer. And basically, I, I was like... I mean, when I, when I fell asleep, it was like here, so there was no spike at all. And I woke up, and my graph looked like this, and I was like literally crying- **Alessio** [56:26] Ahhh **Elie Bakouch** [56:26] ... because all my training was diverging. **Alessio** [56:28] This is the quantum observer effect on training runs. When you're watching it, nothing happens. Um- **Elie Bakouch** [56:34] Yeah. And, and when you, you fell asleep, like you, you woke up the, the, the like the day after and i-it's a nightmare. But, uh, we solved the bug. I think i-it was a bug, uh, in our code base, and this was the, the, the main reason for the spike. And, uh, there is some fun stuff. Like, for instance, uh, when I was doing ablation on Muon, I was looking at like the Grad Norm, and the, the Grad Norm was doing this like, uh, M shape. And I was going a bit crazy because like this is like the, the M of Muon, I was saying to people. So it's always funny like, uh, on the Slack that people are... I think that's like one of the, the great thing at Hugging Face, that people are, are like having fun, uh, by doing all those ablation. Yeah. **Alessio** [57:18] All right. I know we're getting to time. Um, you gotta go back to updating this with all the news from the last one hour- ... that you've been here, just to make sure it's up to date. Any parting thoughts on SmolLM, especially? I think there's, you know, obviously, even when GPT-4 OSS came out, a lot of excitement about- **Elie Bakouch** [57:35] Yeah **Alessio** [57:36] ... open models, local models, blah, blah, blah. And then I think, you know, maybe the reality is that there's not as much adoption as people would like. Is there anything you wish people would do more of? Maybe like, uh, you know, underrated use cases for which you think these smaller models are like just as good as the frontier ones that people are not taking advantage of. **Elie Bakouch** [57:55] Yeah. I think even I, to be honest, I'm mainly using a closed model because it's easier, basically. Uh, it's like the same interface when I'm like, uh, for example, researching, uh, very deep on papers and when I'm, for example, doing like translation stuff, I want to translate something from English to French. ### Future Directions **Elie Bakouch** [58:13] But I know some of the heavier user are using the SmolLM three, for example, for translation. There is also obviously all the on-device application which are super important. Uh, there was this Apple keynote, uh, a few hours ago where they, they, they, they have said the word AI a thousand times probably . And yeah, so obviously on-device, uh, application are super important. And, and yeah, for the future of SmolLM, I think our goal is basically to build the model that the community don't have and need. So we are super open to feedback. Uh, don't hesitate to, to tag me on, uh, X or, uh, any place or whatever. If you, if you have like, uh, some... If you think that there is like some gap in the, the, like what the all the open source model are today and you, you wish there was smaller, for example, very small model like under one billion parameters or something like that, yeah, I think we, we are pretty open on what to do next. We want to train our MoE because it's fun and everyone is doing that. And also, I think there is a lot of, uh, different direction, and it's all very good, um, in terms of science to... Because basically the, the, the, the, the Qwen three or even DeepSeek, they don't release, release all the ablation they do. And I know as a researcher that that's something super important. And, uh, so when we do model training, we, we try to be as open as possible and obviously, uh, it's often appreciated by the community and yeah. And so that's, that's one thing we want to explore with, with this, uh, MoE. And we, yeah, also have a project to do, to do some distributed training across like different data center, like, uh, Pinecone did, for example. And, uh, and, and yeah, this is, this is a project I'm looking forward as well. **Alessio** [1:00:06] Awesome. Yeah, I, I do hope that there will be an easier way to just use these SmolLM models through API. **Elie Bakouch** [1:00:13] Yeah. **Alessio** [1:00:13] Um, I found that a little clunky right now, just to find something that is token-based because the Hugging Face API charges by the hour, which- **Elie Bakouch** [1:00:21] Yeah **Alessio** [1:00:22] ... kinda when you run it, it ends up costing the same as some of the other models. Like, uh, I have this like, uh, MCP client I built and we use Llama- **Elie Bakouch** [1:00:29] Yeah **Alessio** [1:00:29] ... um, AB for like a conversation naming, you know. I feel like that's like a great use case for like a five hundred million parameter model. But there's no simple API to use it. I feel like that's another... I-i-- In a way, it's almost like it's not economical to serve these models because they're so small . And so I think that's where they, they get stuck. You know, it's like who, who's really serving the five hundred million to a billion parameter models, you know? I think that's why. Ho-hopefully Hugging Face can do more there too. **Elie Bakouch** [1:00:56] Yeah, that's, that's very true. I think, uh, we should do some stuff, yeah. Yeah, yeah **Alessio** [1:01:02] But thanks so much for the time and the masterclass. I don't know, i-is this file online anywhere? I, I think this might be the most valuable **Elie Bakouch** [1:01:10] Yeah. I mean, it's not online yet, but I will probably share, uh, maybe a, a much, a more polished version of it. Like this is literally my draft, so... **Alessio** [1:01:21] This is probably like the most valuable TL Draw file ever. Like I think the, the, th-this file, this canvas is probably worth millions of dollars of insights- **Elie Bakouch** [1:01:30] Yeah. **Alessio** [1:01:30] -from, from your talk. So, um, excited to see the final product, and we'll link it. **Elie Bakouch** [1:01:35] Yeah. **Alessio** [1:01:35] We'll, we'll make sure to release this maybe once the talk is also out, so that then people can see the full slides. **Swyx** [1:01:41] J-just a, uh, parting question- **Elie Bakouch** [1:01:42] Yeah **Swyx** [1:01:42] ... because I, I s- I see that you, you know, you care a lot about, uh, training, optimizing and all that and, and, and multi-data center and probably like multiple architecture. Like, you know, if you've got some AMD, you, you wanna use that. Uh, have you looked at Modular at all? **Elie Bakouch** [1:01:53] Exactly. **Swyx** [1:01:53] Like, is that interesting to you? How have... Why- **Elie Bakouch** [1:01:56] So your podcast is the- **Swyx** [1:01:59] I'm just wondering, like, you know, what, what, uh, what's the state of that if you, if you took a serious look at it? If you haven't, if you haven't, then I'm just checking. **Elie Bakouch** [1:02:05] You mean like Modular the, the language or the just training on different architecture? **Swyx** [1:02:10] Yeah. M-l-language and runtime, right? Basically, like they're, they're basically offering like a superset of Python that should be good for you espec- **Elie Bakouch** [1:02:17] Yeah **Swyx** [1:02:17] ... you specifically. You are the target audience for this. And if y- you know, and I'm just ki- trying to hear about like feedback or like... **Elie Bakouch** [1:02:25] Yeah. I'm not very knowledgeable about it, but like the, like, uh, like the pre-training code base is something that you, you basically build, uh, through training and you, you iterate on it. So like changing some, uh, some like heavy stuff in it are always like a big deal for a researcher, I think. Uh, especially that everyone, um, is like their own preference and stuff like that. So yeah. But I, I didn't really... I'm not really knowledgeable about all of that, and I need to watch your podcast with them. Uh, it's actually on my, uh, on my, uh, on my- **Swyx** [1:03:00] Yeah, sure **Elie Bakouch** [1:03:00] ... to watch list. **Swyx** [1:03:01] Look- **Elie Bakouch** [1:03:01] So yeah **Swyx** [1:03:02] ... uh, yeah, you know, I, I see, I see Rust in your code base. I see Python. I see, you know, you're m-experimenting- **Elie Bakouch** [1:03:06] Yeah **Swyx** [1:03:06] ... different optimizers. Like, uh, you know, I, I'm not like the perfect ICP for them, but like you are, and so I'm... be curious to, to hear like, uh, you know, someone who doesn't work for them- **Elie Bakouch** [1:03:15] Yeah **Swyx** [1:03:16] ... like what's, what's your opinion, you know? Um, cool. **Elie Bakouch** [1:03:18] And I never tried, but, uh, but I will. I'll take a look. **Swyx** [1:03:20] It's hard to try things. Yeah, you have a thing that works. Like it's... Yeah. Cool. Well, thanks so much for your time. **Elie Bakouch** [1:03:25] Yeah, thanks a lot for the invitation, and see you in Paris, uh- **Swyx** [1:03:29] See you in Paris **Elie Bakouch** [1:03:30] ... sure, sure, yeah. Yeah. **Swyx** [1:03:31] Yeah. Awesome. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.