# Best of 2024: Synthetic Data / Smol Models, Loubna Ben Allal, HuggingFace [LS Live! @ NeurIPS 2024] Latent Space · 2024-12-24 Loubna Ben Allal of Hugging Face explains how synthetic data has become ubiquitous in LLM pipelines, moving from post-training to pre-training, with examples like Cosmopedia’s 30B-token synthetic dataset and NVIDIA’s 1.9T-token Nemotron CC. She addresses model collapse fears, showing that web dumps after ChatGPT’s release actually yield better models, and emphasizes diversity through prompt seeds and webpage extracts. For filtering, FineWeb-Edu and DCLM use LLMs to rate educational content, achieving top benchmark scores. In post-training, she highlights AgentInstruct, Tülu3 with PersonaHub, and Cohere’s multilingual arbitrage using multiple teachers. On small models, SmolM2 (1.7B) outperforms Llama 1B and Qwen 2.5 after 11T tokens of pre-training, and on-device inference via frameworks like llama.cpp enables privacy-preserving use cases such as text extraction and structured generation. She predicts a return to fine-tuning specialized small models over costly prompt engineering. ## Questions this episode answers ### How does synthetic data from AI models affect the quality of web training data and model collapse? Loubna Ben Allal notes that despite concerns about model collapse, Hugging Face's analysis of Common Crawl dumps showed an increase in proxy words like 'delve' post-ChatGPT, indicating more synthetic content. Yet models trained on later dumps performed better on NLP benchmarks, suggesting the web is enriched rather than polluted, as long as synthetic data is carefully curated. [4:07](https://addtry.com/524d779e-3b96-48c7-86aa-c1c540b55523?t=247000) ### How did Hugging Face's Cosmopedia dataset create diverse synthetic textbooks for pre-training, and what were the results? Loubna Ben Allal explains that Cosmopedia used seed extracts from webpages to prompt an LLM to generate textbooks on diverse topics, avoiding similarity. With a search tool to match topics, they created 30 billion tokens of synthetic textbooks. Training on Cosmopedia consistently outperformed training on the larger FineWeb dataset, showing the effectiveness of this approach for small-scale pre-training. [5:57](https://addtry.com/524d779e-3b96-48c7-86aa-c1c540b55523?t=357000) ### What trend is driving improvements in small language models, and how did Hugging Face's SmolM2 benefit from it? Loubna Ben Allal highlights the trend of training smaller models on much more data: SmolM2 was trained on 11 trillion tokens, up from 1 trillion for its predecessor, and continued to improve without plateauing. This approach, as seen with Llama 3 8B trained on 15 trillion tokens, yields dramatic performance gains, making small models competitive with previously larger ones. [20:20](https://addtry.com/524d779e-3b96-48c7-86aa-c1c540b55523?t=1220000) ## Key moments - **[0:00] Synthetic Data Everywhere** - [0:24] Synthetic data in 2024 spans every part of the LLM pipeline, from pre-training to evaluation. - **[1:11] Pre-training** - [1:28] Cosmopedia provides a 150-billion-token synthetic dataset that enables training LLMs on entirely synthetic data. - **[2:50] Model Collapse** - [3:50] LLM-typical words like ‘delve’ spiked in Common Crawl after ChatGPT’s release, yet later dumps yield better models. - [4:35] “I wouldn't say the web is polluted with synthetic data. Maybe it's even making it more rich.” - **[5:18] Textbooks** - [5:57] Cosmopedia reproduces Microsoft’s textbook approach by seeding diverse web extracts for synthetic data generation. - [7:39] College textbooks improve MMLU, while middle-school textbooks boost OpenBookQA and PECO benchmarks. - **[9:15] Rephrasing Web** - **[11:33] Filtering** - [11:49] FineWeb-Edu filters Common Crawl from 15 trillion to 1.5 trillion tokens using an LLM-trained educational classifier. - **[13:51] Post-training** - [14:10] Microsoft’s AgentInstruct dataset fine-tunes Mistral 7B to outperform Mistral’s own Instruct model on multiple skills. - [16:02] Cohere’s Multilingual Data Arbitrage routes prompts to the best teacher model, like arbitrage in finance. - **[16:43] Smol Models** - [17:00] Llama 3.2 1B matches last year’s Llama 2 13B on the LMSYS Arena human evaluation. - **[18:17] On-Device** - [18:19] Pocket Pal runs a 3.8B Phi model on iPhone with acceptable latency for on-device chat. - [20:20] Llama 3 8B trained on 15 trillion tokens, 15x more than original Llama, exemplifies training small models longer. - [22:27] SmolM2’s 1.7B model beats Llama 1B and Qwen 2.5 after training on 11 trillion tokens, fully open-source. - [24:04] On-device small models enhance privacy, supported by frameworks like MLX, llama.cpp, and Transformers.js. - **[26:15] Future Trends** - [26:15] Domain-specific synthetic data for math will grow, and fine-tuning small models will displace prompt engineering. - [27:27] AI is cycling back from prompt engineering to fine-tuning small models as large models’ cost outweighs small performance gains. ## Speakers - **Loubna Ben Allal** (guest) ## Topics Language Models, Inference ## Mentioned Allen AI (company), Apple (company), Cohere (company), Hugging Face (company), Meta (company), Microsoft (company), Mistral (company), NVIDIA (company), Apple intelligence (product), Cosmopedia (product), FineWeb (product), FineWeb-Edu (product), Llama (product), Mistral 7B (product), Nemotron CC (product), Phi (product), Qwen (product), SmolLM (product), SmolM2 (product), Tülu3 SFT Mixture (product) ## Transcript ### Synthetic Data Everywhere **Loubna Ben Allal** [0:06] I'm very happy to be here. Thank you for the invitation. Uh, so I'm gonna be talking about synthetic data in twenty twenty-four, and then I'm gonna be talking about small on-device models. So I think the most interesting thing about synthetic data this year is that, like, now we have it everywhere in the large language models pipeline. I think initially, uh, synthetic data was mainly used just for post-training because naturally that's the part where we needed human annotators to show the models how they should answer instructions, how should they, they be-- they should be helpful and not toxic. And when we had LLMs that were really performant, we replaced the not human annotators just with the synthetic data. And then after that, we realized that we don't really have good benchmarks to measure if model, models follow instructions well, if they are creative enough, or if they are chatty enough. So we also started using LLMs as judges. And I think this year and, uh, towards the end of last year, we also went to the pre-training parts, and we started generating synthetic data for pre-training to kind of replace some parts of the web. ### Pre-training **Loubna Ben Allal** [1:11] And the motivation behind that is that you have a lot of control over synthetic data. You can control your prompt and basically also the kind of data that you generate. So instead of just trying to filter the web, you could try to get the LLM to generate what you think the best web pages could look like and then train your models on that. So this is how we went from not having synthetic data at all in the LLM pipeline to having it everywhere. And so the cool thing is, like, today you can train an LLM with, like, an entirely synthetic pipeline. For example, you can use our Cosmopedia datasets, and you can train one big model on, like, a hundred and fifty billion tokens that are a hundred percent synthetic, and those are also of good quality. And then you can instruction tune the model on a synthetic SFT dataset. You can also do DPO on a synthetic dataset. And then to evaluate if the model is good, you can use a benchmark that uses LLMs as a judge, for example, MT-Bench or Alpaca Eval. So I think this is, like, uh, really mind-blowing because, like, just a few years ago, we wouldn't think this is possible. And I think there's a lot of concerns about model collapse, and I'm gonna talk about that later. But we'll see that, like, if we use synthetic data properly and we curate it carefully, uh, that shouldn't happen. And the reason synthetic data is very popular right now is that we have really strong models, both open and closed. Uh, it is really cheap and fast to use compared to human annotations, which cost a lot and take a lot of time. And also for open models, right now, we have some really good inference frameworks. So if you have enough GPUs, it's really easy to spawn these GPUs and generate, like, a, a lot of synthetic data. Some examples are vLLM, TGI, and TensorRT. Um, now let's talk about the elephant in the room, model collapse. Is this the end? If you look at the media and all of, like, for example, some papers in Nature, it's really scary because there's a lot of synthetic data out there in the web, and naturally, we train on the web. ### Model Collapse **Loubna Ben Allal** [3:04] So we're gonna be training a lot of synthetic data. And if model collapse is gonna happen, we should really try to take that seriously. Um, and the other issue is that, as I said, we think-- a lot of people think the web is polluted because there's a lot of synthetic data. And for example, when we're building fine web datasets here at Guillermin Hynek, we're interested in, like, how much synthetic data is there in the web. Uh, so there isn't really a method to properly measure the amount of synthetic data or to say if a webpage is synthetic or not. But one thing we can do is to try to look for, like, proxy words. For example, expressions like as a large language model or words like delve that we know are actually generated by ChatGPT. We could try to measure, uh, the amount of these words in our dataset and compare them to the previous years. For example, here we measured, like, uh, these words' ratio in different dumps of Common Crawl. And we can see that, like, the ratio really increased after ChatGPT's release. Um, so if we were to say that synthetic data amount didn't change, you would expect this ratio to stay constant, which is not the case. So there's a lot of synthetic data probably on the web. But does this really make models worse? So what we did is we trained different models on these different dumps, and we then, uh, computed their performance on popular, like, NLP benchmarks, and then we computed the aggregated score. And surprisingly, you can see that the latest dumps are actually even better than the dumps that are before. So if there's some synthetic data there, at least it did not make, uh, the models worse. Um, yeah, which is really encouraging. So personally, I wouldn't say the web is polluted with synthetic data. Maybe it's even making it more rich. Um, and the issue with, like, uh, model collapse is that, for example, those studies, they were done at, like, a small scale, and you would ask a model to complete, for example, a Wikipedia paragraph, and then you would train it on these new generations, and you would do that iteratively. I think if you do that approach, it's normal to observe this kind of behavior, uh, because the quality is gonna be worse because the model is already small. And then if you train it just on its generations, you shouldn't expect it to become better. But what we're really doing here is that we take a model that is very large, and we try to distill its knowledge into a model that is smaller. And in this way, you can expect to get, like, better performance for, for your small model. ### Textbooks **Loubna Ben Allal** [5:18] And, um, using synthetic data for pre-training has become, uh, really popular after the textbooks-are-all-you-need papers, um, where Microsoft basically trained a series of small models on textbooks that were using, uh, uh, a large LLM. And then they found that these models were actually better than models that are much larger. So this was really interesting. It was, like, uh, first of its time, but it was also met with a lot of skepticism, which is a good thing in research. It pushes you to question things. Um, because the dataset that they trained on was not public, so people were not really sure if these models are really good or maybe there's just some data contamination. So it was really hard to check if you just have the weights of the models. And at Hugging Face, because we like open source, we try to reproduce what they did. So this is our Cosmopedia dataset. We basically try to follow a similar approach to what they documented in the paper, and we created a synthetic dataset of textbooks and blog posts and stories that had almost thirty billion tokens. And we try, uh, train some models on that. And, um, we found that, like, the key ingredient to getting a good dataset that is synthetic is trying as much as possible to keep it diverse. 'Cause if you just throw the same prompts as your model, like generate, like, a textbook s- about linear algebra, and even if you change the temperature, the textbooks are gonna look alike. So there's no way you could scale to, like, millions of samples. And the way you do that is by creating prompts that have some seeds that make them diverse. Um, in our case, the prompt, we, we would ask the model to generate a textbook, but make it related to an extract from a webpage. And also we try to frame it within, to stay within topic. For example, here we put, like, an extract about cardiovascular bioimaging, and then we ask the model to generate a textbook related to medicine that is also related to this webpage. And this is a really nice approach because there are so many webpages out there, so you can be sure that your generation is now gonna be diverse when you change, uh, the seed, uh, example. Um, one thing that's challenging with this is that you want the seed samples to be related to your topics. So we use, like, a search tool to try to go all of FineWeb dataset and find the pages that are related to the topics we're interested in. And then we also do a lot of experiments with the type of generations we want the model to generate. For example, we ask it for textbooks for middle school students or textbook for college. And we found that, like, some generation styles help on some specific benchmarks while others help on other benchmarks. For example, college textbooks are really good for MMLU, while middle school textbooks are good for benchmarks like OpenBookQA and PECO. Um, this is like a sample from, like, our search tool. For example, you have a top category, which is a topic, and then you have some subtopics, and then you have the topic hits, which are basically the webpages in FineWeb that belong to these topics. And here you can see the comparison between Cosmopedia. We have two versions, uh, V1 and V2 in blue and red. And you can see the comparison to FineWeb. And as you can see, throughout the training, uh, training on Cosmopedia was consistently better. So we managed to get a dataset that was actually good to train these models on. Uh, it's of course so much smaller than FineWeb. It's only thirty billion tokens. But that's the scale that Microsoft Datasets was, so we kind of managed to reproduce a bit what they did. And the dataset is public, so everyone can go there, check if everything is all right. Um, another... Uh, this is a recent paper from NVIDIA, Nemotron CC. They, uh, took things a bit further and they generated not a few billion tokens, but one point nine trillion tokens, which is huge. And we can see later how they did that. It's more of like rephrasing the web. Um, so we can see today that there's, like, some really huge sy- synthetic datasets out there. And they're public, so, like, you can try to filter them even further if you wanna get, like, more high-quality corpuses. ### Rephrasing Web **Loubna Ben Allal** [9:15] Um, so for this rephrasing the web, uh, this approach was, uh, suggested in this paper by Pratyush, where basically in this paper, they take, uh, some samples from C4 dataset, and then they use an LLM to rewrite these samples into a better format. For example, they ask an LLM to rewrite the sample into a Wikipedia passage or into a Q&A page. And, uh, the interesting thing in this approach is that you can use a model that is small because it doesn't... Rewriting doesn't require knowledge. It's just rewriting a page into a different style. So the model doesn't need to have, like, knowledge, uh, that is, like, extensive of what is the rewriting compared to just asking a model to generate a new textbook and not giving it, like, ground truth. Um, so here they re- rewrite some samples from C4 into Q&A, into Wikipedia, and they find that doing this works better than training just on C4. And so what they did in Nemotron CC is a similar approach. Uh, they rewrite so- some pages from Common Crawl, uh, for two reasons. One is to, like, uh, improve, uh, pages that are low quality. So they rewrite them into, for example, a Wikipedia page, so they look better. And another reason is to create more diverse datasets. So they have a dataset that they already heavily filtered, and then they take these pages that are already high quality, and they ask the model to rewrite them in q- question and answer format into, like, open-ended questions or, like, multi-choice questions. So this way, they can reuse the same page multiple times without fearing, like, having multiple duplicates because th- it's the same information, but it's gonna be written differently. So I think that's also a really interesting approach for, like, um, generating synthetic data just by rephrasing the pages that you already have. Uh, there's also this approach, uh, called PROX, where, uh, they try to start from a webpage, and then they generate a program which finds how to write that page to make it better and less noisy. For example, here you can see there's, there's some leftover metadata in the webpage, and you don't necessarily want to keep that for training your model. So they train a, a model that can generate programs that can, like, normalize and remove lines that are extra. So I think this approach is also interesting, but it's maybe less scalable than the approaches that I presented before. ### Filtering **Loubna Ben Allal** [11:33] Um, so that was it for, like, rephrasing and generating new textbooks. Another approach that I think is really good at b- and becoming really popular for using synthetic data for pre-training is basically building, uh, better classifiers for filtering the web. Uh, for example, here we released a dataset called FineWeb-Edu. And the way we built it is by taking Llama 3 and asking it to rate the educational content of webpages from zero to five. So for example, if a page is, like, a really good textbook that could be useful in a school setting, it would get a really high score. And if a page is just, like, uh, an advertisement or promotional material, it would get a lower score. And then after that, we take these synthetic annotations, and we train a classifier on them. Uh, it's a classifier like a BERT model. And then we run this classifier on all of FineWeb, which is a fifteen trillion tokens dataset. And then we only keep the pages that have, like, a score that's higher than three. So for example, in our case, we went from fifteen trillion tokens to just one point five trillion tokens that are really highly educational. And as you can see here, uh, FineWeb-Edu outperforms all the other, uh, public web datasets by a large margin on a couple of benchmarks. Here I show the aggregated score. And you can see that this approach is really effective for filtering web datasets to get, like, better, uh, corpuses for training your LLMs. Um, others also tried to do this approach. There's, for example, the DCLM dataset, uh, where they also trained a classifier, but not to detect educational content. Instead, they trained it on OpenHermes dataset, which is a dataset for instruction tuning, and also the Explain like I Am Five subreddits. And then they also get really high quality, uh, dataset, uh, which is, uh, like, uh, very information dense and can help you train some really good LLMs. Um, and then Nemotron's ComCrawl, they also did this approach, but instead of using one classifier, they used an ensemble of classifiers. So they used, for example, the DCLM classifier and also classifiers like the ones we used in FineWeb educational, and then they combined these scores into a-- with an ensemble method to only retain the best high-quality pages, and they get a dataset that works even better than the ones we developed. ### Post-training **Loubna Ben Allal** [13:51] Um, so that was it for, like, synthetic data for pre-training. Now we can go back to post-training. I think there's a lot of interesting post-training datasets out there. One that was released, uh, recently is AgentInstruct by Microsoft, where they basically tried to target some specific skills and, uh, improve, uh, the performance of models on them. For example, here you can see code, brain teasers, open domain QA. And they managed to get a dataset that outperforms, uh-- that when fine-tuning Mistral 7B on it, it outperforms the original Instruct model that w- was released by Mistral. Um, and as I said, to get good synthetic data, you really have to have a framework to make sure that your data is diverse. So for example, for them, they always see the generations on either source code or w- raw text documents, and then they rewrite them to make sure they're easier to generate instructions from, and then they use that for their, like, uh, instruction data generation. There's also the Tülu3 SFT Mixture, which was released recently, uh, by Allen AI. It's also really good quality, and it covers a wide range of tasks. And, uh, the way they make sure that this dataset is diverse is by using, uh, personas from the PersonaHub datasets, which is basically a dataset of, like, uh, I think over a million personas. And for example, in the Tülu mixture, to generate, like, a new code snippet, they would give, like, the model a persona, for example, a machine learning researcher interested in neural networks, and then ask it to generate, like, a, a coding problem. This way you make sure that your dataset is really diverse, and then you can further filter the dataset, for example, using the reward models. We also released a dataset called Small Talk, and we also tried to cover the wide range of tasks. And as you can see here, for example, when fine-tuning Mistral 7B on the dataset, we also outperformed the original Mistral Instruct on a number of benchmarks, notably on mathematics and instruction following with IFeval. Uh, another paper that's really interesting I w- I wanted to mention is this one called, uh, Multilingual Data Arbitrage by Cohere. And basically they want to generate, uh, a dataset for post-training that is multilingual. And they have a really interesting problem. It's the fact that there isn't, like, one model that's really good at all the languages they wanted. So what they do is that, like, they use, uh, not just one teacher model, but multiple teachers. And then they have a router which basically sends the prompts they have to all these models, and then they get the completions, and they have a reward model that weighs all these generations and only keeps the best one. And this is like arbitrage in finance. So, uh, what I think what's interesting in this, it shows that, like, synthetic data doesn't have to come from a single model. And because we have so many good models now, you could, like, pull these models together and get, like, a dataset that's of a really high quality and that's diverse and that covers all your needs. Uh, I was supposed to put a meme there, but lack of time. Um, yeah, so that was it for, like, uh, synthetic data. And now we can go to see what's happening in the Smol Models field in twenty twenty-four. ### Smol Models **Loubna Ben Allal** [17:00] Um, I don't know if you know, but, like, now we have some really good Smol Models. For example, Llama 3.2 1B, it matches Llama 2 13B from-- that was released last year on the LMSYS Arena, which is basically the default go-to leaderboard for evaluating models using human evaluation. And as you can see here, the scores of the models are really close. So I think we've made, like, a huge leap forward in terms of Smol Models. Of course, that's one-- just one data point, but there's more. For example, if you look at this, um, chart from the Qwen 2.5 blog post- It shows that, that today we have some really good models that are only like three billion parameters and four billion that score really high on MMLU, which is a really popular benchmark for evaluating models. And you can see here that the re-- the blue dots have more than sixty-five on MMLU, and the gray ones have less. And for example, Llama 33B had less, so now we have a 3B model that outperforms a 33B model that was released earlier on MMLU benchmark. So I think now people sh- are starting to realize that, like, we shouldn't just scale and scale models, but we should try to make them more efficient. Um, I don't know if you knew, but you can also chat with a 3B Plus model on your iPhone. ### On-Device **Loubna Ben Allal** [18:19] For example, here, this is an app called Pocket Pal, where you can go and select a, a model from Hugging Face. It has a large choice. For example, here, uh, we loaded the Phi-3.5, which is three point eight billion parameters on this iPhone, and we can chat with it. And you can see that all-- even the latency is also acceptable. Uh, for example, here, I asked it to give me a joke about NeurIPS, so let's see what it has to say. Okay, why did the neural network attend NeurIPS? Because it heard there would be a lot of layers and fun, and it wanted to train its sense of humor. So not very funny, but at least it can run on device. Um, yeah, so I think now we have sm- good small models, but we also have, like, good frameworks and tools to use these small models. So I think we're really close to having, like, really on-edge and on-device, uh, models that are really good. And I think for a while, we've had this narrative that just training larger models is better. And of course, this is supported by science scaling laws. Um, as you can see here, for example, when we scale the model size, the loss is lower, and obviously you get a better model. But, uh... And we can see this, for example, in the GPT family of models, how we went from just a hundred million parameters to more than a trillion parameters. And of course, we all observed the performance improvement when using the latest model. But one thing that we shouldn't forget is that when we scale the model, we also scale the inference cost and time. And so the largest models were-- are gonna cost so much more. So I think, uh, now instead of just building larger models, we should be focusing on building more efficient models. It's no longer a race for the largest model, since these models are really expensive to run, and they require, like, uh, really good infrastructure to do that, and they cannot run on, for example, consumer hardware. And when you try to build more efficient models that match larger models, that's when you can really unlock some really interesting on-device use cases. And I think a trend that we're noticing now is the trend of training smaller models longer. For example, if you compare how much-- how long Llama was trained compared to Llama 3, there is a huge increase in the pre-training length. Llama was trained on one trillion tokens, but Llama 3.8B was trained on fifteen trillion tokens. So Meta managed to get a model that's the same size but it performs so much better by choosing to, like, uh, spend the sacrifice during, uh, training, because as we know, training is a one-time cost, but inference is something that's ongoing. Um, if you wanna see what are, like, the small models reads in twenty twenty-four, I think this mobile LLM paper by Meta is interesting. They try to study different, um, models that are, like, have the less than one billion parameters and find which architecture makes most sense for these models. For example, they find that depth is more important than width, so it's more important to have models that have, like, more layers than just making them more wide. They also find that GQA helps, that, uh, tying the embedding helps. So I think it's a nice study overall for models that are just a few hundred million parameters. There's also the Apple Intelligence, uh, tech report, which is interesting. Uh, so for Apple Intelligence, they had two models, one that was like, uh, on server and another model that was on device. It had three billion parameters. And I think the interesting part is that they trained this model using pruning and then distillation. And for example, they have this table where they show that, like, using pruning and distillation works much better than training from scratch. And they also have some interesting insights about, like, how they specialize their models on specific tasks, like for example, summarization and rewriting. Um, there's also this paper by NVIDIA that was released recently. I think you've already had a talk about, like, hybrid models that was already interesting. And this model, they used, like, a hybrid architecture be-between state-space models and transformers, and they managed to train a 1B model that's really performant without needing to train it on a lot of tokens. And, uh, regarding our work, we just recently released SmolM2. So it's a series of three models, uh, which are the best in class in each model size. Uh, for example, our 1.7B model outperforms Llama 1B and also Qwen 2.5. And how we managed to train, uh, this model is that we spent a lot of time trying to curate the pre-training dataset. We did a lot of ablations trying to find which datasets are good and also how to mix them. We also created some new math and code datasets that we're releasing soon. But we basically really spent a lot of time trying to find what's the best mixture that we can train these models on. And then we spent some time trying to... Like, uh, we also trained these models for very long. For example, SmolM1 was trained only on one trillion tokens, but this model is trained on eleven trillion tokens. And we saw that the performance kept improving. The models didn't really plateau mid-training, which I think is really interesting. It shows that you can train such small models for very long and keep getting performance gains. Uh, what's interesting about SmolM2 is that it's fully open. We also released, like, uh, the pre-training code base, the fine-tuning code and datasets, and also evaluation in this repository. Uh, also there's like really interesting small models for text, but also for vision. For example, here you can see SmolVLM, which is a two B model that's really efficient. It doesn't consume a lot of RAM, and it also has a good performance. Um, there's also Moondream zero point five B, which was released recently. It's like the smallest visual language model, and as you can see, there isn't like a big trade-off c-compared to Moondream two B. Um, so now I showed you that we have some really good small models. We also have the tools to use them. But why should you consider using small models and when? I think like small models are really interesting because of the on-device feature. Because these models are small and they can run fast, you can basically run them on your laptop but also on your mobile phone, and this means that your dataset stays locally. You don't have to send your queries to third parties, and this really enhances privacy. That was, for example, one of the big selling points for Apple Intelligence. Also right now, we really have so many frameworks to do on-device inference. For example, there's MLX, MLC, llama.cpp, Transformers.js. So we have a lot of options, and each of them have like great features, so you have so many options for doing that. Uh, small models are also really powerful if you choose to specialize them. For example, here, um, there's a startup called Numind, which took SmolLM, and then they fine-tuned this on text extraction datasets. And they managed to get a model that's not very far from models that are much larger. So I think text extraction is like one use case where small models can be really performant, and it makes sense to use them instead of just using larger models. You can also chat with these models in browser. For example, here you can go there. You can load the model. You can even turn off your internet and just chat, start chatting with the model locally. Um, speaking of text extraction, if you don't want to fine-tune the models, there's really good method of structure generation. We can basically force the models to follow a JSON schema that you defined. For example, here we try to force the model to follow a schema for extracting key information from GitHub issues. So we can input free text, which is a complaint about a GitHub repository, something not working. And then you can run it there, and the model can extract anything that is relevant for your GitHub issue creation. For example, the priority, for example, here priority is high, the type of the issue, bug, and then a title and the estimation of how long this will take to fix. And you can just like, uh, do this in the browser. You can transform your text into a GitHub issue that's properly formatted. Um, so what's next for synthetic data and small models? I think that domain-specific synthetic data is gonna be... It's al-it's already important. It's gonna be even more important. For example, generating synthetic data for math, uh, I think this really would help improve the reasoning of a lot of models, and a lot of people are doing it. ### Future Trends **Loubna Ben Allal** [26:34] For example, Qwen two point five Math. Everyone's trying to reproduce a one. And, uh, so I think for synthetic data, trying to specialize it on some domains is gonna be really important. And then for small models, I think specializing them through fine-tuning is also gonna be really important 'cause I think a lot of companies are just trying to use these large models because they are better. But on some tasks, I think you can already get decent performance with small models, so you don't need to pay like a cost that's much larger just to make your model better at your task by a few percent. And this is not just for text. I think, I think it also applies for other modalities like vision and audio. And I think you should also watch out for on-device frameworks and applications. For example, like the app I showed, Pocket Pal or Llama, all these frameworks are becoming really popular, and I'm pretty sure that we're gonna get like more of them in twenty twenty-five, and, uh, users really like that. Um, maybe for other I should also say a hot take. Um, I think that like in AI, we j- started like with fine-tuning, for example, trying to make BERT work on some specific use cases and really struggling to do that. And then we had some models that are much larger, so we just switched to like prompt engineering, uh, to get the models to solve our tasks. And I think we're going back to fine-tuning, where we realize these models are really costly. It's better to use just a small model. We try to specialize it. So I think it's a little bit of a cycle, and we're gonna start to see like more fine-tuning and less of just like, uh, prompt engineering the models. Um, so that was my talk. Thank you for following, and if you have any questions, we can take them now. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.