# Training Llama 2, 3 & 4: The Path to Open Source AGI — with Thomas Scialom of Meta AI Latent Space · 2024-07-23 Thomas Scialom, Llama 2 lead and Llama 3 post-training lead at Meta AI, explains why scaling laws must go beyond Chinchilla—training models longer on more tokens yields better inference efficiency than bigger models, a lesson that drove Llama 3's 15 trillion token pre-training. He details how synthetic data from Llama 2 bootstrapped Llama 3's post-training, eliminating human-written SFT data, and why RLHF outperforms imitation learning: humans are better discriminators than generators, enabling superhuman outputs. Scialom defends the dense 405B architecture over MoE, calls tokenizer vocab size underrated (128k tokens vs Llama 2's 32k), and reveals Llama 4's focus on agentic capabilities—tool use, multi-step reasoning—as the path to open-source AGI. Meta AI is hiring researchers with rigorous first-principles thinking. ## Questions this episode answers ### What is the "Chinchilla trap" that Thomas Scialom warns about, and how does it affect model scaling? Scialom explains that the original Chinchilla scaling laws optimize only for training compute, neglecting inference cost. Since inference cost depends solely on model size, it's better to train a smaller model on many more tokens (over-train) to achieve better performance at lower inference cost. Llama 1 and Llama 3 intentionally over-trained relative to Chinchilla optimal ratios to create more inference-efficient artifacts. [11:25](https://addtry.com/15464dfd-8fe7-4718-b394-467a1b1b4901?t=685000) ### How much does RLHF matter for Llama 3's performance compared to supervised fine-tuning? Thomas Scialom says that 100% of the post-training improvement comes from RLHF. In fact, for Llama 3, they didn't use any human-written answers for the initial supervised fine-tuning stage; instead, they generated all SFT data synthetically using Llama 2. This bootstrapping, combined with RLHF, enabled the model to surpass human-quality annotations and achieve superhuman abilities in some tasks like writing. [33:03](https://addtry.com/15464dfd-8fe7-4718-b394-467a1b1b4901?t=1983000) ### Why is the tokenizer vocabulary size an underrated design choice, and how did Llama 3's 128k vocabulary help? Scialom notes that a larger vocabulary (Llama 3 uses 128k tokens, up from 32k in Llama 2) improves multilinguality by capturing more characters, but it also compresses text into fewer tokens. This allows the model to see more knowledge with fewer training steps and provides longer effective context (about 30% more text for the same token limit) at inference. For small models, the larger embedding matrix is more costly, but for large models it's negligible. [55:22](https://addtry.com/15464dfd-8fe7-4718-b394-467a1b1b4901?t=3322000) ### What is the planned focus for Llama 4, and how does it relate to agents? Scialom says that now with a strong instruction-following base in Llama 3, the next step is to build agents—systems that can plan, backtrack, use tools, and navigate the web. He points to the GAIA benchmark, where orchestrating an LLM with tools already achieves 60% on level 1, and expects orders-of-magnitude scaling by interconnecting world models into agentic systems. Llama 4 will emphasize these agentic capabilities. [45:51](https://addtry.com/15464dfd-8fe7-4718-b394-467a1b1b4901?t=2751000) ## Key moments - **[0:00] Intro** - [1:05] Thomas Scialom left quant finance after the AlphaGo moment to pursue a PhD in deep learning at Sorbonne. - [2:47] Thomas Scialom: 'If I called them RLHF without human in the loop, I would have got much more citations today.' - **[4:16] Llama 2 Origins** - [4:20] Galactica's release two weeks before ChatGPT forced Meta to urgently build a ChatGPT-like model. - [6:08] Thomas Scialom: 'I had it on Overleaf integrated. I tested it. And it works.' (on Galactica Instruct citation generator) - [7:18] For Llama 2, the team had no idea how to scale RLHF annotations and had to reinvent the wheel in a very short time. - [8:32] Model size decisions trade off scaling laws, GPU memory constraints, and inference costs for community use. - [9:54] Thomas Scialom warns against the 'Chinchilla trap': to optimize inference efficiency, train models on more tokens rather than making them larger. - **[9:56] Chinchilla Trap** - **[12:15] Model Decisions** - [12:43] The Llama 3 405B model was built to be the best open-source model and directly compete with GPT-4. - [13:26] Thomas Scialom hopes community quantization will let 405B run on consumer hardware, as previous Llamas were made to run on a Raspberry Pi. - **[17:48] Data & MoE** - [17:54] Llama 3 uses the same architecture as Llama 2 but trained on 15 trillion tokens with significantly better data quality. - [18:33] Q: How was synthetic data used for Llama 3 pre-training? A: Llama 2 classified web documents to select good tokens and add topic tags for diversity. - [21:04] Thomas Scialom predicts that tool-augmented synthetic data (e.g., calculator, search) will enable continual model learning beyond initial training. - [24:42] Llama 2 focused only on helpfulness; Llama 3 post-training targeted code, reasoning, and multilinguality, achieving state-of-the-art open results. - **[27:05] RLHF** - [27:19] No MoE yet: dense model is just one expert; MoE is a hyperparameter to explore in future models. - [27:58] RLHF works because humans are better at judging outputs than creating them, allowing models to achieve superhuman performance in some areas. - [33:03] Q: How much post-training improvement comes from RLHF vs SFT? A: Thomas Scialom says 100% from RLHF. - [33:18] Llama 3's initial supervised fine-tuning used synthetic data generated by Llama 2, not human-written answers. - [34:54] Llama 3 introduced 'teacher critic', combining SFT and RLHF by having humans edit poor model outputs to exit local minima. - [35:28] RLHF scaling is not plateauing yet; more data and compute continue to yield better models. - **[37:06] Evals** - [37:37] Thomas Scialom: Llama 3 405B is the best open source model, outperforming the original GPT-4 on many benchmarks. - [38:38] Q: What benchmark scores does Llama 3 Instruct achieve? A: MMLU 86, HumanEval 84, GSM8K 94, MATH 57.8. - [39:10] Evaluating language models on post-training capabilities is super hard; benchmarks risk overfitting and don't tell the full story. - [40:13] Meta tracks RLHF progress by measuring automatic win-rate between successive model versions on annotation prompts. - [41:54] Thomas Scialom did not expect the Llama 3 70B preview to achieve such a high ELO score on the Lmsys Arena leaderboard. - [43:11] Current evaluations lack confidence estimation; models should be benchmarked on their ability to say 'I don't know'. - **[45:52] Agents** - [48:06] On the Gaia agent benchmark, GPT-4 alone scores 5% on level one, while agent frameworks like Autogen reach 60%. - [49:55] Thomas Scialom envisions future models that think in latent space without generating all intermediate tokens, for efficient reasoning. - [52:59] Training of Llama 4 started in June 2024, with a focus on agentic capabilities and balancing research with product needs. - **[53:55] Tokenizer** - [53:55] Q: Why did Llama 3 increase tokenizer vocabulary to 128k? A: To support multilinguality, compress text into fewer tokens, and extend effective context length. - [56:03] Larger vocabulary size benefits larger models proportionally less, but speeds up training by reducing sequence length. - **[59:12] Startups** - [59:48] Thomas Scialom shouts out agent startups Lindy and Open Devin as promising companies. - [1:00:41] Thomas Scialom warns that deep learning is a self-destructive technology; startups should not bet on the weaknesses of current models. - [1:03:05] Meta AI is hiring researchers who bring first-principles thinking, rigor, and meticulousness over deep LLM expertise. - [1:03:51] Thomas Scialom invites GenAI founders to reach out on LinkedIn. ## Speakers - **Alessio** (host) - **Swyx** (host) - **Thomas** (guest) ## Topics Language Models, Agents ## Mentioned Hugging Face (company), Lindy (company), Meta (company), Mistral (company), Open Devin (company), Autogen (product), Bloom (product), ChatGPT (product), Chinchilla (product), Claude (product), Devin (product), GPT (product), Gaia (product), Galactica (product), H100 (product), Llama (product), OS Copilot (product), Overleaf (product), Raspberry Pi (product), Toolformer (product) ## Transcript ### Intro **Alessio** [0:05] Hey, everyone. Welcome to the Latent Space Podcast. This is Alessio, partner and CTO in residence at Decibel Partners, and I'm joined by my co-host Swyx, founder of Small AI. **Swyx** [0:14] Hey, and today we have a very special episode with Thomas Scialom. I don't know how to describe. You've, you've done so much work in a very short amount of time at Meta, but, uh, you were most notably leading Llama 2 and now today we are with-- we're also coordinating on the release of Llama 3. So welcome. **Thomas** [0:29] Thanks for having me. **Swyx** [0:30] To be clear, obviously, the Llama 3 405B. Is that the official size number that we're going with? Uh, or is it-- do we just say 400B ? **Thomas** [0:37] For the text, uh, text model only, yes. A bit of additional- **Swyx** [0:41] Yeah **Thomas** [0:42] ... more parameters for the multi-model version that will come later. **Swyx** [0:44] Awesome. Awesome. Just to quickly go over your background, you actually we had a slightly similar path. So I was also a quantitative trader. Uh, and it looks like you did five years in quant finance working at Trading Timer in Soc Gen. And then you transitioned into natural language, getting your PhD at Sorbonne, working on Recital as well, and then right after your PhD, joining Meta. **Thomas** [1:05] No, it's exactly that. But basically, I think it's at the AlphaGo moment where I was doing some trading. I say like, what? I need to understand what's the technology behind that, and I wanted to study machine learning. I did first some training, like six months degree, executive degree, at the end of which I knew like what XGBoost at the time and nothing about, uh, deep learning at all. So... And you know, most of the people around were like PhD, uh, people. Then I went, "Okay, PhD seems really cool. Deep learning seems really cool, so I want to do a PhD in deep learning." That's where I joined, um, what-- We have this PhD program in France within a company and, uh, academia, and so I did my PhD with the Recital and, uh, Sorbonne University on natural language generation reinforcement learning. I guess it was a good topic. I was not like a visionary. It was very random. I had the company that offered me like, uh, this topic, and it was something like I started two weeks before BERT. **Swyx** [2:03] Excellent timing. Yeah, we actually also just released our episode with Clementine Forget, who also did her PhD with a company in kind of like a very similar format. I think, yeah, very underrated. Very underrated, this sort of PhD with industry, uh, expertise, because you're also, like, publishing papers the whole time. I, I looked at your publishing history. You were, you were doing, like, summarization work. You were doing factual consistency work. You released some benchmarks. And then you worked on language GANs, uh, before the transformers took over . **Thomas** [2:32] We, we can come back to that later, but I should have-- I mean, papers have like ten, fifty citations. If-- I'm pretty sure that if I call them, uh, like RLHF without human in the loop, but like a discriminator which, uh- ... synthetic a human in the loop, I will have got much more citations today because, like all... And all the inspiration for this paper were from actually the original OpenAI paper of RLHF. But at academia, we don't have the way to pay, uh, annotation online like that, so how to simulate it? **Swyx** [3:06] Yeah. A lot of these ideas are repeated, like discriminator, generator. We just call them different names now, like verifier- **Thomas** [3:11] Yeah. Exactly **Swyx** [3:12] ... whatever. Well, I think your, your, your progress into NLP was like really strong 'cause like the first thing you worked on at Meta was Bloom. **Thomas** [3:18] Yeah, yeah. Actually, I started on to work on that before joining Meta. I was not like, uh, one of the main contributors, but it was at the intersection of multilinguality which was very important to me, large language modeling, and, uh, that's why actually my first big project at Meta and the team I was working on was Galactica. And actually, interesting step back from Bloom was like we did a lot of mistakes, but it was expression that's expected and we learned a lot. But like trying to scale towards like multilinguality, in fact, we learned later that multilinguality almost emerged naturally with very, very few data, which was really surprising and not, not expected at all for us at the time. **Swyx** [3:58] I mean, my learning from that is just there's a natural harmony of language that is abstract from English. When you learn English, you learn language, and then language just translates to other forms of languages, uh, especially if they're the same family, right? Like, uh, yeah. So maybe we should get right into Llama 2, spend a little bit of time there, and then, and then we'll go into Llama 3. So like, uh, what is the story of Llama 2 from your point of view? ### Llama 2 Origins **Thomas** [4:20] Yeah. So as I was saying, I started to Meta on Galactica. That was one of the first large language model at Meta. It's a language model for science. We released it in, I think, December or end of November, I don't remember, one year and half ago. I don't know if people remember, but it was huge on Twitter, both with people like thinking it's the end of science and like that will be a lot of hallucination papers. Others were like, "It's super awesome." I still think it was super awesome, but you know, we didn't do like instruction tuning, RLHF techniques at the time. It was a weird moment because two weeks later, ChatGPT came out and that's the moment where like I think all the FAANG companies went upside down and where we had a, a huge traction from leads to now work on that and make a ChatGPT as soon as possible. So we had this one, two months of like what to do. I actually was working on Galactica Instruct, which basically you could connect it. We had a partner with Overleaf, the Google Doc of like scientists where you can write papers. And it's in a... You write there in LaTeX. You have to do a lot of citations. So the idea was that you can just like ChatGPT or GPT Instruct ask, uh, or swap two columns in a LaTeX table. That's something very, very, uh, time-consuming, I can promise. You could like say, "Oh, find me a citation about LLMs and bias." It will find you some papers, insert automatically the bib in like LaTeX. So that was pretty cool, but because of the backlash, we never like opened it in the end. **Swyx** [5:49] Oh, because the, the Galactica backlash. Oh, yeah. **Thomas** [5:52] Yes. **Swyx** [5:52] Like I was just saying like today it's not solved 'cause Lukas Bayer is still asking for this citation generator . **Thomas** [5:57] I saw these tweets. I was, "Dude, we had that two years ago," and I promised I tested it. It works so well. I had it on Overleaf integrated. I tested it. **Alessio** [6:07] Wow. **Thomas** [6:08] And it works. **Alessio** [6:08] Okay. **Thomas** [6:09] Yeah, yeah. No, it went quite far, in fact. And actually, about citations, like it's anecdotal, but because the way Galactica was trained to cite papers with all the references in paper, that's what made it emerge so easily at instruction timing. Actually, uh, Galactica Instruct was the first annotation project for RLHF at Meta. That was a follow-up of Galactica that we were preparing. And at the same time, my friends from, uh, Paris office created Llama 1. It's like to connect the dot with what we said before, the last author was Guillaume Lample who founded Mistral. The first author is Hugo Touvron who worked with me on Llama 2, still at Meta, and both did a PhD program within Meta as a company and, uh, in academia. So that's a pretty good program, uh, yeah, indeed. And so we worked on Llama 2 from th-that point. We had all the support from the company leadership. That was one of the main priority. We had Llama 1 and Galactica as like backbone of good language model. We started from Llama 1, and we worked mainly with Hugo on how to make instruction following and chat models that will follow instructions. So all the supervised fine-tuning stage, then the RLHF, there are some papers, so you had some intuition from there we could use. But in fact, at large scale, and that was probably the most challenge for us, there's no research anymore. We don't know how much to scale. **Alessio** [7:35] Can you describe what scale you're talking? **Thomas** [7:36] Yeah, yeah. To what level of annotation to scale the annotation. Like, do you need hundred thousand, one million, ten million annotation of supervised fine-tuning, of RLHF preference? We had no idea. What is the actual algorithm to do? How often to retrain the models? You have just the basic, but then when it comes to like ChatGPT or GPT Instructor, uh, Claude, no one published the details there. And so we had to reinvent the wheel there in a very short amount of time. **Alessio** [8:04] And what about parameter size? This is one question that a lot of folks had about Llama 3. So Llama 1, you had 7B, 13B, 33B, 65B model sizes, and then Llama 2, seven, 13, 70. How do you kinda evaluate what's worth training, especially when you think about data? It's like, you know, maybe a hundred thousand is enough for like a 7B model, but it's not enough for a 70B model. How do you decide model size, especially when you're maybe annotation constrained on some of these things? **Thomas** [8:32] That's a very good question, and there's no good answer. There's so many parameters to take into account from the scaling laws, training time to get the b- the best performance, the GPU constraint, and on what different hardwares. And we think about Meta, but also of the community, and like people are not just using H100, but there's H100, there's, uh, there's different size of GPUs, uh, memory. So which size will fit in what and what is the most useful also at inference time, not just at fine-tuning time. Then you can maybe do some tricks at inference time to quantitize it a bit or FP16 or FP8 now. All those constraints makes it very, very challenging. At inference time, you have a lot of costs, so how to trade off between inference cost and, uh, training cost. It's a very challenging problem. In general, we tend to think in particular for Llama 3. Llama 2, maybe I would say it's like Llama 1. We had a flagship model which was 70B. It's also because the project was taking some routes to reproducing, uh, Chinchilla, which was a 70B. For Llama 3, we also like moved to one size more, the flagship model for 05B. I think that there was also the question of we want a model at this time. We have this amount of compute. Given the scaling laws and the amount of tokens, we have to train it. What would be the right balance to still like fits in at inference time? So we try to, to have some trade-off like that. ### Chinchilla Trap **Alessio** [9:57] Yeah. And you mentioned Chinchilla is the best way to go, but then you tweeted recently, "Don't fall into the Chinchilla trap if you want your model to be used by billions of people." So what's the updated state of, of scaling laws? I, I think there was obviously the Kepler and then there was Chinchilla, and then people kind of called the Llama scaling law, like the 100 to 200x kind of parameter to token ratio. What's your updated thinking on how to think about scaling laws when you pick model size and, and training data? **Thomas** [10:24] Right. So y-you know, there's this, uh, as you said, like this Kaplan paper with scaling laws, but they figured out... Basically, they tried two dimensions: the model weights and the number of training time, like number of steps, training tokens, epochs. And for that, they figured that model size is what matters, so GPT-3 was way too big compared to the actual number of training tokens because they did a mistake not adapting the scheduler. That's what, uh, Chinchilla emphasized and discovered. To be fair, I think OpenAI knew that at the time of Chinchilla paper. But yeah, basically, Chinchilla said we have to revisit the scaling laws, uh, uh, generally, uh, published by Kaplan and emphasize much more the importance of training tokens. And they did like some really good scaling laws showing that there's an optimal-- Basically, you need to double the number of training tokens every time you double the training weights to get an optimal ratio so that for a finite number of compute, you will end with the best results in your paper. And what I call the Tr- Ch-Chinchilla trap is that that good if you want the best flagship model that obtains the highest performance on your paper. But if you want to use your model at inference time, inference, the two dimensions, one remains the model weights, but one drops the number of tokens you train it, number of steps. And so to be compute efficient at inference time, it's much better to train it much longer training time, even if it's an effort to, uh, an additional effort, than to have a bigger model. That's what I call, like I refer to the Chinchilla trap. Not that Chinchilla was wrong, but if you consider inference time, you need to go beyond Chinchilla And in fact, that's what Llama 1 folks did by over-training in the sense they could have get a better performance in paper, but they preferred to create the best artifact that will be used by the community. ### Model Decisions **Swyx** [12:15] So that's the scaling thinking. What other went into Llama 3 kind of planning, you know? So Llama 2, you have a pretty good model. People really liked it. In Llama 3, you dropped like the intermediate weight, so it's, uh, 8, 70 and now 405B. What was the thinking behind going so large? I mean, you talked about the hardware capabilities at inference. Like, I cannot run a 405B model at home for sure, and it, it might be hard to even get the, the cloud resources to do it. What was the decision there? **Thomas** [12:43] The decision is super simple. We want the best model. We want to be number one and number two. We started one year and half ago and we did quite some journey. We filled the gap with GPT-4, so that will be the first, uh, open source model that actually compares to GPT-4. There's now GPT-4.0, of course, and we're close but we're not there yet, uh, not in all capabilities, but the gap is, uh, getting smaller and smaller. There's also like what compute we had at the time when we started the journey in January. We put a lot of effort there, but as like Mark announced, we have more and more GPUs, so the next generation will be bigger. So that's what drives the decision. Now maybe let me reflect two things you said. You cannot use it at home. That's probably true, but quantizing it to FP8 can run on node even with a long context of 128k, uh, tokens. Second thing is I'm hopeful that the community will lead to a lot of findings by open sourcing it and very smart way to actually make you use it on, uh, your computer. If you remember Llama 1, Llama 2, like when we published models, people were saying it's too big and after two weeks it was running on a Raspberry. I don't know if it will be the same, but I hope the same kind of trend. And by releasing those models, we are enabling that. Now the last thing I want to add is having bigger models enable us to collect better data, for instance, at RLHF stage because that's the model we use for the annotation. And so we distillate straight forward like those annotation from this better model to the other models. So I can guarantee you that the quality of the smaller models we are releasing with Llama 3 are also thanks to having these artifacts where we can collect and training. **Swyx** [14:28] Yeah. There's a lot of really good info there. One thing I'll, I'll just briefly touch on for quantization. There was a recent Noam Shazeer blog post. No- Noam is writing again for, for some reason and, uh, you know, he was talking about sort of native FP8 training. It seems like that is most useful for inference. That is what you expect the open source community to do with your weights once you release them anyway. Is there any movement or thinking about just moving to FP8 or, you know, whatever oth- other new format is in vogue these days? **Thomas** [15:00] Also these papers like to train like some, I, I forget the name, but like there's two follow papers on like just, uh, zero one or minus one weights and like there's a lot of work there. I think it's promising directions overall. Regarding FP8 in particular, there's also the possibility for the community to try FP8 or other methods, uh, that are very easy at fine-tuning time for the model. So I'm really looking forward what the community can build there. Overall, like scaling, I don't know if it's all you need, but I will not bet against scaling and one of the way to get more scale is by having, uh, better algorithms that we can train for this same level for less compute. **Swyx** [15:41] Less compute and less memory. Yeah, like it, like inference time memory is, is becoming a real constraint. **Thomas** [15:46] Yeah, yeah. But also training with, uh, FP8. If you're not training with FP8 or, I mean, FP0 is, uh, probably nonsense, but to what extent, how far we can go, you know? And every time like you unlock compared to, uh, what we had two, three years ago on, uh, 32 or 64, it's like huge progress in term of scaling. **Swyx** [16:05] For me, it's interesting to say-- to see you mention the ternary quantization, like the 1.58, uh, bit thing. **Thomas** [16:12] Mm-hmm. **Swyx** [16:13] 'Cause I didn't know that... I don't know how much to believe, you know. Like there, there's a lot of these kinds of papers where it makes a lot of noise but it doesn't actually pan out or doesn't scale. **Thomas** [16:21] I totally agree with you. It's so hard for researchers, at least for me, to see all those papers published, all those cool ideas, all those results that are preliminary and in all those massive amount of research, what will scale or not, what will resist the test of time or not. And are we like losing maybe some gems that are not just people are not working on them, but because there's too much research around? I don't know. Maybe. And that's like some problems to have. That's cool to have these problems nowadays compared to probably what Yann LeCun and the others had 30 years ago, but still it's a problem. **Swyx** [16:58] You know, for what it's worth, like I do think that FAIR is putting out like incredible research. You know, uh, it probably doesn't seem like it's your group but, you know, you also recently published MobileLLM, which on the small model side is, uh, is a really good research on just small model architecture that, uh, it looks like Hugging Face is also replicating it and it, like it's doing quite well. Like, you know, there's a lot of ideas on shared weights and shared matrices and, you know, model architecture stuff that we can talk about for smaller scale models. Like Llama is not at that scale, but it seems like one of the big themes of this year is like on device, in browser small models that are like good enough for, you know, daily use. I do wanna talk about architecture, right? Like, uh, I'm not sure when you're releasing the Llama 3 research paper, but in Llama 2 you talked a little bit about, uh, the architecture choices. Like any, any- **Thomas** [17:46] It will be released the day, I think, of the release. ### Data & MoE **Swyx** [17:49] Okay. What should people know or what are the, the major choices of Llama 3 versus Llama 2? **Thomas** [17:54] There's not like a, a lot of changes in term of architectures. I think we can di- do a lot better in the future and not just like with transformers but, for instance, to me like it doesn't make sense to use the same amount of compute per token for every token. Like those architecture lack of flexibilities, there's a lot of research to go there. But still that's the best thing we have for now. And so it's the same recipe than in term of architectures and training than Llama 2. But we put so much effort on scaling the data and the quality of data. There's now 15 trillion tokens compared to 2 trillions. So it's another magnitude there as well, including for the smaller models. **Swyx** [18:33] One of the things I noticed on the paper is that you used Llama 2 to do the data cleaning for what went into Llama 3. I think there's a lot of chatter obviously about synthetic data and like, uh, there was the rephrase the web paper that came out maybe a, a few months ago about using, you know, Mistral to make training data better. Any learnings from that? It's like, is there... How much can you rewrite with the models? Like, uh, I'm sure people would love to hear more about it. **Thomas** [18:59] Right. So it's a very interesting re- research direction, synthetic data in general, synthetic data for pre-training. My intuitions is that the web is full of shit in term of text, and training on those tokens is a waste of compute. Just having a good classifier that labelize that is cool, and Llama was the be- at the time before Llama 3, the best model we had access to, uh, legally to labelize the web and select what are the good tokens and the bad token. The additional thing is that it also enabled to have a topic tag, like is it about law, is it about politics, is it about chemistry, math, reasoning? So that you can also adapt a bit the mixture to like balance a bit more the diversity. **Swyx** [19:48] To me, you know, I, I'm not exactly sure what you guys did, but like I, I feel like when people say synthetic data, there needs to be different categories of synthetic data now because I think there's so many different usage of this thing. But specifically synthetic data for pre-training, it feels almost like you're running multiple epochs on the raw data when it's-- while it's rephrased or reformatted by a language model, right? And in my mind it's very similar to computer vision where you do data augmentation on an item, right? Like we're, we're doing data augmentation. That's the less cool name for synthetic data. **Thomas** [20:24] That's very interesting. I, I totally agree with you related to pre-training. Totally, uh, stamp what you said. I think it's very different though for post-training in the future direction on synthetic data that I'm personally excited. Like for instance, what I'm excited about is we had this survey on augmented LLM a year ago, and all the idea is like if you augment your LLM with something else, it can be a retriever, it can be search, it can be a tool, it can be a calculator, it can be a code execution, then you are not just distillating like d- doing some data augmentation with your model, but you're actually adding some expert skills that possibly goes beyond the model weights. For instance, if your model like can calculate something it was wrong before, and now it has access to a calculator, and you can retrain your model on that, then you're learning something new. If your model didn't know something about Llama 2, probably it doesn't know a lot about Llama 3, but now if it can search online about it, and then you train the model on that, then you have a feed-- a positive feedback loop, like what we call expert iteration, targeting directly the weakness of the model. It's like continual augmentation of the language model much beyond just that augmentation. **Swyx** [21:36] How related is this to tool use? Like are you teaching it to use tools to augment the model, or are you saying like, uh, "Do active learning. Do, do-- Like where it's weak, go augment the model with, with extra data and then memorize that da- that new data," right? **Thomas** [21:51] What I said is more like in term of directions, not for Llama 3. But like when it knows how to use a tool and correct itself, this is like a very promising direction that goes much beyond the augmentation for like the-- in the future to keep collecting new data, new token. People are saying like we are lacking of tokens. But if you think about those kind of tokens where the model always go to correct its own weakness, it can say like, "Okay, that's ten plus ten." Okay, that's an easy example probably the model knows, but for imagine for something more complex. "Ten plus ten, I expect this to be twenty. Let's verify with a calculator," which is easy for a basic agent now powered by an LLM. And then you verified with respect to what you expected that it's correct. If it's not, you can back propagate those example directly to the weights, and the model will keep learning new things. **Swyx** [22:40] It makes sense. What if you... The insights, you know, you, you mentioned about just like using calculators. Uh, what are the insights, uh, I think just in general a lot of that is just driven using code generation and apart from just tool use. What are the insights on just like the data mix of how much code, how much multilinguality, which is something that you're also passionate about? We know that that's changed for between Llama 2 and Llama 3. Um, is it changing for different stages between the different sizes of Llama 3? Like, you know, anything, anything like of that sort? **Thomas** [23:08] No, it, it didn't. For the different size we use the same mostly. What happened is, uh, we changed the data mix during the training of Llama 3 with some findings that happened that... I mean, training is long, so you have to do something while it's training. And what the team did, I was working on my side mo- mostly on post-training, but so the pre-training team did quite a lot of work to find some-- have some new findings, improve the data mixture along the way, and they intersected before the end of the training. **Swyx** [23:36] I sense a movement in terms of like the curriculum, uh, that people are adopting doing pre-training and even post-training, uh, about, you know, what the mix should be like. Uh- **Thomas** [23:45] Yeah **Swyx** [23:45] ... Snowflake is doing some interesting work with, uh, enterprise, uh, intelligence or whatever they call it. What are your goals when post-training? Like just at a high level, you know, like what do you work with like the pre-train team? **Thomas** [23:56] I think it's quite easy for now because there's not yet like this kind of continual augmentation where it could feedback like pre-training, thing like that. One of the big continuum between pre-training and post-training in particular is continual pre-training where you actually, uh, continue the pre-training before RLHF in a self-supervised way but on expert-level domains, like for to have a, an expert in code, an expert in like reasoning or an expert in multilinguality that enables to collect even better RLHF annotation after. So that's one thing. And then you start from those models to actually do the RLHF stage. And goal about your question, like goal was to get the best model in those dimensions. That's actually one thing very different to, I, I can comment, compare to Llama 2. Llama 2, you know, as I said, we were nowhere. We build entirely end-to-end all the stack from data annotation contract, methodology, protocol, algorithms for RLHF at Meta, and we had to limit our scope. We were, like, not allowed also to work on that. We focused mainly on helpfulness, following instructions for Llama 2. And you can see that as in the following months after Llama 2, a lot of open source models came distillating GPT-4 mainly, but obtaining better reasoning, math, coding, chat models. And we didn't annotate at all for code, neither for, uh, reasoning or multilinguality. And o-one thing I'm quite proud is with the early, uh, preview release we did of Llama 3 back in, uh, February, May or March, don't remember, it led quickly to instantly to, like, state-of-the-art results for the model size, almost competing with GPT-4 on the, the Arena leaderboard, where, like, human fight each other, compare, like, uh, two models and select their preference. And no one since then had been able to put, like, a Llama 3 model better than what we did on most of the domains from code reasoning, multilinguality, uh, helpfulness. So that to say that this time, as opposed to Llama 2, we tackle, like, all those different aspects. **Alessio** [26:01] Do you have any other thoughts on the more synthetic data-focused models, kind of like, uh, Nemotron? I think folks were asking if you see that as an interesting, uh, direction too, kind of having specific synthetic data generation things. **Thomas** [26:14] I don't know about this model exactly, but I think, like, Llama had better performance overall. I'm very bullish on synthetic data generation, but I think just gets better when you have a better model. I'm not really bullish on having, like, a model only for synthetic data generation. I understand the need of having, like, bigger models that then you can rationalize saying, "Yeah, maybe people will not use them for inference, but to distillate some specific knowledge of synthetic data." That narrative is, I think I totally, uh, agree with that, but having a model purely for that and not, like, good at other things, I don't think it's the case. **Alessio** [26:51] That makes sense. One of the architecture questions that I forgot to mention in there was so just the architecture choice of, like, a very big, you know, 400B dense model. I actually honestly thought that maybe 175 or, like, you know, was kind of the, the, the peak, you know, what-whatever can fit on, like, an H100. ### RLHF **Alessio** [27:08] So basically, I think the common question that people have is, like, why no MoE in a way that Mistral and the others have, have gone and, you know, it seems like the trend has been MoEs, and you guys have bucked the trend there. **Thomas** [27:19] Uh, I heard that question a lot. Different aspect there. Why not MoE in the future? The other thing is, I think a dense model is just one specific variation of the model for an hyperparameter for an MoE with basically one expert. So it's just an hyperparameter we haven't optimized a lot yet, but we have some stuff ongoing, and that's an hyperparameter we'll explore in the future. **Alessio** [27:44] Let's make sure we run through everything on, on post-training. You also had a recent tweet about RLHF versus imitation learning explained in one tweet. So we'll put this in the show notes, but it's basically, like, two charts about, uh, doctor opinions. On one side, there's, like, whether or not the suggestion is good from, like, a content perspective, and the chatbots rank really highly, and the physicians are kind of like, you know, a bell curve, as you might imagine. But then the empathetic voting, most physicians are rated not empathetic or slightly empathetic versus all the model responses are rated very empathetic and empathetic at, at worst. You know, most people might look at it and not really get much from it, but obviously it resonated with you. Can you run people through, like, some of the choices you make in post-training to, like, optimize for one of the two and getting the best responses? **Thomas** [28:33] I think the tweet was about, like, the intuition of why reinforcement learning with human feedback works. When we started, uh, Llama 2, I had, like, this budget of annotations in millions of dollar and, okay, what to do? I'm responsible of that. I'm accountable for a model at the end that can follow instructions and compete with GPT 3.5 at the time. What to do? You can annotate supervised fine-tuning data, which refers to a human to create a prompt and to also write itself, himself the, the answer expected by the model. So then you train on that and in a supervised manner. Uh, that's, like, very classic and standard on fine-tuning machine learning. The other thing is reinforcement learning with human feedback, where the annotators type a prompt, but this time you sample two different answers from your model, and you ask the annotator which one he prefers, and then you will train on the preference basically to, to simplify it. When you ask to train on the preference of the model, that seems very weird and, uh, not really robust training on synthetic model by generated by the model. So I was like, let's annotate 100,000 more of supervised fine-tuning data, and let's annotate a bit of preference to do RLHF because everyone is doing it. And we had this human evaluation after a few weeks in, uh, Llama 2 projects where our model was already better than the annotation from the humans. So you'd get a prompt, you check what the human will have annotated as an answer, you check what the model generates, and most of the time the model was better. I was like, "Oh, maybe the annotators are pretty bad. Let's look at that." And no, like, the model was pretty good. And so I understood the intuition behind RLHF. Like, this model are already super good at some tasks. And with RLHF then what you have is imagine a distribution, a Gaussian distribution, which was, like, basically the tweet. And you have on the left, like, bad outputs, and on the right good outputs. And the same by, like, medical diagnostics from a doctor. You have good outputs on the right and the bad diagnostic on the left. But the... You have the distribution then when you collect all the diagnostic from doctors Hopefully it's mostly on the right. There's better m- a lot of time good diagnostics, but human makes mistake, right? So there's bad diagnostics. On the left you have still a bit of, uh, examples which makes like your curves not at zero, uh, the distribution. And the same way for humans, like they make mistakes when they annotate, and so training on behavioral cloning to reflect humans, so they will learn to, to do also some mistakes just like humans. And so you will have some bad outputs from the model time to time reflecting humans, and you cannot go beyond that if you train on human outputs. But now if I ask a doctor to check a sample from my model or a sample from two doctors, one diagnostic and this, and another diagnostic, one is better than the other. It's easy for doctor to say which one is better. The same way if I sample from my model that learns a human distribution of answers, and there's one bad time to time like humans, but most of the time good answers. And ask a human to choose which one he prefers. Personally, I'm really bad at creating poems. The example I give a lot of time, try to write a haiku in three lines of about large language models. I don't know you. Take like five seconds to think what you could come. I'm terrible. But yet if I check two poems generated by models or human, I can tell which one I prefer. I'm good at discriminating. And because of that, you can have a model that flats the bad outputs and learns to only shift towards the best and better and better outputs. And you can even aim to superhuman abilities since that I'm bad at writing a poem, but I'm good at judging which one is better. So I can actually annotate data beyond my own skills at creating them. That's the magic of RLHF. **Alessio** [32:19] Yeah, this is-- We have one episode, RLHF 201, with, uh, Nathan Lambert from the Allen Institute, who was at Hugging Face leading RLHF before. And he mentioned one of the things that makes RLHF work is that humans are not maybe great at creating a lot of things, but they're usually very good at giving an opinion on wha- which one, two they prefer. So they're able to actually annotate data of things they, they would never create from scratch. One question actually that he asked me to ask you, how much in post-training you attribute improvement to the RLHF side versus the instruction fine-tuning side, and maybe how you think about prioritizing the two and what areas they impact the most? **Thomas** [32:56] You mean between supervised fine-tuning, like supervised fine-tuning annotation and preference annotation? **Alessio** [33:02] Yeah. **Thomas** [33:03] So 100% to RLHF. In fact, that's quite interesting. You start for Llama 2 with a pre-trained model, and you have to have an instruction model to chat model. Otherwise, like the model is just like continue finishing sentences. So you need that to start RLHF. So we had to annotate like 10,000 examples. What did we do for Llama 3? You start with a new pre-trained model, and then you want, before starting the RLHF, to have a now a chat model that is not too bad. The option one was let's do human annotation again, like SFT stage. But in fact, by the principle I said before, the annotation will be actually worse than Llama 2. So what we did is that we generated all the data and the prompts with Llama 2, and we applied like basically the last round of Llama 2 we had to kick off and start Llama 3 post-training. So Llama 3 post-training doesn't have any like human written answers there basically almost. It just leveraging pure synthetic data from Llama 2. **Alessio** [33:58] Do you have an intuition on which areas work better for which? For example, you mentioned the physicians are expert. What about maybe like code or... Yeah, you also have a multi-model working on, so like image generation is like or does this apply to any modality, any subject? **Thomas** [34:12] That's an open research question. The intuition in general is that like for instance for code because this is factual, you can check if the code is correct or not. RLHF is not the way to go. You prefer to do like supervised fine-tuning as a human to write the code. But in fact, because humans make mistakes because actually even in code there are some preferences that emerge from like that, and maybe for some other reasons that we don't know, RLHF is so much more scalable. It costs less, it's easier than it leads in general to just better performance. And maybe we can come with a compromise. We actually, uh, suggested teacher forcing in Llama 3, a new method that kind of fills a gap between... Not teacher forcing, sorry, teacher critic. Teacher forcing is a way to train the models. Teacher critic where it reconciliate and unify supervised fine-tuning and RLHF such that when you do human preference and you have two outputs, but both are very bad in the code, for instance, you will ask a human to edit the best answer to make it correct now. So now you are doing SFT when all the answer was really bad, so that you can get out from the local minimum of your model. **Swyx** [35:18] I think this is like super promising and it, it seems like there is just... Well, do, do you have an idea, uh, you know, you started with this question of how much scale you need. Uh, do you now have a better idea? **Thomas** [35:28] No. What we know is it's not plateauing yet. **Swyx** [35:31] It's not plateauing yet. Yeah. **Thomas** [35:32] No. **Swyx** [35:32] So just infinite amounts more. Well, you know, Scale AI and all the an- the annotation providers are very happy to hear that. And, uh, so you mentioned at the start of the conversation about the AlphaGo moment, and I feel like this is very interesting to reflect on, right? Like, uh, we're, we're basically saying that I think that one of the lessons from AlphaGo is that people thought that human interest in Go would be, would be diminished because computers are better than humans. But then we have this sort of centaur model where like humans and computers are actually doing better than either humans and computers would be alone. And I think we're seeing that with this R... W- where you're talking about this RLHF, uh, improvement, right? That we're s- kind of building human preference into the model and like the blending of the, the human preference and the model capability is actually doing better than we could on our own. I just think it's pretty fascinating. **Thomas** [36:24] It is fascinating. **Swyx** [36:25] The other thing is RLHF came, came from the alignment community, and I think there's a lot of conception that maybe it's like due to s- safety concerns. But I feel like it's like really over the past like two, three years expanded to just this produces a better model, period, even if you don't really are not that concerned about existential risk. I always feel like it's so interesting to see this Like, people who take alignment super seriously, they, they are the first to consider super alignment. And now we're consider, like... Yeah, I'm almost thinking about this as, like, super quality, that we are training models that are higher quality than humans, and it's not really about alignment so much as, like, we now see that this is actually possible. ### Evals **Thomas** [37:06] Yeah. **Swyx** [37:07] And it's not even for alignment purposes. We, we just think it's, like, better at reasoning, better at knowledge, better at everything. **Thomas** [37:12] Well, I don't know how much better yet it is at, on those, but clearly it's superhuman on some writing skills, and it's super useful. I think that's great, to be honest. **Swyx** [37:21] Yeah. Perhaps we can transition to evals. We, we had some questions about, uh, the 400B details that we want to disclose. You know, by the time this podcast comes out, we, you know, we'll have disclosed them. Yeah, I think last time you disclosed, like, the evals while those, while you were still training. What should people know about the, the high level headlines for the new Llama 3? **Thomas** [37:43] At a high level, it's the best open source model ever. It's better than GPT-4. I mean, what version, but by far compared to the version originally released, uh, even now, I think there's maybe the last clouds on a 3.5 and GPT-40 that are outperforming it, and that's it, period. So for the 405B, that's a flagship. That's pretty, pretty good model. Not yet the number one. We still have a journey to, to get there. For the 70B and 7B, they are like world-class model for this size for general models. **Alessio** [38:17] And are the benchmark numbers from the initial checkpoint still right, so the April 15 checkpoint MMLU on instruct is, like, 86, GPU A 48, HumanEval 84, GSM8K 94, math 57.8. Is this still roughly the same performance or... You know, I haven't seen the numbers yet either. Uh we're just breaking the news right now, so. **Thomas** [38:40] No, it's roughly that. **Alessio** [38:42] Awesome. So talking about evals, we just had an episode with Clementine from Hugging Face about leaderboards and arenas and evals and benchmarks and all of that. How do you think about evals during the training process? And then when the handoff happens, do you already know exactly what you wanna improve? And I know that, for example, to improve, like maybe an arena score, you need different than like an MMLU score. How do you think about prioritizing the, the post-training improvement based on benchmarks? **Thomas** [39:10] That's a super hard and good questions. There's no good answer. Like, I mean, eval's an open research problem, like in particular when you're trying to tackle so many capabilities and, you know, it's also like as soon as a benchmark you're trying like to, to push numbers on a benchmark, it stops to be a good benchmark because then you don't know if you're overfitting it, and it will transfer to similar capabilities. So evaluation for language models, in particular on post-training, is very hard, uh, problem. We tackled that, uh, playing with different methods like reward models evaluation, model as a judge, having a diversity of prompts, diversity of benchmarks as well for a lot of different capabilities. That limits the possibility of hacking them, of course. We do also l- lot of human evaluation. I do also a lot of, uh, model test quality analysis, like testing myself some prompts. I feel it was much easier during Llama 2 when the model was, like, worst than today. Now the model are s- getting so good that it's hard to get to some prompts to break them and to compare models and see the edge cases, so it's getting harder. And a great way also to compare models is, you know, through f- uh, the different round we have done for RLHF. Every time we upload a new model, for all the annotation we are doing, we have the win rate between the previous model and the new model by just sampling for every prompt we annotate mo- uh, prefer sample A with the old model, sample B with the new model. And so we can calculate automatically win rate. **Alessio** [40:45] Interesting. What are areas that you had to work the hardest to catch up to, like, the private models? Maybe like there's, you know, not as good public data or whatnot, or is performance improvement just kinda even across the, the spectrum? **Thomas** [41:00] Honestly, all of them. We are behind all of them with, uh, between Llama 2 and, uh, GPT-4. I mean, it's different challenges every time. Like being good at code or reasoning is something we didn't do at Llama 2, so we had to build everything from scratch. Improving on helpfulness and which is one of the main dimensions that people look at, I think, in the arena. But which is, by the way, a very interesting evaluation because when we did the preview, and I don't know yet what will be the results for this new Llama 3, but we ended very high in this blind test leaderboard. And to be honest, I didn't expected that. I knew we had good results internally, but how that will transfer to perception from the community, people like using it in practice and comparing it to the other models, I didn't expect that, uh, positive feedback, that high ELO score, uh, on this benchmark. It doesn't say like everything. As I said before, which is also interesting because it's a community that judge the prompt and create the prompt and judge the answers. We are limited. We are not, like, good to do that. And so it gives you a very good indicator of how good helpful, how on the main curve the distribution, simple prompts about the tone of the model compared to the others. But for much more complex prompt, much more intelligence reasoning, coding of complex stuff, it doesn't tell the, the full story. You know, like while we had 70B preview at the level of GPT-4 even be the better at the time, I think it was partly true, but clearly we were not at like GPT-4 level on code or reasoning. We, we are now. **Swyx** [42:37] There's some conversation about like the, the math score. S- s- apparently like the next-- GPT Next or whatever is has, has like reached 90, which is a big, big jump from the current state of the art. It will be interesting. One of our previous guests rounding out the topics on like just potential models, areas of development, and evals, Clementine is looking for a confidence estimation or uncertainty benchmark. Uh, one of our previous guests, Brian Bischof, is also asking about, like how do we think about evals for practical things like confidence estimation, structured output, you know, s- stuff like that? **Thomas** [43:11] Yeah, I think we lack actually of such evaluations. When numbers are-- I was just releasing like two days ago to, to the team to, to report at some point is, okay, we have this accuracy on MMLU, on, uh, whatever, uh, on math and GSM8K. What if we change a bit the prompt and instead of telling the model you have, uh, this question, you have to answer A, B, C or D? What if we tell the model you have to answer A, B, C or D or you don't know? And maybe the accuracy will be a bit lower, but I'm curious to see if some models will have difference of calibrations where maybe model A have fifty percent correct, model B has fifty percent correct, but model A answered a hundred percent of the questions, so fifty percent are not correct. Model B actually said like answered only sixty percent, so for forty percent of the time, he said, "I don't know." I prefer model B, and we are not like reflecting that in evaluations. **Swyx** [44:04] I think this is very relevant for post-training in particular because it, it seems that the general consensus is that base models are more calibrated than post-train models, right? Something like that. **Thomas** [44:15] Exactly. **Swyx** [44:16] That seems to be the research from OpenAI as well. I, I don't know the degree of this and, and like maybe we can invert it, right? Maybe post-training can help to increase calibration rather than decrease it. I feel like this is a little bit of being too similar to humans because humans are not calibrated very well. **Thomas** [44:34] Yeah, no, but that's the goal of post-training, I think to, to ma-make models more calibrated, to not be biased toward like answering A, B, C or D as often as possible, uh, to follow the uniform distribution. **Swyx** [44:45] And on the structured output tool calling side, do you think that it's not a explicit part of the evals? Obviously, you, you know, you worked on Toolformer and y-you on-- on the language augmentation. Like, do you encourage pe-- the, the open source community to fine-tune Llama 3 to do tool calling or do you want to just have that in the model from day one? **Thomas** [45:05] We have that from day one. Good news for the community. We are state-of-the-art there. I think the model will be pretty good at, at that. We, we have a lot of gems about, uh, tools in the paper, but the model is fine-tuned to do tool usage to zero-shot function calling. There's some system prompt if you tell the model to do. It can use a search and imagination, can do a lot of stuff like code execution as well in even in a multi-message way, so almost multi-step agents, which kind of sparks off agents. **Swyx** [45:38] Okay. You talked about agents, so I, I guess we should probably mention the, the work on agent stuff. And you also in our pre-conversation, uh, mentioned that you're already starting work on Llama 4. What does agents have to do with Llama 4? How does your work on Gaia, you know, inform all this work? **Thomas** [45:51] Yeah, you know, so we published one year ago Gaia General Assistant Benchmark, uh, that followed the direction I really like pursuing. I mean, everyone passionated about AI and trying to build Jarvis will go there. So I did Toolformer, uh, and this survey on augmented models. ### Agents **Thomas** [46:08] In fact, you know, reflecting back, I was okay, we have Galactica, we have, uh, Llama 1, we have Toolformer, and there's like GPT 3.5 at the time and then 4. If you don't have a good instruct model to follow instructions, the extension and the future of Toolformer is limited. So we need to work on that, and we did Llama 2 and then now Llama 3. And it's very interesting. On General Assistant Benchmark, so Gaia, agents powered by language models perform to zero with GPT 3.5 and to something very, um, significant like thirty, forty percent, sixty percent with GPT-4. So there's a gap of intelligence here. And I think this gap of intelligence, this threshold that you pass in term of zero-shot function calling, following complex instruction that can, uh, span over a page of constraints, all those things that makes the cur-- the nowadays agents with react loops, pre-planning, multi-steps reasoning, function calling work in practice is like this gap of intelligence. So now that we have Llama 3, I'll be back to agents. I expect some incremental and, uh, significant progress on pre-training and post-training, but I'm really hopeful that we can gain some order of magnitude of scaling by interconnecting world models into agents as a more complex system that can do planning, that can do backtracking, that can, uh, take actions, navigate the web, execute code. **Swyx** [47:37] W-- okay, there's a lot there . When you say, when you say integrating world models, um, is there anything from JEPA? Is that like, is that something that we're, we're talking about or is that a different line of research? **Thomas** [47:49] No, not directly. That's the same goal I would say, but JEPA is very, very fundamental research which has some promising early results. And, uh, what I was looking right now on state-of-the-art results on Gaia, there's a leaderboard. By the way, uh, you mentioned Clementine before. She contributed to Gaia as well and Hugging Face puts a leaderboard there on, uh, their website. There's, uh, some state-of-the-art results. What is interesting is like GPT-4 alone has zero percent, but or like five percent I think on level one. There's three level of difficulties. But OS Copilot then and, uh, Autogen from Microsoft and recently Hugging Face agent obtains on level one up to sixty percent. So connecting an LLM to an agent that can do all those things moves much forward new capabilities. Uh, this is kind of a breakthrough. And those models are purely based on instruction tuning models following instructions where like you have an orchestrator and you say to your LLM, "Okay, this is your task. You have access to these tools. You can navigate the web. Can you do a plan of what you should do?" And then, "Okay, that's the plan. Now execute the first step. Did you manage to succeed for the first step or do you want to rethink your plan because you enter in a dead end?" And you have kind of all this, uh, orchestration by system prompting Instruction following. And just that, which is quite suboptimal and probably you need to go later in latent space and more Java style, but just that is getting us to some really impressive results already. **Alessio** [49:28] And do you see the planning and review to always be needed in the future? This is kind of like Andrej Karpathy's idea of like more tokens equal more thinking. So like the more you're having it write tokens and like think about the, the outcome and like the better result you're probably gonna get to. Do you think that's always gonna be the case or that in the future, like the model, you can just say, "This is the task," and then it'll just return the answer directly and do all of that in the, in the latent space, so to speak? **Thomas** [49:55] Right. I, I think in the future it should be-- it should hopefully go more as this is a task and I return it. But we need to teach that to the model to train that, which is far from now. Very medium, long-term directions that could be really relevant here is thinking into latent space. I know some early works are doing that, and that's a way probably to move to first you think and then-- and you don't have to write all the tokens like it's in your head. It doesn't have to be as constructed than a plain text LLM. And once you have done your thoughts, you can just write the final answer or take an action. **Alessio** [50:31] Just a commentary on that. Anthropic actually sit-- cheats at this right now. If you look at the system prompt in, in the Claude artifacts, they actually have a thinking section that is, uh, explicitly removed from the from the output, which is-- I mean, they're still spending the tokens, but like, uh, that is before training it is, you know, it's the-- at, at the prompting level you can, you can simulate this. And then at iClearer there was like the pause token, the backtrack token. I feel like all these are token level stopgap measures. I feel like it's still not the final form. Like we still need to have, at the architecture level, some kind of variable inference length thing that lets you actually think in latent space like you're talking about. I, I don't know if you-- there's any papers that you're thinking about. **Thomas** [51:14] No, but that's interesting because that's what we said at the beginning of, uh, the, the discussion. If you remember, like we are lacking of flexibility for pre-training architecture transformers, where we spend the same amount of compute per token. And so because of that, how can you like mitigate this? By generating more tokens, so more thoughts, more compute, because you have like only access to this dimension. Ideally, you want an architecture that will enable naturally to, to make this emerge, basically. **Alessio** [51:43] Any papers come to mind there that you would recommend people read, or this is like completely new science that we have to do? **Thomas** [51:50] No, it's-- I mean, it's earlier science. I don't know any work that managed to get there. I know like for instance you had, um, universal transformer had this idea of a number N... You can like compute on the layer, uh, N times, N being, uh, decided by the architecture itself with respect to the complexity of the token. I think recently there's a paper from DeepMind on mixture of expert with like a skip layer mixture of... Is it this one? **Alessio** [52:17] Uh, mixture of depths. **Thomas** [52:18] I don't-- I'm not sure it's this one. Maybe. But like basically the idea was that you-- with a mixture of expert, you have an expert that is an identity matrix that you can skip and so like you can... But you know, that's early works, pre- very preliminary works. Like for instance, I haven't seen yet a lot like putting the compute generating a token into the loss. That's gonna be interesting when we start to do that. **Alessio** [52:40] I know we're getting up on time, but we got just a, a few more questions we, we definitely wanna ask you. So as you think about there were reports about Llama 4 started training in, in June. If you think about the evolution of the models, I think up until Llama 3, you know, with Meta AI and some of these things, I'm like, it makes sense that they wanna build their own models and their multi-models. Sounds like Llama 4, maybe a lot of the focus will also be a more agentic behavior and evolve this. I'm curious like at what point it's like, okay, this is a research direction that we still wanna take even though, you know, it doesn't fit right into the product. Like what's that discussion internally about what to focus on as you keep scaling these models? **Thomas** [53:17] Yeah. I think it's a balance, you know, between, well, we want to be number one. Mark want to be with number one there. And there's this understanding also that, you know, this is a critical technology in the future. And even if two nowadays that like research, if nowadays it's not like directly intersecting product, we don't want to be late in the game be as we had in the past. So that's the first thing. Uh, the second thing is, and we think like this technology will change the world. Uh, we want to work towards AGI, and AGI will change the world. And if Meta develop an AGI, it will probably intersect pretty easily the, the products. Now, the third thing is, with that in mind, we have to balance with product needs, and there's always this ongoing discussion and this balance to find for like between a flagship model, between maybe a model that will be more adapted to product needs, and it doesn't have to be decorrelated. ### Tokenizer **Thomas** [54:13] As I said before, like you can leverage also the big models to distillate some capabilities to a smaller one that will be maybe more suited like research. There's always this back and forth. There's also the fact that the product kind of hide this to the recent evaluations that are grounded in actual use cases so that we can also measure ourselves with respect to is there some progress or is it just on, uh, an academic benchmark, you know? **Alessio** [54:36] So one-- Be-before we transition off, I think there's the hidden side maybe of these LLMs that most people don't think about, which is the, the tokeniser and the vocab size especially of them. So Llama 3 is 128k tokens vocab tokeniser. GPT-4 was 100k. 4o is 200k. How should people think about the impact that it has? So basically like, I, I mean, the TLDR is like in the vocab you have these kind of like concepts represented as tokens, so usually the larger the vocab size, the more nuanced the model can be about thinking about different things. What are the scaling laws of those tokenisers? You know, is 120k kinda like very large and it doesn't really matter? Like do you wanna double it? Like any thoughts there would be great. **Thomas** [55:22] There's a lot of dimensions to take into account here, I think. The first thing obvious to say is Llama 3 compared to Llama 2 is multilingual, has multilingual capabilities. We worked on that. And so because you have, uh, languages that are not just Latin languages like English, there's a lot of different characters, uh, you want to include them, uh, to represent like special word there. And so you need to have a bigger, uh, vocabulary size. That's the obvious thing, which is also probably why GPT-4.0, uh, has a much bigger vocabulary as it's like naturally multilingual, multimodal, uh, and speech. So that's why we went to from 30 to 128, uh, vocabulary size. The interesting thing I think to discuss about tokenizer is both scaling laws related to that. If you increase your, uh, vocab size, well, you have a bigger matrix which takes longer to compute. It depends on the model size, but for a small model it has a much bigger impact than a bigger model. So increasing that relat-- basically saying otherwise, the number of vocabulary size 428 is the same than the 8, 70, or 405B. But so relatively in percentage of the total number of weights, for the 7B it's much more than the 405B. Way it's small compared to total number of weights. So that has more impact in term training speed there. But what is interesting is with a bigger vocabulary for the same text you have s- less tokens, right? And so you can train your model on the same amount of knowledge with fewer steps. So for the same compute, you can see more knowledge if you don't epoch. That's one cool thing. The second thing is at inference time, you know that the context length is not a number of text or, or the size of the text, but number of tokens. And so you can compress more such that now with a bigger tokenizer, 128 more, more vocabulary, you can get to longer text for the same, uh, number of tokens. 8K basically or 128K now with this tokenizer means 30% about less text to, to, to encode. **Alessio** [57:35] How are tokenizer vocabs built? I actually don't know that. What's the work that goes into it and then, like, why are people using smaller ones? Is it harder to make them or is it just about some of the things you mentioned around scaling the, the training and, and all of that? **Thomas** [57:49] Oh, it's... No, there's different methods but became quite standard although it will change in the future. **Swyx** [57:55] BPE. **Thomas** [57:56] Yeah, exactly. **Swyx** [57:57] Well, BPE is for text. I don't know about multimodal vocab. Uh, that, that's-- I haven't read anything about. **Thomas** [58:03] Yeah. That's a question that we-- I'm not expert there and I don't remember exactly what they ended to do. **Swyx** [58:08] Now that you're saying this, right, okay, so now we have 100K vocab, 200K vocab. It-- do we see a million vocab? Do we see infinity which is no tokenizer? You know, like what's the natural limit of tokenization? **Thomas** [58:22] Yeah, that's a good question. I don't know. I, I think there's a limit with respect that will grow with respect to the model size. So bigger models means possibly bigger, uh, vocabulary without affecting too much the training. But yeah, there's a lot of people, uh, that's not my domain of expertise, but a lot of people are discussing the interest of having this kind of tokenizer which doesn't fit like natural. Could we go to character level tokenizer? Could we go to actually multimodal tokenizer which will like decompose at pixel level? I don't know. Future directions that could be very promising. **Swyx** [58:59] Yeah. I would say the diffusion people ha- have actually started to swing back to- **Thomas** [59:03] Right **Swyx** [59:03] ... uh, pixel level and probably that will presage the language people also moving towards, you know, 1 million vocabulary and then, you know, whatever the, the natural limit is for character level. ### Startups **Alessio** [59:15] I think we can maybe transition towards some of your personal stuff. We kept you here for a long time. We also-- This is a very distributed podcast. You know, I'm in the Bay Area, you're in France, Shawn is in Singapore, so, uh, everybody's on a, is on a different time zone. You also do, you know, some startup investing and advising. You know, we also meet Chintan on the podcast. He also mentioned he, he always enjoys kinda working with founders and researchers. Any company you're involved with that you wanna shout out that you think is super promising, requests for startups that you've had, anything kinda around that space would be, would be awesome. **Thomas** [59:48] Two cool companies I can think now is, one is Lindy which is based in the Bay Area with Flo Crivello. **Swyx** [59:54] Mm-hmm. **Alessio** [59:55] Yeah, yeah. **Thomas** [59:55] Very cool one. **Swyx** [59:57] Yeah, he's a good friend. **Thomas** [59:58] Uh, Flo. **Swyx** [59:58] Why do you like it? **Thomas** [59:59] Flo is really good, like the French martini is. And, uh, number two, very recently, uh, I really liked Open Devin which is, uh, basically trying to reproduce Devin. **Swyx** [1:00:11] We interviewed him at iClear. Both are agent startups. What do you think is like the direction that startups should be working on, you know, agent-wise and maybe what, what is not working? **Thomas** [1:00:21] That's a tough question. One thing I said, uh, quite often is deep learning has these very specificities that makes it challenging to predict that it's self-destructor, self-destructive technology such that things like, you know, Grammarly, this technology like where, uh, the startup you plug play and it corrects your grammatical errors. Everyone told them, "Guys, deep learning create a barrier to entrance, annotate data, create data." And they had a lot of data for that. And the next day with the same exact technology, deep learning, someone comes with ChatGPT and tell them, "Yeah, I can do the same better and so many other things." Zero barrier to entry a day from yesterday to today. And what is crazy here is that it's based on the same technology. And so there's a lot of people working nowadays to try to mitigate issues with current generation of models. And I'm telling them like assume always the next generation will get better. So if your business will benefit from a new generation with better abilities, that's a good business. If your business may be replaceable and if all the work you have done may vanish and be like wasted because there's better models than maybe change. **Swyx** [1:01:35] Yeah. I mean, yes, but better is so unpredictable. Like, if you asked me before, let's say March of this year, I would have said that maybe, you know, voice chat is still very defensible, and then suddenly, you know, OpenAI demoed their sort of real-time voice thing. It's sort of natively multimodal. **Thomas** [1:01:55] It's easy to, to not anticipate a dimension where it gets better, but find another one that resisted, it's harder. I would say in general, assume you will have progress everywhere. It may not be r- right, but it's, uh, a bit dangerous to bet against that. **Alessio** [1:02:12] Is there any space that you think is overrated by founders that are trying to build something that like, yeah, either, you know, the new models are just gonna do or like you just don't think there's that much interest from folks? **Thomas** [1:02:24] It's a challenging time for founders. It's very exciting. There's a lot of funds, a lot of applications as well, a lot of, of stuff to build. That's pretty cool. But what is hard is because this technology is moving so fast, I see like now a lot of fundamental stacks that are like the unicorn of today. Foundational models, foundational like clusters, data annotations, like that. There's a lot, but less successful yet, for now at least, application company. And it's hard to build an application when it changes so fast, uh, as we discussed before. So it is both crowdy and yet like we haven't find a good like use case that is like the new FAANG company, uh, there. And I want to see that. **Alessio** [1:03:07] Yeah, yeah. We definitely see the same. You know, all of our agent companies, or at least, you know, building agents are the ones getting the most traction. Most companies are like, "Hey, I actually don't have that much expertise and, uh, I'm just waiting for the models to get better, so I, I'm not really sure if I need this now." So it's an interesting time to be investors. Anything else we missed? This was kind of like a master class in how to build state-of-the-art LLM, so it's gonna be a- ... a highly, highly played episode, I'm sure. Any final thoughts you wanna share? **Thomas** [1:03:35] There's two things I can... I guess I can say. One is Lama is hiring, uh, talents worldwide. And two, you can contact me or reach me out on LinkedIn, looking for GenAI technology that, and founders that will create the, the future. **Swyx** [1:03:51] Okay. Hiring one role that you're like, "Man," like, "we really need this, this kind of person." If you describe it, th- that person will be f- will, will be referred to you, right? Like, because we're, we're trying to broadcast it to e- to the whole world. **Thomas** [1:04:05] Researchers with good common sense, first principle thinking, not necessarily like huge expertise on LM, but more being super rigorous, meticulous, uh, structured. **Alessio** [1:04:15] Azam, thank you again for coming on, and hope everybody gets to enjoy Llama 3 today since it just came out. And we'll have you again for Llama 4. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.