# LLM Asia Paper Club Survey Round Latent Space · 2024-05-22 The episode surveys four recent papers on LLM reasoning, uncertainty, interpretability, and efficiency. 'Let's Think Dot by Dot' shows that filler tokens (dots) inserted between input and output enable hidden computation, outperforming no-token baselines on tasks like 3-Sum and 2-Sum Transform. 'Uncertainty Estimation' trains a random forest on hidden-layer activations to predict response confidence, achieving higher AUC than unsupervised methods on Q&A and translation. 'Monosemanticity' uses sparse autoencoders to identify interpretable features (e.g., a DNA-detection feature) in a toy transformer, advancing mechanistic interpretability. 'Medusa' attaches multiple prediction heads to the final hidden state to speculate future tokens, enabling faster decoding without a separate draft model and training in five hours on 60K samples. ## Questions this episode answers ### How can filler tokens like dots improve LLM reasoning without semantic meaning? Experiments by the authors on a scaled-down Llama model show that inserting filler tokens (dots) between input and output improves performance on complex tasks like 3-Sum and 2-Sum Transform. The dots carry task-related structural information, enabling hidden computation, even though they lack semantic meaning. Performance gap widens as task complexity increases, and using filler tokens can approach chain-of-thought reasoning accuracy. [5:08](https://addtry.com/ec96c7c0-4c99-42bf-b403-b0a8f32a4470?t=308000) ### How can you estimate uncertainty of LLM responses using a secondary model? Gabriel’s group trains a random forest regressor on a white-box LLM’s hidden layer activations (especially middle and last layers) to predict a task-specific confidence score for each response. This supervised method beats unsupervised baselines on Q&A and translation AUC, and can be adapted for black-box models by using a proxy white-box model’s activations. [13:04](https://addtry.com/ec96c7c0-4c99-42bf-b403-b0a8f32a4470?t=784000) ### What is the main contribution of Anthropic's Towards Monosemanticity paper? Casper explains that Anthropic trained sparse autoencoders (SAEs) to reconstruct MLP layer outputs with a sparsity penalty and expansion factor, successfully identifying interpretable features in a toy model, such as DNA sequence detectors and language-specific tokens. The work spurred open-source SAE training on models like GPT-2-small, and is a leading method in mechanistic interpretability. [32:44](https://addtry.com/ec96c7c0-4c99-42bf-b403-b0a8f32a4470?t=1964000) ### How does Medusa speed up LLM inference without a draft model? Medusa adds multiple lightweight MLP heads to the base model’s hidden state, each predicting a future token. It generates candidate token trees and verifies them in parallel against the base model’s output, avoiding the need for a separate draft model. Training can be done by freezing the base LM and training only the heads on a task-specific dataset, achieving speedups with minimal quality loss. [48:37](https://addtry.com/ec96c7c0-4c99-42bf-b403-b0a8f32a4470?t=2917000) ## Key moments - **[0:00] Dot by Dot** - [5:30] Filler tokens carry task-related structural information, boosting accuracy as task complexity grows. - [6:20] Increasing the percentage of filler tokens kept in the final layer raises accuracy to 100%, then tapers off past 60%. - [7:02] Chain-of-thought reasoning achieves the highest accuracy, while filler tokens approach that and no tokens reach 78%. - [7:32] Filler tokens require prior knowledge of chain-of-thought token placement to be effective, implying configuration dependence. - **[12:41] Uncertainty Estimation** - [13:31] A random forest regressor trained on LLM hidden layer activations estimates response uncertainty for downstream tasks. - [15:31] Supervised uncertainty estimation improves task performance and offers a potential hallucination detection mechanism. - [23:11] Black-box uncertainty estimation maps a white-box model's activations to a proprietary model's outputs, shortcutting access. - [25:30] Using the middle layer activations yields better uncertainty estimation than the last layer, as middle layers summarize better. - [26:50] Q: How does measuring uncertainty relate to model calibration? A: A well-calibrated model's confidence scores align with true probabilities. - **[32:14] Monosemanticity** - [36:34] Casper presents sparse autoencoders (SAEs) as an effective dictionary learning method for finding monosemantic features in language models. - [38:38] An SAE feature fires precisely when the model encounters a lowercase DNA sequence, strongly upweighting subsequent DNA-like tokens. - [40:18] Training SAEs is now accessible via off-the-shelf libraries, with GPT-2 small being a popular target for researchers. - [42:05] SAEs are now considered the most promising direction in mechanistic interpretability by both Anthropic and DeepMind. - [43:17] Q: What advantages do SAEs have over probing classifiers? A: They are unsupervised, learn many features at once, and leverage sparsity with reconstruction loss. - [44:43] A host introduces Medusa as a speculative decoding method that replaces the draft model with extra prediction heads on the large model. - **[44:58] Medusa** - [48:02] Medusa adds multiple MLP heads that each predict a future token from the same hidden state, generating multiple candidate continuations. - [51:14] Training Medusa heads while freezing the base LLM requires only 5 hours on 60k samples for a 7B model. - [53:55] Q: Does using five Medusa heads increase cost like beam search? A: Medusa changes token speculation, not beam search; cost depends on implementation. ## Speakers - **Casper** (guest) ## Topics Inference, Interpretability, Reasoning ## Mentioned Anthropic (company), OpenAI (company), GPT two small (product), Gemma (product), Kaggle (product), Llama (product), Medusa (product), scikit-learn (product) ## Transcript ### Dot by Dot **Guest** [0:01] Okay, great. Great. Okay, I will now start on presenting this paper. Uh, basically, I found out about this paper recently, which is a paper called Let's Think Dot by Dot. And the motivation behind choosing this paper is that basically, I always have the burning question of to-- as to like, how do LLMs actually think? Now, we know that chain-of-thought reasoning process or chain-of-thought prompting actually shows that by allowing LLMs to think out loud before answering, their performance actually improves considerably when compared to direct answering techniques. This actually provides us some intuition as to how LLMs reason through their task. However, recent work, as found in this paper, suggests that for chain-of-thought reasoning, LLM answers could also be unfaithful to the intermediate reasoning steps. Simply put, the answers do not really tally with their workings. Now, here are some guiding questions be-before moving on to the related work section. The first question is that, do LLMs actually need to think out loud, basically like write down their thoughts, or are they able to think internally like humans? The second question is that, are there any specific tasks to show that LLMs are not simply relying on the semantic information and possibly relying on other kinds of information found in the tokens in their inputs? So, I'll just skim through the related work section just to provide some additional context on this work. The first section in the related work is talking about computational complexity. Basically, they are discussing that there are different levels of computational complexity and task complexity. Basically, the lowest level is something called TC0, and the higher levels of task complexity include graph connectivity, solving Sudoku, et cetera. And this requires some form of recursion and checking multiple constraints at the same time. Now, the second part of the related work section is talking about how transformers use tokens for reasoning. In the chain-of-thought prompting example, we can see that transformers can indeed solve higher complexity problems when additional reasoning tokens are provided. These tokens actually help to decompose the problem into smaller problems and improve the model reasoning capabilities. So, the question is that, are the LLMs making full use of these extra tokens as part of their reasoning process? And the thir-third part in the related work section is talking about potential model performance improvements on more complex tasks. Some work also suggested that using filler tokens could improve performance on nested quantifier tasks. Basically, tasks that have some form of constraints, nested constraints that depend on each other. Now, what the authors want is that they actually wanted to create new dataset that specifically target this kind of problems to test their filler tokens hypothesis. Now, let me move on to the methodology section and talk about the tasks created by the authors. I have a short, short section here talking about a nested quantifier, and then I will send the link over later. But this will just be a short summary about what exactly are nested quantifiers. So, due to time constraints, I'll just move on to the first task created by the authors. So, the first task is called 3-Sum. Basically, is that you have a following list of numbers, and you want to check if there's a group of three numbers inside that list that adds up to zero modular ten. So, the example question is that you have a list of two, three, five, seven, one, eight, and the answer could be like two, three, five or two, seven, one because the result will be ten. And ten divide by ten, uh, the remainder is zero. And for the second task, uh, it's called 2-Sum Transform, which is also regarding a sequence of numbers. And basically, they have one list, and then they have a hidden transform list, basically, uh, using the original list, and then they are adding some form of permutation to each element in that list, and that permutation is only revealed at the end. So, for example, you have an original list, and then you are plus three to each number in the transform list. And then what this task actually requires is that it requires model to remember transformation information and how to integrate this in its computational thinking process. So, let me move on to the experiment section of the paper. So the-- basically, the authors are using some form of scaled-down randomly initialized Llama model in, in their training process using three different model configuration for the setup. And the first configuration is basically using filler tokens, basically a series of dots as intermediate tokens between the input tokens and the output tokens. Now, the second configuration is that instead of using dots, they are using the chain-of-thought reasoning tokens as the intermediate tokens. Now, the third configuration is the sort of like the control setup, where there are no filler tokens in between the input and output. And then the results show that, uh, for the 3-Sum task, the, the first task created by the authors, as the length of the sequence of numbers increases, the difference in performance between the configuration with the filler tokens and the one with no filler tokens actually widens the gap quite drastically. And then this basically shows that as the task complexity increases, the filler tokens actually do contain some form of task-related information. And the next figure is talking about a different experiment, which is basically the authors trying to freeze all the previous models of the transformer and only preserving the final attention layer. And then basically, they are trying to prove that whatever changes they make in the final layer, this will cause the results to change. Basically, that the performance improvements or performance changes are only resultant from the changes in the final layer. So, in the final layer, what they try is that they try to increase the percentage of filler tokens in the final layer, and they've seen that in their task, actually it increases the accuracy up to like, uh, 100%. And then, uh, it increases sharply from 60% of the filler tokens kept until it tapers off, and then it has some diminishing returns past the 60% mark. Now, for the two-sum transform task, which is the second task, the author lists down a table with comparing the chain-of-thought, the filler tokens, and the no tokens configuration. And then this shows that, yeah, as expected, the chain-of-thought is still triumphing and then at the very top, and the filler token is sort of getting quite close to the chain-of-thought reasoning tokens, while the no, no tokens is showing like a 78% accuracy. Now, both of the aforementioned configurations are noticeably exceeding the performance of no intermediate tokens. And some additional insight that the author actually found out in their experiment is that they-- before using the filler tokens to replace the intermediate tokens, they actually needed the chain-of-thought configuration to be set up to see how the chain-of-thought tokens are placed as the intermediate tokens before directly replacing one-to-one with the filler tokens. So this basically means that this actually quite challenging. This is a quite challenging learning problem whereby prior knowledge of how to put the intermediate tokens are needed before you can use the filler tokens. This also implies that the filler tokens must be placed in some particular arrangement or configuration in order to showcase good performance. So at the end of the paper, some food of-- some food for thought is like basically saying that the experiment results imply that although filler tokens do not contain any semantic information or instructions for the language models to follow, they still contain some form of structural information or syntactic information. So a possible question is that why use dots as the filler tokens? We can also use other kinds of tokens, right? As long as the tokens do not really contain any kind of semantic meaning, it could all be valid choices for the model. So I would love to see, like maybe we can try like different type of tokens to see if there are any different kind of results. Although dot tokens are technically filler tokens, but they also do serve as some form of ellipsis in the English language, acting, acting as some form of continuation. So in summary, this pap- this paper basically shows that how large language models can use filler tokens to sort of achieve higher performance in certain tasks when compared to like, uh, not using any filler tokens at all as the intermediate tokens. The results also show that filler tokens have some form of computational benefit, although not semantically, but maybe some form of structural information. Yeah. So that's the end of my, uh, survey. I think it's an interesting paper, and then I'm open to any questions. Yep. **Guest 2** [9:19] I think Casper had a question inside the chat itself, whether the, the filler tokens were special tokens introduced after the model was trained, or was it just the same token used for period filler that tokenizer had learned perhaps during the pre-training phase or- **Guest** [9:34] Uh, they actually- **Guest 2** [9:34] ...that's it **Guest** [9:35] ... include the filler tokens as like sort of the-- in the pre-training phase also. Yeah. So they were not like a special token added in afterwards. **Guest 2** [9:49] There's another question from Warren, which is, uh, why not just use unknown token or something instead of dots in this case? **Guest** [9:55] Yeah, that's an interesting question also. Yeah. **Guest 2** [9:58] Mm-hmm. **Guest** [9:58] I think it will probably... Yeah, you'll probably have different results. Yeah. Seems some form of any rare form of tokens, yeah, should be fine. Yeah. **Guest 2** [10:07] It, it seems to-- I don't know if you've seen like the memes on Twitter where they like, uh, if you, if you prompt the model and you put a space in between or like one or two spaces- **Guest** [10:15] Mm **Guest 2** [10:16] ... and then you just let it like, uh, and then you, you let it do the normal inference call, the performance actually increases sometimes because the, the space itself is another token, right? **Guest** [10:24] Oh, okay. **Guest 2** [10:25] And then that modify-- that, that sort of like modifies the computation itself. Yeah. **Guest** [10:29] Oh, yeah. That's quite interesting. Yeah. It's all these, uh, hidden complexities that, that we don't really know. Yeah. This is quite interesting. **Guest 2** [10:43] I just comment, could it be also be because of positional biases? Uh, do you wanna maybe elaborate on that? I'm not too sure about what positional biases are. I mean, you can drop it in the chat or you can unmute yourself. **Guest 3** [10:53] Yeah, sure. So what I meant was because, like, there's a position encoding the transformers, right? So because the models were trained on a lot of sequences, then you-- it's like, I think maybe if it trains too much, it gets saturated, then it kinda like learns the positional encoding as well. So, like, if you put a space or two, right, then just nice, the most of the training sequences, that particular word also is aligned with, like, the rest of the positions with, like, how it was observed in the training data, then it might s-improve in performance or something like that. I, I just know that this kind of bias exists, but I don't really know how to actually explain it. Let me see where I can find an article on it. **Guest 2** [11:29] For sure. For sure. Mm, I don't think the positional biases of the, the prompt and the original input will change, but I think perhaps what you're thinking about is the additional space itself plus the positional bias might re- might better reflect the training data or the, or where the answer might be. Is that what you're trying to say? Something like this. Okay. I'm looking. Let's see. Pull this link up. Large language models are not very valid. Let's see. Okay. I'm looking too. Yeah. Cool. Uh, I guess if there's no more questions, then maybe we can move on to the next paper. **Guest** [12:14] Okay. **Guest 2** [12:14] Do you wanna go next, Gabriel? **Guest** [12:15] Yeah. I-- Okay, I'll stop sharing my screen. **Guest 2** [12:22] Oh, uh, you want me to go next? Okay. **Guest** [12:24] Yeah. **Guest 2** [12:25] Okay. Uh, give me a moment **Guest 4** [12:32] Can you see the screen? **Guest 5** [12:33] Yep. **Guest 4** [12:35] Just give me a while. ### Uncertainty Estimation **Guest 4** [12:41] Okay. Uh, yeah. Hi, everyone. Uh, yeah. Uh, this paper's a bit different. So it's about how we use a second model actually to, uh, estimate the uncertainty of LLM's response. So it's quite an obscure paper actually. I, I spent a few days trying to think of a paper to sort of talk about and but I did enjoy reading this paper for when, uh, it approaches the problem quite differently. It's from a bunch of people in, uh, operations research, which is actually where I did grad school. And, uh, the sort of approach of doing it is actually more of just getting data, fitting a model in, and then trying to make use of that model for a real-world outcome. So let's just dive right in. So the TLDR is actually just three points. Uh, number one, here they train a regression model, and in fact, actually it's a classical one, regr- random forest, to underest- to estimate the uncertainty of an LLM's response. And the input for that is just the LLM's hidden layers of the last token, activations of the last token, or in the case of a model that, let's say, uh, or OpenAI's APIs or some other LLM provider where you don't actually get the actual hidden layers, you can actually... or, or a gray box model, you can actually use, uh, some of the probability related output as an input to this, uh, random, this regression model. And the output, it's actually not the logits of the, the language model, but instead it's actually a task-specific score, and typically between zero and one about the certainty of the answer. So I'll just elaborate a bit more about that, uh, later. Uh, the paper covers a, a bit about the existing methods, and I think most of the methods right now for quantifying uncertainty, they tend to be based directly on the output of the language model. Uh, what do I mean by that? Say g- given a, a fixed prompt, you might get different outputs and you try to, like, do something there to see what the variation in the output is. Or you might add some perturbations to your prompt and see how that, that results in variation your out- your output, output, and you measure it thereafter. The paper keeps specify-- keeps mentioning that these are unsupervised methods and in contrast to what they're doing with a supervised method where they actually have ground truth to some degree. And I'll share again a bit more about that when I explain the problem mathematically. And this act-- work has some been somewhat applied where they take a second model to quantify the base, like, language model, and it's been done for transformers, but not for recent large language models like Llama 3 or Gemma. So before we go into like the paper itself, like actually why does this matter? I think, uh, the paper sort of points out, uh, point two and point three in my list here of four points. Uh, they show that by doing this, you actually can get improved performance on certain tasks, and it's a potential use case in detecting hallucinations because by getting a sort of confidence score for an answer, you can then, uh, perhaps hedge, do a different sampling approach. Uh, point one and point four is, um, actually my own i- uh, ideas upon reading the paper where actually if, say, you have a language model interface, uh, uh, app and actually w- w- with providing an, a, a sort of certainty score about the r- the response, imagine your typical chatbot, people talk about RAG chatbots. If you're able to provide like, uh, the certainty score, the UI/UX of it could incorporate that into explaining the, uh, certainty of your response. And number four is interesting because, uh, as you'll see later on in the, in this quick sharing, what we're doing is basically predicting the performance of a LLM's answer, and that actually opens up possible use cases of auto evals where you're already building a sort of, uh, hang... You've already done your evals for like maybe a good training set and then now you can scale this up to auto evals. And actually when you deploy a LLM system live, this auto eval could actually help sort of, uh, highlight cases where your system might potentially be giving, uh, low confidence answers. So let's just try to formalize this a little bit. So that's where we just go this part where we express the problem mathematically. So the first thing is an LLM is just abstracted into, I give it an input and it generates a response. And the prompt here denoted by X is a series of tokens and it's X one to XK, and they all belong to this set chi, which is the vocabulary size of the language model. Thereafter, it would generate a, a response Y, which is, uh, over in the second line here. Uh, Y, it's a vector Y, where it consists of multiple tokens, like say M tokens, and again they belong to the set, uh, Y which also could be the vocabulary size, but it's actually also the probability distribution, which is the third line here where each token, YJ, is a probability of the conditional probability of the input, uh, prompts, which is vector X and all the previous earlier outputs Y all-- Y one all the way to YJ minus one. So that's how we set up the problem where it's just X and Ys. Typically then if you use your language model, it's not you-- when-- we're not just using it to like do, uh, completions. We actually want to then use it for downstream tasks like Q&A, MCQ, translations. And here we actually have a scoring function. So this could be blue, uh, Ru. And so here it's actually this function S, where it sort of takes in the true Y, like let's say the true answer, and then model's generated answer Y, and then it sc- maps it to a zero to one. So it's just a sc-- it's just any generic scoring function that you can think of. So then the task of uncertainty estimation is effectively learning a function G, so in this third, third step over here that actually then ma- uh, sort of predicts the score given the input prompt and the output response. So given the mo- the prompt that I feed into the language model and the response that the language model gives, can I then predict how good that answer is? So then the paper then explains how they can apply this approach to all sorts of language models, white box language models where you have the weights, gray box language models where you don't have the weights but maybe you have some out- you have more details about output, like say, the, uh, the probabilities of the various tokens, the log props, or completely black box models like, uh, an, like maybe a, a API provider that doesn't give any sort of, uh, indication of what goes on behind the scenes and it just only give a final chat completion. So let's just share a bit more about the white box language model because these sort of methods thereafter extend from here. So the sort of, uh, for the white box language models, we sort of want to first build a data set that is called, uh, D_raw, where it consists of a four things. Number one, the input prompt, which is Xi, the output an- the, the answer that was generated by the language model Yi. We want Y, the Y tr- the true answer, which is Yi true, and lastly, the fourth item, which is the score, the evaluation of that answer vis-a-vis the, mm, answer that the model prompt, uh, generated. And notice how, like, even if I give the same Xi, I might get different Yi's because of the probabilistic nature of a language model, so that actually gives me more training data as well. So then from each sort of, uh, data, each, uh, row entry in this raw data set, I want to extract out features to construct what they call the uncertainty data set. So the uncertainty data set, uh, over here under s- un- denoted by D_un, is a tuple of your Vi and your scoring. And Vi here is a vector of selected features. So we on- I mean, there are billions of parameters in a language model, and what they suggest is, um, to use the hidden layers of the activation and then for this experiment, they actually use the activation from the middle layer and the last layer. They even, uh, suggest, uh, other ways of getting features such as, uh, directly, uh, asking the model to, like in the pro- input prompt, adding, ap- appending the term, appending the phrase, "How certain are you about the response?" And then trying to get the activations with respect to that, uh, prompt. They suggest that as a method that other people have done it, but I don't think they implemented it in their, in their approach when I read through the paper and the code. So once you have the uncertainty data set, which to recap is, say, just sup- a supervised learning example, where you have just a series of feature vectors and then a score. You then train any other good old-fashioned supervised learning model to predict that score. And then once you have that trained model, which is now in point four of what we see here, I can use that inference time where I get a res- where I have an input text, I feed it through the LLM, I extract the features, and I use my trained machine learning model to then predict the s- uncertainty score. So they do spend a bit of time in the paper, actually only in the appendix. So this is the kind of paper where you read the appendix to actually get the algor- how they actually implemented it. Uh, so they use three hundred and twenty features. So these three hundred and twenty features consists of twenty features from the, that they use from the gray box LLM, which I'll explain later. And another three hundred features of which they get a hundred by running a lasso regression, one hundred by calculating the mutual information, and one hundred by calculating the, uh, correlation coefficient, and then they train a random forest regressor with these three hundred and twenty features. Uh, you don't actually, I don't think, and I, they, they prob- didn't probably, they probably didn't like fine-tune, I mean, uh, what do you call it? Uh, op- optimize the hyperparameters for this, uh, regression model. So, and it wasn't the point of that paper. It was just to demonstrate that it's possible. So it's probably an area of, uh, uh, work if people are interested to tinker. So that's what they do. To recap, prompt, feed language model, get the activations, those activations, select a few, train another regression model. The second thing that they explain is how you can then i- incorporate this for gray box language models. So here they actually then just ki- come up with twenty features from, related to the output probabilities and, uh, from both the response but also the question or the prompt, and it's sort of like covered there in the paper. It's not too interesting, but, uh, what I thought was somewhat interesting was how they could get it from twenty features. Then the last category is for the black box language models. So it's stuff like, let's say, OpenAI or Anthropic. How do you actually can, uh, get some uncertainty estimates from there? So that's where I thought it was a bit clever, where they take a input prompt and they feed it to the proprietary, let's say the black box model, and they get an output response. At the same time, they take that original input prompt and they feed it through a white box model, say Llama3, uh, Llama3, and get the activations of Llama3, and they're trying to map the, uh, activations in Llama3 to the outputs of a, a proprietary model, and somehow that seems to work. In the paper, they treat, they do the experiments with Llama Seven-B and Gemma... I think Llama Two-Seven-B and Gemma, uh, Seven-B as black boxers and vice... using the other open source model as the white box for the uncer- uncertainty estimation. So just going into the results, they cover three tasks, Q&A, MCQ, and translation, and they sort of give the results here. And they compare it against other sort of, uh, methods for sort of getting, uh, uncertainty estimation, because what you want is not just a s- a sort of answer, you want a, something like a probabil- a probability between zero and one. And here what the, the figures you see are actually the AUC scores. So then they s- they sort of demonstrate that their method is, uh, better in getting a higher AUC for these various tasks for Q&A and translation. So the AUC of the score vis-a-vis the answer being correct, uh, binary yes or no correct. So just to give an example of what this looks like, so let's say over here in this screenshot, the question is, uh, what musical features the songs, "A secretary is not a toy" and "The company we"? I have no idea what this, that is, but apparently the answer is, "How to succeed in business without really trying." And if you look at the table over here, if you were to take some sort of greedy approach, the, and take the, let's say the max probability, the, um, I believe- Yeah, you would sort of get a, uh, probably, maybe it's the wrong result. But yeah, you would get a probability of, uh, you get a confidence score of 0.9. But if you look at, say like, the white box method, you would see that the correct answer over in this column, the wh- WB-dash S, the greedy answer gets a, sort of, confidence score of 0.14. But answer one, which is the correct answer, gets a probabilities, gets a confidence score of 0.22. So in this case, they, they found, they found that this confidence score approach could actually yield higher accurate, more accurate answers. So this is just an example. Uh, just wrapping up quite soon. So remember how I said that they, what they're doing is training a regression model on the, uh, hidden activations. So, and I said that they were using the middle layer and the last layer. They sort of found that actually you get better performance when using the middle layer, and they apparently cite this thing from the literature suggesting that actually the middle layer of these language models are better at summarization. So just to quote, "This may come from the fact that the last layer focus more, focuses more on generation of next token instead of summarizing information of the whole sentence as discussed by sort of other, uh, authors." So that was, uh, new information for me. I thought it was quite interesting. So, uh, last thing is that actually there's right now a Kaggle competition, uh, where you're trying to predict how people, uh, uh, rank the, uh, chatbot responses on the arena. So I thought that was some interesting, uh, sort of, uh, uh, synergies here in that paper, and that was interest- pretty cool. And if sort of we can sort of, uh, automate some of the eval work and more from my perspective where I, I build a language mod- large language model powered application, and I deploy it. I also want to know is my li- large language model actually doing well in real time, and maybe something like this could come in useful. So yeah. Uh, thanks for the... Thanks for listening. **Guest 2** [26:34] Awesome, dude. That was a great presentation, honestly. A super interesting paper. I, I think there were two questions inside the, the chat itself. Uh, first one was from Hong Nan, which was, uh, how is measuring of the uncertainty related to the concept of a well-calibrated model? How does? Yeah. **Guest 4** [26:51] Hmm. Yeah. Uh, so the paper does cover the concept of calibration. So, uh, maybe to just share, uh, the, to get everyone up to speed about what calibration is. So a m- a model is considered well-calibrated. Uh, let me give an example of a w- well-calibrated model. So if let's say the model says it's gonna, uh, rain 40% of the time, the probability of it raining is 0.4, then 40% of the time it would rain. So it's... Calibration is, uh, sort of, uh, means that the, sort of, numerical score that a num- a machine model produces gives a probability. Uh, so in this case, they do sort of, um, mention calibration in the paper, and they effectively are saying that, um, number one, a model that has a good confidence score is more likely to be a good, well-calibrated score. And number two, that, uh, usual methods of calibration, like, I think isotonic regression or I believe some other binning method can be applied, and they do have some, uh, numerical results on calibration as well in the paper. **Guest 2** [27:54] So I guess, would it be right to say that then a well-calibrated model is a model that performs well on the task that you've assigned it, or is it slightly more complex than that? **Guest 4** [28:03] Uh- **Guest 2** [28:03] That's it. **Guest 4** [28:04] Ye- yes and no. Uh, yes, it, it effectively means that. Uh, well, you can just think of it as a model that's calibrated is a model that, uh, gives a probability just 'cause a model. So just because a model spits you a number between zero and one doesn't mean it's a probability. Yeah, so if, for folks that are not too, uh, familiar with this, you can go to the scikit-learn, uh, documentation, and it do give an example of calibration, yeah, of how you can calibrate sort of scores to become more like probabilities. **Guest 2** [28:34] I see. **Guest 4** [28:34] Yeah. **Guest 2** [28:34] Uh, I think there are a few more questions, um, i- if, if it's okay with you. **Guest 4** [28:39] Uh, no, I, I'm sorry. **Guest 2** [28:40] Yeah, yeah. **Guest 4** [28:40] I was reading it. Yeah, sorry. How does false- **Guest 2** [28:42] Yeah, yeah **Guest 4** [28:42] ... and so white box model to a black box creator? I am a bit doubtful as well then. Uh, yeah, so Warren, yeah. So it, it doesn't mean, yeah. I, I, initially, it took me a while to understand that because, yeah, how can you use one model for another model given that the architectures are different? So I think what they're simp-- all they're doing is, um, okay, for, let's say, OpenAI, I run my prompt through OpenAI, and I get a result, and I keep that result. At the same time, I take my original prompt, and I run it through Llama 370B or whatever, whatever, and then I get the activations of Llama 370B, and then I just then use those activations as the input for my, uh, secondary model to predict, uh, the uncertainty. So they're trying to, like, sort of shortcut their way into, like, predicting, to getting a sort of repres- they're using, effectively using, if I understand, if I sort of, like, see it intuitively, is that they're using the open source or the white box model as a way to, uh, create some sort of representation of the prompt, and then they're, in any case, they are using a secondary regr- a secondary downstream model to learn that mapping of how Llama 370B would map the prompt into the outputs of, uh, uh, OpenAI's models. Yeah. So for Casper's question about how this would compare to methods just using log props, that's also another thing that, uh, took me a while to eventually realize is that if, let's say, I do log props, it's sort of predicting the next, the probability of the next token. But the probability of the next token isn't necessarily the probability that my answer is correct because, uh, it's, it's task specific. So let's say I want to do a question and answer, and then I, I get a probability of like, "This is a MacBook." You can get some probability for that statement, but I don't get a probability for whether the statement, "This is a MacBook," is the correct statement for my given task. Yeah. And next one. For the mathematical models that measure uncertainties, is it dependent on the loss function? For example, when comparing a simple model trained on L2 loss versus one that goes through additional complexity like RLHF mo- model-based scoring, does the mathematical theory still hold? Okay, I'm not too sure, but the, there was a part of, uh, this paper that did have some theory about how, uh, given, uh- Certain conditions, the, um, I believe, like, this problem, the, the optimal solution is your, uh, optimal base classifier, but there are certain, uh, conditional independence properties that aren't satisfied and also because, uh, if I recall... I actually have the paper open here. Yeah, so if you-- because you don't-- the... When we train a language model, the loss function is actually a different loss function. Uh, over here. Sorry. Yeah, over here. So they basically say that... Yeah. So because, um, if the large language models aren't trained on the cross-entropy loss, the sort of, uh, theorem doesn't hold, but they do see like, oh, because, uh, large language models are trained on the larger data and so on and so forth, they do... There's apparently some, uh, uh, what do you call that? Uh, approximations that can be done. Yeah. So I didn't fully understand, uh, section three point three, but maybe, uh, you could take a, take a read and see if it, it helps that. So they are say- it's trying to justify, uh, why they use hidden layers as features. Yeah. All right. Um- **Guest 2** [32:03] All right. **Guest 4** [32:03] That's about it. **Guest 2** [32:04] Yeah. **Guest 4** [32:04] Thanks everyone. **Guest 2** [32:05] Thanks. Thanks for presenting, dude. Yeah, I think Nix is Casper with mono-semanticity, I think. ### Monosemanticity **Casper** [32:14] All right. Uh, do you mind, uh... Oh, there. Okay. So I promise it's not just a just... Um, but let me just get through some basic stuff first before we go through some interesting dashboards and visualizations. Um, so I'm covering the paper Towards Monosemanticity, uh, which Anthropic released, I believe it was late last year. Um, and basically the, the primary contribution of this paper was demonstrating that it's possible to use, uh, sparse autoencoders to identify features in a language model. Um, so that's, that's really fuzzy, but m-maybe, maybe just to, just to level set and make sure everyone has an idea of what I'm talking about here. Um, you know, this is, you know, probably one of the most important works in the field of mechanistic interpretability or mech interp, um, over the past sort of few years. Um, and, and so what is mechanistic interpretability? You know, it's basically making the inner workings of a neural network human interpretable, right? And, you know, this is in contrast to other interpretability or, you know, AI explainability approaches, which take more of a behavioral approach, right? If you think of like behaviorals, behaviorist in psychology, that's often the approach that's taken, um, by some researchers. Uh, mech interp, on the other hand, is very much concerned with, you know, the, the activations within a model and figuring out at a, at a very sort of granular level, uh, how it is a model is, is working. Um, so, so really what we're trying to do here, or what mech interp is trying to do, is trying to find features, right? Um, and, you know, it's a bit of a fuzzy definition, but think of a feature as sort of a property of a token or a group of tokens. Um, you know, for example, there might be a feature that fires, um, on pronouns where, uh... it's, and this might be an attention feature, and, uh, it, uh, it, you know, that, that, that feature firing, that feature activation might signify that, you know, the pronoun is attending to another proper noun, um, somewhere in the sequence. Or, you know, it could be a, a sort of more fuzzy sort of feature where, um, if this feature is firing, then the, the sort of sequence sounds angry, right? There's all sorts of features. Basically, you know, if you think about what a model is capable of, what it can represent, uh, you should have a feature underlying all of those capabilities, right? Um, now, now the problem here is that it's actually really hard to identify features, um, uh, because models are so big, and in addition to that, you don't have nice clean features where one feature corresponds to a single neuron. Um, and the, the sort of hypothesis is that it's because a model represents far more features than there are neurons available in the model. Right? If you think about, you know, every single feature that you'd want to represent, to represent the world, um, then it's, it's, it's sort of like, um... it's sort of obvious that a model can't be large enough such that you have one neuron that corresponds to each feature, right? Um, but empirically, you know, it, it's been shown that it is possible to identify features, and this was initially done through really hard manual work, um, and just, you know, people eyeballing feature activations. Um, you know, and, and, and broadly, this approach of figuring out what features a model represents is, uh, sometimes also called dictionary learning. So, so with, with that sort of background, you know, maybe I'll jump into the paper. So Towards Monosemanticity was one of the first papers to really introduce and demonstrate that sparse autoencoders, um, uh, appear to be a pretty effective dictionary learning method. Um, and what was done in this paper is that, uh, the authors at Anthropic- ... trained, uh, sparse autoencoders to reconstruct the outputs of an MLP layer. Um, but in this reconstruction, there's two things added. One is a sparsity penalty to remove noise and find, you know, more interpretable, um, feature activations. And two, uh, an expansion factor, um, which, uh, you know, makes it, makes it so that you can, uh... It's easier to sort of represent more features or identify more features using the SAE, right? And the way they did that in the paper was, um, using a toy model with a single layer, so a single layer MLP. Um, and they took the sort of hidden representations from that MLP and trained SAEs with a range of expansion factors from one X to two hundred and fifty six X. Um, so, you know, I think, uh, now that I've gone through that, let me actually just skip to... Let me actually share my other screen where, um, I've actually pulled up the dashboards of the SAEs, right? And then you can actually get a sense of what's actually being identified by these SAEs, right? So, you know, one example feature which they talk about in the paper is a feature that happens to fire when, um, the model, uh, thinks that something is a DNA sequence in lowercase, right? And that looks pretty accurate. Um, you know, what's interesting about these features is that if you look at, uh, what they do in terms of, uh, affecting downstream output, is that if this feature fires on any single token, um, it has a very strong impact, uh, in terms of upweighting, uh, subsequent tokens, um, which are also, you know, DNA-like, right? Um, you know, you have other sort of... There's all sorts of features. Some of them aren't so, um, aren't so interpretable, some of them are. You know, you'll have some that fire on some languages. You'll have some that fire on Caesar shifted encoded words. Um, and I'd encourage you to actually look at the visualization. Um, it's linked in the, in the towards Monosemanticity paper, um, and super interesting. Um, maybe just to give you more, uh, you know, another sort of, um, give you a sense of what other work is going on. So that, you know, that, that work was done on a toy model with a single layer. It's not a real, uh, language model. Um, but on the open source side, there's people training SAEs now using the same techniques, um, on all sorts of models. Uh, GPT two small is a favorite just because it's very well understood by interpretability researchers, and you can see all sorts of features identified in GPT two small. Um, and it's actually super easy to, to train an SAE now. You can just use an off-the-shelf library, uh, and play around with it. Um, but, but maybe just going back to like, you know, why, uh... Going back to, you know, why this matters and, and what the limitations are and, and where we're heading in the future. Let me just switch back to my other screen, and here we go. You know, so, so maybe let me just talk about limitations first, right? These are pretty new... It's a pretty new technique. Less than a year old now, really. Um, at least in the context of MechInterp. Um, you know, it's, it's pretty clear that the feature set identified by SAEs isn't complete. There's rare features. There's features that might get ignored for some reason or another due to, um, training methods not being that sort of, uh, refined. You know, you also have this limitation where as you increase the expansion factor, uh, in the SAE, you get this phenomenon of feature splitting, where, you know, what was previously a, a single feature now becomes a family of like twenty features, for example, um, all of which are like sub-features, and this, it gets a bit complicated. So it's... You don't get like these very nice clean representations necessarily. Everything's sort of fuzzy. Um, but there is work going towards, like, improving them, right? And there's, there's quite a lot of work improving, on improving SAEs. Um, this is now seen as the most, uh, promising line of MechInterp work within both Anthropic and Deep Mind. Um, and there's also folks at OpenAI working on SAEs, and OpenAI actually open sourced their SAEs for GPT too small, which is pretty interesting. Um, but yeah. You know, I don't think this is... This is, like, very much like an exploratory paper, right? It's sort of like, uh, I think the, the, the sort of point is that, "Hey, there's this new technique. It looks pretty interesting," um, but the jury is still out on whether this ends up being, you know, um, the technique that solves MechInterp. Um, but yeah, that's, that's it. **Guest 2** [42:43] Awesome. Thanks for the presentation. I think Warren had two questions. Uh, what is the difference between SAE versus a probing classifier? I think the other one he was saying is, what makes SAE such a good fit for, for this specific task? Why not use other models to extract Monosemanticity? **Casper** [43:00] Yeah, sure. So, uh, it is similar in some regards to linear probes. Um, but I think the nice thing about SAEs is that, and this sort of kind of covers the second question to some extent, is that, um, you get a few nice features. One is that it's, it's unsupervised, so you can learn a lot of features at once, which is nice. It doesn't need any sort of humans eyeballing things. Um, uh, and two, uh, so, so why SAEs in particular? Um, you know, I think... you know, the, the features that are, are helpful or rather not features, the, the elements on the SAE which are, are helpful in this regard. One is like the sparsity, which is important, and also the ability to just reconstruct the inputs, right? Because you want that, and that helps you actually, um... If you're reconstructing hidden activations, you get a very nice, natural way to assess the goodness of an SAE because you can measure the reconstruction error. So, you know, uh, how you typically evaluate these models. One, one metric that you measure them on is actually reconstruction error, right? So you ablate the, the actual sort of activations in the model, and you replace them with actually the reconstructed, um, activations from the SAE. Um. All righty. Any other questions? **Guest 2** [44:43] Seems like I think that's about it. Uh, I guess if that's the case, then let me just share briefly on Medusa. Uh, okay. Let me just see if I can find the screen. Awesome. Thanks for the reminder though, Gabriel. Appreciate it. All right. Uh, sorry about that. So today, I'll just be presenting quickly on Medusa. So the concept behind Medusa is just basically, um, it's a better way to do speculative decoding. And so I think before we move into what speculative decoding is, uh, I think it's important to see what are the main problems with model inference. ### Medusa **Guest 2** [45:17] Uh, just so we're all on the same page, I'm just gonna paste the link that I'm looking at in the chat itself. So if you're familiar with any large language model, a large chunk of the time that we spend, it's basically running, transferring the weights from the, well, I guess the high bandwidth environment, like what they say over here, over to the cache, which is where there's a very limited amount of space and where most of the calculations happen. So what that means is that if we run inference one time, we load in all parameters, we load in all our inputs, uh, and we get out the, the final output of one. We only get one token, and we have to repeat the whole step. Um, there are a lot of optimizations that have been done around it, but basically the main problem is still not fixed that you, you do all this work and you only get one token out. So what people have done is this thing called speculative decoding. So what does that mean? So let's say we have a huge model like a Llama seventy B, and we have a smaller model which we call a draft model called a Llama seven B. Um, a Llama seven B can take an initial prompt and quickly generate a whole bunch of, uh, tokens itself. So you can think, let's say you feed in some prompt, it generates like N tokens that are supposed to be-- that it thinks are gonna co- follow the specific prompt itself. So now we have a bunch of candidates, right, for a proposed sequence inside this draft model. So how do we know what percentage of these tokens are, are correct? And I guess along the same lines, how many of these tokens we should reject? The way to do this is to basically just batch all these tokens and feed it into the model itself. So what this could look like is, let's say our prompt is, "The capital of France is," and our smaller model says, "Hey, the capital of France is Paris, and it's a beautiful city." For inference, we would pass in, "The capital of France is," "The capital of France is Paris." And each step what we're basically trying to see is, does the completion from the smaller model match the big model? And so what we're doing here is we're able to batch all of these inside a single, like, forward pass, and we don't incur a huge back and forth transfer. And so we see this huge speed up, uh, in terms of the decoding speed itself, uh, when we're generating tokens. So there's some problems with this, though. The first one is, of course, optimization, because now we need a smaller model. We need to feed it through this whole chunk of data, and we need to somehow do the reconciliation between the original completions and the new completions. The second, which I think is the bigger problem, is that the draft model might not accurately reflect the capabilities or the world knowledge of the, well, the larger model itself. Uh, if you ever play, let's say Gemma two B, it might not really be the same as a Llama seventy B, even if it's able to decode like six times as fast. So in comes Medusa. So traditionally, we have, uh, some input, passes through an embedding. It goes through your transformer layers, and we get our hidden state. This is gonna be a single vector of some dimensions, uh, that's gonna be the same as your embedding dimension itself. And what we would always do is we would say, okay, like, um, you might take the-- you, you might then have a, a linear layer, and you get out basically a vector with a whole bunch of probabilities, uh, that correspond to the probability that each individual token, uh, for that position is the next token that should be selected. So that's the original transformer flow itself. What Medusa does is that it slaps on a whole bunch of new MLPs that operate on the same hidden state and try to make the same prediction itself. So you can see this LM hit predicts it I-S. Medusa predicts is the second hit. The first hit which predicts the second token, uh, in the completion, uh, it goes for is comma the is a prosophaeria and, and so on. These heads aren't anything special. They're really just NLP networks that generate a distribution over the vocabulary. So you can see over here that all it does is, is just-- Well, the final hidden state, this is probably gonna be a one times D vector, right? It's multiplied by a single weight matrix, W1K, which is D by D. So that gives you a one times V, right? And then we add the residue itself. And so then once we add these two residues together, we do a softmax. I feel like I might be messing up the dimensions, but basically it's a... You're gonna get out a probability distribution at the end that's equal to the number of tokens that you have. Each head is essentially gonna produce a probability distribution over all these different choices. And so you, you're gonna get, well, basically SK different options for each token. I think the best way to see it is sort of over here, we can see the completions, right? So this is what the original language head sort of predicts. These are gonna be the next tokens that are predicted by these. First head, the first Medusa head, the second Medusa head, the third Medusa head, and so on. And so what they do is that they always choose the first token that's generated by the original language modeling head to guarantee that your entities get some completions. But then when it comes to the, the other tokens being chosen, and that by itself, there's some new way that you have. Um, so I think they also mentioned that they do some sort of greedy algorithm during training on the training data set, whereby they try to determine the optimal size for this tree itself at each individual node level. So there are two ways that they do this training. One is that you freeze the base LM and you only do the hits. And what this does is that you basically are just doing the same, uh, cross-entropy loss. But you apply this, um, sort of biased constant term here that is a constant taken to power of K. So what this means is that, uh, for the overall loss that you, that you're calculating, like they call it L Medusa one, uh, this is the first way that you train the Medusa level, Medusa head. Uh, you're essentially weighting the token, the, the hits that are predicting tokens that are further and further out into input sequence less and less. Uh, and this is actually super fast because you can get around five hours. You just need about five hours with sixty K samples and your 7B model is good to go. The harder way that, uh, yields a lot of better results is basically for you to train the LM and the individual hits that results in new loss equation of this is your original language modeling equation, and this is your, well, what we had over here. Uh, except this time they have the small little term called L naught, which is basically a small, small, small, small term, so that the, the head prediction doesn't mess up the overall loss because the Medusa head is gonna be super wrong at the start since it's, it's not trained on the data set, hasn't seen anything. It's just an NLP. So they do some sort of linear warm-ups whereby the learning rate is slowly increased over time and then maybe decreased, and there's some scheduling that's going on with that. Uh, last part is just this data set, which I think was, was pretty interesting. If you train your model on the new-- on, on this data set itself or the Medusa hits, um, it's not really a problem. Uh, you can just take a, a public seat-- public data set and you can train it. Um, and in this case, if you just-- If you're worried that the data set that you're training your model on doesn't actually reflect what the model has learned, you can just basically take a, a public data set with a whole bunch of prompts and just get your model to generate, uh, completions itself. And for certain models that have basically like, basically the ability to, to train on the, uh, the assistant, the user, the, the system, uh, you can have multi-turn conversations which is great. Uh, so that generally works pretty well from, from what they say about the Medusa hits. Uh, if you just freeze the base LM and, and you just train the Medusa hits. Uh, but they do say that, um, if you are training the whole model itself, uh, plus the Medusa hits, which is, uh, this step over here, you probably wanna also include like the little KL divergence term so that, um, when you run the loss itself, the model parameters don't change so much, and you wanna sort of minimize the difference of your model from your original model, uh, so that essentially you're still outputting like high quality, uh, completions. So yeah, that's basically the Medusa paper, uh, summarized pretty fast. There's a whole bunch of stuff that I've skipped over. Uh, but this is basically the, the, the main high-level idea behind the Medusa paper itself. Uh, so yeah, happy to take any questions. Uh, let me just try to pull out the chat if there's any questions. Uh, but yeah. Okay. "What does it cost more than increasing the beam search parameter?" So if I use five Medusa hits, is it like five times six by thirty? So I don't think they actually use beam search inside this itself. Um, so the way that I've seen-- So I looked at the code before this to try and see if they provide a few different ways. The first one is, yeah, some greedy new clip and sampling. Basically, the idea is that, uh, all you're doing with Medusa is that you're changing, uh, the use of a, of... You're changing the way that you generate these speculated tokens itself, right? So you originally use the draft model, but with Medusa, you use these hits. You're still running it through a separate, separate search. So depending on how you use your beam, it's-- how your beam search is implemented with the Medusa head itself, um, I guess it will, it will really determine like the completion. Um, but I, I think it's not super clear in the paper, um, how exactly they, they do the final computation. They just say that they try to find the long-- the longest prefix length that's common across all the different, uh, potential completions that are generated. Hope that answers the question, Bennett. Um, I'm just adding one, pulled up some stuff. Uh, let's see. Multi-token prediction. I've seen this paper. I haven't read it yet. So I, I think, uh, perhaps, uh, I will look at it after this and figure it out. Yeah. But, uh, I guess if not, then I'm just gonna end the recording over h- If anyone has any questions, happy to answer, and I'm just gonna end the recording over here. All right. Uh, Yong-Shin, I think I need you to end the recording. **Guest** [55:22] O-okay. Yes, sure. **Guest 2** [55:25] Yeah. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.