A Comprehensive Overview of Large Language Models - Latent Space Paper Club

All right, let's go.

All right, cool. So hey, guys. Thanks so much for coming by the, uh, Paper Club. As usual, um, this is a Paper Club we run on Asia, where we go through one paper every week. Uh, so today, we're just, um, recording it for the first time, and, uh, we hope that you'll benefit from it.

So as usual, if you guys got any questions, you can either, like, let me know, and I can invite you guys on stage, or you can drop it in the chat, uh, which you can access by just clicking the button on the top, uh, just the little, like, message icon.

And yeah, do you wanna take it away, Brian?

Sure. Thanks, Ivan. So, um, today we'll be going through, um, the comprehensive overview of large language models. Uh, but on top of that, I think what we wanna do also is just to share, you know, the reason why, um, attention actually came about, uh, before the transformers paper.

So we'll have a little bit of, um, a history lesson on that, on why it was developed, and then we will go through, um, the paper talking about what has happened, um, post the transformers era. Uh, in fact, it's when the GPT era started.

So I'm gonna begin. Um, as you can see, the link, um, has two parts. So I'll use the first part to talk about pre, I would say, GPT, and then I'll use the second link to talk about the paper prompt.

So, um, let's begin. So essentially, what models have been trying to do recently is this idea of language modeling, where given a previous sequence of words, which is your input or your prompt, you want to find out the next word in the prompt.

Right. In this case, it can be question and answers. So, uh, it can be modeled essentially by this probability of the next token, given the sequence of tokens. So that's when you can see, um, the next token, which is t plus one over, um, the-- given the sequence over here up to t-- uh, time equals to t, position equals to t.

And of course, your t plus one is a sample, uh, a sample from the vocabulary that you have, which is basically your sub-words or the tokens that you have. Okay. So, um, why is this the case? I think for us, uh, who are doing NLP, beyond just thinking about looking at what the sequence is, what, what, what's being generated in the sequence, we-- it's good to think about what kind of use case or what kind of task we are doing.

And I'll say this is very useful when it comes to thinking about the, uh, evaluation metrics for each of these evaluation tasks. So it can be things like-

Uh, your, your screen just kinda, like, cut out for me.

Is it? Okay. Let me see.

Oh, wait. Sorry. No, no, it's okay. It works again. So my bad. It just suddenly disappeared from me. Yeah. It works again.

Okay.

My bad. Yeah.

No problem. So, um, things like machine translation, they'll be talking about, um, you've got question and answer, summarization, so on and so forth. So essentially, um, good to think about what task, uh, we are trying to attack, uh, when we are using the different models.

Right. So while we think about language models as, uh, predicting the next token, it's also useful to think from a linguistic perspective, um, what is being learnt by these models. So, um, there's a list over here. Uh, I'll just go through a few that is useful.

Um, things like facts, which is trivia. So these are the ones where, you know, you can say, um, the penalty for getting the prediction wrong is relatively higher, because if you have, uh, if you output something that's false, then, um, your language model is probably not truthful.

Um, things like, um, sentiment, which we have, uh, seen before. Things like reasoning. So in this case, if you look at the sentence, uh, "Iroh went to the kitchen to make some tea. Standing next to Iroh, Zuko pondered his destiny.

Zuko left the..." So in this case, the idea is that, um, there is some sort of spatial, um, understanding. Uh, uh, the model needs to understand some spatial understanding of the sentence. In this case, um, Zuko is currently in the kitchen, so he left the kitchen.

Right. So these are some of the things, um, that from a synthetic perspective or from a linguistic perspective, we observe, uh, models are learning in terms of patterns. Right. So from, uh, language models, we talk about conditional language models.

So essentially, the idea is that we are trying to, um, generate a target sequence in a target sentence, given some sequence in the source sentence. So that is why, uh, you see over here that we are not just generating our yt, given some y1 to yt minus one, which is basically the, um, sequence that has been generated by the model before.

But also we want to condition it on the source sentence. Right. So that is essentially what translation does. You give-- If you think about it, you give, um, the model a source sentence, you pick the target, uh, language, and then you observe the model generate the sequence in the target language.

Right. So it's more than just language modeling, but it's also conditional. Right. And one of the key things that we will notice in language-- uh, conditional language modeling is that we don't necessarily see, um, that the first word in the source sentence corresponds to the first word in the target sentence.

So as you can see, this might be it, uh, first word to first word, but just the second word onwards, you start to see that there is this sort of, um, crisscross, uh, relationship Where you might need to-- where maybe the second word over here corresponds to the third word, and the third word over here corresponds to the second.

So essentially, the idea is that, um, we want to find a way to be able to model this relationship. And, um, this relationship has actually been studied before in this idea of alignment, where if you think about it, if let's say we've got the source sentence, let's say, um, on the top and the target sentence on the bottom or on the, on the left, then if we've got this very linear one-to-one relationship, uh, or this monotonic relationship, then we will see that there is these-- there will be a white box over here from the top left to the bottom right, indicating that the first word corresponds to the first word, second word corresponds to the second word, so on and so forth.

But as you can see, just from English to French, um, there is this idea where words, um, that is, uh, later in the sequence corresponds to words that's earlier and vice versa. So that is how we can, um, visualize attention.

Right. So then the question is, okay, what-- how are we, um, in a sense, modeling it, or what does it look like from the encoder-decoder perspective? So naturally, when we look at the encoder-decoder blocks, um, this can be, um, eh, let's, let's look at this as an RNN.

Right. We say that the hidden states, uh, the last hidden state in the encoder block, um, contains all the information of the entire sentence. But there's this information bottleneck problem, which means that if, let's say, this is a longer sentence, the last hidden state might not contain information of the earlier tokens.

And therefore,

there's this idea of attention where you have-- Given that you've got all the hidden states of all the input tokens, the decoder, when during the, during the language generation, uh, component, will pay attention or attend to a weighted sum of all the hidden states.

So if, let's say, I've got something that is, uh, later in my sequence, um, that corresponds to a token that is earlier in my source sentence, then I will see, uh, the attention weights giving more weight to the, uh, hidden states in the source sentence.

So essentially, that's the idea, um, of attention that has been implemented in the, um, encoder-decoder kind of paradigm or the kind of architecture. So the problem with that is that when we, uh, create these or we calculate these individual hidden states, we realize that it has to be calculated sequentially.

That means in this diagram, you can see that the second hidden state has to only-- can only be calculated after the first hidden state is being output. And the third hidden state can only be calculated after the second hidden state has been output.

So the question is, um, can we remove or break free from this idea where there is a dependency of the previous state? Because if we're able to do so, then we are able to run our forward pass and collect our gradients and run backprop on the architecture, uh, con, uh, concurrently across the whole sequence.

Right. So essentially, that's the, uh, idea of your key query value attention. Uh, and that essentially forms, um, one of the building blocks, uh, of the transformer architecture. Right. So, um, I think from here, what we're just gonna talk about is, um, there are other components to the transformer architecture beyond just our, uh, key query value attention.

Um, there is also this idea of understanding, um, the position of the text, and that's basically an idea of adding position representations that you will see in the paper later. Um, adding some sort of non-linearity, uh, when you're doing the calculation, and that's essentially just adding a feedforward layer o-on top of it.

So the idea is that if you're just calculating key query value pairs, you're always looking at linear, uh, combinations, uh, of your, you can say, your values, because you're just, um, getting a weighted sum of the values calculated by attention.

So we want to a-add a, a layer of non-linearity to it, which is, uh, taken care of by the feedforward network. And of course, the last part is, uh, when you're doing the decoding step, when you're generating the tokens, um, you want to not let the model see the future tokens.

And essentially, that's when masking comes into play. Uh, attention masking comes into play. So you will start to see that, uh, in the decoder architecture, um, later down the road. Okay. So a couple of things, uh, on top of what we are talking about in terms of the, uh, the language modeling component for transformers.

Um, one topic is subword models. So this is when you have things like, uh, tokenization, uh, your byte pair encoding. So essentially, what are we trying to solve over here? If you look at this, um, table at the bottom, we start to see that for words that exist outside the vocabulary, that can be things like a variation of an existing word.

In this case, you add many A's in between, um, the word T, uh, between T and A for tasty to probably indicate that it's very tasty. Or misspellings of words, which is also very common in input, or novel words over here where we understand the word transformify might mean, um, adding maybe a transformer, uh, block into an existing architecture, but it's a word that we might not see in the existing dictionary.

So for them, for these words over here, if you just use a traditional vocabulary or a dictionary vocabulary, the index will be some sort of an UNK token. Right. Um, and essentially what goes on with, uh, byte-pair encoding is that it starts to learn these shorter, um, combinations of letters that can sometime, um, represent either prefixes or suffixes of a word.

Um, and then essentially you are able to generate the embeddings for them. So if you see over here, you've got this T-A-A, um, and then anything after that, an A-A-A and anything after that, an S-T-Y. So this guy, probably you've seen it in other, uh, existing words.

Um, and therefore, there is an existing embedding that's associated with it, and therefore, we are able to represent it over here. You can think of it maybe as a... You, you're essentially creating-- You're essentially generating three tokens from this, uh, source sequence over here.

So essentially, that's the idea of, um, sub-word models. Or in this case, you've got things like, uh, byte-pair encoding, sentence piece, word piece, and things like that. Essentially, that's the problem that they're trying to solve. Okay. So, um, three types of, uh, architectures.

Um, the key thing over here to note is that what we have in the transformer block is essentially replacing, um, the recurrent neural network blocks that we had previously. So when we talk about recurrent neural networks, of course, we add things like LSTMs, GRUs, bidirectional models, multilayer models.

So it encompasses all that. And essentially, what we have over here are the three types of archi-- dominant architectures. Uh, we've got the encoder models, and examples of this will be things like BERT, where you learn via masked language modeling, which has been covered before.

Um, encoder-decoder models, where we've seen earlier, we have an encoder, um, that maps your sequence into, um, a space or a position in latent space. And then from there, you perform your, um, sampling or your autoregressive sampling of tokens to form your target sequence, which is what we've seen in T5.

And the decoder models, which I think all of us are familiar with, things like GPT-2, GPT-3, they are all there. So you essentially learn, um, the language or patterns, and then you directly just, um, do your, uh, autoregressive, uh, sampling or decoding from there.

Okay. So from there, right, we will lead to this paper that we have over here, which is the comprehensive overview of large language models. If you take a look at this paper, um, it seemed to me that there were multiple updates to the paper.

And that sig-signals to me that there's probably going to be updates along the way. So I think what's useful is beyond looking at just the paper itself, um, understand-- Or for me, what I did was I tried to understand what was the framework that the authors were using to attack, um, understanding of, um, the knowledge, um, then dividing it and then giving us a, a, a reader to understand it.

Right. Um, it's a very dense paper. Um, it's got, I think, over four hundred and fifty citations. So, um, I think it's more of a pick your own adventure, pick your own journey, pick your own, um, learning process, uh, kind of, I would say, direction.

Um, so that along the way, you'll be able to build that foundation knowledge and then, uh, add layers on it, add layers on it. At the end of the day, we all know that new models are always developed and new models are always announced.

Um, so going back to the first principles and fundamentals are useful. So, um, let's just go through the paper very quickly. Um, let's just start from the top over here. So essentially, we'll just talk about the last point over here, where we are seeing that large language models, uh, in particular things like GPT-3, are able to perform your downstream task without specific fine-tune.

So that's the first key point, because if we looked at T5, um, we saw that the performance of T5 on downstream tasks, in this case, it can be translation, it can be your GLUE task, it can be your SQuAD task.

Um, their performance only will get better once you fine-tune on that particular task. Right. And you've seen there, there are multiple experiments where they've done, um, which demonstrates that that's the better way. That's the better alternative. So, um, what GPT-3 demonstrated was that they are able to perform zero-shot transfer learning on these tasks.

So what does that mean? That means that if you just give the prompt, uh, from the downstream task, GPT-3 is able to give the answer. So that kind of changed things where we actually might not need to fine-tune for a particular task.

Of course, when we look later down the road, we see that there's, um, very, very specific ways of doing things like instruction tuning, right? But that was one of the big discoveries that they had back then. On top of it, um, they were able to show things like reasoning.

They were able to show things like planning. They were able to show things like in-context learning. So we, we get to see demon-- uh, examples of this later when you do things like, uh, chain-of-thought prompting. So they are able to understand, you know, like given certain patterns, um, when they ask for the next-- when they ask for, um, a question that or ask for a task that follows a similar pattern from the prompt, they are able to answer.

Um, the problem that we see today is that, um, the cost of training them or pre-training them is relatively high, usually in the tens of millions. Um, so the question is that can we get better at pre-training these models?

Um, can we look at things like better architectures? Can we look at things like, uh, more, um, efficient ways of, um, fine-tuning our parameters? Um, are there ways that we can represent these vectors in a lower, uh, vector state or, or a low-- or a state that is, that uses less, um, granularity, right?

So that's essentially where things like architectures come into play, uh, quantization comes into play. So, um, the way I div-- I saw this paper was that we had the background, which talks about some of the key concepts, um, and then the different types of LLMs and their particular use cases, um, the datasets that have been used to train them, at least the public ones.

What kind of evaluation tasks are they looking at? So probably that's what we call evals. Um, and the different types of applications, uh, for these LLMs in the commercial world. And of course, um, from there we talk about, you know, what, uh, probably researchers are looking at, um, going into maybe, say, the next three months or the next year.

So let's look at some of the fundamentals. So I'm gonna start from the left side. Um, the paper has covered-- We have covered some of these topics from the paper: tokenization, um, attention mechanisms, um, the different types of activation functions.

So those are stuff that we've learnt. Uh, you can get a recap when you do your traditional, uh, deep learning, um, topics. Then, of course, we talked about the different types of architectures, which was covered earlier. Your encoder only, your encoder-decoder, your decoder only.

And naturally, each of them will have their own associated way of doing attention masking. So that's this part over here. Um, we talked about the different types of pre-training objectives. Naturally, things like masked language modeling are things that we see in, um, your encoder-only models.

Uh, language modeling are things that we see in your encoder-decoder models. So masked language modeling, basically in this diagram is, uh, you feed-- you give the model this token and these tokens over here, and the model is expected to predict these targets over here, the ones that have been highlighted.

Whereas in full language modeling, so essentially it's like a fill-in-the-blank kind of problem. Whereas for full language modeling, you give the first token, and then the model is expected to predict the second, third, fourth, fifth token, so on and so forth.

So that's that. There have been also, um, research into this thing called prefix language modeling, where you feed the model, um, one part of the sequence, and then you're asking the model to, um, generate the remaining, uh, parts of the sequence.

And what's useful over here is that when they do prefix language modeling, they use this thing called, um, a causal mask with, a causal mask with prefix, which means that the, for the input tokens, the model is able to see or attend to all the previous tokens in the input before it starts to generate output.

And that's why when you see, uh, as the model generates the output, you still, you still have that, um, element of masked attention. So essentially, that's this part over here. Um, things that are, I would say if you look at the transformer paper, which is, which is covered over there, will be things like, um, layer normalization, where you, um, div-- where you divide the weights by the mean.

Sorry, sorry. You minus the mean from the weights, and you divide it by the s-- the standard deviation of the weights. Essentially, what we're doing is that we're trying to achieve, uh, numerical stability, uh, of the weights, right?

So that as when you, when you do a forward pass and you do, and you do your backpropagation, you don't have numbers that go, uh, all over the place. So that's, uh, layer normalization. Um, positional encoding was something we talked about earlier.

Um, in the original paper, they had this idea of sinusoidal position representations. So how to read this, uh, graph.

Okay. So essentially how to read this graph is that as you go from left to right in the, uh, as the index of the sequence increases, essentially you are applying some sort of, uh, sinusoidal function on top of it, such that every, uh, token in the sequence has a positional representation.

It's augmented by a positional representation. So essentially from left to right, all these vectors actually look different, right? Um, but what happens is that, um, this way of encoding positional representations, um, is not learnable because there is no such way to do, um, it's no such way to have a gradient and then to, um, update the positions.

So therefore, uh, it has been changed to something as simple as just adding, um, a positional representation on top of the, uh, embeddings. And of course, if you look at the paper, there are also newer ways to do it.

Things like LeVi, things like, um, RoPE. So that's the left-hand side. Now, on the right-hand side over here, we are looking at newer ways or, uh, ways that can help with training or implementation. So things like, um, the libraries that we're using, um, JAX, PyTorch, TensorFlow, amongst others.

Um, there's this idea of distributed training, which means that, um, can we use multiple GPUs to train, uh, our models so that we are able to learn the weights faster. So amongst others, there's this idea of data parallelism, where you duplicate your model in two GPUs.

Let's say I've got two GPUs. I duplicate my model in both GPUs, and then I run separate batches on top of them. So let's say if I've got a batch of, I don't know, a hundred thousand, right? I split it into fifty thousand, fifty thousand.

I run the first batch of fifty thousand in the first GP-- in, in the first model in the first GPU, and then the other fifty thousand in the same model in the second GPU, calculate the gradients, average them, and then perform my backprop.

So that's what data parallelism is. Um, tensor parallelism, essentially the idea is that, um, you calculate the, uh, matrix multiplication steps, uh, in multiple GPUs, and then you add them up. So what happens is that, as you can see, uh, we know that for each row, um, the calculation-- the multiplication with a column can be done concurrently, and therefore it splits it up such that the first-- that this, uh, the, the matrix on the left multiplies with only one column, matrix on the right multiplies on the second column, and then you combine them together, or in this case, you concatenate the results together.

So that again also helps us, uh, with getting, uh, the results from the forward pass a lot better. Okay. So that's that. Um, other kinds of tricks that we are using are things like flash attention, where, um, it's a very smart way of utilizing, uh, memory.

So what happens is that instead of calculating-- instead of a series of steps that is very memory intensive when they load your, um, your query, your key query matrix, let's do the calculation, perform the softmax, and then get your results, um, they are doing some way of-- they are iterating it, uh, and they are using very smart functions, um, to calculate things like the, uh, softmax on the fly.

So essentially, that's what they're doing over here. So it's an optimization of, um, using your high bandwidth RAM and also the, the RAM in your, your GPU, right? Because in your GPUs, you've got very fast computation, but relatively lower memory.

Um, just a little bit extra. This is one of those very common topics that they would like to, um, start off as they go into things like your, uh, Mamba models.

So that's just the first part. So the second part in terms of, um, the background will be how do we adapt these models for specific tasks. So, um, there are things like transfer learning, which we've seen before, where we pre-train, um, our T5 based model, and then we fine-tune individual tasks.

Um, there's also things like instruction fine-tuning, where we are-- where the model is given a series of instructions and outputs, and then the model wi-will fine-tune, uh, its outputs based on that. So examples of this can be things like, if let's say I ask GPT to explain the moon landing to a six-year-old in a few sentences.

Generally, in-- if the model is pre-trained, there is this way where, um, GPT outputs the steps in this way, right? So explain the the-the theory of gravity, explain the theory of relativity to a six-year-old, and then explain the Big Bang to a six-year-old, and then, uh, explain evolution to a six-year-old.

So that's how GPT-3 will output its sentences. But if we're able to do some sort of instruction fine-tuning where, um, there is some sort of emphasis on things like six-year-old in a few sentences, then, um, this is the kind of outputs that you can get.

And so that's the kind of, uh, variations of different models that we can see when we download them, uh, from open source, um, let's say, repositories, right? Things like Hugging Face. So that's instruction fine-tuning over here. Um, and something called alignment tuning, where you want to ensure, um, that your model, uh, fulfills what people call the three H, the three Hs of, uh, model behavior.

So your models will be harmless, your models will be honest, and your models are helpful. So things like harmlessness will be things like, if let's say, um, how can I-- if let's say you ask the model, how can I, um, let's say bake a cake with cyanide in it, right?

If let's say your model is not align-alignment tuned, um, the model might give the, the instructions. But let's say if you do alignment fine-tuning to tell the model, "Hey, this is something that you should not output or you, you should, um, you shouldn't give instructions for," then, uh, the model will learn accordingly from that.

So these are some of the methods that we want to fine-tune our models with such that our models are able to demonstrate a certain behavior. Then how are we doing it? We can use things-- We can use skills like, uh, reinforcement learning to do it, um, where essentially you-- for each of the different outputs, you have a certain kind of reward.

In this case, the reward is just a scalar value. Um, and then you learn some sort-- you, you learn some sort of, uh, policy such that when the model outputs text based on this policy, uh, you get to maximize the reward.

So the key thing over here is that, um, the policy has to be differentiable, so that when you get some results from, um, the model output and you get some reward, uh, sometimes your reward may not be good or you're comparing rewards.

You're able to get the loss of the reward and back propagate it, back propagate it through the gradients to update the weights, uh, in the policy. So that's essentially what reinforcement learning is. Um, so in-- typically for, I think when, when reinforcement learning was a hot thing back then, it wa-- it, it's, it's one course by itself, right?

So this is just a very high level, um, five, six, uh, five-minute, uh, overview of it. On top of it, I think one of the things you are more familiar with is things like prompting. So we've got zero-shot prompting, where you just ask for tasks.

Uh, you, you just give a task and the model answers directly. Uh, but also you have things like chain-of-thought prompting, where you give the mo-- you give the model some examples before, and then from there, um, ask the model to mimic the behavior of the examples above.

So that's essentially what you have over here. You've got, uh, in-context learning of, I would say, translation on the, on the right, and you've got, uh, in-context learning of correcting, uh, spelling mistakes on the left.

So that is essentially, um, this part over here. And you got to see that, uh, a few shots or things like, uh, five shots or three shots usually have better performance against, um, your zero shot or one shots.

So that's this part over here. Um, and then the question, of course, is, you know, how do you craft, um, this, uh, these prompts such that you'll be able to get the results that you want. So that's essentially, um, the idea of what people like to call prompt engineering.

Okay. So that's essentially the part of the background that we wanna cover. Um, the next part over here, I would say, is, uh, a very brief list of some of the models that we have. Now, the-- keep in mind that, um, a lot of these models, the, the, the list always is updated every two or three weeks.

Um, so good to understand. Uh, so, so naturally, I think when the paper is going to be updated in the future, you will see add-additional models. Um, some of the high level, I would say, purposes that we see, um, these models are trying to achieve can be things like your general purpose ones.

So that's when you, you get your model to do all sorts of things. Um, there's also, of course, your multimodal ones, right? When you, when you ask the model-- When you give the model some, uh, image, then you-- if you ask the model to, um, decipher some fact or draw some conclusion from the image, right?

There's also, of course, your video related ones. Um, there are some that are very specific to code generation. So here are some of them. Um, some they are very specific in the finance, uh, domain. Some that are very specific in the science domain.

Um, and of course, um, there are some that are very useful for chatbots, right? So this is the list over here. Uh, there's a much more, um, detailed list, uh, in the paper itself. Um, having said that, of course, as, as, as mentioned, um, there are also additional papers, um, that come out, right?

Uh, and so some of-- There are also some, I would say, missing, um, models, mis- models that were not mentioned. So these are some of them that are not mentioned. So good to understand that this is always an evolving, um, list.

Okay. Um, so what are some of the features that we see in these models? Uh, you've got things like your, uh, instruction tuning, which was talked about earlier. Uh, we notice that models are able to have, uh, increasingly high context windows.

Now, the context windows are in the six figures, uh, sometimes even in the seven figures, right? Um, there are also, um, other ways in which LLMs can be used. Uh, I think a very popular one is RAG. Um, so there are, I would say, beyond just your general purpose use, you can always fine-tune them for very specific purposes or purposes that, uh, are very specific to maybe your own corpus or your own knowledge base.

So that's essentially what we're doing over here. Uh, other topics for the reader explore. So essentially what we are doing over here is, uh, we're talking about ways-- Actually, most of these topics over here, if you look at them, are about parameter efficient fine-tuning.

So things like quantization, uh, where, let's say instead of representing a number in thirty-two bit, I represent my number in eight bit or four bit and see if I can, uh, still maintain the model accuracy. Generally, the model accuracy will go down.

Uh, but the thing is, if you are able to get lighter models, smaller models, uh, that's actually very useful. Uh, multimodal LLMs that we talked about earlier that take in things like, uh, images and video as inputs. Um, adapter tuning essentially is when you just add another layer on top of the output, and then you perform fine-tuning on it.

Um, there are more sophisticated ways to use it, where your adapter is used in mul-- in two or more, um, models. That means the same adapter is being used in, let's say, a general model and also, let's say a, uh, GPT model.

Um, I've seen that in the, um, talk about embedding, uh, representation learning. Uh, mixture of experts, uh, something that we've seen before also, where instead of just, um, having one feed forward, uh, layer over here, you are actually able to route them to different, uh, feedforward, uh, layers.

And then from there, uh, you'll be able to, in a sense, uh, then once you multiply them together, so you, you'll be able to leverage on different, uh, I would say compo-- different vertical workflows of the model, where each of the vert-- uh, vertical workflows will learn, uh, different aspects.

Right. So that's essentially your MoE. Uh, low-rank adaptation or LoRA, um, this looks very popular recently. So essentially what we're trying to do is if you are able to reduce the number of, uh, parameters during your gradient updates, then you actually use less compute to get your, uh, fine-tune models.

And the idea behind it is that instead of-- So let's say over here, instead of, um, calculating gradients for sixty-four parameters for an eight by eight matrix, what you can do is that if you can decompose this matrix into a, uh, eight by two and a two by eight matrix.

The key thing over here is that when you multiply this by this, you get back, uh, the sixty-four-- you get back sixty-four weights, or the resultant is, uh, is an eight by eight matrix, which is sixty-four weights. If you're able to decompose it, uh, with weights in a smaller dimension, essentially this idea is that-- And of course, the how small it is, is, uh, a hyperparameter for you to tune, then the cost of, uh, fine-tuning, uh, will go down.

Okay. So essentially that's what we're doing over here.

Yeah. So that's pretty much it for this segment. Um, the last-- The next few segments, uh, this next segment essentially is about your datasets that can be used for training, at least the public ones that we see. Uh, we've got a-- These are things that we've seen before, Wikipedia dataset, C4 dataset, Common Crawl, uh, which is used for your, I would say, more general purpose models.

Um, and then of course, you've got some, um, datasets that can be used for very task specific, uh, models, for example, code generation. You've got, um, datasets that is used for instruct-- uh, instruction fine-tuning, and you've also got datasets that's used for alignment.

So, um, essentially what happens is that if you go to maybe say, um, TensorDataset, TensorFlow Datasets or Hugging Face, you'll be able to download them, um, and then you'll be able to observe, um, these datasets, uh, by itself.

And if, let's say, you want to maybe say, um, fine-tune a model for specific use, uh, these are actually useful, I would say, templates or schemas that you can use, uh, to prepare your dataset so that you can do fine-tuning.

Um, so this is instruction tuned, um, and this is for, um, getting the model to be more-- to, to, to have-- to display, uh, behavior that is more aligned to, uh, our use. So naturally, this one I'm okay to share, um, some examples, but this one you can go ahead and click on the link.

You'll be able to see the kind of examples, um, that's over there.

So let's say we've done our training, our fine-tuning. We find-- We found a way to, um, get our-- to, to, to update our parameters in a more efficient way. The final part, um, is of course evaluation. So, um, I think we'll cover, uh, at a high level, um, two classes of, uh, model evaluations.

You've got things like your single task evaluations, so very popular ones will be things like SQuAD, uh, StoryCloze, MATH, uh, MNLI, uh, which is for question answering, uh, understanding context of words where you're filling in the blanks, um, answering questions, answering math questions, uh, so mathematical reasoning.

Um, and this is, I believe, uh, natural language inference. So essentially whether your-- whether the two sentences are, um, they, they follow each other or not, right? Essentially, whether, uh, the next sentence logically follows the first sentence or not.

Um, and also things like TruthfulQA, which validates whether a sentence, uh, whether the model outputs facts, uh, instead of maybe just say other kinds of, um, maybe trivia that's not true, not truthful. So these are some of, I would say, your, uh, single task, uh, evals, and then you've got your multitask evaluation, things like GLUE, things like MMLU, um, things like SuperGLUE, and of course, there are a couple more that's inside, um, the list.

So what happens over here is that if we just take look-- just take a look at GLUE, um, there are-- This is divided into multiple, uh, m- multiple individual evaluations. So you've got things like, uh, natural language inference.

You've got things like, um, whether a sentence, uh, makes sense or not. So that's your QALU. You've got things like semantic similarity. So essentially that's what's going on over here. Uh, MMLU, which is one of the more popular ways of, uh, doing benchmarks right now.

Um, so there's a big number of, uh, knowledge intensity tasks that you can see over here. Um, and of course, SuperGLUE, which is the second generation from GLUE, which has, um, more, I would say, I would say questions that mimic, um, human behavior more or things that are a bit trickier for models to understand.

Okay. So, um, that is the part on evaluations. Uh,

so different kinds of applications, I think we've seen many kinds. So beyond just things like, uh, what's in the list, uh, we also see things like music generation. We see things like, uh, video generation. And naturally, what happens is that for each of them, um, there are also certain guardrails that need to be in place.

So what are some examples? If, let's say, for a, uh, music generation, uh, model, uh, it is important to ensure that when we submit lyrics for, uh, the model to output, um, these lyrics should- shouldn't be under any cop-copyright.

If not, then there might be legal consequences, right? So, um, this is something that, um, I would say depending on the domain that you're in, um, you will be looking at models that are very specific to your domain.

So finally Last part before we go into Q&A. Um, what are some of the things that we see models exhibit? So things like biases are very common. Stereotypes are very common. Uh, and I guess the reason why is, is based on some of the training data that we see.

If the training data exhibits a certain behavior, naturally we see the model, um, exhibit, exhibiting this behavior. So, so that's I think one of the things that we want to be aware of. Um, and also things like, uh, models, um,

memorizing private content. So if let's say I've got a GPT model and I, I type in a particular prompt, uh, and this GPT model sees some email, and then it outputs some sort of phone number that is supposed to be private.

Um, and let's say a user takes this and does a search on-- So essentially, the idea is that this is the output from the model, and you can see there's actually some information over here that might be, that might be private.

You might have a phone number that's not supposed to be, um, exposed to the public. Um, and then maybe someone searches for that phone number, and there you might have an additional contact, uh, that maybe you can use, right?

So, um, these are some of the things that we want to, I would say, be aware of, um, when it comes to, uh, the component about human alignment. So on top of the three Hs, help-- making helpful, being harmless, and being honest, uh, you also want to ensure that your models, um, do not have-- do not, do not leak out or do not learn certain private information.

Um, and generally what happens is that there is, uh, teams like-- There are, there are teams that are behind, uh, all these ways of conducting adversarial attacks. You can call them white hat attacks or what people like to call, uh, red teaming these models.

So essentially trying to generate adversarial, um, prompts or find ways such that the model will leak out something. Um, and then if they're able to do so, they will fix it. Okay. I think there's a few, um, interesting articles about that recently.

So essentially, uh, that is the paper. Uh, it sounds like a firehose of information. Uh, so if there's anything, any topic you wanna deep dive into, feel free to, um, take a look at the paper or take a look at this, um, and go into the topics that you're looking at.

So if, let's say, I want to just do something on parameter efficient tuning, feel free to just go into that segment. So, um, I've linked all the papers, uh, over here. I've also linked, uh, some of the external sources that have been useful for me, uh, over here.

So, uh, yeah, feel free to take this as a reference guide for yourself. Uh, and I think with that, I've come to the end, uh, and I'm leaving about ten more minutes if there's any Q&As. So, uh, Ivan?

Uh, yeah, dude. Thanks so much for, thanks so much for giving the-- such a detailed, like, walkthrough. I think there was a question by Hong Nan in the chat about, uh, parallelization of-- like what exactly is the benefit of using a transformer versus a, I guess, in this case, a RNN or LSTM.

Do you wanna maybe start with that? Like how the parallelization works.

Hmm. Let me just take a look. Okay. So if you think about it, um, let's look at this example over here.

One second. Let me just... Okay. So the idea over here is, um, if you think about the traditional RNNs, what happens is that, let's say I've got a sequence of ten tokens, and I want to calculate the hidden state of the, the entire sequence, in this case, the sequence-- the, the hidden state of the tenth token.

There is a dependency, um, of the ninth token, and the dependency of the ninth token is the-- Sorry, the ninth hidden state and the dependency of the ninth hidden state, the eighth hidden state, so on and so forth.

And essentially, that's what's going on over here, where, um, if, let's say, I want to calculate the second state, the second hidden state of the second token in the sequence, uh, I need to calculate the first, I need to calculate the first hidden state as an input.

Um, so that goes, uh, back to, um, either your RNNs or LSTMs, where, um, the hidden state is, um, calculated. The, the, the input to that hidden state is the hidden state of the previous, uh, token and also the input token.

So the thing is that because there is this, uh, dependency, um, there is this reliance on, uh, the future hidden states rely on the, the previous hidden states, and because of that, there is no ability to parallelize, um, from a sequence perspective on the wall clock perspective.

And therefore, you see the first line, backforward and backward pass per step O sequence length. That means for how long the sequence length you have, um, you have to do that number of calculations. Does it make sense?

I think it makes sense to me. At least the way I like to think about it is that, let's say I had five sentences, and they're not the same length. In order for me to get the final hidden state before I can start evaluating its predictions, I need to run, like-

Hmm

... five passes and for each character in each sequence or each token, in this case. For a transformer itself, I can just pad everything to the same length and pass it through in one time step. So I can get everything out in like one s- one like output step, one forward pass.

At least that's my understanding-

Correct

... of the parallelizability of it.

Yeah, that makes sense. I agree. Um-

Yeah.

I would say for this, um, di- this diagram, we think of it during the training state. Naturally, during the inference stage, um, we still have to-- There, there is still, um, there's still this need of passing the hidden state of the current token back into, um, the transformer, the, the model to get the, the next token.

Yeah.

Hmm. For sure. For sure. I was thinking more about the training stage too, but I think in terms of inference-

Exactly

... you, you incur the additional costs at the transformer of each additional token that a RNN doesn't-

Hmm

... I think, ultimately. Actually, for me, like one of the questions I had about the classification in this paper was that of prefix versus full language modeling. Because if you look at the example-

Hmm

... that they give in the text, uh, I think they give the example of-- They have just a cute, cute little example, which is if it's full language modeling, they give the word may, and then you output the word the force be with you.

If it's prefix-

Yes

... language modeling, it's may the force and, and then the model is asked to predict the be with you. But-

Hmm

... that just both seems like the same thing because my understanding of prefix-

Hmm

... language modeling was that, oh, we're gonna specify specific token. For example, like, uh, like a bracket classify, bracket like sentiment, sort of like in T5, and the model learns that if it sees this specific like prefix, then it should like sort of the-- it should perform differently.

And so that was why I was a bit confused by in this specific paper. Yeah.

Hmm. Yeah. That makes sense. Um, I didn't look at the paper in particular, so it's a little bit hard to, uh, comment on that. I understand when you are saying that, you know, this, this, and this really doesn't show a lot of difference.

Um, I think what I will-- I think what I can comment is that, um, generally in full language modeling, what happens is that you, um... Okay, this is, this is of course the encoder-decoder, uh, phase of things, uh, beyond the GPT stuff.

So, um, generally what happens is that for full language, what hap-- uh, you, you generate everything. So in fact, right, maybe in this case, uh, you might just start with a beginning of sentence token, and then you take maybe some hidden state, and then you generate from there.

And then you auto-regressively sample from there. Uh, which is different from the prefix language modeling, where you are given a series of to-- the, the beginning of sentence token naturally, and then a series of tokens before you do your, uh, generation.

And then, of course, when you do your learning, you are learning based on that, that particular sequence, uh, of text more than just the, uh, beginning of sentence. I'm not very sure. I think we-- this one, this one we've got to take a look at the paper to, to fully understand.

It was also the-- It was-- The guy also was the author of the T5 paper, I believe.

Oh, really? The guy who did this paper?

Yes. Uh, I think his name is Colin. Yeah.

Yeah.

But let, let... We have to go again. We have to go and check. Yeah.

Yeah. And I think we can, we can, we can talk about this some other time. It was just, I was just-- It was just something that confused me quite a good amount. I guess the other thing that surprised-

Hmm

... me was just, uh, like learned positional encodings. 'Cause I-- When we-

Hmm

... covered the original transformer paper, I think there was a section where they said, "Oh, we experimented with learned and frozen positional encodings." But it seems like, you know, like you mentioned, that newer papers are starting to use learned positional encodings instead, and it's shown like an increase in performance.

And I was wondering if maybe like, you know, what sort of change, in your opinion, to make this happen, if that makes sense.

Mm. To be very honest, uh, it's-- I'm not very sure what, um, what were the changes that inspired it. Um, maybe the way I would comment is that, you know, once they are able to do so, um, they are able to represent-- they are able to efficiently represent, um, an input with a much longer context window.

So I think there wa-- probably what happened was that there was, uh, innovation in that space. Because the thing is that if let's say I've got maybe say 500 tokens or 1,000 tokens, um, there might be a limitation on how you, um, you, you, you, you, uh, model the po- model the positions because maybe the positions might all be just clustered in one area.

But I think once they have figured out how to do so, that's when they open up the window to longer context windows. So maybe how they, how they learn, uh, how they learn, uh, positional encodings might be one of the tricks that they use to, to, to have, uh, longer context windows.

But again, I might be wrong. Uh, didn't really go into the details of this, uh, part of research.

Yeah. For sure. For sure. Yeah. I was just wondering about it-

Mm-hmm

... 'cause that was just something that, uh, I was intrigued by. Yeah. I think we're-

Hmm

... almost at time. If anyone has any other questions, you can drop it in the chat. Uh, if not, maybe we can just end it here. Anyway. Okay. Um, it seems like there's no more questions. Uh, so anyway, I think moving on to next week's paper, uh, I was thinking of doing the DeepSeek MOE paper.

Uh, that was one thing I'd like to present, to propose, sorry. Uh, 'cause it-- I, I found it super interesting, and, uh, there are a whole bunch of these ideas that they're, they're experimenting with, like always-on experts, randomly routing ex- randomly routed experts.

So I thought it's a good paper. Um, so as usual, if, if, if anyone wants to present on the paper itself for the upcoming week, then, um, you know, happy to help you with it. I think you, you generally learn a lot more when you s- when you actually do the paper.

I learn like at least like 10 times more if I, if I actually have to sit down and present the paper. So, um, I think as usual, I'll probably just drop, um, like a thread inside the paper, Paper Club.

And then, uh, if you guys have any other papers that you'd like to suggest, you can add it on to the thread, and then we can upvote for that. Yeah. Do you have any papers in mind, Brian? Anyone has any other papers that you guys wanna read?

Mm. I'll take a look. I'll take a look at them. There are some, there are some, I would say, um, very open source models. Um, so we'll see, we'll see how... Maybe one day next month I can take a look at them.

Yeah.

Okay. Cool. Cool. Sounds good to me. Yeah. Then otherwise-

Sounds good

... thank you so much, guys, for tuning in today's, uh, session. Really appreciate it. And, um, yeah, looking forward to next week, guys. Ciao.

Thanks, everybody. See you guys. Bye-bye. Have a good evening.