Breaking down the OG GPT Paper by Alec Radford

Okay. Sure. So hey, everyone. Uh, my name is Amget. I'm a machine learning engineer. I generally do ML consulting services to startups. I help them, like, ship AI, uh, powered products, especially in the field of NLP and speech-to-text applications.

And I run a pr- a blog where I, like, publish posts about ML stuff, so feel free to check it out. I've done some posts about Whisper. Uh, yeah. So with that out of the way, let's get d-directly to what we want to discuss today, which is, like, the GPT-1 paper by the folks at OpenAI.

So, uh, the paper is titled Improving Language Understanding by Generative Pre-Training. It was published in June 2018, and these are, like, the authors. Let me switch to presentation on. Yeah. And these are, like, the authors of the paper, so, uh, mainly Alec Radford and Ilya Sost- uh, Sutskever, very well-known publishers in the ML field.

So let's get started. Uh, back in 2018, like, uh, deep learning was, was, was becoming very popular, but the main, uh, thing with deep learning is, like, it is very, very data hungry. Uh, there are, like, good news and bad news.

The good news is, like, there is a ton of data available everywhere on the internet and just online. We have, like, tons and tons of data. The bad news is, is this data is not annotated, and it's not curated, and it's basically very, very messy.

And if you want to train machine learning models back then, the only way to do it is just to annotate data yourselves or hire, like, data annotators, and these tend to be very, very expensive and difficult to scale and hire.

Like, if you think GPUs are expensive, you have not worked with data annotators. That's what I'd like to say. Uh, so this, this makes, like, deep learning very restricted. You cannot use it everywhere. Like, you're only restricted to fields that have good, high quality annotated datasets.

Uh, and this is a very big bottleneck. So people, uh, back then in 2018 are trying to solve this problem. Like, how do we, how do we get, like, over the fact that we need labeled data?

And one potential solution to this problem is, like, unsupervised learning. So the question is: what if we can leverage, like, just the linguistic information from the unlabeled data? So we have, like, a bunch of text, uh, like novels, books, articles.

How, how can we leverage, like, the ling- uh, linguistic information from this? And the answer to this could be using unsupervised learning. And if you can do this successfully, this alleviates the need for large amounts of labeled data, uh, because basically you can utilize Wikipedia, which is very big, or even, uh, all the papers published on Archive and so on.

And even if you have, like, lots of labeled data, using unsupervised learning as a step one or step zero is gonna make your model actually perform better than just training on this labeled data because of transfer learning, where you can transfer the things that your model has learned during pre-training into your actually objective.

And a good evidence of this is back in 2017 and '20, or even a b- a bit before 2017, is, like, people have been using, uh, pre-trained word embeddings, so things like Word2Vec, GloVe, FastText, to achieve very good performance on many tasks like classification and machine learning.

Uh, sorry, machine translation. So this, this is a good evidence that actually unsupervised learning works, and it's a very good approach. So, um, this is the premise of the, the paper actually. Like, unsupervised learning is very, very promising.

So let's talk a bit about word embedding. So the idea behind word embedding is, like, you want to project words, so just text into an N-dimensional space, and N is usually three hundred or one thousand, uh, back then.

And this space has a very special property that words that are similar in meaning have very similar products. Uh, sorry, very similar vectors. And by, by, by similar vectors, I mean w-with, like, you can measure similarity by things like cosine similarity, dot product, or, like, L2 distance.

Uh, so we can capture simi-similarity between words that don't each other, but have similar meanings. For example, the word booking and reservation. Just from the syntax, they are very different words, but they have very similar meanings. We're just booking something.

And similarly, the word Adam and SGD, these are, like, completely different words, but they are both actually, like, optimizers and used in machine learning, so they are very similar, and their vectors are gonna be, uh, probably similar. The most common implementation of word embeddings, like Word2Vec by Google, uh, this is what popularized, uh, popularized the usage of word embedding, and then GloVe by Stanford and FastText by Facebook.

And the way these, these word embedding models are trained is, like, uh, leveraging co-occurrence between words that have, like, similar context. For example, similar words tend to be, like, tend to occur in similar context. Like, you would generally find the word Adam associated with learning rate or machine learning.

Similarly, SGD is, is associated with learning rate as well. So we can conclude that these two words are kind of similar. And the way these word embedding are, are used is that they are, like, utilized by training a head.

Like, for example, if you wanna classify a word as being positive or negative, having a positive sentiment or a negative sentiment, you can train a classification head on top of the frozen embeddings. So you use the word embeddings, like, as a fixed, uh, feature input, input features, uh, just frozen things without training the word embeddics, uh, embedding themselves.

And this has, like, a significant drawback or a bunch of significant drawbacks. The first thing is it, it doesn't utilize the context of the te- uh, of the text, so you're just using the word itself. And some, some words, even the same word can have m- very different meanings depending on the context.

For example, the river bank, uh, like the Amazon bank or the Nile bank is, is different from the HSBC bank or like the JPMorgan bank, even though these are the exact same words. So if you're gonna use just wave2vec, these two words will have the same vector, even though they are like have very different context.

And, and even beyond this, like natural language has like nuances that cannot just be captured by using words like the way you write things like Llama 3 win and skill issue and OpenAI, the way you write it this way, like you, you have a very specific intonation compared just to writing OpenAI in the normal way.

Uh, so are there any questions about word embedding so far?

Uh, someone says in the chat, "Nope." Um, I, I cannot read the chat, so if just someone can say, uh, this in, using the microphone, that'd be great.

I think, um, I don't think we have any questions. Um, I think Sean said no, but if anyone has any questions, you can maybe drop it in the chat, and then I can just surface them as and when.

Uh, but it seems I think everyone's okay right for now.

Yeah. That's, that would be great.

Yeah. I'll keep an eye on the chat for you.

Yeah. That'd be great. Yeah. Thank you. That's appreciated. So this, this was word embedding. The, the main limitation or drawback is like you cannot use context. So can we go beyond word embedding? Like word embeddings are very like too local.

We need something that's more global that can capture the higher level semantics. But this also has, has its complication. It's like how do you leverage more than word level information from un-unlabeled text? You're gonna have like some, some questions that, questions that you need to answer.

First is like which objectives should you use while training? Do you wanna use language modeling or machine translation or like dos- discourse coherence or something else? And like in, in twenty twenty-four right now, we definitely know the answer, like language modeling works very well, very well because this is what we've, we've been using for the past three years.

But back then in twenty seventeen and twenty eighteen, this was not very clear. Like for example, this paper came out before BERT, and even the, the original machine learning, uh, the original transformer paper, Attention Is All You Need used, uh, machine translation.

So back then, this definitely was not very obvious to people. The second question is how should we transfer these embeddings like to the target task? Uh, there are like bunch of ways to do this. Do you wanna... You...

Like do you wanna modify the model architecture? Uh, very... Like each ta-task is gonna require each specific modification to the, the model itself. This is one approach, and this like requires very deep knowledge of model architecture and like being just a wizard to, uh, modify the architecture.

The second approach is like to use a specific recipe or schema to do transfer learning, and a very popular example of this is ULMFiT, uh, by Jeremy Howard and Sebastian Ruder, and we're gonna cover this in the next two slides, I think.

The third option is also you can add like an auxiliary learning objective during pre-training. So whi-while you're pre-training on language modeling, you can have an au- an auxiliary learning objective like machine translation or discourse coherence. Uh, so these are like some of the approaches you might wanna take when you are deciding on the target task.

So these, these... All these questions made just, uh, going beyond word embedding not, not so straightforward, and they made it like difficult to utilize semi-supervised learning or un- or unsupervised pre-training. So let's take a look at ULMFiT and how they did this.

So this is, this is a, uh, a paper titled "Universal Language Model Fine-Tuning for Text Classification." So their, their objective is text classification, but, uh, do you wanna also build a universal language model? And this is a very seminal work in NLP.

It's a very well-known paper, and it has a big impact. So the question they are trying to answer is, or the question they are li- raising is, and instead of just utilizing the word embeddings, uh, like you're gonna have a classifier, we're gonna have an embedding layer and a classification layer.

And the old way of doing this, you're gonna use the embedding layer from wave2vec, and you are going to randomly initialize the classification head. So they are asking wh-why not just, uh, have a good init-initialization point for all the layers, not just the embedding layer?

And the, uh, their answer to this question is an approach called ULMFiT, and it is a three-step recipe for like state-of-the-art text classification back then. So we have like three steps. First step is like to train the language model on general domain data, and we call this pre-training on lar- on large corpus these days.

So you just train, uh, your language model on a very big corpus like Wiki-Text, and this was big back then, but it's probably small now. You can do pre-training on like fifteen t-trillion tokens of data if you are like a big, uh, organization like Meta, for example, or even more.

Uh, so this is like the first step. The second step is you do fine-tune the language model on your target data, uh, and you, you keep doing language modeling at this step. So this is kind of like similar to what we call continued pre-training on target domain.

So you just take your Llama two seventy B base, and you just do language modeling on, let's say, financial books to try to make your model like Bloomberg GPT, for example. Uh, but you're still doing language modeling. You're not doing any task-specific training.

And the third and final step is to train a classifier on your label data, and this is like the fine-tuning step that we are all familiar with. So if you have... If you wanna l- like let's say you wanna classify Amazon reviews as being positive, neutral, or negative, you're just gonna get maybe a thousand reviews and the, the labels for this, for each review and just train the model on, in a supervised fashion.

And this was a very good paper, and w-was very influential and, uh, made a big, like a big buzz in the, the ML field. But this paper was, was released in twenty eighteen, but it did not mention the word transformer at all.

Like, they did not even reference the paper. The architecture of the model used were, was an RNN-based model. I think it was an LSTM. So no transformers at all. Uh, and this is like, uh, this is kind of a big, big gap in this work that, uh, GPT folks are going to fill.

Otherwise, this paper would have been, like, a very, very good paper. It still is a good paper. So that was ELMo/PET. Uh, so let, let's talk a bit about GPT, uh, the thing that we want to talk about today.

So GPT stands for Generative Pre-trained Transformer, uh, and the key word there is transformer. It was developed by OpenAI, uh, by these awesome folks, and was actually one of the first things that made OpenAI, like, a popular organization in the ML field.

And the whole premise is to use semi-supervised, uh, a, a semi-supervised approach for NLU tasks, uh, natural language understanding tasks. Uh, so the goal is to learn a un-universal representation that transfers v- transfers very well to other, uh, downstream tasks.

Uh, so basically have a good starting point instead of, like, starting from scratch every single time. And their approach is basically two-step. The first step is to do unsupervised pre-training and then supervised fine-tuning. And this is kind of like where the word semi, semi-supervised is, comes from.

Like, you have a bunch, a mixture of unsupervised training and supervised training. And their architecture is transformers, of course, because, uh, we have GPUs, and GPUs love transformers, so that's a good thing. Uh,

so to do this approach, you, you're gonna need two things. The first thing is, like, a very big corpus of unlabeled text, and then the second thing is, like, a dataset that is annotated, that is, like, ready to use for, uh, supervised fine-tuning.

Or you could have multiple datasets if you, you wanna train your model for several tasks.

But the good news is your target tasks do not need to be in the same domain as, like, the unlabeled text. So for example, if, let's say, you wanna train on financial task, like, you're gonna give the model some information about a stock and ask it about how it performs, uh, whether or whether you should buy or sell, so like a, a financial classifier.

Actually, you can pre-train on just, like, uh, normal general data. Like, you can pre-train on a corpus from the internet and then just fine-tune on the, on your desired task. Like, your, your, your, your unlabeled corpus does not need to be in the same domain as the, uh, your objective.

And this is good news because we have a lot of general purpose text that, that we can use for pre-training, while obtaining very domain-specific corpus is more involved. And a sm- a very minor note here is, like, the name can be misleading in this work.

Like, the word generative here mainly refers to, like, the pre-training step. The actual tasks that they had in mind are more discriminative, so like classification, question answering, and semantic similarity. Uh, that is natural language understanding tasks. So they did not work, uh, like they did, they did not discuss machine translation or just being a, a chatbot in this work.

And they actually released their blog post under a different, and I think more fitting title called Improving Language Understanding with Unsupervised Train- uh, Learning. And this is, like, the key, th-this is the key idea here, like unsupervised learning.

And this is a very nice blog post, uh, on their, on their blog.

So, uh, now we have discussed ELMo/PET and GPT, let's, let's also discuss some of the other, uh, related work in this domain. So let's first start with, like, semi-supervised learning. Uh, the work GPT falls under this domain, and back then, this was becoming very popular, like sequence labeling task, uh, text classification tasks.

People were doing semi-supervised learning for these. And you can, uh, you have, like, different levels for this, uh, approach for... So, like, the first and basic level is just using, uh, statistics, language statistics to do, to get the features, like use bag of words, TF-IDF, and all these cool classical machine learning stuff.

You can use them as input features and just train a classifier on top of this. Uh, but that's not very helpful because, like, two words that are different in syntax but are similar in, in semantics are gonna have very different features.

Uh, so it makes it a bit, uh, the model is a bit limited. The second step is to use the word embedding, like we discussed previously. And this a-approach allows you to capture the semantics, but it's also very limited in that it's, uh, based on words.

We-we're not capturing the higher-level semantics. The s-third level is, like, sequence embeddings. So instead of just using the, the word to get the embedding, you are gonna utilize the entire sequence, uh, so like the sentence or the paragraph, get the embedding.

And this actually allows us to utilize the context to understand, like, the high-level semantics of the text. And this is, like, level three is, is wh-where GPT falls. Like, we are using the, the entire sequence to generate embeddings that we can use for classification or any other task.

So this is regarding semi-supervised learning. The second field, and the more specific field, is, like, unsupervised, uh, learning. So it's a special case of semi-supervised learning, and these terminology c- uh, these terminologies can be confusing, I know. But the goal is like to find an, a good initial starting point instead of just, uh, instead of just, uh, going directly to do un- uh, supervised learning objective.

Uh, this is a typo probably. So the early works explored the, uh, uh, the early works in unsupervised learning w- was actually used in, in vision, in image classification. So for example, you can use a ResNet that was trained to classify ImageNet.

You can take the backbone, the ResNet as a backbone, and then do like, uh, segmentation or just try to detect pneumonia in chest X-rays, and this is a very good starting point. Even though ResNet was, was, uh, even though ImageNet is just classify- classifying images as being cats or dogs or, uh, or humans or horses.

Uh, so it's more of a general purpose data set that you, you learn a very good representation that can, uh, be used in your downstream task. And, uh, in the field of vision, this actually proved to be quite well, uh, because people have found out that pre-training acts as a regularization scheme.

Y- your model tends to have better generalization if you pre-train it on a very large corpus. And the authors mentioned in the paper that the closest line of work to their work is actually what we discovered, uh, what we covered so far, the ULMFiT by Howard and Ruder, and also another, uh, work by Dai et al.

Uh, but I did not go into, uh, I did not go, go in detail into this work. However, the, the main, the main drawback of these two works and everything else, almost everything else back then is like they are using RNNs.

And we know that by twenty eighteen, like, Transformer reigns supreme because we have GPUs, uh, so we can train Transformers more efficiently. And also RNNs have like, they are limited in their ability to process large context, uh, because of like the gradient vanishing problems and so on.

So this is like the most common approach back then. Another approach is also to use the hidden representation from a pre-trained language model or machine translation model as auxiliary features while training your model. So you can have, let's say you have a, uh, a machine learning model.

You c- you're gonna get the representations or like the, uh, the hidden state of this, this model and then just, uh, give it to your classifier as like, uh, additional features. And as you can imagine, this is like, this involves like a, a substantial amount of work and new parameters for each task.

Uh, so this is not very universal.

So any questions so far before we actually go in detail into the approach?

Um, I think there was a question from Six. Uh, did AlexNet use GPUs? And were the Transformers-

AlexNet?

-the first ones to use the GPUs? I think he just put it in the chat. Yeah. But I, I did... Yeah. Were Transformers like some of the first models to use GPUs?

Yeah, Alex, AlexNet, yeah. Yeah, yeah, AlexNet did definitely use, uh, GPUs, but back then I think they were writing, uh, CUDA code. I think Alex Krizhevsky was the person doing the GPU programming stuff, uh, himself. Uh, so yeah, I think AlexNet did definitely use GPUs, if I remember correctly.

Yes. Okay. Control code.

Oh, seems like they did. Yeah. Okay. I, I will just surface-

Great. Uh-

-if anyone else has any other questions, but I think we should be good for now.

Yeah. S- someone say according to papers with code, they did, yeah. I think they did use GPUs. So let's, let's go into details about the, the GPT approach. So we have two steps. The first step is unsupervised pre-training.

So basically, the goal is to train a high capacity language model on a large corpus of text. Uh, the training objective here is language modeling. That is, given a sequence of tokens, try to p- uh, correctly predict the next token.

Uh, so l- let's say, "The cat sat on the," you're trying to predict the next token, which, which I think should be, uh, "the mat." So this is basically language modeling. Uh, the loss function they used is negative log likelihood.

Uh, it's also called cross entropy. Uh, basically, uh, this equation, the negative log likelihood of the correct label. And you do this over... A very, very important note here is you do this over the entire sequence. So if you have a sentence that said, "The cat sat on the mat," you do this over every single token in this sentence, and this is very important.

Uh, so you are training on your entire input to- uh, uh, your entire corpus, uh, not just the last, uh, token. So the architecture is like a Transformer, a multi-layer Transformer, but they are using the decoder only. They are not using the encoder.

So... And the difference between the encoder and the decoder, I think can be summarized to, like, how you do attention. So in enc- in encoders, uh, every token has access to every other token, but in decoders, every token has access only to the tokens that came before it.

So you only have one directional attention. I think it's also called, uh, left to right attention or right to left, or like you only have attention in one direction.

And basically, the a- the Transformer ar- architecture applies like, uh, multi-headed, masked, itself attention operation over the input tokens, and, and then this is followed by a position-wise feed forward layer. And you keep doing this for like N, where N is the number of Transformer layers, and then you just, uh, generate an output distribution over the vocabulary.

So let's take this into, uh, more detail. You have your input text that has been tokenized, uh, into tokens, and these tokens have been encoded. So you have like the token integer or token ID, so it's a number.

So you, you take this number and you pass it through the embedding layer, the token embedding layer, also called the, the semantic embedding layer, and you get an, a vector for this, uh, token, and you al- you also get the positional embedding for this token.

You, you sum these up, uh, by just vector addition, and you get the, your input to the, to the first transformer block, which is H zero. So just the positional, uh, uh, the positional embedding and the token embedding added together, and you get your input H, H zero.

And then you take H zero and, and pass it to each block, each transformer block, and the output of the first block is gonna be the input to the n- the second block and so on. And this is what this equation says.

Uh, and at the end, once you've done going through all these blocks, you're gonna go to the, the output layer, which is actually a reverse embedding. So you have a vector, but you want to go back to a token or, or like, I should say, a probability distributions of, over all the tokens.

And once you have this distribution, you just pass it through a softmax to get, uh, actual probabilities that sum up to one instead of just logits or scores. And I think, uh, we've covered the transformer paper, so I think people are fa-familiar with this, but if you have other questions, please go ahead.

A small remark about, about this is that we use right input and output token embedding, so the embedding layer in this step is the same as the one used at the final step. Um. Yeah. Uh, someone says causal masking.

Yes, exactly. Causal language modeling. So the second step is supervised fine-tuning. So the goal of this, uh, this step is to adapt the parameters of the pre-trained model to your supervised target task. And for this, we need a label data set where each instance is a pair of like you have a sequence of input tokens and a label.

So for example, uh, an input, uh, sequence could be, "This product is very bad," and your label could be like a negative sentiment. Uh, so the inputs are passed through the pre-trained model to obtain the final transformer blocks activation.

So if you go back a bit, you take your input sequence and pass it to-- through this same transformer and get the output of the final token, and then you compare this to your label. Uh, so you're gonna, uh, get the hidden representation of the last encoder layer at the M token, where M is like the final token in the input sequence.

And you just pass this through your classification head or whatever head you're using. So for example, in classification, we're gonna use softmax, uh, on top of a linear layer to get your output and compare it with, with the label.

Uh, and you are using, uh, roughly the same loss function, which is negative log likelihood estimation or cross-entropy.

And, uh, a, a key distinction here between this and the previous step is you are only calculating the loss over only the output token, not the entire sequence. So the loss is only on Y, not on X one or, or XM and so on.

So y- the only extra parameters you need for this is, is the, your classification head. So for example, WI in, in, if you're trying to do classification, the, uh, the parameter metrics of the, the metrics of the parameters of the output layer, and also embeddings if you are adding new tokens, and we're gonna see this in a bit.

Uh, something they also experimented with is auxiliary training objective. So they also use language modeling as an auxiliary objective in fine-tuning. So not just this, uh, not just this classification, but also language modeling. And they say this helped them, uh, by improving the generalization of the supervised model and accelerating convergence.

They also th-say this is in line with prior work, who also observed better performance, uh, when u- when using int- uh, auxiliary objective. And the way you do this is your loss function is now a sum of multiple loss functions, where one loss function is this one, the classification loss function, and also you have the language modeling loss function, uh, with a certain weight, like lambda here is like a weight.

And lambda could be, for example, z- zero point five or zero point three. So you have like, uh, a summation of multiple losses. And a, a small note from myself here. Uh, I'm not sure if auxiliary langu- like auxiliary objectives are popular today.

I think people just do, uh, supervised fine-tuning without an auxiliary objective. That's just my take.

Uh, so any questions so far?

Mm.

I think the, I think the chat seems to not have any questions. So maybe you wanna just, we can just continue for now. Um.

Sure.

Yeah.

So now we have discovered, uh, we have covered the approach, the basic two steps. You will now get a very good, let's say, classifier because you have done unsupervised pre-training and supervised fine-tuning on classifiers. But GPT-1 is actually trying to be more, it's trying to be a more of a universal model than just a classifier.

So they are trying to handle multiple tasks like classification, entailment, semantic similarity, and answering multiple choice questions. And these are all, as you can see, discriminative tasks rather than generative tasks as we discussed before. So for tasks like classification, this is very easy.

You can do what we have covered so far, just add the head on top and do the classification. But other tasks have, like, different structured ou- inputs and outputs. So for example, text entailment has ordered sentence pairs. MCQs have a question with multiple answers, uh, and so on.

So each task, each task has its own specific structure. And the way people have dealt with this previously is, like, just learn a specific, uh, uh, architecture for each task on top of your, uh, model. And this, this, like, this defeats, this defeats the whole purpose of the GPT work.

Like, we're trying to do something that's global, general, and a general purpose model rather than having multiple task-specific ta- architectures. So instead of using this approach, they opted to use a different approach where we con- they convert the, the structured inputs into, like, just tokens.

Uh, so they are trying to create a multitask format, and this is similar to, like, what people have used in feature work like T5 and Whisper, where basically you're trying to, uh, model different tasks just using tokens and special tokens.

These input transformations allows us to use, like, the same architecture for different tasks. So you, you don't need to do a lot of modification. Uh, and we're gonna go into this into details in the next slide. So for example, let's take textual entailment.

This task, uh, involves reading a pair of sentences and judging the relationship between them. So the relationship could be one of entailment, contradiction, or neutral. Uh, and a small note is that this task is, is still st- challenging because your-- you need-- your model needs to have good understanding and reasoning of the language, uh, because it can be confusing sometimes.

So you have your premise, and you have your hypothesis, and you're trying to classify or try to predict the relationship as being entailment, contradiction, or neutral. So the way to do this is just to concatenate the premise and the hypothesis.

So this could be a sentence, and this could be a sentence. You just concatenate them and add a special delimiter token in between them. And obviously, you add your start token at the beginning and the, your end token at the end, and just try to train a classifier on top of this.

And your classifier should classify this, uh, this sequence of input to- input tokens as one of entailment, contradiction, or neutral. So this has become just a classification task, uh, by just doing transformation. Uh, the second ta-task they cover is semantic similarity, and I think this is very popular nowadays because of retrieval, augmented generation, generation, and embe- and embeddings in general.

So all this cool RAG stuff. Uh, so this task is about predicting how semantically similar two sentences are, and semantic similarity just means how close in meaning they are. Do they talk about the same thing? Do they mean, do they mean the same thing?

Do they have similar meaning or not? Uh, and again, this can be challenging because you can have, uh, two paragraphs that have very different usage of words, but they convey the same meaning. So this can be challenging if your model is not smart enough.

And one note about this task is there is no inherent ordering of the sentences, so you can just, uh, there is no sentence A, sentence B. You just have two sentences. Unlike, for example, in the entailment, you have a very specific order.

Like, you have a distinction between the premise and the hypothesis, but here you have just two random sentences. And the way they approached this is using a, a Siamese architecture, where we have-- where we generate two input to, uh, sequences and pass them through, through the transformer and compare between them.

So basically, Siamese architecture is a fancy way of, of saying that we are using the same model twice or two identical versions of the model. So you get your sentence, first sentence and second sentence, and then concatenate them and add your special tokens, and you pass them through the transformer, and you get some vector at the end.

And also, you do the same thing, but you reverse the order of that sentences. Uh, so you get the second input sequence, pass it to the transformer, and you, you get your vector. So basically, at the end, you're gonna have two vectors.

You just add them, do vector addition on top of them, and then you just add a, a head on top of the, the output vector. Uh, so you just multiply the vector by n- by some, uh, layer, a linear layer, for example, that has parameters W, and you get your output.

And for example, if you're doing, uh, like if you s- just have-- if you're just interested in knowing wh-whether these sentences are similar or not similar, so you have only two labels, you can train a classifier. If maybe you're interested in having, like, more of a s- or, or like a scale of similarity, like from zero to ten, you can train a regressor on top of this.

So that's very cool. Uh, we're still using the same architecture, by the way, just the transformer here. We're not modifying it. We're just, uh, being smart about how to approach this, uh, task. Uh, you can extend this, uh, to do question answering.

So let's say you have a question, and you have four choices. Or let's say you have a document, you have a question about the document, uh, and you have four potential answers to this document, and you want your model to pick one answer of these.

So the way to do this is you take your document or context and add the, the question and then add the, uh, the first answer, and then you get your first input sequence, and you pass this to the transformer.

And similarly, you get your context or document and add the question and add the second answer and make this into a sequence and give it to the transformer. And you do this for all the answers And then you just compare between, uh, the scores the model gives to each of these, uh, potential answers.

So for example, we have a, a Wikipedia article, and we have a question, and we have answer A. We concatenate all of these with some special tokens, and we give the-- we pass them to the transformer, uh, and to the final linear layer, and we get a score.

So score for answer A and score for answer B and score for answer C, and just do a softmax to get a proper probability distribution. And that's it. You've got your model to do question answering. And you can do this also for other tasks like commonsense reasoning.

So any questions about this, uh, this like str-- these transformations for each task?

Uh, I think there was a question about modifying the input sequence to have both possible orderings, at least when it comes to semantic similarity. Um, I think someone had a question about that. Uh,

so if-

Sure, uh-

So he just said that he doesn't really understand what it means to modify the input sequence with both possible orderings.

Yeah, sure. So if you're doing textual entailment, you have very s-- like you have distinction between the premise and the hypothesis. Like you have a prem-premise and the hypothesis. Uh, so you cannot just mix them up. You, you have a very, uh, specific premise and a very specific hypothesis.

And the ordering here matters, like you should put the premise first and then the special token and then the hypothesis. You cannot put the hypothesis first and then the delimiter and then the premise. So this is what they mean by the ordering here matters.

But for semantic similarity, you just have two, two sentences. You can... There is no inherent, uh, ordering for the semantic similarity task. Uh, I hope this answers the question.

Uh, I think it, uh... He said, "Ah, I see. Thanks." So I think, uh, probably that's about it. We could... I don't see any other questions. Oh, he said that does it also combat things like positional biases in transformers by ordering, by switching the order of the sentences itself?

Any value to implementing-

Yeah, yeah, exactly. I think, yeah. I think this is like one of the motivations they did this. Like they wanna say there is no inherent order for this task, so maybe the transformer will just have a bias, so let's try both ways.

Like let's give it the first sentence and then the second, and let's give it the other way around, uh, to like get over the, the positional bias. Because I think for some models, uh, they, they pay more attention to the last few tokens in the input, and they disregard the earlier tokens.

This, this tends to happen sometimes, yes.

I think, yeah, I think this, this, uh, the good way to think about this.

Awesome. I, I think, uh, I don't see any other questions in the chat, so I can just surface them as and when they come. So I think we should be good for now by the looks of it.

Sure. Sure. So that was like the whole approach. We can now discuss some details about the training and back then OpenAI actually did release info about their training and models. They don't do this now, unfortunately. Uh, but anyways, so the data set they use for training is BookCorpus for training the language model.

So this is in step one, which is unsupervised training, BookCorpus. And back then, this was huge. It has seven thousand unique unpublished books from different genres. So like the variety here helps as well. They have adventure, fantasy, and romance.

So kind of like a very good, big, diverse data set. And, and the main, the main advantage of this data set and the reason, reason they chose it is because it has long stretches of text. So if you have a, a book or a novel, you have like a paragraph that's maybe ten lines or more, and this helps the model to learn how to handle long, long-range information and how to handle long context.

For example, there is also another data set that's called WordBenchmark, which is also big and diverse, but it doesn't have, uh, this long context. It's just a bunch of small sentences, I would say. Uh, uh, and this way y-y-your model not learn how to handle long context.

And, uh, a side note here is like ELMo is also a very seminal, seminal work in NLP, and it falls under the same domain of like unsupervised pre-training and having good embeddings. So it's one of the, uh, the, the fundamental papers and works in NLP.

So they say their model achieves a very low token-level perplexly, uh, perplexity of eighteen point four on this corpus. But I don't think this is actually low today. Like I think perplexity of eighteen is a bit high, uh, but I'm not sure.

And their model is about, uh, th-their data set is about five gigabyte in size, so not quite big by today's standard, of course. Uh, their, their architecture, as we mentioned, they have a transformer model. They use a tokenizer.

They use byte-pair encoding, uh, back then, which is also I think what's used right now. Uh, so this is still not changed from today. They have a surprisingly big vocab size by their time's standard, like they have forty thousand tokens.

I think Llama 2 had also something that's in the same range, like forty thousand, fifty thousand tokens. So back then, this was actually quite big, I would say. And they use the ftfy library to clean the raw text and then do some standardization using spaCy tokenizer.

Uh, so this is good work, uh, in the tokenization area, I would say. And their model is just, uh, a typical transformer model, like typical by today's standard, just a transformer, decoder-only transformer with like twelve layers and masked self-attention.

And it has A very big size of one hundred and seventeen millions, and this was actually big back then, although this is trivial now, of course. Uh, their embedding, they used learned positional embeddings compared to the sinsoid- sinusoidal embedding in the original transformer paper.

So this was actually much simpler to implement, and the model just has to learn the, the embeddings in training. They have a context size of five hundred and twelve. Again, back then th-that was very big, and they also used tied input and output token embeddings, as we mentioned.

And for the attention block, they have twelve attention heads. Each head, each head has sixty-four, uh, each head has a dimension of sixty-four for a total of seven hundred and sixty-eight, uh, dimension for the, for the entire attention block.

And after the attention, you have the MLP layer, also known as position-wise feedforward network, and the size of the, the inner state of this network is three thousand and seventy-two, uh, the size three thousand and seventy-two. And this means that actually this is a expansion and then contraction, uh, network.

So you go from seven sixty-eight to three K, and then you go back from three K to seven sixty-eight, and all this is happening inside this MLP. And the activation layer they used is, is the GeLU activation layer.

Uh, the optimizer, the optimizer, they used Adam, which was becoming very popular back then. I think Adam was, was developed in twenty fourteen, twenty fifteen, so it was getting a lot of traction back then. And the maximum learning rate is this learning rate, which is also very popular nowadays.

I think this was used in one of the Llama Two models. So this is very familiar. The warm-up, they do warm-up as well from zero to maximum to the maximum learning rate over two K step, two K steps, and then they just, uh, cosine annealing to the, uh, they do cosine annealing from the maximum learning rate to zero.

So just, uh, learning rate decay. Their compute, uh, they used, um, uh, a, um, one machine that has eight X P600 GPU. Uh, this is, uh, the same family of GPUs as V100, I think. But, uh, I don't have much information about this GPU.

They trained for thirty days, uh, which actually is not bad. That's, uh, that's not very long. But back then, this was very long, I think, I think. Their utilization, they mentioned, is zero point thirty-three, and the total FLOPS is zero point six petaflop, so almost one petaflop.

And for a reference, I think the tiny grad, uh, machine is trying to give you one pe- one petaflop. Uh, or I could be wrong. Uh, I could be wrong about this. So anyway, the, the compute they used is almost one petaflop, uh, days, and this is like the, the way they calculated this.

And this is actually how the model looks like if you try to load in the transformers framework. So you have like token embedding, position embedding, and then some dropouts, and then you, you have your actual transformer blocks. And this is like the, the, the, the architecture of the language model.

So there is no head in here. So the, the block is just attention and then layer norm, and then MLP, and then layer, layer norm.

Uh, so there are no-- if there are no questions, we can move on to the second step.

Uh, I think there was a question-

Which is supervised fine-tuning.

Oh, sorry. There was, uh, one question-

Sure.

-about from Sean, which is, uh, are perplexity numbers comparable across different models? And I think we, we discussed just now that the model itself has a perplexity of eighteen point four. And so in this case, uh, would it be okay to compare?

Like is it, is it an, is it a, a number that's a metric that's invariant across different models in this sense?

Yeah. So that's a good question. I think perplexity is just a metric. Like you're gonna measure perplexity on a certain dataset. Like let's say you have this dataset that is one trillion tokens or one billion tokens, and you measure perplexity of GPT one on this dataset, and you can measure the perplexity of Llama Two on this dataset.

And I think it's-- you can compare this, uh, this metric. It's like saying, uh, Llama Two has, uh, a human eval of seventy and GPT four has a human eval of maybe ninety. It's not totally fair, but we can do it, I guess.

We can get away with doing it. Yeah, but it might not be a hundred percent fair.

Uh, yeah. So that's, uh, that's a good point.

Okay, cool. I, I think, um, that's probably the only question. So I think we could probably just move on to supervised fine-tuning. Yeah.

Sure. So the second step is supervised fine-tuning, and they use these datasets for this task. So for sentence similarity, they use these datasets, and for classification, z- they used GLUE, which I think is popular now as well. Uh, I'm not gonna go into details about this because, uh, I don't have much information about these.

And the architecture is just use the same backbone as pre-training, and you add your head, mostly classifier head with a dropout of zero point one. They train for three epochs with a batch size of thirty-two. This is also still, uh, standard nowadays as well.

Uh, people usually train, do fine-tuning for three epochs with a batch size of maybe thirty-two or sixty-four. So yeah, that's pop-- that's, uh, common nowadays as well. The learning rate is six point two five E to the negative five, and also people use very similar learning, learning rates nowadays as well.

So this is very familiar. This architecture for fi-fine-tuning is very familiar even today. And they also use, uh, learning rate decay with a warm-up. And when they use the auxiliary language, uh, the auxiliary objective, they used a lambda of zero point five.

So the weight of the auxiliary language modeling objective was zero point five. But, uh, this is not popular nowadays. Most people don't do this.

Uh, any question about this?

I think it should be okay.

Before we go to the benchmarks?

Yeah, the chat. There's no questions in the chat, so I think maybe we can just move on to the benchmarks.

Sure. So the benchmark is like they do a lot of SOTA performance on many tasks, like you can see here almost every single task except this one. And there are ones where they have significant improvement like this, uh, in QNLI.

They have like six percent absolute im-improvement, and I think... Let's go back a bit. QNLI, uh, where is it? In here.

Uh, I think the QNLI is, is more of a, uh, natural language understanding where it requires reasoning, and this is where they make the most improvement, I think. But overall, like they are, they are like doing a very good performance on many tasks, and they are comparing to LSTMs and other models.

So good news, GPT-1 tends to be a good model. Uh, yeah, this is for natural language inference. For question answering and common sense, they also make very big gains, as you can see here. So seventy-six compared to eighty-six.

Uh, they compare also to multiple models, and one of them is actually an ensemble of nine models as they denote here by 9x. And they also, also the same happens for semantic similarity and classification, although the-- I would say the, the boost in, in performance is not as, as big as the previous ones.

Uh,

some-- they are good on some, uh, some metrics and bad on other metrics.

Uh, so this is my favorite part of the paper. They did-- After the benchmarks, they now have a good model, so they are tr- they try- they are trying to understand why their model is good and why their GPT-1 is, is suddenly SOTA.

So they do analysis and try to understand why this is happening. So the first step is try to-- they trying to analyze the impact of the transfer learning, the number of layers transferred from the pre-trained model to the fine-tuned model.

So they just take the pre-trained transformer, and they take all the layers and do fine-tuning, and then they take eleven layers, and then they do fine-tuning, and then they take ten layers and do fine-tuning and so on. Uh, they compare the performance of these models.

And as you can see, the more layers you add, the more performance you get. And if you do zero layers transfer, you, you're not taking any of the pre-trained weights. You're just starting from scratch. Uh, you get this performance.

But if you just add one layer, you get a very significant boost. Uh, the solid lines here, lines here are the dev datasets. The dashed line is the training dataset. So you can see that a very big improvement in performance just by adding one layer, and this is almost like a continuous trend.

So, uh, we-- they observe the obvious fact that transferring more layers improves performance, and just only transferring the embedding layers al-also improves performance significantly. And they mention that each layer adds a boost of nine percent for, for, for each layer you add on this task, MultiNLI.

And this, this analysis actually indicates that each layer in the pre-trained model actually has a purpose, and it's, it learns something that's useful, and it, it's very helpful in, in the fine-tuned model. So different layers le-learn different things.

They learn different scales, and they are all helpful. Uh, this is a very good finding of this work. So each layer in the pre-trained model contains useful information on functionality for solving for target tasks like classification.

Uh, the second piece of analysis they did is zero-shot evaluation. And this was, uh, I would say radical back then. Like, they are trying to evaluate how good the pre-trained model is on, on, on these tasks without even fine-tuning.

Uh, and they want to answer the questions like why is the language model pre-training effective? Wh-why does pre-training a language model on-- why does pre-training a transformer on language modeling help us when we are doing classification? Um, they have a hypothesis is that the underlying generative model learns actually multiple tasks in, in the pre-training.

So it's not just learning language modeling. Because if, if you're, if you think about it, if you're learning how to lang- model language properly, you, you, you, you're more likely to under- have a, like a deeper understanding of the, the language and just la- natural language.

Like if you're, for example, if you, if you speak only English and French, uh, and you, you cannot speak, let's say, Spanish, you won't be able to, uh, do classifications in Spanish. But if you speak English and you speak French, you probably, uh, have the understanding to do these tasks.

So this is, uh, something cool and something to think about a bit. Another thing is like attention is very helpful. Like transformers, uh, show very big improvement compared to LSTMs, or that's what they are hypothesizing about and what they are trying to test.

So to test these two hypotheses, they designed a series of heuristics. Uh, they basically tried to evaluate the pre-trained model on these tasks before doing the supervised fine-tuning. And they have different modifications for each, uh, dataset and each task.

For example, for linguistic acceptability, for the CoLA, they, they used the examples, and they take the average token log probability for e- for each token and use this as a score, and they have a threshold. And they just determine if the model, if the model got this right or wrong.

Uh, something that's, that's very cool is how they handle sentiment analysis. So you get your, let's say, Amazon review, and you append the token very to the review and restrict the language to generate only two tokens. One of two tokens, positive or negative.

And you, you see which token has higher score or higher probability, and this is the prediction of the model on this, uh, sentiment or this paragraph. So just restrict the language model prediction to these two to- these two tokens and see which one is higher.

Uh, this is actually very cool. It's kind of like, uh, similar to constrained grammar and constrained output. Another example is, is, uh, question answering. So for each answer, y- as you said, concatenate the document question and answer in the input, and you average the token log probability of just the answer, and use this as the score for this answer, and do this for multiple answers and just pick the one with the highest score.

Uh, and for, for Winograd schemas, uh, for, for this task, you have, let's say you have an, uh, a sentence that says, "The city councilman refused the demonstrators a permit because they advocated violence." So the goal h- here is to find this word they.

What does it refer to? Does it refer to the demonstrators or the city councilman? Uh, this is what they mean by Winograd schema, and I think it's a very popular task. So the way they do this is they take this word they and replace it with the two possible answers.

In this case, councilman and demonstrators. So you have, uh, uh, example A, you use the word councilman, and example B, you use the word demonstrators, and just, uh, do just average the token probability of, of what comes after this word.

And then you just take the one with the higher probability score.

Uh, so this is like the how they wanted to evaluate the model. Let's see actually the evaluation results. And surprise, surprise, the, the more-- the longer you train the model and the more data you give the model, the more it tends to perform, the m- the more it tends to perform on this task.

So the better zero-shot performance it has, basically. Uh, this is a very cool graph. So we have different tasks here, and the solid line is the GPT transformer, and you can see with more steps, more training steps, the better performance you have.

And they also did a very cool analysis where they trained an LSTM, which is the dashed line. And you can see that for almost every single task, the transformer is better than the LSTM.

So they observe the obvious that more training gives better performance, more as- more pre-training gives better performance even in zero-shot settings. Uh, and this suggests that gener- generative pre-training, uh, actually... In generative pre-training, the model learns a lot of tasks, not just language modeling.

And they mentioned something here. They said they observed that the LSTM exhibits higher variance in its zero-shot performance, suggesting that the inclusive bits of the transformer assists in transfer. But I'm not sure what they mean by high variance in this graph, actually.

Uh, so if anyone has any clue about this, please share it with us.

Yeah, it's, it's better than LSTM one, and GPUs are all we need.

Exactly. So the final, uh, like, the final section in the paper is the ablation studies they did. Uh, they have three ablation studies. They want to first see the effect of the auxiliary language modeling objective during fine-tuning. Uh, so in this case, we have the, the, the normal transformer training w- with auxiliary language modeling, which is row one, and we have the one without auxiliary language modeling, which is row three.

Uh, they say that the trend suggests that, uh, larger datasets benefit from auxiliary training compared to smaller datasets. And this is kind of like actually why, probably why people stop doing auxiliary language modeling. It doesn't seem to be quite, uh, important, uh, probably.

The second ablation study is the effect of the transformer. So they u- they just train the usu- usual transformer, and they compare it with an L- LSTM, so row one and row four. Uh, and you can see, yeah, the transformer has generally better performance than the LSTM on most tasks.

So they mentioned that they observe a five point, five point six average drop when they use the LSTM.

And it only outperforms the transformer in, on one dataset, which is MRCP, MRPC. Uh, but I'm not sure what this is. The second ablation study, and, uh, I would say the most important one is the effect of pre-training.

So they compare the transformer that has been trained in what the, the framework they proposed, the two-step framework, and they, they also compare it to a transformer that was directly trained on the supervised task. That is without any pre-training.

So we have row one and row two. You can see that this is where the huge difference shows up, like seventy-four compared to fifty-nine, fift- forty-five compared to eighteen, eighty-eight compared to seventy-one. So this is like a very big difference.

Uh, yeah, and we observed that no pre-training actually hurts the performance quite a lot on almost all tasks. Uh, and this results in f- f- fifteen p- percent decrease, and this is actually the worst-performing model out of all these models.

Uh, yeah. So I would say that the conclusion from this ablation study is that pre-training is very, very important.

I think this is the final section before the future studies. Do you wanna discuss? Any questions so far?

Mm.

I think chat seems good. Do you wanna... You have one more slide, I think. You said you-- future studies, right?

Yeah.

Do you wanna just finish it up, then we can open up the questions if people have any?

Sure.

Yeah.

Sure. So this section, I don't think it was in the paper. I think it was only on the blog post. Uh, they discuss what future work could be, and the first approach is surprise, surprise, s- just scale up.

So they mentioned that they noticed improvement in language modeling, and it correlates with downstream tasks, and they're only using very limited hardware, only eight GPUs on a machine and training on a very small dataset. So maybe there is room for improvement if you scale the model, the training, and the dataset, and I think we know the answer to this question, uh, or the answer to this, uh, hypothesis.

Uh, the second approach is, uh, the second futuristic approach they mi- they want to try is like, uh, try to see if there is like you can tweak the fine-tuning approach instead of just doing vanilla fine-tuning. Maybe you can use adaptation or any of the cool...

one of the other fancy ways of doing fine-tuning. And I don't think this, this... I don't think this is as important as they might have thought about this because people now- right now are doing just simple fine-tuning, and it works.

So yeah, not quite as, as promising as the first approach, which is just scaling up. And the third one is understanding of why generative pre, pre-training helps. Uh, and they've done some ablations and analysis about this. They want to do even more further targeted experiments and research to understand this.

So basically, observability and explainability in machine learning.

Uh, and one very good question they ask is like, how much does longer context help, like, compared to just improved world knowledge when you are doing pre-training? So for example, this GPT model is able to process a longer context because it's a transformer.

So is this the thing that makes it sort of the performance, or is it the fact that they train on a bigger dataset with a longer training time, and it obtains world knowledge? And I think both are, both are important, I would say.

But that's a very good question. Uh, so yeah, that was, that was it in the GPT. They introduced a framework for, like, achieving sort of performance by doing two-step approach, doing pre-training and then fine-tuning. Uh, pre-training, the goal of pre-training is to, is to, like, to obtain a very good world knowledge and a very good starting point, and you do the pre-training on a diverse corpus with long stretches of text, so the model acquires actually world knowledge.

And then you actually do transfer learning by fine-tuning on, like, tasks, like question answering and so on. And the results that they improve the state-of-the-art performance on nine datasets out of 12, and they used, uh, they successfully have utilized the unsupervised approach to do transfer learning to discriminative tasks.

So now we have a clear way of how to do, uh, unsupervised training and semi-supervised training.

And the work also highlights that transformers is very, is a very good architecture, and larger datasets are good. And this is a very important push in the direction of, of scaling up and pre-training. And this is actually what people are going to do for the next, uh, six years from twenty eighteen to twenty twenty-four.

Yeah, so very good paper. Very good work. And with that, we'll- we've come to the end of this paper, so if you have any questions, I think we can take questions now.