# llm.c's Origin and the Future of LLM Compilers - Andrej Karpathy at CUDA MODE Latent Space · 2024-09-21 Andrej Karpathy explains llm.c, his from-scratch C implementation of GPT-2 training that outperforms PyTorch by 20% in speed and 30% in memory, arguing that LLMs will eventually act as compilers for custom code. He recounts starting the project out of frustration with PyTorch compile errors, writing all forward and backward passes manually in C, then porting to CUDA with kernel optimizations. A community of contributors (Eric, Arun, Alex R) helped achieve 50% MFU on a single H100 node, training GPT-2 1.6B in 24 hours for $600. Ongoing work adds Llama 3.1 and FP8 support. Karpathy sees llm.c as proof that LLMs can automate such low-level optimization, making Python and PyTorch a temporary crutch. ## Questions this episode answers ### What motivated Andrej Karpathy to start the llm.c project? Andrej Karpathy was frustrated when Torch compile errors prevented him from running evaluation and inference on a small GPT model. After two hours of fighting, he felt powerless and decided to take control by writing the entire training code in C. He wanted a simpler, dependency-free implementation that he could fully understand and trust, leading to the creation of llm.c. [0:03](https://addtry.com/2db337f6-8faa-4b3c-9fde-0b36c49288e3?t=3350) ### What are the performance benefits of llm.c compared to PyTorch for training GPT-2? When training GPT-2 1.6B, llm.c used 30% less memory and was 20% faster than the equivalent PyTorch implementation, as reported by Karpathy. On a single node of H100s, it trains the model in about 24 hours costing roughly $600. This was achieved through manual optimizations like custom CUDA kernels, mixed precision, and zero/1 optimizer sharding, without PyTorch overhead. [0:19](https://addtry.com/2db337f6-8faa-4b3c-9fde-0b36c49288e3?t=19190) ### How does llm.c relate to the future of AI-assisted code generation? Karpathy sees llm.c as proof that hand-written C and CUDA can outperform PyTorch for a specific task. He argues that as LLMs improve at coding, they could act as compilers that generate custom, optimized binaries for any application, using projects like llm.c as few-shot examples. This could make high-level frameworks obsolete, removing the human bottleneck of limited knowledge and attention. [0:21](https://addtry.com/2db337f6-8faa-4b3c-9fde-0b36c49288e3?t=21280) ## Key moments - **[0:00] Intro** - **[1:46] The Grief** - **[3:47] Pure C** - [3:47] Karpathy decides to rewrite GPT entirely in C after Torch compile errors block his evaluation and inference. - [7:00] llm.c rejects tensor abstractions, using only float arrays and pre-planned memory allocation to keep code simple. - [9:48] "You're not gonna ship PyTorch code on a Lunar probe. But I think llm.c is great." - **[10:32] CUDA Kernels** - [11:52] Karpathy finds CUDA learning unexpectedly hard: textbooks are too basic, official guide is unreadable, and only blog posts help. - [13:02] llm.c attracts a team of volunteer 'Avengers'—Eric, Arun, Alex R—who write and optimize the CUDA kernels Karpathy struggled with. - **[13:50] Avengers Assemble** - [15:44] Karpathy 'nukes from orbit' all CUDA stream overlap code in llm.c after it causes race conditions and complexity. - **[17:52] GPT-2 Training** - [18:05] llm.c trains GPT-2 1.6B on a single H100 node in 24 hours for ~$600, reaching near 50% model FLOPs utilization. - [19:19] llm.c uses 30% less memory and trains 20% faster than Karpathy's best PyTorch equivalent for GPT-2 training. - **[20:19] In Progress** - **[21:26] LLM Compiler** - [21:51] Karpathy predicts LLMs will evolve into automatic custom compilers, outputting optimized CUDA binaries for any application. ## Speakers - **Andrej Karpathy** (guest) ## Topics Compute, Open Source Tools ## Mentioned AMD (company), Lilab (company), Zephyrtropic (company), CUDA (product), GPT-2 (product), H100 (product), Llama (product), NCCL (product), PyTorch (product), cuBLAS (product), cuDNN (product), llm.c (product) ## Transcript ### Intro **Host** [0:00] To introduce to you a legendary person, someone who has been hacking and educating at the forefront of AI for over a decade. From neural networks to computer vision, from natural language processing to reinforcement learning, he has pushed the boundaries and inspired millions all over the world, including, I think, all of us here. He is a distinguished machine learning superstar, a founding member of OpenAI, the reference human for ImageNet. An ex-Google Brain, ex-DeepMind, ex-Tesla, Mr. Autopilot. He's really seen it all. And some months ago, on a memorable day, this special person joined the CUDA Mode Discord to start hacking with others on llm.c, which became one of the greatest and most active community projects on our server. But I guess it's best if he, if he tells the story himself. So please join me in welcoming the incredible, one and only, Andrej Karpathy. **Andrej Karpathy** [1:19] Wow, okay. Uh, okay, yeah, I'm very excited to be here. This is my favorite kind of event to present at. So, um, yeah, thank you for the invitation, and thank you for running CUDA Mode and putting this on. This is a wonderful event. Okay, so I'll tell you a bit about llm.c. Uh, so what are we doing? We're training transformers, uh, in C and a pinch of C++. Uh, how do I go next here? **Host** [1:44] Mm-hmm. ### The Grief **Andrej Karpathy** [1:46] Okay, so I'd like to tell a story a little bit of how this project came about and what this looked like from my perspective. So roughly a year ago, I was trying to add a video to my YouTube series, and I was trying to teach people, um, LLM training, GPT training, and so on. And I was basically hacking on that GPT, trying to get it to work. So, uh, that was me. And then, um, you've, you've all worked with PyTorch, of course, right? So the trickiness comes that, okay, you have your model, which you've written. That makes sense. But now you have to keep track of a number of abstractions here at the same time. So you have to put it to a device. You wanna compile it. You wanna wrap it in DDP. And, uh, suddenly things start to be a little bit more complicated because I'm not even sure, like, in what order do you do these? What exactly happens? What are these abstractions? What do they do to your model? So I don't fully understand how any of this works. And then what happens is, um, you wanna use your model in different ways. So you wanna use it in evaluation, in training or model inference, and so on. And, uh, what happens to me is that I was able to train the model, but for some reason, eval and inference was not working. And what happened was I was getting some kind of a Torch compile error when I was trying to, uh, run my eval and my inference. And this is just an illustrated example of a Torch compile error. It was something else. I don't remember. I didn't capture it. Uh, but both of them were giving me error, inference and eval, and a different error, and I had no idea what was going on. So I did what anyone would do in, uh, my position. I went to Torch Discuss. I'm looking for PTR VLCK to solve my issue. Uh, unfortunately, PTR VLCK did not have any guidance that I could see on that specific error. So I was kind of stuck, honestly. So, uh, two hours later of fighting with Torch compile and trying to figure out what the hell was going on, I'm kind of a sad panda. I don't know exactly how to solve this. And so I felt like I was going through these steps of grief. In the beginning, I was in denial. I was like, "This can't be happening to me. I'm not doing anything crazy. I'm just training a little GPT. Like, why is this not working? This seems really simple. I'm not doing anything crazy." And then eventually I entered the state of anger. And I was like, "Okay. You know what? I'm just gonna write the whole thing." ### Pure C **Andrej Karpathy** [3:47] I understand in my mind that what I'm trying to do, like the computation itself, the algorithm itself, is totally clear in my mind. Uh, and for some reason, Torch compile doesn't let me, like, use it, run it, et cetera. So I felt a little bit powerless, and I was like, "Okay, I'm gonna take life into my own hands and, uh, be in control of my destiny. I'm gonna just write this in C. How bad could it be?" So, uh, let's think about, like, what is PyTorch offering you really? And there's many things, but maybe some of the things that are relevant here. I don't know why these bullet points are... Um, I don't know what-- On my slides, there's too many points. I don't know what really happened here. Uh, okay. But number one, we're getting an array, right? So a very useful N-dimensional array that we can manipulate with operations. If we're gonna abandon this, then we're gonna have to do a lot of, uh, pointer arithmetic, basically, making sure that we rattle and unravel indices correctly. Uh, second, we're getting autograd for free. So if we don't have autograd, we need to do forward and backward passes of all the layers. Uh, we don't have devices, so we have to worry about memory being on the host or on the device and shuttling memory around your different, uh, devices between CPU and GPU and so on. We don't have simple DType conversions, so we have to be very mindful what tensors are stored in what precisions and convert explicitly between them. We don't have Torch compile, so we're gonna have to do all the kernel fusions that we want, uh, manually, and we're gonna have to optimize for space, time, performance manually. Uh, and finally, we don't have distributed, so we're gonna have to manually spin up all of our processes, make sure that they can find each other, communicate with NCCL, et cetera. So PyTorch is really, really nice, and this is just some of the things that PyTorch offers. So without PyTorch, we're kind of naked in the world, right? But maybe it's okay. So, uh, yeah, how could-- how bad could it be? So step one, we have our PyTorch code, which now isn't the primary thing we're working with. It's only a reference that we check correctness with respect to. And so we're in PyTorch land. Everything is nice and clean. We have a little transformer, a few modules, and we're just calling them, so everything is great. And that now becomes our reference in PyTorch. Um, I'd like to just take you through one example of a layer. So for example, LayerNorm here is like a PyTorch layer. And, uh, we'd like to basically port this over to C. So what kind of process do we go through? Well, we're gonna iterate through all the layers. Number one, we need the forward pass. And actually, I had to write a forward pass of LayerNorm, um, because, uh, PyTorch doesn't just have this kind of implementation in PyTorch of LayerNorm because it's kind of like a block, and eventually it calls into some CUDA kernels. Uh, so I had to write a forward pass of LayerNorm and make sure it's equivalent to the LayerNorm in PyTorch. And then of course, I had to write the backward pass of LayerNorm. This is where you kinda take out your pen and paper, do some backprop. Uh, this is, uh, for BatchNorm, but LayerNorm would be similar. And yeah, we had to write the backward pass. And again, you're-- this is all still in PyTorch, but it's explicit, and you're just making sure the LayerNorm of PyTorch forward and backward matches this represent-- this, uh, um, basically, um, manual, uh, tensor-based, um, implementation. So now we have PyTorch code forward, backward. So next thing we do is we try to port it to C. And this is actually like a lot simpler in many cases than you might think. So on the left we have the PyTorch code, and on the right we basically have the equivalent of LayerNorm forward in C. Uh, and it's not that crazy, right? So, uh, unlike in PyTorch, we just have a bunch of, uh, float star arrays. Uh, so we have a float star out, float star inputs, outputs, means, standard deviations, weights and biases, and some hyperparameters. And, uh, one thing I really liked to do in llm.c, I just wanna keep things simple. I don't wanna create a tensor abstraction. I don't wanna create any abstraction really. It's just float arrays and operations on float arrays. Like why should it be a lot more complicated than that? So everything's just float arrays. Uh, everything is fully self-contained. There's no underlying representations, abstractions to call, import, et cetera. This is the LayerNorm forward on float arrays, and that's it. So that's the forward. Um, and then you also do the backward for all the layers. Once we've done that for all the layers and converted everything to C- -to make sure that everything matches our, uh, reference implementation, we have to start to string it together. So, uh, we go into, uh, our, uh, C code in main, and we have to allocate all of the memory that we're gonna be using. In llm.c, all of the allocation happens a single time at the beginning. So we pre-plan all of the memory that we're gonna ever use. Then it's fixed, and from then on it's just dynamics of just feeding data through it and, uh, training the model. So we have to pre-plan all the tensors, their sizes, and we have to do that for the parameters. And we have the data grad and the mmb for the AdamW, uh, buffers, and then for the activations as well. Like we need space for both data and grad. And so you just pre-plan all the memory, you allocate all of it, and then we need to stitch it all up. So, uh, we have all these layers, and they have a forward and a backward pass in a backward passation. And so on the forward, uh, the forward pass, just kind of like you allocate all these tensors and you're very careful and you index into them properly and you make sure everything flows correctly through. And you just call all the forwards and then all the backwards. And then, uh, you're kind of done and you're left with gradient and can do an update. So stringing that together is the second piece of work. And then once we sort of like strung it together, you get something that you can just compile and run. So, uh, we, um... on the top left is everything that's required. We download a starter pack, which is really just the GPT-2 weights in a single binary file. Very simple. Uh, and also we need the dataset, in this case tiny Shakespeare, and a tokenizer and stuff like that. And then we just compile and run this little C code file. It's a single file of C at this point. And, uh, I think it's like two thousand lines or something like that if I remember correctly. And, uh, you run that program, and it like does a little training and outputs out Shakespeare at the end. And then we can verify that the PyTorch code is identical to the C code and everything is great, and we're just running in C. And at this point, I'm actually feeling quite great- -because this is amazing. So we have a single file C. There's no dependencies whatsoever. It compiles instantly. It runs instantly. All the memory is just allocated, uh, in a single blob. So if you start stepping, there's no way you're gonna boom later. It's all pre-planned. It's fully deterministic. It, in principle, can train GPT-2. It's complete. It will train GPT-2, you just have to wait a long time. And it can run on a potato. It can just run on anything. It's just a single file of C with no dependencies. And in principle, this could run-- this would be a great candidate to run on a, on a Loongson probe, uh, because in, in space if we just hardened it a little bit more. Uh, because you know, you're not gonna ship PyTorch code on a Loongson probe. But I think llm.c is great. So I was feeling great at this point. Uh, fun side note, by the way, all of this work that I described so far- **Host** [10:01] Whoa. **Andrej Karpathy** [10:03] -I worked on on vacation and while I was jet lagged in Maldives. Uh, so, uh, I-- basically it's perfect because you wake up at one AM- -and there's nothing to do. Uh, so you write stuff like llm.c, and then in sunrise you go do all the water activities. So that is the villa where most of llm.c was, uh, was trained. Uh, so that was perfect. This is a picture of it. And this is a rec-- this is the-- I think the moon is about to set and the sunrise is about to happen. This is a recommended way to do software development. ### CUDA Kernels **Andrej Karpathy** [10:32] Okay, so now we have C code, but it's inefficient, so we'd like to run it faster. For that we reach for GPUs. So we need to convert all of our C code to GPU. So this is where we go to the depth CUDA, uh, part of the repo, and we start to develop all the kernels. So here's the LayerNorm forward pass, as I mentioned. And now we're gonna develop a number of kernels that have the identical functionality, but now run on a GPU, and they're gonna be faster. And so usually we have versions one, two, three, four, five, six, et cetera. And these are all different kernel implementations that are a bit faster usually over time, but they match, uh, the specification exactly and give the exact same numbers. Uh, so we develop all those layers and port them to CUDA. And, uh, this is... uh, I don't know what this is. I'm gonna skip that. I think it's like, uh, one of the kernels. Uh, basically the point here is the first kernel is trivial to do usually because you're parallelizing over batch and time, and then you're basically copy-pasting the C code into your CUDA kernel. And you're already getting speedups because you're parallelizing over the batch time tokens and each thread just handles a single output element. So the first kernel is usually trivial, but then optimizations are usually pretty, uh, elaborate. So by the end we get to kernel six, for example, in LayerNorm, and we're doing a lot of things that are a bit more complicated. So we have some, uh, you know, um, more produce operations. We have some, uh, we also communicate through shared memory, through global memory. We're orchestrating it correctly. Uh, some cache streaming hints and a bunch of little tips and tricks for, uh, dealing with everything. Um, and I'm gonna go into a bit more detail later, but this-- you can get arbitrarily complicated here writing the CUDA code. One thing that I sort of found in this project is that it's not exactly trivial to learn CUDA, unfortunately, and, uh, it was like a little bit harder than I expected. Um, I mean, I, I knew some CUDA going in, but getting better at it, I think is not trivial. I think some of these books, unfortunately, are, are a bit out of date, as you might know. Uh, PMDP is actually quite good, uh, but also I think still kind of like, um, mostly on the beginner level because a lot of the CUDA code that we ended up developing in the lifetime of the llm.c project, you would not find those things in, in this book, actually. Uh, so a lot of the kernels that, uh, we en-ended up, uh, adding would just, uh, not be covered. And then on top of that, you have this CUDA C++ programming guide, which frankly is not exactly readable for someone who is like a bit new to that, uh, to the-- to CUDA. Uh, and then you have this amazing blog post from Simon at Zephyrtropic that is like way better than anything we deserve just like randomly on the internet. So that was incredible, and if there was just more of that, that would be so amazing. But, uh, yeah, so I think I found it a little bit difficult. Um, but, uh, I mean, I'm hoping that things like CUDA MODE can, can definitely, uh, speed up, uh, the, um, uh, availability of, of writing CUDA. Okay, so next what happened is, uh, I was basically struggling with the CUDA code, uh, a little bit, and I was reading through the book, and I was implementing all these CUDA kernels. And they're like okay CUDA kernels, but they're not, uh, great. And so, uh, a team of Avengers assembled from the internet, uh, when they saw llm.c and started contributing. So specifically, uh, Eric, Arun, Alex R are like, I would say, core devs of llm.c and have contributed a ton of work to llm.c. And they, they started to like really, uh, optimize and write all these kernels, and this was incredible to watch and learn a lot from. And there's many more, uh, Ross Buehler and, uh, Chen Thistle and a few others. Uh, there's-- But over time, we have sixty contributors to the llm.c project. Shout out to Lilab, our sponsor in llm.c. Uh, they con-contribute compute so that we can run and optimize all these kernels. Uh, so it was amazing for me that, uh, people just came from the internet and helped out on the project. ### Avengers Assemble **Andrej Karpathy** [13:50] And you know, this is one of the favorite things that can happen. My, my favorite things that can happen within open source MIT license repo, people just come from the internet and all contribute. It's amazing. Okay, so we've converted all the layers to CUDA. We have now all the kernels, and we can now train on a single GPU in FP32 so far. So that's great. So from then on, we start to make more and more optimizations. So number one, we don't wanna have matmul in FP32 when you roll your own code. Uh, y-we actually switch to cuBLAS. Uh, step two, we don't wanna write our own flash attention. I think that'll be pretty complicated. Turns out cuDNN has a very good flash attention implementation, so we switch to that. Um, next, you wanna definitely reach for mixed precision, uh, so that, uh, to speed up, um, the code. So you wanna go over all your tensors for parameters and also for activations and so on. And you have to start to think about, okay, which ones are in float thirty-two, which ones are in bfloat sixteen, and, uh, what precision are they in? And then we do all the conversions, uh, automatically. So we, uh, reach for that and implement that. There's many, many other optimizations that we ended up implementing over time. So as an example, we did all the kernel fusions, uh, different recompute settings to recompute a piece of the forward pass during the backward. Uh, we, um-- There's been a lot of optimizations from Eric, especially on minimizing the amount of memory that you need during the backward pass. Um, we have this like packed one twenty-eight data structure, which basically, in our experience, forces the compiler to use the one twenty-eight bit load and store instructions that are available. But somehow the compiler is unwilling to use in many cases. Uh, so I think Arun did a lot of, uh, work here where you just look at the SAS, and you look at, um, the SAS as the assembly, and you are looking at what the instructions are being used for a loop. And you figure out that, okay, this should be a one twenty-eight bit load and store, but it happens to be a thirty-two bit or something else because something in the NCC compiler is not going very well. So it-- we found that this data structure kind of forces the compiler's hand a bit more. Um, we implemented all kinds of CUDA streams to overlap, uh, the part of the computation, and this ended up creating like a total disaster. Um, and so that's why I scratched it out because at one point of llm.c as Arun would say, I basically went in, and I nuked it from orbit. I just went in. I did Control + F for all mentions of stream, and I just delete, delete, delete. And basically, I deleted all the streams, made everything single-threaded because we ended up getting all kinds of really weird race conditions and errors and so on, and I just didn't wanna deal with it. So llm.c is not actually as overlapped as it could be, but it's just like it's, it's too much complexity for not enough gain at this point. So, uh, but maybe the-- maybe we can slowly reintroduce some of it. We have stochastic rounding. We have full determinism. Full determinism turns out to be pretty hard because some of the kernels complexify a lot because you can't use atomics. Like the encoder backward was especially crazy because encoder backward, um, is trivial with atomics but non-trivial without it. Anyway, so a lot of the optimizations went into, uh, with, with a lot of, uh, efficiency and determinism in mind, and accuracy, like stochastic rounding and so on. Next, you want to use multiple GPUs, not just a single GPU. So this is where you, uh, bring in the NCCL. You start to, um, do all reduce between all the different workers. And this is where you also start to reach for like sharded optimizer state, uh, zero/one. So basically, you take your optimizer states which are in float, and these are really large, um, buffers for AdamW, and you can actually spread out a lot of the stuff across all the GPUs, and it really helps to keep, uh, your requirements down per GPU in terms of memory. So very helpful to reach for that. So currently, llm.c uses zero/one, uh, which is a sharded optimizer state. There's a PR for zero/two, but I don't believe I merged that, uh, yet because it gets a little bit messy but might be merged, uh, eventually. A lot of llm.c is just kind of like, um, balancing the improvement and speed with the complexity of what you're actually introducing. And so I've actually rejected a lot of PRs because of that, because the code starts to get crazy, and I think that decreases the amount of people that can be onboarded to contribute. Uh, and then after multi-GPU, you have multi-node. So now you are running across multiple machines. You have to make sure that you synchronize, uh, all of them, that they can find each other and so on. So we implemented all that. And where that lead us to is that we can actually train GPT-2, and we can actually reproduce it after all of that work. So there's a post in the- ... discussions of llm.c. We can train the one point six billion GPT-2, which was state-of-the-art LLM as of twenty nineteen or so. ### GPT-2 Training **Andrej Karpathy** [18:05] Uh, and you can train it on a single node of H100s in about twenty-four hours, and that costs roughly six hundred dollars. And the way you do that is it's extremely dependency-free. There's no need for Python, no need for PyTorch. Uh, so, uh, you do need cuDNN, which is the most heavy dependency, but cuDNN is optional. So if you'd like to roll your own manual attention, that is possible in llm.c. But cuDNN is kinda like the hairiest dependency. But after that, it's just a bunch of C code. You compile it, and you run it. There's no need for really anything. Uh, so there's no need for conda environments, pip installs. There's just nothing, which is amazing. Uh, and then you compile your code, and you run it, and it starts stepping, and you wait twenty-four hours. And then this is it stepping, uh, printing some diagnostics. We have almost, uh, fifty percent MFU here on one, uh, node, which is quite good. Um, and you get really nice plots, and you beat GPT-2 on HellaSwag. And, uh, basically, this just indicates that the optimization went well, no crazy numerical issues, lost pipes or anything like that for this size. And, uh, yeah, achieving a really good model in llm.c, in llm.c. Um, we can still compare to PyTorch because remember, we have PyTorch implementation for all this stuff in parallel on the side. And so you can run the equivalent training loop almost in PyTorch, and we can compare the two implementations side by side. And in particular, at the time of writing that post, and I don't know if this has changed because the PyTorch team continues to optimize things over time, but at the time of that post, we were using in llm.c thirty percent less memory, and we were twenty percent faster in training, just the throughput. And I don't know if I fully super-duper optimized the PyTorch implementation. I did my personal best. Um, but, uh, this is-- We were able to, I think, beat PyTorch in training of specifically GPT-2 in llm.c. If you wanna train anything else, you're in a lot of trouble. You'd have to change your code a lot. And we're not doing that, nor come back to it. But for GPT-2 training, we're better after all that work. Um, and it also compiles and runs much faster, which is beautiful. Torch compile actually takes, like, quite a bit of time, like a minute or something. You're just waiting. Uh, so that's also something that I personally don't like to work with usually. Okay. So looping back around, turns out it wasn't all that simple. There was a lot of stuff involved, and it took a few months for a few people. Um, but, uh, it was fun. We learned a lot and made friends along the way. **Guest** [20:19] Aw. ### In Progress **Andrej Karpathy** [20:19] This is the llm.c core devs. You know, so it was, it was great. Um, ongoing work, uh, we are adding Llama Three support. We actually thought maybe we would have it done by today, but, uh, there's a few more, uh, few more... a little bit more work to do. But we will have Llama Three point one, um, training in llm.c very, very soon. Uh, we will have FP8 support, so Haroon has been working on this. Uh, and, and, uh, there's a big PR that's coming for FP8 support, which is also interesting. Um, and there's a lot of notable forks of llm.c. They're all listed on the GitHub repo. Uh, the AMD fork is very active, as far as I understand, and, and quite good. Uh, I think also the C plus plus, uh, CUDA fork is, is quite nice. And so a lot of, uh, forks. Um, uh, so encourage you to also fork llm.c. It's fairly readable, I think. I try to keep it clean, well-documented. I think it's pretty well understood what's in there. It's only maybe like, I think, uh, three thousand lines of code of basically C mostly. Uh, and one more thought I think that I wanted to get across is, um, it wasn't all that haphazard to start the project. I had another, uh, motivation for starting the project, and that's that I think, um... And what is llm.c? ### LLM Compiler **Andrej Karpathy** [21:28] Like, if PyTorch is, especially with Torch compile, is a bit like GCC for software two point O, it's a compiler. Then llm.c is a bit like writing assembly. We're doing everything manually, right? And basically, I think, um, we wrote llm.c as multiple people over a duration of three months and got something that was faster than PyTorch in a specific setting of GPT-2 training. And so what this, this exercise basically proves that this is possible. Now, the problem is you need to spend multiple people several months. But if LLMs are about to become much better at coding over time, then I think you can expect that the LLM could actually do this for any custom application over time. And so the LLMs could, could act as a kind of compiler for any custom application you're interested in. They're gonna do all the LLMC work, and they're gonna output a binary, uh, that you can compile and run for your specific applications. So I don't actually know if we-- like the use of Python and PyTorch and everything else is just a crutch because we humans are finite. We have finite knowledge, intelligence, and attention. Uh, but actually, don't you wanna write all code in custom CUDA kernels and so on? Like maybe. And so, um, the other thing that I think is interesting is, uh, the llm.c repo might be useful because in the early stages of these LLMs and their intelligence, uh, they might not be able to write this code from scratch. You just prompted them, "Write GPT-2 in C." You probably won't get llm.c. But you're a lot more likely to get it if you put llm.c in the context of a certain LLM, and you can expect that the few-shot learning will be very helpful for the LLM to basically give it example code. And so I think llm.c could be very useful for this example code to give to the LLMs that they're about to write all of our custom applications. And so I think this is actually not unlikely to happen. Um, yeah, this is kind of likely to happen. So I think software development, in general, will probably change a lot. And to me, llm.c is an exploration of whether this is even possible because if it is possible, then maybe this crazy world's gonna happen. So yeah, that's it. Thank you. All right. Like, well, the morning sessions are done --- This library is powered by PodHood (https://podhood.com), the podcast website platform.