NCCL

llm.c's Origin and the Future of LLM Compilers - Andrej Karpathy at CUDA MODE

Sep 21, 2024 · 23:29

Andrej Karpathy explains llm.c, his from-scratch C implementation of GPT-2 training that outperforms PyTorch by 20% in speed and 30% in memory, arguing that LLMs will eventually act as compilers for custom code. He recounts starting the project out of frustration with PyTorch compile errors, writing all forward and backward passes manually in C, then porting to CUDA with kernel optimizations. A community of contributors (Eric, Arun, Alex R) helped achieve 50% MFU on a single H100 node, training GPT-2 1.6B in 24 hours for $600. Ongoing work adds Llama 3.1 and FP8 support. Karpathy sees llm.c as proof that LLMs can automate such low-level optimization, making Python and PyTorch a temporary crutch.

Ep 18: Petaflops to the People — with George Hotz of tinycorp

Jun 20, 2023 · 1:23:24

George Hotz of tinycorp argues that a simplified, open-source ML framework (tinygrad) can democratize AI compute and replace NVIDIA, Google, and AMD. Tinygrad uses 25 ops (vs. XLA's 250) and fuses kernels automatically, achieving 2x speed on Qualcomm GPUs over their library. Hotz reports AMD kernel panics fixed after emailing CEO Lisa Su, but laments AMD's open-source as 'dumped on GitHub.' For hardware, tinybox is a six-GPU desktop at 350W per GPU, aiming for 5x better price-performance than NVIDIA H100 for 90% of training. He criticizes OpenAI's secrecy, revealing GPT-4 is an 8-way mixture of 220B models, and cites the Bitter Lesson. He also outlines FLOPcoin using hardware identity to prevent cheating, and his goal of an AI girlfriend as the third company, merging humans via data rather than implants.