DeepSpeed

[Paper Club] Weight Streaming on Wafer-Scale Clusters (w/ Sarah Chieng of Cerebras)

Dec 7, 2024 · 43:54

Sarah Chieng of Cerebras presents the weight streaming technique for training giant neural networks on Cerebras wafer-scale clusters, arguing it enables near-linear scaling by separating parameter storage from primary compute. The system uses the Wafer Scale Engine (WSE-2/3) with 900,000 cores and 44 GB on-chip SRAM, avoiding off-chip memory bottlenecks that limit NVIDIA GPUs. MemoryX provides external storage for weights and optimizer states (up to 2.4 petabytes), streaming weights to compute units via the SwarmX interconnect fabric, which aggregates gradients. This design allows training models with up to 120 trillion parameters without complex hybrid parallelism. Cerebras also leverages unstructured weight sparsity to prune 90% of data during transmission and skip zero-value computations on-chip, reducing bandwidth and improving efficiency.

[LLM Paper Club] Llama 3.1 Paper: The Llama Family of Models

Jul 29, 2024 · 1:23:46

This episode examines Meta's Llama 3.1 paper, detailing the 405B dense model, its scaling laws grounded on the ARC reasoning benchmark rather than perplexity, and the decision to train on 15 trillion tokens. Vibhu explains the training infrastructure: 16,000 H100s over 54 days with 419 interruptions, 78% from GPU hardware failures. Eugene Yan walks through the synthetic data pipeline—using Llama 2 for filtering, stepwise reward models, and Monte Carlo tree search to improve reasoning traces. Hassan shares building LlamaTutor.com with Together API, serving 4,000 visitors and 5,900 requests for about $12. The group also discusses quantization trade-offs (larger models degrade less), inference provider variability (Groq's non-deterministic temperature zero), and compares Llama 405B's performance to GPT-4o and Claude.

State of the Art: Training 70B LLMs on 10,000 H100 clusters

Jun 25, 2024 · 1:32:05

Josh Albrecht, CTO of Imbue, and Jon Frankle, Chief AI Scientist of Databricks, detail the challenges of training 70B models on 10,000 H100 clusters, from stolen Infiniband cables to GPUs silently returning wrong math. Imbue open-sources infrastructure scripts, health checks, the CARBS hyperparameter optimizer, and cleaned versions of 11 benchmarks plus 450,000 human judgments on ambiguity. Databricks released DBRX, a 132B-parameter mixture-of-experts model with 36B active per token, and a text-to-image model trained exclusively on Shutterstock's curated dataset. They discuss typical 3% weekly hardware failure rates, the difficulty of MoE training pushing network bandwidth, and why eval datasets need human scrutiny—most benchmarks are saturated once ambiguous examples are removed. Both emphasize that success requires deep understanding from firmware to final loss, not just fancy libraries.