CodeLlama

[LLM Paper Club] Llama 3.1 Paper: The Llama Family of Models

Jul 29, 2024 · 1:23:46

This episode examines Meta's Llama 3.1 paper, detailing the 405B dense model, its scaling laws grounded on the ARC reasoning benchmark rather than perplexity, and the decision to train on 15 trillion tokens. Vibhu explains the training infrastructure: 16,000 H100s over 54 days with 419 interruptions, 78% from GPU hardware failures. Eugene Yan walks through the synthetic data pipeline—using Llama 2 for filtering, stepwise reward models, and Monte Carlo tree search to improve reasoning traces. Hassan shares building LlamaTutor.com with Together API, serving 4,000 visitors and 5,900 requests for about $12. The group also discusses quantization trade-offs (larger models degrade less), inference provider variability (Groq's non-deterministic temperature zero), and compares Llama 405B's performance to GPT-4o and Claude.

How to train a Million Context LLM — with Mark Huang of Gradient.ai

May 31, 2024 · 1:12:14

Mark Huang of Gradient.ai explains how his team extended Llama 3 to a 1 million token context window using curriculum learning, RingAttention, and EasyContext, achieving near-perfect GPU utilization. They employed theta scaling from the RoPE paper to interpolate positional encodings, trained on carefully curated datasets including SlimPajamas and synthetic data from GPT-4, and validated with benchmarks like Ruler and needle-in-a-haystack. Huang details the trade-offs between full fine-tuning and LoRA adapters, the challenges of pushing to 4M tokens (degradation from floating-point precision limits), and why long context matters for state management across sessions and grounding multimodal inputs. He calls for community collaboration on long-context evaluations and pairwise multimodal datasets.