Chinchilla

The AI Frontier: from Gemini 3 Deep Think distilling to Flash — Jeff Dean

Feb 12, 2026 · 1:23:31

Jeff Dean, Google's Chief AI Scientist, explains how the company's dual strategy of frontier "Pro" models and distilled "Flash" models, alongside co-designed TPUs and latency-optimization, keeps Gemini at the Pareto frontier. Distillation compresses high-capability large models into smaller, efficient ones, enabling Flash models to exceed earlier Pro generations. He emphasizes energy-based thinking: data movement costs thousands more picojoules than arithmetic, making batching and speculative decoding essential. TPU development requires predicting ML workloads 2–6 years out, often speculatively adding hardware features. Gemini was born after Dean wrote a one-page memo arguing that fragmented model efforts across Google Brain and DeepMind should merge into a single unified multimodal effort. Looking ahead, he predicts personalized models attending to all user data and reasoning speeds of 10,000 tokens/second will transform software engineering.

Training Llama 2, 3 & 4: The Path to Open Source AGI — with Thomas Scialom of Meta AI

Jul 23, 2024 · 1:04:34

Thomas Scialom, Llama 2 lead and Llama 3 post-training lead at Meta AI, explains why scaling laws must go beyond Chinchilla—training models longer on more tokens yields better inference efficiency than bigger models, a lesson that drove Llama 3's 15 trillion token pre-training. He details how synthetic data from Llama 2 bootstrapped Llama 3's post-training, eliminating human-written SFT data, and why RLHF outperforms imitation learning: humans are better discriminators than generators, enabling superhuman outputs. Scialom defends the dense 405B architecture over MoE, calls tokenizer vocab size underrated (128k tokens vs Llama 2's 32k), and reveals Llama 4's focus on agentic capabilities—tool use, multi-step reasoning—as the path to open-source AGI. Meta AI is hiring researchers with rigorous first-principles thinking.