4o Mini

Sleep-Time Compute — Letta AI (Charles Packer, Charlie Snell, Kevin Lin)

Apr 21, 2025 · 34:02

This episode covers Letta AI's new paper on Sleep-Time Compute, a scaling direction that applies compute during model idle periods (sleep time) rather than only at test time. Charles Packer, Kevin Lin, and Charlie Snell explain that sleep-time compute precomputes inferences from static context before queries arrive, yielding Pareto improvements on math benchmarks like GSM8K by shifting the accuracy-to-token curve leftwards. They distinguish it from test-time compute by emphasizing that sleep-time tokens incur no user-latency cost, and show that the benefit is largest when questions are predictable from context. The team also ties the concept to stateful agents and memory systems (building on MemGPT), releasing two implementations: one for low-latency chatbots and another for document-driven agents.

The AI Architect: Bret Taylor

Feb 11, 2025 · 1:35:59

Bret Taylor, CEO of Sierra and chairman of OpenAI, recounts engineering AI's future across three eras — rewriting Google Maps' frontend in a weekend (cutting un-gzipped code to 20K), building Sierra's in-house agent platform for consumer brands like Sonos and SiriusXM, and navigating OpenAI's leadership crisis. He argues that AI agents' core abstractions are still pre-React, compares today's agents to jQuery, and predicts that outcome-based pricing will dominate as agents complete jobs rather than assist humans. He urges software engineers to move up the stack toward domain-specific agents and to embrace Rust-like languages for AI-generated code, emphasizing formal verification and safety. Taylor also shares how OpenAI prioritizes mission over product features, and why the relationship with Microsoft remains OpenAI's most important partnership.

[LLM Paper Club] Llama 3.1 Paper: The Llama Family of Models

Jul 29, 2024 · 1:23:46

This episode examines Meta's Llama 3.1 paper, detailing the 405B dense model, its scaling laws grounded on the ARC reasoning benchmark rather than perplexity, and the decision to train on 15 trillion tokens. Vibhu explains the training infrastructure: 16,000 H100s over 54 days with 419 interruptions, 78% from GPU hardware failures. Eugene Yan walks through the synthetic data pipeline—using Llama 2 for filtering, stepwise reward models, and Monte Carlo tree search to improve reasoning traces. Hassan shares building LlamaTutor.com with Together API, serving 4,000 visitors and 5,900 requests for about $12. The group also discusses quantization trade-offs (larger models degrade less), inference provider variability (Groq's non-deterministic temperature zero), and compares Llama 405B's performance to GPT-4o and Claude.