Infrastructure

Podcast Crossover: AIE, AGI, frontier lab strategy with ​ ⁨@matthew_berman⁩ and @swyxtv

Jul 10, 2026 · 28:03

Shawn 'Swyx' Wang, founder of the AI Engineer conference, tells Matthew Berman how he seized the industry shift by buying ai.engineer and partnering with a veteran conference organizer, crediting Andrej Karpathy's early endorsement. He argues Etched's ASICs are a natural next-gen bet for transformer inference, not an NVIDIA disruptor, and that Fable 5's slowness and cost signal the end of the LLM scaling era—making model efficiency the next problem. Swyx interprets OpenAI's reported 5% equity offer to the US government as a pragmatic multi-turn negotiation, likening it to Singapore's Temasek model, but warns against premature utility regulation. He pegs his own P(doom) at ~5% over 50 years, rejecting near-term doomerism as egotistical. For founders, he advocates building 'agent labs' that solve specific customer problems (e.g., for lawyers, dentists) rather than betting on model routing, which he dismisses as a marketing line that fails to exploit a single model's full stack as deeply as frontier labs do.

The Future of AI Infra: from Kubernetes to Agent Sandboxes — Akshat Bubna, Modal CTO

Jul 8, 2026 · 59:10

Modal CTO Akshat Bubna explains how the cloud platform evolved from a serverless runtime to an AI cloud built for elastic inference, agent sandboxes, and post-training workloads. He argues that Kubernetes was never designed for bursty GPU-heavy AI workloads, so Modal built a decorator-based infrastructure that co-locates compute requirements with code. The company added GPUs a year before ChatGPT and now powers inference for custom models at Suno, Runway, and robotics firms, using GPU snapshotting to slash cold starts. Modal's new Auto Endpoints incorporate open-source DeFlash speculative decoding for frontier-level performance. For reinforcement learning rollouts, Modal provides up to 100,000 sandboxes simultaneously. The platform spans 17 cloud providers, features private IPv6 networking via eBPF, and supports serverless multi-node training with RDMA. Bubna also reveals Modal's shift from developer experience to agent experience, noting that agents benefit from the same minimalist SDK and observability tools that humans use.

Why AI Labs With Unlimited GPUs Still Fail — Anjney Midha, AMP

Jun 18, 2026 · 1:00:37

Anjney Midha, CEO of AMP, argues that AI labs with unlimited GPUs still fail due to misaligned culture and infrastructure waste, proposing a compute grid modeled on independent system operators to pool demand and supply. At Google, 95% node utilization was considered an outage, yet most clusters today don't reach that, with waste compounding at scale. AMP’s grid, starting at scheduling, aims to make FLOPs flow like megawatts, having secured 1.3 gigawatts of demand. Midha explains Anthropic cracked coding because 'luck favors the prepared mind'—their four years of paranoia and scarcity created a culture that OpenAI’s abundance couldn't replicate. He also shares a 14-year mission in end-of-life prediction, arguing AI can reduce the 30% of Medicare/Medicaid spend on end-of-life care. He warns that too much capital too early makes labs fragile because without hardship they fail to define their P0.

⚡️ Google's Open AI Strategy — Omar Sanseviero, Google DeepMind

May 24, 2026 · 29:59

Omar Sanseviero, Google DeepMind's Head of Developer Experience, explains Gemma 4's novel architecture with per-layer embeddings that enable effective parameter offloading: only 2B of 5B parameters need GPU memory, ideal for on-device inference on phones and Raspberry Pis. The model matches 1.5-year-old state-of-the-art in most areas, with Gemini Nano integrated into Pixel and Samsung phones. Gemma 4 supports multimodal input (audio, images, short video) but not audio output or combined audio-video prompts. Sanseviero notes fine-tuning is declining as out-of-box capabilities improve, but remains relevant for specialized domains like healthcare. He contrasts dense (31B) and MoE (27B) variants, highlighting MoE's inference speed but fine-tuning challenges. The team is expanding globally, with Kaggle joining DeepMind to create community-driven benchmarks for model evaluation.

Retrieval After RAG: Hybrid Search, Agents, and Database Design — Simon Eskildsen of Turbopuffer

Mar 12, 2026 · 1:00:32

Simon Eskildsen, founder of Turbopuffer, explains how his obsession with the cost of vector search at Readwise led him to build a search engine on object storage and NVMe SSDs, cutting costs by 95% for customers like Cursor. He outlines the three conditions for a major database company—a new workload (AI-driven search), a new architecture (object storage first, no consensus layer), and evolving to support every query plan. Turbopuffer's architecture uses S3's strong consistency and compare-and-swap for metadata, avoiding stateful systems. Eskildsen shares how they won Notion by absorbing latency issues with dark fiber, and discusses the shift from single RAG queries to agents making many concurrent searches. He also describes the "P99 engineer"—obsessive, trade-off minded, able to bend software to first principles. Future plans include expanding full-text search, reaching Common Crawl scale, and eventually adding simpler OLAP queries.

Agent Inference at the "Speed of Light" — How NVIDIA moves like a $4.3 Trillion Startup

Mar 8, 2026 · 1:26:00

NVIDIA's Nader Khalil and Kyle Kranen join Swyx and Vibhu to explain how the company moves like a $4.3 trillion startup through speed-of-light (SOL) first-principles thinking, agent security boundaries, and the Dynamo inference engine. They argue agents should only do two of three things (files, internet, code) to prevent vulnerabilities, and detail Brev's acquisition to improve developer UX with one-click GPU access and DGX Spark integration. Kyle describes Dynamo as a data center scale inference engine that optimizes serving by scaling out, using prefill/decode disaggregation, Kubernetes-based scheduling, and model-hardware co-design to improve cost, latency, and quality. The episode covers SOL's role in creating urgency, long-context limits and potential 'unhobblers' like multi-head latent attention, and the shift toward CLI-first agent workflows for enterprise tools.

Dylan Patel Explains the AI War While Cooking | In-Context Cooking

Feb 26, 2026 · 55:13

Dylan Patel, CEO of SemiAnalysis, argues hyperscalers like Google, Amazon, and Meta will sacrifice all profits to build AI infrastructure, spending $180–$200 billion in capex this year alone, because the AI adoption explosion—Claude Code driving 4% of GitHub commits in one month, Anthropic adding $2.5 billion monthly revenue—makes it a Pascal's wager: spend or die. He details how Taiwan's semiconductor geopolitics create endgame scenarios, from a KMT win placating China to full invasion, with TSMC's output critical. Patel explains Nvidia's paranoid founder Jensen Huang is responding to vertical integration threats from hyperscalers by diversifying into chips like CPX and Groq, but warns moats are shallow. The real bottleneck in AI progress? Semiconductors themselves: fabs take years to build, and no one can buy enough GPUs through 2028. He also predicts a massive AI backlash from the public and financial markets, as capital consumption outpaces revenue and labor displacement accelerates.

Claude Code for Finance + The Global Memory Shortage: Doug O'Laughlin, SemiAnalysis

Feb 24, 2026 · 2:07:40

Doug O'Laughlin, SemiAnalysis founder, explains his Claude Code awakening in December 2024: Anthropic's Opus 4.5 one-shotted tasks that used to take 24 hours, leading him to predict Claude Code will write 25–50% of all GitHub code by year-end (already 4% of commits in two weeks). He argues this marks the death of Excel and Bloomberg for analysts—'you can just do things'—but warns of a 'hygiene' crisis as junior analysts lose meta-learning. On the semiconductor side, O'Laughlin details a severe memory squeeze: HBM demand consumes 4× the DRAM capacity, DRAM prices could double again, and CXL is reviving as a workaround. He compares AI infrastructure buildout to the railroad boom (25% of U.S. gross fixed capital investment) and says Microsoft faces an innovator's dilemma—renting GPUs to 'barbarians at the gate' while its own office franchise is disrupted by Claude Code. Google's TPU v7 enjoys a temporary TCO advantage, but Nvidia's supply chain dominance (Jensen doing shots with SK and Samsung) will reassert with Rubin. O'Laughlin also reflects on his 2,800-mile Continental Divide Trail hike as essential self-mastery.

The AI Frontier: from Gemini 3 Deep Think distilling to Flash — Jeff Dean

Feb 12, 2026 · 1:23:31

Jeff Dean, Google's Chief AI Scientist, explains how the company's dual strategy of frontier "Pro" models and distilled "Flash" models, alongside co-designed TPUs and latency-optimization, keeps Gemini at the Pareto frontier. Distillation compresses high-capability large models into smaller, efficient ones, enabling Flash models to exceed earlier Pro generations. He emphasizes energy-based thinking: data movement costs thousands more picojoules than arithmetic, making batching and speculative decoding essential. TPU development requires predicting ML workloads 2–6 years out, often speculatively adding hardware features. Gemini was born after Dean wrote a one-page memo arguing that fragmented model efforts across Google Brain and DeepMind should merge into a single unified multimodal effort. Looking ahead, he predicts personalized models attending to all user data and reasoning speeds of 10,000 tokens/second will transform software engineering.

The Future of Email: Superhuman CTO on Your Inbox As the Real AI Agent (Not ChatGPT) — Loïc Houssier

Dec 11, 2025 · 1:11:02

Loïc Houssier, CTO of Superhuman Mail (recently acquired by Grammarly), joins hosts Alessio and Swyx to detail how Superhuman builds AI into email without adding latency. He explains their agentic framework of small tools vs. a single agent, the fight against “agent laziness,” and their eval process anchored by Rahul’s infamous “what wood was my table?” test. Houssier reveals Superhuman uses local-first caching and Baseten’s box pricing for cost control, stores embeddings in TurboPuffer, and has only three engineers on AI—yet PR throughput rose from four to six per engineer per week. He argues the inbox will power your future AI executive assistant, and that AI will widen the gap between engineers with real fundamentals and those faking it.

After LLMs: Spatial Intelligence and World Models — Fei-Fei Li & Justin Johnson, World Labs

Nov 25, 2025 · 1:00:39

World Labs co-founders Fei-Fei Li and Justin Johnson argue that spatial intelligence is the next frontier beyond LLMs, with their product Marble generating editable 3D worlds from text or images using Gaussian splats. They claim language is a lossy channel for describing the rich 3D/4D world, while spatial intelligence handles everything from picking up a mug to inferring DNA's double helix. Marble already serves gaming, VFX, and film, with precise camera control and real-time rendering on phones and VR headsets. They discuss scaling compute from AlexNet to today's million-fold increase, why world models can 'soak up' modern GPU clusters better than language alone, and the gap between pattern-fitting and causal understanding in physics. Fei-Fei worries more about under-resourced academia than open vs. closed models, advocating for national AI compute clouds and open benchmarks like the BEHAVIOR dataset. Justin notes transformers are natively set models, not sequence models, opening architectural possibilities for new spatial data.

How Zyphra went all-in on AMD + Why Devs feel faster with AI but are slower — with Quentin Anthony

Nov 3, 2025 · 1:00:02

Quentin Anthony, head of model training at Zyphra and advisor at EleutherAI, explains why his company moved all training to AMD MI300X GPUs, outperforming Nvidia H100s on certain workloads thanks to 192GB VRAM and higher memory bandwidth, and describes his kernel development approach of writing directly in ROCm or GPU assembly rather than using Triton. He also shares his experience in the METR study on AI coding productivity, where he was one of the few developers with measurable speedup, and offers tips: timebox AI use, avoid the slot machine effect, maintain context hygiene, and use direct API over tools like Cursor. Additionally, he argues that open source AI research benefits from siloed focused teams with guaranteed funding over grand collaborations, and notes that kernel datasets alone won't solve GPU programming due to evaluation challenges.

⚡️Raising $1.1b to build the fastest LLM Chips on Earth — Andrew Feldman, Cerebras

Oct 1, 2025 · 29:14

Andrew Feldman, CEO of Cerebras, joins Latent Space to discuss their $1.1B fundraise at an $8.1B valuation and their wafer-scale chip that delivers 20x faster inference than NVIDIA's B200 GPUs. He explains how their architecture uses SRAM instead of HBM, providing 2,625x more memory bandwidth by eliminating the narrow straw between compute and memory. Feldman details the decision to accelerate sparse linear algebra rather than specialized convolutions, enabling support for transformers and diffusion models unseen during design. He discusses the explosive growth in AI inference demand, the shift from closed-source to fast open-source models, and the importance of speed—citing Paul Graham's observation that ChatGPT's slowness drives users away. The conversation covers enterprise trends in the 10-30B parameter space, the complexity of building data centers that pull gigawatts of power, and the often-overlooked routing and caching systems that make AI work seamlessly.

⚡️ Beyond Transformers with Power Retention

Sep 23, 2025 · 32:59

Jacob Buckman, CEO of Manifest AI, discusses their solution to AI's expensive computational bottleneck: the ever-growing KV cache for long context inference. Power Retention replaces attention in transformers with a fixed-size memory, achieving 10X training speedup at 64K tokens and 100X inference speedup over FlashAttention. Manifest releases Vidrial, a just-in-time CUDA kernel framework that finds optimal configurations, and Metamorphosis, a process to convert existing transformers like StarCoder-3B into Power Retention models in just two hours of mid-training. Buckman demonstrates PowerCoder, a 3B code model that matches baseline loss after 10K steps and handles contexts up to 32K and beyond. The open-source release includes kernels and tools, aiming to build community trust and encourage adoption by inference providers. He also highlights the need for genuinely long-context datasets beyond internet text, such as human trajectories.

A Technical History of Generative Media

Sep 8, 2025 · 1:04:44

Fal.ai founders Gorkem and Batuhan detail their pivot from dbt pipelines to generative media inference, which now serves 2M developers and 350 models, crossed $100M ARR, and raised a $125M Series C. Key inflection points included Stable Diffusion 1.5 (company pivot), SDXL (first $1M revenue), Flux (jump from $2M to $10M monthly revenue), and Veo 3 (text-to-video with perfect lip-sync). They built a proprietary inference engine with 100+ custom kernels and a serverless GPU stack managing 10,000+ H100 equivalents across 6 cloud providers, typically delivering 1.5-10x speedups over stock PyTorch. Video models now drive 50% of revenue, up from 18% in February, fueled by open-source models like Hunyuan and partnerships with closed labs such as Play.ht and Google DeepMind. They argue advertising is the killer application for generative media and that image/video RL, specialized data pipelines, and cheaper conversational video models are underexplored startup opportunities.

⚡️Accelerators @ 3x NVIDIA H200 perf, Made in the USA - Thomas Sohmers + Mitesh Agrawal, Positron AI

Aug 18, 2025 · 47:56

Thomas Sohmers and Mitesh Agrawal of Positron AI argue memory bandwidth, not compute, is the true bottleneck in AI inference, and their accelerator achieves 93% memory bandwidth utilization—triple NVIDIA's efficiency—enabling 70% faster token generation at 150W. The founders, both Lambda Labs veterans, shipped an FPGA product 15 months after founding, then raised a $51M Series A for an ASIC in late 2026. Their hardware requires no recompilation: it ingests raw binary weights from NVIDIA training and outputs an OpenAI-compatible API. Positron focuses on the decode phase of transformers, where memory-bound matrix-vector multiplication dominates, and already counts Cloudflare and Parasail as customers. The company sells systems directly, prioritizing capital efficiency and ROIC over operating its own cloud.

The AI Agenda: GPT5 leaks and the business of AI News — Steph Palazzolo, The Information

Aug 6, 2025 · 1:14:36

Steph Palazzolo, AI journalist at The Information, explains how she covers the secretive AI industry, from OpenAI's GPT-5 to the inference market where startups like Modal raise billions. She reveals that inference providers are essentially GPU resellers facing margin pressure, and that GPT-5's performance will signal whether pre-training scaling or reinforcement learning drives progress. She critiques Meta's super intelligence ambition as misaligned with its ad-revenue business, and details the talent war with rumored hundred-million-dollar offers from Meta to poach researchers. Palazzolo also discusses the Windsurf acqui-hire backlash, the rise of coding agents like Claude Code, and how journalists protect sources while navigating leaks from CEOs like Sam Altman.

The Shape of Compute (Chris Lattner of Modular)

Jun 13, 2025 · 1:18:18

Chris Lattner of Modular explains how his company is breaking the CUDA monopoly with Mojo and MAX, matching NVIDIA's best inference performance on AMD and NVIDIA GPUs. After three years in R&D, Modular open-sourced its stack and now delivers state-of-the-art Llama 3 serving at over 800 tokens per second. Mojo, a Python-family language, runs kernels faster than Rust and can extend Python without bindings, while MAX provides a full inference framework with automatic kernel fusion and cluster management. Unlike VLLM or SGLang, Modular's container is just a gigabyte, fully open source, and not reliant on proprietary CUDA blobs. DeepSeek's low-level PTX work validated the approach, but Lattner emphasizes that Mojo's portability avoids rewriting kernels for each new GPU architecture. He also details his daily routine, using Cursor for coding, and hiring "elite nerds" to grow the team.

DeepWiki: The GitHub Encyclopedia

May 21, 2025 · 32:05

Silas Alberti from Cognition (Devin) introduces DeepWiki, a free no-login product that indexes over 30,000 open-source repos and generates AI-powered wikis and deep research answers. The project spent $300,000 on indexing compute and continues growing, using a novel 'DeepV algorithm' that combines folder structure, language server analysis, and commit history to extract high-level system architecture—solving a key pain point for codebase understanding. Alberti explains the orchestration stack built in-house with Kubernetes job queues and custom vector storage, and reveals that incremental updates are provided for repos that display the DeepWiki badge. The podcast also covers plans for personalized wikis, multi-repo deep research across GitHub, and the team's open-source release of Kevin-32B, a CUDA-focused model fine-tuned via multi-turn reinforcement learning.

What is an RL environment? w/ Nous Research's Roger Jin

Apr 29, 2025 · 15:27

Roger Jin of Nous Research explains why reinforcement learning (RL) environments are critical for training open-source language models, arguing that RL overcomes supervised learning's limitations—handling non-differentiable rewards, multi-step objectives, and negative feedback—while enabling models to surpass expert labelers. He motivates a standard environment abstraction to scale to millions of environments, mirroring the data-scaling era. Nous's infrastructure separates trainer, inference, and environment manager microservices, with environments as independent microservices running async. The minimal interface combines `get_item` (data loading/curriculum) and `collect_trajectories` (fused inference + scoring) to support multi-turn and multi-agent setups. Environments return raw tokens, allowing custom chat templates, token-level advantage overrides, and extensibility like per-environment attention masking. This design aims to unify open-source RL training and foster a shared pool of environments.

Tiny Teams: $6m ARR, 5m users with 4 employees — Sid Bendre, Oleve (Quizard AI/Unstuck AI)

Apr 23, 2025 · 42:59

Sid Bendre of Oleve (formerly Quizzer AI and Unstuck AI) explains how his four-person team built $6M ARR and 5M users across multiple consumer apps. Starting with Quizzer AI in January 2023, a viral TikTok gained 10K users in 30 hours; by August 2024, Unstuck AI hit 1M users in nine weeks. The company has been profitable since its first nine months. Bendre describes their platform engineering strategy: product engineers own individual apps while a platform team builds reusable systems, including internal shadow-org agents for growth and marketing. He shares technical hacks like using LaunchDarkly feature flags to load-balance across Azure OpenAI endpoints and a de-indexer for Azure AI Search to manage costs. The key engineering principle is deterministic LLM workflows—classifying user intent then routing to pre-built deterministic flows—rather than open-ended agent tool use.

SF Compute: Commoditizing Compute

Apr 11, 2025 · 1:12:02

Evan Conrad, co-founder of SF Compute, argues that GPUs behave like a real estate business, not a traditional cloud, because price-sensitive customers value every incremental GPU and will switch for a 10% margin. CoreWeave succeeded by selling locked-in long-term contracts to low-credit-risk customers like Microsoft and OpenAI, ignoring short-term demand. He predicts hyperscalers and providers like Together and DigitalOcean will lose money on GPU clusters because software margins cannot match the hardware costs. SF Compute started as an AI lab forced to sublease its cluster monthly to avoid bankruptcy, then evolved into a market where anyone can buy H100s by the hour via dynamic pricing—often below $1/hour for short bursts. Utilization stays near 100% as prices adjust. Future plans include cash-settled futures to reduce financial risk across the industry, while the brand deliberately stays anti-hype and calm.

LIVE from GTC: DGX Spark Insides First Look

Mar 20, 2025 · 12:39

Israel from NVIDIA gives a first look at the DGX Spark, a $3,000–$4,000 mini AI supercomputer that fits up to 200B parameter models in FP4 with 128GB unified memory, powered by the NVIDIA GB10 Superchip. It uses the same Grace Blackwell architecture and software stack as data center systems, so code written on the Spark deploys to production without modifications. The device includes a ConnectX-7 dual-port 200Gb ethernet for clustering two units, and runs DGX OS (Ubuntu 24.04). Storage is 1TB or 4TB NVMe. Partners ASUS, HP, Dell, and Lenovo will offer their own cases. Israel stresses the Spark is a developer box, not a server, but delivers enterprise-grade networking and shared memory in a compact form.

Browserbase: Browser Infrastructure For Your AI Agents

Feb 28, 2025 · 1:01:33

Paul Klein IV, CEO of Browserbase, argues that headless browser infrastructure is a critical primitive for AI agents to automate the web. Browserbase solves the hard problem of running thousands of browsers in the cloud—too big for Lambda, requiring Firecracker microVMs—and exposes APIs via its open-source framework Stagehand (act, extract, observe). It handles proxies, CAPTCHA solving, and offers a live view iframe for human-in-the-loop control. Primary use cases include browser automation and web scraping as a heavy-hitter renderer when simpler methods fail. Paul explains his choice to go solo founder and how Browserbase's in-person, 10–5 culture enables fast execution. He predicts the future of software will involve software acting on other software via browsers.

S1: the $6 DeepSeek R1 Competitor (ft. Entropix)

Feb 26, 2025 · 17:43

Tim Kellogg joins hosts Alessio and Swyx to break down his viral blog post on S1, showing how Stanford researchers cloned DeepSeek R1's reasoning for $6 by inserting a 'think token' and fine-tuning on just 1,000 examples. Kellogg argues the real innovation isn't the cost but the technique's simplicity—making advanced inference-time scaling accessible. He explains Entropix, a dynamic sampler that uses model entropy and varentropy to adjust generation on the fly, and notes its creators are starting a company. The conversation contrasts supervised fine-tuning's targeted results with RL's broader reasoning growth, using an analogy: 'RL causes a lot of growth, then SFT trims it.' Kellogg predicts that embedding introspection signals into RL training could solve agent 'doom loops' and unlock higher agency—a key challenge behind products like OpenAI's Deep Research.

Outlasting Noam Shazeer, Crowdsourcing Chai AI w/ 1.4m DAU — with William Beauchamp, Chai Research

Jan 26, 2025 · 1:13:33

William Beauchamp, founder-CEO of Chai AI, explains how he pivoted from algorithmic trading to build a character chatbot platform before Character.ai, growing to 1.4M DAU and $22M+ revenue by crowdsourcing model improvements through Chaiverse. Starting with GPT-J in 2021, he found product-market fit with a therapist bot and shifted to user-generated content, letting users define prompts, images, and names. Competing against well-funded rivals like Character AI and Talkie, Chai ships over 100 LLMs weekly via its developer platform, spending $10M on compute in 2024 and tripling it. Beauchamp argues AI follows an S-curve, not scaling laws, and focuses on inference optimization using rejection sampling and reward models to serve better responses. He prioritizes 'insanely great' products over technology-driven features, noting that audio and image features failed to move metrics, while a data flywheel and aggressive user acquisition drove rapid growth.

DeepSeek V3, SGLang, and the state of Open Model Inference in 2025 (Quantization, MoEs, Pricing)

Jan 19, 2025 · 57:18

Amir Haghighat and Yineng Zhang from Baseten explain how they serve DeepSeek V3, a 671-billion-parameter MoE model requiring H200 clusters with FP8 support, making it the top open-weights model. They detail Baseten’s dedicated inference model (no shared endpoints) and its reliance on SGLang for performance gains like Radix prefix caching and MLA attention support. Yineng highlights SGLang’s edge over vLLM and TensorRT-LLM in usability and customization, especially for large models. Amir outlines three pillars for mission-critical inference: model-level performance (via frameworks like SGLang), horizontal scaling across regions and clouds, and low-latency multi-model workflows. The episode also covers quantization trends, MoE architecture, and speculative decoding as key enablers for open model deployment.

Beating Google at Search with Neural PageRank and $5M of H200s — with Will Bryk of Exa.ai

Jan 10, 2025 · 55:54

Will Bryk, CEO of Exa.ai (formerly Metaphor), details building a neural search engine from scratch using link prediction as 'Neural PageRank' — predicting documents rather than keywords. Exa's new product offers near-perfect lists (e.g., 'startups working on hardware in SF') by scaling compute per query, from milliseconds to a day, like o1 for search. Bryk argues LLMs will become the interface to search, while Exa provides the 'super knowledge' that even AGI will need. He contrasts Exa's neural approach with Google's keyword-based system and Perplexity's reliance on Bing. The company recently purchased a $5M H200 cluster and maintains a culture of nap pods and first-principles thinking.

Best of 2024: Synthetic Data / Smol Models, Loubna Ben Allal, HuggingFace [LS Live! @ NeurIPS 2024]

Dec 24, 2024 · 28:08

Loubna Ben Allal of Hugging Face explains how synthetic data has become ubiquitous in LLM pipelines, moving from post-training to pre-training, with examples like Cosmopedia’s 30B-token synthetic dataset and NVIDIA’s 1.9T-token Nemotron CC. She addresses model collapse fears, showing that web dumps after ChatGPT’s release actually yield better models, and emphasizes diversity through prompt seeds and webpage extracts. For filtering, FineWeb-Edu and DCLM use LLMs to rate educational content, achieving top benchmark scores. In post-training, she highlights AgentInstruct, Tülu3 with PersonaHub, and Cohere’s multilingual arbitrage using multiple teachers. On small models, SmolM2 (1.7B) outperforms Llama 1B and Qwen 2.5 after 11T tokens of pre-training, and on-device inference via frameworks like llama.cpp enables privacy-preserving use cases such as text extraction and structured generation. She predicts a return to fine-tuning specialized small models over costly prompt engineering.

Best of 2024: Open Models [LS LIVE! at NeurIPS 2024]

Dec 23, 2024 · 37:29

Luca Soldani of AI2 and Sophia Yang of Mistral argue that open models in 2024 have exploded in number and performance, closing the gap with closed models, but face growing compute and data-access challenges. Soldani notes 2024 saw models like Qwen and Deepseek rivaling frontier closed-model performance, and fully open models like OLMo release full recipes including data, code, and intermediate checkpoints. He points out that pre-training requires 10K+ GPUs for state-of-the-art, while post-training can be done with as few as eight. Sophia highlights Mistral's release of over a dozen models in 2024, including Pixtral multimodal and Le Chat—a free chat interface with image understanding, canvas code execution, and web search. Soldani warns of data access diminishing due to content owners blocking crawlers and lobbying efforts labeling open source AI as dangerous, emphasizing the need to protect the open ecosystem.

[Paper Club] Weight Streaming on Wafer-Scale Clusters (w/ Sarah Chieng of Cerebras)

Dec 7, 2024 · 43:54

Sarah Chieng of Cerebras presents the weight streaming technique for training giant neural networks on Cerebras wafer-scale clusters, arguing it enables near-linear scaling by separating parameter storage from primary compute. The system uses the Wafer Scale Engine (WSE-2/3) with 900,000 cores and 44 GB on-chip SRAM, avoiding off-chip memory bottlenecks that limit NVIDIA GPUs. MemoryX provides external storage for weights and optimizer states (up to 2.4 petabytes), streaming weights to compute units via the SwarmX interconnect fabric, which aggregates gradients. This design allows training models with up to 120 trillion parameters without complex hybrid parallelism. Cerebras also leverages unstructured weight sparsity to prune 90% of data during transmission and skip zero-value computations on-chip, reducing bandwidth and improving efficiency.

Why Compound AI + Open Source will beat Closed AI — with Lin Qiao, CEO of Fireworks AI

Nov 25, 2024 · 55:49

Lin Qiao, CEO of Fireworks AI, argues that compound AI systems combining multiple open-source models will outperform closed monolithic models like OpenAI's. She explains Fireworks' evolution from a PyTorch platform to a full-stack inference and customization engine serving 40+ customers including Cursor. Qiao details their distributed inference engine, Fire Optimizer for quality-latency-cost tradeoffs, and upcoming o1-like model built on open-source foundations. She defends their quantization approach after public criticism from rivals, emphasizes specialization over general intelligence, and invites developers to test their free LoRA adapter hosting.

Singapore: the AI Engineer Nation — with Minister Josephine Teo

Oct 19, 2024 · 56:40

Singapore Minister Josephine Teo outlines the city-state's refreshed national AI strategy and the Ministry of Digital Development's approach to balancing innovation, safety, and talent development. She details plans to triple AI practitioners from 5,000 to 15,000, the AI Verify testing tools and AI Safety Institute for governance, and internal government AI bots like AI Bot for RAG-based knowledge retrieval. On sovereign AI, she discusses hybrid cloud infrastructure and pragmatic data center expansion. Teo also explains Singapore's new law requiring transparency for AI-generated election content, emphasizing fact-based political discourse.

llm.c's Origin and the Future of LLM Compilers - Andrej Karpathy at CUDA MODE

Sep 21, 2024 · 23:29

Andrej Karpathy explains llm.c, his from-scratch C implementation of GPT-2 training that outperforms PyTorch by 20% in speed and 30% in memory, arguing that LLMs will eventually act as compilers for custom code. He recounts starting the project out of frustration with PyTorch compile errors, writing all forward and backward passes manually in C, then porting to CUDA with kernel optimizations. A community of contributors (Eric, Arun, Alex R) helped achieve 50% MFU on a single H100 node, training GPT-2 1.6B in 24 hours for $600. Ongoing work adds Llama 3.1 and FP8 support. Karpathy sees llm.c as proof that LLMs can automate such low-level optimization, making Python and PyTorch a temporary crutch.

[Paper Club] Writing in the Margins: Chunked Prefill KV Caching for Long Context Retrieval

Sep 19, 2024 · 53:53

Umar Jamil from Writer presents 'Writing in the Margins,' a paper on improving long-context retrieval in LLMs by leveraging chunked prefill of the KV cache. The method splits a long prompt into chunks, generates query-relevant annotations (margins) after each chunk, and appends them at the end before generating the final answer. This approach solves the 'lost in the middle' problem without fine-tuning, and unlike traditional RAG or separate summarization, it avoids re-prefilling the entire context, reducing cost from double to single prefilling. The technique is compatible with any transformer model and includes overlapping margin generation and classification within the same batch request. Benchmarks show consistent improvements across models, and the implementation is open-source on GitHub.

Answer.ai & AI Magic with Jeremy Howard

Aug 17, 2024 · 1:11:00

Jeremy Howard of Answer.AI argues that continuous pre-training should be treated as a continuum, not separate phases, and demonstrates how FSDP+QLoRA enables training a 70B model on just two NVIDIA 4090s. He reveals Answer’s non-hierarchical, manager-free R&D lab model that recruited unusual talent like Benjamin Warner and Ben Claviez, who independently launched BERT 24 to revive encoder-only architectures. Howard introduces FastHTML, a pure-Python web framework built on HTMX and Starlette, for creating modern SPAs without JavaScript. He previews “AI Magic,” a dialogue engineering system that moves beyond teletype-style chat interfaces and code editors, aiming to make AI-assisted development more interactive. The episode also covers Answer’s Public Benefit Corporation structure designed to resist hostile takeovers, and critiques decoder-only hype while advocating for encoder-decoder and state-space models.

[LLM Paper Club] Llama 3.1 Paper: The Llama Family of Models

Jul 29, 2024 · 1:23:46

This episode examines Meta's Llama 3.1 paper, detailing the 405B dense model, its scaling laws grounded on the ARC reasoning benchmark rather than perplexity, and the decision to train on 15 trillion tokens. Vibhu explains the training infrastructure: 16,000 H100s over 54 days with 419 interruptions, 78% from GPU hardware failures. Eugene Yan walks through the synthetic data pipeline—using Llama 2 for filtering, stepwise reward models, and Monte Carlo tree search to improve reasoning traces. Hassan shares building LlamaTutor.com with Together API, serving 4,000 visitors and 5,900 requests for about $12. The group also discusses quantization trade-offs (larger models degrade less), inference provider variability (Groq's non-deterministic temperature zero), and compares Llama 405B's performance to GPT-4o and Claude.

State of the Art: Training 70B LLMs on 10,000 H100 clusters

Jun 25, 2024 · 1:32:05

Josh Albrecht, CTO of Imbue, and Jon Frankle, Chief AI Scientist of Databricks, detail the challenges of training 70B models on 10,000 H100 clusters, from stolen Infiniband cables to GPUs silently returning wrong math. Imbue open-sources infrastructure scripts, health checks, the CARBS hyperparameter optimizer, and cleaned versions of 11 benchmarks plus 450,000 human judgments on ambiguity. Databricks released DBRX, a 132B-parameter mixture-of-experts model with 36B active per token, and a text-to-image model trained exclusively on Shutterstock's curated dataset. They discuss typical 3% weekly hardware failure rates, the difficulty of MoE training pushing network bandwidth, and why eval datasets need human scrutiny—most benchmarks are saturated once ambiguous examples are removed. Both emphasize that success requires deep understanding from firmware to final loss, not just fancy libraries.

How to train a Million Context LLM — with Mark Huang of Gradient.ai

May 31, 2024 · 1:12:14

Mark Huang of Gradient.ai explains how his team extended Llama 3 to a 1 million token context window using curriculum learning, RingAttention, and EasyContext, achieving near-perfect GPU utilization. They employed theta scaling from the RoPE paper to interpolate positional encodings, trained on carefully curated datasets including SlimPajamas and synthetic data from GPT-4, and validated with benchmarks like Ruler and needle-in-a-haystack. Huang details the trade-offs between full fine-tuning and LoRA adapters, the challenges of pushing to 4M tokens (degradation from floating-point precision limits), and why long context matters for state management across sessions and grounding multimodal inputs. He calls for community collaboration on long-context evaluations and pairwise multimodal datasets.

LLM Asia Paper Club Survey Round

May 22, 2024 · 55:25

The episode surveys four recent papers on LLM reasoning, uncertainty, interpretability, and efficiency. 'Let's Think Dot by Dot' shows that filler tokens (dots) inserted between input and output enable hidden computation, outperforming no-token baselines on tasks like 3-Sum and 2-Sum Transform. 'Uncertainty Estimation' trains a random forest on hidden-layer activations to predict response confidence, achieving higher AUC than unsupervised methods on Q&A and translation. 'Monosemanticity' uses sparse autoencoders to identify interpretable features (e.g., a DNA-detection feature) in a toy transformer, advancing mechanistic interpretability. 'Medusa' attaches multiple prediction heads to the final hidden state to speculate future tokens, enabling faster decoding without a separate draft model and training in five hours on 60K samples.

Personal AI Meetup - Bee, BasedHardware, LangChain LangFriend, Deepgram EmilyAI

Apr 6, 2024 · 58:54

This episode features Damien Murphy of Deepgram, Ethan of Owl/Bee, and Harrison of LangChain demonstrating how to build personal AI with real-time voice bots, wearable life-recording devices, and memory-enhanced journaling apps. Damien shows building a voice bot with subsecond latency using Deepgram, OpenAI, and open source code, costing about 6.5 cents per five-minute call. Ethan presents his Owl wearable that continuously records audio, triggers actions via hot word 'Scarlett,' and discusses challenges in adding vision and open source adoption. Harrison introduces LangFriend, a journaling app using conversational, semantic, and knowledge graph memory, referencing the Generative Agents paper for recency and importance weighting. The episode also highlights open source projects Whomane, Friend, and ADeus, arguing that hardware, voice, and memory are all necessary components for personal AI.

A Brief History of the Open Source AI Hacker - with Ben Firshman of Replicate

Feb 28, 2024 · 1:21:09

Ben Firshman, CEO of Replicate, explains how the inference platform grew from a research reproducibility tool into a 2M-user API business by embracing the generative image community and treating open-source AI as a hacker-friendly ecosystem. They accidentally discovered their API when a user reverse-engineered their web form, leading to their first $1k/month customer. Cog, their container standard for ML models, was born from lessons at Docker and the need to make models tinkerable. Ben argues that fine-tuning's low cost makes open-source models sustainable, and that AI engineers (orders of magnitude more than ML engineers) just need to start playing with models. He also discusses GPU scarcity, preferring sustainable pricing over price wars, and reveals that demand is not outpacing supply thanks to aggregating demand.

Truly Serverless Infra for AI Engineers - with Erik Bernhardsson of Modal

Feb 19, 2024 · 1:08:58

Erik Bernhardsson, creator of Annoy and Luigi, founded Modal to build a 'postmodern data stack' with a self-revisioning runtime that eliminates container cold starts and lets developers attach GPUs with a Python decorator. His serverless platform can fan out to thousands of GPUs within seconds, becoming a natural fit for GenAI workloads like Stable Diffusion and fine-tuning (e.g., RAMP fine-tuning 100 models in parallel). Modal differentiates from Replicate by targeting custom models and workflows, and from Modular by offering a managed cloud service rather than licensed software. Erik argues that buying hardware is inefficient for startups and that infrastructure founders must be willing to spend years on load balancing, page faults, and DNS. He shares how his IOI Gold Medal background shapes Modal's talent culture, hiring competitive programmers for complex scheduling and bin-packing problems.

Building an open AI company - with Ce and Vipul of Together AI

Feb 8, 2024 · 1:15:18

Together AI co-founders Vipul Ved Prakash and Ce Zhang explain why openness is core to their mission, detailing their journey from Apple (Vipul) and Stanford research (Ce) to building an open AI platform. They discuss RedPajama’s evolution into a modular dataset with 40 quality signals, the need for 5,000 tokens/second inference speed, and their investment in state space models like Mamba and Hyena as alternatives to transformers. The company, which runs 7,000-8,000 GPUs (mostly H100s), sees training as a larger workload than inference, with fine-tuning driving top models. They advocate for independent inference benchmarks, publish open research like FlashAttention, and keep some software proprietary. With 38 employees and 45% researchers, they are hiring across the stack, from CUDA to DevOps.

The Four Wars of the AI Stack - Dec 2023 Recap

Jan 26, 2024 · 1:20:58

Swyx and Alessio recap December 2023 by framing the AI landscape as four wars: the Data War (NYT lawsuit demanding destruction of GPTs, OpenAI's data partnerships, synthetic data from DeepMind); the GPU/Inference War (Mixtral price dropping 90% to $0.27/million tokens, benchmark drama between Anyscale and Together, alternative architectures like Mamba); the Multimodality War (Midjourney reaching $200M ARR, ElevenLabs hitting unicorn status, OpenAI and Google building god models); and the RAG/Ops War (LangChain vs LlamaIndex, Pinecone's $750M valuation, Qdrant serving OpenAI and Anthropic internally). They argue agents and open source are not active wars, and predict 2024 will see AI finally reach production, with hardware like Rabbit R1 and Tab capturing unique context.

The AI-First Graphics Editor - with Suhail Doshi of Playground AI

Jan 2, 2024 · 1:09:23

Suhail Doshi, co-founder of Mixpanel and founder of Playground AI, argues that image generation is still in a GPT-2 moment and that training open-source foundation models from scratch is necessary to unlock real utility, exemplified by Playground v2's 2.5x preference over Stable Diffusion XL on a 1K prompt benchmark. He explains the pivot from Mighty (cloud-streamed browser) to AI, driven by the belief that shifting compute elsewhere aligns with AI's parallel computation. The episode covers Playground's unique UI—not just a prompt box but a full canvas with preview rendering, seed control, and style filters—and the difficulty of balancing safety with artistic expression, especially around NSFW content. Suhail discusses the under-investment in graphics AI compared to language, the open-source community's role, and his choice to release pre-trained weights for academic research. He also shares lessons from running GPU infrastructure (harder for training than inference) and advises founders to follow curiosity and build projects rather than relying on books.

The State of Silicon and the GPU Poors - with Dylan Patel of SemiAnalysis

Dec 5, 2023 · 1:07:55

Dylan Patel of SemiAnalysis argues the 'GPU poor vs GPU rich' divide defines AI, with Google's TPU ramp (millions of units) dwarfing others and Nvidia selling over a million H100s this year. He explains that training costs are irrelevant compared to inference costs, and that memory bandwidth is the key bottleneck for LLM inference (e.g., Llama 70B needs ~2.1 TB/s for human reading speed). Patel advises GPU-poor players to focus on on-device innovations like speculative decoding (Medusa), asynchronous training, or fine-tuning for edge use—not on gaming benchmarks or fine-tuning small models. He details why AI hardware startups (Cerebras, Graphcore) bet wrong on on-chip SRAM, while Google’s TPU/Broadcom partnership and Nvidia’s yearly cadence make alternatives tough. He also warns that rebuilding the semiconductor supply chain in the US is unfeasible due to its fragmentation, and that safety through obscurity doesn't work—open innovation is better.

FlashAttention-2: Making Transformers 800% faster AND exact

Aug 3, 2023 · 1:04:06

Tri Dao, creator of FlashAttention and FlashAttention-2, explains how his I/O-aware algorithm makes attention 2x faster by fusing kernels and using online softmax, achieving near-matrix-multiply efficiency. He argues that transformer alternatives like state space models and RNNs (e.g., RWKV) could surpass transformers for long sequences and high-throughput generation, though attention still dominates. Dao discusses the hardware lottery, where NVIDIA's CUDA ecosystem entrenches transformers, and advocates for open-source AI, praising Meta's Llama 2 for shifting enterprise adoption despite its restrictive license. He emphasizes that understanding both algorithms and systems is key to scaling AI, and that academia should pursue risky bets that industry cannot.

Ep 18: Petaflops to the People — with George Hotz of tinycorp

Jun 20, 2023 · 1:23:24

George Hotz of tinycorp argues that a simplified, open-source ML framework (tinygrad) can democratize AI compute and replace NVIDIA, Google, and AMD. Tinygrad uses 25 ops (vs. XLA's 250) and fuses kernels automatically, achieving 2x speed on Qualcomm GPUs over their library. Hotz reports AMD kernel panics fixed after emailing CEO Lisa Su, but laments AMD's open-source as 'dumped on GitHub.' For hardware, tinybox is a six-GPU desktop at 350W per GPU, aiming for 5x better price-performance than NVIDIA H100 for 90% of training. He criticizes OpenAI's secrecy, revealing GPT-4 is an 8-way mixture of 220B models, and cites the Bitter Lesson. He also outlines FLOPcoin using hardware identity to prevent cheating, and his goal of an AI girlfriend as the third company, merging humans via data rather than implants.