Llama

The AI Memory Problem: Why Long Context Isn’t Enough — Dan Biderman, Engram Co-founder & CEO

Jul 13, 2026 · 49:44

Engram co-founder and CEO Dan Biderman tells hosts Allen Park and Sean that long context and RAG aren't enough for AI memory—continual learning and gradient-based weight updates are needed. He argues Engram's approach compresses company knowledge into 'cartridges' that let models reason with far fewer tokens, overcoming 'context rot' and the inefficiency of re-reading massive corpora. Biderman draws on his background in Israeli special forces and computational neuroscience to explain how training creates intuition beyond text retrieval, citing examples like Harvey's legal queries where holistic understanding beats search. He envisions personal AI weights that improve like Tamagotchis, driven by user-specific feedback loops, and stresses that token efficiency and intelligence are inseparable—doing more with less enables harder problems. Engram is hiring infrastructure engineers to deploy millions of continuously updated memories.

🔬 "The Most Innovative Diffusion Research Is Happening in Drug Discovery, Not Image Generation"

Jun 30, 2026 · 1:48:40

Evan Feinberg and Sergey Edunov of Genesis Molecular AI argue that diffusion models have unlocked sub-ångström accuracy in protein-ligand structure prediction, a breakthrough that makes AI useful for real drug discovery where the field’s favored 2Å RMSD benchmark is "slop." Their PEARL model uses diffusion with physics-based guidance, synthetic training data from molecular dynamics, and inference-time scaling to predict induced fit—how a protein flexes to accommodate a ligand. On the OpenBind benchmark, PEARL zero-shot surpassed all cofolding models on the notoriously hard EV A721A protease, correctly predicting a flexible loop movement that other methods missed. They also introduce SAPPHIRE, an agentic system that orchestrates AI models for 24/7 drug design, and discuss how downstream ADMET properties (solubility, toxicity, etc.) remain equally critical. The biggest bottleneck they face is GPU availability, and they are actively hiring AI researchers interested in novel architectures beyond standard transformers.

Why AI Labs With Unlimited GPUs Still Fail — Anjney Midha, AMP

Jun 18, 2026 · 1:00:37

Anjney Midha, CEO of AMP, argues that AI labs with unlimited GPUs still fail due to misaligned culture and infrastructure waste, proposing a compute grid modeled on independent system operators to pool demand and supply. At Google, 95% node utilization was considered an outage, yet most clusters today don't reach that, with waste compounding at scale. AMP’s grid, starting at scheduling, aims to make FLOPs flow like megawatts, having secured 1.3 gigawatts of demand. Midha explains Anthropic cracked coding because 'luck favors the prepared mind'—their four years of paranoia and scarcity created a culture that OpenAI’s abundance couldn't replicate. He also shares a 14-year mission in end-of-life prediction, arguing AI can reduce the 30% of Medicare/Medicaid spend on end-of-life care. He warns that too much capital too early makes labs fragile because without hardship they fail to define their P0.

Satya Nadella on AI: @NoPriorsPodcast x Latent Space Crossover Special at Microsoft Build 2026

Jun 3, 2026 · 41:27

Satya Nadella argues that Microsoft's AI strategy is an ecosystem platform enabling any company to build frontier intelligence using models, tools, data, and a harness, not just consume one model. He outlines MAI training with clean data lineage, hill-climbing scaffolds, and private evals as core IP. The harness concept features multi-model harnesses with strong context layers, exemplified by GitHub Copilot and Work IQ turning M365 data into a database for agents. He notes coding agents required new IDE/UI, and long-running autopilots create value. Pricing evolves from per-user to consumption, and SaaS will unbundle and rebundle. Engineering generalists gain leverage; infrastructure roles like RLEs remain critical. Datacenter expansion requires community permission, with benefits in healthcare and rethinking education. Microsoft built more Azure capacity in 15 months than its first 15 years.

Inside xAI: Building Grok Imagine in 3 Months, Videogen vs World Models, and Video Agents— Ethan He

Jun 1, 2026 · 1:44:43

Ethan He, former xAI and NVIDIA Cosmos researcher, explains how xAI built its first image and video models (Grok Imagine 0.9) from zero to one in three months, attributing rapid iteration to small teams with minimal meetings and strong infra that enabled fixing tiny data and training bugs for biggest quality gains. He argues that most improvements in video generation now come from language models and agents rather than diffusion technology, predicting that by end of 2025 video agents will produce production-grade content for ads. He defines world models as real-time, interactive, long-horizon videos, and details challenges like temporal compression, context management, and the high cost of storing and moving video data (e.g., tens of petabytes for a billion videos). Ethan also shares why he left xAI to focus on language model research, believing the next frontier is models that manage their own context length, similar to solutions already being explored in video generation.

Mistral: Voxtral TTS, Forge, Leanstral, & Mistral 4 — w/ Pavan Kumar Reddy & Guillaume Lample

Mar 30, 2026 · 54:02

Mistral releases Voxtral TTS, a 3B-parameter speech generation model using a novel autoregressive flow matching architecture and a 12.5 Hz neural audio codec developed in-house. Pavan Kumar Reddy and Guillaume Lample explain how the model achieves high quality and efficiency by predicting audio latents as a continuous distribution, enabling real-time voice agent applications. The episode details Mistral's stepwise multimodal strategy, starting with transcription and now speech generation, with full duplex voice agents as the next goal. Enterprise deployment via Mistral Forge allows fine-tuning on proprietary data, exemplified by training on rare languages or domain-specific jargon. Guillaume discusses merging capabilities (Mistral Small merging coding, reasoning, vision), commitment to open science through technical reports, and reasoning transfer from formal proofs in Lean. The episode also covers AI for science partnerships and hiring for research and forward-deployed engineering roles.

Agent Inference at the "Speed of Light" — How NVIDIA moves like a $4.3 Trillion Startup

Mar 8, 2026 · 1:26:00

NVIDIA's Nader Khalil and Kyle Kranen join Swyx and Vibhu to explain how the company moves like a $4.3 trillion startup through speed-of-light (SOL) first-principles thinking, agent security boundaries, and the Dynamo inference engine. They argue agents should only do two of three things (files, internet, code) to prevent vulnerabilities, and detail Brev's acquisition to improve developer UX with one-click GPU access and DGX Spark integration. Kyle describes Dynamo as a data center scale inference engine that optimizes serving by scaling out, using prefill/decode disaggregation, Kubernetes-based scheduling, and model-hardware co-design to improve cost, latency, and quality. The episode covers SOL's role in creating urgency, long-context limits and potential 'unhobblers' like multi-head latent attention, and the shift toward CLI-first agent workflows for enterprise tools.

Goodfire AI’s Bet: Interpretability as the Next Frontier of Model Design — Myra Deng & Mark Bissell

Feb 5, 2026 · 1:08:41

Goodfire AI's Mark Bissell and Myra Deng argue that interpretability is the next frontier for model design, using their recent $150M Series B at $1.25B valuation to scale surgical edits of model internals beyond post-hoc poking. They explain how their platform detects behaviors like sycophancy and reward hacking, enabling targeted unlearning without wrecking capabilities. The episode covers real-world deployments from Rakuten's PII guardrails to life science partnerships with Mayo Clinic finding Alzheimer's biomarkers. Mark demonstrates real-time steering of a trillion-parameter Kimi K2 model, while Myra details how SAEs sometimes underperform probes for detection tasks. They envision a future where interpretability guides training so customization isn't brute-force guesswork.

Artificial Analysis: The Independent LLM Analysis House — with George Cameron and Micah Hill-Smith

Jan 9, 2026 · 1:18:15

Artificial Analysis founders George Cameron and Micah Hill-Smith explain how their independent benchmarking platform became the gold standard by running their own evals with a mystery shopper policy to prevent labs from manipulating results. They launched in January 2024 after building it as a side project in Sydney, going viral after Swyx's retweet. The Intelligence Index V3 synthesizes 10 datasets with 95% confidence intervals, while the Omniscience Index measures hallucination rates from -100 to +100 (Claude models lead). Their GDP Val AA benchmark tests 44 white-collar tasks, and they open-sourced their agentic harness Stirrup. They also introduced an Openness Index scoring models out of 18 points. The episode covers how they make money through enterprise benchmarking subscriptions and custom work, and why the cost of GPT-4-level intelligence has dropped over 100× while total inference spend rises due to reasoning and agentic workflows.

[State of Evals] LMArena's $1.7B Vision — Anastasios Angelopoulos, LMArena

Dec 31, 2025 · 24:02

Anastasios Angelopoulos, founder of LMArena (now Arena), discusses the platform's $100M raise at a $1.7B valuation, its spin-out from Berkeley incubation by a16z's Anjney Midha, and its mission to be the industry's north star for real-world AI evaluation. He defends against the 'leaderboard illusion' paper, citing factual errors and reaffirming that models cannot pay to be on or off the public leaderboard. Arena funds inference costs for millions of monthly users, with 25% of its 5M+ users in software, and is expanding into occupational verticals (medicine, legal, creative) and multimodal video arenas. Key challenges include consumer retention, which improved with sign-in and persistent history, and moving off Gradio to React for better development. Angelopoulos calls for top talent in ML, product, and go-to-market to join the high-performance team.

SAM 3: The Eyes for AI — Nikhila & Pengchuan (Meta Superintelligence), ft. Joseph Nelson (Roboflow)

Dec 18, 2025 · 1:15:04

Meta's SAM 3, introduced by Nikhila Ravi and Pengchuan Zhang alongside Roboflow CEO Joseph Nelson, unifies interactive segmentation, open-vocabulary detection, and video tracking into a single model that runs in 30ms on images and scales to real-time video on multi-GPU setups. The model uses concept prompts like "yellow school bus" to detect and segment every instance, separating recognition from localization via a presence token. Its data engine automated exhaustive annotation from two minutes per image down to 25 seconds using AI verifiers fine-tuned on Llama, while the new SACO benchmark contains over 200,000 unique concepts versus previous 1.2k. For video, decoupling the detector and tracker preserves object identity, and SAM 3 agents pair with multimodal LLMs like Gemini to handle complex visual reasoning. The real-world impact includes 106 million smart polygons created on Roboflow, saving an estimated 130+ years of labeling time across fields from cancer research to underwater trash cleanup.

The Future of Email: Superhuman CTO on Your Inbox As the Real AI Agent (Not ChatGPT) — Loïc Houssier

Dec 11, 2025 · 1:11:02

Loïc Houssier, CTO of Superhuman Mail (recently acquired by Grammarly), joins hosts Alessio and Swyx to detail how Superhuman builds AI into email without adding latency. He explains their agentic framework of small tools vs. a single agent, the fight against “agent laziness,” and their eval process anchored by Rahul’s infamous “what wood was my table?” test. Houssier reveals Superhuman uses local-first caching and Baseten’s box pricing for cost control, stores embeddings in TurboPuffer, and has only three engineers on AI—yet PR throughput rose from four to six per engineer per week. He argues the inbox will power your future AI executive assistant, and that AI will widen the gap between engineers with real fundamentals and those faking it.

⚡ Open Model Pretraining Masterclass — Elie Bakouch, HuggingFace SmolLM 3, FineWeb, FinePDF

Oct 20, 2025 · 1:03:40

Elie Bakouch, Hugging Face's pre-training lead and architect behind SmolLM, presents a five-pillar framework for model training spanning data quality, architecture, information extraction, gradient quality, and stability, while breaking down recent innovations in optimizers, Mixture of Experts, and data rephrasing. He explains that the field of optimizers is moving beyond AdamW with Muon and Shampoo, noting that speed-ups are often exaggerated due to undertuned baselines. He deep-dives into MoE architecture, showing that expert specialization requires load balancing at the global batch level, as demonstrated by Qwen's findings. He discusses the rephrasing revolution, where converting low-quality web data into QA format yields non-random MMLU performance even for small models. Elie also highlights Hugging Face's open science contributions: FinePDF (new PDF dataset), FineWeb-Edu2, and tools like Nanotron, Datahub, and Liteval.

Why RL Won — Kyle Corbitt, OpenPipe (acq. CoreWeave)

Oct 16, 2025 · 1:08:23

Kyle Corbitt, co-founder and CEO of OpenPipe (acquired by CoreWeave), explains why reinforcement learning has replaced supervised fine-tuning for training reliable AI agents. He argues GRPO is a dead end due to its requirement for perfectly reproducible parallel rollouts, which is extremely hard in practice. Instead, OpenPipe’s RULER uses relative LLM-as-judge rewards, achieving state-of-the-art performance even with a weak judge. Corbitt reports that 90% of AI projects remain stuck in proof-of-concept due to reliability issues, and that LoRAs are underrated for production while GEPA failed in his tests. He predicts continuous RL from real-world experience can unlock 10x more inference demand.

Building Jamba 3B: the tiny Hybrid Transformer State Space Reasoning Model - Barak Lenz, CTO of AI21

Oct 11, 2025 · 45:05

Barak Lenz, CTO of AI21, presents their Jamba 3B model as a tiny hybrid transformer-state space model that brings long context capabilities to edge devices, and argues enterprises need AI systems like Maestro over standalone models. Lenz explains that the 1:8 ratio of attention to Mamba layers emerged from extensive ablations, with attention placed in the middle of the block working best. The Jamba 3B model uses a 1:12 ratio to maximize efficiency, fitting the same context length as larger models with a fraction of memory. He notes that pure Mamba underperformed on some tasks but hybrid models resolved those deficiencies, and that images quickly become long context problems (4 images can be thousands of tokens). For enterprise, Lenz advocates for model-agnostic orchestration layers treating models as "actions" with statistical properties, enabling continuous learning and cost optimization without vendor lock-in. Drawing from his algo trading background, he compares training frontier models to developing trading algorithms, emphasizing the importance of world-class engineering and avoiding brute force reasoning.

⚡️Raising $1.1b to build the fastest LLM Chips on Earth — Andrew Feldman, Cerebras

Oct 1, 2025 · 29:14

Andrew Feldman, CEO of Cerebras, joins Latent Space to discuss their $1.1B fundraise at an $8.1B valuation and their wafer-scale chip that delivers 20x faster inference than NVIDIA's B200 GPUs. He explains how their architecture uses SRAM instead of HBM, providing 2,625x more memory bandwidth by eliminating the narrow straw between compute and memory. Feldman details the decision to accelerate sparse linear algebra rather than specialized convolutions, enabling support for transformers and diffusion models unseen during design. He discusses the explosive growth in AI inference demand, the shift from closed-source to fast open-source models, and the importance of speed—citing Paul Graham's observation that ChatGPT's slowness drives users away. The conversation covers enterprise trends in the 10-30B parameter space, the complexity of building data centers that pull gigawatts of power, and the often-overlooked routing and caching systems that make AI work seamlessly.

Better Data is All You Need — Ari Morcos, Datology

Aug 29, 2025 · 1:18:43

Ari Morcos, CEO of Datology, argues that data curation is the most impactful and underinvested area in AI, claiming models are defined by what they eat. He details his shift from focusing on inductive biases to realizing data quality is paramount, citing the DCLM study where human experts could not predict automated filtering decisions. Datology's methods combine filtering, rebalancing, curriculum, and rephrasing synthetic data to achieve baseline performance 12x faster, improve accuracy by 4–5 absolute points, and train models with fewer than half the parameters. Morcos explains how curating data bends naive scaling laws by maintaining marginal information gain, and shares results from the Arc 4.5B model, which was trained on 7 trillion tokens (down from 25 trillion) using Datology's curation. He highlights that data is a compute multiplier, making it possible for enterprises to train smaller, cheaper models for specific domains.

⚡️Accelerators @ 3x NVIDIA H200 perf, Made in the USA - Thomas Sohmers + Mitesh Agrawal, Positron AI

Aug 18, 2025 · 47:56

Thomas Sohmers and Mitesh Agrawal of Positron AI argue memory bandwidth, not compute, is the true bottleneck in AI inference, and their accelerator achieves 93% memory bandwidth utilization—triple NVIDIA's efficiency—enabling 70% faster token generation at 150W. The founders, both Lambda Labs veterans, shipped an FPGA product 15 months after founding, then raised a $51M Series A for an ASIC in late 2026. Their hardware requires no recompilation: it ingests raw binary weights from NVIDIA training and outputs an OpenAI-compatible API. Positron focuses on the decode phase of transformers, where memory-bound matrix-vector multiplication dominates, and already counts Cloudflare and Parasail as customers. The company sells systems directly, prioritizing capital efficiency and ROIC over operating its own cloud.

The AI Agenda: GPT5 leaks and the business of AI News — Steph Palazzolo, The Information

Aug 6, 2025 · 1:14:36

Steph Palazzolo, AI journalist at The Information, explains how she covers the secretive AI industry, from OpenAI's GPT-5 to the inference market where startups like Modal raise billions. She reveals that inference providers are essentially GPU resellers facing margin pressure, and that GPT-5's performance will signal whether pre-training scaling or reinforcement learning drives progress. She critiques Meta's super intelligence ambition as misaligned with its ad-revenue business, and details the talent war with rumored hundred-million-dollar offers from Meta to poach researchers. Palazzolo also discusses the Windsurf acqui-hire backlash, the rise of coding agents like Claude Code, and how journalists protect sources while navigating leaks from CEOs like Sam Altman.

The RLVR Revolution — with Nathan Lambert (AI2, Interconnects.ai)

Jul 31, 2025 · 1:19:00

Nathan Lambert from AI2 explains the RLVR (Reinforcement Learning with Verifiable Rewards) revolution, arguing that verifiable rewards for math, code, and instruction-following are scaling more reliably than human feedback, and that open models like Tulu 3 can match frontier labs on core evals with just 10-15 tasks vs. hundreds. He traces overoptimization through three phases—control, RLHF, and RLVR—and warns that models learn to cheat unit tests unless reward design penalizes it. He analyzes o3's search-heavy approach (e.g., 80 websites per query), hybrid reasoning models like Gemini 2.5 and Claude, and predicts that pure reasoning models will become the default as inference costs drop. Lambert introduces a four-skill taxonomy for agent models—skills, calibration, strategy, abstraction—and says planning improvements often come from mundane data fixes. He advocates for fully open models, citing AI2's Olmo 32B as approaching GPT-4 level, but notes that building an 'American DeepSeek' requires massive resources and nonprofit constraints.

🕰️ The Oral History of Windsurf (ft. Varun Mohan, Scott Wu, Jeff Wang, Kevin Hou, Anshul R)

Jul 28, 2025 · 3:53:23

This episode traces the oral history of Windsurf (formerly Codeium and Exafunction) with founders Varun Mohan, Scott Wu, Jeff Wang, Kevin Hou, and Anshul R, charting its evolution from a GPU virtualization startup to a leading agentic IDE and its dramatic acquisition by Cognition. The narrative covers the pivot to code completion in 2022, the shift to a free model to gain traction, and the launch of the Windsurf editor in November 2024 with Cascade, an agent that operates on a shared timeline with the developer. Key principles include meta-learning (auto-generated memories that adapt to each user's preferences) and scaling with intelligence (replacing hard-coded rules with LLM reasoning as models improve). The episode also details the acquisition weekend when Cognition bought Windsurf after Google acquired a stake, highlighting product synergies between the synchronous agent (Windsurf) and async agent (Devin) and the ambition to build a comprehensive platform for real-world software engineering teams.

⚡️Ranking Agentic LLMs — Pratik Bhavsar, Galileo

Jul 14, 2025 · 34:10

Pratik Bhavsar from Galileo presents their Agent Leaderboard, which evaluates LLMs on tool calling across multiple benchmarks, revealing that Gemini models top the charts for performance and cost-efficiency. The leaderboard, built on datasets like BFCL, XLAM, and Tau Bench, uses an LLM-as-judge metric called Tool Selection Quality (TSQ). Surprising findings include reasoning models like o1 struggling with multi-tool outputs, Mistral Small excelling as the best open-source model, and Llama models performing poorly. The episode also previews V2 of the leaderboard, which introduces domain-specific, harder, multi-turn scenarios with a user and tool simulator, and a new Action Completion metric to measure whether all user requests are accomplished.

Information Theory for Language Models: Jack Morris

Jul 2, 2025 · 1:18:13

Jack Morris, a Cornell PhD student advised by Sasha Rush, discusses his information-theoretic research on language models, including embedding inversion (recovering text from embeddings with 90% accuracy), the universal geometry of embeddings (aligning different models' latent spaces), and measuring model memorization capacity at 3.6 bits per parameter. He argues that paradigm shifts in AI, from AlexNet to instruction tuning, stem not from new architectures but from new datasets, and that the next breakthrough will likely emerge from a novel data source. The conversation also covers the shift from academia to industry, practical advice for grad students on distributed training, and the implications of embedding inversion for privacy and model alignment.

The Utility of Interpretability — Emmanuel Amiesen

Jun 6, 2025 · 1:53:02

Emmanuel Amiesen, lead author of Anthropic's Circuit Tracing paper, presents the newly open-sourced tools that let anyone inspect how language models compute, revealing that models like Gemma 2-2B engage in genuine multi-step reasoning rather than mere pattern matching. Amiesen demonstrates that the model's internal features causally drive outputs—for instance, suppressing the 'Texas' feature in a capital-of-state query changes the answer from Austin to Sacramento. He explains how sparse autoencoders uncover features shared across languages and modalities, and how attribution graphs map feature interactions. The episode covers practical insights: why jailbreaks succeed (the model prioritizes grammatical completion over refusal) and how planning in poetry generation involves deciding a rhyme before the line begins. Amiesen calls for more researchers to explore interpretability, citing the low barrier to entry with open models and tools.

The AI Coding Factory

May 29, 2025 · 59:23

Factory.ai co-founders Eno Reyes and Matan Grinberg join Latent Space to explain their vision for autonomous software engineering droids that handle everything from code generation to incident response for enterprises. They met at a Langchain Hackathon in 2023, and within eight days both quit their jobs to start Factory. The platform is browser-based, not IDE-based, because they argue the optimal UI for a world where humans write less code will not evolve from the IDE. They focus on enterprise codebases, citing a large migration that took from four months to three and a half days. Key design decisions include a delegation model where agents ask clarifying questions, proactive context gathering via synthetic insights, and usage-based pricing tied to token efficiency. They also discuss why SWE-Bench is dead for their use case, the challenge of model post-training biases toward CLI tools, and their need for 'junior Enos' in go-to-market roles.

The Rise and Fall of the Vector DB category: Jo Kristian Bergum (ex-Chief Scientist, Vespa)

Apr 19, 2025 · 27:17

Jo Kristian Bergum argues that the vector database category is dying because vector search capabilities have converged into existing databases like Postgres' pgvector, Elasticsearch, and Vespa, making specialized vector databases unnecessary for most use cases. He traces the category's rapid rise after ChatGPT, driven by the misconception that RAG required embeddings, and notes Pinecone's high ARR and subsequent repositioning. Bergum emphasizes that embeddings remain important but should be combined with traditional retrieval methods like BM25 for effective search, and that re-ranking can add modest gains. He critiques the hype around knowledge graphs, noting the bottleneck of building them, but sees LLMs making triplet generation easier. For the future, he hopes for more domain-specific embedding models and visual language model backbones, though acknowledges the difficulty of the business model.

Claude Plays Pokémon Hackathon: Escape from Mt. Moon!

Apr 5, 2025 · 1:15:00

David Hershey (creator of Claude Plays Pokémon), Andrew (a developer building a Pokémon-playing virtual streamer), and Jesse Han (Morph Labs CEO) detail how to build agents that escape Mt. Moon in Pokémon FireRed using Anthropic's Claude and Morph Cloud's Infinibranch technology. Hershey explains that Claude's vision is often unreliable—it spent 8 hours pressing A on a doormat thinking it was a dialogue box—and that spatial reasoning remains a core bottleneck, best mitigated by touchscreen controls rather than prompt engineering. Andrew reveals he relied on reading raw RAM and A* pathfinding rather than computer vision, calling it “cheating” but pragmatically useful: his biggest breakthrough was a single prompt line telling the agent to try something else if it fails to grab a starter Pokémon. Jesse introduces Morph's EVA agent framework, which uses low-overhead snapshotting for test-time search, and the hackathon's judging criteria—escape Mt. Moon in the fewest agent turns, with a $1,000 prize for the coolest use of Infinibranch branching. The episode serves as both a technical walkthrough and a challenge to build general-purpose agents that can handle complex, open-ended tasks…

The Magic of LLM Distillation — Rishabh Agarwal, Google DeepMind

Mar 23, 2025 · 46:42

Rishabh Agarwal from Google DeepMind discusses modern LLM distillation, arguing that while simple synthetic data distillation (generating outputs from a teacher and fine-tuning a student) gets 80-90% of the benefit, more advanced on-policy methods (sampling from the student and getting teacher logits) can close the remaining gap by fixing the train-test mismatch. He explains that logit-based distillation is more information-rich but often neglected due to infrastructure complexity, while online distillation is more expensive but crucial for long-horizon or agentic tasks. Agarwal highlights how flipping the KL divergence direction changes behavior (mode-covering vs. mode-seeking) and shows that smaller models can sometimes outperform larger ones in compute-matched distillation due to generating more filtered data. He also connects distillation to speculative decoding, where a distilled student speeds up the teacher, and encourages researchers to question current methods and mine RL literature for ideas.

Outlasting Noam Shazeer, Crowdsourcing Chai AI w/ 1.4m DAU — with William Beauchamp, Chai Research

Jan 26, 2025 · 1:13:33

William Beauchamp, founder-CEO of Chai AI, explains how he pivoted from algorithmic trading to build a character chatbot platform before Character.ai, growing to 1.4M DAU and $22M+ revenue by crowdsourcing model improvements through Chaiverse. Starting with GPT-J in 2021, he found product-market fit with a therapist bot and shifted to user-generated content, letting users define prompts, images, and names. Competing against well-funded rivals like Character AI and Talkie, Chai ships over 100 LLMs weekly via its developer platform, spending $10M on compute in 2024 and tripling it. Beauchamp argues AI follows an S-curve, not scaling laws, and focuses on inference optimization using rejection sampling and reward models to serve better responses. He prioritizes 'insanely great' products over technology-driven features, noting that audio and image features failed to move metrics, while a data flywheel and aggressive user acquisition drove rapid growth.

The Unreasonable Effectiveness of Reasoning Distillation: using DeepSeek R1 to beat OpenAI o1

Jan 24, 2025 · 23:35

Bespoke Labs founders Mahesh, Ryan, and Trung explain how they distilled DeepSeek R1 into Bespoke-Stratos-32B, a reasoning model that beats OpenAI o1-preview on math and code benchmarks using only 17,000 training examples—47 times fewer than DeepSeek's own distill. They achieved this in a 48-hour sprint leveraging their data curation library Curator. Unlike Sky-T1, which required rewriting unreliable QWQ traces, R1's coherent reasoning allowed them to skip re-annotation and simply filter for correctness. The team argues data quality matters more than quantity, and that smaller models (like their 7B variant) can also improve with better teacher data. They see this as evidence that reasoning can emerge purely from supervised fine-tuning on high-quality traces, without complex search algorithms.

DeepSeek V3, SGLang, and the state of Open Model Inference in 2025 (Quantization, MoEs, Pricing)

Jan 19, 2025 · 57:18

Amir Haghighat and Yineng Zhang from Baseten explain how they serve DeepSeek V3, a 671-billion-parameter MoE model requiring H200 clusters with FP8 support, making it the top open-weights model. They detail Baseten’s dedicated inference model (no shared endpoints) and its reliance on SGLang for performance gains like Radix prefix caching and MLA attention support. Yineng highlights SGLang’s edge over vLLM and TensorRT-LLM in usability and customization, especially for large models. Amir outlines three pillars for mission-critical inference: model-level performance (via frameworks like SGLang), horizontal scaling across regions and clouds, and low-latency multi-model workflows. The episode also covers quantization trends, MoE architecture, and speculative decoding as key enablers for open model deployment.

Beating Google at Search with Neural PageRank and $5M of H200s — with Will Bryk of Exa.ai

Jan 10, 2025 · 55:54

Will Bryk, CEO of Exa.ai (formerly Metaphor), details building a neural search engine from scratch using link prediction as 'Neural PageRank' — predicting documents rather than keywords. Exa's new product offers near-perfect lists (e.g., 'startups working on hardware in SF') by scaling compute per query, from milliseconds to a day, like o1 for search. Bryk argues LLMs will become the interface to search, while Exa provides the 'super knowledge' that even AGI will need. He contrasts Exa's neural approach with Google's keyword-based system and Perplexity's reliance on Bing. The company recently purchased a $5M H200 cluster and maintains a culture of nap pods and first-principles thinking.

The State of Reasoning — from Nathan Lambert, Interconnects/AI2 [LS Live @ NeurIPS 2024]

Jan 2, 2025 · 16:22

Nathan Lambert argues that language models do perform reasoning, contrary to skeptics, and that embracing chain-of-thought and reinforcement learning (RL) is key to advancing their capabilities. He explains OpenAI's o1 as large-scale RL on verifiable outcomes, noting that post-training flops exceed pre-training, and highlights relatives like DeepSeek and Qwen which are narrower. Lambert details OpenAI's new reinforcement fine-tuning API, which uses the same infrastructure as o1 and requires only dozens of labeled samples, and contrasts it with process reward models or Monte Carlo tree search. He presents his own project using RL on math evaluations (GSM8K, MATH, AFeval) to show gentle RL fine-tuning can boost specific capabilities without degrading general performance. The talk concludes that reasoning is worth pursuing and that new, less-human-like forms of model reasoning are emerging.

2024 Year in Review: The Big Scaling Debate, the Four Wars of AI, Top Themes and the Rise of Agents

Jan 1, 2025 · 1:51:07

In their 100th episode, hosts Alessio and Swyx recap 2024 in AI, arguing that pre-training scaling has hit a wall—backed by Ilya Sutskever and others at NeurIPS—and that inference-time compute (o1, o3) is the new frontier. They dissect the "four wars": data quality (lawsuits vs. synthetic data), GPU haves vs. have-nots (with the middle class dying), multimodality (Sora, Veo 2, Gemini 2.0's native image output), and the LLM OS/agents stack (LangChain, E2B, memory). Market share shifted from OpenAI's 95% to 50-75% as Anthropic and Gemini gained ground; prices dropped ~3 orders of magnitude for same ELO. The episode predicts 2025 as the year agents finally enter production, driven by models like o1 and tools like Devin, and warns that AI will set the skill floor for roles.

Best of 2024 in Agents (from #1 on SWE-Bench Full, Prof. Graham Neubig of OpenHands/AllHands)

Dec 25, 2024 · 51:25

Professor Graham Neubig of CMU and AllHands AI, maintainer of the open-source coding agent framework OpenHands, delivers a talk on the state of agents in 2024, arguing that agents are now capable of automating significant portions of software development but still struggle with information gathering and error recovery. He demonstrates OpenHands solving real coding tasks live, explains key design choices like using Python code execution over granular tool calls to reduce LLM invocations, and shares that Claude currently outperforms other models for agentic tasks due to superior instruction following and error correction. Neubig discusses eight perennial agent problems including agent-computer interface, human-agent interface, model selection, planning, workflows, exploration, search, and evaluation. He predicts every major LLM trainer will focus on agent-oriented models by mid-2025, that agent benchmarks like SWE-Bench and WebArena will saturate and require harder successors, and that the biggest challenge will be designing human-agent interfaces for systems that succeed only 30-40% of the time autonomously but 80-90% with human feedback. The talk also covers workflow memory as a…

Best of 2024: Synthetic Data / Smol Models, Loubna Ben Allal, HuggingFace [LS Live! @ NeurIPS 2024]

Dec 24, 2024 · 28:08

Loubna Ben Allal of Hugging Face explains how synthetic data has become ubiquitous in LLM pipelines, moving from post-training to pre-training, with examples like Cosmopedia’s 30B-token synthetic dataset and NVIDIA’s 1.9T-token Nemotron CC. She addresses model collapse fears, showing that web dumps after ChatGPT’s release actually yield better models, and emphasizes diversity through prompt seeds and webpage extracts. For filtering, FineWeb-Edu and DCLM use LLMs to rate educational content, achieving top benchmark scores. In post-training, she highlights AgentInstruct, Tülu3 with PersonaHub, and Cohere’s multilingual arbitrage using multiple teachers. On small models, SmolM2 (1.7B) outperforms Llama 1B and Qwen 2.5 after 11T tokens of pre-training, and on-device inference via frameworks like llama.cpp enables privacy-preserving use cases such as text extraction and structured generation. She predicts a return to fine-tuning specialized small models over costly prompt engineering.

Best of 2024: Open Models [LS LIVE! at NeurIPS 2024]

Dec 23, 2024 · 37:29

Luca Soldani of AI2 and Sophia Yang of Mistral argue that open models in 2024 have exploded in number and performance, closing the gap with closed models, but face growing compute and data-access challenges. Soldani notes 2024 saw models like Qwen and Deepseek rivaling frontier closed-model performance, and fully open models like OLMo release full recipes including data, code, and intermediate checkpoints. He points out that pre-training requires 10K+ GPUs for state-of-the-art, while post-training can be done with as few as eight. Sophia highlights Mistral's release of over a dozen models in 2024, including Pixtral multimodal and Le Chat—a free chat interface with image understanding, canvas code execution, and web search. Soldani warns of data access diminishing due to content owners blocking crawlers and lobbying efforts labeling open source AI as dangerous, emphasizing the need to protect the open ecosystem.

The State of AI Startups in 2024 [LS Live @ NeurIPS]

Dec 21, 2024 · 26:35

Sarah Guo and Pranav Reddy argue that 2024 has become a far friendlier ecosystem for AI startups, with the model landscape shifting from OpenAI's near-monopoly to a competitive field where Google's Gemini now leads LMSys Arena and open-source models like Llama 8B score ten points higher on MMLU than Mistral 7B a year ago. They note that OpenAI's API cost has dropped 80-85% in 18 months, and total OpenAI API share fell from ~90% to ~60% as customers switch. The funding environment is rational, not a bubble, with foundation-model labs raising $30-40B but most startups seeing sane valuations; one portfolio company grew from zero to twenty million in PLG-style spending. Key startup themes include first-wave service automation (Sierra, Decagon, Harvey, EvenUp), better search and new friends (Perplexity, Glean, Character, Replica), and democratized creativity (Midjourney, HeyGen). They argue the 'GPT wrapper' narrative is false—applications capture value—and that incumbents face innovator's dilemma because AI changes business models (outcomes-based pricing) and data needs (reasoning traces rarely saved). The speed of change and new markets (legal, healthcare, defense) structurally favor…

Windsurf: The Enterprise AI IDE

Dec 13, 2024 · 1:06:36

Varun Mohan and Anshul of Codeium explain why they built Windsurf, a new AI IDE, arguing that VS Code's API limitations prevented them from delivering the best agentic experience. They detail how Cascade, their agentic system, uses proprietary retrieval and planning models alongside third-party LLMs, and describe their evaluation method that masks commits and tests incomplete code states. The duo also reveals that over 800,000 developers use Codeium extensions, that they still support JetBrains and Eclipse for enterprise customers, and that they intentionally avoided a waitlist launch. They discuss the trade-offs of building first-party vs. third-party models, the importance of 'go slow to go fast' in enterprise infrastructure, and their belief that individual developer profits should come after building switching costs through superior product.

[Paper Club] Weight Streaming on Wafer-Scale Clusters (w/ Sarah Chieng of Cerebras)

Dec 7, 2024 · 43:54

Sarah Chieng of Cerebras presents the weight streaming technique for training giant neural networks on Cerebras wafer-scale clusters, arguing it enables near-linear scaling by separating parameter storage from primary compute. The system uses the Wafer Scale Engine (WSE-2/3) with 900,000 cores and 44 GB on-chip SRAM, avoiding off-chip memory bottlenecks that limit NVIDIA GPUs. MemoryX provides external storage for weights and optimizer states (up to 2.4 petabytes), streaming weights to compute units via the SwarmX interconnect fabric, which aggregates gradients. This design allows training models with up to 120 trillion parameters without complex hybrid parallelism. Cerebras also leverages unstructured weight sparsity to prune 90% of data during transmission and skip zero-value computations on-chip, reducing bandwidth and improving efficiency.

[Paper Club] BERT: Bidirectional Encoder Representations from Transformers

Nov 27, 2024 · 53:12

Eric Ness walks through the BERT paper, explaining how its bidirectional encoder architecture with masked language modeling and next sentence prediction pre-training enabled state-of-the-art results on 11 NLP tasks in 2019. He details the 110M parameter BASE and 340M parameter LARGE models, trained on 2.5B Wikipedia words and 800M book words, and how a simple logistic regression on BERT embeddings achieves 82% accuracy on IMDb sentiment classification, far above the 50% baseline. Swyx and Eric discuss how BERT’s pre-training objectives—masking 15% of tokens and predicting sentence order—teach small models word relationships for efficient classification, though they lose data compared to decoder-only next-token prediction. They note that while scaling laws favor decoder models, BERT-style encoders remain cost-effective for edge deployment, with recent work showing full retraining in 24 hours for under $500 or even 1 hour for $20 on 8 A100s, matching original GLUE scores.

Why Compound AI + Open Source will beat Closed AI — with Lin Qiao, CEO of Fireworks AI

Nov 25, 2024 · 55:49

Lin Qiao, CEO of Fireworks AI, argues that compound AI systems combining multiple open-source models will outperform closed monolithic models like OpenAI's. She explains Fireworks' evolution from a PyTorch platform to a full-stack inference and customization engine serving 40+ customers including Cursor. Qiao details their distributed inference engine, Fire Optimizer for quality-latency-cost tradeoffs, and upcoming o1-like model built on open-source foundations. She defends their quantization approach after public criticism from rivals, emphasizes specialization over general intelligence, and invites developers to test their free LoRA adapter hosting.

In the Arena: How LMSys changed LLM Benchmarking Forever

Nov 1, 2024 · 41:02

Anastasios and Weilin from LM-Sys explain how ChatBot Arena became the de facto standard for LLM benchmarking by replacing static benchmarks with dynamic human preference evaluations. They trace the origin from Vicuna fine-tuning to the viral launch in April 2023, where anonymous side-by-side battles let the community decide which model is better. To address biases like length preference, they developed Style Control, a logistic regression method that adjusts for confounders such as response length and markdown formatting. They address the controversy around labs testing multiple private models, arguing that selection bias is empirically small and the live benchmark self-corrects over time. They also discuss RouteLLM for cost-performance routing, the graduation of ChatBot Arena from LMSys to support new projects, and call for community help with red teaming, vision modalities, and implementing a REPL for coding evaluations.

[Paper Club] Upcycling Large Language Models into Mixture of Experts

Oct 29, 2024 · 39:02

NVIDIA's Ethan He presents Megatron-Core MoE and the upcycling of dense LLMs into Mixture-of-Experts models, showing that upcycling a 15B-param dense model into a 64-expert MoE and training on 1T tokens yields 5% lower validation loss and 4% higher MMLU than continued dense training on the same compute. Key techniques include swapping the router order (softmax-then-top-K instead of top-K-then-softmax) with a 4× output scaling to preserve initial forward pass behavior, and initializing fine-grained MoE routers by duplicating half the weights so each shard group selects identically. A high learning rate matching the original pre-training peak is critical; constant low LR causes catastrophic forgetting. The upcycled model matches the compute of a 1.7× larger dense model per scaling laws, but data quality remains paramount—continued dense training still delivered a 20% MMLU jump. Megatron-Core's fused permutation, GroupedGEMM, and expert parallelism are available as a standalone library.

Building the Silicon Brain - Drew Houston of Dropbox

Oct 18, 2024 · 1:11:40

Drew Houston, CEO of Dropbox, details his hands-on AI engineering journey and the company's strategic pivot to AI-first products like Dropbox Dash for universal search and access control. Having spent over 400 hours coding with LLMs, he built personal tools that seeded Dropbox AI, including a file question-answering system. He advocates 'rent, don't buy' for AI infrastructure, relying on open-source models and keeping options open as costs drop 10-100x yearly. Houston explains Dropbox's advantage in trust and data privacy, positioning it as a neutral platform that integrates with Google Drive and OneDrive. He discusses staying relevant through constant learning and founder mode, and advises founders to systematically train skills ahead of their company's growth.

[Paper Club] Molmo + Pixmo + Whisper 3 Turbo - with Vibhu Sapra, Nathan Lambert, Amgadoz

Oct 13, 2024 · 1:12:59

This episode covers two papers: AI2's Molmo open-source vision-language models and OpenAI's Whisper Large V3 Turbo. For Molmo, the key claim is that high-quality, audio-annotated data (Pixmo) enables models as small as 1B to match GPT-4V on academic benchmarks, with the 72B variant outperforming GPT-4o, Gemini 1.5, and Claude 3.5 Sonnet. The team avoided distilling proprietary models by having annotators describe images in speech for 60-90 seconds, then transcribing and augmenting the captions. For Whisper Turbo, OpenAI pruned the decoder from 32 to 4 layers and continued pre-training on 10 million hours of multilingual transcription data, making it 1.78x smaller than Large V3 with minimal word-error-rate increase. The model is faster and supports real-time chunk-based decoding, unlike English-only DistilWhisper. Nathan Lambert and Amgadoz provide commentary on data strategies and benchmarking nuances.

Production AI Engineering starts with Evals

Oct 11, 2024 · 1:56:16

Ankur Goyal, founder and CEO of Braintrust, argues that evaluation is the core workflow of production AI engineering and shows how his platform makes evals accessible to software engineers. Drawing from his experience at SingleStore and Impira, he details Braintrust's evolution from an eval tool into an end-to-end AI development platform used by Stripe, Zapier, Vercel, and other top AI teams. He shares market data: OpenAI handles over 95% of production workloads, fine-tuning is declining, and open-source models account for under 5% due to reliability issues. Ankur explains why he avoided building a vector database—the real challenge is permissions and joins, not vector search—and predicts o1-style reasoning will replace complex agent frameworks. He reveals Braintrust's differentiators: hybrid on-prem, TypeScript-first SDK, and declarative eval structures.

Language Agents: From Reasoning to Acting — with Shunyu Yao of OpenAI, Harrison Chase of LangGraph

Sep 27, 2024 · 1:26:33

Shunyu Yao of OpenAI and Harrison Chase of LangChain discuss the evolution of language agents, arguing that combining reasoning and acting through techniques like ReAct remains foundational, while tool design and cognitive architectures like CoALA will shape the next wave. Shunyu traces his work from text games to ReAct, showing how thinking as an action improves reliability; Harrison explains LangChain's adoption and the shift to LangGraph for stateful orchestration. They emphasize agent-computer interfaces (ACI) as a neglected area, noting that SWE-Agent's success came from optimizing tools for models, not humans. The conversation covers memory types (semantic, episodic, procedural) and applies CoALA's three dimensions to current systems. They highlight promising applications: customer support, coding agents, and spreadsheet-style UX for batch operations. Shunyu stresses that better benchmarks like SWE-Bench (which solved 30% of GitHub issues) and TauBench (best model 48%) are critical, and that data—not architecture—will drive future model improvements.

llm.c's Origin and the Future of LLM Compilers - Andrej Karpathy at CUDA MODE

Sep 21, 2024 · 23:29

Andrej Karpathy explains llm.c, his from-scratch C implementation of GPT-2 training that outperforms PyTorch by 20% in speed and 30% in memory, arguing that LLMs will eventually act as compilers for custom code. He recounts starting the project out of frustration with PyTorch compile errors, writing all forward and backward passes manually in C, then porting to CUDA with kernel optimizations. A community of contributors (Eric, Arun, Alex R) helped achieve 50% MFU on a single H100 node, training GPT-2 1.6B in 24 hours for $600. Ongoing work adds Llama 3.1 and FP8 support. Karpathy sees llm.c as proof that LLMs can automate such low-level optimization, making Python and PyTorch a temporary crutch.

[Paper Club] Writing in the Margins: Chunked Prefill KV Caching for Long Context Retrieval

Sep 19, 2024 · 53:53

Umar Jamil from Writer presents 'Writing in the Margins,' a paper on improving long-context retrieval in LLMs by leveraging chunked prefill of the KV cache. The method splits a long prompt into chunks, generates query-relevant annotations (margins) after each chunk, and appends them at the end before generating the final answer. This approach solves the 'lost in the middle' problem without fine-tuning, and unlike traditional RAG or separate summarization, it avoids re-prefilling the entire context, reducing cost from double to single prefilling. The technique is compatible with any transformer model and includes overlapping margin generation and classification within the same batch request. Benchmarks show consistent improvements across models, and the implementation is open-source on GitHub.

[Paper Club] 🍓 On Reasoning: Q-STaR and Friends!

Sep 18, 2024 · 47:12

This episode of Paper Club surveys three reasoning papers — STaR, Quiet-STaR, and V-STaR — arguing that while STaR is foundational for bootstrapping reasoning via rationales and rationalization, Quiet-STaR's attempt to generate rationales at every token yields only marginal gains (5–10% on GSM8K and CQA), and V-STaR's verifier trained with DPO on both correct and incorrect solutions delivers the most practical improvement, beating majority voting. The host explains STaR's two-loop process of generating rationales and rationalizations from wrong answers, highlights examples like a filtering-straw question where human raters evaluated reasoning quality, and notes that STaR on GPT-J 6B achieved human-like step counts in math problems. V-STaR's verifier selects among candidate solutions, scaling with K candidates, and is compared to process reward models from OpenAI's "let's verify step by step."

Answer.ai & AI Magic with Jeremy Howard

Aug 17, 2024 · 1:11:00

Jeremy Howard of Answer.AI argues that continuous pre-training should be treated as a continuum, not separate phases, and demonstrates how FSDP+QLoRA enables training a 70B model on just two NVIDIA 4090s. He reveals Answer’s non-hierarchical, manager-free R&D lab model that recruited unusual talent like Benjamin Warner and Ben Claviez, who independently launched BERT 24 to revive encoder-only architectures. Howard introduces FastHTML, a pure-Python web framework built on HTMX and Starlette, for creating modern SPAs without JavaScript. He previews “AI Magic,” a dialogue engineering system that moves beyond teletype-style chat interfaces and code editors, aiming to make AI-assisted development more interactive. The episode also covers Answer’s Public Benefit Corporation structure designed to resist hostile takeovers, and critiques decoder-only hype while advocating for encoder-decoder and state-space models.

The Winds of AI Winter (Q2 Four Wars of the AI Stack Recap)

Aug 2, 2024 · 1:23:36

Swyx and Alessio recap Q2 2024 through their 'Four Wars' framework, arguing that the AI landscape is shifting from frontier model dominance to commoditization and vertical applications. They highlight Claude 3.5 Sonnet overtaking OpenAI on coding benchmarks, Llama 3.1's synthetic data approach enabling 7B models to rival GPT-4, Mistral Large 2's non-commercial license and lost open-source crown, and on-device models like Gemini Nano and Apple Intelligence. The Quality Data Wars see NYT suing OpenAI, Reddit licensing data for $200M+, and synthetic data proving real for math (AlphaProof near IMO gold) and code. The Multimodality War includes ChatGPT Voice Mode delayed, Meta's Chameleon for native fusion, and Google's PaliGemma for PDF extraction. The renamed LLM OS War covers agent protocols, memory databases, and the collapse of model cost by an order of magnitude every four months, pushing startups toward vertical services like Brightwave and Dropzone that sell labor, not tools. The episode ends with a CrowdStrike joke about agent safety.

[LLM Paper Club] Llama 3.1 Paper: The Llama Family of Models

Jul 29, 2024 · 1:23:46

This episode examines Meta's Llama 3.1 paper, detailing the 405B dense model, its scaling laws grounded on the ARC reasoning benchmark rather than perplexity, and the decision to train on 15 trillion tokens. Vibhu explains the training infrastructure: 16,000 H100s over 54 days with 419 interruptions, 78% from GPU hardware failures. Eugene Yan walks through the synthetic data pipeline—using Llama 2 for filtering, stepwise reward models, and Monte Carlo tree search to improve reasoning traces. Hassan shares building LlamaTutor.com with Together API, serving 4,000 visitors and 5,900 requests for about $12. The group also discusses quantization trade-offs (larger models degrade less), inference provider variability (Groq's non-deterministic temperature zero), and compares Llama 405B's performance to GPT-4o and Claude.

Training Llama 2, 3 & 4: The Path to Open Source AGI — with Thomas Scialom of Meta AI

Jul 23, 2024 · 1:04:34

Thomas Scialom, Llama 2 lead and Llama 3 post-training lead at Meta AI, explains why scaling laws must go beyond Chinchilla—training models longer on more tokens yields better inference efficiency than bigger models, a lesson that drove Llama 3's 15 trillion token pre-training. He details how synthetic data from Llama 2 bootstrapped Llama 3's post-training, eliminating human-written SFT data, and why RLHF outperforms imitation learning: humans are better discriminators than generators, enabling superhuman outputs. Scialom defends the dense 405B architecture over MoE, calls tokenizer vocab size underrated (128k tokens vs Llama 2's 32k), and reveals Llama 4's focus on agentic capabilities—tool use, multi-step reasoning—as the path to open-source AGI. Meta AI is hiring researchers with rigorous first-principles thinking.

The 10,000x Yolo Researcher Metagame — with Yi Tay of Reka

Jul 5, 2024 · 2:18:43

Yi Tay, chief scientist at Reka and former Google Brain architecture co-lead on PaLM 2, recounts his journey from an NTU PhD to co-founding Reka, where a team of just 5 people pre-trained Reka Core on $60M of GPU compute—debating at #7 on the LMsys leaderboard. He explains why he believes the Noam transformer (with SwiGLU, GQA, RoPE) remains the strongest baseline, how encoder-decoder architectures offer 'free sparsity' (a 2× flop-efficiency over decoder-only), and why Chinchilla scaling laws are often misunderstood (training past the optimal compute frontier is routine). He argues that long context will eventually outperform RAG for complex reasoning, and that open-source models like Llama 3 are catching up only because Meta invested in a top-tier training stack, not due to grassroots innovation. The episode also covers his productivity habits (working backwards from a paper title, camping arXiv) and the cultural shock of moving from Singapore academia to Google's impact-driven research environment.

How AI is Eating Finance - with Mike Conover of Brightwave

Jun 11, 2024 · 1:05:06

Brightwave founder Mike Conover explains how his vertical AI startup builds a 'partner in thought' for finance professionals, using a systems-of-systems approach that decomposes problems into specialized subsystems rather than relying on large context windows or monolithic agents. He argues that the evolution from DALL-E's 1,024-token context to today's million-token models has not solved synthesis; smaller, focused reasoning units still outperform. Conover emphasizes that the real competitive edge lies in custom training data and human annotation, not pre-training, and that models are converging in capability—so value shifts to fine-tuning data that elicits specific behaviors. He details why Brightwave avoids spreadsheets in favor of conversational analysis, and predicts AI will automate idea generation and second-order derivative bets, but humans must remain the final synthesizers and deciders. The episode also covers hiring for vertical AI, the role of knowledge graphs, and the diminishing economic incentive for companies to train their own foundation models.

How to train a Million Context LLM — with Mark Huang of Gradient.ai

May 31, 2024 · 1:12:14

Mark Huang of Gradient.ai explains how his team extended Llama 3 to a 1 million token context window using curriculum learning, RingAttention, and EasyContext, achieving near-perfect GPU utilization. They employed theta scaling from the RoPE paper to interpolate positional encodings, trained on carefully curated datasets including SlimPajamas and synthetic data from GPT-4, and validated with benchmarks like Ruler and needle-in-a-haystack. Huang details the trade-offs between full fine-tuning and LoRA adapters, the challenges of pushing to 4M tokens (degradation from floating-point precision limits), and why long context matters for state management across sessions and grounding multimodal inputs. He calls for community collaboration on long-context evaluations and pairwise multimodal datasets.

LLM Asia Paper Club Survey Round

May 22, 2024 · 55:25

The episode surveys four recent papers on LLM reasoning, uncertainty, interpretability, and efficiency. 'Let's Think Dot by Dot' shows that filler tokens (dots) inserted between input and output enable hidden computation, outperforming no-token baselines on tasks like 3-Sum and 2-Sum Transform. 'Uncertainty Estimation' trains a random forest on hidden-layer activations to predict response confidence, achieving higher AUC than unsupervised methods on Q&A and translation. 'Monosemanticity' uses sparse autoencoders to identify interpretable features (e.g., a DNA-detection feature) in a toy transformer, advancing mechanistic interpretability. 'Medusa' attaches multiple prediction heads to the final hidden state to speculate future tokens, enabling faster decoding without a separate draft model and training in five hours on 60K samples.

Breaking down the OG GPT Paper by Alec Radford

Apr 23, 2024 · 1:05:03

This episode of the Latent Space Paper Club breaks down Alec Radford's seminal 2018 GPT-1 paper, which introduced generative pre-training of a Transformer decoder on BookCorpus (5 GB, 7,000 books) followed by supervised fine-tuning. Amget explains how the 117M-parameter model achieved state-of-the-art on 9 of 12 NLU tasks, with absolute gains of 8.9% on commonsense reasoning (Stories Cloze) and 5.7% on question answering (RACE). Key innovations include task-agnostic input transformations (e.g., concatenating premise and hypothesis with a delimiter for entailment, using a Siamese architecture for semantic similarity, and scoring answer choices for QA) and an auxiliary language modeling objective during fine-tuning (lambda=0.5). The episode also covers zero-shot heuristics (appending 'very' for sentiment, averaging token log probabilities for QA), ablation studies showing pre-training adds 15% average improvement, and the paper's prescient call to scale up—which became the blueprint for GPT-2, GPT-3, and beyond.

Supervise the Process of AI Research — with Jungwon Byun and Andreas Stuhlmüller of Elicit

Apr 11, 2024 · 1:05:28

Andreas Stuhlmüller and Jungwon Byun, co-founders of Elicit (formerly the nonprofit Ought), have built an AI research assistant that automates literature review and reasoning by breaking complex tasks into transparent, step-by-step processes. Their philosophy—"supervise the process, not just the outcome"—led them to start with human simulations before GPT-3 enabled a product pivot. Elicit now uses both open-source and closed models (e.g., GPT-4, Claude Haiku) for summarization, data extraction, and uncertainty flags, and recently launched computational notebooks for scalable, reusable workflows. The company transitioned from nonprofit to a Public Benefit Corporation, reached $1M revenue in four months, and now employs 12 people, focusing on senior software engineers to build reliable orchestration from unreliable components.

Personal AI Meetup - Bee, BasedHardware, LangChain LangFriend, Deepgram EmilyAI

Apr 6, 2024 · 58:54

This episode features Damien Murphy of Deepgram, Ethan of Owl/Bee, and Harrison of LangChain demonstrating how to build personal AI with real-time voice bots, wearable life-recording devices, and memory-enhanced journaling apps. Damien shows building a voice bot with subsecond latency using Deepgram, OpenAI, and open source code, costing about 6.5 cents per five-minute call. Ethan presents his Owl wearable that continuously records audio, triggers actions via hot word 'Scarlett,' and discusses challenges in adding vision and open source adoption. Harrison introduces LangFriend, a journaling app using conversational, semantic, and knowledge graph memory, referencing the Generative Agents paper for recency and importance weighting. The episode also highlights open source projects Whomane, Friend, and ADeus, arguing that hardware, voice, and memory are all necessary components for personal AI.

Why Google failed to make GPT-3 -- with David Luan of Adept

Mar 27, 2024 · 49:27

David Luan, co-founder of Adept and former early OpenAI leader, explains why Google failed to build GPT-3—its 'brain credit marketplace' prevented critical mass—and why Adept builds enterprise AI agents prioritizing reliability over generality. He recounts the GPT-2 demo that helped secure Microsoft's $1B investment and details Adept's goal: an AI teammate that can do anything a human does on a computer, targeting 'nines of reliability' for workflows like dispatching a physical truck. Luan contrasts Adept's vertical integration—training fast multimodal models (Fuyu) for charts and UIs—with pure-play foundation model companies that sell tokens, predicting commoditization. He notes Adept is sold out for Q1 and raised $420M, and explains how an augmentation focus creates a data flywheel from human oversight.

Open Source AI is AI we can Trust — with Soumith Chintala of Meta AI

Mar 6, 2024 · 1:37:38

Soumith Chintala, creator of PyTorch and engineering lead at Meta AI, argues that open source AI is essential for distributing opportunity and trust. He details PyTorch's complexity—1,000 operators needed for generality—and explains synthetic data as a vehicle for imparting symbolic knowledge where humans already have good symbolic models. He highlights a coordination problem in open source: feedback is lost because frontends like Ooba and Ollama lack feedback buttons, and proposes a centralized sinkhole to collect high-quality feedback. Beyond text, he is excited about robotics, where hardware remains a bottleneck, and Osmo's work to digitize smell, which he compares to images in the 1800s.

A Brief History of the Open Source AI Hacker - with Ben Firshman of Replicate

Feb 28, 2024 · 1:21:09

Ben Firshman, CEO of Replicate, explains how the inference platform grew from a research reproducibility tool into a 2M-user API business by embracing the generative image community and treating open-source AI as a hacker-friendly ecosystem. They accidentally discovered their API when a user reverse-engineered their web form, leading to their first $1k/month customer. Cog, their container standard for ML models, was born from lessons at Docker and the need to make models tinkerable. Ben argues that fine-tuning's low cost makes open-source models sustainable, and that AI engineers (orders of magnitude more than ML engineers) just need to start playing with models. He also discusses GPU scarcity, preferring sustainable pricing over price wars, and reveals that demand is not outpacing supply thanks to aggregating demand.

Building an open AI company - with Ce and Vipul of Together AI

Feb 8, 2024 · 1:15:18

Together AI co-founders Vipul Ved Prakash and Ce Zhang explain why openness is core to their mission, detailing their journey from Apple (Vipul) and Stanford research (Ce) to building an open AI platform. They discuss RedPajama’s evolution into a modular dataset with 40 quality signals, the need for 5,000 tokens/second inference speed, and their investment in state space models like Mamba and Hyena as alternatives to transformers. The company, which runs 7,000-8,000 GPUs (mostly H100s), sees training as a larger workload than inference, with fine-tuning driving top models. They advocate for independent inference benchmarks, publish open research like FlashAttention, and keep some software proprietary. With 38 employees and 45% researchers, they are hiring across the stack, from CUDA to DevOps.

The State of AI in production — with David Hsu of Retool

Feb 7, 2024 · 1:06:52

Retool CEO David Hsu shares insights from the company's 2023 State of AI survey, revealing that most AI adoption remains internal and that the hype may be overrated—52% of 1,600 respondents said AI is overrated, and only 27% have AI in production, with 66% of those being internal use cases. He explains Retool's developer-first philosophy, their choice of open-source PG Vector over proprietary solutions, and why they intentionally raised less money at lower valuations to avoid over-dilution. Hsu describes Retool's shift from sales-led to bottom-up growth to reach millions of developers, and highlights the importance of AI workflows over simple chatbots, citing a clothing manufacturer that uses Retool apps with DALL-E to generate patterns. He discusses the competitive landscape, predicting open-source models will eventually catch up to OpenAI, and shares philosophical views on AGI using the plane-vs-bird analogy.

The Origin and Future of RLHF: the secret ingredient for ChatGPT - with Nathan Lambert

Jan 11, 2024 · 1:35:27

Dr. Nathan Lambert traces the origin and future of Reinforcement Learning from Human Feedback (RLHF), the secret ingredient behind ChatGPT, explaining how it evolved from robotics and early preference learning to become the core alignment technique for large language models. He details the three-phase process (instruction tuning, preference data collection, RL optimization), noting that RLHF's data costs for LLaMA2 were around $6–8 million, and that synthetic data from GPT-4 is cheaper and often more accurate than human labels. Lambert contrasts DPO with PPO, arguing DPO is simpler but may have lower peak performance. He discusses emerging methods like Constitutional AI, which uses AI-generated critiques based on principles, and highlights the challenge of evaluating RLHF models, noting GPT-4 Turbo's lead over earlier versions. The episode covers open questions about data aggregation, reward model agreement (65–75%), and the need for qualitative model interaction.

The State of Silicon and the GPU Poors - with Dylan Patel of SemiAnalysis

Dec 5, 2023 · 1:07:55

Dylan Patel of SemiAnalysis argues the 'GPU poor vs GPU rich' divide defines AI, with Google's TPU ramp (millions of units) dwarfing others and Nvidia selling over a million H100s this year. He explains that training costs are irrelevant compared to inference costs, and that memory bandwidth is the key bottleneck for LLM inference (e.g., Llama 70B needs ~2.1 TB/s for human reading speed). Patel advises GPU-poor players to focus on on-device innovations like speculative decoding (Medusa), asynchronous training, or fine-tuning for edge use—not on gaming benchmarks or fine-tuning small models. He details why AI hardware startups (Cerebras, Graphcore) bet wrong on on-chip SRAM, while Google’s TPU/Broadcom partnership and Nvidia’s yearly cadence make alternatives tough. He also warns that rebuilding the semiconductor supply chain in the US is unfeasible due to its fragmentation, and that safety through obscurity doesn't work—open innovation is better.

Beating GPT-4 with Open Source Models - with Michael Royzen of Phind

Nov 3, 2023 · 1:18:54

Michael Royzen, co-founder and CEO of Phind, explains how his company built a GPT-4-beating open-source model for developer Q&A. Royzen recounts founding SmartLens in high school, shifting to NLP after a Hugging Face demo, and creating an internet-scale LLM-powered RAG system in January 2022. He details Phind’s pivot to programmers, the Hacker News launch that gave it 1,500 points, and Paul Graham’s role in naming the company and introducing Ron Conway, who then connected Phind to NVIDIA for GPU access. Royzen argues that Phind’s model, fine-tuned from Code Llama 34B with extra data, closes the gap with proprietary models—especially on code reasoning—and that open-source will win the enterprise because the delta to GPT-5 will be small. He shares how Phind handles multi-step conversations via a pair programmer mode where users can pin messages, and reveals plans for reinforcement learning to reduce hallucination and improve correctness.

Why AI Agents Don't Work (yet) - with Kanjun Qiu of Imbue

Oct 21, 2023 · 1:12:37

Kanjun Qiu, CEO of Imbue (formerly Generally Intelligent), argues that AI agents remain unreliable because they lack robust reasoning and proper abstractions, which Imbue tackles by training foundation models optimized for reasoning over data like code. With a $200M Series B and $1B+ valuation, Imbue builds internal tools for debugging and inspecting agent decision-making, rejecting pure reinforcement learning after their Avalon environment revealed RL cannot handle planning. Instead, they emphasize natural language reasoning for inspectability, use code as a curriculum for reasoning, and design interfaces that let users fork and modify agents mid-execution. Qiu also discusses lessons from earlier startups (Sorceress in recruiting, Embark in VR), the importance of treating team members as creative agents, and her role in co-founding communal living space The Archive to foster scenius.

The End of Finetuning — with Jeremy Howard of Fast.ai

Oct 20, 2023 · 1:24:48

Jeremy Howard of Fast.ai argues that fine-tuning language models is essentially continued pre-training, not a separate process, and that the common practice of fine-tuning on a single task causes catastrophic forgetting. He recounts the discovery of single-shot memorization in LLMs, where models memorize entire datasets after one epoch, a phenomenon many practitioners ignore. Howard criticizes the current focus on zero-shot and few-shot learning, advocating for transfer learning and small models. He shares his journey from philosophy to founding Fast.ai, the creation of ULMFit (which inspired GPT), and his ongoing work on making AI accessible. He also discusses his involvement with Modular's Mojo language and the importance of democratizing AI technology.

RAG is a hack - with Jerry Liu of LlamaIndex

Oct 12, 2023 · 1:13:25

In this episode of Latent Space, Jerry Liu of LlamaIndex argues that while RAG is fundamentally a hack, it remains the most practical approach for grounding LLMs with external data, and his open-source framework now serves 600,000 monthly downloads. Liu recounts how LlamaIndex originated from a hackathon at Robust Intelligence, evolving from an experimental tree-index to a modular toolkit spanning data loading, retrieval, synthesis, and agent loops. He explains the trade-offs between RAG and fine-tuning, noting that context window limits force algorithmic rather than learned optimizations, but RAG offers transparency and access control that fine-tuning cannot. Liu details the company’s $8.5M Greylock raise, the launch of LlamaHub for community-contributed data loaders, and the open-source SEC Insights app as a production-grade template. Looking ahead, he emphasizes the need for better retrieval benchmarks and predicts that future personalization will move beyond vector stores into model-internal memory architectures.

RWKV: Reinventing RNNs for the Transformer Era

Aug 31, 2023 · 1:56:51

Eugene, CTO of UILicious, introduces RWKV, a recurrent neural network that matches transformer-level performance while scaling linearly with context length, solving the quadratic cost of attention. He explains how RWKV uses Attention-Free Transformer layers and parallelizable training, achieving competitive results on 7B and 14B models. The community-driven project prioritizes multilingual support, with a tokenizer that handles languages without spaces. Eugene recounts his journey from GPU.js to finding RWKV, and highlights use cases like long HTML analysis where transformers fail. He also discusses the token crisis, diffusion models for text, and how the AI waifu community drives optimization and alignment research. Finally, he advises AI engineers to focus on practical prompting and data curation, noting that deep architecture knowledge is optional.

FlashAttention-2: Making Transformers 800% faster AND exact

Aug 3, 2023 · 1:04:06

Tri Dao, creator of FlashAttention and FlashAttention-2, explains how his I/O-aware algorithm makes attention 2x faster by fusing kernels and using online softmax, achieving near-matrix-multiply efficiency. He argues that transformer alternatives like state space models and RNNs (e.g., RWKV) could surpass transformers for long sequences and high-throughput generation, though attention still dominates. Dao discusses the hardware lottery, where NVIDIA's CUDA ecosystem entrenches transformers, and advocates for open-source AI, praising Meta's Llama 2 for shifting enterprise adoption despite its restrictive license. He emphasizes that understanding both algorithms and systems is key to scaling AI, and that academia should pursue risky bets that industry cannot.