Media & Creativity

Inside xAI: Building Grok Imagine in 3 Months, Videogen vs World Models, and Video Agents— Ethan He

Jun 1, 2026 · 1:44:43

Ethan He, former xAI and NVIDIA Cosmos researcher, explains how xAI built its first image and video models (Grok Imagine 0.9) from zero to one in three months, attributing rapid iteration to small teams with minimal meetings and strong infra that enabled fixing tiny data and training bugs for biggest quality gains. He argues that most improvements in video generation now come from language models and agents rather than diffusion technology, predicting that by end of 2025 video agents will produce production-grade content for ads. He defines world models as real-time, interactive, long-horizon videos, and details challenges like temporal compression, context management, and the high cost of storing and moving video data (e.g., tens of petabytes for a billion videos). Ethan also shares why he left xAI to focus on language model research, believing the next frontier is models that manage their own context length, similar to solutions already being explored in video generation.

Mistral: Voxtral TTS, Forge, Leanstral, & Mistral 4 — w/ Pavan Kumar Reddy & Guillaume Lample

Mar 30, 2026 · 54:02

Mistral releases Voxtral TTS, a 3B-parameter speech generation model using a novel autoregressive flow matching architecture and a 12.5 Hz neural audio codec developed in-house. Pavan Kumar Reddy and Guillaume Lample explain how the model achieves high quality and efficiency by predicting audio latents as a continuous distribution, enabling real-time voice agent applications. The episode details Mistral's stepwise multimodal strategy, starting with transcription and now speech generation, with full duplex voice agents as the next goal. Enterprise deployment via Mistral Forge allows fine-tuning on proprietary data, exemplified by training on rare languages or domain-specific jargon. Guillaume discusses merging capabilities (Mistral Small merging coding, reasoning, vision), commitment to open science through technical reports, and reasoning transfer from formal proofs in Lean. The episode also covers AI for science partnerships and hiring for research and forward-deployed engineering roles.

Taste is your Moat (Dylan Field of Figma)

Oct 2, 2025 · 1:01:43

Dylan Field, CEO of Figma, argues that as AI accelerates software creation, design becomes the key differentiator—and Figma Make, a prompt-to-app tool, is Figma's bet on lowering the barrier for everyone. Field recounts his first AI-pill moment with Chris Olah in 2014 and later GPT-3 convincing him of exponential progress. He explains Figma's strategy: treat natural language as today's interface (calling it the 'MS-DOS era of AI'), use Figma as a 'context repository for aesthetics' to ground AI outputs in a user's taste, and solve the blank-canvas problem by letting AI generate initial designs. Field pushes back on 'fast fashion' software, arguing complex products like Workday can't be vibe-coded. He shares Thiel Fellowship lessons (start with 'what could this be' before finding flaws), recruiting advice from John Doerr (obsess over the funnel), and caution about get-rich-quick AI hype, drawing parallels to the NFT cycle he exited.

A Technical History of Generative Media

Sep 8, 2025 · 1:04:44

Fal.ai founders Gorkem and Batuhan detail their pivot from dbt pipelines to generative media inference, which now serves 2M developers and 350 models, crossed $100M ARR, and raised a $125M Series C. Key inflection points included Stable Diffusion 1.5 (company pivot), SDXL (first $1M revenue), Flux (jump from $2M to $10M monthly revenue), and Veo 3 (text-to-video with perfect lip-sync). They built a proprietary inference engine with 100+ custom kernels and a serverless GPU stack managing 10,000+ H100 equivalents across 6 cloud providers, typically delivering 1.5-10x speedups over stock PyTorch. Video models now drive 50% of revenue, up from 18% in February, fueled by open-source models like Hunyuan and partnerships with closed labs such as Play.ht and Google DeepMind. They argue advertising is the killer application for generative media and that image/video RL, specialized data pipelines, and cheaper conversational video models are underexplored startup opportunities.

Personalized AI Language Education — with Andrew Hsu, Speak

Jul 11, 2025 · 1:04:10

Andrew Hsu, CTO of Speak, explains how the company built the "third generation" of language learning by betting on AI speech and language models before they were ready. Speak focused on South Korea early, a counterintuitive move for a San Francisco startup, and now 6% of the Korean population has tried the app. The turning point came with Whisper and GPT in 2022, which enabled Speak to evolve from a listen-and-repeat tool into a full-featured AI tutor that gives real-time feedback and adapts to learners. Speak generates over $50M ARR primarily from consumer subscriptions, and Hsu details the product decisions that made it work: abandoning a free version, building custom ASR for low latency, and emphasizing functional fluency over textbook language. He also reveals that Speak is building a real-time voice platform and a knowledge graph for personalized fluency scores, with an eye toward expanding beyond language into general AI-powered education.

AI Video Is Eating The World — Olivia and Justine Moore, a16z

Jul 9, 2025 · 49:28

Alessio and Shawn (hosts of Latent Space) talk with a16z partners Justine and Olivia Moore about the explosion of AI-generated video on TikTok, Instagram, and YouTube. The Moores trace how trends moved from Reddit to consumer platforms, with examples like Italian Brainrot, Kim the Gorilla, and fruit-slicing ASMR. They detail their own hands-on creation: Olivia used Veo 3 and MiniMax to make viral clips (gold bar squishing, Tide Pod consumption), spending up to 8 generations per usable video and hitting Veo 3’s $125/month plan limits. They explain monetization paths—creator fund payouts (~$20 per million views), merch (Breadclump sweatshirts), consulting, and licensing to Netflix—and recommend tools like Overlap (clips 30–140 seconds) and ComfyUI for control. The episode closes with prompt theory: AI characters questioning their existence, and a bet that Gen Alpha’s brainrot consumption will reshape long-form content.

Voice AI Masterclass — Kwindla Hultman Kramer and swyx

May 6, 2025 · 20:51

Shawn Wang and Kwindla Hultman Kramer announce a voice AI masterclass course, diving into the landscape of models like Dia and Parakeet, production deployment challenges, and future trends such as speech-to-speech and real-time video. Kwindla explains that telephony (Twilio) drives 99% of current monetizable voice AI, while open source frameworks like Pipecat enable low-latency multi-modal apps. The course covers turn detection, context management, and evals, with 28 sessions featuring partners like OpenAI, Google, and NVIDIA. The goal is to jumpstart builders from prototype to production, with real-time video expected to hit its inflection point by year-end.

Snipd: The AI Podcast App for Learning — with CEO Kevin Ben-Smith

Mar 14, 2025 · 1:17:46

Snipd CEO Kevin Ben-Smith joins Swyx to explain how the AI-powered podcast app turns passive listening into active learning. Snipd transcribes and diarizes episodes, generates chapters, identifies mentioned books, and summarizes insights on demand — all with a four-person team. Kevin describes moving from self-hosted open-source models (Wave2vec, Pyannote) to a mix of Whisper, OpenAI, Google Gemini, and Perplexity APIs, balancing cost against the need for predictable batch processing vs. real-time user requests. He details technical hurdles like dynamic content resyncing (building a 'Shazam for podcasts') and using LLMs as judges to filter hallucinated book recommendations. The episode also covers Snipd's origin as a hackathon-winning podcast search tool, Swyx's own switch from Overcast, and Kevin's vision for voice-driven post-episode reflection and AI-assisted discovery as the next frontier.

AI Engineering for Art - with comfyanonymous

Jan 4, 2025 · 52:09

Comfy Anonymous, the anonymous creator of ComfyUI, details in his first-ever podcast interview how his node-based image generation tool overtook Automatic1111 through superior memory management and early support for SDXL, becoming the de facto interface for advanced diffusion workflows. He started coding on January 1, 2023, and released the first version on January 16, driven by a desire to chain models and experiment with custom samplers. His work at Stability AI from June 2023 ensured ComfyUI efficiently handled SDXL, which leaked via early access and forced users from less powerful GPUs to adopt his tool. He discusses model preferences: Flux for consistency, SD 3.5 for creativity, and SD1.5 remaining popular. For video, he highlights Mochi as a 'true' video model with 3D latents, already implemented in Comfy. Now with a core team, ComfyUI is targeting a v1 release with an easy installer on Windows and Mac, while planning monetization through cloud and enterprise features—but keeping the open-source core free.

2024 Year in Review: The Big Scaling Debate, the Four Wars of AI, Top Themes and the Rise of Agents

Jan 1, 2025 · 1:51:07

In their 100th episode, hosts Alessio and Swyx recap 2024 in AI, arguing that pre-training scaling has hit a wall—backed by Ilya Sutskever and others at NeurIPS—and that inference-time compute (o1, o3) is the new frontier. They dissect the "four wars": data quality (lawsuits vs. synthetic data), GPU haves vs. have-nots (with the middle class dying), multimodality (Sora, Veo 2, Gemini 2.0's native image output), and the LLM OS/agents stack (LangChain, E2B, memory). Market share shifted from OpenAI's 95% to 50-75% as Anthropic and Gemini gained ground; prices dropped ~3 orders of magnitude for same ELO. The episode predicts 2025 as the year agents finally enter production, driven by models like o1 and tools like Devin, and warns that AI will set the skill floor for roles.

Best of 2024 in Vision [LS Live @ NeurIPS]

Dec 22, 2024 · 55:46

Isaac Robinson and Peter Robicheaux from Roboflow and Vik Korrapati from Moondream recap the biggest shifts in computer vision in 2024, naming Sora as the year's most significant paper despite being a blog post, and highlighting its replication efforts (OpenSora) that use MAGViT V2 and diffusion transformers. They detail how SAM 2 extends the SAM approach to video with a memory bank and real-time performance, and how DETR-based detectors (RT-DETR, LW-DETR, DeFine) now surpass YOLOs on COCO by 4.6 AP at the same latency. Peter explains the CLIP-blind phenomenon: MMVP shows leading VLMs (ChatGPT, Gemini, LLaVA) fail at fine-grained tasks like reading clock hands, but models like PaliGemma 2 and AimV2 improve by incorporating pixel-level understanding and autoregressive image reconstruction, with AimV2 approaching 60.2 mAP on COCO. Vik introduces Moondream's 0.5B parameter pruned VLM and demonstrates a grounded chain-of-thought approach that dramatically improves sample efficiency for analog gauge reading, reducing the need for millions of synthetic examples.

How NotebookLM Was Made

Oct 25, 2024 · 1:13:57

Raiza Martin (NotebookLM lead PM) and Usama Bin Shafqat (AI engineer) explain how Google's NotebookLM built the viral 'Deep Dive' audio overview feature. They reveal the product evolved from Project Tailwind, using Gemini 1.5's long context and DeepMind speech to create a two-persona dialogue format that transforms documents into engaging podcasts. The team learned from 65,000 Discord members, leaned on best-selling author Steven Johnson for a 'tool for thought' workflow, and prioritized a single format over exposed controls to preserve unpredictability and delight. Humor and tension are not explicitly prompted but emerge from giving personas different angles. Evaulation relied on internal taste ('potatoes for chefs') before formal raters, with a Likert scale on dimensions like entertainment and groundedness. Future plans include multilingual support, API access, real-time chat, and codebase podcasting, while managing non-determinism by accepting occasional bad rolls.

[Paper Club] Molmo + Pixmo + Whisper 3 Turbo - with Vibhu Sapra, Nathan Lambert, Amgadoz

Oct 13, 2024 · 1:12:59

This episode covers two papers: AI2's Molmo open-source vision-language models and OpenAI's Whisper Large V3 Turbo. For Molmo, the key claim is that high-quality, audio-annotated data (Pixmo) enables models as small as 1B to match GPT-4V on academic benchmarks, with the 72B variant outperforming GPT-4o, Gemini 1.5, and Claude 3.5 Sonnet. The team avoided distilling proprietary models by having annotators describe images in speech for 60-90 seconds, then transcribing and augmenting the captions. For Whisper Turbo, OpenAI pruned the decoder from 32 to 4 layers and continued pre-training on 10 million hours of multilingual transcription data, making it 1.78x smaller than Large V3 with minimal word-error-rate increase. The model is faster and supports real-time chunk-based decoding, unlike English-only DistilWhisper. Nathan Lambert and Amgadoz provide commentary on data strategies and benchmarking nuances.

Personal AI Meetup - Bee, BasedHardware, LangChain LangFriend, Deepgram EmilyAI

Apr 6, 2024 · 58:54

This episode features Damien Murphy of Deepgram, Ethan of Owl/Bee, and Harrison of LangChain demonstrating how to build personal AI with real-time voice bots, wearable life-recording devices, and memory-enhanced journaling apps. Damien shows building a voice bot with subsecond latency using Deepgram, OpenAI, and open source code, costing about 6.5 cents per five-minute call. Ethan presents his Owl wearable that continuously records audio, triggers actions via hot word 'Scarlett,' and discusses challenges in adding vision and open source adoption. Harrison introduces LangFriend, a journaling app using conversational, semantic, and knowledge graph memory, referencing the Generative Agents paper for recency and importance weighting. The episode also highlights open source projects Whomane, Friend, and ADeus, arguing that hardware, voice, and memory are all necessary components for personal AI.

Making Transformers Sing - with Mikey Shulman of Suno

Mar 14, 2024 · 58:58

In this episode of Latent Space, hosts Alessio and Shawn Wang interview Mikey Shulman, CEO of Suno, about making transformers sing. Suno uses transformers to predict audio tokens end-to-end, avoiding baked-in musical knowledge, with a tokenization secret sauce that also includes non-music audio for better vocal realism. Their models are relatively small (far below 175B parameters) due to latency needs, and they prioritize scaling research over brute-force size. Over half of Suno users employ expert mode, tweaking lyrics and style prompts, rather than easy mode. Shulman argues Suno is not the 'Midjourney of music' because music is inherently social and synchronous, unlike images. He demoed live generation, showing control via tokens like [beat drop] and style modifiers, and revealed future plans for collaborative concerts, continuous DJ modes, and personalized models. He also advocates for hiring economists to avoid Goodhart's law pitfalls in ML benchmarks, especially in audio where aesthetics matter most.

A Brief History of the Open Source AI Hacker - with Ben Firshman of Replicate

Feb 28, 2024 · 1:21:09

Ben Firshman, CEO of Replicate, explains how the inference platform grew from a research reproducibility tool into a 2M-user API business by embracing the generative image community and treating open-source AI as a hacker-friendly ecosystem. They accidentally discovered their API when a user reverse-engineered their web form, leading to their first $1k/month customer. Cog, their container standard for ML models, was born from lessons at Docker and the need to make models tinkerable. Ben argues that fine-tuning's low cost makes open-source models sustainable, and that AI engineers (orders of magnitude more than ML engineers) just need to start playing with models. He also discusses GPU scarcity, preferring sustainable pricing over price wars, and reveals that demand is not outpacing supply thanks to aggregating demand.

The Four Wars of the AI Stack - Dec 2023 Recap

Jan 26, 2024 · 1:20:58

Swyx and Alessio recap December 2023 by framing the AI landscape as four wars: the Data War (NYT lawsuit demanding destruction of GPTs, OpenAI's data partnerships, synthetic data from DeepMind); the GPU/Inference War (Mixtral price dropping 90% to $0.27/million tokens, benchmark drama between Anyscale and Together, alternative architectures like Mamba); the Multimodality War (Midjourney reaching $200M ARR, ElevenLabs hitting unicorn status, OpenAI and Google building god models); and the RAG/Ops War (LangChain vs LlamaIndex, Pinecone's $750M valuation, Qdrant serving OpenAI and Anthropic internally). They argue agents and open source are not active wars, and predict 2024 will see AI finally reach production, with hardware like Rabbit R1 and Tab capturing unique context.

The Accidental AI Canvas - with Steve Ruiz of tldraw

Jan 5, 2024 · 1:27:49

Steve Ruiz, founder of tldraw, recounts how his open-source whiteboard accidentally became a premier multimodal AI canvas, enabling visual prompting with GPT-4V. Originally built as a personal project with obsessive attention to arrows and ink, tldraw gained traction as a developer SDK for infinite canvas apps. The 'Make It Real' feature, which turns wireframes into working HTML using GPT-4V, exploded with 22 million Twitter views, showcasing iterative design through spatial branching and annotation. Ruiz explains how the canvas supports multiple input layers—UI sketches, state charts, screenshots, even API endpoints—and how his fine arts background guided his taste-driven approach. He argues tldraw's moat lies in its web-native, hackable canvas infrastructure, not ephemeral AI integrations, and positions it as the battlefield for comparing multimodal models like OpenAI's and Gemini.

The AI-First Graphics Editor - with Suhail Doshi of Playground AI

Jan 2, 2024 · 1:09:23

Suhail Doshi, co-founder of Mixpanel and founder of Playground AI, argues that image generation is still in a GPT-2 moment and that training open-source foundation models from scratch is necessary to unlock real utility, exemplified by Playground v2's 2.5x preference over Stable Diffusion XL on a 1K prompt benchmark. He explains the pivot from Mighty (cloud-streamed browser) to AI, driven by the belief that shifting compute elsewhere aligns with AI's parallel computation. The episode covers Playground's unique UI—not just a prompt box but a full canvas with preview rendering, seed control, and style filters—and the difficulty of balancing safety with artistic expression, especially around NSFW content. Suhail discusses the under-investment in graphics AI compared to language, the open-source community's role, and his choice to release pre-trained weights for academic research. He also shares lessons from running GPU infrastructure (harder for training than inference) and advises founders to follow curiosity and build projects rather than relying on books.

Generating your AI Media Empire - with Youssef Rizk of Wondercraft.ai

Sep 20, 2023 · 1:28:55

Youssef Rizk, co-founder of Wondercraft.ai, explains how his five-month-old startup uses hyperrealistic AI voices to turn blogs, newsletters, and other content into podcasts, arguing that the application layer—not the underlying API—is the real moat. He details Wondercraft's origin from a failed sports-investing startup, the rapid MVP that generated $3K in one day, and the decision to build script generation and audio production tools rather than train its own models. Rizk also announces a new AI dubbing feature for video, discusses the challenges of scaling to 28 languages, and reveals that the company runs on just four people pushing two major features per week. The conversation covers the daily Hacker News Recap podcast—which reached Spotify's top 30 tech podcasts—and Rizk's view that AI-generated content must be clearly disclosed to avoid backlash.