Whisper

Mistral: Voxtral TTS, Forge, Leanstral, & Mistral 4 — w/ Pavan Kumar Reddy & Guillaume Lample

Mar 30, 2026 · 54:02

Mistral releases Voxtral TTS, a 3B-parameter speech generation model using a novel autoregressive flow matching architecture and a 12.5 Hz neural audio codec developed in-house. Pavan Kumar Reddy and Guillaume Lample explain how the model achieves high quality and efficiency by predicting audio latents as a continuous distribution, enabling real-time voice agent applications. The episode details Mistral's stepwise multimodal strategy, starting with transcription and now speech generation, with full duplex voice agents as the next goal. Enterprise deployment via Mistral Forge allows fine-tuning on proprietary data, exemplified by training on rare languages or domain-specific jargon. Guillaume discusses merging capabilities (Mistral Small merging coding, reasoning, vision), commitment to open science through technical reports, and reasoning transfer from formal proofs in Lean. The episode also covers AI for science partnerships and hiring for research and forward-deployed engineering roles.

Personalized AI Language Education — with Andrew Hsu, Speak

Jul 11, 2025 · 1:04:10

Andrew Hsu, CTO of Speak, explains how the company built the "third generation" of language learning by betting on AI speech and language models before they were ready. Speak focused on South Korea early, a counterintuitive move for a San Francisco startup, and now 6% of the Korean population has tried the app. The turning point came with Whisper and GPT in 2022, which enabled Speak to evolve from a listen-and-repeat tool into a full-featured AI tutor that gives real-time feedback and adapts to learners. Speak generates over $50M ARR primarily from consumer subscriptions, and Hsu details the product decisions that made it work: abandoning a free version, building custom ASR for low latency, and emphasizing functional fluency over textbook language. He also reveals that Speak is building a real-time voice platform and a knowledge graph for personalized fluency scores, with an eye toward expanding beyond language into general AI-powered education.

Snipd: The AI Podcast App for Learning — with CEO Kevin Ben-Smith

Mar 14, 2025 · 1:17:46

Snipd CEO Kevin Ben-Smith joins Swyx to explain how the AI-powered podcast app turns passive listening into active learning. Snipd transcribes and diarizes episodes, generates chapters, identifies mentioned books, and summarizes insights on demand — all with a four-person team. Kevin describes moving from self-hosted open-source models (Wave2vec, Pyannote) to a mix of Whisper, OpenAI, Google Gemini, and Perplexity APIs, balancing cost against the need for predictable batch processing vs. real-time user requests. He details technical hurdles like dynamic content resyncing (building a 'Shazam for podcasts') and using LLMs as judges to filter hallucinated book recommendations. The episode also covers Snipd's origin as a hackathon-winning podcast search tool, Swyx's own switch from Overcast, and Kevin's vision for voice-driven post-episode reflection and AI-assisted discovery as the next frontier.

DeepSeek V3, SGLang, and the state of Open Model Inference in 2025 (Quantization, MoEs, Pricing)

Jan 19, 2025 · 57:18

Amir Haghighat and Yineng Zhang from Baseten explain how they serve DeepSeek V3, a 671-billion-parameter MoE model requiring H200 clusters with FP8 support, making it the top open-weights model. They detail Baseten’s dedicated inference model (no shared endpoints) and its reliance on SGLang for performance gains like Radix prefix caching and MLA attention support. Yineng highlights SGLang’s edge over vLLM and TensorRT-LLM in usability and customization, especially for large models. Amir outlines three pillars for mission-critical inference: model-level performance (via frameworks like SGLang), horizontal scaling across regions and clouds, and low-latency multi-model workflows. The episode also covers quantization trends, MoE architecture, and speculative decoding as key enablers for open model deployment.

[Paper Club] Molmo + Pixmo + Whisper 3 Turbo - with Vibhu Sapra, Nathan Lambert, Amgadoz

Oct 13, 2024 · 1:12:59

This episode covers two papers: AI2's Molmo open-source vision-language models and OpenAI's Whisper Large V3 Turbo. For Molmo, the key claim is that high-quality, audio-annotated data (Pixmo) enables models as small as 1B to match GPT-4V on academic benchmarks, with the 72B variant outperforming GPT-4o, Gemini 1.5, and Claude 3.5 Sonnet. The team avoided distilling proprietary models by having annotators describe images in speech for 60-90 seconds, then transcribing and augmenting the captions. For Whisper Turbo, OpenAI pruned the decoder from 32 to 4 layers and continued pre-training on 10 million hours of multilingual transcription data, making it 1.78x smaller than Large V3 with minimal word-error-rate increase. The model is faster and supports real-time chunk-based decoding, unlike English-only DistilWhisper. Nathan Lambert and Amgadoz provide commentary on data strategies and benchmarking nuances.

Breaking down the OG GPT Paper by Alec Radford

Apr 23, 2024 · 1:05:03

This episode of the Latent Space Paper Club breaks down Alec Radford's seminal 2018 GPT-1 paper, which introduced generative pre-training of a Transformer decoder on BookCorpus (5 GB, 7,000 books) followed by supervised fine-tuning. Amget explains how the 117M-parameter model achieved state-of-the-art on 9 of 12 NLU tasks, with absolute gains of 8.9% on commonsense reasoning (Stories Cloze) and 5.7% on question answering (RACE). Key innovations include task-agnostic input transformations (e.g., concatenating premise and hypothesis with a delimiter for entailment, using a Siamese architecture for semantic similarity, and scoring answer choices for QA) and an auxiliary language modeling objective during fine-tuning (lambda=0.5). The episode also covers zero-shot heuristics (appending 'very' for sentiment, averaging token log probabilities for QA), ablation studies showing pre-training adds 15% average improvement, and the paper's prescient call to scale up—which became the blueprint for GPT-2, GPT-3, and beyond.