Mistral 7B

Mistral: Voxtral TTS, Forge, Leanstral, & Mistral 4 — w/ Pavan Kumar Reddy & Guillaume Lample

Mar 30, 2026 · 54:02

Mistral releases Voxtral TTS, a 3B-parameter speech generation model using a novel autoregressive flow matching architecture and a 12.5 Hz neural audio codec developed in-house. Pavan Kumar Reddy and Guillaume Lample explain how the model achieves high quality and efficiency by predicting audio latents as a continuous distribution, enabling real-time voice agent applications. The episode details Mistral's stepwise multimodal strategy, starting with transcription and now speech generation, with full duplex voice agents as the next goal. Enterprise deployment via Mistral Forge allows fine-tuning on proprietary data, exemplified by training on rare languages or domain-specific jargon. Guillaume discusses merging capabilities (Mistral Small merging coding, reasoning, vision), commitment to open science through technical reports, and reasoning transfer from formal proofs in Lean. The episode also covers AI for science partnerships and hiring for research and forward-deployed engineering roles.

Why RL Won — Kyle Corbitt, OpenPipe (acq. CoreWeave)

Oct 16, 2025 · 1:08:23

Kyle Corbitt, co-founder and CEO of OpenPipe (acquired by CoreWeave), explains why reinforcement learning has replaced supervised fine-tuning for training reliable AI agents. He argues GRPO is a dead end due to its requirement for perfectly reproducible parallel rollouts, which is extremely hard in practice. Instead, OpenPipe’s RULER uses relative LLM-as-judge rewards, achieving state-of-the-art performance even with a weak judge. Corbitt reports that 90% of AI projects remain stuck in proof-of-concept due to reliability issues, and that LoRAs are underrated for production while GEPA failed in his tests. He predicts continuous RL from real-world experience can unlock 10x more inference demand.

Best of 2024: Synthetic Data / Smol Models, Loubna Ben Allal, HuggingFace [LS Live! @ NeurIPS 2024]

Dec 24, 2024 · 28:08

Loubna Ben Allal of Hugging Face explains how synthetic data has become ubiquitous in LLM pipelines, moving from post-training to pre-training, with examples like Cosmopedia’s 30B-token synthetic dataset and NVIDIA’s 1.9T-token Nemotron CC. She addresses model collapse fears, showing that web dumps after ChatGPT’s release actually yield better models, and emphasizes diversity through prompt seeds and webpage extracts. For filtering, FineWeb-Edu and DCLM use LLMs to rate educational content, achieving top benchmark scores. In post-training, she highlights AgentInstruct, Tülu3 with PersonaHub, and Cohere’s multilingual arbitrage using multiple teachers. On small models, SmolM2 (1.7B) outperforms Llama 1B and Qwen 2.5 after 11T tokens of pre-training, and on-device inference via frameworks like llama.cpp enables privacy-preserving use cases such as text extraction and structured generation. She predicts a return to fine-tuning specialized small models over costly prompt engineering.

Best of 2024: Open Models [LS LIVE! at NeurIPS 2024]

Dec 23, 2024 · 37:29

Luca Soldani of AI2 and Sophia Yang of Mistral argue that open models in 2024 have exploded in number and performance, closing the gap with closed models, but face growing compute and data-access challenges. Soldani notes 2024 saw models like Qwen and Deepseek rivaling frontier closed-model performance, and fully open models like OLMo release full recipes including data, code, and intermediate checkpoints. He points out that pre-training requires 10K+ GPUs for state-of-the-art, while post-training can be done with as few as eight. Sophia highlights Mistral's release of over a dozen models in 2024, including Pixtral multimodal and Le Chat—a free chat interface with image understanding, canvas code execution, and web search. Soldani warns of data access diminishing due to content owners blocking crawlers and lobbying efforts labeling open source AI as dangerous, emphasizing the need to protect the open ecosystem.

[Paper Club] BERT: Bidirectional Encoder Representations from Transformers

Nov 27, 2024 · 53:12

Eric Ness walks through the BERT paper, explaining how its bidirectional encoder architecture with masked language modeling and next sentence prediction pre-training enabled state-of-the-art results on 11 NLP tasks in 2019. He details the 110M parameter BASE and 340M parameter LARGE models, trained on 2.5B Wikipedia words and 800M book words, and how a simple logistic regression on BERT embeddings achieves 82% accuracy on IMDb sentiment classification, far above the 50% baseline. Swyx and Eric discuss how BERT’s pre-training objectives—masking 15% of tokens and predicting sentence order—teach small models word relationships for efficient classification, though they lose data compared to decoder-only next-token prediction. They note that while scaling laws favor decoder models, BERT-style encoders remain cost-effective for edge deployment, with recent work showing full retraining in 24 hours for under $500 or even 1 hour for $20 on 8 A100s, matching original GLUE scores.

[Paper Club] 🍓 On Reasoning: Q-STaR and Friends!

Sep 18, 2024 · 47:12

This episode of Paper Club surveys three reasoning papers — STaR, Quiet-STaR, and V-STaR — arguing that while STaR is foundational for bootstrapping reasoning via rationales and rationalization, Quiet-STaR's attempt to generate rationales at every token yields only marginal gains (5–10% on GSM8K and CQA), and V-STaR's verifier trained with DPO on both correct and incorrect solutions delivers the most practical improvement, beating majority voting. The host explains STaR's two-loop process of generating rationales and rationalizations from wrong answers, highlights examples like a filtering-straw question where human raters evaluated reasoning quality, and notes that STaR on GPT-J 6B achieved human-like step counts in math problems. V-STaR's verifier selects among candidate solutions, scaling with K candidates, and is compared to process reward models from OpenAI's "let's verify step by step."