BERT

The AI Frontier: from Gemini 3 Deep Think distilling to Flash — Jeff Dean

Feb 12, 2026 · 1:23:31

Jeff Dean, Google's Chief AI Scientist, explains how the company's dual strategy of frontier "Pro" models and distilled "Flash" models, alongside co-designed TPUs and latency-optimization, keeps Gemini at the Pareto frontier. Distillation compresses high-capability large models into smaller, efficient ones, enabling Flash models to exceed earlier Pro generations. He emphasizes energy-based thinking: data movement costs thousands more picojoules than arithmetic, making batching and speculative decoding essential. TPU development requires predicting ML workloads 2–6 years out, often speculatively adding hardware features. Gemini was born after Dean wrote a one-page memo arguing that fragmented model efforts across Google Brain and DeepMind should merge into a single unified multimodal effort. Looking ahead, he predicts personalized models attending to all user data and reasoning speeds of 10,000 tokens/second will transform software engineering.

Information Theory for Language Models: Jack Morris

Jul 2, 2025 · 1:18:13

Jack Morris, a Cornell PhD student advised by Sasha Rush, discusses his information-theoretic research on language models, including embedding inversion (recovering text from embeddings with 90% accuracy), the universal geometry of embeddings (aligning different models' latent spaces), and measuring model memorization capacity at 3.6 bits per parameter. He argues that paradigm shifts in AI, from AlexNet to instruction tuning, stem not from new architectures but from new datasets, and that the next breakthrough will likely emerge from a novel data source. The conversation also covers the shift from academia to industry, practical advice for grad students on distributed training, and the implications of embedding inversion for privacy and model alignment.

Outlasting Noam Shazeer, Crowdsourcing Chai AI w/ 1.4m DAU — with William Beauchamp, Chai Research

Jan 26, 2025 · 1:13:33

William Beauchamp, founder-CEO of Chai AI, explains how he pivoted from algorithmic trading to build a character chatbot platform before Character.ai, growing to 1.4M DAU and $22M+ revenue by crowdsourcing model improvements through Chaiverse. Starting with GPT-J in 2021, he found product-market fit with a therapist bot and shifted to user-generated content, letting users define prompts, images, and names. Competing against well-funded rivals like Character AI and Talkie, Chai ships over 100 LLMs weekly via its developer platform, spending $10M on compute in 2024 and tripling it. Beauchamp argues AI follows an S-curve, not scaling laws, and focuses on inference optimization using rejection sampling and reward models to serve better responses. He prioritizes 'insanely great' products over technology-driven features, noting that audio and image features failed to move metrics, while a data flywheel and aggressive user acquisition drove rapid growth.

[Paper Club] Embeddings in 2024: OpenAI, Nomic Embed, Jina Embed, cde-small-v1 - with swyx

Dec 1, 2024 · 41:46

Shawn Wang (swyx) surveys four key embedding developments: OpenAI's text-embedding-3 models, Nomic Embed, Jina Embed v3, and CDE-small-v1. OpenAI's models introduce Matryoshka embeddings, compressing output from 1024 to 64 dimensions (94% storage reduction) with only an 8% performance drop. Nomic Embed is a fully open-source BERT-based model trained on English data, offering reproducible code. Jina Embed v3 supports 89 languages and uses task-specific LoRA adapters—e.g., a retrieval adapter yields a huge boost over the base model. CDE-small-v1 (143M parameters) employs a two-stage adaptation: first conditioning on the corpus, then embedding; it outperforms 7B-parameter models on many tasks, though its stateful API complicates deployment.

[Paper Club] BERT: Bidirectional Encoder Representations from Transformers

Nov 27, 2024 · 53:12

Eric Ness walks through the BERT paper, explaining how its bidirectional encoder architecture with masked language modeling and next sentence prediction pre-training enabled state-of-the-art results on 11 NLP tasks in 2019. He details the 110M parameter BASE and 340M parameter LARGE models, trained on 2.5B Wikipedia words and 800M book words, and how a simple logistic regression on BERT embeddings achieves 82% accuracy on IMDb sentiment classification, far above the 50% baseline. Swyx and Eric discuss how BERT’s pre-training objectives—masking 15% of tokens and predicting sentence order—teach small models word relationships for efficient classification, though they lose data compared to decoder-only next-token prediction. They note that while scaling laws favor decoder models, BERT-style encoders remain cost-effective for edge deployment, with recent work showing full retraining in 24 hours for under $500 or even 1 hour for $20 on 8 A100s, matching original GLUE scores.

Breaking down the OG GPT Paper by Alec Radford

Apr 23, 2024 · 1:05:03

This episode of the Latent Space Paper Club breaks down Alec Radford's seminal 2018 GPT-1 paper, which introduced generative pre-training of a Transformer decoder on BookCorpus (5 GB, 7,000 books) followed by supervised fine-tuning. Amget explains how the 117M-parameter model achieved state-of-the-art on 9 of 12 NLU tasks, with absolute gains of 8.9% on commonsense reasoning (Stories Cloze) and 5.7% on question answering (RACE). Key innovations include task-agnostic input transformations (e.g., concatenating premise and hypothesis with a delimiter for entailment, using a Siamese architecture for semantic similarity, and scoring answer choices for QA) and an auxiliary language modeling objective during fine-tuning (lambda=0.5). The episode also covers zero-shot heuristics (appending 'very' for sentiment, averaging token log probabilities for QA), ablation studies showing pre-training adds 15% average improvement, and the paper's prescient call to scale up—which became the blueprint for GPT-2, GPT-3, and beyond.

A Comprehensive Overview of Large Language Models - Latent Space Paper Club

Mar 15, 2024 · 54:30

Brian presents a detailed walkthrough of the survey paper 'A Comprehensive Overview of Large Language Models,' which systematically covers the evolution of LLMs from early attention mechanisms to modern transformer architectures. He explains the shift from sequential RNNs to parallelizable transformers, the three dominant architectures (encoder-only like BERT, encoder-decoder like T5, decoder-only like GPT), and key training objectives (masked language modeling, full language modeling, prefix language modeling). The paper also reviews fine-tuning techniques (instruction tuning, alignment tuning via RLHF), prompting strategies (zero-shot, chain-of-thought), efficient adaptation methods (LoRA, quantization), evaluation benchmarks (GLUE, MMLU), and challenges such as bias, memorization, and privacy leaks. This episode serves as a comprehensive primer for anyone seeking to understand the foundational concepts and latest trends in large language model research.