Interpretability

[State of MechInterp] SAEs in Production, Circuit Tracing, AI4Science, "Pragmatic" Interp — Goodfire

Dec 31, 2025 · 21:48

Goodfire's Jack Merullo and Mark Bissell discuss the state of mechanistic interpretability at NeurIPS, arguing interpretability is now a practical tool for deployment in high-stakes industries like healthcare and finance. They introduce paint.goodfire.ai, which lets users paint directly into Stable Diffusion's internal concept map via unsupervised feature discovery. At Rakuten, Goodfire's interpretability-based PII detection proved 500x cheaper than GPT-5 as a judge with higher recall. Merullo presents a memorization vs. reasoning spectrum, showing factual recall sits between rote memorization and logical reasoning. They highlight cross-layer transcoders and circuit tracing for scaling interpretability across all layers. Neil Nanda's pivot to 'pragmatic interpretability' is seen as validation, not retreat, and Goodfire's Pasteur's Quadrant philosophy balances foundational research with applied use cases like novel biomarker discovery in genomics.

⚡️Automating Scientific Discovery - Jessica Rumbelow, Leap Labs

Nov 2, 2025 · 27:32

Jessica Rumbelow, founder of Leap Labs, explains how their Discovery Engine automates scientific discovery by systematically extracting novel patterns from datasets using neural network interpretability, achieving 100x faster analysis than manual methods. The tool has uncovered unexpected findings: novel T-cell receptor markers for tumor reactivity, a synergistic effect between manganese and genotype on root architecture for drought-resistant crops, and a 20% violation of the foundational surface layer assumption in meteorological modeling, worth billions if improved. Rumbelow contrasts Claude 4.1 Opus alone (which hallucinated and overgeneralized) vs. Claude with Discovery Engine (which became powerful at synthesis), arguing that language models need specialized tools for unbiased, hypothesis-free discovery. The platform is domain-agnostic, free for academics publishing their data, and expanding to multimodal datasets and industry pilots.

The Utility of Interpretability — Emmanuel Amiesen

Jun 6, 2025 · 1:53:02

Emmanuel Amiesen, lead author of Anthropic's Circuit Tracing paper, presents the newly open-sourced tools that let anyone inspect how language models compute, revealing that models like Gemma 2-2B engage in genuine multi-step reasoning rather than mere pattern matching. Amiesen demonstrates that the model's internal features causally drive outputs—for instance, suppressing the 'Texas' feature in a capital-of-state query changes the answer from Austin to Sacramento. He explains how sparse autoencoders uncover features shared across languages and modalities, and how attribution graphs map feature interactions. The episode covers practical insights: why jailbreaks succeed (the model prioritizes grammatical completion over refusal) and how planning in poetry generation involves deciding a rhyme before the line begins. Amiesen calls for more researchers to explore interpretability, citing the low barrier to entry with open models and tools.

LLM Asia Paper Club Survey Round

May 22, 2024 · 55:25

The episode surveys four recent papers on LLM reasoning, uncertainty, interpretability, and efficiency. 'Let's Think Dot by Dot' shows that filler tokens (dots) inserted between input and output enable hidden computation, outperforming no-token baselines on tasks like 3-Sum and 2-Sum Transform. 'Uncertainty Estimation' trains a random forest on hidden-layer activations to predict response confidence, achieving higher AUC than unsupervised methods on Q&A and translation. 'Monosemanticity' uses sparse autoencoders to identify interpretable features (e.g., a DNA-detection feature) in a toy transformer, advancing mechanistic interpretability. 'Medusa' attaches multiple prediction heads to the final hidden state to speculate future tokens, enabling faster decoding without a separate draft model and training in five hours on 60K samples.