ImageNet

After LLMs: Spatial Intelligence and World Models — Fei-Fei Li & Justin Johnson, World Labs

Nov 25, 2025 · 1:00:39

World Labs co-founders Fei-Fei Li and Justin Johnson argue that spatial intelligence is the next frontier beyond LLMs, with their product Marble generating editable 3D worlds from text or images using Gaussian splats. They claim language is a lossy channel for describing the rich 3D/4D world, while spatial intelligence handles everything from picking up a mug to inferring DNA's double helix. Marble already serves gaming, VFX, and film, with precise camera control and real-time rendering on phones and VR headsets. They discuss scaling compute from AlexNet to today's million-fold increase, why world models can 'soak up' modern GPU clusters better than language alone, and the gap between pattern-fitting and causal understanding in physics. Fei-Fei worries more about under-resourced academia than open vs. closed models, advocating for national AI compute clouds and open benchmarks like the BEHAVIOR dataset. Justin notes transformers are natively set models, not sequence models, opening architectural possibilities for new spatial data.

Better Data is All You Need — Ari Morcos, Datology

Aug 29, 2025 · 1:18:43

Ari Morcos, CEO of Datology, argues that data curation is the most impactful and underinvested area in AI, claiming models are defined by what they eat. He details his shift from focusing on inductive biases to realizing data quality is paramount, citing the DCLM study where human experts could not predict automated filtering decisions. Datology's methods combine filtering, rebalancing, curriculum, and rephrasing synthetic data to achieve baseline performance 12x faster, improve accuracy by 4–5 absolute points, and train models with fewer than half the parameters. Morcos explains how curating data bends naive scaling laws by maintaining marginal information gain, and shares results from the Arc 4.5B model, which was trained on 7 trillion tokens (down from 25 trillion) using Datology's curation. He highlights that data is a compute multiplier, making it possible for enterprises to train smaller, cheaper models for specific domains.

Information Theory for Language Models: Jack Morris

Jul 2, 2025 · 1:18:13

Jack Morris, a Cornell PhD student advised by Sasha Rush, discusses his information-theoretic research on language models, including embedding inversion (recovering text from embeddings with 90% accuracy), the universal geometry of embeddings (aligning different models' latent spaces), and measuring model memorization capacity at 3.6 bits per parameter. He argues that paradigm shifts in AI, from AlexNet to instruction tuning, stem not from new architectures but from new datasets, and that the next breakthrough will likely emerge from a novel data source. The conversation also covers the shift from academia to industry, practical advice for grad students on distributed training, and the implications of embedding inversion for privacy and model alignment.