DCLM

Better Data is All You Need — Ari Morcos, Datology

Aug 29, 2025 · 1:18:43

Ari Morcos, CEO of Datology, argues that data curation is the most impactful and underinvested area in AI, claiming models are defined by what they eat. He details his shift from focusing on inductive biases to realizing data quality is paramount, citing the DCLM study where human experts could not predict automated filtering decisions. Datology's methods combine filtering, rebalancing, curriculum, and rephrasing synthetic data to achieve baseline performance 12x faster, improve accuracy by 4–5 absolute points, and train models with fewer than half the parameters. Morcos explains how curating data bends naive scaling laws by maintaining marginal information gain, and shares results from the Arc 4.5B model, which was trained on 7 trillion tokens (down from 25 trillion) using Datology's curation. He highlights that data is a compute multiplier, making it possible for enterprises to train smaller, cheaper models for specific domains.

Best of 2024: Open Models [LS LIVE! at NeurIPS 2024]

Dec 23, 2024 · 37:29

Luca Soldani of AI2 and Sophia Yang of Mistral argue that open models in 2024 have exploded in number and performance, closing the gap with closed models, but face growing compute and data-access challenges. Soldani notes 2024 saw models like Qwen and Deepseek rivaling frontier closed-model performance, and fully open models like OLMo release full recipes including data, code, and intermediate checkpoints. He points out that pre-training requires 10K+ GPUs for state-of-the-art, while post-training can be done with as few as eight. Sophia highlights Mistral's release of over a dozen models in 2024, including Pixtral multimodal and Le Chat—a free chat interface with image understanding, canvas code execution, and web search. Soldani warns of data access diminishing due to content owners blocking crawlers and lobbying efforts labeling open source AI as dangerous, emphasizing the need to protect the open ecosystem.