Lindy

Beating Google at Search with Neural PageRank and $5M of H200s — with Will Bryk of Exa.ai

Jan 10, 2025 · 55:54

Will Bryk, CEO of Exa.ai (formerly Metaphor), details building a neural search engine from scratch using link prediction as 'Neural PageRank' — predicting documents rather than keywords. Exa's new product offers near-perfect lists (e.g., 'startups working on hardware in SF') by scaling compute per query, from milliseconds to a day, like o1 for search. Bryk argues LLMs will become the interface to search, while Exa provides the 'super knowledge' that even AGI will need. He contrasts Exa's neural approach with Google's keyword-based system and Perplexity's reliance on Bing. The company recently purchased a $5M H200 cluster and maintains a culture of nap pods and first-principles thinking.

Training Llama 2, 3 & 4: The Path to Open Source AGI — with Thomas Scialom of Meta AI

Jul 23, 2024 · 1:04:34

Thomas Scialom, Llama 2 lead and Llama 3 post-training lead at Meta AI, explains why scaling laws must go beyond Chinchilla—training models longer on more tokens yields better inference efficiency than bigger models, a lesson that drove Llama 3's 15 trillion token pre-training. He details how synthetic data from Llama 2 bootstrapped Llama 3's post-training, eliminating human-written SFT data, and why RLHF outperforms imitation learning: humans are better discriminators than generators, enabling superhuman outputs. Scialom defends the dense 405B architecture over MoE, calls tokenizer vocab size underrated (128k tokens vs Llama 2's 32k), and reveals Llama 4's focus on agentic capabilities—tool use, multi-step reasoning—as the path to open-source AGI. Meta AI is hiring researchers with rigorous first-principles thinking.