Fuyu

The 10,000x Yolo Researcher Metagame — with Yi Tay of Reka

Jul 5, 2024 · 2:18:43

Yi Tay, chief scientist at Reka and former Google Brain architecture co-lead on PaLM 2, recounts his journey from an NTU PhD to co-founding Reka, where a team of just 5 people pre-trained Reka Core on $60M of GPU compute—debating at #7 on the LMsys leaderboard. He explains why he believes the Noam transformer (with SwiGLU, GQA, RoPE) remains the strongest baseline, how encoder-decoder architectures offer 'free sparsity' (a 2× flop-efficiency over decoder-only), and why Chinchilla scaling laws are often misunderstood (training past the optimal compute frontier is routine). He argues that long context will eventually outperform RAG for complex reasoning, and that open-source models like Llama 3 are catching up only because Meta invested in a top-tier training stack, not due to grassroots innovation. The episode also covers his productivity habits (working backwards from a paper title, camping arXiv) and the cultural shock of moving from Singapore academia to Google's impact-driven research environment.

Why Google failed to make GPT-3 -- with David Luan of Adept

Mar 27, 2024 · 49:27

David Luan, co-founder of Adept and former early OpenAI leader, explains why Google failed to build GPT-3—its 'brain credit marketplace' prevented critical mass—and why Adept builds enterprise AI agents prioritizing reliability over generality. He recounts the GPT-2 demo that helped secure Microsoft's $1B investment and details Adept's goal: an AI teammate that can do anything a human does on a computer, targeting 'nines of reliability' for workflows like dispatching a physical truck. Luan contrasts Adept's vertical integration—training fast multimodal models (Fuyu) for charts and UIs—with pure-play foundation model companies that sell tokens, predicting commoditization. He notes Adept is sold out for Q1 and raised $420M, and explains how an augmentation focus creates a data flywheel from human oversight.