Reka

Captaining IMO Gold, Deep Think, On-Policy RL, Feeling the AGI in Singapore — Yi Tay

Jan 23, 2026 · 1:32:05

Yi Tay, who leads Google DeepMind's Reasoning and AGI team in Singapore, explains how Gemini Deep Think achieved IMO Gold by abandoning symbolic AlphaProof for an end-to-end RL-trained model. He details the on-policy RL philosophy—models learn from their own generated outputs rather than imitating others—and the critical role of self-consistency through parallel sampling and internal verification. Tay describes the IMO effort: four co-captains in different time zones, a one-week training sprint, a live competition in Australia where researchers punched in problems as they were released, and the tension of waiting for human scores to determine the gold threshold. He discusses why the team believes one model must subsume everything for AGI, the data efficiency gap compared to humans, and his hiring focus on raw talent and research taste. Tay also shares his personal fitness transformation—losing 23 kilos and improving HRV—as integral to research productivity.

The 10,000x Yolo Researcher Metagame — with Yi Tay of Reka

Jul 5, 2024 · 2:18:43

Yi Tay, chief scientist at Reka and former Google Brain architecture co-lead on PaLM 2, recounts his journey from an NTU PhD to co-founding Reka, where a team of just 5 people pre-trained Reka Core on $60M of GPU compute—debating at #7 on the LMsys leaderboard. He explains why he believes the Noam transformer (with SwiGLU, GQA, RoPE) remains the strongest baseline, how encoder-decoder architectures offer 'free sparsity' (a 2× flop-efficiency over decoder-only), and why Chinchilla scaling laws are often misunderstood (training past the optimal compute frontier is routine). He argues that long context will eventually outperform RAG for complex reasoning, and that open-source models like Llama 3 are catching up only because Meta invested in a top-tier training stack, not due to grassroots innovation. The episode also covers his productivity habits (working backwards from a paper title, camping arXiv) and the cultural shock of moving from Singapore academia to Google's impact-driven research environment.