Ali Pullen

Building AGI in Real Time (OpenAI Dev Day 2024)

Oct 4, 2024 · 2:09:35

OpenAI's DevDay 2024 unveils the Realtime API with WebSocket-based voice and function calling, Vision Finetuning, Prompt Caching, and Model Distillation. Product leads Olivier Godement, Romain Huet, and Michelle Pokrass detail design choices like zero-code caching and WebSockets for human-level latency. Alistair Pullen of Cosine explains how fine-tuning GPT-4o on custom reasoning traces beat o1 on SWE-bench Verified, while Sam Altman and Kevin Weil discuss AGI levels (o1 as level 2 reasoner), the coming agent era (level 3), safety through iterative deployment, and the importance of building for the frontier. The episode also covers moderation policy relaxation, plans for o1 system prompts by year-end, and the competitive admiration for NotebookLM.

Is finetuning GPT4o worth it?

Aug 22, 2024 · 1:01:16

Ali Pullen, founder of Cosine, discusses Genie, an AI software engineering colleague that achieved 30% on SWE-Bench and 43.8% on SWE-Bench Verified via fine-tuning GPT-4o on billions of tokens of synthetic data. Cosine built Genie by training on the process of software engineering—not just working code—using synthetic runtime errors and a self-improvement loop. The agent follows a four-stage workflow: code retrieval (66% accuracy), planning, writing diffs, and running CI tests. Pullen reveals that model performance degrades linearly beyond 60K tokens in the context window, and that OpenAI dynamically sizes LoRA adapters to handle Cosine's massive dataset. He explains why Genie isn't on the SWE-Bench leaderboard (refusing to publish trajectories to prevent distillation) and outlines plans to scale the dataset and fine-tune on customer codebases for personalized performance.