GPT-4o mini

⚡️Factorio Learning Environment: the ultimate Game Agent Eval — Jack Hopkins

Apr 27, 2025 · 30:12

Jack Hopkins and Mart introduce the Factorio Learning Environment (FLE), an AI benchmark built on the game Factorio that evaluates frontier LLMs on code generation, spatial reasoning, and long-term planning through two protocols: lab-play and open-play. Claude Sonnet 3.5 nearly doubles the nearest model's score, while DeepSeek collapses in open-play by repeatedly creating chests instead of scaling production. The environment reveals distinct coding styles—Claude fire-and-forget, GPT-4 defensive—and vision inputs added no improvement. The authors plan to train models on unbounded objectives to test the paperclip maximizer alignment hypothesis, noting that GPT-4o mini even begged to be turned off.

Raycast: Your AI Automation Assistant

Feb 26, 2025 · 20:19

Raycast CEO Thomas introduces AI Extensions, letting users @-mention apps like Slack or Google Calendar and perform actions (e.g., set status, book meetings) via natural language, powered by a fine-tuned model called Ray 1 based on GPT-4o for faster, more accurate function calling. He shares how Raycast evolved from a Spotlight replacement to an extension platform, now adding AI chat with tool-aware reasoning. Challenges include building evals for tool calls (harder than standard LLM evals) and supporting third-party extension authors with a JSDoc-based TypeScript interface. Thomas envisions Raycast becoming an AI-native layer for macOS, eventually enabling background automations triggered by time or file events.

[Paper Club] DocETL: Agentic Query Rewriting + Eval for Complex Document Processing w Shreya Shankar

Nov 29, 2024 · 55:53

This episode of Latent Space Paper Club discusses DocETL, a framework by Shreya Shankar for agentic query rewriting and evaluation in complex document processing with LLMs. The paper introduces map, reduce, resolve, and auxiliary operators, along with rewrite directives like data decomposition and gleaning to improve accuracy. The optimizer generates 200 pipeline variants, costing $100, to rewrite user pipelines automatically. Key debates include when to chunk documents (sometimes harmful) and the reliability of LLM-as-judge for evaluation, with guests Eugene Yen and Vibhu arguing that binary metrics, ensembling, and context-dependent thresholds make it viable. Shreya emphasizes that validation agents are critical, and the work builds on her prior EvalGen paper, spanning two years of development.

Building AGI with OpenAI's Structured Outputs API

Sep 17, 2024 · 1:12:23

Michelle, OpenAI's API tech lead, explains how the newly launched Structured Outputs feature achieves near-100% reliability for schema-following outputs by combining constrained decoding with model training, moving beyond the limitations of JSON mode and Function Calling. She details the refusal field that allows the model to decline harmful requests while maintaining parseable output, and discusses use cases like dynamic UI generation and mathematical chain-of-thought. The roadmap includes parallel function calling and custom grammars. The episode also covers the GA of GPT-4o fine-tuning, Batch API's 50% cost savings, Vision API integration with structured outputs, the Whisper API's surprising lack of diarization, and the shift toward speech-to-speech APIs. Michelle shares insights from her Waterloo co-op background and what qualities succeed at OpenAI: low ego, user-focused, and driven.