DSPy

[State of Research Funding] Beyond NSF, Slingshots, Open Frontiers — Andy Konwinski, Laude Institute

Dec 31, 2025 · 22:34

Andy Konwinski, co-founder of Databricks and Perplexity, launches Laude Institute—a dual venture fund and nonprofit to accelerate the path from open research to breakout companies. At NeurIPS, he explains how Laude's Slingshot program funds projects like DSPy, Terminal Bench, and LMArena, and why the NSF's $1B/year for CS is insufficient, needing $10-100B for frontier AI. He argues Chinese labs (Moonshot, DeepSeek) now outpublish US because OpenAI and others stopped sharing; Open Frontiers, a live-streamed conference in SF, aims to unite top open researchers (Yann LeCun, François Chollet, Jan Leike) to reclaim global leadership. The episode also spotlights the emerging 'post-post-training' layer—prompt optimization, context management, RAG—and why research teams with multiple co-founders (like Databricks' eight) are the new gold standard for AI startups.

The Agent Network — Dharmesh Shah, Agent.ai + CTO of HubSpot

Mar 28, 2025 · 1:42:28

Dharmesh Shah, founder of Agent.ai and co-founder of HubSpot, defines an AI agent broadly as AI-powered software that accomplishes a goal, and argues that the next frontier is multi-agent systems and shared memory across agents. He reveals Agent.ai has 1.3M users and 3,000 published agents, positioning it as a professional network where agents have profiles, post release notes, and can be composed via MCP servers. Shah contrasts work as a service (paying for the work done) versus results as a service (paying for outcomes), warning that results-based pricing only works for objectively measurable, low-variance tasks like customer support tickets. He advocates for MCP as the standard for agent-tool discovery, and shares his personal engineering philosophy of preferring under-engineering over over-engineering because the cost to fix later trends toward zero with AI code generation. Shah also discusses his domain investing strategy (owning chat.com, prompt.com, crew.ai), his rule against competing with Sam Altman, and the 'Sorry Must Pass' framework for managing overwhelm by defaulting to no.

[Paper Club] Who Validates the Validators? Aligning LLM-Judges with Humans (w/ Eugene Yan)

Sep 28, 2024 · 1:00:55

Eugene Yan presents the paper 'Who Validates the Validators? Aligning LLM-Judges with Humans' by Shreya Shankar, introducing EvalGen, a framework that helps developers iteratively align LLM evaluators with human labels by grading outputs, refining criteria, and tracking coverage (recall) and false failure rate (1−precision). Yan demonstrates his own prototype 'Label', which forces users to label 20 samples before unlocking evaluation mode and 50 samples plus one evaluation run before optimization, gamifying the alignment loop. The discussion covers binary vs. pairwise evaluation, the subjectivity of criteria, and Shreya Shankar's plans to integrate natural language feedback into EvalGen's next version. Yan advocates that labeling data is essential before deploying LLM evaluators in production, turning the process into prompt fine-tuning rather than vibe checks.

The Ultimate Guide to Prompting - with Sander Schulhoff from LearnPrompting.org

Sep 20, 2024 · 1:07:09

Sander Schulhoff, author of The Prompt Report and founder of LearnPrompting, presents a comprehensive taxonomy of 58 prompting techniques and argues that prompt engineering is a skill everyone should have, not a specialized role. He categorizes techniques by problem-solving strategy (zero-shot, few-shot, thought generation, decomposition, ensembling, self-criticism) and debunks role prompting for accuracy-based tasks, showing an 'idiot' prompt outperformed a 'genius' prompt on MMLU. Schulhoff shares his experience running Hack-a-Prompt, which collected 600,000 malicious prompts and won best paper at EMNLP, and differentiates between prompt injection and jailbreaking. He advocates for automatic prompt engineering with DSPy, which beat his manual effort in 10 minutes, and previews Hack-a-Prompt 2.0, aiming for a $500,000 prize pool to generate real-world harms for safety tuning.

High Agency Pydantic over VC Backed Frameworks — with Jason Liu of Instructor

Apr 24, 2024 · 1:02:59

Jason Liu, creator of the Instructor library, explains why structured outputs from LLMs are best handled by a simple requests-like wrapper rather than a VC-backed framework, arguing that Pydantic-defined schemas via function calling outperform JSON Mode for typed responses. He details his journey from being bearish on LLMs at StitchFix to building Instructor on a bullet train to Japan after GPT-3 proved him wrong. Liu advocates for workflow-based DAGs over reactive agent loops, recommends using rankers rather than cramming 60+ tools into an API call, and credits high agency—trying many experiments and documenting conditions for revisiting failures—as key to his success. He also critiques the MLE hiring hype, urging startups to empower motivated AI engineers instead.