BFCL

⚡️Ranking Agentic LLMs — Pratik Bhavsar, Galileo

Jul 14, 2025 · 34:10

Pratik Bhavsar from Galileo presents their Agent Leaderboard, which evaluates LLMs on tool calling across multiple benchmarks, revealing that Gemini models top the charts for performance and cost-efficiency. The leaderboard, built on datasets like BFCL, XLAM, and Tau Bench, uses an LLM-as-judge metric called Tool Selection Quality (TSQ). Surprising findings include reasoning models like o1 struggling with multi-tool outputs, Mistral Small excelling as the best open-source model, and Llama models performing poorly. The episode also previews V2 of the leaderboard, which introduces domain-specific, harder, multi-turn scenarios with a user and tool simulator, and a new Action Completion metric to measure whether all user requests are accomplished.

[Paper Club] Berkeley Function Calling Paper Club! — Sam Julien, Writer

Oct 5, 2024 · 41:29

Sam Julien (Writer) walks through the Berkeley Function Calling Leaderboard (BFCL) and its three versions, detailing how each iteration improves evaluation of LLM function calling capabilities. BFCL v1 (March) introduced abstract syntax tree evaluation across 2,000 function-answer pairs, covering simple, multiple, and parallel calling. v2 (August) added a live dataset of 2,251 pairs, reducing bias and contamination through Rouge-score deduplication and filtering. v3 (September) introduced multi-turn and multi-step evaluation with state-based metrics, using a custom API (vehicle control, stock trading, etc.) and human-validated ground truth. Sam also highlights common model failures—misunderstanding current state, unnecessary planning—and mentions the Gorilla CLI and Open Functions as related tools in the Berkeley ecosystem.