Writer

[Paper Club] Berkeley Function Calling Paper Club! — Sam Julien, Writer

Oct 5, 2024 · 41:29

Sam Julien (Writer) walks through the Berkeley Function Calling Leaderboard (BFCL) and its three versions, detailing how each iteration improves evaluation of LLM function calling capabilities. BFCL v1 (March) introduced abstract syntax tree evaluation across 2,000 function-answer pairs, covering simple, multiple, and parallel calling. v2 (August) added a live dataset of 2,251 pairs, reducing bias and contamination through Rouge-score deduplication and filtering. v3 (September) introduced multi-turn and multi-step evaluation with state-based metrics, using a custom API (vehicle control, stock trading, etc.) and human-validated ground truth. Sam also highlights common model failures—misunderstanding current state, unnecessary planning—and mentions the Gorilla CLI and Open Functions as related tools in the Berkeley ecosystem.

[Paper Club] Writing in the Margins: Chunked Prefill KV Caching for Long Context Retrieval

Sep 19, 2024 · 53:53

Umar Jamil from Writer presents 'Writing in the Margins,' a paper on improving long-context retrieval in LLMs by leveraging chunked prefill of the KV cache. The method splits a long prompt into chunks, generates query-relevant annotations (margins) after each chunk, and appends them at the end before generating the final answer. This approach solves the 'lost in the middle' problem without fine-tuning, and unlike traditional RAG or separate summarization, it avoids re-prefilling the entire context, reducing cost from double to single prefilling. The technique is compatible with any transformer model and includes overlapping margin generation and classification within the same batch request. Benchmarks show consistent improvements across models, and the implementation is open-source on GitHub.