Kaggle

[Paper Club] SWE-Bench [OpenAI Verified/Multimodal] + MLE-Bench with Jesse Hu

Oct 19, 2024 · 1:01:40

Jesse Hu walks through SWE-Bench, SWE-Bench Verified, SWE-Bench Multimodal, and MLE-Bench — benchmarks that evaluate AI agents on real-world coding and ML tasks, from fixing GitHub issues to winning Kaggle competitions. SWE-Bench scrapes 2,000+ pull requests from 12 Python repos to get free unit tests, with leaderboard scores rising from ~2% to 43% thanks to improved agent scaffolding. SWE-Bench Verified uses human raters and three criteria—well-specified problem, valid test, difficulty on a 0–3 scale—to filter 500 solvable tasks. SWE-Bench Multimodal targets 617 UI tasks from 17 JavaScript/TypeScript libraries, each including an image, but achieves only 12% accuracy. MLE-Bench repurposes Kaggle competitions: agents given 24 hours and an A10 GPU, with o1-preview scoring bronze in 17% of competitions and surpassing grandmasters (7 gold medals vs 5 required), though each run costs $4,000.

LLM Asia Paper Club Survey Round

May 22, 2024 · 55:25

The episode surveys four recent papers on LLM reasoning, uncertainty, interpretability, and efficiency. 'Let's Think Dot by Dot' shows that filler tokens (dots) inserted between input and output enable hidden computation, outperforming no-token baselines on tasks like 3-Sum and 2-Sum Transform. 'Uncertainty Estimation' trains a random forest on hidden-layer activations to predict response confidence, achieving higher AUC than unsupervised methods on Q&A and translation. 'Monosemanticity' uses sparse autoencoders to identify interpretable features (e.g., a DNA-detection feature) in a toy transformer, advancing mechanistic interpretability. 'Medusa' attaches multiple prediction heads to the final hidden state to speculate future tokens, enabling faster decoding without a separate draft model and training in five hours on 60K samples.

The End of Finetuning — with Jeremy Howard of Fast.ai

Oct 20, 2023 · 1:24:48

Jeremy Howard of Fast.ai argues that fine-tuning language models is essentially continued pre-training, not a separate process, and that the common practice of fine-tuning on a single task causes catastrophic forgetting. He recounts the discovery of single-shot memorization in LLMs, where models memorize entire datasets after one epoch, a phenomenon many practitioners ignore. Howard criticizes the current focus on zero-shot and few-shot learning, advocating for transfer learning and small models. He shares his journey from philosophy to founding Fast.ai, the creation of ULMFit (which inspired GPT), and his ongoing work on making AI accessible. He also discusses his involvement with Modular's Mojo language and the importance of democratizing AI technology.