BART

[Paper Club] BERT: Bidirectional Encoder Representations from Transformers

Nov 27, 2024 · 53:12

Eric Ness walks through the BERT paper, explaining how its bidirectional encoder architecture with masked language modeling and next sentence prediction pre-training enabled state-of-the-art results on 11 NLP tasks in 2019. He details the 110M parameter BASE and 340M parameter LARGE models, trained on 2.5B Wikipedia words and 800M book words, and how a simple logistic regression on BERT embeddings achieves 82% accuracy on IMDb sentiment classification, far above the 50% baseline. Swyx and Eric discuss how BERT’s pre-training objectives—masking 15% of tokens and predicting sentence order—teach small models word relationships for efficient classification, though they lose data compared to decoder-only next-token prediction. They note that while scaling laws favor decoder models, BERT-style encoders remain cost-effective for edge deployment, with recent work showing full retraining in 24 hours for under $500 or even 1 hour for $20 on 8 A100s, matching original GLUE scores.

Beating GPT-4 with Open Source Models - with Michael Royzen of Phind

Nov 3, 2023 · 1:18:54

Michael Royzen, co-founder and CEO of Phind, explains how his company built a GPT-4-beating open-source model for developer Q&A. Royzen recounts founding SmartLens in high school, shifting to NLP after a Hugging Face demo, and creating an internet-scale LLM-powered RAG system in January 2022. He details Phind’s pivot to programmers, the Hacker News launch that gave it 1,500 points, and Paul Graham’s role in naming the company and introducing Ron Conway, who then connected Phind to NVIDIA for GPU access. Royzen argues that Phind’s model, fine-tuned from Code Llama 34B with extra data, closes the gap with proprietary models—especially on code reasoning—and that open-source will win the enterprise because the delta to GPT-5 will be small. He shares how Phind handles multi-step conversations via a pair programmer mode where users can pin messages, and reveals plans for reinforcement learning to reduce hallucination and improve correctness.