ELMo

[Paper Club] BERT: Bidirectional Encoder Representations from Transformers

Nov 27, 2024 · 53:12

Eric Ness walks through the BERT paper, explaining how its bidirectional encoder architecture with masked language modeling and next sentence prediction pre-training enabled state-of-the-art results on 11 NLP tasks in 2019. He details the 110M parameter BASE and 340M parameter LARGE models, trained on 2.5B Wikipedia words and 800M book words, and how a simple logistic regression on BERT embeddings achieves 82% accuracy on IMDb sentiment classification, far above the 50% baseline. Swyx and Eric discuss how BERT’s pre-training objectives—masking 15% of tokens and predicting sentence order—teach small models word relationships for efficient classification, though they lose data compared to decoder-only next-token prediction. They note that while scaling laws favor decoder models, BERT-style encoders remain cost-effective for edge deployment, with recent work showing full retraining in 24 hours for under $500 or even 1 hour for $20 on 8 A100s, matching original GLUE scores.

Breaking down the OG GPT Paper by Alec Radford

Apr 23, 2024 · 1:05:03

This episode of the Latent Space Paper Club breaks down Alec Radford's seminal 2018 GPT-1 paper, which introduced generative pre-training of a Transformer decoder on BookCorpus (5 GB, 7,000 books) followed by supervised fine-tuning. Amget explains how the 117M-parameter model achieved state-of-the-art on 9 of 12 NLU tasks, with absolute gains of 8.9% on commonsense reasoning (Stories Cloze) and 5.7% on question answering (RACE). Key innovations include task-agnostic input transformations (e.g., concatenating premise and hypothesis with a delimiter for entailment, using a Siamese architecture for semantic similarity, and scoring answer choices for QA) and an auxiliary language modeling objective during fine-tuning (lambda=0.5). The episode also covers zero-shot heuristics (appending 'very' for sentiment, averaging token log probabilities for QA), ablation studies showing pre-training adds 15% average improvement, and the paper's prescient call to scale up—which became the blueprint for GPT-2, GPT-3, and beyond.