Eugene

[Paper Club] Weight Streaming on Wafer-Scale Clusters (w/ Sarah Chieng of Cerebras)

Dec 7, 2024 · 43:54

Sarah Chieng of Cerebras presents the weight streaming technique for training giant neural networks on Cerebras wafer-scale clusters, arguing it enables near-linear scaling by separating parameter storage from primary compute. The system uses the Wafer Scale Engine (WSE-2/3) with 900,000 cores and 44 GB on-chip SRAM, avoiding off-chip memory bottlenecks that limit NVIDIA GPUs. MemoryX provides external storage for weights and optimizer states (up to 2.4 petabytes), streaming weights to compute units via the SwarmX interconnect fabric, which aggregates gradients. This design allows training models with up to 120 trillion parameters without complex hybrid parallelism. Cerebras also leverages unstructured weight sparsity to prune 90% of data during transmission and skip zero-value computations on-chip, reducing bandwidth and improving efficiency.

[Paper Club] Embeddings in 2024: OpenAI, Nomic Embed, Jina Embed, cde-small-v1 - with swyx

Dec 1, 2024 · 41:46

Shawn Wang (swyx) surveys four key embedding developments: OpenAI's text-embedding-3 models, Nomic Embed, Jina Embed v3, and CDE-small-v1. OpenAI's models introduce Matryoshka embeddings, compressing output from 1024 to 64 dimensions (94% storage reduction) with only an 8% performance drop. Nomic Embed is a fully open-source BERT-based model trained on English data, offering reproducible code. Jina Embed v3 supports 89 languages and uses task-specific LoRA adapters—e.g., a retrieval adapter yields a huge boost over the base model. CDE-small-v1 (143M parameters) employs a two-stage adaptation: first conditioning on the corpus, then embedding; it outperforms 7B-parameter models on many tasks, though its stateful API complicates deployment.

[Paper Club] SWE-Bench [OpenAI Verified/Multimodal] + MLE-Bench with Jesse Hu

Oct 19, 2024 · 1:01:40

Jesse Hu walks through SWE-Bench, SWE-Bench Verified, SWE-Bench Multimodal, and MLE-Bench — benchmarks that evaluate AI agents on real-world coding and ML tasks, from fixing GitHub issues to winning Kaggle competitions. SWE-Bench scrapes 2,000+ pull requests from 12 Python repos to get free unit tests, with leaderboard scores rising from ~2% to 43% thanks to improved agent scaffolding. SWE-Bench Verified uses human raters and three criteria—well-specified problem, valid test, difficulty on a 0–3 scale—to filter 500 solvable tasks. SWE-Bench Multimodal targets 617 UI tasks from 17 JavaScript/TypeScript libraries, each including an image, but achieves only 12% accuracy. MLE-Bench repurposes Kaggle competitions: agents given 24 hours and an A10 GPU, with o1-preview scoring bronze in 17% of competitions and surpassing grandmasters (7 gold medals vs 5 required), though each run costs $4,000.

[Paper Club] Writing in the Margins: Chunked Prefill KV Caching for Long Context Retrieval

Sep 19, 2024 · 53:53

Umar Jamil from Writer presents 'Writing in the Margins,' a paper on improving long-context retrieval in LLMs by leveraging chunked prefill of the KV cache. The method splits a long prompt into chunks, generates query-relevant annotations (margins) after each chunk, and appends them at the end before generating the final answer. This approach solves the 'lost in the middle' problem without fine-tuning, and unlike traditional RAG or separate summarization, it avoids re-prefilling the entire context, reducing cost from double to single prefilling. The technique is compatible with any transformer model and includes overlapping margin generation and classification within the same batch request. Benchmarks show consistent improvements across models, and the implementation is open-source on GitHub.

RWKV: Reinventing RNNs for the Transformer Era

Aug 31, 2023 · 1:56:51

Eugene, CTO of UILicious, introduces RWKV, a recurrent neural network that matches transformer-level performance while scaling linearly with context length, solving the quadratic cost of attention. He explains how RWKV uses Attention-Free Transformer layers and parallelizable training, achieving competitive results on 7B and 14B models. The community-driven project prioritizes multilingual support, with a tokenizer that handles languages without spaces. Eugene recounts his journey from GPU.js to finding RWKV, and highlights use cases like long HTML analysis where transformers fail. He also discusses the token crisis, diffusion models for text, and how the AI waifu community drives optimization and alignment research. Finally, he advises AI engineers to focus on practical prompting and data curation, noting that deep architecture knowledge is optional.