Molmo

Best of 2024: Open Models [LS LIVE! at NeurIPS 2024]

Dec 23, 2024 · 37:29

Luca Soldani of AI2 and Sophia Yang of Mistral argue that open models in 2024 have exploded in number and performance, closing the gap with closed models, but face growing compute and data-access challenges. Soldani notes 2024 saw models like Qwen and Deepseek rivaling frontier closed-model performance, and fully open models like OLMo release full recipes including data, code, and intermediate checkpoints. He points out that pre-training requires 10K+ GPUs for state-of-the-art, while post-training can be done with as few as eight. Sophia highlights Mistral's release of over a dozen models in 2024, including Pixtral multimodal and Le Chat—a free chat interface with image understanding, canvas code execution, and web search. Soldani warns of data access diminishing due to content owners blocking crawlers and lobbying efforts labeling open source AI as dangerous, emphasizing the need to protect the open ecosystem.

[Paper Club] Molmo + Pixmo + Whisper 3 Turbo - with Vibhu Sapra, Nathan Lambert, Amgadoz

Oct 13, 2024 · 1:12:59

This episode covers two papers: AI2's Molmo open-source vision-language models and OpenAI's Whisper Large V3 Turbo. For Molmo, the key claim is that high-quality, audio-annotated data (Pixmo) enables models as small as 1B to match GPT-4V on academic benchmarks, with the 72B variant outperforming GPT-4o, Gemini 1.5, and Claude 3.5 Sonnet. The team avoided distilling proprietary models by having annotators describe images in speech for 60-90 seconds, then transcribing and augmenting the captions. For Whisper Turbo, OpenAI pruned the decoder from 32 to 4 layers and continued pre-training on 10 million hours of multilingual transcription data, making it 1.78x smaller than Large V3 with minimal word-error-rate increase. The model is faster and supports real-time chunk-based decoding, unlike English-only DistilWhisper. Nathan Lambert and Amgadoz provide commentary on data strategies and benchmarking nuances.