Best of 2024: Open Models [LS LIVE! at NeurIPS 2024]
Dec 23, 2024 · 37:29
Luca Soldani of AI2 and Sophia Yang of Mistral argue that open models in 2024 have exploded in number and performance, closing the gap with closed models, but face growing compute and data-access challenges. Soldani notes 2024 saw models like Qwen and Deepseek rivaling frontier closed-model performance, and fully open models like OLMo release full recipes including data, code, and intermediate checkpoints. He points out that pre-training requires 10K+ GPUs for state-of-the-art, while post-training can be done with as few as eight. Sophia highlights Mistral's release of over a dozen models in 2024, including Pixtral multimodal and Le Chat—a free chat interface with image understanding, canvas code execution, and web search. Soldani warns of data access diminishing due to content owners blocking crawlers and lobbying efforts labeling open source AI as dangerous, emphasizing the need to protect the open ecosystem.
