Computer Vision

SAM 3: The Eyes for AI — Nikhila & Pengchuan (Meta Superintelligence), ft. Joseph Nelson (Roboflow)

Dec 18, 2025 · 1:15:04

Meta's SAM 3, introduced by Nikhila Ravi and Pengchuan Zhang alongside Roboflow CEO Joseph Nelson, unifies interactive segmentation, open-vocabulary detection, and video tracking into a single model that runs in 30ms on images and scales to real-time video on multi-GPU setups. The model uses concept prompts like "yellow school bus" to detect and segment every instance, separating recognition from localization via a presence token. Its data engine automated exhaustive annotation from two minutes per image down to 25 seconds using AI verifiers fine-tuned on Llama, while the new SACO benchmark contains over 200,000 unique concepts versus previous 1.2k. For video, decoupling the detector and tracker preserves object identity, and SAM 3 agents pair with multimodal LLMs like Gemini to handle complex visual reasoning. The real-world impact includes 106 million smart polygons created on Roboflow, saving an estimated 130+ years of labeling time across fields from cancer research to underwater trash cleanup.

Best of 2024 in Vision [LS Live @ NeurIPS]

Dec 22, 2024 · 55:46

Isaac Robinson and Peter Robicheaux from Roboflow and Vik Korrapati from Moondream recap the biggest shifts in computer vision in 2024, naming Sora as the year's most significant paper despite being a blog post, and highlighting its replication efforts (OpenSora) that use MAGViT V2 and diffusion transformers. They detail how SAM 2 extends the SAM approach to video with a memory bank and real-time performance, and how DETR-based detectors (RT-DETR, LW-DETR, DeFine) now surpass YOLOs on COCO by 4.6 AP at the same latency. Peter explains the CLIP-blind phenomenon: MMVP shows leading VLMs (ChatGPT, Gemini, LLaVA) fail at fine-grained tasks like reading clock hands, but models like PaliGemma 2 and AimV2 improve by incorporating pixel-level understanding and autoregressive image reconstruction, with AimV2 approaching 60.2 mAP on COCO. Vik introduces Moondream's 0.5B parameter pruned VLM and demonstrates a grounded chain-of-thought approach that dramatically improves sample efficiency for analog gauge reading, reducing the need for millions of synthetic examples.

[Paper Club] Molmo + Pixmo + Whisper 3 Turbo - with Vibhu Sapra, Nathan Lambert, Amgadoz

Oct 13, 2024 · 1:12:59

This episode covers two papers: AI2's Molmo open-source vision-language models and OpenAI's Whisper Large V3 Turbo. For Molmo, the key claim is that high-quality, audio-annotated data (Pixmo) enables models as small as 1B to match GPT-4V on academic benchmarks, with the 72B variant outperforming GPT-4o, Gemini 1.5, and Claude 3.5 Sonnet. The team avoided distilling proprietary models by having annotators describe images in speech for 60-90 seconds, then transcribing and augmenting the captions. For Whisper Turbo, OpenAI pruned the decoder from 32 to 4 layers and continued pre-training on 10 million hours of multilingual transcription data, making it 1.78x smaller than Large V3 with minimal word-error-rate increase. The model is faster and supports real-time chunk-based decoding, unlike English-only DistilWhisper. Nathan Lambert and Amgadoz provide commentary on data strategies and benchmarking nuances.

Segment Anything 2: Memory + Vision = Object Permanence — with Nikhila Ravi and Joseph Nelson

Aug 7, 2024 · 1:00:44

Nikhila Ravi from FAIR and Joseph Nelson from Roboflow discuss Segment Anything 2 (SAM 2), which extends zero-shot object segmentation to video with memory and real-time interactive tracking. The model is one-third the size of SAM 1 (224M vs 630M parameters) and six times faster, using a novel memory attention mechanism with six-frame spatial memory and longer-term object pointers. Ravi explains the three-phase data engine that built the SA-V dataset of 51,000 videos, enabling SAM 2 to track arbitrary objects like a T-shirt or an octopus even when occluded. At Roboflow, users labeled 49 million images with SAM in its first year, saving an estimated 35 years of manual annotation time. The demo features swim lanes that show object visibility and allow refinement clicks to correct tracking mistakes, a key improvement over prior video segmentation models. SAM 2 also handles out-of-distribution domains like underwater footage, though screenshots remain challenging and may require fine-tuning.