Segment Anything 2: Memory + Vision = Object Permanence — with Nikhila Ravi and Joseph Nelson
Aug 7, 2024 · 1:00:44
Nikhila Ravi from FAIR and Joseph Nelson from Roboflow discuss Segment Anything 2 (SAM 2), which extends zero-shot object segmentation to video with memory and real-time interactive tracking. The model is one-third the size of SAM 1 (224M vs 630M parameters) and six times faster, using a novel memory attention mechanism with six-frame spatial memory and longer-term object pointers. Ravi explains the three-phase data engine that built the SA-V dataset of 51,000 videos, enabling SAM 2 to track arbitrary objects like a T-shirt or an octopus even when occluded. At Roboflow, users labeled 49 million images with SAM in its first year, saving an estimated 35 years of manual annotation time. The demo features swim lanes that show object visibility and allow refinement clicks to correct tracking mistakes, a key improvement over prior video segmentation models. SAM 2 also handles out-of-distribution domains like underwater footage, though screenshots remain challenging and may require fine-tuning.
