# SAM 3: The Eyes for AI  — Nikhila & Pengchuan (Meta Superintelligence), ft. Joseph Nelson (Roboflow)

Latent Space · 2025-12-18

<https://addtry.com/faf5679a-390a-4420-9286-d708fe71b204>

Meta's SAM 3, introduced by Nikhila Ravi and Pengchuan Zhang alongside Roboflow CEO Joseph Nelson, unifies interactive segmentation, open-vocabulary detection, and video tracking into a single model that runs in 30ms on images and scales to real-time video on multi-GPU setups. The model uses concept prompts like "yellow school bus" to detect and segment every instance, separating recognition from localization via a presence token. Its data engine automated exhaustive annotation from two minutes per image down to 25 seconds using AI verifiers fine-tuned on Llama, while the new SACO benchmark contains over 200,000 unique concepts versus previous 1.2k. For video, decoupling the detector and tracker preserves object identity, and SAM 3 agents pair with multimodal LLMs like Gemini to handle complex visual reasoning. The real-world impact includes 106 million smart polygons created on Roboflow, saving an estimated 130+ years of labeling time across fields from cancer research to underwater trash cleanup.

## Questions this episode answers

### What is SAM 3 and how does it improve upon previous SAM models?

SAM 3 is a unified model for concept-prompted segmentation, detection, and tracking in images and video, using atomic visual concepts like 'yellow school bus'. Nikhila Ravi explains it runs in 30ms on a single image with 100 objects on an H200, and unifies tasks that previously required separate models. It adds concept prompting, drastically reducing the need for manual clicks per instance.

[5:40](https://addtry.com/faf5679a-390a-4420-9286-d708fe71b204?t=340000)

### How did Meta's SAM 3 team reduce per-image annotation time from two minutes to twenty-five seconds?

Pengchuan Zhang describes that initial human annotation took over two minutes per data point. By using a model-in-the-loop to propose masks and then human correction, it dropped to 45 seconds. They then fine-tuned AI verifiers on Llama to check mask quality and exhaustivity, further reducing annotation to 25 seconds per data point.

[40:35](https://addtry.com/faf5679a-390a-4420-9286-d708fe71b204?t=2435000)

### What is the presence token in SAM 3 and why does it separate recognition from localization?

Nikhila Ravi and Pengchuan Zhang explain that the presence token is added to the model to handle the binary question of whether a concept exists in an image, separate from locating it. This simplifies the task: the detector no longer has to simultaneously determine presence and location, improving performance and reducing false positives.

[23:11](https://addtry.com/faf5679a-390a-4420-9286-d708fe71b204?t=1391000)

## Key moments

- **[0:00] Introductions**
  - [1:03] SAM 3 is not a 3D model; it was launched alongside separate SAM 3D Objects and SAM 3D Body models, clarifies Nikhila Ravi.
  - [4:25] Roboflow's RF-Detr achieves real-time object detection and segmentation on edge devices with a transformer model.
- **[5:40] SAM 3 Demo**
  - [9:13] SAM 3 achieves 30ms inference per image on an H200 and real-time video tracking with 10 objects on 2 H200s.
  - [10:50] SAM 3's concept prompting was the most requested feature after SAM 1, leading to a focused approach on atomic visual concepts.
- **[11:04] Concept Segmentation**
  - [12:01] SACO benchmark contains over 200,000 unique visual concepts, compared to 1,200 in previous benchmarks.
- **[13:33] Real-World Impact**
  - [14:25] Roboflow users created 106 million smart polygons with SAM, saving an estimated 130+ years of labeling time.
  - [17:18] Nikhila Ravi: 'The best eval is if it works in the real world.'
- **[19:16] Fine-Tuning & Tokens**
  - [19:37] Fine-tuning SAM 3 for new domains can be done with as few as 10 data points.
  - [20:16] A small number of negative examples (3-5) significantly helps fine-tuning SAM 3 for new concepts.
  - [22:43] SAM 3's training data includes over 70% negative concept annotations to train recognition vs. localization.
  - [23:16] SAM 3 uses a presence token to explicitly separate recognition from localization in its architecture.
- **[24:15] Architecture**
  - [24:39] SAM 3 unifies interactive segmentation, open-vocabulary detection, and tracking into a single model.
  - [25:23] SAM 3 decouples detection (identity-agnostic) and tracking (identity-preserving) to resolve task conflict.
- **[27:42] SAM Agents**
  - [27:42] SAM 3 can serve as a visual agent for multimodal LLMs, enabling complex visual reasoning beyond atomic concepts.
  - [29:55] Pengchuan Zhang: 'SAM 3 provides the eye for large language models to solve complex visual grounding tasks.'
- **[32:43] Model Showdown**
  - [35:32] Meta uses SAM 3 in production applications where fast inference speed is critical, without needing an LLM.
  - [36:42] SAM 3 performs well on OCR tasks like recognizing numbers, even though the data engine intentionally avoided OCR-heavy images.
- **[38:04] Data Engine**
  - [41:31] AI verifiers fine-tuned on Llama 3.2 achieved superhuman performance on mask verification in SAM 3's data engine.
  - [44:04] To surpass human performance in computer vision, the field will need to adopt RLHF-like learning paradigms, predicts Zhang.
  - [45:34] Video annotation remains a major challenge; SAM 3 did not achieve a fully automated data engine for video.
- **[45:43] Future Directions**
  - [47:55] SAM 3's MaskLiT detection score improves video stability by aggregating information over time, at the cost of some streaming latency.
  - [53:14] Pengchuan Zhang: the key debate is whether SAM 3 should be a tool call for LLMs or natively integrated into multimodal models.
  - [58:04] Pengchuan Zhang: future SAM work includes smaller efficient models, end-to-end video training, and deeper integration with reasoning systems.
  - [1:00:36] Robotics companies are already adopting SAM 3 for video perception in their systems.
  - [1:01:30] Joseph Nelson predicts the open text prompt in SAM 3 will reveal user demand for OCR, spatial reasoning, and action recognition.
  - [1:04:42] Hundreds of domain-specific fine-tunes of SAM 3 have already been created, such as Med-SAM 3 for medical imaging.
- **[1:12:01] Calls to Action**
  - [1:13:18] The SACO benchmark aims to guide the community to human and superhuman performance on concept segmentation, says Pengchuan Zhang.

## Speakers

- **Swyx** (host)
- **Joseph Nelson** (guest)
- **Nikhila Ravi** (guest)
- **Pengchuan Zhang** (guest)

## Topics

Computer Vision, Benchmarks

## Mentioned

Meta (company), Roboflow (company), Detectron (product), Detr (product), ELVIS (product), Florence (product), Gemini (product), Llama (product), Mask R-CNN (product), PyTorch 3D (product), RF-Detr (product), Rapid (product), SA1B (product), SACO (product), SAM (product), Segment Anything (product)

## Transcript

### Introductions

**Swyx** [0:04]
Okay, we're here in the remote studio with the grand return of the Roboflow and Latenz space and SAM combo. Uh, welcome to Joseph, my, my sort of, uh, vision co-host, I guess.

**Joseph Nelson** [0:16]
Thanks. Great to be here.

**Swyx** [0:16]
Welcome back. We also have welcome back Nikhila Ravi, who's, uh, the lead on S-SAM 2-- I guess just SAM in general, right? Um, and we have, uh, jo-joining us Pengchuan, uh, who's also a researcher on SAM.

**Joseph Nelson** [0:28]
Yeah.

**Nikhila Ravi** [0:28]
Great to be here.

**Joseph Nelson** [0:28]
Nice to meet you guys. Yeah.

**Swyx** [0:29]
So congrats on SAM 3's launch. I mean, like, the, the demo, each time you set it up, like, really a-amazingly. And I think, like, every time... my, my general impression or, or takeaway when I tell people about SAM is, like, just the, uh...

every time you have a new release, like, it's like once a year you show up, you drop a banger, and then you, you like, you know, you just, like, drop the mic and, and, and, and go, go for next year.

And you also add a dimension. So I was entirely, like, really not surprised when SAM 3 had the 3D thing. 'Cause I'm like, "Well, yeah, which, which is the next dimension to go?" It's like 3D. Like

**Nikhila Ravi** [1:03]
Yeah, actually, maybe just on that, w- I think that's actually a common misconception. We- ... launched actually three separate models this time. It was SAM 3-

**Swyx** [1:12]
You're correct

**Nikhila Ravi** [1:13]
... SAM 3D Objects, and SAM 3D Body.

**Swyx** [1:16]
Yes.

**Nikhila Ravi** [1:17]
Those were two completely separate models, and SAM 3 is just the image and video understanding model.

**Swyx** [1:23]
Which is on a, on a better backbone and is sped up. Yeah. Uh, sorry, I didn't, I didn't mean to, uh, sort of pre-preface all, all, all this. But maybe for-- just to remind our audience or maybe for, for, for people in-new to the SAM series of a podcast that we've done so far, maybe each of you can sort of go around and, uh, intro, like, your or your sort of entry into computer vision or sort of your relationship with SAM.

Uh, go ahead, Nikki.

**Nikhila Ravi** [1:45]
Okay, cool. Hi, everyone. I'm Nikhila. I'm a researcher at Meta. I've been at Meta for eight and a half years, so really been through m- evolution of the field, um, in that time. I really started working on a range of different problems in computer vision, worked briefly on 3D.

We built this library called PyTorch 3D. But really started on this Segment Anything as a project in around sort of late 2021. So it's actually, you know, been almost four years since I've been, like, working on this Segment Anything space.

And you know, we started with SAM 1 in 2023, SAM 2 last year in July 2024, and then now SAM 3. So it's been, you know, a culmination of a lot of work of a lot of people over the years.

So yeah, really, really excited to be at this point and, you know, get to share it with all of you. Um, I'll hand it over to Pengchuan. Yeah. Hello, everyone. So I'm Pengchuan. I'm a researcher at the SAM team.

I have been working in computer vision, this field, for nearly nine years, starting from 2017. Kind of I think it's a long time. I have been working in, uh, MSR for five years and then kind of moved to Meta Reality Lab to work on, uh, egocentric foundation models, uh, on AI glasses for a while.

And then in 2023, near, I moved to SAM team, and that time is exactly the start time of SAM 3. And really kind of I think that's the lifetime experience I have on the SAM 3 team. And, uh, it's glad that SAM 3 is out and I kind of achieve my original grand goal of computer vision to reach kind of human performance of detection, segmentation, tracking image and videos.

**Joseph Nelson** [3:33]
I'm Joseph, co-founder, CEO at Roboflow, where our mission is to make the world programmable. We think software should have the sense of sight, and models like SAM and others are critical to unlocking that capability. Now, millions of developers, half the Fortune 100, build with Roboflow's tools and infrastructure to create and deploy models to production.

We've been big believers of the Meta family of open source models all the way back to, like, Mask R-CNN and Detectron 2, all the way to present of SAM 1, SAM 2, and SAM 3. The work that the Meta team does to advance state-of-the-art and open source computer vision has been bedrock to enabling developers and enterprises globally to adopt AI.

So we've been big fans of the work, and I'm pleas-pleased to be joining you today, Swyx, to co-host the episode-

**Swyx** [4:20]
Yeah

**Joseph Nelson** [4:20]
... on SAM 3.

**Swyx** [4:22]
And you guys shipped your own Detr model too.

**Joseph Nelson** [4:25]
Yeah. We've been, we've been doing some work to advance, uh, machine learning research too. Like, one of the-- for example, Detr detection transformers, which was born out of NeurIPS last year. I think, Swyx, you actually challenged us. You were like, "Hey, what are some of the advancements that are happening in computer vision and in visual AI?"

And we had this observation that transformers had surpassed a lot of CNNs in vision tasks, but they hadn't been made to run real time, uh, as in, you know, s- over thirty frames per second, for example, on like a small T4...

or excuse me, small, like, edge device and hundreds of frames per second on, like, a T4. We did some research and published RF-Detr, Roboflow Detection Transformer, which is, you know, we kinda joke the greatest of all time model for doing real-time segmentation and object detection on the edge.

Now, in RF-Detr, it's, you know, you have to have a fixed class list and need to know some of the objects that you want to segment at a time. But for anyone that's running on, like, constrained compute and on an edge device and wants, like, an Apache 2 model to do that, RF-Detr and its family of models are key to fulfilling that mission and that goal.

**Swyx** [5:27]
Yeah, amazing. Okay, I think we are going to just go into a SAM 3 demo. Um, I think, Nikki, uh, you've prepped some stuff to show us. And this is great 'cause, uh, obviously there's nothing better than the creator of the tool showing off the tool.

**Nikhila Ravi** [5:40]
So just to start with, like, what is SAM 3? So SAM 3 is a model that can detect, segment, and track objects in images and videos using what we call concept prompts. So I'm gonna start with a simple image example, and then we'll show you a video example.

### SAM 3 Demo

**Nikhila Ravi** [5:58]
So a concept can be anything That is a short text phrase. So here, for example, we can use something like watering can, and you can see the model predicts a mask for the watering can. You can also then, um, refine the prompts using clicks or additional, um, visual exemplars, which I'll show you in a different image.

But essentially, idea of a concept prompt opens up the ability to find all instances of an object category without having to manually click on every single instance as you would've had to do if you were using SAM 2 or SAM 1.

Now, if the model misses any of the, um, any of the instances, you can add visual exemplars. So a visual exemplar is also a way to describe a concept to the model. So here I can add a positive box here and show the model that, you know, this is, also an instance of a flower that we want to detect.

So this is just in images, but what's really cool is you can now also do this in video. And so here I'll show you an example. Um, maybe this is a football match. You wanna track all the players in white, for example.

So red jersey or white jersey, you can provide a concept prompt, and the model will find the objects in the first frame and then track and de-detect the new instances that appear later on in the video. So it's not just detecting on the first frame, but both tracking those detections and finding new instances that appear throughout the video.

And one of the things we love to do in our demos is also show some real world applications of this. And so one idea here is that you can use this for video editing or adding effects. So here is a really simple mask effect, but you can imagine, for example, you might want to add a trail around the players.

You know, you can follow them around. Maybe you want to clone them, so you've got multiple players running around. You can also do background effects, um, for example, spotlighting players. And so these are just fun things you can do on top of the SAM 3 outputs, and this is just, like, a way to show people, like, what you can do.

There's also some templates which basically are pre-populated with a text prompt and an effect. Um, and these are just some fun ways you can use the outputs, but really, you know, the crux of it is in this, like, create from scratch where you can upload any image or video and try SAM 3 on that.

And we'll share the link so you can try it out as well.

**Joseph Nelson** [8:42]
One of the other demos that I have is, like, a busy, a busy scene for, like, doing labeling, which we can do later on, but just to give you a preview. It's like if you wanted to find tablecloth and maybe, like, back there there's, like, airplanes, so I'll do airplane, and you kind of get the ability to start to-

**Swyx** [9:00]
Do you find the confidence thresholds-

**Joseph Nelson** [9:03]
They do. I don't know why tablecloth wasn't as good. I've used that one in the past. Table maybe?

Yeah. Cool.

**Swyx** [9:11]
Wow, look at that.

**Joseph Nelson** [9:13]
Yeah.

**Swyx** [9:13]
I think, I think the other impressive thing that you guys emphasized in your launch is also, like, the latency. Um, I don't know where this, this particular inference is running, uh, but it says something like, "SAM 3 runs in thirty milliseconds on single image with one hundred detected objects on an H200."

Obviously, this isn't an H200, but-

**Joseph Nelson** [9:32]
Yeah

**Swyx** [9:33]
... it's also, like, just impressively fast.

**Joseph Nelson** [9:34]
Yeah.

**Swyx** [9:35]
And, and sometimes... Basically, you can be real-time if you want.

**Nikhila Ravi** [9:37]
Yeah, definitely on images. On images it's really fast, and then on video it kind of scales with the number of objects, but it's... For a set, limited number of objects, it's still, still real.

**Pengchuan Zhang** [9:49]
Yeah. Also add even for video, if you, if you can afford the kind of GPUs implement a very kinda good kind of parallel inference algorithm. So even you have a lot of object to track, you can still get kinda real time, uh, tracking performance as long as you scale up the GPUs there.

**Swyx** [10:07]
So, so I'm reading in the paper it's, uh, ten objects on two H200s, twenty-eight on four H200s, and sixty-four on eight 200s, something like that. I don't think there's an architecture... There, there-- I don't know if this, this is the parallelism demonstration-

**Nikhila Ravi** [10:20]
Yeah, yeah

**Swyx** [10:20]
... that we're talking about.

**Pengchuan Zhang** [10:21]
Yeah. In fact, when you kind of try the demo, the video to the kind of parallel implementation of the kind of video grounding. So it's already kinda in that fast mode.

**Nikhila Ravi** [10:32]
Yeah. It's-- You try it with a video with, like, lots of objects, and then you can notice that it's actually not very slow and you get the sense that we are doing the multi-GPU inference. Yeah. Everyone should try it out and see for them.

**Swyx** [10:50]
Uh, so okay. Amazing. So this, this thing about, uh, concept segmentation, I feel like you had a prototypical version of this and, and in your paper you really talk about, like, sort of generalizing it. I guess, like, what was the planning like in SAM 3?

Like, w- at the start of this, did you... You know, is, is, is what we have today exactly what you planned for or did you kinda-- did it emerge as you discovered capabilities?

### Concept Segmentation

**Nikhila Ravi** [11:12]
Maybe I could quickly talk about... Yeah. In SAM 1 we did have a proof of concept of text prompting, but that was just a very early exploration. It wasn't really built out and, you know, became the most highly requested feature since then.

Um, and so we, you know, in SAM 3 we really wanted to do it properly and actually do this in a way that it works in all different, um, scenarios. And so we had to really think about how to formulate the problem.

So we-- It could have been that we took open-ended text input and it works for all open-ended text, or we could have be more, be more focused, which is what we chose to do, and really focus on these atomic visual concepts like yellow school bus or a purple umbrella, and really focus on nailing the problem for these, like, atomic visual concepts.

But Pengchuan, maybe you wanna talk a little bit about kind of the benchmarks that existed previously and how we had to actually fully redefine the task and the benchmark that we wanted to solve. Yeah, and maybe just to, to add to Pengchuan's point, like if you look at the size of these benchmarks, the previous benchmark Pengchuan mentioned, ELVIS, that everyone uses, it has about 1.2K unique concepts, and the benchmark that we created, which we're calling Segment Anything with Concepts or SACO, COCO for short, SACO has more than 200,000 unique concepts.

If you think about the w-way natural language that people use, we don't just use 1,000 words, we use... We have a very large vocabulary, and we really wanted to build a benchmark that can capture that diversity and size.

**Swyx** [12:50]
Yeah, it's, it's, it's really impressive and also like very formulaic, I guess, or classic that every great model work starts with a lot of data work. I think basically it's, you know, very s- the scaled up version of the same process for SAM 2.

**Nikhila Ravi** [13:05]
Yeah, in some ways. I think the, in SAM 3, data engine really was like a very novel and, and critical component. I think, you know, to your point, competitive advantage in AI is, is not just about, um, the models, but really about the data and maybe even more so is actually the data engine to generate that data.

And we put a lot of effort in SAM 3 specifically to try and automate that process a lot.

**Joseph Nelson** [13:33]
One of the things that we're really impressed by is the diversity and depth as well as breadth of uses that we see with models like SAM in production. Basically, when you think about computer vision, you know, folks kind of like always classically think about like dogs and cats and, and simple sorts of things.

### Real-World Impact

**Joseph Nelson** [13:49]
And the reality is like computer vision is where AI kind of meets the real world. So any sort of thing that needs to be seen and understood, you need to have understanding of that thing. So a model like SAM expanding the concepts from like, you know, a few thousand closed-form concepts max in a single model to tens of thousands of concepts means that you're going to see such a huge acceleration of the number of fields and, um, applications of the model.

So this is SAM 3, right? So we've already seen and measured some of the impact of the SAM family of models, and we pulled some of the updated stats on how impactful SAM is being across the Roboflow community.

I think, I think Roboflow might maintain one of, if not the largest hosted instances of SAM. And we're seeing basically 106 million kind of smart poly created examples that are, that are SAM 1, 2 or 3 powered. And we estimate that that saved humanity collectively like 100, maybe 130 years, depending on exactly how you want to do the calculation of time, just curating data.

And each of those use cases, right, isn't dogs and cats on the internet. It's things like, I don't know, we see medical labs across the world that are accelerating cancer research by doing things like counting and identifying the automation of neutrophils after a given experiment.

Or we see folks that are using aerial imagery for things like helping a drone navigate through the world or maybe counting and seeing, you know, solar panels from, from above, or maybe even doing like insurance estimates. We see folks that are building underwater trash cleaning up robots.

So like you can imagine an autonomous underwater bot that's navigating through the Pacific Ocean and identifying and grabbing on and grabbing plastics and, and cleaning up the world's ecosystem. Relatedly, we've seen some work with aquariums across the US like MBARI, who are doing work for keeping track of species and identifying the impact of ensuring given steps that are taken are increasing the populations of given fish with like underwater fish cameras.

We see folks in industrial settings like doing work to produce electric vehicles or get products from point A to point B. At the time of recording this, it's like near Christmas time, and it's like high time for holidays for folks that are doing gift giving, and that ends up being really, really high time for making sure goods and services show up where they're supposed to be at the, at the given point in time.

One of the statistics that we track is the frequency with which folks cite, um, works like SAM or, or Roboflow or blogs that we publish. And there's now basically like a little over two research papers published every day citing some of the work- ...

across like the, the Roboflow community. And that's folks that are like publishing in, in Nature and, uh, ScienceDirect and a fairly nu- prestigious number of journals. And each of those-- You got to think about it. Each one of those publications is someone's like seminal work, often six, 12, 24 months of effort that's been accelerated from models like SAM.

So it's not exaggeration to say like models like SAM are speeding up the rate at which we, you know, solve global hunger or find cures to cancer or make sure critical medical products make their way to people all across the planet.

And at the infrastructure level, we're like thrilled and surprised constantly by the breadth and depth of adoption that we see from the community. I mean, in the first five days of SAM 3, there was like 8 million inferences of folks that were running across all diverse sets of fields, and that's actually only increased because we-- it was released, and then there was like Thanksgiving, and now it's back and folks are like hitting it pretty hard.

So it's been incredibly encouraging to see the both depth of adoption and how much, uh, the community takes and uses and relies on models like SAM in prod.

**Nikhila Ravi** [17:18]
Yeah, and I think from... maybe just to add to that from like Meta's side, like we don't usually get as much visibility into all of these real world use cases. They're, you know, being able to kind of hear that from Roboflow and having these models available on the platform is like so valuable for us, is also, you know, we get, we get to know how these models actually work in the real world, which is, you know, ultimately the best eval for a model.

So I think, you know, it's definitely awesome to hear about all these things that we're empowering.

**Joseph Nelson** [17:48]
Nikhila, you had this, you had this comment of like the best eval for a model is like, it's not necessarily benchmarked. What, what was it? It's like if it works on, on real world things, I think it's a really good soundbite.

**Nikhila Ravi** [17:58]
Probably something like the best eval is if it works in the real world.

**Joseph Nelson** [18:02]
Yeah, true. Yeah.

**Nikhila Ravi** [18:03]
And that's like the ultimate goal for all of our models, like SAM 1, SAM 2, SAM 3, we want to people to use it out of the box as much as possible. And I think, you know, with language in SAM 3 specifically, there, there does need to be, in some cases, some domain adaptation.

But we have sort of tried to make that easy. I know, Pengchuan, you want to talk a little bit about, about that, like the fine-tuning aspect?

**Swyx** [18:29]
I wanted to also endorse like the real world thing. Uh, I was just so s- happily surprised when I was visiting the, uh, CZI Imaging Institute for-- in preparation for our pod with, uh, Mark, that they were using SAM in imaging the human cell.

And they showed us like how... Like in reality, all these sort of masses are actually like really undifferentiated, and it's really hard for the human eye to track. This is actually a simpler one where you can actually s- there's not-- this is like pretty clean here.

In reality, a lot of it is just like, just gray mush, and you have to like segment individual lysosomes out of these. And they, they showed us how they were using SAM and fine-tuning SAM to do it. Uh, yeah, really, really, really complicated and also like very meaningful, right, uh, for, for, for, uh, basic hu- basic science research.

### Fine-Tuning & Tokens

**Swyx** [19:16]
And I was also maybe mention like, um, th-this in the paper, the, the data distribution, you can actually see, uh, what SACO does. So a lot of, lot of animals, lot of animals, and then very surprisingly few maps.

I'm like, maybe there should be more maps. Also, Hugging Face has been doing, doing a lot here, uh, on, uh, and, and other, other companies.

**Nikhila Ravi** [19:37]
Yeah, this is actually one thing-- some-something we get asked a lot is like, "What's the minimum amount of data I need to fine-tune?" And, you know, being able to do that with just sort of 10 data points is hopefully will unlock a lot more than we can do ourselves.

**Swyx** [19:50]
Yeah, I mean, the more the merrier, obviously. This is where ablations are really helpful. You probably didn't have any fine-tune ablations in here. I think this is all data and model training or-oriented. But, uh, yeah, I mean, like, uh, very, very, very ob-very clear.

And I just have a cheeky curious point. Is there a neg-- uh, a ratio of-- what is the ratio of the negative examples of positive example, right? So in Nikhila's example, when you were, when you were demoing just now, you only selected positive examples.

Obviously, there's gonna be a lot more negative examples of not class than positive example of class. So there should be some exchange ratio where like negative examples contribute smaller than a positive example, or is that not the case?

**Joseph Nelson** [20:30]
For positive and negative examples, I don't know that I have seen like a golden ratio that, that works well or not works well, but I can offer anecdotally that a single negative example goes a long way. A common place where fine-tuning is really helpful is like data that's out of distribution that might, might have been impossibly in distribution.

Like one of my favorite fine-tune examples is like counting Waymos. There's not that much data that have like Waymos labeled throughout the streets of San Francisco, but SAM does a really good job to identify Waymo as like a vehicle.

If you, um, prompt with Waymo, it doesn't find anything. You find vehicle, it find-- it labels a Waymo as a vehicle, which is valid, but a Waymo is a specific type of vehicle, right? Usually, from even just like a ten-second video clip, you can actually start to have SAM 3 learn what should have been seen versus as a Waymo versus what should have been seen as, as a vehicle.

And even on a single image example, we see that like SAM 3 starts to adapt because it takes the text and image prompt into ac-account when it makes a subsequent inference. From like three to five negative examples alongside positive examples, you start to see the model update its priors, if you will, for where it would predict things from what the user provided.

All this is written with caveats, right? 'Cause like when you talk about visual world, the negative example and the positive examples could have been a very different perspective or a very different type of object. Um, like maybe you're like labeling dog breeds and suddenly a new dog breed appears, or maybe you have a perspective where it's overhead and then suddenly you have a side-by-side view.

So usually, the best way is to like have these things meet the real world data and, and try. But I'll offer maybe the note that a small number of negative examples goes a really long way, like small like three to five, not like hundreds.

**Nikhila Ravi** [22:09]
Yeah. The other place where negatives play a big role is just is it in the image or not. And that was one of the things that we did was really separate the problem into a recognition problem and a localization problem.

So first, can you answer the question, is this object or is this concept in the image? And then if it's in the image, where is it in the image? And so to really, uh, to, to really build in that capability, we had to annotate a lot of negative, um, phrases and images.

So basically, a lot of phrases that don't exist in the image, in addition to the concepts that exist in the image with the corresponding mask pair. So we have... You know, if you look at our, uh, ta- one of the tables in the paper which shows the training dataset distribution, I think it's table twenty-four, we have about seventy-- more than seventy percent of the annotations are these like negative phrases that are not present in the image.

So we have to really train the model to not detect stuff that is not in the image.

**Swyx** [23:16]
Yeah, I think that the separation of localization and... So it's basically precision recall, right, but in the vision domain.

**Nikhila Ravi** [23:25]
We basically add this presence token to the model, which explicitly separates the task of recognition and, and localization. So-

**Swyx** [23:37]
Yeah, we know.

**Nikhila Ravi** [23:39]
Basically-

**Swyx** [23:39]
Yeah

**Nikhila Ravi** [23:40]
... it, it simplifies the task, and so the model doesn't have to try to do everything with just the, um, with just the proposals in the, in the detector.

Be able to have this global like sort of learned token just for the recognition part.

**Swyx** [24:01]
Yeah. Uh, i-in general, I find that you, you guys did a lot of extra net new work. You had a really nice chart in here about like the yellow boxes being like the, the new stuff. I forget where-

**Nikhila Ravi** [24:13]
Yeah, the architecture diagram.

### Architecture

**Swyx** [24:15]
Yeah. I'm like, "Holy crap." Last time it was, it was like, you know, there was like the memory stuff, uh, this is SAM 2, and here it is all this Um, obviously we, we, you know, it's hard to cover it all, but, you know, I, I wonder if there's any other interesting stories or, or tricks like the presence token that you might, you might wanna focus on.

**Nikhila Ravi** [24:39]
Yeah, I mean, this, this is nice, this diagram. I'm glad you brought it up because SAM 3 isn't just a version bump. It's, you know, an entirely new approach to do segmentation. It's like this, you know, new interface for segmentation, and it combines so many different tasks where previously you would've needed a task-specific model for each of these tasks.

You know, interactive segmentation, um, text prompting, um, open-vocabulary detection, tracking. Like, all of these tasks you would've needed a separate model. And so, you know, really had to do a lot of work to bring it together. I think one of the things we did was really decouple the detection component and the tracking component.

So you can see, you know, we still preserved the tracking components from SAM 2, but the, the detector is, is separate. And the reason we do this is if you think about what a detector has to do and what the tracker has to do, the detector needs to be identity agnostic.

So if you have a concept dog, it needs to be able to find all instances of that dog, and it needs to sort of have this representation of dog that is the same for all dogs. But when you're tracking those dogs through the video, each dog needs to have a separate representation such that we're able to preserve the identities.

And so there's this kind of task conflict that emerges between the detector and the tracker. And so we really had to, you know, we experimented a lot. We really tried to build kind of a unified approach to do things, but then what we found was having, um, the separate detector and tracker really worked.

But we share-- we use the perception encoder as a shared visual backbone, um, and this is sort of a text and image aligned encoder. You can see the, the green boxes there. They're from, it says from PE, that's perception encoder.

That was also from our group in FAIR at the time. This was released earlier this year in April. And so this really is bringing together components from, like, the entire FA- FAIR and Meta ecosystem. We have perception encoder.

We have a deep detector. We use SAM 2. Um, we also use Llama and our data engine, so we really like using all the components from, um-

**Swyx** [27:02]
Yeah, it's like any, any third film in a trilogy. Like, you always see, like, the, the previous recurring characters come back.

**Nikhila Ravi** [27:09]
Yeah, well, if it works, you gotta s- continue using it .

**Joseph Nelson** [27:12]
And to connect to something we just got discussed earlier, you mentioned that a video component, each object needs to be tracked independently. That's why the compute scales linearly with the number of classes, right? 'Cause each of those instance types needs to be maintained.

**Nikhila Ravi** [27:24]
Each of the d-- Well, it, it scales with the number of detected objects.

**Joseph Nelson** [27:27]
Yeah, so e- so for example, like each dog that appears in the video, each one of those needs to be tracked independently. There was something else that you started to allude to in the paper that I was hoping we would spend some time discussing, and it's interaction of SAM 3 and LLMs, Llama and others.

### SAM Agents

**Joseph Nelson** [27:42]
So using SAM 3 to almost be like a tool call for LLMs to give them better grounding and give them better visual understanding. And there's a, a paper in the table where you describe the increase in, in performance.

It's kind of alluding, I think, to maybe where things are going for using SAM 3 as a component part of multimodal architectures. Do you want to describe a bit about what the introduction of that work was meaning to showcase and, uh, how the interaction of SAM 3 and LLMs is envisioned to be important?

**Nikhila Ravi** [28:12]
Yeah, maybe I can just do a quick intro and, and I'll hand over to Pengchuan to, to do the deep dive. But essentially, as I mentioned, SAM 3, we constrain the text input to these atomic visual concepts like, you know, yellow school bus or yellow watering can.

But obviously people want to interact with the model of natural language, and we want to enable that as well. And so that really, uh, beco-- segues into being able to use SAM 3 as this, like, visual agent for an MLLM.

And so I'll hand over to Pengchuan. Maybe you can explain h- about the SAM 3 agent setup and then talk through some of the results that we got there.

**Pengchuan Zhang** [28:53]
Yeah, yeah. So as Nikhila mentioned, the big picture is that SAM 3 is focused on this kind of atomic kind of concept, but people definitely want to try kind of much more complex phrases like, okay, uh, um, kinda, kinda, "Would you locate the bigger kind of character for me?"

What kind of... For example, this kind of my example, what is the kind of the, the feature that distinguish male and female, uh, uh, in this picture? Then these are more kind of complex language. This is exactly kinda SAM 3 cannot do, but SAM 3 agents target to kinda solve.

In this case, you can see that it's need much more advanced, uh, language understanding and reasoning. The SAM 3 currently do not have this kind of, uh, capability because it's small language encoder. Uh, but we know that large language models definitely gonna have, kinda, was trained on a lot of this data and has this kind of word knowledge and the reasoning capability.

Kinda SAM 3, uh, SAM 3 agent is exactly using SAM 3 as the eye for the large language models to solve this kind of complex, uh, visual grounding tasks.

**Swyx** [30:08]
Is there any sort of, um, insights that you or surprises that you have o- other than, I guess, like SAM is-- SAM 3 is a very good tool? Is that the main conclusion?

**Pengchuan Zhang** [30:19]
Um-

**Joseph Nelson** [30:19]
If you go to t- go to table eight in the paper as you, as you describe this-

**Pengchuan Zhang** [30:23]
Yeah

**Joseph Nelson** [30:23]
... if you don't mind. Yeah.

**Pengchuan Zhang** [30:24]
Table eight. Okay. Yeah.

**Joseph Nelson** [30:26]
Yeah. Here we go.

**Pengchuan Zhang** [30:29]
Yeah, please. Uh, maybe kind of quickly reply to kind of Swig's, uh, kind of question. I would say that first, besides that SAM 3 is really a good tool, kind of provides the eye for large language model, the other thing we definitely have found is that SAM 3 is not perfect.

It's not, like, kind of as robust as kind of human eye. The large mo- language model also kind of helps to correct the, kind of some error. Kind of they have a synergy between each other instead of just, okay, large language model provides the brain and the SAM 3 provides the eye.

**Swyx** [31:03]
Interestingly, you use Llama 4. I, I saw there's a mix of Llama 3 and Llama 4 here, uh, but it, it looks like it does best with Gemini 2.5, which makes sense, uh, given this comparable set of MLMs.

I, I'm just like, I, I think, like, the baseline also is just that, like, well, what extra addition does this add on top of just the MLM? Like, I, I would maybe, like, want to do that ablation. Maybe, maybe you already, you've already done it somewhere.

**Pengchuan Zhang** [31:27]
What do you mean by additional thing?

**Swyx** [31:29]
So, so basically, like, without a tool call, there's some native capability inside the MLM itself-

**Pengchuan Zhang** [31:35]
Wow

**Swyx** [31:36]
... to, to draw.

**Pengchuan Zhang** [31:36]
In fact, is that... Yeah, yeah. That's a really kind of good question. In fact, our kind of reviewer even asked that question. So without kind of... You can imagine that without kind of large language models, kind of without VLM, kind of SAM 3 kind of only, for kind of reason seg, it only achieves about, kind of on the validation set, if I remember it correctly, it's only achieved kind of 30, kind of, kind of numbers there.

And also, kind of it's very intuitive. You can see that for reason seg, it has this kind of, uh, kind of short, long untested. It has kind of different subset, short, long. Short, then it's very close to SAM 3's training data, like it's kind of atomic phrases, short phrases.

Long is this kind of very kind of complex reasoning. You will see that kind of, for short, SAM 3 only is very close to kind of the kind of SAM 3 agent. But for long, the gap is so large, which indicates that, okay, that is exactly kind of the capabilities that large language model bring in.

**Swyx** [32:40]
Got it.

**Joseph Nelson** [32:41]
I can show an example here that might be insightful too.

### Model Showdown

**Swyx** [32:43]
Go for it.

**Joseph Nelson** [32:45]
So even comparing, even comparing like SAM 3 and Gemini, let's say that we just want to have them do like a, an object detection task here of finding... Here, we're going to prompt with a speedometer and RPMs, and we're going to ask for things like indicator light, number, and needle.

And if we run SAM 3 head-to-head with Gemini 3 and Florence 2, almost as a baseline of like where things have been, and we see each of the results, first things first, you'll note that the speed of inference of SAM 3, uh, is quite quick.

Um, this is just calling the Gemini 3 Pro API, so whatever is provided from hosted compute is sort of what you get on the, on the response time. And then the second thing you'll note is, in addition to speed, is some of the accuracy of, of results.

Ooh, might get a, we might have a timeout error. Let's see.

**Swyx** [33:37]
Do you have ELO scores?

**Joseph Nelson** [33:38]
Uh, what scores?

**Swyx** [33:39]
ELO scores, like-

**Joseph Nelson** [33:41]
ELO, yeah

**Swyx** [33:41]
... like the, yeah.

**Joseph Nelson** [33:42]
Uh, we do.

**Swyx** [33:43]
You had, you had, you had the arena. Okay. I was wondering what the ELO was, was, 'cause you said you were blind testing this.

**Joseph Nelson** [33:49]
Yeah. Um, that's actually interesting because we had blind tested SAM 3 before it was released not as SAM 3, just for people to try and compare. I think we called it like a potential SAG or SAG preview or something.

And we allowed users to vote, and they kind of unanimously voted for what they didn't know at the time was, was SAM 3. We actually got like emails of people being like, "Hey, like, where can I use that?"

And we just sort of ignored them until the model came out. But so here with, with the responses, you see that, um, the, uh, grounding capabilities of, of SAM and, uh, SAM 3 compared to even Gemini, uh, are out ahead currently.

So not only is it doing grounding, but if you look closely, you can actually see it's making segmentation masks too, whereas, um, uh, Gemini 3 struggles to do, it just does detection by comparison. Um, and then the other thing is just the, uh, richness of detections, like the recall is, is high as well as the precision.

And if we compare here, it does almost as well, right? But you see that it misses some of the numbers and has kind of these, some of these erroneous boxes that it's, that it's predicted, and then it also doesn't do segmentation, so it just does, uh, detection of, of the task.

So you can envision that the same way the SAM 3 paper introduces the idea of using SAM 3 in tandem with MLMs, I would expect that to be the case pretty soon, and maybe the Google team taking some notes to improve Gemini and other series of models based on what SAM 3 demonstrates here.

So in other words, not only is it faster, but it seems to be more comprehensive for concept segmentation.

**Nikhila Ravi** [35:32]
And I think the speed actually is a, is a huge factor for many use cases. I, I think like even Meta were using SAM 3 for various different products use cases, and fast inference speed is very critical to, to enable that.

And so I think that's something that I think in many cases you don't even need an MLM for. It's, it's kind of overkill to use an MLM for some applications.

**Joseph Nelson** [36:00]
The other interesting thing is the Florence 2 results. And, you know, Florence 2 is a little bit older of a model now, so maybe it's not fair to put up head-to-head with state-of-the-art. But it is useful as a way to just see how far we've come.

'Cause Florence 2, by comparison, labels the entire region as, uh, a single class without seeing individual detection of numbers and indicator lights and needle. And not only that, but it actually runs at about three times the speed as, as SAM 3.

So SAM 3, again, is faster doing a task that the other models are not doing in segmentation and more accurate both in recall and precision of the things that it's intended to find, which I think really showcases the Capabilities of, of the model

**Pengchuan Zhang** [36:42]
In fact, I even got kind of a little surprised about this because this domain, this more like kind of an OCR like, because recognition numbers is nearly OCR. We do not prioritize this domain of data collection. It works.

So we know that it roughly works, but I, I think I got surprised that kind of it works so well.

**Joseph Nelson** [37:04]
That's encouraging. Even a task that wasn't expressly prioritized, it still does a great job on.

**Pengchuan Zhang** [37:11]
Yeah. In fact, during our data engine, we intentionally do not sample OCR-heavy images.

**Joseph Nelson** [37:18]
Wow. On an easier one, glass mug, SAM 3, Gemini 3, Florence 2. SAM 3 loaded first and has-- really impressively, it sees even this glass mug in the corner, which I think is something SAM 3 does a great job of, is occlusion and partial objects.

Gemini 3 struggles a bit with this one, I think maybe 'cause the opacity of the objects by comparison. And then Florence 2 does a good job at finding one of the glass mugs. So again, another s- type of task that shows the power and veracity of the model.

**Nikhila Ravi** [37:53]
Yeah, I mean, exhaustivity, like finding every instance is something we heavily prioritized and is really built into the data engine design. Um, you know, maybe Pengchuan, you want to talk about how we design the data engine to really scale exhaustivity.

### Data Engine

**Nikhila Ravi** [38:12]
'Cause, you know, if a human was to annotate every single instance, it would take a really long time, and, and verify, but we put a lot of effort into trying to, to automate and speed up that process s-such that we could get to the data scale and diversity needed to get to a step change.

**Pengchuan Zhang** [38:30]
Yeah, yeah. I think definitely kind of I would say data engine is the kind of the cr- the critical component that we achieve sensory performance like now. So maybe we can go to the data en-engine picture. I think we have a kind of illustration there.

**Nikhila Ravi** [38:48]
Yeah, page five.

**Joseph Nelson** [38:49]
Oh, yeah, sorry.

**Pengchuan Zhang** [38:50]
Yeah, yeah, yeah. Here.

**Joseph Nelson** [38:51]
Yeah.

**Pengchuan Zhang** [38:51]
You can see that this is our annotation kind of pipeline. So we first source the kind of the images, and it kind of generates the noun phrases. So this is the input of this task, source images, and it generates kind of noun phrases from, for example, kind of Llama generate caption, and we pass the caption to get the noun phrases.

This is the kind of input distribution. Then we use kind of sensory model in the loop to generate kind of candidates kind of masks that we kind of, uh, that, that should be the candidate, but it's not perfect, especially in the beginning.

Then we go to kind of... You can see go to the next step is verification. So since we gave you this mask, then we need to first do mask verification to verify each mask, whether it's good or not.

And then kind of after we kind of filter all the bad masks, we kind of-- there are some good ma-masks left, and we verify whether this kind of, this good masks are exhaustive or not, like your mug example.

So for example, the kind of by the model do not predict that kind of, that partial mask, then the exhaustivity check will be kind of, uh, failing there. Kind of then, kind of if, kind of the exhaustivity is failed, then we go to next step.

You can see that we can go to the kind of the pipeline, go to this kind of so-called kind of human manual correction. Human kind of manually annotate kind of all this kind of missing masks to make this data point exhaustive.

So you can see that exhaustivity is a very big factor there, and we play it as the kind of center place in this data engine. And, uh, but you can see that if we ask human annotator to annotate every mask from scratch, it will take a lot of time.

I remember kind of each data point in the beginning will take about more than kind of two minutes to finish. But if you use model in the loop, then it's reduced to about kind of fi- forty-five seconds. Kind of you kind of use model to propose mask and then just a human to kind of, to annotate the missing mask, then it's forty-five minutes.

Another very key kind of innovation in this data engine is that we really found that this verification steps, like to verify a mask is good or not, or to verify now the good masks are exhaustive or not, can be done by AI, can be done by large multimodal model.

That is a breakthrough, and then kind of we kind of fine-tune our kind of, for example, Llama 3.2 with our kind of verification, human annotated verification data. We get kind of superhuman performance on these two verification tasks. And then we do not need a human on these two tasks.

This further bring our kind of per data point annotation time to about kind of twenty-five seconds. So you can see that from the original kind of all human to kind of about two minutes to finally kind of twenty-five minutes for one kind of data point.

How can... This is kind of our kind of

the journey of our data engine to make it super efficient.

**Joseph Nelson** [41:54]
Did you maintain statistics on how many images were specifically hard? For example, like we had N many objects that were very difficultly occluded, or we had some number of images where the comprehensive test was, was really hard. Or did you just bet that by having a large scale, you would encompass occlusion and exhaustive cases?

**Pengchuan Zhang** [42:16]
In fact, we know we kind of maintain this kind of information exhaustivity, which one is hard, which one is easy, because first, kind of in our data engine, when human annotates, then we exactly know which kind of, which data point are exhaustivity by the model, which part we need a human intervene.

In fact, we have that kind of, uh, metadata in our data sets. The second one is that the Better kind of the more beautiful part is we have this kind of exhaustively AI annotator. Then we can kind of give a new data point, we can automatically decide whether this is a difficult, uh, kind of data point or can, uh, easy data point by this AI annotator.

**Swyx** [42:57]
Yeah, I think the, the sort of bootstrapping annotation story was very strong last time around, and it's, it, you know, it's even stronger this time. What, what are you gonna do when you run out of humans? Like, you know, next year you're gonna have superhuman level of everything, right?

Like PCS and PVS. What, what then?

**Pengchuan Zhang** [43:20]
I'm not, not so optimistic about this. And at first, indeed, kind of our current plan for next project is kind of this kind of fully automated data engine without humans. That's our dream. I would say that's-- that will kind of...

I think that is the kind of perfect thing, but still we need some kind of useful information. There's no free lunch. There's kind of something kind of no kind of-- no model can do well, and we need human to inject that useful information.

I would say that what kind of practically can do is really min-minimal human intervention. Human only do the task that kind of the model cannot do, the most kind difficult task. So that's the kind of, kind of first one, kind of in term of data engine.

The second one is about human performance and this kind of PCS task. A... My feeling is that this kind of computer vision is going to enter this kind of... When we get to human performance, we will enter this RLHF domain of computer vision.

So you can see that language models kind of before, kind of in the birth, uh, age, kind of the language model are not hu-human performance. Kind of SFT, kind of really imitation learning really do their job, get to very good performance.

But if you only do SFT and the SFT data is annotated by human, then your performance is bounded by human. You cannot get kind of superhuman performance just by kind of, uh, this kind of data engine approach to use human to annotate data, then learn from that.

You need to go to this RLHF domain that human really just tell which two point, which one is better. This is exactly kind of the philosophy that to, kind of to tell which one is better is easier to really kind of to construct the data point from scratch.

So you can get kind of higher performance. Can, can get better performance from human draw from scratch. I would say that kind of-- I hope that after sensory, we can see kind of new research emerge from kind of in computer vision, which is, okay, how we go beyond human performance.

Sensory is close to that, but I would say that new learning paradigm is needed to go beyond human performance for sensory task and for computer vision.

**Nikhila Ravi** [45:34]
Yeah, no, just to add to that, this is, Pengchuan was only, was only talking about images. I think video is a whole nother challenging beast, and-

### Future Directions

**Swyx** [45:43]
Yeah

**Nikhila Ravi** [45:43]
... getting to that fully automated data engine is something that, you know, we tried to do in SAM 2. We actually didn't get to that fully automated approach. In SAM 1, we did. We, you know, fully-- our SA1B data set that we released was fully annotated automatically.

We didn't really get to that in SAM 2 for video and in SAM 3 for video. I think there's still, like, a lot of room to push on the sort of pseudo-labeling for video and really be able to get to that same step change as we had on images.

**Swyx** [46:13]
What are the biggest changes to see the same step cha-change in video that you've seen in images for automated data pipeline?

**Pengchuan Zhang** [46:20]
Yeah, yeah. I would say that's really kind of good video, large language kind of video mo- large multimodal model. So when we do sensory is kind of earlier this year or kind of mmm, last year, you can see that image, large multimodal model is very good, but video large multimodal model, I think really kind of it get-- become good or practical later this year, like kind of Quench 3, this kind of model gets kind of roughly kind of okay in that stage.

So we have a good kind of base model to fine-tune our, our data and to get human performance for this recognition or verification task. I would say that you can see that we need definitely kind of sensory-like effort in the perception side, but we also need kind of this kind of m-multimodal, large language model kind of efforts, kind of good foundation model on the kind of vision language side.

I think it's ready. It's ready now.

**Nikhila Ravi** [47:14]
Yeah. Also, video annotation is just so much more time intensive to, to get to that, to, you know, be able to ge- tr- annotate enough data to train a verifier. Like video mask annotation, we just found it was like very time intensive.

So maybe there are more efficient video annotation strategies. I think there's, you know, a lot of exploration that could be done there too.

**Swyx** [47:36]
Yeah, uh, you know, spending a bit of time on video, I wanted to also talk about, you know, obviously last time we were focused a lot on memory attention. I think this time there was this sort of, um, uh, MaskLiT thing that I, I, I, I, I wanted to just like get more ideas of or just to share the idea just generally.

What, what was it called?

**Nikhila Ravi** [47:56]
The MaskLiT detection.

**Swyx** [47:58]
MaskLiT detection score, exactly. Um, and, and how it's basically smoothing within a temporal window, which I think basically, you know, a lot of computer vision models don't have this, and they could just simply add it, and it would be a lot more stable when it comes to video, and I don't know why they don't do it.

**Pengchuan Zhang** [48:16]
Yeah, maybe I can comment on this. First, why they didn't do that, I think, um, one big reason is kind of the streaming requirement. You can see when you want to gather information across the entire mask net, then you need to wait for the mask net in the end and then, and kind of get the strategies.

So that will sacrifice some streaming kind of capability. So you can See that the streaming requirement is kind of somehow kind of limits we kind of traditional measure to do this. But I would say that this is definitely kind of beneficial.

The kind of the reason why is that I think even human do this. You can imagine that when something just appears kind of at the corner of the video, like a hand appears at the corner of the window, kind of the video, you just do not know whether this a man or woman.

So the human even make mistakes. Of-- also for Sam 3, it will make these mistakes. But kind of when you get more and more information, the person really enters the video fully, then you get to know, okay, whether this a man and woman.

So this kind of, kind of gather kind of more information to really kind of know whether kind of this concept is kind of the concepts you query, uh, is the idea here. Uh, so there is a trade-off between kind of the, kind of the latency and the accuracy here.

If you care more about accuracy, then you can use kind of this kind of overall kind of information can, of course, the Mask Net to get kind of more robust signal about, uh, the concept. But if you care about kind of latency, then you, you need to make a decision in the very beginning, and then you will sacrifice some accuracy.

**Nikhila Ravi** [49:58]
I think also in many video use cases, I think Joseph, you were showing on Roboflow, users care more about detecting the objects rather than having unique identities. So in m-in some cases, this-- maybe it's, this isn't required to preserve the identities throughout the video, and you just want to essentially do detection per frame, like for the Roboflow Rapid examples you were sharing.

**Joseph Nelson** [50:23]
Yeah. There's cases where being able to count and you know the objects are all gonna be the same, so you don't care as much about unique classes, you just want to know the full presence. Things like that matter.

But then there's other cases, like you mentioned, where, I don't know, like in sport, you care about individual players versus just knowing that there's eleven players on the pitch. One thing that might be useful actually to discuss with some of our time is, we talked a little bit about how SAM 3 and MLMs will play nicely together, but there's probably, like a greater discussion about how SAM 3 fits into the broader AI ecosystem and, like what bigger picture trends it might fit into.

Um, do you have some thoughts on what this represents about where things are headed?

**Nikhila Ravi** [51:06]
Yeah, maybe I could say one point, and then Pengchuan feel free to add. One, you know, as we mentioned before, SAM 3 isn't just a version bump. We are really having a unified model that can do many different tasks in the same unified architecture.

And so, you know, in the same way that LLMs can do many different tasks without needing a task-specific model, like w-with SAM 3, we're able to do, you know, image promptable concept segmentation, video promptable concept segmentation. We can do-- we don't need a specialist model for counting, we can do interactivity.

So really, it's like multi-capability visual models that are on par or better than the single task state-of-the-art models. So that's really one place in which SAM 3 fits into the AI ecosystem. In terms of MLMs, I don't know if, Pengchuan, you want to talk about the agent approach.

**Pengchuan Zhang** [52:05]
Yeah, yeah, definitely. I, I would-- you can see-- let me give, kind of-- I would say that Sam 3 kind of now kind of really get a big step change in vision, how it really helps the general AGI fit into general AGI or frontier model landscape is very, very kind of, um, exciting for me.

We always have this example, kind of give this kind of six finger, kind of hands up picture, ask how many fingers do we have in this picture, and then kind of nearly all the frontier models say five. And y-you can imagine that with Sam 3, then we can just kind of first detect how many fingers we have that very robustly, kind of six fingers.

And then the multimodal model should know that, okay, this is six finger hand instead of five, five finger. You can see that first of all, the, the errors make, made by frontier models can be solved if we use kind of Sam 3 as a tool.

But then how really kind of-- is Sam 3 as a tool is the end of the picture, or should really somehow Sam 3 even just be n-more naturally embedded into this frontier models, the frontier models have really the Sam 3 capability by themselves.

I, I would say that there's a lot of possibility there. Kind of my picture is that now we have a very good brain with this kind of frontier models, and we have a very good eye with Sam 3.

Now let's see kind of whether the eye really kind of is kind of working together kind of natively with the brain together, or eye is really kind of a, a different kind of, uh, organism, kind of organ, and then need to kind of somehow like a tool to kind of work with the brain.

I think th-this a very exciting kind of research area.

**Joseph Nelson** [53:44]
And so in your analogy, if you think about, like the visual cortex compared to like a human, human brain, like, you know, we have rods and cones in our eyes that do kind of very fast, we joke, like lizard brain level detection, simple stuff.

And then you have your brain that reasons about some of the visual information that your eyes see. In your example of SAM 3 as a tool call or SAM 3 as natively a part of the multimodal models, which future do you think is more likely?

**Pengchuan Zhang** [54:11]
I think at least I want to bet on kind of really they work natively together, the future for simple, I would say for simple or even intermediate difficult vision tasks. For example, count, kind of counting with less than twenty objects.

I think for this kind of simple task, this is like system one kind of visual reasoning, uh, with our brain. This should be kind of our brain kind of should do it by or kind of by themselves. But with very, very difficult tasks, you can see that if we are counting kind of maybe kind of thousands of objects kind of in the picture, so crowded, then we kind of even need to kind of draw something there.

I would say that at that time, maybe we need a kind of some extra model kind of for difficult tasks. This is-- You can see that this is a hybrid approach, but I'm more excited, kind of I think for most of the cases should be native.

The reason why there is, is You can think that I would say kind of perception or grounding kind of really kind of know where it is, how many it is, is like a fundamental capability of our brain. It's-- I'm just not happy that kind of the frontier model just cannot count how many fingers immediately, and instead of need to call a tool to do that.

I think this kind of should be system one thing, and this should be kind of natively in our brain. And also, if our brain cannot do this task, which means that it's definitely kind of missing some kind of very critical kind of visual capability by itself.

So that's kind of... I would say that it just feels that, uh, the intuition just feels that it's not correct to do not have this capability by itself.

**Joseph Nelson** [55:53]
So for very simple system one questions, things like how many fingers on a hand, that should be native. But for maybe more complex things that are maybe long-running tasks and long-running reasoning, then maybe there's a bit more of, like a tool call approach.

**Pengchuan Zhang** [56:07]
Yeah, yeah. Exactly. For example, you can see that we already kind of in our sensory agents or kind of in our AI annotator, we even demonstrate this approach. Uh, kind of for simple cases, the model can do it by itself that, okay, I can detect, for example, ten people here, and then the large mo- language model can even...

the AI annotator can even know that, okay, this ten people is not exhaustive. Okay, there are more people there. So if you want to do kind of well, then maybe kind of you need to do more step, for example, kind of to call an expert model.

So you can see that this is a very, very kind of native, kind of true, uh, kind of reasoning process for, for more advanced or complicated vision questions.

**Joseph Nelson** [56:47]
I have a related but maybe slightly different question. SAM 3 is an incredibly f- powerful piece of work, um, and it's open source as a part of, uh, now MSL. Open source critical to achieving AGI?

**Nikhila Ravi** [57:04]
Maybe I can comment on SAM specifically, but in SAM 3, we did leverage many of the open-source contributions people have made on top of SAM 2. There were new, there were new datasets, there were new benchmarks, there were new kind of inference time optimizations.

We adopt a lot of the things that the community built on top of the models, on top of the datasets. Um, and so the-- all those contributions helped make SAM 3. So for SAM series, we've really benefited a lot from, you know, being very generous with all the open source and then leveraging what the community builds on top of that.

But that, that's just from the SAM perspective.

**Swyx** [57:49]
I, I think it's clear what the community brings and offers, and I think, uh, you know, every time we do this, we always shout to the community to, like, you know, try it on their use cases and report, like, weird findings and, like, you know, if it doesn't do what you are trying to make it do, well, let's, let's talk about it, right?

And then maybe sort of implement it in, in the next version. Like you already said, uh, Pengchuan, you already hinted at, like, what might be coming for SAM 4, which is at least a little bit more of the document and OCR work.

Any other directions are interesting. I guess obviously a lot more video work as well.

**Pengchuan Zhang** [58:19]
Yeah.

**Swyx** [58:19]
What, what, what is the talk of the to- the town in, in, like, the CV community that, like, you know, "Oh, it'd be really great," or, like, super obvious, like, next year is gonna be the year of what?

**Pengchuan Zhang** [58:28]
Yeah. Maybe kind of I can first talk something and then Nikila can add. First, definitely kind of I think even it's not SAM 4, it's SAM 3 something, a SAM 3 point something, like small models. SAM 3 currently only have really kind of one model, kind of one size model, kind of more kind of efficient model that's kind of fit for kind of edge cases and also kind of m- a more efficient model for video.

I think currently kind of the video model is not efficient. You either-- you can achieve very good kind of throughput, but you need GPUs to do that. So first, kind of small and efficient models, that's one kind of big thing.

The second big thing is definitely kind of video.

**Swyx** [59:07]
Roboflow can do that for you.

**Pengchuan Zhang** [59:09]
Yeah, yeah. Mm. The second thing is video. I would say that we... Video is f- still far from, I would say, have a big gap from human performance. Right now, there is kind of still kind of a lot of research need to be done there, how to do end-to-end training with video.

We do not have end-- kind of we have this kind of decoupled approach, but we do not end-to-end train this model, and we expect definitely kind of it will be kind of, uh, a benefit from kind of end-to-end training.

And also as we dis-- kind of on video side, really kind of how to scale up the data engine. We need to kind of definitely kind of AI annotators for video. We tried that, but yeah, we kind of...

I, I think that's, that's something kind definitely, uh, worthwhile to do. The third one, we also discussed about that how sensory, how perception fit into AGI, this big landscape. Now we have the eye, how the eye work with the brain to do, to solve real kind of reasoning tasks.

Not only output segmentation, but really kind of answer how many cats are here or even answer the question, okay, I kind of, I have an example of kind of bio, um, biology labs. Kind of the robots need to decide whether kind of the, kind, liquid in the, kind, test tube is, uh, kind of at the kind of correct level or not.

You can see that this is kind of involves perception, but also involves reasoning. How to kind of solve this more kind of, kind of visual reasoning task with some is kind of a very big direction.

**Nikhila Ravi** [1:00:36]
On the robotics topic, it was exciting to hear from, like, several friends that work at, you know, different robotics companies on how they're, like, immediately starting to use SAM 3. And I th- I think especially for the video use case, I think robotics is probably one of the domains where I think improving video performance will have a lot of impact.

And so I think, yeah, that's definitely an area that we could improve on further. But yeah, to Pengchuan's point, I think there's still another step change to be achieved on video PCS.

**Swyx** [1:01:08]
Yeah, I-I-- just, just a quick comment on the robotics things. I know we, we're interviewing a bunch of robotics folks here, as, as well as, like, uh, Fei-Fei Li, who obviously started ImageNet. Uh, a lot of people are betting on explicit world models, and SAM is not, for better or worse.

And I, and I wonder when that crossover might happen. That's, that's an open question if you want guys want to take any world models, uh, discussions.

**Joseph Nelson** [1:01:30]
Uh, re where things are going based on, like, community questions, similar how Nikhila mentioned after SAM 1, the, like, almost obvious thing that people wanted was, like, open concept prompting because people are like, "Great, this model can see things, but I want to tell it what I want it to see."

And now with the introduction of SAM 3, you have this stepwise component, which feels like a key component of, you know, the ChatGPT era for vision is, is arriving as a result. What's gonna happen is now you've provided people with an open text box and media, and so you're gonna get all sorts of queries from people that maybe the model isn't primed to be able to perform particularly well on yet.

For example, earlier we were talking about document understanding and document reasoning being a place where there's known improvements to be made. And so you'll have people that will probably prompt to try to OCR things, or you'll have people that want to do work with spatial reasoning, like, "Give me the object to the left of this other object," or, "Give me a sense of where things are in relation to one another," which is critical for robotics like we were discussing because that's how you navigate throughout the real world.

You'll also have-- I think people will want action recognition and, and vision language action models, VLAs. Like, the same things that where you have these tasks where people are used to providing open text prompts and getting, "Here's the part of the scene where the player kicked the ball," or, "The tennis player made the serve."

Those are interesting for the purposes of how to understand and synthesize visual inputs. And so now that you've kind of given this open text box for media, there's gonna be a flood of the types of things users are gonna want to try to do, some of which SAM is already gonna be really well adapted to do, some of which not.

And I think that that's gonna be-- it's gonna reveal itself of the types of things that are, that are obvious. One of the things that we wanted to discuss was, like, where to, um, use SAM and, and discover how to build with SAM.

So in addition to the Meta team building a tremendous playground for being able to interact with images and video and kind of apply effects with, like, a video emphasis, I think one of the things that we're pretty excited about with SAM 3 is how much it positively impacts each part of building a system for visual understanding.

So, for example, the very first step of historically aggregating and collecting a dataset because you think that there's not a model that understands the slice of the world that you want to understand is where automating away lots of labeling can exist.

Basically, if you collected a bunch of data of something that's, is already in the SAM 3's knowledge, then you can prompt for SAM 3 to automatically label all that data for you. And so we've actually made a bet on SAM 3 being a core part of auto label at Roboflow, giving users a first pass of saying, "Hey, if you have a new image or you have a new video, start providing just a text prompt and allow SAM 3 to find and automatically label those regions of interest for you."

Downstream, I think there's areas for fine-tuning, like, you know, within a week of releasing SAM 3, Med-SAM 3 came out for adapting SAM into medical contexts, and I think that's a harbinger of what's to come. Like, there will be lots of domain-specific adaptations of SAM in places where maybe there's a specific ontology that someone wants to understand, or maybe there's a place where just the model doesn't have great awareness yet.

And I think we're already beginning to see that with hundreds of fine-tunes that users are creating for various domains. And then the last area is like, "Okay, I've got my model, now I wanna use it." And so one of the things that we're really proud of is to be ready on launch day to showcase the infrastructure we've built to burst and scale, like, infinitely large as folks have, uh, models that they want to deploy and make it readily available.

Having an endpoint that serves other fine-tune model or a model as is, or even a model that might be able to run on edge hardware as smaller models come out, or maybe distillation comes to rise, is, I think, also an awesome place of where we're seeing SAM 3 being impactful at each part of, like, the computer vision life cycle and pipeline.

**Swyx** [1:05:20]
That's awesome. Yeah, I think especially the impact on speeding up annotation, I think we've seen that consistently on Roboflow, and I'm really curious to see how, um, SAM 3-- the introduction of SAM 3 really helps, uh, speed up that process even further.

I mean, just from playing around with it, it's so much faster than having to manually annotate every single object. So yeah, you're really curious to see how that improves the experience.

**Joseph Nelson** [1:05:45]
One of the things that we were pretty excited about is we were kind of able to build an entirely new product in the world of, of SAM 3, and we, we called it, we called it Rapid. But basically, it's like there's probably a model that already understands the objects in the world that you want to see.

So here I'm, I'm screen sharing an example of, like, these are vehicles next to our office in San Francisco that go by. And you can see here's a Waymo, and here's, like, other vehicles. And, like, if I just have, like, this ten-second clip and let's say, you know, the first thing I want to do maybe is just, like, count cars, and I want to get a sense of, like, each of the vehicles.

What's really awesome is I can just, you know, of course, text prompt and say, "I want vehicle." And as I toggle through different frames in my video, SAM 3 already recognizes and understands those objects. Now, one thing that I think is really interesting, there was a conversation earlier about how much you want to rely on a model versus humans output of the model for what you care about.

So, for example, let's pretend in this scene, maybe the only cars that we care about are the ones that are, like, before the crosswalk and maybe not far in the distance. Then you'd get people that would say, "Hey, you know what?

I actually want the objects that are, like, most confident." And I would like, you know, move my slider down to, like, getting a fewer number of objects. Whereas maybe others might say, "Hey, I want, like, every single presence of a potential object in the scene," which even gets, like, reflections on the building of objects.

As computer vision approaches this world where we increasingly have, like- Models that can understand and improve themselves, and we rely on what human output and human preference from the models is, we're gonna get these funny scenarios where things aren't als- all, like, immediately deterministic of what a human cares about.

And I think that's where, like, tooling fills a big gap, but it also is gonna be a place where it'll be really interesting to see where users kind of start to use and apply the models and why you need sort of this last mile work to put the model in context in the domain that someone is trying to solve and tackle.

Um-

**Swyx** [1:07:44]
So let me, let me... Since you're here, right?

**Joseph Nelson** [1:07:46]
Yeah.

**Swyx** [1:07:46]
This is one of those things where I'm like, I'm not sure this concept, concept, the concept of, of labeling concepts can scale, only because I don't know if I ever-- if, if this slider between less and more is the way if, uh, ultimately I need to tell you whether or not to include, uh, reflections, right?

'Cause if reflections sometimes is great, that's exactly what I want. Most of the time it's not gonna be what I want. I don't know if some RLHF thing is gonna solve any of that because it-- you just need more prompting.

It's-- just, just saying vehicle is not gonna do it.

**Joseph Nelson** [1:08:23]
Yeah.

**Swyx** [1:08:23]
I don't know. Feel free to disagree.

**Pengchuan Zhang** [1:08:26]
Can you imagine such kinda s- such pipeline coming, for example, as kinda Swig said, that maybe kinda the reflection is exactly what I want, then you need some kinda iterations with the kinda the interface or the model or to kinda-- to get finally what you need.

So you need to specify the concept kind of more clearly through kind of multiple iterations. Can human not be involved in this iteration, but just kinda models i-i-in-- just kinda do it automatically? I think that's kinda something gonna definitely gonna-- it's...

I would say I'm quite interested in that. You can imagine this workflow, and then I want kinda reflections, and then I can kinda with the kinda the default kinda threshold, maybe kinda the kinda the model will get that output.

Then another kind of very strong kind of perception model on their kind of,

like, kinda Gemini 3 will then kinda ask-

**Swyx** [1:09:24]
Yeah

**Pengchuan Zhang** [1:09:24]
... we ask Gemini 3 whether there's kinda some reflections there. And if it says-

**Swyx** [1:09:28]
Yeah

**Pengchuan Zhang** [1:09:29]
... yes, then we can-- then you can see that we can automatically s- not kinda move the threshold kinda lower, and gonna ask again Gemini again and again to see whether kinda the reflection's now included or not. So somehow this process can conceptually should be done kinda completely with AI, kinda un- kinda-

**Swyx** [1:09:48]
I see. I see

**Pengchuan Zhang** [1:09:48]
... unless... Yeah, yeah, exactly.

**Swyx** [1:09:51]
So for now, the answer is Image.

**Pengchuan Zhang** [1:09:54]
Yeah.

**Swyx** [1:09:54]
Um, and we can, we can sort of tie it, tie it closer. I think, I think Joseph is showing us the, the sort of Waymo annotation. Yeah, it's, it's nice. Now you have a Waymo-

**Joseph Nelson** [1:10:01]
Yeah

**Swyx** [1:10:01]
... model.

**Joseph Nelson** [1:10:03]
Yeah, I was just doing an example where maybe we want to find an object that's not already represented in the training data.

I think, I think prompting-

**Swyx** [1:10:10]
That's a difficult problem

**Joseph Nelson** [1:10:12]
... could solve... Yeah. I think prompting could solve the problem of, like, reflections, 'cause maybe you could say, like, vehicles on the street. But to your point, like, you would have to, like, see that that's a failure case, right?

Like, if I was, like, just setting up a camera and saying, "Count cars," I wouldn't anticipate realizing that reflection could be a problem. And so I think this is why, like, in some ways human-in-the-loop, because identifying human intention, not necessarily human knowledge, is what's gonna be important for a lot of last-mile use.

**Swyx** [1:10:43]
Mm.

**Joseph Nelson** [1:10:43]
But yeah, I'm, I'm pretty excited about-

**Pengchuan Zhang** [1:10:45]
Yeah, maybe I want to echo kind of what Joseph said. This is also my experience, just different people have quite different kinda definition of even a visual concept. For example, for some kind of dataset, even hand, kinda some people would like to kinda just kinda annotate the palm kinda part as kinda the hand, and some people will kinda include the arm kinda also kinda as hand.

Then when we kinda first test SAM 3 on kinda some very kinda customized dataset, we found, okay, the performance is not that good. And when we kind of finally look into kinda the kind of performance, we found, okay, this is kinda just the user have a def- different definition or ex-explanation of the concepts.

But kind of both explanation are okay. Then in this case, you can see that really need a human-in-the-loop to do the kind of few-shot fine-tuning or to adapt to the user's definition of this concept.

**Joseph Nelson** [1:11:40]
That's exactly right. It's not always, like, deterministic of what someone really wants, which is why I think, like, even if you have a fully comprehensive omniscient model, putting the model into the context of what the user's trying to do is where a lot of tooling and infrastructure becomes really, really helpful.

Anyway, I found, I found our Waymos.

**Swyx** [1:12:01]
You continue to great-- build, like, excellent tooling for, for vision, and I think the world is very grateful for that. Let's get to call to action. Uh, you know, I think, you know, we, we, we've sort of given a, a good overview, and people obviously should read the paper and try out the playground, try out Roboflow if they're, if they're interested in di-diving deeper.

### Calls to Action

**Swyx** [1:12:20]
What is your call to action from, from each of you?

**Nikhila Ravi** [1:12:23]
I mean, try the demo, try the code. We've, we've got a lot of resources on GitHub repo. Um, and, you know, please-

**Swyx** [1:12:31]
It's a very well-managed launch, by the way. Like, kudos. I, I don't know. This probably takes a lot of effort just on the launch itself, even after the model's done.

**Nikhila Ravi** [1:12:39]
Yeah, and actually just on that, maybe one thing, just shout out to the whole team. I think this is-

**Swyx** [1:12:43]
Yeah

**Nikhila Ravi** [1:12:43]
... SAM 3 was our biggest and most ambitious project to date, and it really took a huge team of scientists, engineers, interns, software engineers, you know, across, across the company. So, you know, really huge shout out to the entire team that made not just the model successful, but also the demo and then all the, the launch and everything.

So it was a, a huge team effort. Definitely, like, would love to hear from people on what you're using the models for, where it's failing. You know, raise GitHub issues, message us on Twitter. We'd, you know, love to hear from you on, on where we should go next as well.

**Pengchuan Zhang** [1:13:18]
Yeah. And on top of that, definitely gonna try out also our benchmark, the SACO benchmark. I would say that it's likely that it's gonna-- the benchmark will last longer than our SAM 3 model. Maybe kinda next year there will be a stronger model, but the benchmark is kinda the one that I hope to guide the community to kinda get better and better models, kinda to get to a kinda we measure human performance on the benchmark.

I think maybe we are the first one to do that for this kind of very kinda segmentation and the kind of video or kinda grounding in the past. It's very difficult to measure human performance on this task. Hopefully, kinda this benchmark can guide the community to achieve human performance for this task and even kinda surpass human performance there.

**Joseph Nelson** [1:14:03]
We, uh, we set out to be one of the best places, if not the best place, to build with SAM 3 and the SAM family of models. So we're eager to see what people build with SAM and computer vision models to move the whole field forward.

We have infrastructure for everything from deploying SAM 3 zero-shot, to making your own fine-tunes, to auditlating-- automating labeling of data with SAM. And we continue to see the impact with each subsequent release expand the number of use cases and the amount of use and accelerate the time to value.

So excited to see what folks can build on, on Roboflow with SAM.

**Swyx** [1:14:32]
Thank you all so much. Like, this is, uh, you know, really great comprehensive coverage, great work, and just, uh, ob-obviously, like, always expands my mind as to what is possible with machine learning. Yeah, I mean, you know, as we're not, we're not at ASI yet or AGI yet, but we're-- every day we're getting closer.

**Nikhila Ravi** [1:14:48]
Awesome. Thank you so much.

**Joseph Nelson** [1:14:50]
Thank you.

**Pengchuan Zhang** [1:14:50]
Yeah. Thank you.

---

This library is powered by PodHood (https://podhood.com), the podcast website platform.