# Best of 2024 in Vision [LS Live @ NeurIPS] Latent Space · 2024-12-22 Isaac Robinson and Peter Robicheaux from Roboflow and Vik Korrapati from Moondream recap the biggest shifts in computer vision in 2024, naming Sora as the year's most significant paper despite being a blog post, and highlighting its replication efforts (OpenSora) that use MAGViT V2 and diffusion transformers. They detail how SAM 2 extends the SAM approach to video with a memory bank and real-time performance, and how DETR-based detectors (RT-DETR, LW-DETR, DeFine) now surpass YOLOs on COCO by 4.6 AP at the same latency. Peter explains the CLIP-blind phenomenon: MMVP shows leading VLMs (ChatGPT, Gemini, LLaVA) fail at fine-grained tasks like reading clock hands, but models like PaliGemma 2 and AimV2 improve by incorporating pixel-level understanding and autoregressive image reconstruction, with AimV2 approaching 60.2 mAP on COCO. Vik introduces Moondream's 0.5B parameter pruned VLM and demonstrates a grounded chain-of-thought approach that dramatically improves sample efficiency for analog gauge reading, reducing the need for millions of synthetic examples. ## Questions this episode answers ### How have DETR-based object detectors overtaken YOLOs in real-time performance in 2024? Isaac Robinson from Roboflow explains that DETR variants like RT-DETR, LW-DETR, and DeFine now outperform YOLOs. DeFine achieves +4.6 AP on COCO at equal latency. Key advances include efficient transformer encoders (RT-DETR), effective pre-training (LW-DETR shows YOLOs don't benefit as much), and adding specialized losses (DeFine). This marks a Pareto improvement in real-time object detection. [15:03](https://addtry.com/62ef4adc-ddea-4a76-b6f9-88d16c6eaeb7?t=903000) ### Why do large vision-language models like ChatGPT fail at reading analog clocks or other fine visual details? Peter Robicheaux highlights that VLMs initialized with CLIP lack fine-grained visual features because CLIP contrastive training doesn't require pixel-level detail. The MMVP paper demonstrates this with 'CLIP-blind pairs'—images similar in CLIP space but distinct to humans. On the MMVP benchmark, GPT-4V scores only slightly above random guessing, while humans achieve 94% accuracy. This reveals a fundamental limitation in how current VLMs perceive details. [20:51](https://addtry.com/62ef4adc-ddea-4a76-b6f9-88d16c6eaeb7?t=1251000) ### How does Moondream's 0.5B parameter model achieve accurate analog gauge reading using chain-of-thought? Vik Korrapati explains that Moondream's 0.5B model was pruned iteratively from a 2B general-purpose VLM, preserving accuracy. For gauge reading, they generated synthetic data with grounded chain-of-thought—having the model predict the needle tip, tick marks, and labels step by step before the final reading. This approach improved sample efficiency dramatically and allows error debugging by inspecting the reasoning steps. [47:13](https://addtry.com/62ef4adc-ddea-4a76-b6f9-88d16c6eaeb7?t=2833000) ## Key moments - **[0:00] Intro** - **[1:00] Sora & Video** - [1:00] "Sora is the biggest paper of 2024," says Isaac Robinson, even though it came out in February and has no paper. - [6:09] Sora and OpenSora show that simply scaling compute improves video generation quality, with hyperparameters mattering little. - **[7:45] SAM 2** - [8:05] SAM has saved Roboflow users 75 years of labeling time, says Isaac Robinson. - [10:30] SAM 2’s HERA hierarchical encoder delivers 6x faster inference than the original ViT while maintaining accuracy. - [13:23] In SAM 2, storing more past frames in memory does not significantly boost segmentation accuracy but slows inference. - **[14:40] DETRs Rise** - [15:56] DeFine DETR achieves 4.6 higher AP on COCO than YOLO at the same latency. - [17:23] LW-DETR shows that pre-training significantly boosts DETR performance but not YOLO after extended training. - [19:05] Combining RT-DETR, LW-DETR, and DeFine reaches nearly 60 AP on COCO at 10ms latency. - [20:18] "I make the claim that LLMs can't see," says Peter Robicheaux, demonstrating that ChatGPT and Claude fail at reading an analog watch. - **[20:21] LLM Blindness** - [24:39] LLaVA performs worse than random on the MMVP benchmark, confirming severe CLIP-induced blindness to fine details. - **[27:10] Florence 2** - [29:34] Florence 2 reaches 60 mAP on COCO, close to state-of-the-art, with faster convergence from pretrained weights. - **[31:00] PaliGemma 2** - [33:03] PaliGemma 1 saturates after seeing only 300 million examples, indicating limited model capacity despite large-scale pretraining. - [34:28] PaliGemma 2 achieves 47.3% on MMVP, far below human 94% but surpassing ChatGPT’s vision performance. - **[34:56] AIMv2** - [36:43] AIMv2 shows no saturation; performance continuously improves as model size and training data increase. - [39:06] Q: Why are foundation models worse at object detection than RT-DETR? Isaac Robinson points to domain-specific architectures and lack of pre-training benefit for YOLOs. - **[43:07] Moondream** - [45:24] Moondream released a 0.5B parameter vision-language model last week, distilled from a 2B model via pruning and retraining. - [46:58] Vision models fail at analog gauge reading because internet training data is biased toward product images with needles at zero. - **[47:13] Gauge Reading** - [48:35] Moondream needed millions of synthetic gauge images to teach a model to read analog gauges. - [49:20] Grounded chain-of-thought for gauge reading lets Moondream output reasoning steps and improves accuracy. - [53:13] Vik Korrapati hypothesizes VLMs lag in visual reasoning because the web has abundant reasoning text but almost no data teaching perception. - **[53:17] VLM Reasoning Gap** - **[54:58] Outro** ## Speakers - **Isaac Robinson** (guest) - **Peter Robicheaux** (guest) - **Vik Korrapati** (guest) - **Wanjun** (guest) ## Topics Computer Vision, Video Generation, Benchmarks ## Mentioned CLIP (product), COCO (product), DALL-E (product), DINOv2 (product), DeFine (product), Florence (product), LLaVA (product), LW Deter (product), MAGViT (product), MMVP (product), Moondream (product), OpenSora (product), PolyGemma (product), RT Deter (product), Roboflow (product), SAM (product), Sora (product), Stable Diffusion Video (product), VQGAN (product), YOLO (product) ## Transcript ### Intro **Isaac Robinson** [0:07] Hi. We're Isaac and Peter from Roboflow, and we're gonna talk about the best papers of twenty twenty-four in computer vision. Um, so for us, we define best as what made the biggest shifts in the space. Uh, and to, to determine that, we looked at what are some major trends that happened, uh, and what papers most contributed to those trends. So I'm gonna talk about a couple of trends. Peter's gonna talk about a trend, and then, uh, we're gonna hand it off to Moondream. So the trends that I'm interested in talking about are a major transition from models that run on per image basis to models that run using the same basic ideas on video, and then also how Deters are starting to take over the, uh, real-time, uh, object detection scene from the YOLOs which have been dominant for years. Uh, so as a highlight, um, we're gonna talk about Sora, which from my perspective is the biggest paper of twenty twenty-four, even though it came out in February. Um, is the what- Yeah. Yeah. So just... It's, uh, Sora is just a, uh, a post. ### Sora & Video **Isaac Robinson** [1:17] Um, so I'm going to fill it in with details from replication efforts, including OpenSora and related work such as, uh, stable diffusion video. Um, and then we're also gonna talk about SAM two, which applies the SAM strategy to video, and then how Deters are ... The, the improvements in twenty twenty-four to Deters that are making them a Pareto improvement to yellow-based models. Um, so to start this off, we're gonna talk about, uh, the state-of-the-art of video generation at the end of twenty twenty-three, MAGViT. Uh, MAGViT is a discrete token, video tokenizer akin to VQGAN, but applied to video sequences, and it actually outperforms, uh, state-of-the-art, uh, handcrafted video compression frameworks, uh, in terms of the, uh, bit rate versus human preference for quality, and video is generated by autoregressing on these discrete tokens. Um, generates some pretty nice stuff, but up to, like, five seconds in length and, you know, not super detailed. And then suddenly, a few, uh, months later, we have this, which when I saw it was totally mind-blowing to me. Um, ten ADP, a whole minute long. We've got light reflecting in puddles that's reflective. Uh, re-reminds me of, uh, those RTX demonstrations for next generation video games such as Cyberpunk, but with better graphics. You can see some issues in the background if you look closely, but they're kind of with a lot-- as with a lot of these models, the issues tend to be things that people aren't gonna pay attention to unless they're looking for, in the same way that, like, six fingers on a hand you're not gonna notice is a, uh, giveaway unless you're looking for it. Um, so yeah, as we said, Sora does not have a paper, so we're gonna be filling it in with, uh, context from the rest of the, uh, computer vision scene attempting to replicate these efforts. Um, so the first step, you have an LLM caption a huge amount of videos. Um, this, this is a trick that they, uh, introduced in DALL-E 3, where they train a, uh, image captioning model to just generate very high-quality captions for a huge corpus, and then train a diffusion model on that. Their, uh, Sora and the replication efforts also show a bunch of other steps that are necessary for good video generation, including, uh, filtering by aesthetic score and filtering by making sure the videos have enough motion so they're not just, like, kind of ... The generator's not learning to just generate static frames. Um, so then we encode our video into a series of space-time latents. Once again, Sora, very sparse in details. So, um, the replication related works, uh, OpenSora actually uses a MAGViT V2 itself to do this, but swapping out the, uh, disc-discret-discretization step with a classic VAE, uh, autoencoder framework. Um, they show that there's a lot of benefit from getting the temporal compression, which makes a lot of sense as, uh, the each sequential frames and videos have mostly redundant information. Um, so by compressing against, compressing in the temporal space, you allow the latent to hold a lot more semantic information while, uh, avoiding that duplicate. Um, so we've got our space-time latents possibly, but via there's some three-D VAE, presumably a MAGViT V2. Um, and then you throw it into a diffusion transformer. So, um, I, I think it's personally interesting to note that OpenSora is using a MAGViT V2, which originally used an autoregressive transformer decoder to model the latent space, but, uh, is now using a diffusion, uh, diffusion transformer. So it's still a transformer happening. Just the question is, like, is it parameterizing the stochastic, uh, uh, differential equation? Is it parameterizing a, uh, conditional distribution via autoregression? Um, it's als- um, it's also worth noting that most diffusion models today, the, the very high performance ones are switching away from the classic, like, DDPM, uh, denoising diffusion probability modeling framework to rectified flows. Um, rectified flows have a very interesting property that as they converge, they actually get closer to being able to be sampled with a single step, which means that, uh, in practice you can actually generate high-quality samples much faster. Um, major problem of DDPM and related models for the past four years is just that they require many, many steps to generate high-quality samples. So, and naturally, the third step is throwing lots of compute at the problem. So, uh, I didn't, I never figured out how to manage to get this video to loop, but we see very little compute, medium compute, lots of compute. This is so interesting because the, uh, the original Diffusion Transformer paper from Facebook actually showed that, in fact, the specific hyperparameters of the transformer didn't really matter that much. What mattered was that you d- were just increasing the amount of compute that the model had. So I love how in the, you know, once again, little blog post, they don't even talk about, like, the specific hyperparameters. They say, "We're using a diffusion transformer, and we're just throwing more compute at it, and this is what happens." Um, OpenSora shows similar results. The, uh, primary issue I think here is that no one else has 32X compute budget, so we end up with these, uh, uh... We, we end up in the middle of the domain in most of the, uh, uh, related work, which is still super, super cool. It's just a little disappointing considering the context. Um, so I think this is a beautiful extension of the, uh, framework that was introduced, uh, in '22 and '23 for these very high-quality per image generation, and then extending that to videos. ### SAM 2 **Isaac Robinson** [7:45] Uh, it's awesome. And it's GA as of Monday, except no one can seem to get access to it because they keep shutting down the login. Uh, the next, so next paper I wanted to talk about is SAM. S- so we at Roboflow allow users to label data and train models on that data. SAM, for us, has saved our users 75 years of labeling time. Um, we are the, to the best of my knowledge, the largest, uh, SAM API that exists. We also, SAM also allows us to have our users train just pure, uh, bounding box regression models and use those to generate high-quality masks, um, which has the great side effect of requiring less training data to have a meaningful convergence. So most people are data limited in the real world, so anything that requires less data to get to a useful thing is, uh, super useful. Um, most of our users actually run their object, uh, per frame object detectors on every frame in a video. Or maybe not most, but many, many. And so, uh, SAM, uh, falls into this category of take- SAM 2 falls into this category of taking something that really, really works and applying it to a video, which has the wonderful benefit of being plug-and-play with most of our, many of our users' use cases. Um, we're, we're still building out a sufficiently mature pipeline to take advantage of that, but it, it's, it's in the works. Um, so here we've got a, a great example. We can click on cells and then follow them. And you even notice the cell goes away and comes back, and we can still, uh, keep track of it, um, which is very challenging for, uh, existing obj- uh, object trackers. Um, high-level overview of how SAM 2 works. We, uh, uh, uh, there's a simple pipeline here where we can give, provide some type of prompt, and it fills out the rest of the likely masks for that object throughout the rest of the video. So here we're giving a bounding box in the first frame, a set of positive and negative points, or even just a simple mask. Um, I'm gonna assume people are somewhat familiar with SAM, so I'm gonna just give a high-level overview of how SAM works. You have a vi- uh, an image encoder that runs on every frame. Um, SAM 2 can be used on a single image, in which case the only difference between SAM 2 and SAM is that image encoder, which SAM used a standard VIT. Um, SAM 2 replaced that with a, uh, a HERA hierarchical encoder, which gets approximately the same results, but leads to a six times faster inference, which is excellent, especially considering how in a, a trend of '23 was replacing the VIT with more efficient backbones. Um, in the case where you're doing video segmentation, the, the difference is that you actually create a memory bank, and you cross-attend the features from the image encoder based on the memory bank. So the, uh, feature set that is created is essentially, uh... Well, I'll, I'll go more into it in a, in a couple slides. But we take the features from the past couple frames, plus a set of object pointers and the set of prompts, and use that to, uh, generate our new masks. Then we then fuse the new masks for this frame with the, um, image features and add that to the memory bank. It's... Well, I'll, I'll say more in a minute. The um, just like SAM, the SAM 2 actually uses a data engine to create its, uh, data set in that people are, they assembled a huge amount of reference data, used people to label some of it, and train the model. Uh, used the, the model to label more of it, and asked people to refine the predictions of the model. And then ultimately, the data set is just, uh, created from the final output of the model on the, uh, reference data. It's very interesting. Uh, this paradigm is so interesting to me because it, uh, it- Uh, unifies a model and a dataset in a way that is very unique. It seems unlikely that another model could come in and have such a tight relationship with the training set. Um, yeah. So brief overview of how the memory bank works. Th-th-their paper did not have a great visual, so I'm just-- I'm going to fill in a bit more. Um, so we take the last couple frames from our video, and, uh, we take the last couple frames from our video, uh, attend that along with the set of prompts, uh, that we provided. They could come from the future, they could come from anywhere in the video, as well as reference object pointers saying, "By the way, here's what we've found so far." Uh, attending to the last few frames has the interesting benefit of allowing it to model complex object motion, uh, without actually... Uh, you, by limiting the amount of frames that you attend to, you manage to keep the model running in real time. This is such an interesting topic, talk, topic for me, because one would assume that attending to all of the frames is super essential, or having some type of summarization of all the frames is super essential for a high performance. Um, but we see in their later ablation that that actually is not the case. So here, just to make sure that there is some benchmarking happening, we just compare it to some of the stuff that's came out, uh, prior, and indeed, the SAM2 strategy does improve on the state-of-the-art. Um, this ablation, deep in the dependencies, was super interesting to me. Uh, we see in section C the number of memories. Um, one would assume that increasing the count of memories would meaningfully increase performance, and we see that it has some impact, but not the, the type that you'd expect, and that it meaningfully decreases speed, which justifies, in my mind, just having this FIFO queue of memories. Um, although in the future, I'm super interested to see a more dedicated summarization of all of the last video, not just a, uh, stacking of the last frames. So that, another extension of beautiful per frame work into the, uh, video domain. The next trend I'm interested in talking about is, uh, this interesting w- a-at Roboflow, we're super interested in training real-time object detectors. Those are our bread and butter, and so we're doing a lot to keep track of what is actually happening in that space. ### DETRs Rise **Isaac Robinson** [15:03] Uh, we are finally starting to see something change. So for years, YOLOs have been the dominant f- way of doing real-time object detection, and we can see here that they've essentially stagnated. The, the performance between ten and 11 is not meaningfully different, at least, you know, in, in this type of high-level chart. And even from the last couple series, there's not a major change. Uh, so YOLOs have hit a plateau. Deters have not. So we can look here and see the YOLO series has this plateau, and then these R-RT Deter, LW Deter, and Define have meaningfully changed that plateau so that, in fact, the best Define models are plus four point six AP on Coco at the same latency. So three major steps to accomplish this. Uh, the first, RT Deter, which is technically a twenty twenty-three paper preprint, but published officially in twenty-four, so I'm going to include that. I hope that's okay. Um, Deters showed that, uh, RT Deter showed that we could actually match or outspeed YOLOs. Um, then LW Deter showed that pre-training is hugely effective on Deters and much less so on YOLOs. And then Define added the types of bells and whistles that we expect from, uh, these types, this, this, uh, arena. So the major improvements that RT Deter shows was, uh, taking the multi-scale features that Deters typically pass into their encoder and decoupling them into a much more efficient, uh, transformer encoder. Uh, the transformer is, of course, quadratic complexity, so decreasing the amount of stuff that you pass in at once is super helpful for increasing your runtime or, uh, increasing your throughput. So that change basically brought us up to YOLO speed, and then they do a hardcore analysis on, uh, benchmarking YOLOs, including the NMS step. Once you re- uh, once you include the NMS in the latency calculation, you see that, in fact, these Deters are outperforming, at least at this time, the, uh, the, the YOLOs that existed. Then LW Deter goes in and suggests that, in fact, the, uh, um, it, this frame, the huge boost here is from pre-training. So this, uh, is the Define line, and this is the Define line without pre-training. It's within range. It's, it's still an improvement over the, uh, YOLOs, but the really huge boost comes from the benefit of pre-training. Uh, in tw- when YOLOx came out in twenty twenty-one, they showed that they got much better results by run- having a much, much longer training time. But they found that when they did that, they actually did not benefit from pre-training. So you see in this graph from LW Deter, in fact, YOLOs do have a real benefit from pre-training, but it goes away as we increase the training time. Then the debtors converge much faster. LWDetr trains for only fifty epochs, RTDetr is sixty epochs. So one could assume that in fact the entire extra gain from pre-training is that you're not destroying your original weights by reques-relying on this long training cycle. Um, and then LWDetr also shows superior performance to our favorite dataset, Roboflow one hundred, uh, which means that they do better on the real world, not just on COCO. Uh, then DeFine throws all the bells and whistles at it. Uh, yellow models tend to have a lot of very specific, uh, complicated loss functions. This, uh, DeFine brings that into the Detr world and shows consistent improvement on a variety of Detr-based frameworks. So bring these all together, and we see that suddenly we have almost sixty AP on COCO while running in like ten milliseconds. Huge. Huge stuff. So we're spending a lot of time trying to build models that work better with less data, and Detrs are clearly becoming a promising step in that direction. The w-w-w-what we're interested in seeing from the Detrs in this, this trend next is CODetr and the, the, the models that are currently sitting on the top of the, uh, leaderboard for large scale inference scale really well as you switch out the backbone. We're, we're very interested in seeing and, and, and having people publish a paper, potentially us, on what happens if you take these real-time ones and then throw a swing G at it. Like, do we have a Pareto curve that extends from the real-time domain all the way up to the, uh, uh, super, super slow but high-performance domain? We also wanna see people benchmarking an RF one hundred more because that type of data is what's relevant for most users. Um, and we wanna see more pre-training because pre-training works now. It's super cool. **Peter Robicheaux** [20:18] Yeah. Sure. All right. So yeah. So in that theme, uh, one of the big things that we're focusing on is how do we get more out of our pre-trained models. Um, and one of the lenses to look at this is through sort of this, this new requirement for, like, fine-grained visual details and your representations that are extracted from your foundation model. ### LLM Blindness **Peter Robicheaux** [20:42] So as sort of a hook for this, um... Oh yeah, this is just a list of all the, the papers that I'm gonna mention. I just wanted to make sure I cited the actual paper so you can find it later. Um, yeah, so sort of the big hook here is that I make the claim that LLMs can't see. If you go to, if you go to Claude or, um, ChatGPT, and you ask it to, to see this, uh, uh, watch and tell me what time it is, it fails, right? And so you could say, like, maybe, maybe the, um, like this is like a very classic, uh, test of an LLM. But you could say, "Okay, maybe this, this image is like too zoomed out and it just, like, it'll do better if we increase the resolution, and it has easier time finding these fine gra- fine grain features like where the watch hands are pointing." No dice. And you can say, "Okay, well maybe, uh, the model just doesn't know how to tell time from knowing the position of the hands." But if you actually prompt it textually, it's very easy for it to tell the time. So this to me is proof that these LLMs literally cannot see the position of the watch hands, and it can't see those details. So the question is sort of why. And, uh, for you anthropic heads out there, Claude fails too. Um, so the, the... My first pick for best paper of twenty twenty-four Envision is this MMVP paper, which tries to investigate why do LLMs not have the ability to see fine grain details. And so for instance, it, it comes up with a lot of images like this where you ask it a, a question that seems very visually apparent to us, like which way is the school bus facing, and it gets it wrong, and then of course it makes up details to support its wrong claim. Um, and so the process by which it finds these images is sort of contained in its hypothesis for why it can't, uh, see these details. So it hypothesizes that models that have been initialized with, with Clip as their vision encoder, they don't have fine grained details and the, the features extracted using Clip because, um, Clip sort of doesn't need to find these fine grain details to do its job correctly, which is just to match, um, captions and images, right? Um, and sort of at a high level, even if ChatGPT wasn't initialized with Clip, um, and wasn't trained contrastively at la- the vision encoder wasn't trained contrastively at all, still, in order to do its job of capturing the image, uh, it could do a fa- a pretty good job without actually finding the exact position of all the objects and fe- visual features in the image, right? So this paper finds a set of difficult images for these types of models. Um, and the way it does it is it looks for embeddings that are similar in Clip space, but far in DINOv2 space. So DINOv2 is a foundation model that was trained, um, self-supervised purely on image data. Um, and it kind of uses like some complex student-teacher framework, but essentially, and like it patches out like certain, uh, areas of the image or like crops the certain areas of the image and tries to make sure that those have consistent representations, which is a way for it to learn very fine grained visual, uh, features. And so if you take things that are very close in Clip space and very far in DINOv2 space, you get a set of images that, um, basically are pairs of images that are hard for ChatGPT and other big language models to distinguish. So if you then ask it questions about this image, well, as you can see from this chart, it's gonna answer the same way, um, for both images, right? Because to, to- from the perspective of the vision encoder, they're the same image. And so if you ask a question like, "How many eyes does this animal have?" It answers the same for both. And like all these other models, including Lava, um, do the same thing, right? And so this is the, the benchmark that they create, which is like finding- CLIP, like, CLIP line pairs, which is pairs of images that are similar in CLIP space, and c- creating a data set of multiple choice questions based off of those. Um, and so how do these models do? Well, really bad. Um, LLaVA, I think... So, so ChatGPT and Gemini do a little bit better than random guessing, but like half of the performance of humans who find these problems to be very easy. Uh, LLaVA is interestingly extremely neg- negatively correlated with this data set. It does much, much, much, much worse than random guessing, which means that this process has done a very good job of identifying hard images for, for LLaVA specifically. Um, and that's because LLaVA is basically not trained for very long and is initialized from CLIP, and so you would expect it to do poorly on this data set. So one of the proposed solutions that this paper, uh, attempts is by basically saying, okay, well, if CLIP features aren't enough, what if we train the visual encoder of the language model also on DINO features? And so it, it, um, proposes two different ways of doing this. One addi- additively, um, which is basically interpolating between the two features, and then one is interleaving, which is just kind of like training one on the combination of both features. So there's this really interesting trend when you do the additive mixture of features. So zero is all, um, CLIP features, and one is all, uh, DINOv2 features. So it-- as you inc- so I think it's helpful to look at the rightmost chart first, which is as you increase the number of DINOv2 features, your model does worse and worse and worse on the actual language modeling task. And that's because DINOv2 features were trained completely from a self su- self-supervised manner and completely in image space. It knows nothing about text. These features aren't really compatible with these text models. And so you can train an adapter all you want, but it seems that it's in such an alien language that it's, like, a very hard optimization for this, these models to solve. Um, and so that kind of supports what's happening on the left, which is that, yeah, it gets better at answering these questions if, as you include more DINOv2, uh, features up to a point, but then you, when you oversaturate, it completely loses its ability to, like, answer language and, and do language tasks. Um, so, uh, you could s- also see with the interleaving, like they essentially double the number of tokens that are going into these models, um, and just train on both, and it still doesn't really solve the MMVP task. It gets LLaVA 1.5 above random guessing by a little bit, but still not close to, um, ChatGPT or any, like, human performance, obviously. Um, so clearly, this proposed solution of just using DINOv2 features directly isn't gonna work. And basically, what that means is that as a, um, as a vision foundation model, DINOv2 is gonna be insufficient for language tasks, right? ### Florence 2 **Peter Robicheaux** [27:10] So my next pick for best paper of 2024, um, would be Florence 2, which tries to solve this problem by incorporating not only this dimension of spatial hierarchy, which is to say pixel-level understanding, but also in making sure to include what they call semantic granularity, which ends up... The goal is basically to have features that are sufficient for finding objects in the image, so they're, they're, they have p- enough pixel information, but also can be talked about and can be reasoned about, um, and that's on the sema- semantic granularity axis. So here's an example of, um, basically three different paradigms of labeling that they do. Um, so they, they create a big data set. Um, one is text, which is just captioning, and you would expect a model that's trained only on captioning to have similar performance like ChatGPT and, like, not have s- uh, spatial hierarchy, not have features that are meaningful at the pixel level. And so they add another type, which is region text pairs, which is essentially either classifying a region or, um, s- um, doing object detection or doing s- uh, instance segmentation on that region or captioning that region. And then they have text phrase region annotations, which is essentially a triple. Um, and basically, not only do you have a region that you've described, you also find its, like, its place in a descriptive paragraph about the image, which is basically trying to introduce even more, like, semantic understanding of these regions. And so, like for instance, if you're saying a woman riding on the road, right? You have to know what a woman is and what the road is, and that she's on top of it, and that's, um, that's basically composing a bunch of objects in this visual space, but also thinking about it semantically, right? Um, and so the way that they do this is they take, um, basically, they just dump, uh, features from a vision encoder straight into a, uh, encoder-decoder transformer. Um, and then they train a bunch of different tasks like object detection and so on, uh, as a language task. And I think that's one of the big things that we saw in 2024, is these, these, um, vision language models operating in, on pixel space linguistically. So they introduce a bunch of new tokens to point to locations in, um, in pixel space. So how does it work? How does, how does it actually do? Um, we can see, uh, if you look at the graph on the right, which is using the, the DINO, the d- uh, the DINO framework, um, your, your pre-trained Florence 2 models transfer very, very well. They get 60 per... 60% map on COCO, which is like approaching state-of-the-art, and they train with- **Vik Korrapati** [29:56] Recording in progress. **Peter Robicheaux** [29:58] You're good. And they train with, uh, much more, um, uh, much more efficiently. So they, they converge a lot faster, which both of these things are pointing to the fact that they're actually leveraging their pre-trained weights effectively. Um, so where is it falling short? So these models, I forgot to mention, Florence is a 0.2 billion and a 0.7 bill- billion parameter count, so they're very, very small in terms of being a language model. Um, and I think that- ... this framework, you can see saturation. So what this graph is showing is that if you train a Florence 2 model purely on the image level and region level annotations, and not including the pixel level annotations, like segmentation, it actually performs better as an object detector. And what that means is that it's not able to actually learn all the visual tasks that it's trying to learn because it doesn't have enough capacity. So I'd like to see this paper explore larger model sizes. Which brings us to our next big paper of 2024, um, or two papers. ### PaliGemma 2 **Peter Robicheaux** [31:00] So PolyGemma came out earlier this year. PolyGemma 2 was released, I think, like, a week or two ago. Um, oh, I forgot to mention, you can actually train, like, label text datasets on Roboflow, and you can train a Florence 2 model, and you can actually train a, train a PolyGemma 2 model on Roboflow, which we got into the platform within, like, 14 hours of release, which I was really excited about. So anyway, so PolyGemma 2 and Po- so PolyGemma is essentially doing the same thing, but instead of doing an encoder-decoder, it just dumps everything into a decoder-only transformer model. Um, but it also introduced the concept of location tokens to point to, uh, objects in pixel space. PolyGemma 2-- So PolyGemma uses Gemma as the language encoder, and it uses Gemma 2B. PolyGemma 2 introduces using multiple different sizes of language encoders. Um, so the way that they sort of get around having to do encoder-decoder is they use the concept of prefix loss, which basically means that when it's generating tokens, um, autoregressively, it's-- all those, uh, tokens in the prefix, which is like the image that it's looking at, and like a description of the task that it's trying to do, they're attending to each other fully, full attention. Um, which means that, you know, it can sort of bind high level, uh... I-I-It's easier for the, the prefix to color, to color the output of the suffix, and also to just find, like, features, uh, easily. So this is sort of an example of, like, one of the tasks that was trained on, which is like you describe the task in English, um, and then you give it all these, like... You're asking for it to segment these two classes, um, of objects, and then it finds, like, their locations using these lo- tokens, and it finds their masks using, uh, some encoding of, uh, the masks into tokens. And yeah, so one of my critiques, I guess, of PolyGemma 1, at least, is that, um, you find that performance saturates as a pre-trained model after only 300 million examples seen. Um, so what this graph is representing is each blue dot is a performance on some downstream task. And you can see that after seeing 300 million examples, it sort of does equally well on all of the downstream tasks that they tried it on, which was a lot, um, as 1 billion examples. Which to me also su- kind of suggests a lack of capacity for this model. Um, PolyGemma 2, you can see the results on object detection. So these were transferred to, um, to COCO. Um, and you can see that this sort of also points to an increase in capacity being helpful to the model. You can see as both the, uh, resolution increases and the parameter count of the language model increases, performance increases. So resolution makes sense. Obviously, it helps to find small images or small objects in the image. But it also makes sense from another reason, which is that it kind of gives the model a thinking register, and it gives it more tokens to, like process when making its predictions. Um, but yeah, you could, you could say, "Oh, 43.6, that's not that great. Like, um, Florence 2 got 60." But this is not training a DINO or a DETR on top of this, uh, language or this image encoder. It's doing the raw language modeling task on COCO. Um, so it doesn't have any of the bells and whistles. It doesn't have any of the fancy losses. It doesn't even have, um, bipartite graph matching or anything like that. Okay. The big result, and one of the re- reasons that I was really excited about this paper, is that they blow everything else away on MMVP. I mean, 47.3, sure, that's nowhere near human accuracy, which again, is 94%. But for, you know, a two billion language-- two billion parameter language model to beat ChatGPT, that's quite the achievement. Um, and that sort of brings us to our final pick for paper of the year, which, um, is AimV2. ### AIMv2 **Peter Robicheaux** [34:56] So AimV2 sort of says, "Okay, maybe this language model, like, maybe coming up with all these specific annotations to find features and with high fidelity in pixel space isn't actually necessary. And, uh, we can come up with an even simpler and more beautiful idea for combining, um, you know, image tokens and pixel tokens in a way that's interfaceable for language tasks." Um, and this is nice because it can scale. You can come up with lots more data if you don't have to come up with all these annotations, right? So the way that it works is it does something very, very similar to PolyGemma, where you have a vision encoder that dumps image tokens into a decoder-only transformer. Um, but the interesting thing is that it also autoregressively tries to learn the mean squared error of the image tokens. So instead of having to come up with fancy object detection or semantic or segment-- or segmentation labels, you can just try to reconstruct the image and have it learn fine-grained features that way. Um, and it does this in kind of, I think, a beautiful way that's kind of compatible with the PolyGemma line of thinking, which is randomly sampling a prefix, prefix length and using only this number of image tokens as the prefix. Um, and so doing a similar thing with the, uh, causal... So the causal with prefix is the, the attention mask on the right. So it's doing full block attention with some randomly sampled number of image tokens to then reconstruct the rest of the image and the downstream caption for that image. Um, and so, uh, this is the data set that they train on. It's image... or internet-scale data, very high-quality data created by the, um, data filtering networks paper essentially, um, which is maybe the best CLIP data that exists. Um, and we can see that this is finally a model that doesn't saturate. Um, it's... Even at the highest parameter count, it's- it appears to be... Well, at the highest parameter count, it- it, uh, appears to be improving in performance with more and more samples seen. And so you can sort of think that, uh, you know, if we just keep bumping the parameter count and increasing the examples seen, which is the- the- the line of thinking for language models, then it'll keep getting better. Um, so how does it actually do at finding... Oh, it also improves with resolution, which you would expect for a model that, um... This is the ImageNet classification accuracy, but yeah. It- it does better if you increase the resolution, which means that it's actually leveraging and finding, um, fine grain visual features. Um, and so how does it actually do compared to CLIP on COCO? Well, you can see that if you slap a transformer, uh, detection head on it and train it on COCO, it gets to 60.2, which is also within spitting distance of SOTA, which means that it does a very good job of finding, um, visual features. But y- you could say, "Okay, well, wait a second. Uh, CLIP got to 59.1, so, like, how does this prove your claim at all?" Because doesn't that mean, like, CLIP, which h- is known to be CLIP-blind and do badly on MMVP, it- it's able to achieve a very high performance on fine gr- on this fine grain visual features task of object detection. Well, they train on, like, tons of data. They train on, like, Objects 365, COCO, Flickr, and everything else. And so I think that this benchmark doesn't do a great job of selling how good of a pre-trained model AMV2 is, and we would like to see, uh, performance on fewer data as examples and not trained to convergence on object detection. So seeing it in the real world on, like, a data set like Roboflow 100 I think would be quite interesting. And our, our, I guess, our final, final pick for paper of 2024 would be Moondream, so introducing Vik to talk about that. **Host** [38:48] Uh, but overall, that was exactly what I was looking for. Like, best of 2024. Amazing job. Um, uh, yeah, you can... Does anyone have questions while Vik gets set up, like vision stuff? Yeah. Vik, go ahead. **Wanjun** [39:06] Hi. Well, while we're getting set up, hi over here. Thanks for the really awesome talk. One of the things that's been weird and surprising is, um, that the foundation model companies, uh, even these MLMs, they're just, like, worse than RT-Detr at detection still. Like, if you wanted to pay a bunch of money, uh, to auto label your detection data set, if you gave it to OpenAI or Claude, that would be, like, a big waste. Um, so I'm curious, just, like, even PaliGemma 2, like, uh, is worse. So, so I'm curious to hear your thoughts on, like, how come nobody's cracked the code on, like, a generalist that really, uh, you know, beats a, a specialist model in computer vision like they have in, uh, in LM land? **Isaac Robinson** [40:00] I can... Can you hear me? **Host** [40:01] Yeah. You gotta press the speak button. **Isaac Robinson** [40:04] Okay. **Host** [40:04] Oh, yeah. You can. **Isaac Robinson** [40:07] Um, it's very, very interesting question. I think, um, it depends on the specific domain. Uh, for image classification, it's basically there in that AMV2 showed a simple, uh, attentional probe on the pre-trained features, gets, like, 90%, which is as well as anyone does. Um, the- the- the- the bigger question, like, why isn't it transferring to, uh, uh, object detection, especially, like, real-time object detection, um, I think in my mind there are two answers. One is object detection is really, really, really, uh, the architectures are super domain specific. You know, we see these, all these super, super complicated things, and it's not super easy to- to- to build something that just transfers naturally like that, whereas image classification, you know, CLIP pre-training transfers super, super easily. Um, and the other thing is, until recently, the real-time object detectors didn't even really benefit from pre-training. Like, you see the YOLOs that are, like, essentially saturated, showing very little difference with, uh, pre-training improvements, uh, w- with using pre-trained model at all. It's not surprising necessarily that people aren't looking at the effects of better and better pre-training on real-time detection. Maybe that'll change in the next year. Does that answer your question? **Peter Robicheaux** [41:29] Cool. Uh, ch- can you guys hear me? Uh, yeah, one thing I wanna add is just, like... or just to summarize basically is that, like, until 2024, you know, we haven't really seen a combination of transformer-based, uh, object detectors and, uh, fancy losses, and PaliGemma suffers from the same problem, which is basically to say that, um, these ResNet or, like, the convolutional models, they have all these, like, e- extreme optimizations for- for doing object detection. But essentially, I think it's kind of been shown now that convolution models, like, just don't benefit from pre-training and just don't, like, have the level of intelligence of transformer models. **Host** [42:12] Awesome. Wanjun. **Vik Korrapati** [42:15] Hi, can you hear me? **Host** [42:17] Cool. **Isaac Robinson** [42:17] I hear you, see you. Are you sharing your screen? **Vik Korrapati** [42:21] I might have forgotten to do that. Let me do that. **Isaac Robinson** [42:23] Sorry, should've done that. **Vik Korrapati** [42:24] One moment. Um, let me... Uh, share screen. Uh-oh, classic. Um, you might have to quit Zoom and restart. **Vik Korrapati** [42:41] What? Um, it's fine. I- yeah, it's, it's like we, we have, we have a capture of your screen. I'll just make sure this is visible. So let's skip this one. **Vik Korrapati** [42:52] Okay. Easy enough. How do we make it that way for you? **Vik Korrapati** [43:01] Quit Zoom. No. Yeah, yeah. There you go. **Vik Korrapati** [43:05] There we go. Yeah, perfect. **Vik Korrapati** [43:07] All right. Hi, everyone. My name is Vik. Um, I've been working on Moondream for almost a year now, like Shawn mentioned. I just went and looked, and it turns out the first version I released December twenty-nine, twenty twenty-three. ### Moondream **Vik Korrapati** [43:19] Um, it's been fascinating journey. So Moondream, um, started off as a tiny vision language model. Since then, we've expanded scope a little bit to also try and build some tooling, client libraries, et cetera, to help people really deploy it. Um, unlike traditional large models that are focused at assistant-type use cases, we're laser-focused on building, um, capabilities that developers can... Sorry, it, it's, uh... Yeah, we're laser-focused on building capabilities that developers can use to build vision applications, uh, that can run anywhere. So in a lot of cases for vision, more so than for text, you really care about being able to run on the edge, run in real-time, et cetera. So, um, it's very important. We have, uh, we have different output modalities that we support. There's query, where you can ask general English questions about an image and get back human-like answers. There's captioning, which a lot of our users use for generating synthetic data sets to then train diffusion models and whatnot. Um, we've done a lot of work to minimize hallucinations there, so that's, um, used a lot. We have open vocabulary object detection built in, similar to a couple more recent models like PaliGem, et cetera, where rather than having to train a dedicated model, you can just say, "Show me shoc- soccer balls in this image," or, "Show me if there are any deer in this image." It'll detect it. Uh, more recently, earlier this month, we released pointing capability, where if all you're interested in is the center of an object, um, you can just ask it to point out where that is. This is very useful when you're doing UI automation type stuff. Um, let's see. La-- We, we have two models out right now. There's a general purpose 2B param model which, um, runs fair. Like it's, it's, uh, it's fine if you're running on server. It's, uh, good for our local Llama desktop friends, and it can run on flagship, flagship mobile phones, but it never really fulfilled the promise of being able to run anywhere. Uh, last week we released a new 0.5B param model, uh, which should be seen more as a distillation target as opposed to a general purpose model. Uh, it's very good if you're running on like older mobile phones or edge devices. Uses less memory even with our not yet fully optimized inference client. Um, so the way we built our 0.5B model was to start with the two billion parameter model, um, and prune it while doing continual training to retain performance. We-- Our objective during the pruning was to preserve accuracy across a broad set of benchmarks. So the way we went about it was to estimate the importance of different components of the model, like attention heads, channels, um, MLP rows and whatnot, um, using basically a technique based on the gradient. I'm not sure how much people wanna know details. We'll be writing a paper about this, but, uh, feel free to grab me if you have more questions. Uh, then we iteratively prune a small chunk that will minimize loss in performance, uh, retrain the model to recover performance and bring it back. Um, the 0.5B we released is more of a proof of concept that this is possible. I think the thing that's really exciting about this is it makes it possible for, um, for developers to build using the 2B param model and just explore, build their application, and then once they're ready to deploy, uh, figure out what exactly they need out of the model and prune those capabilities into a smaller form factor that makes sense for their deployment target. Um, so yeah, very excited about that. Let me talk to you folks a little bit about, uh, another problem I've been working on recently, which is similar to the clocks example we've been talking about. We had a customer reach out who was, uh, talking about like... who, who had a bunch of gauges out in the field. This is very common in manufacturing and oil and gas, where you have a bunch of analog devices that you need to monitor. It's expensive to s- have humans look at that and monitor stuff and make sure that, um, the system gets shut down when the temperature goes over eighty or something. ### Gauge Reading **Vik Korrapati** [47:30] So I was like, "Yeah, this seems easy enough. Happy to, happy to help you distill that. Uh, let's, let's get it going." Turns out our model couldn't do it at all. Uh, I went and looked at other open source models to see if I could just generate a bunch of data and learn from that. That did not work either. So I was like, "Let's look at what the folks with hundreds of billions of dollars in market cap have to offer." And yeah, that doesn't work either. Um, my hypothesis is that like the, the way these models are trained are, are using a large amount of image text data scraped from the internet, and that can be biased. In the case of gauges, most gauge images aren't gauges in the wild. They're product detail images like these, where it's always set to zero. It's paired with an alt text that says something like GIVTO pressure sensor PSI zero to thirty or something. And so the models are fairly good at picking up those details. It'll tell you that it's a pressure gauge, it'll tell you what the brand is, but it doesn't really learn to pay attention to the needle over there. Um- And so yeah, that, that's a gap we need to address. So naturally, m-my mind goes to like, let's use synthetic data to solve this problem. Um, that works, but it's problematic because it turned out we needed millions of synthetic gauge images to get to reasonable performance. And thinking about it, reading a gauge is like not a one... Like, it's not a zero shot process in our minds, right? Like, if you had to tell me the reading in Celsius for this real world gauge, uh, there's two dials on there. So first you have to figure out which one you have to be paying attention to, like the inner one or the outer one. Um, you look at the tip of the needle, you look at what labels it's between, and you count how many and do some math to figure out what that probably is. So what happens if we just add that as chain of thought, um, to give the model better understanding of the different sub... to allow the model to better learn the subtasks it needs to perform to accomplish this goal. Um, so you can see in this example, this was actually generated by the latest version of our model. Uh, it's like, okay, Celsius is the inner scale. It's between 50 and 60. There's 10 ticks. It's, it's at the second tick. It's a little debatable here, like there's a weird shadow situation going on. The dial is off, so I, I don't know what the ground truth is, but it works okay. Um, there's points on there that... The points over there are actually grounded. I don't know if this is easy to see, but when I click on those, there's a little red dot that moves around on the image. The model actually has to predict where, uh, those points are. I was originally trying to do this with bounding boxes, but then Momo came out with pointing capabilities, and I was like, "Pointing is a much better paradigm to, uh, to represent this." S- we see pretty good results. This one's actually for clock reading. I couldn't find our chart for gauge reading at the last minute, so, um, the light blue chart is with, uh, our grounded chain of thought. Um, this measures... We, we have a... We built a clock reading benchmark about 500 images. This measures accuracy on that. Um, you can see it's a lot more sample efficient, uh, when you're using the chain of thought to, to help the model. Um, yeah. Another big benefit from this approach is like you can kind of understand how the model is doing it and how it's feeling. So in this example, the actual correct reading is 54 Celsius, the model output 56. Not too bad, um, but you can actually go and see where it messed up. Like it got a lot of these right except, um, instead of saying it was on the seventh tick, it actually predicted that was... it was the eighth, eighth tick, and that's why it went with 56. So n-now that you know that this is failing in this way, you can adjust how you're doing the chain of thought to maybe say like, "Actually count out each tick from 40 instead of just trying to say it's the eighth tick." Or you might say like, "Okay, I see that there's that middle thing. I'll count from there instead of all the way from 40." Um, so helps a ton. The other thing I'm excited about is, uh, few short prompting or test time training with this. Like if a customer has a specific gauge that, uh, like we're seeing minor errors on, they can give us a couple of examples where like if it's misdetecting the needle, they can go in and correct that in the chain of thought, and hopefully that works the next time. Um, now exciting approach, we've only applied it to clocks and gauges. The real question is, is it gonna generalize? Um, probably like there's some signs from text models that when you train on a broad number of tasks, it does generalize, and, um, I'm seeing some signs with our model as well. Um, so in addition to the image-based chain of thought stuff, I also added some spelling-based chain of thought, uh, to help it understand, uh, better understand OCR, I guess. Um, I don't understand why ev-everyone doesn't do this, by the way. Like it's trivial benchmark question that's very, very easy to nail. Um, but I also wanted to support it for stuff like license plate partial matching, like, "Hey, does any license plate in this image start with W-H-A?" Or whatever. Um, so yeah, that sort of worked. Um, all right, that, that ends my story about the gauges. If we think about what's going on over here, um, it's interesting that like LLMs are showing enormous progress in reasoning, especially with the latest set of models that we've seen, but we're not really seeing... ### VLM Reasoning Gap **Vik Korrapati** [53:17] I, I have a feeling that VLMs are lagging behind, as we can see with these tasks that should be very simple for a human to do, that are very easy to find, um, VLMs failing at. Uh, my hypothesis on why this is the case is because on the internet, there's a ton of data that talks about how to reason. There's books about how to solve problems. There's books critiquing the books about how to solve problems. But humans are just so good at perception that we never really talk about it. Like maybe in art books where it's like, "Hey, to show that that mountain is further away, you need to desaturate it a bit," or whatever. But, um, the a- the actual data on how to like look at images is, isn't really present. Also, the data we have is kind of sketch. The best source of data we have is like image alt text pairs on the internet, and that's pretty low quality. Um, so yeah, I, I, I think our solution here is really just we need to teach them how to operate on individual tasks and figure out how to scale that out. Um, all right. Yep. So in conclusion, uh, at Moondream, we're trying to build amazing VLMs that run everywhere. Very hard problem, much work ahead, but, uh, we're making a ton of progress that I'm really excited about. Um, if anyone wants to chat about more, um, technical details about how we're doing this or is interested in collaborating, please, please hit me up. **Host** [54:40] Mm-hmm. Uh, yeah, like, uh, always... When people say, when people say multimodality, like, you know, I always think, think about vision as the first among equals in all the modalities. So I really appreciate, um, having the experts in the room. This is the year that vision language models became mainstream with every model from GPT-40 to 1, to Claude 3, to Gemini 1 and 2, to Llama 3.2, to Mistral's Pixtral, to AI2's Pixmo going multimodal. We asked Peter and Isaac to highlight the best work in computer vision for 2024, and they blew us away with the complete overview. ### Outro **Host** [55:22] As a special bonus, we also got a bonus talk from Vik Korrapati at Moondream, who gave an incredible talk at this year's AI Engineer World's Fair on his tiny 0.5 billion parameter pruned vision language model that absolutely slaps. As always, don't forget to check the show notes for the YouTube link to their talk, as well as their slides. Watch out and take care. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.