# [Paper Club] Molmo + Pixmo + Whisper 3 Turbo - with Vibhu Sapra, Nathan Lambert, Amgadoz

Latent Space · 2024-10-13

<https://addtry.com/d1db6e55-f12c-4ddb-8c86-168ac2b89868>

This episode covers two papers: AI2's Molmo open-source vision-language models and OpenAI's Whisper Large V3 Turbo. For Molmo, the key claim is that high-quality, audio-annotated data (Pixmo) enables models as small as 1B to match GPT-4V on academic benchmarks, with the 72B variant outperforming GPT-4o, Gemini 1.5, and Claude 3.5 Sonnet. The team avoided distilling proprietary models by having annotators describe images in speech for 60-90 seconds, then transcribing and augmenting the captions. For Whisper Turbo, OpenAI pruned the decoder from 32 to 4 layers and continued pre-training on 10 million hours of multilingual transcription data, making it 1.78x smaller than Large V3 with minimal word-error-rate increase. The model is faster and supports real-time chunk-based decoding, unlike English-only DistilWhisper. Nathan Lambert and Amgadoz provide commentary on data strategies and benchmarking nuances.

## Questions this episode answers

### How does the Molmo paper create an open-source VLM without distilling from proprietary models?

They ask human annotators to verbally describe images for 60–90 seconds each, guided by prompts like 'What objects are there?' Transcriptions (via Whisper) are cleaned and summarised with an LLM, generating 1.3 million captions for 712,000 images. This Pixmo dataset, free of any proprietary model outputs, trains the Molmo models to match or exceed GPT‑4V and Gemini on vision benchmarks.

[0:06](https://addtry.com/d1db6e55-f12c-4ddb-8c86-168ac2b89868?t=6000)

### What makes OpenAI's Whisper Large V3 Turbo faster and smaller?

Whisper Turbo prunes the decoder from 32 layers to just 4, reducing model size by 1.78×. It then undergoes continued pretraining on the same multilingual data as Large V3 (omitting translation data) to recover accuracy. This avoids knowledge distillation and yields a model nearly as accurate but much more efficient.

[0:54](https://addtry.com/d1db6e55-f12c-4ddb-8c86-168ac2b89868?t=54000)

### Why did AI2 include synthetic analog clock images in the Pixmo dataset?

Nathan Lambert says the lead researcher was frustrated that no vision‑language model could read analog clocks, so they created a synthetic dataset to teach Molmo this skill. Interestingly, the model learned to read clocks well but still struggles with other dials like oven or pressure dials, showing limited generalization.

[0:28](https://addtry.com/d1db6e55-f12c-4ddb-8c86-168ac2b89868?t=28000)

## Key moments

- **[0:00] Molmo Overview**
  - [1:13] Most open-weight VLMs are distillations of proprietary models like GPT-4V, notes the host.
  - [2:23] Molmo's models are on par with GPT-4V and surpass it at larger sizes despite being much smaller, claims the paper.
- **[3:04] Audio Annotations**
  - [6:07] Annotators speak for 60-90 seconds to describe images, providing far more detail in less time than writing, reveals the Pixmo dataset method.
  - [7:39] Molmo 1B, a 1-billion-parameter model, performs on par with GPT-4V on both academic benchmarks and user preference.
- **[9:03] Model Architecture**
  - [12:15] Nathan Lambert: 'The whole vision space is so underdeveloped that many surprising results probably are not that surprising.'
  - [14:29] Nathan Lambert: Molmo has no chat template; it concatenates messages and is like 'Go. Look, good luck.'
- **[16:55] Data Curation**
  - [19:30] Q: Is Molmo's text performance inferior because it lacks synthetic text data? Eugene Chia asks.
  - [22:59] Q: How is the Pixmo Ask Anything dataset created without giving the LLM the image? Eugene Chia asks.
  - [28:38] Nathan Lambert on the clock dataset: The lead was upset it didn't work on clocks and no models work on clocks, so they made it work.
- **[30:34] Benchmarks & Openness**
  - [33:10] Nathan Lambert: This model tests the definition of open source AI because the vision encoder is the only closed component.
- **[38:11] ELO & Highlights**
  - [38:33] The human preference evals for Molmo used 15K image-text pairs and 870 annotators — the largest multimodal evals to date.
  - [40:28] Nathan Lambert on no RLHF: 'Cross-team alignment and incentives are hard.'
  - [41:06] Q: How granular should image caption pairs be to achieve human-like AI performance? Sam asks the panel.
  - [43:56] Eugene Chia: Captioning tends to overfit; question-answer pairs force the model to pick up details it would otherwise ignore.
- **[50:09] Whisper Turbo**
  - [50:50] Whisper Large V3 Turbo is faster and more efficient with minimal accuracy loss, using pruning and continued pre-training, says Amgadoz.
- **[54:24] Pruning & Pre-training**
  - [54:26] Whisper Turbo prunes decoder layers from 32 to 4, making the model 1.78× smaller, and uses continued pre-training, not distillation.
  - [56:24] Whisper Turbo's continued pre-training uses the same 5-million-hour-per-epoch dataset as the original, over 2 epochs (10M hours total).
  - [57:24] Whisper Turbo has 4 decoder layers and is multilingual; DistilWhisper has only 2 and is English-only, explains Amgadoz.
- **[57:58] Distillation vs Pruning**
- **[1:03:34] Real-Time & Benchmarks**
  - [1:04:16] Q: Is Whisper Turbo fast enough for real-time? A: Yes, on modern GPUs, you can transcribe 1 minute of audio in less than a minute, says Amgadoz.
  - [1:10:01] On Karpathy's talks, Distil V2 is least accurate; Large V3 can hallucinate; Whisper Turbo performs very well, reports Amgadoz.

## Speakers

- **Amgadoz** (guest)
- **Eugene Chia** (guest)
- **Nathan Lambert** (guest)
- **Sam** (guest)

## Topics

Audio & Music, Computer Vision, Benchmarks

## Mentioned

AI2 (company), Hugging Face (company), Meta (company), OpenAI (company), CLIP (product), Claude (product), DistilWhisper (product), Faster Whisper (product), Flamingo (product), GPT (product), Gemini (product), LLaVA (product), Llama (product), Molmo (product), OLMo (product), OpenCLIP (product), Phi (product), Pixmo (product), Qwen (product), Whisper (product)

## Transcript

### Molmo Overview

**Host** [0:01]
I'll, I'll go through here. Um, let me set up real quick. If anything's in the chat, let me know. Okay. So, uh, Molmo and Pixmo. This is from, uh, AI21. Basically really, really good people. If people haven't read research papers before-

**Nathan Lambert** [0:18]
Yeah. AI2-

**Host** [0:18]
This is probably one of the-

**Nathan Lambert** [0:19]
AI2 and AI21 are different things.

**Host** [0:23]
AI2, my bad.

**Nathan Lambert** [0:24]
It's hilarious. So many people mess it up. We also get A12.

**Host** [0:28]
A12. Oh, are they another one, or different?

**Nathan Lambert** [0:30]
No, it's if you misread the I as a one.

**Host** [0:33]
Okay, so AI2.

**Nathan Lambert** [0:34]
Makes it even more confusing with the 20-

**Host** [0:37]
Is there an AI21? Okay, I've been showing the wrong one. But, um, good correction, good to know. So, uh, AI2 though. So basically this is a proper open source, open weights model, and it's a VLM. So if anyone hasn't read papers before, this is one that I highly recommend reading.

Clear problem, c- clear solution, clear explanation of what they do, the data set and everything. Very, very easy read, no crazy math. Um, so yeah, highly, highly recommend. But TLDR, it's a set of different models at different sizes that are good vision language models that solve a different problem.

So the high level of this is most VLMs are distillations of proprietary closed source models, right? So if you need to generate synthetic data, like most open weight models rely heavily on synthetic data from private models. So think GPT 4V, GPT 4O, Gemini, whatnot, and they, they label a lot of data and then they, they create a vision language model that's open weight, something like Llava.

But all they're doing is kind of distilling that proprietary knowledge into an open knowledge. But this is kind of the key highlight and takeaway, that comm- the community is still missing foundational knowledge about how to build these models from scratch.

So they kind of go into everything from data set, training, how they label data, and how they can do this whole mix. And some of the other papers we covered in Paper Club are stuff like Clip, OpenClip, which are captioners, the whole history of these, and how captioning is pretty important for this.

But basically, yeah, they do that. They, um, explain the data set mixture, how to do this fine-tuning. They're gonna release it all. There's a set of different, um, a class of different models that are gonna come out. And the, the really interesting thing is they're good.

So Llava is kind of starting to lag behind, but, um, this thing is outperforming. So there's two ways to benchmark this. You've got standard vision benchmarks, and then you've got like actual usage, and their stuff is on par with GPT 4V.

They're starting to surpass it with some of the larger models, and they're very small. So yeah, diving into that, we talk about the data set, how they do the training data, the architecture of it, which is kind of what you would expect if you joined our other Paper Clubs on this.

But let's, let's go a little bit more into that. So point of this is how can we generate this data from non-proprietary models? So we can't use a closed source model to do this captioning. How do we do it?

### Audio Annotations

**Host** [3:04]
Some of the background that we learned from other mo- from other papers like Clip and whatnot is that it's hard to generate really good, diverse, like robust descriptions from images. And this is something that like the solution that they found was to have audio labels.

So they kind of guided people towards here's a set of about a million images, here's some prompting questions, here's a timeline so people answer them concisely. From that, they have a data set. From that, they train it into a model that's based on their open source Molmo models.

So that's the big thing. They, they can do this end to end without anything proprietary. And yeah, let's, let's like go through it a little bit more. So instead of being a distillation, it's independently pre-trained. There's a vision encoder and a language model that's jointly trained.

There are some interesting stuff that they call out, like traditionally in stuff like Llava and different adapters, what they'll do is they'll train this vision encoder. They'll have an LLM. They'll freeze weights, and then they'll kind of just merge them.

They avoid doing that multi-stage pre-training. They don't freeze parts of it. They just kind of do it all as one training run. Um, they show how important high quality data is, so they do this, I think, on roughly a million samples of data.

There's some augmentation, like they pass through these, uh, labeled data sets through LLMs. They augment them. They do augment it a little bit, but point being, they can get better than Gemini 1.5, better than Claude 3.5 Sonnet, better than GPT 4V at a much smaller size with about a million samples of data, which is very impressive, right?

So good high quality data, and then how to do it from scratch. Now, the, the other little interesting thing here was it's not as accessible as it seems. Like they still had to pay for a million labeled samples, but point being still, still very useful.

And then we can distill it down to other models. So, um, yeah, it's, it's... This is one of the main challenges, right? It's hard to collect captioning data. So for example, like if you're given an image here and you're told to describe it, like I would probably just say, "This is Seattle."

I could add in things like, you know, "I see the Space Needle. I see Mount Rainier. I see trees." But if you just blindly give images to either LLMs or whatever, multimodal models or people, we don't give good descriptions, right?

We don't give like robust, you know, a city landscape with a lot of skyscrapers, trees in the foreground, water in the background, construction on the side. We don't do that. But they kind of broke down how they get this type of data and the different segments, and then I think soon the data set is coming out.

But that's basically the, the challenge, right? So they don't wanna get this from larger proprietary models, which can... Like, you can write a really good robust prompt and get good descriptions, and then you can create a Llava style model, but you're no longer doing this from scratch.

You're doing it from a distillation of proprietary work. So they wanted to do it with annotators. Here's kind of the solution. "We ask annotators to describe the images in speech for 60 to 90 seconds rather than asking th- them to write descriptions.

They prompted them to describe everything in great detail, including descriptions of spatial positioning and relationships." So stuff like, you know, there's the Space Needle in the middle of a bunch of buildings. Um- With that, the, the modality switching trick annotators provide far more detailed descriptions in less time.

So part of this is, you know, you keep it 60 to 90 seconds. They gave them guiding questions, but that was basically the, the takeaway. Like, we're not good at writing descriptions, but we're good at talking about and describing what we want.

Um, and then from there they, they have, like, a little bit more into what this data set is. So they've got positional data in some of the prompts, they've got high level, they've got different types of data. They've got, like, part of the data set is about, uh, charts, tables, descriptions of that so the model can pick it up.

And then there's some use cases where this stuff just blows out GPT-4V from, like, out of the water. So there's a good, uh, blog post on the website that kind of shows how you could do this real time.

It's a very Apple Intelligence or Google Lens type example, where I think the team put it on the Apple Vision Pro, and they just look at stuff and they're like, "Hey, where is this in the image?" And it can, like, actually annotate the image because that's part of the training set.

So one more use case of high quality, well labeled data. Um, benchmarks-wise, they did the traditional 11 academic benchmarks. It does well. Um, the size of the models they put out, so there's Molmo 1B, which is based on their open source language model, which is a MoE one billion active parameter model.

That performs on par with GPT-4V on both, uh, user preference and academic benchmarks. Then there's the 7B, which is based on their 7B language model, and Qwen 7B. That outperforms GPT-4V and 4o. Then there's a large one, which is Molmo 72B.

72B is based on Qwen 72B, and that, um, you know, it, it outperforms stuff on academic benchmarks, and then on human preference it's behind 4o. So preference-wise, if you guys remember how we do, um, vision benchmarks, there's stuff like Vision QA, where you can QA, you know, what's in this image.

It's similar to Trivia QA, where it's just understanding. There's 11 benchmarks, and they're not realistic of how people use it, so they also have this user preference stuff. User preference is basically like an ELO score of models, but TLDR, these are very good small models.

Um, this last sentence here is pretty interesting, right? "Our best models outperform state-of-the-art proprietary systems, including Gemini 1.5 Pro and Flash and Claude 3.5 Sonnet." That's pretty huge. Um, architecture, if you guys know about other multimodal models, it's, it's an interesting adapter with a little twist of traditionally, like we said, um, there's, there's this mix of you have a, a pre-trained off-the-shelf vision encoder and a language model joined in one space.

Typically, we freeze the language model and we kind of do contrastive, um... There's a term for this, but you basically merge these embedding spaces. I'm blanking on the term. In this case, they're like, "No, we don't need to do that."

### Model Architecture

**Host** [9:16]
Um, so they go over the four steps of this architecture. Uh, pretty straightforward. So the first step is a preprocessor. It converts the image into a multi-scale, multi-crop images. This kind of allows for diversity in inputs. So stage one of architecture is a preprocessor.

Second, it's a ViT image encoder that maps these embedd- these, um, images into vision tokens in an embedding space. Three, there's the connector that maps and merges these two embedding spaces. So the vision encoder, which I believe is based on CLIP, and they, they mentioned, you know, we know CLIP is trained on proprietary data, but there's a paper called OpenCLIP from Meta which shows you can do OpenCLIP.

They just use regular CLIP, which is... I think it's fine. Um, so stage one, preprocessed images. Stage two, vision encoder. Stage three, contrastive embeddings, which merge the, um, embedding dimension of the vision and the language model, and that's not frozen.

And then stage four is a decoder-only transformer LLM to produce text. That's the architecture. If, um, people have questions on what's going on here, we can probably dig into it, but for people that joined our other multimodal stuff, um, you should kind of understand how this works.

There's a good blog post we went over as well, I think from Chip Huen, or another one that we can also share as a reference. Um, but yeah.

**Host 2** [10:39]
Yeah, the Flamingo post.

**Host** [10:41]
There's what?

**Host 2** [10:43]
I think that was our Flamingo post, uh, we said with the models.

**Host** [10:45]
Yeah, Flamingo. Our Flamingo is a really good one on how multimodal stuff works, so highly recommend, but also we can dig into it here. And then this is kind of that part of the vision encoder. Uh, they use OpenAI's, uh, ViT Large CLIP model, and we know that CLIP is trained on closed source data, and they understand this.

But recently, Meta CLIP came out from, um, Meta, and they show that you can reproduce it from scratch. They use OpenAI because it's trained for higher resolution. Meta's was-

**Host 2** [11:17]
Yes

**Host** [11:17]
... more of a proof of concept. Um, proof of concept meaning, you know, here's how you can do it. We do it efficiently at low resolution. For the actual production, they just use CLIP, and I'm fine with that.

If people disagree, that, that's a fair topic, too. And then the, the LLM they use is their recently put out Olmo models, which are very similar to this. They're open language models. They've got the range, and then for the big one, it's, um, OpenWeight Qwen.

Now, the, the data set and training was kind of inter-

**Host 2** [11:46]
Do you wanna, do you wanna pause for any commentary from Nathan?

**Host** [11:49]
Yeah.

**Host 2** [11:50]
Otherwise, yeah. Nathan?

**Nathan Lambert** [11:54]
I was just gonna kind of let him cook and then go back to the beginning.

**Host 2** [11:58]
Oh, okay. All right.

**Nathan Lambert** [11:59]
I mean-

**Host 2** [11:59]
I can always run an interleave commentary.

**Nathan Lambert** [12:01]
I thought about interrupting. I had unmuted. I just think that it's... I mean, I have this whole take that I've written is that the whole vision space is just so underdeveloped that a lot of these things that seem surprising I actually think probably aren't surprising.

And the things that this model is good at are things that all the foundation comp- like, they're just gonna take our data and train on it, and it's like, it's fine-tuning, so it's like a million data points is actually kind of a lot and all this stuff.

And it's just like- A lot, I think that is just, like, good data works, and they figured out some cool new applications, like clock reading and pointing, and then the data works. There is actually, like, a niche reference of somebody that did this kind of voice annotation earlier.

I can find it, 'cause I had ... Someone had mentioned it after we released it, saying that we were the first people to do it. So then, like, somebody's like, "Oh, but you have to cite this old paper," which was kind of fun.

So I can- I'll then go find that.

**Host** [12:55]
Okay. I love the take of me being like, "I love it. It's so straightforward, and it works, and people shoulda done it." And then Nathan here like, "Yeah, they'll do it too." Um, a- and it's cool to see the actual use cases.

The website has a pretty good blog post on this, and there's a good, very professionally shot video on how this excels, and it's always nice to see that niche of, like, there's so much underdeveloped alpha left in vision space that, you know, you train the right data set and it gets good.

But, um, it, it's still cool to see how it's, like, significantly better than Llava. It, it's all open, so I like it. Um, any other questions from chat and whatnot? I haven't been following. Is there, is there anything we should, um, pause and break on, or ...

**Nathan Lambert** [13:42]
Something that's worth saying is that I think the other models are much better as text models. Even, like, Llama, where their, like, vision fine-tuning is generally accepted as being, like, kind of meh for whatever drama reasons you wanna attribute to it, like, their text scores are still much better.

So they have, like, more of a text focus, which OpenAI probably does as well.

**Host** [14:02]
Is there a drop in performance on these? Like, the Qwen 72 versus Molmo 72, is the text performance also taking a hit?

**Nathan Lambert** [14:10]
I honestly don't even know, but probably to some extent. Like, it's ... I find it so unlikely it's gonna get better.

**Host** [14:16]
Yeah.

**Nathan Lambert** [14:17]
Or it's ... Even, like, some of these other models are probably using, like, instruction models as their base before doing the vision stuff, but this is just, like, straight base model, no real instruction tuning. There's literally, like, no chat template for multi-turn.

It just concatenates the messages together and, like, is like, "Go. Look, good luck."

**Host** [14:36]
So for, for other people that don't follow, there's base models and then there's instruction fine-tuning, right? So you can ... Base models predict next token. Instruction models are chat models. So, like, we have Llama base, which is not a chat model.

You can't chat with it. It'll complete your words. Uh, basically these are just those adapted to vision, which means that they're not good chat models, but they do perform well, and they do do what they're supposed to do.

But TLDR, yeah, you, you know, they've, they've s- done so much open source work, so maybe sh- someone should recreate. I don't know. Have you guys put the data set out yet? I, I hear it's coming soon.

**Nathan Lambert** [15:12]
It's just, like, needing to clean it up.

**Host** [15:15]
Yeah.

**Nathan Lambert** [15:15]
So it was, like, the rush of get the model out. I mean-

**Host** [15:19]
Makes sense

**Nathan Lambert** [15:19]
... back our leadership probably knew Llama was coming on a specific day, if you look at the timing. Like, there's some of that needed to happen. I mean, the data set's just being cleaned.

**Host** [15:30]
Okay.

**Nathan Lambert** [15:30]
And I mean-

**Host** [15:32]
Well-

**Nathan Lambert** [15:32]
When you do anything with images, something, like, data set releases are a little bit messier. You have to check for, like ... You don't ... Like, we're not releasing the images, we released the links type of thing. Do some checks.

But yeah, it's coming. I think there is, like, hundreds of people that filled out the silly Google form asking for interest, which I'm not surprised. I'm just like ...

**Host** [15:55]
Yeah. Uh, well, that's a plug for, you know, they started 90% of the open source stuff. Now take a, take a chat instruction tune model and do the same thing if anyone does it.

**Nathan Lambert** [16:04]
We did a little bit of experimenting on it, but n- like, in a rushed way, and we didn't find it was better, so they didn't add it in.

**Host** [16:11]
Hmm. Interesting.

**Nathan Lambert** [16:12]
But, like, this is very visual oriented. The demo, you can't send a ... You need an image.

**Host** [16:19]
Yeah.

**Nathan Lambert** [16:19]
And you, like, can't send a text-only query to it. Partially for saving money, but partially because, like, that's how the model is trained.

**Host** [16:28]
Good to know. Good to know. I, I think the interesting thing there is, like, as you mentioned, uh, yeah, Gemini and OpenAI and stuff, they'll basically train on this data set, and it'll, it'll kind of improve what they do too.

But, um, it, it also kind of shows a flaw in benchmarks, 'cause our, our vision benchmarks don't account for, like, text output as well, which is, you know, pretty flawed because people, people do both. Um, yeah. This is, this is the fun part since it's primarily vision-based.

### Data Curation

**Host** [16:55]
How did they, how did they do the vision? How did they generate the data? So, um, they've got a split of different data sets. Uh, step one is opti- obviously caption generating from the images. So they source web images from diverse sets of data, 70 high-level topics.

So high-level topics: street signs , street signs, food, meme, drawings, websites, blurry photos, and whatnot. They asked annotators to describe images by, uh, speaking in detail for 60 to 90 seconds using a single annotator per image. This was more effective.

But yeah, questions were, like, guided questions. So, uh, what's the image at first glance? What objects are there and their counts? What does the text say? What's the background? What's the style and color? I think this is, like, pretty useful, 'cause I'm pretty bad at describing images, and 60 to 90 seconds is a good, like, you know, answer a bunch of stuff.

People will be verbose, concise. Then after this, uh, basic pre-processing, so transcribe it using off-the-shelf speech-to-text. Um, do you know what that was? Was that, like, you know, Whisper or something, or ... It just probably doesn't matter, I'm guessing.

**Nathan Lambert** [18:02]
Yeah, it was, like, Whisper, and I don't remember the details. And I, I, I had asked, and it's like, I don't think they actually logged the audio. I was like, "Oh, this would be super cool for, like, other types of multimodal," but I don't think it was in the terms of the-

**Host** [18:15]
Ah, fair

**Nathan Lambert** [18:15]
... data annotation to also have the audio with the images, 'cause it'd be a super cool, like ... I don't know, like, you can, you can see how all these voice features come about.

**Host** [18:25]
That's what I was about to say. I was gonna say, like, it, it would've been a pretty cool audio data set too, but it looks like the data set that'll be pr- uh- ... distributed is just gonna be the trans- the captioning and image pair, so not audio, which would've been fun.

But, uh, yeah, there's, there's a very straightforward pre-processing, so use something like Whisper or whatever off-the-shelf they wanna call it to speech-to-text. Um, then, uh, process it through an LLM to improve text quality, so remove artifacts, normalize style.

Then, uh, create a fourth image description by using an LLM to summarize the three original descriptions transcripts into a single description. So, you know, basic augmentation. Um, from that we use these four images to do natural data augmentation.

So trained on 712,000 dis- uh, distinct images with 1.3 million captions, including the augmentation. So not a lot. It's basically just fine-tuning after that. So, um-

**Guest** [19:25]
I, I know Eugene Chia has a hand raised. You have a question, Eugene Chia?

**Host** [19:29]
Hop in.

**Eugene Chia** [19:30]
Yeah. It, it wasn't urgent for me. I- because there was a lot of commentary regarding text performance and, and the... a- and I guess the presumption that the Molmo one is inferior in text performance. That's what I kind of understood.

Um, I'm actually wondering how much of it is potentially for, especially the closed Molmos, just purely synthetic data, because I can imagine for text in particular, we can ch- uh, we can generate a lot of synthetic images, essentially signboards, prompt paper, all these kind of scenarios, and just put var- various permutation of text, and we ask questions about the text through the model, and that essentially forms the training set.

**Host** [20:16]
Thoughts?

I, I think a lot of it's also just, yeah, it's a base model. Um, I think that there's a decent bit you could do. I think also at the scale that they're at, like with a 1B and 7B, um, they could be deployed in a system such that it's primarily image processing, like zero-shot, few-shot, you know, convert this to JSON, answer stuff about this.

I, I don't see this really being like a chat with style model because, yeah, there is like... there's Llama if you want that, right? If you want images and chat with it. But the applications of these edge small image processing models, I, I see this more as like, you know, OCR output in a pipeline.

But that's my two cents. Um, I'm... I guess it's interesting to see that if you did it with a chat instruct tune model, there wasn't any crazy performance difference. But yeah, the other thing here is just that there's not much benchmarking on, um, the language model side.

So I guess there is MMLU, and they're not the best. Um, actually quite, quite roughly quite bad. Um-

**Guest** [21:34]
That's MMMU.

**Host** [21:35]
MMMU, my bad.

**Guest** [21:37]
That's a multimodal version.

**Host** [21:39]
So there's no, there's no pure text, um, benchmarks, right? Should we benchmark? Someone benchmark it. We've got 40 people on the call that are here for the paper. So some, someone report back to us. Um, yeah, any other interesting chat stuff?

Should we-

**Guest** [21:55]
I do

**Host** [21:55]
... crunch the paper quick?

**Guest** [21:57]
Me?

Um, Molmo?

**Host** [22:05]
Is someone talking?

**Guest** [22:07]
Mm-hmm.

**Host** [22:08]
Okay, they're muted. Um, let's go to that data set then. So TLDR, people yap. Here's guidance questions. It's transcribed. They got about a million captions for 700,000 images, which I'm like, "Okay, that's, that's kinda expensive. I can't run this experiment myself."

I'm glad they'll put it out. Um, basically, there's this, uh, Pixmo is the data set, and then there are subsets of it. So Pixmo Ask Anything Model, it's, uh, basically collection of answer diverse set of questions a user might ask.

So for this image, what are questions people might ask? Um, here's how they do it. They have this whole OCR stage. I'm gonna go through these pretty quick, but then here's kind of the subset. So 162,000 question-answer pairs of images.

Uh, Pixmo Points is the next one.

**Guest** [22:56]
Can, can I ask a quick question on the Pixmo Ask Model Anything?

**Host** [22:59]
Mm-hmm.

**Guest** [22:59]
So I had a really hard time understanding this. Um, what we do is we use the stage one model to generate a dense caption. So I guess what this means is that give an image, get a caption, and then we pass that caption an OCR image for the...

and the OCR output for the image. Does this mean we just apply OCR on the image? How would that work if the image is a image of scenery or something? Um, I mean, if... uh, someone can just share the intuition behind this.

**Host** [23:29]
Nathan, if you have intuition, you'd probably know better. My intuition here is that, uh, when you look at their 70 different high-level topics, like drawings-

**Guest** [23:40]
Yeah

**Host** [23:40]
... websites, there's another thing here like street signs, and then there's one-

**Guest** [23:44]
Yeah

**Host** [23:44]
... that's like primarily charts and tables. So I don't have the exact input-output of this, but I'm assuming it's more text-based if there's OCR on it, so it's probably, you know, um, documents-

**Guest** [23:58]
I-

**Host** [23:58]
Receipts. That's what I would assume this subset of the data set is.

**Guest** [24:03]
Yeah.

**Host** [24:03]
I see.

**Guest** [24:04]
Yeah. But that makes sense.

**Host** [24:06]
But there-

**Guest** [24:06]
So-

**Host** [24:06]
I mean... Yeah.

**Guest** [24:08]
Yeah. So we provide caption, we provide OCR output in case there's charts and tables, and then we ask a question, and the LM is supposed to answer the question purely solely based on text. It does not have the image.

Um, yeah, that's, that sounds crazy to me. I just don't know how we can teach a model to answer anything about an image without actually having an image. Um, but if, if it works, it works. Uh, but that was quite mind-blowing to me, e- especially this data set and how they created it.

Yeah

**Nathan Lambert** [24:36]
So I kind of interpreted this as that's like a starting point, and then the annotator would... I d- I wasn't sure why they did it this way, and maybe Nathan can speak to this, but, uh, uh, that, that you, like, have these captions and then you're kind of just selecting and combining annotations as a human who has access to the, the, the vision of the model or the vision of the image.

**Guest** [25:04]
That makes sense. Yeah. Or maybe this is one of the earlier images where we're just, uh, bootstrapping our synthetic data and the quality is maybe is not as high or not as intuitive as compared to the point, uh, Pixmo- Pixmo points, Pixmo clocks, which was easy for me to understand.

So this is the one thing I got stuck on. Thank you.

**Host** [25:22]
Are you... Wait, the, the model was trained with the image and the question pairs, though. It's saying that the LLM that's doing the question answering didn't have access to the image. No? Like, it's still a output pair is still image to QA pairs that you train Molmo on, but the, the annotation-

**Guest** [25:46]
I see

**Host** [25:47]
... of the image. That's what I... So, like, I think that's what you were saying. Like, it's interesting how you can train, uh, Molmo to answer questions on an image without giving it the image. But no, it's given the image and question answer pairs.

It's just the annotation-

**Guest** [26:00]
I-

**Host** [26:00]
... answering the question is image to OCR to GPT-4, write question answers on this OCR to, um... You know, there's a... What is it? There's a annotator to approve the question answers. Now we've got question answers with image.

When we train, we have image question answers.

**Guest** [26:20]
I see. Yeah. Okay. That, uh, that, that actually is a... That's actually very intuitive. So essentially what we're saying is that right now the model can't answer questions. What we're gonna do... But the model can generate captions because that's the previous Pixmo Cap layers learned on.

So we're gonna get that model to generate captions.

**Host** [26:35]
Mm-hmm.

**Guest** [26:36]
Then we get an off-the-shelf LLM, probably one of their strong LLMs, to create QA pairs. And now based on that, we now have QA pairs on the image.

**Host** [26:45]
Yes.

**Guest** [26:45]
So that's how you bootstrap yourself from no data to, like, step by step. Okay.

**Host** [26:50]
This little niche here was that that off-the-shelf LLM is not-

**Guest** [26:54]
At all

**Host** [26:54]
... vision capable. So that's why-

**Guest** [26:56]
Yeah

**Host** [26:56]
... they have to caption an OCR and provide that to the LLM.

**Guest** [26:59]
Makes sense.

**Host** [27:00]
So that- that's where I'm like, oh, I think you mentioned they're training questioning without giving it the image, but you still do have the reference image.

**Guest** [27:08]
Yeah. The, the, the final Molmo data actually has that, but this is the process of them just generating the image.

**Host** [27:13]
Yeah. So-

**Guest** [27:13]
Cool.

**Host** [27:14]
Thank you. Yeah. And then, uh, honestly, I found this one pretty confusing at first as well. The rest, there's, there's just quite a few subsets of these, um. So points, this is kind of their demo. Um, I don't know if I'm sharing my whole screen or just the...

Yeah, I'm just sharing the-

**Guest** [27:33]
PDF

**Host** [27:33]
... paper with you. Well, um, actually, I'll change it real quick. If people haven't seen the demo, I would recommend checking it out. But this is basically points on an image, so, um, it, it kind of answers, like, if you were to ask this, uh, this example, point to Mount Rainier, the model can drop a dot on the mountain.

So it's, it's labeled data set of that next sample. And then they, they show their example of connecting this thing to Apple Vision Pro and asking it, like, what do I see here? Um, they've got a subset of that.

Then there's, uh, Cap QA. It's basically just here are the other ones. There's docs. Generate, um, code for this much text and heavy figure images. So charts, documents, tables. We prompt it to do QA pairs. There's clocks. I was confused with why there's synthetic analog clocks data set.

Uh, but there are clocks. There's a lot of clock examples in here. Um-

**Guest** [28:32]
I had the same question. Like, Nathan, would you please-

**Nathan Lambert** [28:34]
This is because the model didn't work on clocks, and then the lead was really-

**Guest** [28:38]
Ah

**Nathan Lambert** [28:38]
... upset it didn't work on clocks, and no models work on clocks. So they're like, "We've got to make it work on clocks." One of the interesting things is that it doesn't work on dials even though it works on clocks.

**Guest** [28:48]
It doesn't work on dials, like an oven dial or washing machine dial?

**Nathan Lambert** [28:50]
Yeah, like a temperature dial or su- or pressure dial. Like, it can't read the number off of that, but it'll work on clocks. Which is, like, such a... Like, it should be able to. Like, why can't it generalize?

**Guest** [28:59]
Isn't it supposed to generalize? Oh, my God. Okay. But th- that was very-

**Nathan Lambert** [29:02]
But it does work well on clocks.

**Guest** [29:03]
That was very good, uh-

**Host** [29:04]
Interesting

**Guest** [29:04]
... behind the scenes. Okay.

**Host** [29:05]
I love the answers, though. These are the insights that, like, we'd never get. But is this-

**Nathan Lambert** [29:08]
Yeah

**Host** [29:09]
... not just, like, overfit to a benchmark that nothing else does? Is this not like it should generalize a bit less? But I also love it, like-

**Nathan Lambert** [29:16]
Is there a, is there a clock benchmark?

**Host** [29:18]
Hell yeah, dude. You threw it in. You... Like, now-

**Nathan Lambert** [29:20]
Oh, okay

**Host** [29:21]
... now I know. Now every vision language model that comes out, I'm gonna be like-

**Nathan Lambert** [29:24]
Well, well, it is gonna make... Like, it is something that they should be able to do.

**Host** [29:27]
Yeah. It should generalize it.

**Nathan Lambert** [29:29]
I mean, a big story of this is that, like, Matt De- De... I don't know how to pronounce his name, has just got obsessive over data and just kept scaling it up and up and up, and all the things they were adding to this data set kept on working, and they just spent more and more money because it kept getting better and better and better over, like, six months of just making the data set bigger.

So, like-

**Host** [29:46]
Then-

**Nathan Lambert** [29:47]
L- like, the pointing is probably similar. It's like, why can't they point? And then they're like, "Okay, let's make it work."

**Host** [29:52]
Like, pointing seems useful though, right? Like, uh, the demo you guys showed was actually somewhat useful. Like, uh, you know, translate this table in a whole image to JSON and it, it does it. Or, like, um, stuff like this, like point to Mount Rainier.

It can point, but other models can't. Seems useful. Clocks? I don't know. It, it... This just reminds me of strawberry. You know how many Rs are in strawberry? Um, someone, someone start this with GPT-4V, what time is it?

And it can't tell time. Uh, and then of course the rest, there's academic da- uh, data sets. Once again, the Flamingo blog post goes into all these very deeply. Would recommend. Um, but TLDR, those are the data sets.

Evals are evals. Um, this is where I was like- Okay. It does good on vision. Um, there's academic benchmarks, which are, you know, 11 of the commonly used ones. There's good averages. I love the new rebrand they've done with the whole pink and color theme.

### Benchmarks & Openness

**Host** [30:50]
It looks good. Averages, um, but yeah, I mean, across the board, the 7B, the 1B, the 72B, very on par with 404V, 1.5, Claude Sonnet and whatnot. Um, the human ELO ranking was pretty interesting, where per- preference is decent, but, um, I, I guess the interesting differentiation there is how far of a drop other stuff like the Lava and whatnot is.

Uh, yeah, Phi is doing pretty rough in multimodal even though it got a multimodal update, but it's probably a better text model. I guess I'd just be interested now to see how this thing does in text and if we can get a better text version.

This is kind of back to the theme of this paper of, like, stuff. Most models use a vision encoder that's closed, so we wanted to do full open. Um, this is a little harder to read, but let me digest real quick.

So, um, the vision encoder for Lava is open weights, but the data and code used to generate it was closed. API models, they're, they're basically all closed. A lot of the generation is done with closed stuff. They're in this corner of all green.

We love all green. We don't love red. Um, but this is basically just a visual of the theme of the paper. I didn't look that deep into it, but that's what it's showing that. Once again, the, the captioning that's done is mostly a distillation of proprietary stuff.

For example, all this proprietary stuff sucks at clocks, so nothing that's a distillation will be good at clocks. Nothing can point if you just distill from this. If they can't point, your VLM won't point. So we show how to get good data.

The answer to that was have people talk with guided questions, do pre-processing, and I guess keep scaling it up. It's good to see it still scales. Um, yeah.

**Guest** [32:45]
And also one point on the Molmo one, right? Uh, if you scroll up all the way to the top. Yeah, over here. The only reason why Molmo 7B and 7BD is not open data and code is because it's based off the Qwen data.

I think. Correct me if I'm wrong, Nathan. And that's why it's, it's not-

**Nathan Lambert** [33:01]
Yeah, that's right.

**Guest** [33:01]
It's, it's not because they don't want to open it. It's because we just don't know what was in Qwen data.

**Nathan Lambert** [33:05]
Yeah.

**Guest** [33:05]
Same for vision encoder. If not it's everything, it's everything that the, um, the EI2 team did was-

**Nathan Lambert** [33:11]
Th- this model is test the definition of open source AI because it's like you grab some random vision encoder, and that's the only thing that's not open. It's, like, kind of disappointing.

**Host** [33:20]
Well-

**Nathan Lambert** [33:21]
But it's, like, also a good-

**Guest** [33:22]
The, the weights are open, but we just don't know what the data input is. Yeah.

**Nathan Lambert** [33:26]
Yeah. Yeah, but what's the difference between a vision... Like, if, if the weights of the vision encoder were frozen, I think that it could be fine, but I, I don't know. It's weird because you, like, said that fine tunes from Llama are, like, open weight models.

They're not, like, open source fine tunes and stuff. It's... And this is a discussion that, like, the open source initiative and all their people are, like, having right now. It's kind of a whole other thing that I don't have enough time to answer the emails for, but it's like I, I don't know what to do with it.

**Host** [33:57]
The interesting thing there was the only closed code for the... The closed open data was just they used Clip, large Clip, right? Uh, they, they have the sentence that, uh, "Our vision encoder is based on Clip," right here.

So for the vision encoder, we release, uh, "All of our released models use OpenAI's VIT Large Clip model, which provides consistently good results. While this model uses closed data, it can be reduced from scratch with Meta Clip." So that's what gives them some red because the data that OpenAI trained Clip on, they didn't open source.

So it's closed data, and they talked about this too. Like, the, the paper and the blog post really shows how the thing that made vision language models good was a good captioner, which was the last Clip that they put out.

And this is where OpenAI started to go closed source, where they're like, "Okay, we've, we've scraped the web. We have our own scraper. We have a good captioner," this, that. "Here's the data set." Or no, they don't give out the data set.

They give the, they give the final model, but not the data set. And then a few years later, Meta comes out with Meta Clip, where they're like, "Okay, here's how we can remake Clip." And I, I, I think that's personally...

This is just, like, now the open source debate of they could have used the Meta version and given us a worse model, and it would have given them green here. But I don't know. I think it's fine that they did it.

And then this is closed because, yeah, like Eugene said, uh, Qwen stuff. But that's my two cents. This could have been open if they just used Meta Clip, but Meta Clip's not as good because I think Meta did it as a proof of concept.

**Eugene Chia** [35:34]
I, I think, honestly, I think it's fine because right now, as previously outlined, be... for vision and audio, right, the gap in data set in the open source space is so large, right? The, the biggest, um, the biggest thing is actually more about getting the data set open source and working.

So for me, if in this process, sure, this may be flawed, and we get a strong open image data set, which then can... And then subsequently we can do captioning on top of that strong open image data set, which will create a more complete data set, then is used to train that to the next open model, then this will be a bootstrap essentially towards a all green open model.

Which might be a, which, which might be a better path than trying to do the use the inferior Clip and not have a good data set in the first place.

**Host** [36:26]
Makes sense. I think also, like, this is a proof of concept that open source stuff, like... I don't know how much there's, like, this is a product, use it, use it, use it, versus, like, we were scaling experiments, it kept working better.

And there's, there's still quite a bit that could be done here. But, um- Yeah, it's a really good point out that stuff is a distillation and we don't know what goes into this. We ... It, it, it does have that issue with, uh, Clip, right?

We don't know the data that went into it. Meta Clip would've been cool, but yeah. Um,

evals are next, so benchmarks. Um, if people understand and wanna dig into any of these, we can dig into them. So it, it's a interesting differentiation of open-weight, open-weight distillation where Molmo sits, and then these are all the proprietary, so Claude, Gemini, and GPT.

Here's the averages. Um, you know, Molmo 72B kills it. It's the best average across them. And then across the actual benchmarks themselves, um, they're, they're doing pretty good across. I think when you look a little deeper, they, they mention, like, you know, the 1B is on par with GPT-4V on most benchmarks, but, like, the 7B outperforms 4V.

It's closer to 40 and whatnot. So if anyone has anything interesting they'd wanna dig into on the benchmarks, we can. Otherwise, I think we know where it sits. The interesting part would be digging into the text generation part.

Um, Eugene, you still have your hand up. Are you trying to comment? Okay, hand down. Um-

**Guest** [38:11]
Go Sam.

**Host** [38:11]
Yeah. I, I didn't have much more to dig into in this. I thought the ELO ranking was kind of interesting as well. Um, here's ELO ranking. Um, human preference evals use 15K image text prompt pairs. We queried VLMs for responses, presented the resulting images, triplets to all this to, uh, 870 human annotators.

### ELO & Highlights

**Host** [38:33]
They were given pairwise preference rankings for 325 samples across 27 models. Nice flex. Biggest human preference evals for multimodal models to date. All it took was less than 1,000 people. Their ELO rankings are 3X more than Chatbot Arena from Limsess.

Also, I hear there's some potential tea with Limsess. Um, maybe that's coming out soon, uh, for their vision models. And then here's, here's an ELO ranking.

**Nathan Lambert** [39:00]
You're gonna drop potential tea with ... Limsess not doing scaffolding.

**Host** [39:06]
We'll, we'll leave that without much more unless anyone else wants to comment on it. Um, but yeah. Uh, if anyone knows how a ELO ranking works, um, they're pretty cool. Figure it out. But, uh, yeah, most preference ranked was GPT-4o, then Molmo 72B, then Gemini, then Sonnet, then the 7B.

It's really cool how the fourth one is the 7B, so very cool on-device model. And then, um, RIP our boy Llava 1.5 7B down here. Chameleon down here. Chameleon was cool though. They're fusion models. They're, they're a different way to do this.

They're a little early. I'd love to see a recreation of fusion for this as opposed to adapters. But, um, those are the two benchmarks. There are some, like, takeaways on this as well, so, um, here's a few key points.

Also, there was, there was a line here as well in their pre-training that I wanted to highlight. So i- in this data and training, they talk about how they do this, um, you know, different approach, and then they just add this line here.

So, um, we don't do RLHF, but we've got the RLHF guy here, but they added a line that, "We don't do RLHF." Um, is this meant to say, like, you know, there's no SFT, these are base models, or just ...

Well, there, there is supervised fine-tuning for it, but no RLHF.

**Nathan Lambert** [40:28]
Cross-team alignment and incentives are hard.

**Guest** [40:33]
We have a question from Sam.

**Sam** [40:37]
Thanks, Eugene. Uh-

**Guest** [40:38]
Sorry.

**Sam** [40:38]
Hold on. Is my mic on? Can you guys hear me?

**Guest** [40:40]
Yes. Yes, yes.

**Sam** [40:41]
Great. Great. Cool. Um, so this is, uh, coming from a, a place of vague naivety multimodality stuff, and, um, it's about the granularity that we should, uh, attach to images when creating, say, open source image caption datasets. So right now there's a, a picture, say, from like a satellite view of a parking lot, and I wanna ask a question that, like, Eugene could answer of, like, what is on the dashboard of the car in the second row, third from the left or something.

There's a little Hawaiian guy or something like that on the, the dashboard. So a human can answer that, but I assume a language model only could if, uh, it had been trained, I suppose, on, on image caption pairs that have that level of extreme granularity to them.

So, like, I've seen some really cool demos on the blog post for Molmo. Now, again, I'm not really sure whether to s- uh, ascribe that performance to the pre-chained, uh, vision encoder, the Clip one that we're using, or if it's from some, some data in the Pixmo set that uses that cool, like, audio transcription pipeline.

So I, I guess the question is if we kind of ig- ignored the fact that we're using a pre-trained image encoder that has a bunch of cool abilities on data that we don't know about, how granular should, uh, our image caption ...

our captions for our image caption or image caption datasets be if we want to have ... we would expect to be, you know, quote, "AI performance," you know, if you don't know how this stuff works, um, from our, our vision language models.

And, and why did you guys choose the granularity that you did, I suppose? Uh, and, and the question format as well.

Kind of a lot.

**Host** [42:18]
Do you wanna TLDR that into one, two line, Eric?

**Sam** [42:22]
How granular should our image caption pairs be, period?

How detailed?

**Host** [42:30]
Um, I think Nathan probably has more intuition on this, but their thing wasn't about granularity it seems. It was about one, like yeah, you've got diversity, you've got, um, you know, processing of it. But then it was also about the split, right?

So we talked about these four different ones. Like, part of it is, you know- Generate code from 255,000 text and figure heavy images, including charts and whatnot, then get QA pairs on it. They got clocks. But there's also all the academic data sets, right?

So, um, there's a lot of these if you look into what they are. That blog post goes into them. So I think granularity of specific stuff isn't how I necessarily look at it. It's diversity of the images and what they do with good captions around it.

Um, Nathan, if you wanna chime in, go ahead. Eugene, you have your hand raised. I don't know if you wanna answer, but let's answer this question before moving to another one.

**Guest 2** [43:31]
I don't really have good response. I think most of this was, like, really high detail responses is what worked for them. But, uh, I don't know.

**Host** [43:40]
Okay. Quality-

**Eugene Chia** [43:41]
Uh, my intuition at least is that I suspect it's really the heavy lifting might be actually more towards the Q&A side rather than captioning side. I view captioning as a bootstrap, but at the end of the day, we want the model to generalize what they want to describe.

Captioning tends to overfit what is being described, uh, where our question answers kind of forces the model to pick up details that it would previously ignore, like plots, for example, or, or the one extra finger on that hand, or whatever it is.

And i- as long as you're able to ask questions and it can answer, you're training the model to capture as much information as possible, even if you don't ask. Because even if, let's say, this data set, we didn't ask this question, another data set may ask an, uh, an, an, an alternative question instead, and that will just generalize better.

And that's why I suspect. Like, I think captioning is a problem in overfitting from my point of view.

**Host** [44:37]
That takes... I really don't have that much intuition on this either, though. Um, someone else has hand up. You wanna pop in?

Okay. I'm gonna finish the paper real quick and the takeaways. This is a cool little, like, video we can watch after, and we'll, we'll just save time for Q&A. Um, there was really not much left, though. There's, there's just some highlights.

So they wanted to highlight features. I will share their highlights. Uh, the most efficient model, the 1B, is based on their 1B MoE. That one matches performance of 4V on most, uh, academic benchmarks and their ELO ranking. The 7B and the ...

Wait, so our 7B and Qwen 7B based models comfortably sit between GPT-4V and 4o on both. So 1B, similar to 4V. Um, 7Bs are between 4V and 4o. The big one, the one based on Qwen 72B, is the best benchmarks on academic.

So academic benchmarks-wise, their big one is the best state-of-the-art everything, better than proprietary. But ELO-wise, it sits behind 4o. Um, then our best model outperforms. Yeah, so it's basically best model beats everything. To highlight their potential for action, we tested, um, this on Android control, where it did lower le- I didn't really get this, but we tested it on Android control, where it achieved this low level accuracy, sixty-nine percent high level accuracy.

I don't really know what Android control is, but they highlighted it. I probably should have read more into it, but-

**Guest 3** [46:22]
Uh, excuse me. Do you plan to show us something, or you're just pointing?

**Host** [46:27]
No.

**Guest 3** [46:27]
'Cause we only see the, uh, Molmo page.

**Host** [46:31]
Good catch. My bad. I went back to the slides. Uh, my bad. But yeah, I was just going over these, um, highlights. So highlights was, um, 1B is based on that. It's at 4V. The 7Bs are between the two OpenAI models.

72B is better than anything on their chart, so it's state-of-the-art, better than proprietary. And then there's this last little one, or, yeah, there's this last little point here of it does really good on this Android control, which, no, I'm not gonna lie, I don't know what Android control is, but they wanted to highlight it, so I've highlighted it.

Um, I think that's most of the paper. There's a release plan here. So, um, yeah, LLaMA was coming out. They put it out early. Um, they've put out some stuff. There's more stuff coming, so this technical report is getting an update soon.

The data sets are coming out, but putting out data sets is hard, so give them time. Uh, model weights. Training and eval code. This is a fun one. This typically doesn't come out because you're probably not doing training on the same inference, the same hardware they're training on, but always cool to see training and, uh, eval code.

I guess that's pretty good overview of the paper. Um, if there's anything we wanna dive deeper into, we've got ten minutes. Nathan, if you wanna add anything, um, cook. Go over.

**Guest 2** [48:02]
I just have a question, but, like, I'm ready to get the section about Whisper 3 Turbo if there are, like, no other questions.

**Guest 3** [48:10]
Nathan left.

Yeah, Nathan had to drop.

**Host** [48:20]
Oh, he dropped. Okay, cool. Well, um, that's the paper. Um, there is a little use case thing to just make it useful. We can kind of watch this real quick. Um, and then we've got a little update on Whisper.

So this is their video they put out.

**Guest 2** [48:40]
Is this option vegan?

**Eugene Chia** [48:41]
No, it's not option vegan.

**Host** [48:42]
It's very Apple Intel-like.

**Guest 2** [48:51]
Count the number of people.

**Eugene Chia** [48:54]
Counting the number of people shows a total of twenty-one.

**Host** [48:59]
Very cool. You asked, "Convert this table to JSON."

Here's the table converted to JSON format.

I want to sell my bike on Craigslist. Write me a description for it. Schwinn bike for sale. Blue with white accent. Three hundred dollars or best offer.

**Host** [49:21]
Okay. That's just a, you know, example of what the model can do. I recommend people check out the rest of their blog post. It's, it's pretty cool. Um, here's benchmarks. There's a couple other videos if people are interested, and then there's just a good little TLDR of the paper.

Uh, I wanna save time for questions and stuff, so I'll, I'll share the link. We'll share it in Discord. If anyone has last questions on the paper, we can go over now. Otherwise, we have a little Whisper update too.

Whisper 3 Turbo dropped. I was very impressed. Um, yeah. Questions, or should we pass to Whisper? Last shot.

All right.

**Host 2** [50:08]
Okay. Nice.

**Host** [50:08]
Whisper it is then.

### Whisper Turbo

**Host 2** [50:09]
Whisper, let's go.

**Host** [50:11]
Whisper. Sorry I took up way more time.

Whisper? Who's covering Whisper?

**Host 2** [50:26]
Uh, it's supposed to be Amgadoz, but, uh... Oh, there he is.

**Amgadoz** [50:30]
Yes.

**Host** [50:31]
Okay.

**Amgadoz** [50:32]
Yes. Can you guys hear me?

**Host** [50:33]
Yes.

**Amgadoz** [50:35]
Okay, great. Well, thank you so much for covering the, uh, Molmo paper alongside Nathan. That was great. Uh, so yeah, let's, let's get started quickly.

So can you guys see my screen?

**Host** [50:52]
Yes.

**Amgadoz** [50:55]
Okay, great. Uh, so just like some updates, uh, OpenAI has recently released a new checkpoint or version of Whisper called Whisper Large V3 Turbo. Uh, it's supposed to be like more fa- faster and more efficient, uh, with like very little accuracy degradation, and it was inspired by the work behind DistilWhisper.

So this is like the pull request, uh, then that John Work merged into the OpenAI repository.

So yeah, a little reminder about Whisper. Whisper is like the state-of-the-art ASR model for like transcription, translation, and speech detection as well. The architecture is basically based on the encoder-decoder transformer architecture that was presented in the original Attention Is All You Need paper.

So the input is passed to the encoder, which processes the audio and then gives out hidden states at the end, and these hidden states are sent to the decoder, which it uses to like generate, uh, the text one token at a time.

Uh, so yeah, the encoder and decoder mostly have the same number of layers. Uh, for example, in those for Large V3, it has thirty-two layers in the encoder and thirty-two layers in the decoder. So this was like a quick summary about the architecture of Whisper itself.

Uh, Whisper, like when it was originally released, it had different versions. It had tiny, base, small, medium, and large. And then later on they added Large V2 and Large V3. But for wh- for each of these versions, like the number of layers in the d- in the decoder and the encoder were the same.

So for example, base has six layers in the encoder and six layers in the decoder. Uh, yeah. So the motivation behind Whisper Turbo was actually the same motivation behind DistilWhisper. And the motivation behind DistilWhisper was like based on two observations.

The first one is like the decoder accounts for almost ninety percent of the latency when you are trying to transcribe any audio. Uh, the second observation is that the decoder performs the simpler task of like mapping the hidden states of the audio into the text.

Like the most difficult part is actually extracting these hidden states of the audio and trying to understand what's being said in there. Uh, so yeah, the first observation is, is because, uh, Whisper is like an autoregressive language model.

It generates tokens one token at a time, and you need the token at time T to generate the token at time T plus one. So it is actually a very serial process, and we cannot utilize modern GPUs to parallelize this.

The second observation was made by actually comparing the different vers- versions of Whisper. So for example, the small and tiny versions could transcr-cribe audio when it is loud and clear, and you can actually hear what's being said. But they would struggle with audios that have, have like background noise or where the speaker is not very, very clear.

However, a- all models or all versions, whether be- whether be it like small, tiny, large, they all generate coherent text in English and all the other languages. And this kind of like gives a hint that they are actually capable of like modeling the language, but they just struggle with the audio sometimes.

So yeah. What is exactly Whisper Turbo? Whisper Turbo is like a, a new and more efficient version of the original model with like minimal degradation accuracy. It uses mainly two techniques to satisfy this improvement in, in, in speed and in compute.

### Pruning & Pre-training

**Amgadoz** [54:26]
The first one is like model pruning, and the second one is continued pre-training. But it does not use distillation, which is, I think, a very big misconception about this new release. Uh, so let's, let's talk about model pruning.

Pruning in machine learning mainly refers to like, uh, eliminating some, some parameters from a neural-- from a neural network, uh, to like reduce the compute and memory requirements of a model without impacting the accuracy as much. In, in Whisper Turbo, the pruning was done only in the decoder and like entire layers were pruned.

So we went from thirty-two decoder layers, which were in the original Large V3, to like just four layers. And this resulted in, like, massive reduction in the me-- in the model size so that the Turbo model is, like, one point seven eight times smaller than the original large V3.

Uh, so yeah, after you prune a model and you go from thirty-two layers into four layers, you probably will miss all the capabilities of the model. Like, the model will probably degrade and collapse and will not be able to even generate coherent text.

You'll have, like, to train these four layers to work together again as a single decoder. So the way to do this is to actually do continued pre-training. And in continued pre-training, you train the model on, like, a relatively big amount of data to actually teach it again all the things that it has forgotten or, like, got confused about.

And i-in the case of Whisper Turbo, the, the continued pre-training happened on the same data set that was actually used in the original training of Whisper Large V3. So the... for reference, the data set contained of two epochs.

Uh, each epoch had five million hours. One million out, uh, one million out of these were, like, weakly supervised as in the original Whisper, and four million were, like, pseudo-labeled by the Whisper Large V2 model. So we had, like, five million hours in each epoch.

This means that we have ten million hours in two epochs. So this was, like, the size of the original training data. And the c- the data used in continued pre-training is, is very similar to this, except we're only using the transcription data.

We're not using any of the translation data in the, in the Whisper Turbo pre... uh, continued pre-training. Uh, so yeah, this gives us a hint about, like, the size of the training data of the new model, which is actually quite large.

And some of the fine... some of the fun tips is, like, we used a learn, like, a linear learning rate with the TK starting from five e to negative five.

So, uh, also another confusion is, like, it gets compared a lot with DistilWhisper, uh, but they, like, differ in size and also the tuning strategy. Uh, DistilWhisper was trained using knowledge distillation of, of ASR models. But we, we'll talk about this, like, in a, in a minute or two.

Let's just go over a quick comparison between the two models. So they both ha- they both have the same number of encoder layers, which is thirty-two, but the Whisper Turbo has four decoder layers, while Distil has o- has only two.

The tuning strategy for Whisper Turbo is pruning and continued pre-training, but for Dis- DistilWhisper it's actually knowledge distillation. And for Whisper Turbo, the training data is multilingual. For, for DistilWhisper it's only English, and the task for DistilWhisper is transcription and the same for Whisper Turbo.

**Host** [57:55]
Can I ask a quick question?

**Amgadoz** [57:57]
So... Sure.

### Distillation vs Pruning

**Host** [57:58]
The difference with Whisper Turbo and DistilWhisper is primarily the pruning versus distillation. It seems like we're giving a lot of benefit towards pruning, but the other big distinction is also the difference in data, right? I'm curious on your intuition because I, I also assume pruning versus distillation is a big part of it, and that's why it can stay...

It, it's interesting that it can stay multilingual. I, I would assume the opposite. But, um, how much of this do you think is based on the strategy versus just better data? DistilWhisper came out a while ago, um, versus Whisper Turbo.

OpenAI does good data. Like, how much of the difference do you think is caused... Because... So they're both similar size, but Whisper Turbo is a lot better, right? So how, how much of that better, like, performance at a smaller size comes from pruning versus data?

Because they don't really mention how good this data is. Do you have intuition on this?

**Amgadoz** [58:58]
Uh, yes. Yes. So we kind of, like, have some info about the data set. For, for Whisper Turbo, it uses exactly the same data set for Whisper Large V3, except for the translation section which is, uh, excluded. So this is like a very big data set of like ten mil- or like a few million hours, and it has like many, many different languages.

So this is a very diverse and large and, and robust data set, which is actually quite important. On the other hand, DistilWhisper uses, uh, English-only data. I think it is a few thousand hours of audio. So this is like much, much, much smaller than the Whisper Turbo.

Uh, o-one, one upside is that the DistilWhisper is trying to use actually higher quality data, arguably, but we're not sure about this. So I think there is, like, a very big emphasis about the data size as well, and alongside the training strategy.

For Whi- Whisper Turbo it's a few million hours. For DistilWhisper it's just a few thousand hours, and this is quite crucial I think. For example, DistilWhisper is only for English. You cannot use it like for French, German, Spanish or Arabic or any other language.

And I, I believe, like, if you train a single model on m- on multiple data sets, multiple domains, domains, multiple languages, it becomes more ro-robust, and this is, like, the premise behind, uh, Whisper. Like, it is a robu-robust model because it, it is trained on a very diverse data set, uh, encompassing different domains and languages.

**Host** [1:00:25]
Got it.

**Amgadoz** [1:00:25]
So I hope this kind of answers your question.

**Host** [1:00:28]
Yeah, yeah. Really, really helps with the intuition there. Um, sorry to cut off. Also great slides, by the way.

**Amgadoz** [1:00:34]
Thanks. Uh-

**Host** [1:00:36]
Yeah, amazing slides.

**Amgadoz** [1:00:38]
Uh, so... Oh, thank you. Uh, this one is g- you're gonna like this one a lot more. Uh, so there was a question, uh, from, uh, Swix about, like, how do you actually do distillation for an ASR model?

And I think it's kind of, like, similar to how you distill a language model. So let's say you wanna distill Llama 405B to Llama 3B. The way you do th- do this is you try to, like, extract or compress the knowledge from the big model to the smaller model, and this is exactly what you are doing here.

Uh, the technicalities is the way you train Whisper is a language model that is conditioned on the audio. So it's actually like a language model that predicts the next token So you can have like cross entr- cross entropy loss on like the next token.

And, and you only... Like the difference is, is the input also contains audio, not just text. So, uh, you kind of like train the smaller model on the next token predicted by the bigger model. So kind of like synthetic data or like pseudo data, uh, pseudo label data to be precise.

That's one of the training objectives. The other training objective is, is you're trying to like, uh, make the di- the output distribution of the smaller model as close as possible to the output distribution of the bigger model. So not just the next token, but like the entire distribution of the next token.

Uh, so, uh, if you're training like a language model in a conventional way, you're only like training it on the next token. You only care about the correct next token. You don't care about the top nine or top 10 predictions.

But if you wanna train or like gonna do knowledge distillation and you want to like train the model a lot of information, you might want to do something called KL divergence, uh, training. And in this way, you, you try to teach the smaller model to approximate the probability distribution of the bigger model.

So not just get the n- the next token right, get the top 10 predictions for the next token correctly. And this actually helps the model learn a lot more. You've, you've given it more, much more information for each, uh, data point than just next token prediction.

So this is like how you do a knowledge distillation for an ASR model in like a very brief overview. The Distil Whisper model has a good paper from the Hugging Face team and goes into much more detail about this.

Uh, so yeah.

**Host** [1:02:52]
There, there's another paper that came out recently that talks about distillation and the three types. So there's, there's also distillation where you do pure SFT on outputs and you match input output type distillation, but there's a distillation loss, which is basically this KL, um, KL divergence that you're talking about where you map a whole bunch of outputs.

Um, we, we did have a question in the chat around it about how this is basically token level decoding of audio chunks, right? So, uh, Swix asked, "Is Whisper Turbo fast enough to be real time? Uh, and what's the definition of real time and how does real time work with chunk-based decoding?"

Is something asked if you have intuition around this.

### Real-Time & Benchmarks

**Host 2** [1:03:43]
Uh, I think he mu-- I, I mu- I muted the audio and I think he's gotta unmute. Amgadoz, you have to unmute.

**Host** [1:03:50]
Ugh.

**Host 2** [1:03:52]
Unmute.

**Amgadoz** [1:03:59]
Yeah, I think I got muted. Uh, can you guys hear me right now?

**Host 2** [1:04:02]
Yeah, yeah, yeah, you're good.

**Host** [1:04:03]
Did you get the question?

**Amgadoz** [1:04:04]
Okay. Yes, yes, I got the question. So the, the concept of, of like real time is kind of like debatable, but if you ask me if, if you can transcribe a one minute audio in less than a minute, I think this is kind of like real time.

Uh, so, uh, the question is like, is the, the, the Whisper model fast enough to be real time? If you... The answer is yes, if you run it on the right hardware. If you run it on a T4 or like anything that's bigger, for any modern GPU basically, it's gonna be almost real time.

Like the latency is gonna be like a few, like maybe a second or two max if you've got the parameters correctly and if you size it correctly. And this is even applicable with like the large V2 or large V3, and it's gonna be even faster with like Distil Whisper or like Whisper Turbo.

So yes, I think, uh, Whisper can actually be very, very good at real time kind of, uh, transcription. And-

**Host** [1:04:55]
And how do you do like live transcription with token or chunk-based decoding?

**Amgadoz** [1:05:02]
Uh, so one of the ways to do r- real time transcription, uh, is to select, uh, a small chunk size. To just... B- before answering this question, like one of the key things about Whisper is that you have to give it 30 seconds of audio.

Even if, if the audio is one second, like you have to pad the remaining 29 seconds. So you're always giving it 30 seconds of audio that will be processed by the encoder. And then depending on actually how, how, how long your speech is, it will maybe generate two tokens or 200 tokens.

But if you're giving it o- one second speech and 29 seconds of padding, it will probably generate only two or three tokens or maybe five tokens, not, not 200. So you're spending the same time on the encoder, but you're spending significantly less time on the decoder.

So one way to, to utilize this in, in real time decoding is, uh, continuously give Whisper maybe th- three, 300 milliseconds of audio and give it as the input and pad it, and then ask it to generate like the, the five or 10 tokens, or maybe even less that correspond to 300 milliseconds.

Because probably in, in half a second, someone is, is saying two or three words, which is four or five tokens, which can be decoded relatively quickly and get back the answer in almost real time. So you keep doing this, uh, multiple, multiple times until you run into your first, uh, batch of chunks, and then you can just shift the audio buffer a bit.

It's kind of like difficult to explain without having a graph beforehand, but it's basically about, uh, continuously filling the model, uh, chunks that are like ha- have been shifted by maybe half a second or so. And then you just, if, if, if the buffer is less than 30 second, seconds, you just, uh, pad it and give it to the encoder.

**Host** [1:06:49]
Got it.

**Amgadoz** [1:06:49]
Uh, does this give some clarification?

**Host** [1:06:51]
Yeah, yeah. Very useful intuition. I didn't realize all the padding and stuff. Uh, someone asked if these chunks overlap.

**Amgadoz** [1:06:58]
You can make them overlap. You can make them not overlap. Uh, this is kind of like there's like a who- a whole set of hyperparameters that you can play with. You can use VED. You can not use VED.

You can just keep adding the chunks without overlapping. You can make them overlap. There's like a whole world of doing this.

**Host 2** [1:07:17]
Got it. Um, no, really, really useful intuition. Any other questions, thoughts? This is the last slide, right? We didn't cut off?

**Amgadoz** [1:07:25]
Uh, like, we have two other slides-

**Host 2** [1:07:27]
Oh.

**Amgadoz** [1:07:27]
But this is, like, the main thing.

**Host 2** [1:07:29]
Okay. If you wanna keep going through, go through.

**Amgadoz** [1:07:32]
Sure. So some fine-tuning, uh, Whisper Turbo can be fine-tuned just like any of the other models. I asked, uh, John Walk about which hyperparameters to use. He said you can start with, like, one e negative five as learning rate, but you should do a learning rate sweep and, like, do hyperparameter tuning.

But it can be, like, fine-tuned just like any other model. Just be careful, like, it's probably gonna be fine-tunable only on, like, transcription. Uh, don't try too much with translation or maybe give it a try, but it's not guaranteed to work.

So yeah, it's, it's fine-tunable just like any other model. Uh, the, the final two slides about, like, benchmarks. But just before the benchmarks, people also asked how, how it compares to Faster Whisper and, like, DistilWhisper. So we've answered the question about DistilWhisper.

But for Faster Whisper, like, this is, like, like, a valid question, but because Faster Whisper is like an inference engine, engine. It's like a library. It's, it's, it's code. It can be used to deploy any of the o- the versions of Whisper.

So like Whisper Small, Tiny, Base, Medium, Large, Large V2, Large V3, Large V3 Turbo, Distil Large and, and so on. So kind of like try to make a difference between the inference engine and the ver- the model versions themselves.

Uh, so any question before we go into the benchmarks?

**Host 2** [1:09:00]
No, I guess we can go.

**Amgadoz** [1:09:03]
S- sure. So, uh, uh, for benchmar- for benchmarks, I ha- highly recommend that you benchmark on your own data. Uh, so I think this is quite important. But to, like, give some intuition about the model's performance, I choose...

I chose three data points that I think are very interesting and the... they're, like, very important for me. So one of them is, like, the GPU mode LLM talk by Karpathy. This is very recent, so there is no way it was in the training data, hopefully.

Uh, the second one is, like, a, a chat with, like, Mistral CEO, Arthur Mensch. He is a French guy, so he's speaking English with kind of like an accent. This was, I think, in December last year. Hopefully it was not in the training data as well.

And the final one is my favorite, like, the State of GPT by Karpathy in, I think, twenty twenty-three as well. Uh, so yeah, Karpathy is known to be talking re- relatively fast, so this can, can be challenging. So yeah, let's, let's jump into the benchmarks.

Uh, for, for benchmarking ASR, you generally look at two metrics, WER and CER. WER is word error rate, like how often the model is mis- mistakes... makes a mistake about a certain word in the text. So for the WER, I benchmark the different sizes or versions with, like, different, uh, uh,

different, like, the different data points we discussed. So we kind of, like, see that, uh, the Distil V2 is kind of like an outlier. It's, it's one of the least accurate models out there. Uh, but the other models are, like, kind of similar except maybe Large V3 can be a hit or miss.

So sometimes it, it has relatively higher WER, sometimes it's the best performing model. And this also, uh, aligns or, like, is aligned with, like, what people have experienced it and have shared, like, that Large V3 can sometimes hallucinate.

It can be caught in an endless repetition loop and can make the output, like, really bad. Uh, but yeah, I think that the Turbo model on these data points is doing very, very well, uh, as well as the Distil Large V3.

But on, on other bench- benchmarks that I did for, like, our projects that I cannot share the results, but I can give hints that it was not very good. It was doing much worse than the other models. So yeah, do your own benchmark, and especially if the data set is not English and if it's not transcription, you might see, like, very different numbers from this.

So this is like the WER. The CER is very similar as well. The Distil Large V2 is doing very, very bad compared to the others, and Large V3 can be hit or miss. So yeah, that was, like, a very quick, super quick introduction about Whisper Turbo.

This is, like, the end. If you have any questions, please go ahead. Uh, and thank you for watching.

**Host 2** [1:11:57]
That was really great. Clap, clap, clap. Applause all around.

Klaus says, "No MLX stuff." What do you mean, MLX?

**Amgadoz** [1:12:08]
Uh, no MLX stuff. So I don't have a Mac, so I don't care about MLX. Sorry, but, uh- ... yeah.

That's like the two answers. So yeah, just, uh, as a hint, I think MLX is, like, a framework for deploying or, like, using machine learning models on Apple Silicon hardware, and presumably it's, it's very fast and very efficient.

But I don't have a Mac, so no, no data points from my side.

**Host 2** [1:12:39]
All right. My battery's dying. I'm on 1%, so I gotta end this soon. But thank you so much, Nelson. Fan- fantastic. Uh, always ex- ex- excited by your Whisper updates and implementations. Really, really, like-

**Amgadoz** [1:12:50]
Thanks

**Host 2** [1:12:51]
... those slides, you should share those slides. Um, awesome. Awesome. Thanks everyone.

**Amgadoz** [1:12:55]
Sure. Sure. Thank you, guys. Have a nice day.

**Host 2** [1:12:57]
Thank you, everyone. Bye.

---

This library is powered by PodHood (https://podhood.com), the podcast website platform.