# The Unreasonable Effectiveness of Reasoning Distillation: using DeepSeek R1 to beat OpenAI o1 Latent Space · 2025-01-24 Bespoke Labs founders Mahesh, Ryan, and Trung explain how they distilled DeepSeek R1 into Bespoke-Stratos-32B, a reasoning model that beats OpenAI o1-preview on math and code benchmarks using only 17,000 training examples—47 times fewer than DeepSeek's own distill. They achieved this in a 48-hour sprint leveraging their data curation library Curator. Unlike Sky-T1, which required rewriting unreliable QWQ traces, R1's coherent reasoning allowed them to skip re-annotation and simply filter for correctness. The team argues data quality matters more than quantity, and that smaller models (like their 7B variant) can also improve with better teacher data. They see this as evidence that reasoning can emerge purely from supervised fine-tuning on high-quality traces, without complex search algorithms. ## Questions this episode answers ### How did Bespoke Labs distill DeepSeek R1 into a 32B model that beats o1-preview? In a 48-hour sprint, Bespoke Labs used DeepSeek R1 as a teacher to generate reasoning traces on the Sky-T1 dataset of 17,000 math and code problems. After filtering for correct answers, they fine-tuned a Qwen 32B model with supervised fine-tuning, omitting the re-annotation step because R1’s traces were coherent. The resulting Bespoke-Stratos-32B model outperforms Sky-T1 and o1-preview on reasoning benchmarks, achieving near parity with DeepSeek’s own distilled models trained on 47x more data. [1:35](https://addtry.com/697d398e-d2b5-497f-9aa8-300107015345?t=95000) ### What is the difference between logit-level and data-level distillation? Mahesh explains that logit-level distillation, from a 2016 paper by Geoffrey Hinton, has a student model mimic the teacher’s output probabilities (logits) to capture ‘dark knowledge’ like class similarities. In data-level distillation, a large teacher model generates high-quality training data, which the student then fine-tunes on, without necessarily matching logits. Bespoke Labs used data-level distillation, curating R1’s responses to train their model. [4:53](https://addtry.com/697d398e-d2b5-497f-9aa8-300107015345?t=293000) ### Why did Bespoke Labs' 7B reasoning model succeed when Sky-T1's 7B model failed? Ryan attributes the success to swapping the teacher model from QWQ to DeepSeek R1, which produced much more coherent reasoning traces without rambling. Additionally, Bespoke Labs removed the re-annotation step that Sky-T1 used to clean up QWQ’s outputs, preserving the natural reasoning flow. These changes improved data quality sufficiently for the 7B model to show gains, though it still lags behind larger models. [15:13](https://addtry.com/697d398e-d2b5-497f-9aa8-300107015345?t=913000) ## Key moments - **[0:00] Intro** - **[1:24] The Sprint** - [3:17] Bespoke Labs distilled DeepSeek-R1 into Stratos-32B in 48 hours, outperforming o1-preview on reasoning benchmarks. - **[4:23] Distillation** - [4:53] Q: What is distillation? A: Originally mimicking teacher logits' 'dark knowledge', now often using a teacher LLM to generate fine-tuning data, says Mahesh. - **[6:48] Reasoning** - [7:48] Ryan: Distilling reasoning from R1 is surprisingly easy and effective, dubbing it 'the unreasonable effectiveness of reasoning distillation'. - [8:37] Mahesh: R1's reasoning traces perform implicit search and backtracking entirely through autoregressive text generation, eliminating external search algorithms. - **[10:26] Search Debate** - [10:26] Swyx vs Mahesh debate whether OpenAI's o1 uses inference-time search or relies on distillation, with Swyx suggesting o1-mini is distilled. - **[11:45] Generalization** - [11:45] Mahesh: While Bespoke Stratos trains only on reasoning data, DeepSeek's distillations mix 600K reasoning and 200K general instances to retain chat skills. - **[12:57] Data Scale** - [12:57] Bespoke Stratos-32B was trained on just 17K examples—47× fewer than DeepSeek's 800K—yet nearly matches DeepSeek's 32B distill, highlighting data quality over quantity. - [15:13] Trung: Bespoke's 7B model succeeded because DeepSeek-R1 traces are more coherent than QWQ's, which frequently rambled off-topic before addressing the problem. - [16:04] Bespoke removed the data re-annotation step from Sky-T1's pipeline after finding R1's reasoning traces coherent enough without GPT-4o rewriting. - [17:40] Q: Can a student model surpass its teacher? A: Mahesh confirms Bespoke's Minicheck 7B beat the Llama 405B teacher through data curation. - **[17:59] Student > Teacher** - [19:03] Mahesh predicts that RL-only training (as in DeepSeek-R1-zero) and inference-time search will be major research areas in 2025. - **[19:19] Future Directions** - **[22:40] Synthetic Data** - **[23:12] Wrap-up** ## Speakers - **Alessio** (host) - **Swyx** (host) - **Mahesh** (guest) - **Ryan** (guest) - **Trung** (guest) ## Topics Reasoning, Data Tools ## Mentioned Bespoke Labs (company), Bespoke Stratos (product), Curator (product), DataComp (product), DeepSeek-R1 (product), GPT-4o (product), Llama (product), MiniCheck (product), QWQ (product), Qwen (product), Sky-T1 (product), o1 (product) ## Transcript ### Intro **Alessio** [0:00] Hey, everyone. Welcome to the Latent Space Podcast. This is Alessio, partner and CTO at Decibel, and I'm joined by Swyx, founder of Small AI. **Swyx** [0:09] Hello, and this is another one of our, like, really short sort of news-based episodes that we're trying out to do, and it's with Mahesh and team from Bespoke Labs. Uh, welcome. **Mahesh** [0:20] Thank you. **Swyx** [0:21] Hey. **Mahesh** [0:21] Yeah. Thank you, guys. **Swyx** [0:21] Maybe we'll go one by one and identify voices. Uh, so Mahesh, Ryan, and Trung, you wanna go, you wanna go in, like, who you are, what you do. Uh, yeah. **Mahesh** [0:28] Sure. Yes. Hey, guys. Uh, thanks for inviting us. My name is Mahesh, and I'm, uh, co-founder and, uh, CEO of Bespoke Labs, and I'm here with, uh, Ryan and Trung, who are our, uh, founding engineers. Ryan, you wanna go next? **Ryan** [0:41] Yeah, sure. So I'm Ryan, founding engineer at Bespoke. I'm also an AI researcher. We collaborate with this group with DataComp and we-- Yes. **Swyx** [0:49] Is, is that DataComp, is that DataComp LM or the, the image DataComp? **Ryan** [0:52] Yes. So, so I was actually part of the original image DataComp, which was text and images, and then there was one about language models, which was pre-training, and now we're looking into fine-tuning. **Swyx** [1:05] Nice. Yeah, we, uh, I actually interviewed both at the original DataComp and the new DataComp LMT, uh, at NeurIPS, so that was fun. Uh, Trung, go ahead. **Trung** [1:12] Hi, I'm Trung, a founded, a founding engineer at, uh, Bespoke Labs, and yeah, I work on Curator, which is our data curation library. Please star it. And then also DataComp. Yeah. ### The Sprint **Swyx** [1:24] Yeah. So okay, we're, we're meeting to talk about, uh, what happened after DeepSeek's launch. You guys had a crazy 48 hours, is what I understand. **Mahesh** [1:34] Yes. **Swyx** [1:35] Distilling DeepSeek into Qwen and basically beating o1-preview. That's my TLDR. Obviously, there's more to it. Basically, can somebody summarize what happened for someone who's, who only knows about o1 and DeepSeek? **Mahesh** [1:47] Sure, yeah. I think I can give like a-- I can set this up, right? So actually, at Bespoke Labs, we were building this thing called Curator, which makes it easy to curate, uh, data, and a big part of it is like doing, helping with distillation, right? So we literally announced it last Wednesday, and so it, the timing was perfect. So on Monday, when Monday morning DeepSeek, uh, v1 was released, we were like thinking, "Okay, maybe we can use Curator." Uh, we were trying to think through this to say, "Okay, should we do this, not?" SkyT-1 had released a distilled model, uh, which was using QWQ, and we were thinking, "Okay, maybe with DeepSeek-R1 we can also do a distillation." And around 5:00 p.m. is when we were like, "Okay, let's, yeah, let's just start this." And it, yeah, since Curator was, it's, it's like really well-built, so it took us like five minutes or so to set things up and within one and a half hours the data was almost ready. So after that, things were like, "Okay, let's just do this, see this through." And yeah, Ryan and Trung were working effortlessly and like basically putting night outs and we, we started that. We went through and like did the verification of data and started training run that same night and, you know, within like next morning we started to see that the results are starting to look pretty good. Yeah, so, so basically within 48 hours we were able to get the data, verify it, train a model, eval and, uh, you know, announce it, uh, on Wednesday morning. So that's the, uh, background. I, I can talk more. Yeah, anything else to add, Ryan and, uh, Trung? **Ryan** [3:19] I think that pretty much covers it. **Trung** [3:21] Yeah. **Ryan** [3:22] It was, it was a pretty, it was a pretty crazy sprint. Um, it was pretty exciting, right? SkyT-1 was, was really, really interesting 'cause at least personally, I was like, "Okay, these reasoning models are really cool." There's rumors about how they're built. It seems like it's pretty hard. And then it turns out that you can actually just use another existing reasoning model to answer questions, and it generates those really long responses where it's sort of rambling on chain of thought reasoning traces. And then you can just take those and, and use a very basic training procedure, post-training procedure, supervised fine-tuning and take, take an instruct model and then get a pretty decent reasoning model. So that was very exciting, right? We saw, we saw SkyT-1 release that and we were thinking about it. And what we actually have seen is that the annotator for synthetic data is really, really important. So when we saw DeepSeek-R1 came out and you have access to it, we were really excited 'cause now you have this supercharged teacher annotator model that you can just plug right into the SkyT-1 data pipeline, which is phenomenal, and then basically churn out great results. ### Distillation **Alessio** [4:23] Before we jump in, can we just do a little bit of glossary, so to speak? So I, I think like when people say distill model- **Mahesh** [4:31] Mm-hmm **Alessio** [4:31] ... I think people have different understanding. So one is I'm taking one model, I'm making data, I'm retraining another model on the data. Some people are using it for fine-tuning, some people are just using the data, they're not using the logits. Some people are using the logits. Can you just run through, yeah, your explanation of the process and maybe just explain some of these terms so that then people have them in mind? **Mahesh** [4:53] Yeah. I can actually start with the, I think around 2016 or so there was this paper on, uh, distillation. Geoffrey Hinton wrote this paper on. The, the-- There the idea was like, okay, you train one larger model and then look at the logits that it produces and try to have a smaller model mimic those logits. So that's how distillation started is, was on the logit level, right? So there is some sort of dark knowledge that is encoded in distillation or the, in this, uh, teacher model, which is actually helpful for the student model. So as a rough example, you are trying to do a image classification. You can just tell the model, "Okay, this image is a car or a cheetah or something like that." Whereas if when you're doing distillation, what happens is like the teacher model can actually give you some confidence scores, right? Like a car and a truck are somewhat similar, so the logits are correspondingly, you know, there, there is some non-trivial value for truck. Whereas for, uh, a cheetah or something like that, it's, it's zero. So that's dark knowledge. That's how people started doing distillation. And now what-- And, and, and I guess, yeah, sometime back what distillation became is like you have a teacher model, like a much larger LLM That's basically generating data. So that's at the data level, and then you-- once you have the data, you fine-tune. And now, I think also now people are al- starting to see, like, how do we combine both fine-tuning on the data that the teacher produced, but also looking at the actual logics that the teacher was using. So that's the third level. So what we did was the second one, which is just look at the... take the data that the teacher produced, uh, and it's basically a way of curating data, right? A humans could have written those data, but it's just way more expensive and time-consuming. So now we have a very good, uh, teacher or a very good LLM writing it. So that's the-- that's distillation. So that, yeah. ### Reasoning **Swyx** [6:48] And then just to keep digging on that, a lot of this on the reasoning side you can do now because, of course, o1 doesn't show the reasoning traces, so- **Mahesh** [6:57] Yeah **Swyx** [6:57] ... part of it is also that the, the open models... Like, I, I think maybe before the discussion was like, is the data that you get from Llama 3, 4, 5B that much worse than like for... And I think now with o1, you just cannot get reasoning traces at all. So with R1, it's the first time we actually get the reasoning traces in, in open source. And maybe you guys can run people through how that data looks different than traditional chat completions. You know, like any particular work you have to do it to curate it. Like, is the process the same? I know, you know, you have a whole company that is focused on helping people curate data for fine-tuning, so I'm sure you have a lot of stories. **Mahesh** [7:34] And on Trung, do, do you guys want to take this? **Ryan** [7:37] Well, I think just one thing that is interesting you said there is that the o1 reasoning traces aren't available, and I think we're seeing why, right? The title of our release was called The Unreasonable Effectiveness of Reasoning Distillation, right? It's, it's really surprising just how easy it is and how well it works. **Mahesh** [7:52] Mm-hmm. **Ryan** [7:52] So OpenAI doesn't want to give that out for free. In terms of the actual data, yes, the data is very different, right? The, the responses are much, much, much longer, right? We're using much more inference, and also the style of the response is much different. It-- Rather than a, a direct answer, which is easy to read, you, you get these walls of text, right? So there, there is a difference, right? In terms of the sort of things that you can do with the reasoning trace can be more interesting. So you can, you can rewrite it and reformat it in different ways, right? And I think this is something that people are exploring, we're exploring too. And generally, some of the lessons learned from generating synthetic data previously when you're just doing instruction tuning, these lessons can be applied, but there's also new ground here, which is really exciting. **Mahesh** [8:37] And the other thing is, like, other interesting thing... So chain of thought has always been there, but sometimes when you have a tricky problem, what you are doing is you, you want to be able to backtrack and kind of search through the answer. Initially, when o1 came out, people thought, "Okay, maybe there is some fancy search happening where the model, you know, emits one-- some step, and then you go back and so on." But later on, what it turns out is just it's happening just through the reasoning steps, is like the model is, like, saying, taking some steps and then says, "Okay," it's just literally backtracking, right? "Okay, this is probably not right." And in the next step it says, "Okay, we, we should do-- try something else." So it's just autoregressive decoding. It's just many, many, many, many steps of reasoning where there is implicit search and backtracking happening all through- **Ryan** [9:25] Yeah **Mahesh** [9:25] ... those reasoning steps. So that, that's kind of what makes it very powerful. **Trung** [9:29] Yeah. I, I cannot tell you guys how shocking this was when we were discussing this in the company Slack because we thought, you know, we need all these, like, these very complicated search technologies. So in search, you would need, like, kind of like a driver algorithm that does the search, that tells the model to, like, go down these branches and go up and, you know, do all these things. And then the model kind of generates stuff, and then the search algorithm kind of drives that. But what, what's, like, really, really impressive and really, really surprising about R1, you know, as things are coming out, is that you literally just train the model to solve these math problems, and the model automatically learns the search algorithm. I don't know if you, you, you guys have heard of, uh, the bitter lesson, right? There are two things that work in AI, like learning and search, and it turns out you just need learning because you can learn to search. So to me, that was, that was a big re-revelation that all these emergent properties can emerge out of just doing, like, you know, reinforcement learning on outcomes. **Swyx** [10:26] So I want to push back a little bit. ### Search Debate **Trung** [10:28] Yeah. **Swyx** [10:28] Mostly I agree on, on this discussion, especially from the R1 paper. R1 specifically said, "Yes, we tried MCTS. Yes, we tried PRMs, and none of that is useful." **Trung** [10:38] Yes. **Swyx** [10:38] Okay. But there's the question of whether or not the name Q* was a psyop. **Trung** [10:43] Mm-hmm. **Swyx** [10:44] It just, like, threw us off for a whole year- ... because they were like, "Oh my God, they're doing search," because they said Q*. And it turns out they don't do search. Um, I, I think the, I think there might be a different distinction between- **Trung** [10:53] Sure **Swyx** [10:53] ... um, yes, you can have autoregressive, uh, distillation of this, of this reasoning path, including backtracking, uh, for the distilled models, but maybe the full-size models are different, right? Like, the only way you, you reach the frontier with the full-size models- **Trung** [11:08] Yeah **Swyx** [11:08] ... of o1 and o3 is- **Trung** [11:10] Yeah **Swyx** [11:10] ... with that stuff, and then you can distill to the minis- **Trung** [11:12] Yeah **Swyx** [11:13] ... the o1 mini, o3 mini. So in my write-up I said, like, maybe this is the formula for o1 mini, o3 mini. This is why they're cheaper. **Mahesh** [11:19] Right. Right. So, so there is training. You, you can do all sorts of search, uh, during training, meaning at least to generate training data, right? So that is there. And of course, yeah, for o1- **Swyx** [11:30] Yes **Mahesh** [11:30] ... all of that stuff. **Swyx** [11:32] Inference time. Yeah, even o3 inference time- **Mahesh** [11:34] During, yeah **Swyx** [11:34] ... may be, may not be doing search, right? Yeah. That- **Mahesh** [11:36] Yes **Swyx** [11:36] ... this is something that people are speculating. **Mahesh** [11:38] Right. Right. Yeah. There's a... Yes. We, we don't know the answers to this. Yes. That's why it's actually very interesting. Yeah. **Swyx** [11:45] Okay. Yeah. And I, I think the other thing as well, uh, I wanted to discuss with you was generalization. ### Generalization **Mahesh** [11:50] Mm-hmm. **Swyx** [11:51] The idea that a lot of people were thinking about was, "Okay, we can only write verifiers for code and math." **Mahesh** [11:56] Yeah. **Swyx** [11:56] "So this will only be good in code and math." **Mahesh** [11:58] Yeah. **Swyx** [11:58] It seems like at least R1 is generalizing very well. I don't know if... I mean, I haven't looked through your seventeen thousand rows, but I assume it's mostly code and math. Um- **Mahesh** [12:06] Yeah **Swyx** [12:06] ... but are you finding that it's generalizing well? **Mahesh** [12:09] Right. It's-- Yes, this model is trained for those. It's reasoning benchmarks and reasoning tasks, so it's pretty much, yeah, code, math, those kind of questions. I mean, yes, I don't think it'll be a very good model to write poetry, so. **Ryan** [12:22] But no, I think there's... I mean, I, I think that, that you can do that, right? So if you look at how some of DeepSeek's distilled models are trained, they have 800K instances for SFT. 600K of those are reasoning and 200K are not reasoning. So yes, this is a very interesting question. What is the right data mix to retain the general abilities of a chat model? This is not something that we looked at with the Bespoke Stratos 32B release, but this is, is something that we're looking at. **Mahesh** [12:49] Yeah. **Ryan** [12:50] Um, and I think that the answer to this will be interesting for everyone, as DeepSeek did not release their data, unfortunately. **Trung** [12:57] I mean, so I mean, the, the natural question is, actually, so you did 17K because Sky-T1 did 17K, correct? ### Data Scale **Ryan** [13:03] Yeah. **Trung** [13:05] Do you think that you would get the same results if you just went to 800K rows? Or do you think DeepSeek has other, other secret sauce? **Mahesh** [13:14] Well, uh, we-- So that's actively what we are, you know, planning to work on, is like figuring out the data curation strategy to improve these models, right? So our hope is, yeah, we don't have to actually get to 800K, but somewhere in between. So basically, the quality of data matters a lot. That's something that's abundantly clear because compared to Sky-T1, the new data is better because R1 is a better model. We just want to make sure we can curate it in such a way that we can get a very good data, uh, mixture that, that gets us to the level of what they were able to distill or hopefully meet them. **Ryan** [13:48] Yeah. To answer your question, yes. More data should improve here, right? **Mahesh** [13:52] Should improve. **Ryan** [13:52] So this is something that we're looking at. The question is how much will it improve and then- **Trung** [13:57] And how much- **Ryan** [13:58] How much do you need? **Trung** [13:59] Yeah. I'm interested in a chart. You should sort of do an ablation of like every 10K, every 100K. You know, obviously you're gonna, you know- **Ryan** [14:06] We are doing the charts. **Mahesh** [14:07] There'll be a cool chart. There'll be a cool chart. **Ryan** [14:08] Will be a cool chart. **Mahesh** [14:08] The chart is cooking. The chart is cooking. **Trung** [14:10] The chart is coming. **Mahesh** [14:11] The Pareto frontier, right? The, the Pareto curve, like the model size versus... It looks like smaller models need a lot more data, so that's something also that will be interesting. **Trung** [14:19] Yes. Okay. **Ryan** [14:19] I mean, that's actually something that's, that was actually pretty interesting here, right? Is the Sky- the Sky-T1 release, they released a 32B model, and they said actually they did not see improvements on the 7B model, right? So there was a big question of, oh, can you even do this on 7B models with SFT? And then DeepSeek R1 paper came out, and they were-- they had great distills, right? They had even 1.5B models. **Trung** [14:42] 1.5B. **Ryan** [14:43] It was, it was insane, and it was super cool to see, and the answer is obviously like, yes, you can do it. So we also released Bespoke Strato 7B, which improved from the model it was trained on, which was Qwen 2.5B Instruct. Not as much as the 2B model, and obviously did not at the level of performance of the 32B model. But it, it's, it's, it's interesting, and it's inspiring, right? Like, it seems like you can also do this at the smaller scale, and I think that's pretty interesting. **Trung** [15:10] Yeah, agreed. Sorry, what did you do differently that your 7B worked and their 7B didn't? **Ryan** [15:13] Yeah, this is a, this is a, this is a great question. Um, it wa-- it, it seems like the data quality, right? The, the, the, the, the simple thing that we changed is changing out Quick, QWQ, for DeepSeek-R1. So it... the data is just, is just better. We did do some tweaks to the data pipeline, which Trung can talk about more. Maybe those helped significantly, but it seems like just the better teacher is the major factor. We could- **Trung** [15:39] Yeah. **Ryan** [15:39] You know, we can also ablate these decisions as well. **Trung** [15:41] Yeah. **Ryan** [15:41] Trung, why don't, why don't you tell a little bit more? **Trung** [15:44] Yeah, I think like you... we know that the... I, I think we, based on just looking at the traces, we see that the R1 traces are more coherent than the Quick traces. So we can see this because when we look at the traces from the Sky-T1 dataset, a lot of the time the model is like, you know, given an answer about math and it starts like, you know, "Oh, let's think about marine biology. Let's think about like forestry. Let's think about all these other things." And then like finally it gets to the topic of math. In R1, we see that the model behaves much more coherently. One interesting thing, we did a couple changes in the, in the pipeline. One very interesting thing is that in the original Sky-T1 pipeline, they need to rewrite the, they needed to rewrite the reasoning traces from Quick to improve coherence because, you know, the model was rambling so much from what we see. Uh, with R1, I think it's such a high-quality reasoning model that we found that we didn't need to do it when we read the traces, and we just decided to look... We looked at the traces and say, "You know, this looks really good. Like, let's just, let's just do this." And I think like that shows in, in, in the metrics. There are other- **Ryan** [16:46] You, you just removed the step where you use, uh- **Trung** [16:48] We ablated the, the- **Ryan** [16:50] Oh, okay **Trung** [16:50] ... the re-annotation, sorry, of the, the original Sky-T1 pipeline. So, so immediately we, we can see that, yeah- **Ryan** [16:55] Yeah **Trung** [16:56] ... the quality improvement is pretty significant. **Mahesh** [16:58] But, but we did do- **Trung** [16:59] Just based on eyeballing. **Mahesh** [17:00] Yeah. So we, we did do some, uh, filtering, right? **Trung** [17:03] Yes, we... Sorry, yeah. We, we, we... So there, there are two things. Like first, like there's some filtering to make sure, like given a math problem that you have ground true solutions on, you do filtering so that the, the answer provided by R1 or Quick is the correct answer based on the ground true solution. Uh, so that's very basic, like, uh, correctness, uh, rejection sampling, right? We just filter for correctness. The other thing, which I, I wouldn't call it a filtering step, is like the re-annotation step, uh, from the Sky-T1 pipeline, which rewrites the original Quick traces into something that's much more readable and, you know, easier to parse using GPT-4o Viddy. So the, the one that we, we did not ablate this rejection sampling, and we, uh, we've been discussing internally, maybe that's not necessarily, not even necessary given how good R1 is. But we did, the thing that we did ablate, given like the quality of the R1 reasoning traces, is the annotation step. Got it. Well, thank you for doing that. That's actually very, very comprehensive. One short question and a longer question. The short question is, do you ever see a path where the students become smarter than the teacher? ### Student > Teacher **Ryan** [18:09] We, we have. **Mahesh** [18:10] I mean, there is one other example, right? For the Minicheck model, we trained a 7B model and the da- **Swyx** [18:15] Or maybe you wanna, you wanna introduce the Minicheck model- **Mahesh** [18:17] Yes, yes, yes **Swyx** [18:17] ... 'cause I know about it, but- **Mahesh** [18:19] Right. So we, so sometime back in August or so, we released a model called our Bespoke Minicheck MB, and what it does is it, uh, takes... It's in the tech setting. It takes a, a context, s- for example, a paragraph and a claim, and it tells you whether this claim is supported or grounded in this paragraph or not. It's very useful in RAG to prevent hallucination, right? So the main point, yeah, the, the, that's, uh, the main point I'm getting to is, like, for that particular model, it's a 7B model and the data comes from 70B, uh, sorry, 405B, Llama 5B, and we were able to beat the Llama 405B on that particular data. So just by doing some of the data managing, basic- I, I call it inductive prior. So y- you introduce, like, how to organize the data and that will actually help you beat the teacher, right? So I think yes to, short answer, yes, I think we can do it. But of course it depends on case-by-case, uh, basis and like, you know, what kind of settings we are doing, right? So that's, yeah. ### Future Directions **Swyx** [19:19] I mean, I, I think we have to, yeah. Uh, we're getting to this stage where we have to figure out how to make the student smarter than the teacher. And then, okay, like other reasoning work that you are interested in. For me, you know, this was kind of obvious because I read the Star paper, self-taught reasoning, and I mean, like it made sense. And then we also, uh, on this podcast we interviewed Cosine that, that actually fine-tuned 4o, uh, to on SWE-bench, uh, sorry, uh, to, to, to achieve state-of-the-art on SWE-bench and that score was actually higher than o1 when it came out. And that was- **Mahesh** [19:48] Mm **Swyx** [19:49] ... my first clue that, oh, you can actually take autoregressive models and just give reasoning data and it still- **Mahesh** [19:54] That's true **Swyx** [19:54] ... outperforms reasoning models. And, uh, I mean, now we're seeing this come to life, uh, here. But like basically, like related work, like papers you recommend or other things you're watching that you think will be relevant this year. **Mahesh** [20:03] Yeah, I think this great debate about like, uh, so of course o- one is the role of RL, right? Like DeepSeek, they, they in fact said that at least for one of their R10 model, like they didn't even do SFT on it. So that's, that's, that's very interesting, right? To improve reasoning, we, we did like lot of SFT and it does really well. They have a new recipe where they don't do any SFT, just RL. So that's something I think people have to study and like, you know, there is a lot of, uh, work that's going to happen this year. So that, I think that's something that I'm quite excited about. And yeah, in general, the other thing would be like for inference time scaling, how does inference time scaling, you know, like I w- I was definitely under the impression, okay, we will do some more search, uh, during inference time. Maybe it's not happening, so but I think there is still a role for that to be, you know, for, for that to happen. So that's the other thing that I'm quite excited about. **Swyx** [21:00] Anyone, anyone else? **Alessio** [21:01] I mean, we might just, you know, we, we get surprises like every week, so just hold on for the ride, right? **Swyx** [21:07] Hold onto your papers, as they say. **Alessio** [21:09] Yes. **Mahesh** [21:09] Yeah, just- **Swyx** [21:09] Anything else that we should have asked? Anything, anything else we should have asked that you're, you're just excited about or just in, in the, the sort of Bespoke, uh- **Mahesh** [21:17] Right **Swyx** [21:18] ... model that we didn't ask? **Mahesh** [21:20] For, um, no, the, I think just, just the fact that data quality matters a lot, right? So it's, it's actually what, um, our bet is also is like data curation and curating high-quality data. So basically there are three things we need to look into. People have an idea and they want to fine-tune model, but typically they fail in the data curation stage. So that's something that we want really to help people with. And there predominantly you, you want to tackle like quantity of data, get good qu- lot of data and quality and diversity. So there are these three axis of how to think through data curation. So in this work we fixed, I guess, quantity. We improved on the, uh, quality. So there is more work to be done on these different axis. Yeah, the, the shape of product we are building, we are hoping that this actually with the Curator, that the product we are building called Curator hopefully helps people to kind of turn the knobs on all three of these and get good quality data and therefore good quality models also. **Alessio** [22:17] Yeah, I think to underscore that, we're really interested in what data recipes work and building tools that make it really, really easy to use those recipes. **Mahesh** [22:27] Yeah. And, and also just one more thing is, yeah, we, we are now working very closely with the DataComp team and a lot of great researchers there and hoping to produce some good, uh, work, uh, along the r- you know, along these steps now. ### Synthetic Data **Swyx** [22:40] Yeah. And is there, is there any juice left in the synthetic textbooks stuff or is that done already? **Mahesh** [22:45] Oh, lot of stuff. **Swyx** [22:45] Is that, is that last year's, last year's- **Mahesh** [22:48] Oh, synthetic textbooks? **Swyx** [22:49] Yeah. You know, FineWeb Edu. **Mahesh** [22:51] Yeah, yeah. Fi- **Swyx** [22:53] Phi. Yeah. **Mahesh** [22:56] I think, yeah, just the data quality matters, right? So it's just textbook is a form of, you know, data and high-quality data. Uh, I, I don't know. Yeah, we are not looking into that kind of stuff, but you know- **Swyx** [23:07] Okay **Mahesh** [23:08] ... uh, we are building a platform- **Swyx** [23:09] So more reasoning. Yeah. **Mahesh** [23:10] Yeah, re-reasoning. ### Wrap-up **Swyx** [23:12] So DataComp RM coming. Okay. Uh, Alessio- **Alessio** [23:15] All right **Swyx** [23:16] ... any, any, anything else? Yeah. **Alessio** [23:17] No, this was great. I mean, uh, it's amazing to see how since this R1 model came out every- **Mahesh** [23:23] Yep **Alessio** [23:24] ... three hours or something. So thank you guys for, for jumping on, on short notice and, um, yeah, great, great work. **Swyx** [23:31] Yeah. Thank you guys for, uh- **Mahesh** [23:32] Thanks for hosting us. **Swyx** [23:33] Yeah. Thank you guys. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.