⚡️ Reverse Engineering OpenAI's Training Data — Pratyush Maini, Datology

Hi, welcome, uh, to the Lane Space, uh, Lightning Pod. Uh, we have Pratyush here from Datology, one of the founding team members. Listeners to the pod will remember our episode with Ari Motos. I think it was one of the top episodes of twenty twenty-five.

Uh, w- so I wanna double down on, uh, the interesting work coming on Datology. I think not enough people are looking at the data, and we want to encourage engineers and researchers to think more deeply about data, and I think Datology is one of the best teams that is, like, basically all data nerds.

Uh, Pratyush, uh, y- you- your, your bio has, like, seventy percent California, twenty percent Pittsburgh, ten percent Delhi. You're in South Bay right now, right, with the, at, at the office?

That's correct. I'm at the office.

Yeah. Cool, cool. How, how else do you introduce yourself? Uh, any, any other things that people should know about you before we get started?

No, uh, that was pretty good. Thank you so much for having me. Um, I am, um, I'm basically finishing up my PhD at CMU and have been a part of the founding team at Datology for more than two years now.

It's been a while that we've been building this company. I am very excited about data centric aspects to AI and more of how we can build responsible and very high performance systems by focusing on the data layer. Somehow data is still one of the more undervalued research topics, and, um, I've been...

that's been the focus of my PhD and now also a lot of the time at Datology. I think you very well pointed that we're all data nerds. We love looking at data in general. I think one of the coolest parts about looking at data is, like, we can share, like, strange artifacts about it.

In fact, we even made a channel called Data Is Weird on our Slack channel where we keep posting screenshots and snippets of, like, all of these random artifacts that you see within datasets, and they'll keep surprising you. The internet is so weird in so many ways that you would never expect that, oh, the depths of the internet has this, and we are actually feeding these kinds of data points to our models.

So I think there's, like, a shared passion about really looking into data and, like, an investigative approach around the team in general.

Yeah. Okay. Two follow-up questions, and then we can go into the paper. Uh, one, what, what's the, what's your thesis gonna be about? What's the overall theme of your work? Uh, and it-

My thesis is, uh, called Responsible and Efficient Use of Web Scale Data for Pre-training, which is basically when you're training on this messy web data and you shove all of that into the model, how can we train efficiently, which is like only use things that are useful, but then also the responsible angle to it, which is like how can we make sure we are attributing data to, like, giving the right credit to, like, artists who put data online, or how can we make sure the data is safe for models or model behaviors.

So I do a lot of work on the evaluation aspects of this as well, uh, but then also on the efficiency and performance aspects.

Okay. And then, uh, just in your Data Is Weird channel, any, uh, fun anecdotes over the last... recent anecdotes? Just, just pull something out. Now, we're going to go into, like, the GPT training data stuff, but I'm just kind of curious what, what gets posted there.

So some of the cool things is all from, like, researcher curiosity. I think one of the recent things that we were exploring was that how, uh, do various models, like for instance, you take the Qwen model and then ask it to do completions on certain questions.

So we would just take like... we took some exa- example questions from JEE, which is like an important Olympiad, uh, like an exam in India. And we would like even ask, seed the models with the first two words of a question, like the light, the light bulb, and it would like literally complete the actual questions.

And so then we started comparing this across like family of models, and we consistently saw how many of these like models today are being like massively like kind of fine-tuned on problems from-

Like overfit?

Massively overfit. Like even like imagine if I ask you the light bulb, what comes next in your mind? It won't be a question from an Olympiad or a practice exam, right? And these give the right question. They also give the options after that.

And so I think that is one of the fascinating findings from, uh, Data Is Weird from the last month.

But no- nobody's benchmarking on JEE. That's not a, that's not the thing that people rewards.

There's a JEE bench now, and I think it's also been-

Oh, okay

... repeated by, uh, various, uh, frontier model labs. But like JEE is like one example. I'm sure this happens like across, um, different exams. But it's very clear how the last stage of training for many of these models does have a massive amount of, um, example or exam-

Benchmarking

... example, because the model will not like behaviorally complete exam questions with options if they have not really seen it at the end of training for multiple epochs.

And you're saying there's perf-- there's basically perfect memorization, and I'm wondering if you have ablated this across different model sizes. Like does Qwen be- do the same as 30B? Probably not, right? And I'm always curious about the memorization power.

That's a great question. I think it's a mix of two things. One is the recency of seeing that data, and the second is also the size of this model. I don't-- We don't see this phenomenon in like the earlier versions of like Qwen 1.5 or even Qwen 2, but start seeing it in Qwen 3.

So there's something about a more, like a significantly higher weight on benchmark or like, uh, JEE or like any types of evaluation questions during the final stages of training in these recent models. And then definitely see, we see the effect of like bigger models having memorization to a significantly higher level.

Though I would say that with the coming of the MoE models, we are starting to see these, uh, phenomena at even like models which have a less number of active parameters. So a phenomena that we would observe at a 72B model in the past is probably now visible at a 20B active, 120B MoE also.

So in some way, the active parameters might be lesser in the MoE, um, life cycle, but still be able to like regurgitate exact memorization.

Yeah. It's i- interesting when memorization is done in the MoE, like the router at the MoE actually ends up becoming like an index where you look up like, "Oh, here's the index of the thing that you need," and then it's routed to that expert that is gonna have memorized the answer.

Exactly.

Uh, that's, that's a way to cheat. Okay, so let's, uh, tell us, tell us about the seahorse emoji and GPT.

So around October last year, I saw on Twitter a few people had posted this very fascinating result where they would ask a frontier model such as GPT, like, uh, is there a seahorse emoji? And like that's the only question they would ask.

And what you would see is that the model keeps going on and on. And this was a phenomena that really fascinated me. Like, imagine asking GPT a question, and this, uh, answer does not stop, which was really surprising to me because all of these models like are like...

these are like frontier models. They would not really have this problem of like continually like, uh, going back and forth on a certain answer. If you see over here, it says yes in the beginning, then it starts to think more about it, and then it says no, and then goes yes and no and yes and no.

So there is like a huge amount of internal turmoil in the model and a phenomena where it just keeps saying this and does not really stop. And so then I tried this across models, and I saw consistently across Grok and GPT and Gemini that you were seeing this phenomena where models will like self-correct themselves quite a bit, and these were like non-thinking models.

Like the standard, the auto or the smallest fast model, and they would still do this. And I was like quite perplexed that how is it possible that these models, which are like so strong, they're able to solve these IMO problems and they're like, kind of just getting into this self-narrative existential dilemma or something when you ask them about like is there a seahorse emoji.

And this was also a time when I was thinking a lot about reasoning data and like should like models, uh, learn more self-correction behavior during pre-training or not. And my initial thought was that this behavior that we are seeing of models saying yes and no is very reminiscent of all of these back correction and self-reflection behavior, which is becoming very predominant, um, right, in modern AI models.

Uh, and there are a lot of papers on like if you... the models learn how to correct their wrong traces or like, uh, in their, um, data or in their post-training data, then the models become much stronger at, at reasoning.

And so I was thinking this is very reminiscent of that, and somehow it feels like there is like an emergence of that behavior even when I did not ask for reasoning. So, uh, I thought like it would be very interesting to see like what really causes this behavior, because we totally understand that and maybe there has some tie-in with the prevalence of reasoning data or self-correcting monologues within the mid-training or pre-training phase of non, uh, thinking models.

And so my first thought was, is this something that happened always? What-- or is it just like a GPT 5 phenomena? And because people have been noting this in October and potentially people asked this in GP- in, in like a year ago also, like would the models create this kind of a self-correcting monologue?

And the interesting thing was, and I have like put the number of output tokens on the Y-axis on this graph and the release date of the model on the X-axis. And so I like ran the, uh, OpenAI API across like models released from twenty twenty-three to twenty twenty-five, and you would see like all the models until twenty twenty-four December had very terse and short responses to the question: Is there a seahorse emoji?

They would either say that they exist, sometimes they would say they do not exist, but they would never go into this self-correction loop where they like say one answer in the beginning and then a different answer at the end.

And so something very interesting was happening because we know that in December twenty twenty-four is when the o1 model came, which is the first time where a model that could think and self-correct itself was available to the public.

And then four months after that, we noticed that the GPT 4.1 series is the first model where suddenly this, the response length of all the models had started to increase. And then the ChatGPT 4o model, uh, update that was in the month of May again had like this recursive behavior, and this became significantly more prevalent by the time GPT 5 came.

And so-

And, and, and just to insert a question here, this is specifically like the cutoff date that they have declared themselves, right? Like I don't know if this is, uh, is, is there... This is not... What... Are you ordering by cutoff date or the release date?

Uh, this is ordered by the release date. And so, uh-

Release date.

It's unclear what the cutoff date would be, but very likely that it has seen like exam samples very close to that date.

Got it.

So for instance, ChatGPT 4o is an interesting case. Its original release was somewhere in Decem- uh, like twenty twenty-four, but then they had an update in May, and only the May update shows this behavior, but the previous update does not show it.

And so they kind of potentially post-trained the model on these kinds of reasoning traces, and it certainly started showing the self-reflective behavior. I think it should be important to point out that this seahorse emoji is an interesting question and like why it really brings out this behavior is because the existence of the seahorse emoji is kind of called like a Mandela effect, where a lot of people on Reddit are saying that, "Hey, I've seen the seahorse emoji.

It looked blue in color and had a horse pointing to the left," or something like that. And so there is like enough doubt or like enough conversation on the internet that some people say yes and some people do say no.

And so it's kind of like acting like a trigger to elicit that thinking response, which is great for us because we do want these edge cases that can help us with this discovery or this investigative analysis. So by now, what we have figured out is that around the GPT 4.1 and then GPT 5 series, there is something happening which leads the models to have a lot of self-reflection.

And now the question is like what happened over here, uh, like which led to models starting to do this? And for me, this was interesting because I wanted to know if the frontier model labs also care about putting reasoning data in pre-training or in mid-training phases.

Like you might have seen a lot of papers coming out these days which say that you should do reasoning pre-training and like RL pre-training and all of those papers which are saying that put the thinking tokens in the beginning.

And As academic researchers or like people who are not training frontier models, it's always a curiosity question, are these things really valuable at the frontier level, or are these questions only interesting at the one B scale when you can move your data distribution?

And so that's why I was excited about understanding, like if there is relevant data in the mid-training phase for non-thinking models. And something interesting, uh, that I was able to validate this was with the OLMo series. So the OLMo models are great, uh, for doing investigative science because they release everything.

They release the data, the models, um, and they also have a very fantastic OLMoTrace now. With OLMoTrace, you can trace like what data in the pre-training of OLMo was closest to its response right now.

Yeah, this is... I was thinking of this when you were talking about your PhD thesis. It's like directly the tool to use.

Exactly. Exactly.

Do you, do you agree with their approach?

Yeah, I think like, I honestly, OLMoTrace or the original paper which produced the Infinigram, like I think it came out last year at Colum, uh, 2024. Uh, that was one of my favorite papers that year. I think they did a fantastic job in like being able to like actually trace data back.

It's like a big engineering problem to be able to like look up so fast, and they did like a fantastic job. But I've used it a lot for multiple projects in my PhD. So coming back to this phenomena, uh, and I was like curious like, okay, now we know that this suddenly, uh, happens or like this infection point happens in GPT 4.1 series.

Can we trace it back to an open s-series of models and see if this is like, um, uh, the same phenomena happens there? And so you will see for the OLMo two model, which was released a year ago, like says there exists an emoji, it's the wrong answer, but does not really try to self-correct itself, right?

On the other hand, in the 3.132B series, it has the same self-correction behavior. Like it's a long response, I only have the last part of it, but it first says yes, then says, but wait, there is no one, and then...

So we see the same monologue within the OLMo 3.1 model. And if you look at the OLMo trace, there is no particular example that it is really referencing to over here that leads to this phenomena. So it's more like a capability rather than a regurgitation of the text.

So that's great to know. It's not like someone has poisoned the data. That was another thought in my mind. Has Pliny poi-poisoned the data on the internet and the models are like regurgitating something from there, but that's not happening.

So that's good. That's a good sanity check that what we are seeing is a capability and not like a regurgitation from some website or like Reddit post, which might have said that yes and no. And then the more interesting thing is now we can actually trace the exact difference in the data sets for OLMo two and OLMo three.

And the main difference for the instruct model over here, uh, so OLMo three has multiple variants with some are the thinking variants, some are the instruct variants. So for the instruct variant, they do not have any thinking data in the post-training s- uh, phase.

But they do mention that we are going to put, like there's a line that they write, there is some intentional addition of thinking traces in the mid-training phase of OLMo data, and so that's the information available from their report, and we can also see the actual data sets they use.

This observation really well corroborates with the whole finding about the OpenAI models, where they potentially also did the same thing, where even if in the non-thinking models, the instruct models, where you do not put thinking traces in the post-training phase, you do put thinking traces in the mid-training phase, and that leads to models becoming better.

And so this was like quite interesting for me, uh, because th- what this suggests about the GPT training data is that the self-reflection data has now actually become pretty much core to the training of all frontier models, because we are seeing that happen in non-instruct models across the board.

Which is very interesting to me because for the longest time people would always believe that the idea of foundation models is that you have trained this foundation and now you just post-train it, which meant that there's a general purpose model.

You don't need to really put everything that you care about in, in the general purpose model, and then you can post-train it to get the capability that you want. But this really shows us that even this foundation mod-- foundation needs to have the ingredients that of the capability that you desire at the end.

And so putting thinking traces in the foundation is actually useful for the post-training or f- fine-tuning of, uh, the thinking models. And so that's why they can only have one single backbone, and you would actually prefer to have the foundation also represent the capabilities that you desire.

And so self-reflection is a capability that we desire, and it's become core to the foundation and not left as a cosmetic afterfa- afterthought of just post-training. I think also what's interesting to me that the GPT 4.1 series came about four months after the o1 data was available.

So it's kind of interesting, like how fast do frontier labs move. o1 reasoning traces would have been available to the pre-training team in December, or the mid-training team, I should say, in December. And then from December until, uh, what's like four months from there, end of April, the mid-training team would have potentially used the traces, which they obviously were dis- developing and there was a lot of talk about the Omni models.

But not directly, right? Like it's just went into the, the, the web and the, the pre-training picked it up.

No, I would be very surprised if it went into web. I, I think this was-

Okay

... very intentional because that's exactly how the OLMo series happened, where they have an intentional addition of the thinking, uh, data. So they have added a data set which has this self-reflection behavior where you will, uh, self-correct yourself by first entering a wrong sequence and then the right one.

Yeah. I mean, this is why I wanted to feature this piece because I think like the, the, you know, like how people are training reasoning into their models and when they're choosing to include it and all this, uh, it's very ac- actually very important and not well known.

And so this is one of the first like actual, like somewhat investigative piece, uh, I've seen about it, so.

Yeah, and you, you might have seen all these papers which show that RL is only like kind of enhancing or amplifying a capability that the model already has and you actually need the code to have that capability. So I think this reinforces all of those things that the code actually contain the capability and then you can actually do better RL, you can do better post-training, all of that.

And so I think in general it was very nice to like be able to like see that these are not just things that are relevant to a 3B or a 7B model, but also relevant at the real frontier, uh, where people are training like the best models.

Any reactions, criticisms, or follow-up work?

For now, like, um, in general, I think, um, the community was pretty excited about this. Like a lot of people, uh, felt that this was either reinforcing their beliefs or very interesting investigative work. I think that's something people always appreciate when you just, like, put your detective hat on and try to like dig into, um, whatever is happening.

Something that we are actually following this up, um, with, uh, and also in general we have been working on that for a few months, is a work, uh, which should be out like in a few weeks called The Fine Tuner's Fallacy, which is basically kind of arguing pretty much the same thing, which is that if there is a core capability that you actually care about, that capability should be part of the foundation and not a fine-tuned artifact.

So we kind of show this for various types of domains, and we're actually in general seeing adoption of like AI in a lot of like domain-specific use cases. So the old idea of you have a foundation model, and you can just fine-tune it and get the desired capability is kind of like, uh, we are past that stage now.

And this really means there's gonna be a lot of specialized pre-training going forward. So I think like twenty twenty-six and twenty twenty-seven are going to be the years where different enterprises start doing specialized pre-training because the cost of pre-training really amortizes itself very fast.

When you think of the fact that by doing specialized pre-training, you can train a smaller model which is as capable as a much larger model when fine-tuned. And so that's the core thesis that we have been working towards in the f- past few months and also very relevant to Datology in general.

Okay. Got it. Um, I may need to, uh, go soon because I'm being sort of chased out, I think. But, uh, this is, uh, secretly a Datology pitch I-I'm realizing, you know, because everything you're saying is like exactly the, the thesis of Datology, which is, uh, very fascinating.

You know, you've also done other work. Uh, you are, you are one of the authors on BeyondWeb. Um, I-I don't know, any-anything else you want to sort of plug or, or feature, um, you know, in, in terms of the s- the stuff that's going on?

Yeah. Uh, uh, I can quickly share something about BeyondWeb and how it's been like, uh, making waves. And I also want to talk about the same idea of having the core capability that you care about at pre-training in something called safety pre-training and maybe like a few minutes over there.

So let's go to BeyondWeb first. So this was like one of our big releases last year, where we were trying to showcase how to scale synthetic data to trillion scale. Like works which are like doing like hundred billion token training with synthetic data and two hundred billion training token with synthetic data.

But the actual dynamics of how you work well with synthetic data really change when you go to like a trillion token scale and beyond. And so this work was majorly about like understanding and like giving the lessons back to the community of like how can you do good hybrid model training, where there's like a part of data that's synthetic.

So like, uh, just like as a high level like flagship numbers, this model that we released, these results were very strong. So the Nemotron data set is like one of the top data sets today, which is like based with a lot of synthetic data.

Uh, so the nem... and the Datology, uh, data that we, or model that we released called BeyondWeb, uh, is the blue line over here. As you can see that we achieved the same performance as the NVIDIA model in almost like two point seven x like lesser time, and then much faster than anything that Hugging Face or that Pajama does.

Very interestingly, our three B model is pretty much the same performance as the eight B model of NVIDIA. And so this is quite, uh, like strong given that NVIDIA's Nemotron data is like literally the most downloaded, uh, open source data set.

It's like being downloaded like I think a million times every month, I was just seeing, uh, recently, and so that's huge. So internally, we've been doing a lot to improve, um, uh, synthetic data. So just a little bit of backstory to explain the types of synthetic data approaches that exist today.

So for the longest-- for the beginning of how synthetic data came into picture was through the Tiny Stories and the Phi model family. And so-

Really?

Yeah. So, uh-

That's nice

... so that was like, uh, I think the big push-

Yeah. Textbooks are all you need. Yeah.

The textbooks are all you need. And even before that, there was this paper called Tiny Stories, which was by a couple of researchers at Microsoft on telling like how can you train small models with entirely synthetic data. And so I would bucket these approaches into something that I call the generator-driven paradigm, which means you have a big model.

For instance, it could be the GPT-4 model. And you're trying to query that model to generate a textbook or an essay or a paragraph, but all the information in your data set is coming from the generator or from the model.

And so you really need that the generator is huge and massive and contains information about everything on the world so that you can train models in the generator-driven paradigm. And so a couple of years ago, when I was actually an intern at Apple, we re-released this paper called Rephrasing the Web, and we modeled this alternate paradigm, which is called the source rephrasing paradigm.

And so we said the generator-driven paradigm is exciting and doing well, but it just does not scale. You need the model to be massive, and generating data will be very expensive. Plus, it really depends on you prompting in the right way, because if I ask you to generate a paragraph about the laws of motion, then my data will have it, otherwise it will not.

So it really puts the burden on the researchers to make sure that the data is diverse. But on the other hand, uh, we have the entire corpus of the Internet, which has so much knowledge available. The only th- issue is that the data might not be high quality.

And so we can basically rephrase all of this data into higher quality data. So for instance, the Internet becomes the source of knowledge, and the synthetic data generation model is only modifying that knowledge into styles that you care about.

So for instance, if question answering is something that you care about, then you repurpose the knowledge of the Internet into smaller styles, into, into the question answering styles. So this really trans-- changes the whole, uh, philosophy of generating data by making the cost of synthetic data generation extremely small.

Because now you can use like a very small off-the-shelf model, you can put the actual information inside it, uh, and you can ask it to rephrase it into question answering. And so the capability to transform data is actually very cheap.

Even a 1B model or a 3B model can do this very well. You do not need all the knowledge of the internet to be within this model. And so the source rephrasing paradigm has actually become the dominant paradigm in 2026.

Like, even the Qimi K2 model has like a long section of like how they do rephrasing of internet content. Then the e-even Grok 4 has been using the same source rephrasing paradigm. NVIDIA's Nemotron data that we are benchmarking against is also using this idea of source rephrasing.

And so philosophically, in the pre-training phase, uh, we have kind of almost finalized how we do synthetic data, which is like we transform existing knowledge into patterns that are useful for us. And in BeyondWeb, we really go beyond what we had released at Apple when I was interning over there, which was called Rephrasing the Web, and now we have BeyondWeb, which really like multiplies the advantages of, um, what we were doing in synthetic data work.

For a lightning episode, this is w- fantastic and, uh, it's really good to sort of, uh, get a sense. And I think like I want to also just like use Latent Space to feature this kind of work. People are gonna read the paper on their own.

We're not gonna cover the whole thing, right? So, but if they can reach out to you, I'll leave all the socials and all there. Um, yeah, and, uh, probably join you as well.

Awesome. Yeah, thank you so much for having me and, uh-

Yes

... yeah, it's, uh, very excited to see the response for-

No, thanks for all the great work. Yeah. I, I would say-

Yeah

... like, yeah, I, I, it's, it's, to me it's amazing 'cause like when Red Pajama came out, like I, I realized like that was like the start of something. It's interesting to me that every single generation is a different company, you know?

Like, oh, it was Together AI, and then it's like Microsoft or whatever, and it's Apple, then it's Hugging Face, then it's NVIDIA, now it's you guys, and I'm like, you know, where, where's the, the, the sort of persistence or like is this like such a competitive field or you guys keep changing companies?

So It's the same, it's the same people.

Now we have a hub for all data enthusiasts. I think it's gonna be the same for a while now.

Yeah. Okay. Well, thank you so much. That was great.

Thank you so much. Bye-bye.