# Information Theory for Language Models: Jack Morris

Latent Space · 2025-07-02

<https://addtry.com/75d7fcc3-8b1b-4a74-af57-92dd062e3b6a>

Jack Morris, a Cornell PhD student advised by Sasha Rush, discusses his information-theoretic research on language models, including embedding inversion (recovering text from embeddings with 90% accuracy), the universal geometry of embeddings (aligning different models' latent spaces), and measuring model memorization capacity at 3.6 bits per parameter. He argues that paradigm shifts in AI, from AlexNet to instruction tuning, stem not from new architectures but from new datasets, and that the next breakthrough will likely emerge from a novel data source. The conversation also covers the shift from academia to industry, practical advice for grad students on distributed training, and the implications of embedding inversion for privacy and model alignment.

## Questions this episode answers

### How many bits of information can a language model memorize per parameter?

Jack Morris found that GPT-like transformers trained in 32-bit floating point can store approximately 3.6 to 3.9 bits of information per parameter when memorizing random data. This measurement reveals a substantial inefficiency—less than 12% of the theoretical maximum—suggesting that current architectures and training optimizers are far from optimally compressing training data into weights.

[54:29](https://addtry.com/75d7fcc3-8b1b-4a74-af57-92dd062e3b6a?t=3269000)

### Can text be recovered from text embeddings, and if so, how much?

Jack Morris explains that his research showed it's possible to recover up to 90% of the original text from embeddings of short sentences. Because embedding vectors like OpenAI's contain many dimensions (e.g., 1,536), they can encode significant information. He demonstrated an inversion attack that re-extracts text, highlighting privacy risks for vector databases and prompting some providers to revise their policies.

[23:45](https://addtry.com/75d7fcc3-8b1b-4a74-af57-92dd062e3b6a?t=1425000)

### Do different text embedding models learn similar internal representations?

Jack Morris's work on universal embedding geometry shows that models with different architectures and training objectives converge to highly aligned representation spaces. By adapting a CycleGAN technique, his team mapped between BERT and GPT embeddings without paired data, confirming the Platonic Representation Hypothesis—that models trained on real-world data share a common latent structure, enabling embedding inversion across models.

[31:35](https://addtry.com/75d7fcc3-8b1b-4a74-af57-92dd062e3b6a?t=1895000)

## Key moments

- **[0:00] Intro**
- **[1:12] PhD Journey**
- **[3:54] ChatGPT**
- **[5:07] Embeddings**
  - [5:13] Jack Morris advises grad students to wait for the next paradigm shift and then immediately re-implement it to succeed in academia.
  - [5:54] OpenAI's o1 reasoning model, released September 2024, sparked an explosion of research on data, evals, and training schemes, says Jack Morris.
  - [7:05] At 100M parameters, language models know nothing—they always answer 'George Washington' for any president, while at 8B parameters they know every president, says Jack Morris.
  - [8:36] Academia spent two years working on small models that didn't matter because they lacked emergence, before catching up to 7B-scale models, says Jack Morris.
- **[8:50] GPU Advice**
  - [9:38] Grad students are not taught anything about multi-node distributed training; most only ever use single GPUs, and they must learn from the internet, says Jack Morris.
- **[14:32] Information Theory**
  - [15:18] V‑information: two text files can have identical Shannon information, but one is more usable because the other is encrypted—a theory of information under computational constraints.
  - [18:17] Deep learning lacks a fundamental unit like the 'bit' from information theory; Jack Morris tried to find it, couldn't, and challenges listeners to figure it out.
- **[23:45] Embedding Inversion**
  - [23:45] OpenAI embeddings use 1,536 dimensions, roughly 20 KB of information per vector—many paragraphs of text can be compressed into one embedding, says Jack Morris.
  - [24:23] After many rejected proposals, Jack Morris's advisor Sasha Rush finally approved the idea of reverse-engineering text from embeddings, launching his first major grad-school project.
  - [26:50] Jack Morris's embedding inversion method can recover exactly 90% of the original text from an embedding at long sentence length.
  - [27:21] Researchers used Jack Morris's tool to verify de‑biasing: after removing gender features, 'This woman is a doctor' became 'This person is a doctor.'
  - [28:00] The embedding inversion project was stuck at 30% for months, then jumped to 97%—one of Jack Morris's most rewarding grad-school experiences.
- **[31:08] Universal Geometry**
  - [31:21] The paper 'Harnessing the Universal Geometry of Embeddings' applies the Platonic representation hypothesis to align different embedding models.
  - [33:09] The Platonic Representation Hypothesis posits that all models converge to the same representations as they scale because they are trained on data from the same world.
  - [35:31] Q: How does longer context affect embedding inversion? Jack Morris: longer sequences with fixed-size embeddings cause compression collisions, making inversion impossible beyond a point.
  - [38:23] GPT-style models memorize about 3.6 bits of information per parameter—a stable plateau across scales, found by Jack Morris's team.
  - [46:13] Applying CycleGAN to embeddings, BERT and GPT vectors from different models snap to the same location, supporting the Platonic representation hypothesis.
  - [50:39] Q: Could a 100M-parameter model ever match today's o3 Pro? Jack Morris asks whether we are at the plateau or if future breakthroughs will smash it.
- **[52:14] Model Capacity**
  - [56:19] Transformers trained in 32‑bit can only store 3.6 bits per parameter—roughly 28 bits per parameter are wasted, and we don't understand why, says Jack Morris.
  - [57:21] Q: Is the 3.6‑bits‑per‑param a hard limit? Jack Morris says better architectures could increase it, and better measurement tools may show it's actually higher.
- **[1:01:44] Data Inversion**
  - [1:02:58] Jack Morris's paper uses the weight difference between base and fine-tuned checkpoints to select likely training examples from a web corpus.
- **[1:05:24] Datasets Thesis**
  - [1:06:52] Jack Morris's Substack post 'There Are No New Ideas in AI, Only New Datasets' argues that paradigm shifts come from new data, not just new algorithms.
  - [1:08:15] Jack Morris identifies four AI paradigm shifts: ImageNet, web pre‑training, human preference data, and verification data for reasoning—all driven by new datasets.
  - [1:09:50] The real innovation behind ChatGPT was web‑scale pre‑training, not the transformer architecture—we might have built it with very sophisticated RNNs, says Jack Morris.
  - [1:11:05] Instruction tuning's breakthrough was the human preference data, not reinforcement learning; DPO (direct preference optimization) works with supervised learning.
  - [1:12:04] Jack Morris predicts the fifth AI paradigm shift will come from a new data source, possibly video or embodied AI.
- **[1:14:43] Wrap-up**
  - [1:14:43] Q: Does Sean Wang pass as a grad student? Jack Morris says he passes the 'grad student Turing test'—people would assume he has his own niche research area.
  - [1:16:17] "Substack better than PhD" — Sean Wang jokingly suggests it as the episode title; Jack Morris agrees to "Substack."

## Speakers

- **Sean** (host)
- **Jack Morris** (guest)

## Topics

Language Models, Alignment

## Mentioned

DeepSeek (company), Google (company), Mistral (company), OpenAI (company), BERT (product), CUDA (product), ChatGPT (product), CycleGAN (product), GPT-2 (product), GPT-3 (product), GPT-4 (product), Gemma (product), Hugging Face (product), ImageNet (product), InstructGPT (product), JAX (product), Llama (product), MS MARCO (product), PyTorch (product), o1 (product)

## Transcript

### Intro

**Sean** [0:04]
Hello, this is Late in Space, just Swix today with, uh, our special guest, Jack Morris. I guess from Columbia, that's your affiliation right now?

**Jack Morris** [0:12]
Cornell. It's actually confusing because I go-- I'm in a, the New York City outpost of, of Cornell, so you have the city right. But it's Cornell Tech, which is like a small- ... uh, Cornell campus in New York.

**Sean** [0:29]
I just... You're, you're a student of Sasha Rush, who teaches at Cornell, so I, I should have made that connection. Okay. Yeah, I'm sorry. Um, wow, th-that's, that's a horrible mistake to make right off the bat. But, uh- ...

you're one of... Look, you're one of the, there, there are not that many PhD students that make an impact with their research. The last time someone like this happens was Shunyu from Princeton, and, uh, he joined the OpenAI operator team quite shortly after he graduated.

So, like, you're one of those, like, high-profile PhD students at least that's, like, coming out of the, the program and, like, I figured, like, it was a good time to just, like, talk about your work and also the fact that you're looking for, like, which lab you're, you're gonna join.

That's like a whole interesting meta discussion, especially with, like, the insane market for AI talent these days. What's it like to be a, um, AI grad student these days?

### PhD Journey

**Jack Morris** [1:22]
Yeah, and thanks for having me. I guess maybe we can go back to when things first started or, like, like, put yourself in my shoes. In 2017, 2018, I really learned a lot about machine learning and at my...

I went to a state university. It's a good school, but they didn't have, like, a deep learning research department or anything. They had people doing it, but it was just not as big at that time. But I was getting really interested in those topics, especially as applied to language.

And then in 2019, I kind of s- was starting to do research and I think thinking about my career. I mean, at that point I was 20, 21. I was thinking about, like, where do I wanna be career-wise, or like, who's doing the coolest stuff right now?

Like, looking at, like, what kind of stuff is coming out at that time. I mean, I think AlphaGo, I thought AlphaGo was really good. At that time, I was playing a lot with, like, BERT and BERT-based models. So like, you know, Google, DeepMind, they're doing great work.

GPT-2, GPT-1 from OpenAI were, like, interesting, but I think most people were into BERT at that time. I still have a soft spot for, like, that parameter class of, like, 100 million to one billion scale models. But this is all to say, I think at that time, I felt like the people doing a lot of the most impactful work were, like, professors and PhD students.

Like, just a ton of, like, interesting ideas being explored and cool opportunities in academia. So I ended up applying to grad school. Well, at first, I did this Google AI residency program, which was mostly during the pandemic, like 2020 and then 2021, and then I was also applying to grad school.

Started grad school in 2021. That's still what was going on at that time. Like, a-around when, I guess, GPT-3 one hundred and seventy-five billion had been released, but not InstructGPT. So, like, we had pre-training and sort of the science of pre-training was emerging, but that's where the models were.

And I still think, like, I'm glad that I went to grad school and, like, I had a great experience, but the last five years have changed a lot. Like, the whole meta has shifted, you know? Like, uh, uh, the kind of power dynamics are completely different.

The ideas are coming from different places. Most stuff is open now. Most stuff is not open. Um- ... the types of questions people are asking are different. And so, yeah, I mean, for better or for worse, I did go to do the full grad school thing and, and here I am.

It's been really interesting perspective watching the science kind of emerge with the products. Like, the biggest thing that happened by far was, like, ChatGPT coming out and which was right in the middle, like, what? 2022 before Christmas, like November.

### ChatGPT

**Jack Morris** [4:09]
I remember that year, like, um, like my grandma was asking me about it, and that's when it hit me like, oh, this is actually becoming like a real area-

**Sean** [4:17]
Yeah

**Jack Morris** [4:17]
... that people will know about and understand. Like, I was trying to explain it to my parents, and that's when I think things really started to change in terms of the types of questions you wanted to ask can't always be answered with academic resources, so a lot of the, like, fundamental kind of like boundary pushing and AI science moved into companies.

**Sean** [4:38]
That was the year when like, uh, you know, just around Europe as well, everyone in, in NLP and deep learning were, were, were like very confused at like- I think some people were like kind of expecting this already, uh, in a sense that they had-- they were obviously more clued into large language models.

But I think that the sheer amount of consumer-level interest that had, that, that was around at the time in 2022, that completely changed the world. Like now, now we're just like in a different sphere. Uh, did you have to pivot your research or were you already...

### Embeddings

**Sean** [5:07]
You just went from BERT to, like, other stuff. Uh, you-you've done a lot of embeddings work.

**Jack Morris** [5:13]
I mean, you're always heads down working on a problem, so I don't think most people in academia are the type to say, "Oh, look at this new product that came out. I'm gonna abandon everything I'm doing."

**Sean** [5:25]
That could be the right move, you know.

**Jack Morris** [5:27]
Oh, it definitely can. Honestly, if, if, if I were to give advice to a younger grad student, I think the way to do it would be literally just like sit and wait until the next kind of paradigm shift and then just immediately start working as fast as you can to like re-implement it.

Like I, I don't think that's like maybe the best way to do science, but it's probably the best way to play the sort of academic game in the, in the days of AI. Like you've seen that so many times.

Most recently probably with the reasoning models like o1 came out of OpenAI September 2024 last year. And then there's just been this explosion of like you build like abstraction ladders on top of that. Like first it was re-implementation, like how do we even do this?

And now it's like a lot about the data. What's the right data? What are the right evals? What are the right training schemes? Like there are so many different axes you can test and- ... publish research in. And like, I think the easiest way to do that probably is just work in a field like that, that like has it only existed for less than one year.

**Sean** [6:29]
Yeah.

**Jack Morris** [6:29]
And so no one has any, like, big advantage, I guess.

**Sean** [6:32]
That is mostly correct. I think anyone who jumps on reasoning and RL for LLMs is doing super well. I just saw this morning that one of the recent Stanford grad students who worked on RL, they just started their company and they're worth 500 million.

It's, it's like absolutely bonkers right now. Like just like- ... no product, just three dudes, you know, sitting in some basement somewhere. I mean, undoubtedly cracked, but like also not worth 500 million.

**Jack Morris** [7:00]
Mm-hmm. Yeah, but maybe it's not paying for the product, right? It's like the-

**Sean** [7:05]
The potential

**Jack Morris** [7:05]
... the ideas behind it or the... Yeah. Yeah. There was this big shift from in scale of working with 100 million parameter models. Really what happened is like, I think the companies invested a ton more into training and infra and like we all kind of had to catch up.

Like, you know me, I go to Cornell, work with a professor there. He has to buy GPUs. Like should he buy last year's GPUs or this year's GPUs? How many should he get? That we were kind of like trying to figure that out and there was, there was like a big lag I think where basically the, the 7 and 8 billion parameter scale, like there's a huge difference between the bird size models, which are 125 million parameters to, to-

**Sean** [7:46]
Yeah

**Jack Morris** [7:46]
... 200, and then like the 8 billion parameters. I mean, obviously it's two orders of magnitude, but just like this idea of emergence, like if you're talking to a model that's 100 million parameters, no matter how well it's trained, it knows nothing.

Like if you ask it- ... like what's the capital of a state or like if you ask it who's, who was president of the United States in 1990 or n- whatever, it'll just always say George Washington because it just associates the words like President United States with George Washington.

And then when you get to the 8 billion parameter scale, suddenly it knows every single president. It knows every single capital of every single country. And I really do think that changes the type of research you can do.

And so like it took us a while, I think in academia to catch up, like getting good 7 billion parameter models and then running them and getting GPUs to run them. Now I think things have stabilized a lot.

Like we have access to compute and we can kind of like fine-tune and inference that scale of models and that's like kind of fine. But there was like kind of two years where everyone in academia was working on like smaller models and none of it really mattered.

**Sean** [8:50]
I can sort of branch that discussion in two ways and we should, we should sort of go to your research at some point. But I'm enjoying this because I think like we don't get to talk about this on the podcast too often.

### GPU Advice

**Sean** [8:59]
One is there, there's, there is an often, there's an often bit of advice from the industry people to the grad students, which is give up, don't work on models, just do benchmarks, right? Like a really good benchmarks will, will get our attention and then we'll hire you and then you can switch to models later.

You have, for better or worse, avoided that. Which is cool and we can talk about that as well. But the other thing I think is that round about 7, 8B, maybe 4B is when you start switching from like a single GPU setup to like a distributed setup.

And I'm wondering like do grad students get HPC training? How much do they teach you of like just how to work with like large clusters of stuff?

**Jack Morris** [9:38]
Oh, to be clear, they don't teach you anything. Like anything, like if you see a paper coming out from even, you know, Stanford, they're probably the best school in AI if, if you had to choose, and it's not like they're learning how to do like multi-node distributed FSDP training like with whatever deep speed.

You have to learn that from the internet and from other people and like there's no classes that really do that. I mean, it's... that's hard to facilitate like as one person. I would say most grad students are doing stuff on single GPU.

Some people are doing multi GPU training. There's probably basically no grad students doing multi-node training. I mean, there's probably a few, especially if they have like company affiliations, but that's really unusual I think.

**Sean** [10:25]
Okay. For grad students who are looking to get up to speed on that, I would recommend the GPU mode Discord, where basically the PyTorch team is hanging out in there just waiting to help you. And then the other one would be the Fast.ai team.

If you have some kind of thing, Jeremy Howard will basically help you out and, uh, they, they have some, uh, distributed training. Honestly, try to reach out to the deep speed team at Microsoft. Like actually they're reasonably accessible.

Nobody talks to them. Like it's so, so funny. I, I like met them at NeurIPS- ... and like they had nobody at their... Like they were presenting these speeds. Three-- I was the only one asking questions like, um, yeah.

**Jack Morris** [11:01]
Yeah, that's good, that's good advice. Listen to this guy.

**Sean** [11:04]
Yeah, I mean, it's just basically like the people are there if you wanna ask. This is very, very valuable experience. Once you're like a GPU god, like you're basically, you know, in a, like a different tier as a researcher because you don't rely on someone else helping you out.

Like you can just how to be your own research engineer, you know?

**Jack Morris** [11:21]
Yeah, I'll comment on that quickly because if someone has been listening to this and also following me online for a while, I think I've made a couple comments like saying something like you shouldn't learn about CUDA or things to that nature, and I'll g- I'll give some more color to that.

So it's definitely a great idea to learn CUDA if you can. I think my point was that if you're trying to enter this space, like learn about the models, learn about how they're trained, what the data looks like, what the compute looks like, one axis of that is how to do more efficient training and inference, and one part of doing more efficient training and inference is studying the hardware, which is GPUs.

So like I think that's a very small subset of all possible knowledge that you could acquire, and it's probably not the best place for a lot of people to start. That said, if you do it, you're, you've gotta be one of the most hireable people in the world.

Like if you like really deeply understand the architecture of the new GPUs coming out and, and how to control it, you're in a very small handful of people and like everyone will want to hire you.

**Sean** [12:30]
Actually, the sweet spot is not even CUDA right now. I would say actually it is Mojo. I don't know if you've been paying attention to-

**Jack Morris** [12:36]
Mm

**Sean** [12:36]
... modular Mojo.

**Jack Morris** [12:38]
Oh, I listened to your podcast, man. You had that guy on, uh, uh, the other day

**Sean** [12:42]
The whole story is Chris Lattner, industry legends, LLVM, Swift, all these things, and now he's turned his attention to the Python CUDA relationship, right? And he wants to basically create a viable CUDA replacement. It's basically Python married with Rust.

For the last two and a half years, he was basically kind of stealth, not ready for production. When he came on our podcast, he was basically announcing to the world, like, "We're open for business, like, you can use us now for, for most models, and, like, we actually are faster than, like, the native, like, uh, sometimes the PTX implementation."

I don't know how that works precisely, but he's a compiler languages god. I think there's a n- there's one of those windows now, like, like you said, like, you know, bet early on something that's, there's a, there's a shift.

It's one of those windows now where you try to implement things. You basically like, you know, Modulus is 100 people. If you run into issues, you'll get Chris's personal help on things. Like, I'm not promising it, but, like, probably, you know, like, because he wants to work on improving the toolkit.

And, um, all you have to do is just like... It's not really about becoming a CUDA god, because obviously, like once you ramp up on, on the, the general concepts and principles, you can probably translate ecosystems pretty effectively.

A lot of people switch from like JAX to, to CUDA. But, like, the, the thing is just like being able to experiment very quickly on a limited budget. Like efficiency is not just because you are trying to be an efficiency guru and that's your career and that's kind of boring.

But it's really also just about being able to experiment very quickly, uh-

**Jack Morris** [14:08]
Mm-hmm

**Sean** [14:08]
... and finding these i- these ideas.

**Jack Morris** [14:10]
I also think, uh, VLLM and SGLang seem like really good and important and here to stay. Like they'll probably just get larger and more complex to accommodate future systems. But if I were like, uh, starting out grad student, I-- and working in that area, I'd probably like wanna learn more about how they work.

**Sean** [14:32]
Awesome. Let's go to your research. I like to mention that I, I first came across you because of CDE, the Contextual Document Embeddings paper. You can, uh, tell me the story about that, but I just want to show you proof that, you know, I get one slot per day to highlight the number one AI story, and you were the slot of the day for October fifth.

### Information Theory

**Jack Morris** [14:51]
Oh, no way.

**Sean** [14:53]
I mean, obviously, you were producing work before that, but like, I thought CDE was a really cool exploration of like, oh yeah, you know, embedding models are kind of like stuck in a rut. Like here's actually how to make them very efficient by just doing it in two stages.

That seems like a, you know, relatively simple insight that was done very well. But you have a general maybe information theory thing that maybe we should start with, and then we can sort of grade it and send our way.

**Jack Morris** [15:18]
Yeah, sure. That sounds good. So we can, we can circle back on that. That's, that's really cool that you, uh, wrote about it. What was that? Almost coming up on two years ago. Yeah, this is the post I wrote.

I, I called it A New Type of Information Theory.

**Sean** [15:32]
Okay.

**Jack Morris** [15:32]
We don't need to go into the... There, there's this paper about a concept called V Information. Maybe I'll give like the most simple explanation, which is if you say you have two text files, one text file contains a paragraph of information about New York City, and then the other text file con-contains the same text but encrypted with like SHA whatever encryption algorithm.

So it looks like random letters, but if you decrypt it, it has the same text as the first text file. From the perspective of like Shannon's information theory, these two files contain the same information content, like relative to everything, they have the same number of bits.

But it's, it's very clear to the observer that the first text file, which is plain English text, is like much easier to read and easier to process, even though they have the same information. And so there's this theoretical framework proposed in this paper, which is a theory of usable information under computational constraints from twenty twenty.

It really doesn't have that much press. There not-- aren't as many citations as you would think. But I think it's a really, really neat idea. It's like we should measure information with computational power as a constraint. So like they have this idea, they call V information of how much information is extractable from a given like file or, or code.

So in that case, we could say the left text file actually has more extractable information than the, the right text file. I think that's like really good. That captures a lot of our ideas of how these deep learning systems work.

Like why does pre-training work? Like if you have two sets of weights and you, you wanna train on some downstream data set, one set of weights is pre-trained, one set of weights is randomly initialized. Why is the pre-trained model better at all, even though it's never seen your data?

Maybe one way of looking at that is that it has like-- it makes the information like more extractable somehow. Like there's this concept of like computational processing that you can almost like store up. I like this as a just like a lens to view problems with like how much information is stored where.

Like if you, if you get a, a set of model weights or like an activation vector and you open it up like print some tensor or NumPy array, it looks like- ... random numbers, right? Like there's nothing human intelligible about that.

But really it's this complex combination of like the training data and the training algorithm which get compressed into model weights, and then the actual computation that the model is doing, which involves like manipulating these numbers in ways that we don't understand.

So it's like this really highly compressed, non-linear combination of all these information sources mixed with like computation. And w- I just think we don't have like the right words of, of discussing this. I think I like the information theory analogy because back in the day, you know, we had phones and like telegraphs and, and people were just sort of like building the phone system with these crazy heuristics to like send information across the country or send telegraphs across the Atlantic.

People were just- Like trying stuff. And then, uh We kind of found stuff that worked, and we, we ran with it, but it wasn't really optimal. And it wasn't until someone came along and proposed this concept of like a bit, like a one or a zero that tells you something.

And once we have a bit, we can do all these things. We can like count the amount of information a signal. We can do really good error correction. We can measure properties of distributions of things and, and we can build like a really good system for, for phones, and then eventually which led to computers.

I'm bringing this all up because I don't think we have-- I don't think we know what a bit is yet in terms of like deep learning models. I'm gonna graduate from my PhD this year, but I didn't figure it out.

So if you're listening to this, maybe you can like, I don't know, spend more time on it, or- ... you're smarter than me, or you have a b- you know, a group of collaborators who can all get together and figure out what the right lens to look at this stuff is.

But even by just asking these questions, I think I was able to conduct this research agenda that I, I'm kinda still working on actually.

**Sean** [19:53]
Yeah. Uh, what do you call this field?

**Jack Morris** [19:56]
Oh. I don't know. I don't know. I called the post a, a new type of information theory.

**Sean** [20:02]
Okay.

**Jack Morris** [20:02]
I, I don't think it exists yet, I guess. So maybe it'll, it'll get a name once, uh, someone actually comes up with the right set of definitions. I think the information is a, is a really good start.

**Sean** [20:12]
There's a couple related threads. Uh, so first of all, you don't know this, but I actually have been trying to accumulate in, uh, data about Shannon... Like this is just like a Shannon information theory view of language models.

I have a, a lot of notes. This is actually on my GitHub for people who are watching along. But, um, you know, and like at the limit, if a, a language model has 175 billion parameters using 16-bit, you can-- it will take up 350 gigabytes.

You can compare that to Wikipedia. Wikipedia is about 150 gigabytes. You know, let's say GPT-3 can store two Wikipedias. But like is, is that a relevant measure of information storage, right? It is not because you can compress Wikipedia a lot.

There's a lot of repeated-

**Jack Morris** [20:52]
Mm

**Sean** [20:52]
... patterns. Tokenization is, is like the first form of compression. But I think there's a, there's a related talk from Ilya Sutskever about how deep learning is machine learning, it kind of is, is, is compression. Like you have a dataset, you compress it into a, a model that is smaller than a dataset but generalizes and like, uh, has like, you know, some, some amount of acceptable loss.

I think that one of your commenters on, on the, on the post made this direct comparison with Ko- Kolmogorov complexity-

**Jack Morris** [21:22]
Mm-hmm

**Sean** [21:22]
... which is what Ilya-

**Jack Morris** [21:23]
Mm-hmm

**Sean** [21:23]
... uh, how Ilya sees it. So I think like people have this information theory idea or approach to language models. It is just not precise because exactly what you say, like it's we don't know what a bit really means.

We don't know what like the most legible-- Legibility is, is a word that comes to mind in terms of like, like it, it matters to us that it's human readable. Like even if it's SHA-one, SHA-256, I don't care.

But like that is less readable, and therefore there's more, I guess... I don't know. Entropy is not the word because it's, it's directly-

**Jack Morris** [21:53]
It's a-

**Sean** [21:53]
... convertible, but it's just less useful.

**Jack Morris** [21:57]
Yeah. Yeah, yeah. Useful is a good word. I think maybe useful information or usable information is, is the right lens. And Kolmogorov is a really interesting connection, like Kolmogorov complexity. I think that's a really good concept for, for computer scientists.

**Sean** [22:11]
Yeah.

**Jack Morris** [22:11]
So I'm not sure exactly about this specific talk or like what-

**Sean** [22:15]
Yeah

**Jack Morris** [22:15]
... he was trying to say, but I, I think that we have a very good understanding of language model pre-training, and there's a deep connection between language models and, and compression. Actually, may- maybe let's, let's start with the embeddings.

We can come back to that.

**Sean** [22:30]
Okay. So is this, uh... Are we going to the first paper?

**Jack Morris** [22:33]
Actually, let's go to your Wikipedia, uh, numbers if, if you still ha- have access to that. So this 50 gigabytes for text of Wikipedia, that sounds like pretty high to me. Is that... That's uncompressed like text files or something?

**Sean** [22:46]
I don't know. I, I grabbed it from, uh, Andrew Yang, so I don't know.

**Jack Morris** [22:49]
Oh, no, no. Okay, okay. No, no, I'm probably off. I just sort of have the sense that like when you store text, it's generally like very, very small, especially when you zip it.

**Sean** [22:59]
Maybe he's including all the languages, all the edits. I don't know.

**Jack Morris** [23:02]
Yeah, yeah, that can make sense. That can make sense because I, I guess what you, you say from, you know, if you wanna do apples to apples comparisons, GPT-3 can store two Wikipedias. Is that right? Two, 2.3 Wikipedias?

**Sean** [23:14]
Two point something. Yeah.

**Jack Morris** [23:14]
So I thought it would be a lot more, and this is actually an experiment that you could do. You could like just train a model on Wikipedia and keep training it until you can perfectly extract all of Wikipedia, and that would be like a good way of knowing like how many Wikipedias can GPT store.

I like, I like that idea. But I think this type of like back of the envelope math is, is really useful for thinking about problems and like grounding yourself in the real world, even if you can never quite answer the questions you wanna answer, at least like in four years.

### Embedding Inversion

**Jack Morris** [23:45]
If we think about embeddings, you know, vectors that people use for search, we can do the exact same kind of math. So if you use the OpenAI embeddings, which, uh, last time I checked, I think have 1,536 dimensions, so that if you say there's 16 bits per dimension and like half precision floating point, it's something like 20 kilobytes of information in, in a vector.

And if you wanna store 20 kilobytes of text, that's a lot of text, like many- ... many paragraphs that you can perfectly compress into 20 kilobytes. And so I think this is kind of like the idea we had.

I'll give you the practical explanation, which was I'm-- well, first of all, I'm a second-year grad student. I'm like going to these conferences, seeing all these other things people working on and thinking, you know, like, "What the heck?"

Like, how am I gonna like have my own little area to do work in that no one else is working in already? And so I spent a lot of time coming up with bad ideas, and my advisor would say, "No."

Like, that's not a good idea to work on. Many times this happened, and- Like even my first year and a half of grad school was like a lot of exploration and a lot of like coming up with bad ideas.

And then honestly, I'd be interested to see how he remembers it, but I think I wrote a sequence of proposals about different projects, and then I, I f- came up with this idea. I was like, "Oh, we should just try to do as well as we can to reverse engineer the text that's in embeddings."

And I t- and then we were talking about it. He was like, "Oh yeah, you should just do that." And then that was the end of the proposals, and then that was just working on that problem for a long time, which at the time I was really motivated by that because I was like, cool, like my first, as a grad student, my first sort of like official like sign-off on like coming up with a good research idea.

And at the same time, there was this big rise of this startup business model called like a vector database. And there were all these companies popping up, raising money, raising money, uh, getting like crazy funding, and then actual applications being built that do something where instead of exchanging customer data, they exchange vectors.

So we had this like very grounded question of like, what data are they actually sending when they, when they send the vectors? Like first of all, you have this information theoretic argument that when you send one vector, there should be a lot of text recoverable just in terms of like a lot of these things represent very short documents, but they actually have many, many bits.

So like the problem seems trackable. And then second of all, we had this justification of how the product is actually being used. Like if, if someone hacks into a vector database, what do they actually find? If that makes sense.

**Sean** [26:25]
Wow.

**Jack Morris** [26:25]
So we were working on that for a while.

**Sean** [26:27]
I think I have the, the, the talk that you did from, uh, that Sasha highlighted, uh, is this one.

**Jack Morris** [26:31]
Oh, yeah, yeah, yeah.

**Sean** [26:33]
Um, maybe that has the graphic that, that would kind of-

**Jack Morris** [26:35]
Oh, go one before, I think.

**Sean** [26:37]
One before?

**Jack Morris** [26:38]
This is actually... Yeah, this one's good. Yeah.

**Sean** [26:40]
I like having visual aid. I like how-- I like giving people breadcrumbs to follow up if they, if they're interested in digging more. But yeah, I, I remember this was a pret- pretty, uh, hot area of research, uh, at the time.

**Jack Morris** [26:50]
And there's been some really interesting follow-ups. Like we, we ended up building a system that can do this quite well, like taking and embedding and w- I think our highlight number is like at a certain length, like a long sentence length, we can get ninety percent of the text back exactly.

And, uh, a lot of people were able to do stuff with that. Like they can... They, for example, I, I know these people that work on a problem of like de-biasing embeddings and like in one dataset they do something.

They have a procedure for like removing all latent features that correlate with gender, so they can produce like useful embeddings that from some perspective have no like information or usable information about gender. And they'd been doing that for a while, and then they actually just used our tool, and they-- so like they would put in a sentence like, "This woman is a doctor at Weill Cornell Hospital in New York."

Or say, "This woman is a doctor. She works at Weill Cornell." And then they would run their procedure, and then they run our embedding to text model, and now it would say like, "This person is a doctor. They work at Weill Cornell," which is pretty cool.

So they have like sort of text-based evidence that their method is actually removing gender features. But let me talk for a second about the research phase here, 'cause I thought it would be... I mean, I know if, if you ever heard me talk about this, I probably told you about it, but just for a wider audience, I like thinking back on this because it was probably my, in some sense, like my greatest victory of grad school was like working on this embedding inversion problem for a while, for a l- uh, quite a while and, and proposing a lot of approaches and like testing stuff.

I think sometimes you do stuff and it's clear it was a bad idea. Sometimes you think you should have figured it out earlier, and then sometimes you do stuff and you kind of realize it's really complicated and, and probably not worth it.

So I was testing different decoding algorithms for embeddings that are closer or text that's closer to the text that's in embeddings, and I was testing these kind of like inference time adaptation models for samplers. I think we tried a lot of architecture and like kind of training tweaks.

We should have tried RL. I think that would work. But w- finally, we found something that ended up working, and I guess I'm just saying this all because I thought it was like so rewarding. Like, we were just banging our heads against the, the wall.

I'd have biweekly meetings with my advisor who'd kind of suggest things. Sometimes we would agree we were mutually stuck. Sometimes I would get feedback one way or another and, and try something new or try a couple things and, and we had this idea that it was possible from the information theory arguments and this other thing where we would, we would kind of like take our best guess at what the text was and re-embed it and see that it was kind of far from the true embedding.

So we had this proof that like a better method could like leverage this kind of information. And then when we finally solved it, it was, it was awesome. Like we had this number that was like thirty for months.

I think at one point I got it to thirty-five, and actually I think I was like, "Oh, I'm done." Like I got it to thirty-five and, and my advisor told me, "No, like that's-- you can't really just propose a new problem and show you pushed a metric from thirty to thirty-five.

That's like confusing and probably not that meaningful to people." And I, I think I was, uh, uh, you know, that was kind of like a local minimum for me where I was like bummed. But then we ended up getting the number to like ninety-seven or something, which neither of us knew were possible, and we were all just-- we were just kind of staring at this graph like, oh my God, like who knew you could get this much information from an embedding?

And that was like so great, like, um, just sort of this... It was so rewarding and so it was invigorating, honestly, like that research process of like we picked a good problem and then we spent so long trying stuff that didn't work, which I'm probably forgetting how frustrating that was.

I'm sure it was terrible. But then like actually solving or at least like coming up with a much better way of solving the problem, I don't know if I'd say we solved it, but we definitely learned a lot from where we started, was like, was great and it Completely solidified for me the fact that I should have gone to grad school to have this, like, life experience and, like, makes me wanna do research forever.

### Universal Geometry

**Sean** [31:09]
You're clearly, clearly, um, sort of in love with the, the, the journey, uh, which I think is, is important because it-- this is what keeps you going through the, the tough parts. Is this a good time to talk about the universal geometry side then?

**Jack Morris** [31:21]
Yeah. Yeah, yeah. Let's, let's do that next. I think that's a good idea. So, so we have this more recent follow-up and the... So the first part I was talking about ended up in this paper called Text Embeddings Reveal Almost as Much as Text.

**Sean** [31:35]
Yeah.

**Jack Morris** [31:35]
Which was published in twenty twenty-three, and then we recently had a paper come out on Archive, which will hopefully be published at some point, and it's called Harnessing the Universal Geometry of Embeddings, which was also... That was probably, like, the only other time I felt like we've made, like...

Maybe there have been two more times, but that, that was probably the, the second of three times where I felt like we made, like, a real discovery about, like, the unknown, and it was, like, very rewarding just for its own intrinsic kind of elusiveness.

And I'll start from explaining it in terms of the prior paper. So we, we built a system that can, you know, do embeddings to text and it, and it works very well, and we're all-- we're very pleased with ourselves.

And then we, we went to a conference. We talked to people about it. We talked to, like, the vector databases. I think some of them changed their privacy policies, which was, like, somewhat gratifying. Um, and then we kept getting this perpetual question, which is like, "Well, you're just assuming we use the OpenAI model," or, "You're just assuming we use the most popular text embedding model."

If they fine-tune their own model or if they use a model that you're not training an adversary for, then you can't solve the problem, which is, like, true. Like, none of the vector-to-text stuff works unless you have this assumption of, like, knowing the encoder and also being able to make a lot of queries to it.

But we had this kind of underlying theory that all of the models learn very similar things. Like, we have some preliminary evidence for that, like certain models that are fine-tuned from the same base, you can kind of swap their representations without doing much.

Or if you look at the nearest neighbors, a lot of the models will give you the exact same nearest neighbors even though they have completely different training bases. And then there's this paper that came out last year called the Platonic Representation Hypothesis from some folks at MIT, which is really, really compelling and I think just, like, great intersection of philosophy, representation learning, deep learning research.

Like, I, I love this paper and that it's, it's such a beautiful idea, which is something like all models are trained on data from the world, and there's only one world. And so as the models get better by scaling data and scaling model size, they're sort of converging to learn the exact same thing.

And in this paper, they have evidence ba-based on correlations for doing this with vision and language models. It's very neat. And so we saw this. S-so basically think about, you know, you're us. You see this Platonic Representation Hypothesis paper.

A lot of people have this shared idea, like, you know, Claude and GPT-4 probably do a lot of very similar internal computation because both of them are trained on trillions of tokens of human-written text, even if they have different architectures.

Like, maybe, you know, the actual basis or, like, the, the numbers if you look at them look different. But in some way, they're, like, kind of computing the same thing. And I think it's even more true with these, like, embedding models which have, like, really only one objective that works, and they're probably all trained on, like, MS MARCO, which is a really popular dataset, and pre-trained maybe on Wikipedia.

But we wanted to basically combine this Platonic Representation Hypothesis idea with the vector text thing and produce a system that can, like, align models so that we can do embedding inversion.

**Sean** [34:54]
But you know, it's, it's valuable for more than just embedding inversion. Like, you can use this to kind of-

**Jack Morris** [34:58]
Yeah, yeah

**Sean** [34:59]
... glue together models. Like, that's what actually got me, like, super excited. And by the way, like, I think there's a few related threads. Uh, I think we did an episode with Nicholas Carlini where he had an extraction attack, uh, on, on one of the, the GPT models and, uh, they, uh, they got it fixed.

The other thing I wanna... I just really wanna spell out for people just in case they're not thinking it through. Being able to invert embeddings also means that you can s- you can back out, uh, s- like, secret prompts or context that might leak customer information that's potentially harmful and, like, obviously a, a attack vector issue.

I think one of the things I, I had a question about was whether or not position embedding does affect it and, like, extension of pos-position embeddings affect it because obviously that... Like, contexts are gonna get longer and longer.

Your ability to invert will obviously dec-decrease with longer context. What now? You know, uh, maybe not that important.

**Jack Morris** [35:48]
No, no, no. No, you're totally right. So we're operating in this space in, in our work where the sequences are relatively short and the embeddings are relatively large. Like, I think we're kind of at a great advantage from that perspective, and you're definitely right.

Like, if you embed an entire book to a- ... a five hundred-dimensional vector-

**Sean** [36:13]
Yeah, no way. Yeah

**Jack Morris** [36:13]
... there's just no way you could get the entire book back. Like, there must be this, these kind of collisions. Like, it, you know, in information theory, like, if you have lossy compression, two different inputs map to the same code, which means that you can never determine which input formed the code.

And I think that's probably what will start to happen. Like, if you have two books and you swap just one word and you embed them, I don't know, someone can try this, you'll probably get, like, s- a perfect collision, and in that case, inversion is impossible.

And even, like, when you don't take it to the limit, it probably just gets very, very hard. Like, things get super compressed. So I don't know how well this work scales. Like, it's a great question, like, exactly how much information you can sort of cram into one of these vectors, and I, I don't have a sense of where the boundary is.

**Sean** [36:58]
It'd be interesting to talk to some- one of the, like, linear algebra people from, like, the math department on, like, how literally can we take inversion Like, you know how-

**Jack Morris** [37:07]
Yeah

**Sean** [37:07]
... like, what, what measures of a matrix do they have where we can, like, kind of run that and, like, try to g-get some meaningful information out of that? This is, like, where information theory starts to collide with, uh, linear algebra and all the, all the other stuff.

**Jack Morris** [37:20]
Totally, yeah. There, there's always this, um, this detail where we're, we're running these on computers, and so we don't actually have, like, real decimal numbers or real numbers. We have, like, floating point representations of numbers-

**Sean** [37:34]
Yeah

**Jack Morris** [37:34]
... which are, like, very... It kind of, like, throws a wrench in-into the mix.

**Sean** [37:38]
Do you have any consideration of, like, superposition when, like, sort of nonlinearity. Like, you, you could, like, stuff information in the lower bits, but I don't know if that matters. I, I really don't. It-it's just, like, a nice thing to think about.

**Jack Morris** [37:51]
Yeah, yeah. It is a great question, and, and I get a sense that, like, a lot of the less important bits are more useful for computation, and maybe the higher order bits are more important for, like, storing data or something like that.

But I-I'm not sure. These are the kinds of questions I'm actually- ... hoping to explore over the next few years. Like, um, I'll skip ahead for a second. So we, we have this result that's, like, maybe the th-the third sort of, like, discovery I was alluding to, which is, like, a way to measure the exact capacity of a language model, and we get this number.

If you train a language model on a ton of random data and you measure its rate of memorization... Yeah, can you open the right curve? This is sort of the discovery I'm talking about. Like, no matter how you scale the training size, you hit this, like, perfect, perfect-ish plateau in model memorization, which we call the model capacity.

And the, the question I've been stuck on in the back of my mind for a while is, like, how is that actually implemented? So, like, this is a transformer that is trained for many, many data points and many, many training steps.

And so it, like, it's almost... Like, if you have, okay, the 10 to the 6th point on the x-axis, the capacity, uh, we don't have to actually say the numbers, but it's basically perfectly dividing its computation between all of the data points.

Like, every one of the 10 to the 6th data points gets, like, a tiny sliver of the model parameters because they're completely independent random strings. So I don't really know if superposition is occurring here. Like, it seems possible to me that the model would learn, like, these completely independent columns of computation, one per data point.

But it's also possible it's learning some kind of, like, combined thing where-

**Sean** [39:35]
Mm-hmm

**Jack Morris** [39:35]
... it's maybe it learns like a load and a store, and it's, like, sort of like loading and storing bits using these generic operations, and then in the end, it reconstructs the random string. So even though, like, the data is completely independent, the kind of like compute is, is very similar in terms of, like, predicting random strings.

But yeah, I guess this is all to say, like, about superposition and everything. I have no idea how the mechanisms are actually implemented inside the models, and that's, like, one thing I'm hoping to learn about in the next couple years.

**Sean** [40:06]
It's a reasonable question whether it's meaningful to learn. I think there's a lot of things that is, like, nice to know, but maybe not that useful. Latent space, latent space alignment is very, very useful. Dataset efficiency in theory, cool, but, like, practically, people are just gonna go for the biggest dataset they can.

**Jack Morris** [40:26]
Uh-huh. Uh-huh.

**Sean** [40:27]
Like, like-

**Jack Morris** [40:28]
Yeah, yeah, yeah

**Sean** [40:28]
... the, the scaling laws are kind of worked out insofar as, like, the relationship of compute and data. Amount of memorization, I, I don't know. I think maybe this is a good point to maybe also bring in the idea that Andre has been pushing for the last, like, I think year and a bit of, um, the cognitive core.

Like, the, what is the dumbest possible model-

**Jack Morris** [40:47]
Mm

**Sean** [40:47]
... that knows nothing but is u- is smart enough for tool use to do everything else, right? So you can run it on device and fast inference is open source, whatever. Uh, so Gemma 3N is, like, a really good candidate right now because it's, like, a 4B model that is, like, claimed to be better than Llama 4 and GPT-4.1, uh, according to, you know, certain-

**Jack Morris** [41:08]
Yes

**Sean** [41:08]
... arenas that shall not be named.

**Jack Morris** [41:10]
This is where things get complicated. Like, I- it feels like language models kind of implement things and know things almost in the same way, and it's, like, really difficult to disentangle, like, whether they're memorizing facts from whether they're, like, learning useful ways to generalize about new stuff.

But I, I agree this would be really nice. I don't think we have a lot of evidence that we can build a system like this that, like, is really, really good at reasoning, but really dumb about the world.

Like, I don't know if we have the tools.

**Sean** [41:42]
Yeah, maybe, maybe not. I think the existence proof is hum-humans, right? People always lean on humans as, like, the existence proof. It's not a great existence proof because I think if you talk to people about the number of neurons that we have and you make a neuron roughly equivalent to a parameter, we have something like 100 trillion in our brains.

So, like... and, like, we consume, like, 20 watts of energy. Like, it's nothing. Like, we're so much better than, than, uh-

**Jack Morris** [42:06]
Yeah

**Sean** [42:06]
... language models. It's not even funny. And then the, the last feature of us is that we are self-pruning, which is, uh, not something that language models do as well.

**Jack Morris** [42:13]
Oh, like we forget stuff?

**Sean** [42:16]
No, like we are not deeply, densely connected. Like we-- like connections will drop, therefore we're more efficient, you know?

**Jack Morris** [42:22]
See, see, unlike a language model where everything is always connected all the time.

**Sean** [42:26]
Yeah, or you, like, you, you preset the skip layers or whatever and that, that's it. You know, like it's not- ... it's not really actually anything involved with learning. It's just, like, something you do based on ablations and, like, guesstimates.

**Jack Morris** [42:39]
Even if we did want that, I'm not sure if we have, like, the right frameworks or methods for actually building, like, what you're talking about yet.

**Sean** [42:47]
I think the world is much closer to where you're at than where Andre is at. Andre is, like, kind of wishing for a, a fu- an optimistic world. Our conversation with Noam Brown was like, yeah, reasoning is emergent.

If you gave the o1 harness on top of GPT-2, you would get nothing because GPT-2 didn't know enough. You need a GPT-3 and GPT-4 in order to then get o1, like, as, as GPT-4 as the base model. Which is like, yeah, that's, I mean, that's, that's reasonable.

The way I put it is like, in order to use tools, you need to... Like, in order to search Google, you need to, to know at least search terms-

**Jack Morris** [43:19]
Yes

**Sean** [43:19]
... in order to, like, then search Google and then learn what you need. And if you don't know what, what to search, then, like, you might just be too dumb. Uh

**Jack Morris** [43:28]
Um, I like the kind of, uh, ethos, like maybe you could do some kind of free training where whenever the model doesn't know something, it can just Google for it, and that way you try to encourage it to learn words without...

or, like, to, to guess words correctly without actually storing the information into its weights.

**Sean** [43:47]
Yeah.

**Jack Morris** [43:47]
It seems like a nice, like, goal, at least.

**Sean** [43:50]
Yeah. You need some kind of online learning probably, or memory-

**Jack Morris** [43:54]
Yeah

**Sean** [43:54]
... uh, and some combination of that.

**Jack Morris** [43:56]
Yeah.

**Sean** [43:57]
It's exciting. You know, like, I think, like, if that is the, the direction of, of where this all ends up, that's great. But, like, people aren't, are not doing that. Instead, we're building, you know, $500 billion data centers in the middle of Texas.

And like, you know, all hail the, the, the God cluster, uh- ... that just will, you know, eventually wrap around the sun and consume solar energy because that's- ... that's what we need. Did we finish out the universal geometry thing?

**Jack Morris** [44:24]
Let me finish the kind of, uh, methodological description. So, so we had this goal... So, so, so yeah, back to the embedding universality. We started with going from embeddings to text. We know about this Platonic representation hypothesis, and maybe I'll skip over the details, but basically we had total inspiration from computer vision and this model from 2017 called CycleGAN, which is, among other things, uh, it's a way to map between two different distributions without any underlying notion of, like, which thing should be mapped where.

It's just based on some kind of idea of closeness. So, like, the cool thing about this, if you look at the top left, so I guess the, the top left is Monet, so impressionist paintings, and this picture on the right is a photograph.

So, like, it's learning this kind of like semantic notion of what content goes where just by mapping a distribution of Monet pictures to a distribution of photographs without actually telling it which Monet picture should map to which photograph.

It's kind of a subtle point I'm making. Uh, it takes a little bit of time to wrap your head around or maybe, like, go to the middle one, if you don't mind, the zebras and the horses. So, like, it's clearly learning, like, what an animal is and what legs are, and sort of like more abstract stuff like what, uh, the camera position should be and, and what grass is and stuff like that.

And it's learning, like, what a horse that looks like a zebra is, which is actually like a complicated semantic concept. Like, we don't have a, a dataset that has a horse and then that horse as a zebra. We just have separate horses and separate zebras, but somehow this, this GAN system is able to, like, elicit this sort of mapping property.

It's like kind of a magical connection that it learns. Uh, and I'm still like in awe that it's possible at all. But we mo- more or less, like, repurposed this system and, like, we, we built our own, but, like, this idea, we took it and we applied it to model embeddings, where instead of zebras and horses, we have like BERT embeddings and GPT embeddings or like two completely different models with different architectures.

So I think these are GTR, which is a T5-based retrieval model, and GTE, which is based on BERT. So they have different training data, different architectures, different downstream objectives, different embeddings. But yet when we do this CycleGAN in the embedding space, they just perfectly sort of snap to the same place, which is amazing and has some pretty deep implications of like the Platonic stuff.

Like, maybe the models actually are learning a lot of the same functions or something and in some semantic way they're like very close. And y- yeah, this is a diagram of how our system looks.

**Sean** [47:12]
It's weird to me how profound it seems. Uh, like, uh, you seem, you seem like deeply impressed by it. And then the other thing is like, uh, when we talk to the, uh, the, to Emmanuel from Anthropic, who did the circuit tracing and, uh, mech- mechinterpret-- mechanistic interpretability work, they were like excited that like the same thing in different languages maps to the same circuits.

And I'm like, "What you would expect?"

**Jack Morris** [47:36]
Yeah. Yeah.

**Sean** [47:37]
I don't... Like, I, I don't know, like, why, like... I, I, I, I, I don't know. I think-- I feel like this, this feels more profound to you than it does to me. I'm like, "Yeah, obviously."

**Jack Morris** [47:45]
No, that's, that's so fair. Maybe it's just like self-congratulatory and we're happy-

**Sean** [47:49]
No

**Jack Morris** [47:49]
... that we're like the people-

**Sean** [47:50]
Yeah

**Jack Morris** [47:50]
... that got it to work.

**Sean** [47:52]
Yeah, exactly.

**Jack Morris** [47:53]
Yeah, it does, it does seem obvious in retrospect, and I think that's like constant feedback I've gotten from research from... You know, people tell you that this seems obvious to them, but you have to realize that, like, you came from a perspective of no one ever having done this before-

**Sean** [48:07]
Right

**Jack Morris** [48:07]
... and they're coming from a, a perspective of you telling them it's true. And, like, if someone had told you that this was true, it would be like maybe obvious to you too- ... if that makes sense.

**Sean** [48:18]
The way I put it is that we have the intuition but not the proof. You have to-- You did the work, and you have at least some evidence that it's true, whereas we just have intuitions, right? So, uh, part of research is just confirming intuitions.

The applied part comes from like, okay, now that you know this for a fact, what do you do with it?

**Jack Morris** [48:38]
Yeah. Right. I, I think the details can be really interesting, like the details of the proof, like which models are most similar to one another and to what degree can you get them to align, and on which distributions does this property actually emerge.

And, like, that's why reading papers can be fun sometimes, is because they kind of answer all those little question.

**Sean** [48:55]
Yeah. I would say, uh... Uh, okay, I pulled out something very current, uh, which is Gemma 3N, which launched-- which, uh, sort of was, uh, generally available yesterday. I would say the-- f- for me, and you can correct me if I'm wrong, the most immediate implication is mapping adapters to language models.

So the, the dream is that you have a language model backbone, let's say this one is like a 2B- Language model backbone. And then you offload your vision. So you only, you only load in the vision and para- parameters or the, the vision adapter when you need vision.

You only load in audio, you know, you only load in speech, uh, text to speech whenever you need it, uh, because these are all separately trained. You're, you're just sort of aligning latent spaces, and you can sort of train them separately.

And I think, like, this helps to make us more confident in... One, it's, it's more efficient. That's a, that's a given. Two, it makes... helps makes us confident that we can just, just add capabilities without taking away or catastrophically forgetting others.

**Jack Morris** [49:51]
Hmm. So they're just sort of like stacking more parameters.

**Sean** [49:55]
Just stackable.

**Jack Morris** [49:56]
So that's very cool. Yeah.

**Sean** [49:57]
Swappable, stackable. It's like a, a fatter version of LORAs that is not really-

**Jack Morris** [50:03]
Yeah

**Sean** [50:03]
... that model specific. I would say Apple and, and Google are pursuing this for their on-device stuff is, is where... is my sense.

**Jack Morris** [50:10]
Is this open source?

**Sean** [50:11]
Gemma? Yeah. For a given definition of open source, which is like we released the weights of Hugging Face, here you go.

**Jack Morris** [50:16]
Oh, that sounds like open source to me.

**Sean** [50:19]
Eh.

**Jack Morris** [50:19]
Oh, yeah, I guess it's open weights, but not the data.

**Sean** [50:23]
Not the data, not the code.

**Jack Morris** [50:25]
Not the code, yeah, right.

**Sean** [50:27]
Yeah, I would say that this is quite SOTA in terms of efficient models. Maybe a small LM also from Hugging Face would be also in that, in that category.

**Jack Morris** [50:35]
Hmm.

**Sean** [50:35]
There's not that many people working on very good, very efficient models.

**Jack Morris** [50:39]
Yeah. This is a, a very deeply related question and something that really interests me, which is like what is the limit of like 100 million parameter model? Like, if you imagine, you know, 100 years from now when we have maybe our computers are gelatinous blobs and we all communicate through telepathy, will we have 100 million parameter models that are at the level of today's o3 Pro or whatever?

And like, if so, like how would that even be the case, like based on scaling laws? Like do we have special data? Do we come up with like a brilliant new training scheme or some type of magical architecture?

Like, I really don't know. Or maybe we're, we really are on the, at the plateau already. I don't know.

**Sean** [51:22]
It seems like when we are doing things like calling a small model like a 27B, uh, model as, as small, like that's what Mistral is doing-

You know, we've plateaued a little bit in terms of like what we can do to compress things. I have a fun theory, uh, that this is where we mix quantum computing with models. Like we have to change what a parameter means.

We have to search through very high dimensional space and, and resolve them much quicker than we can with like conventional compute. That would be my pie in the sky thing.

**Jack Morris** [51:54]
I said 100 years. That's very reasonable to me.

**Sean** [51:57]
Throw quantum at it.

**Jack Morris** [52:00]
Yeah, I'd probably have to get a second PhD to know what's going on there. I think that we should establish the definition of small model as being a model that a grad student can infer-

**Sean** [52:13]
Yes

**Jack Morris** [52:14]
... at reasonable time on a single GPU.

### Model Capacity

**Sean** [52:19]
Yeah.

**Jack Morris** [52:19]
Which is probably like 7B maybe. I don't think 27 is small under any reasonable... Is it, wait, is it MoE?

**Sean** [52:27]
Mistral? Uh, I don't think so. I think I'll... I think their stuff is default dense. Don't quote me on that. Uh, this is, uh, this is coming off of, um, just a lot of pre-trained data that is, that is, uh, po- potentially collided.

Okay. There was one more-- there's two more papers that we wanted to cover and then we can, we can sort of wrap it. You had an approximating... You had a language model training data. I think this is a little bit, uh, also newer.

How does this rank in terms of your, your overall work?

**Jack Morris** [52:54]
Yeah. L- let's return to the kind of information theory question. So yeah, maybe we'll, we'll skip over the contextual embeddings in the case of time, but we'll group those papers. Great paper. Hopefully, people start training with that technique.

It's kind of a free lunch. Those questions are all about information and model activations. Like how much can we recover from this given vector or like what data does this vector represent or what com- computation does this vector represent?

And there's really two types of... Like if you wanna taxonomize, there's, there's two types of, whatever you call it, dense information storage mechanisms. One of them is activations or embeddings, which we were discussing already, and then the other is weights, which are the things that are used to perform the computation but not the computation itself.

And so we have now two papers in this direction of what is stored in the weights. The first one is about language model capacity, which is called How Much Can Language Models Memorize or How Much Do Language Models Memorize?

I never remember which one we settled on. And then the other one is called Approximating Language Model Training Data from Weights. The first one is like I, I think has a lot of deep messages about how language models store information and how they work in general.

The second thing is like a proof of concept of maybe like a longer term research project. Let's start with the capacity stuff, if that's good with you.

**Sean** [54:21]
Do I have the paper for that? I don't, I don't know.

**Jack Morris** [54:24]
You know, we can return to the question you asked me, which is something like why do we care or like what is this-

**Sean** [54:29]
Yeah

**Jack Morris** [54:29]
... useful for? And I don't know if I have a good answer for this. I think this is, this is somewhat profound. Like it's kind of like in, in physics, you know, when they try to measure these constants, like gravity.

People tried to measure the rate of acceleration of gravity for a long time, or like those Greek guys like back in, in the BC era when they were trying to, to approximate the radius of the Earth based on shadows.

We're trying to take the GPT architecture, like the main one, and just measure how much information it, it can store. And w- we did this through the, the lens of memorization, which I think we can skip over for the podcast and, and we'll just talk about like information storage and, and weights.

Like-

**Sean** [55:13]
Yeah

**Jack Morris** [55:13]
... these curves to me are, are pretty crazy. Again, maybe, maybe it's like the sort of discoverer's fa- folly or something where I'm like, "Oh, this didn't exist before, so it seems so cool," but then you're saying like it seems somewhat obvious.

**Sean** [55:28]
No, no, no. Don't, don't let me take that away from you. Yeah, no. Again, like I, I independently was asking how come there's not enough people exploring LLMs from information theory. And then like you come along and your embeddings works become like an information theory exploration and I'm like suddenly, like I'm very aligned to like exploring this, promoting this and encouraging more people to, to figure it out because like that's ultimately how we figure out this whole compression issue and the-- you know what, what Andre wants, which is like the cognitive core, right?

The most efficient model for the most capability. Like that is an information theory question.

**Jack Morris** [55:59]
Totally agree with that. We could start here like so, so transformers that are trained in 32-bit precision, we approximate can store about 3.6 bits of information to maybe 3.9 bits somewhere in there per parameter. And like why is this?

I mean from some perspective this is, this is quite bad. Like- ... if you have 32 bits available and you can only use three to four of them, like you're-

**Sean** [56:27]
Just store 32 bro.

**Jack Morris** [56:31]
Yeah, yeah. Then you'll like, you know, you could build your own AI lab if you can make the-- these models, uh, that much more efficient. I don't know how they're implementing this mechanism or where the kind of bottlenecks come from or even now that we know this, what it's necessarily useful for.

I guess the tools that would be interesting to me are knowing like given a data set, if you could predetermine the exact model size and maybe architectural properties required to get a certain level of performance, that would be really neat.

And like we don't even know how to do that. We don't even know what the difference is between doing low retraining, which trains less than one percent of the parameters and full fine-tuning, which trains all the parameters. We don't even really understand the difference there.

So I think this is like maybe like a baby step sort of in that direction, but there's a lot of unknown ahead of us.

**Sean** [57:21]
Okay. Do you think this is a hard limit? Do you think someone can come up with a better algorithm but better architecture and then sort of just change the slope?

**Jack Morris** [57:28]
There are two axes here. One is the ability of the model to store data, and I think we can definitely improve that. I think like maybe even if we tested this with Llama architecture, like there's sort of like a GPT plus plus architecture.

Like I would guess that can store better data just because the kind of numerical flow is a little bit better. The non-linearities are maybe like a little bit more suitable to training. Like that will probably raise the bound a little bit.

And then the second axis is that our measurement tools are just not that good. Like this is, you know me, I'm a grad student, I'm running all these hyperparameter sweeps and sort of like we, we draw conclusions from them.

But even that being said, like there are probably ways to measure this better and-- but all that would do is push the number up. So it's possible like there is a way to store five bits per parameter if you have like a better optimization technique or if you were a super genius and you could just perfectly set the weights to store the data, then maybe you can do better.

And this is just sort of like what we can reach through optimization is this 3.6 bits per parameter. But I would be happy if someone came along with a much better measurement tool. Like-

**Sean** [58:37]
Yeah

**Jack Morris** [58:38]
... uh, this is just sort of like the first measurement. I mean, I would, I would guess in the future like, you know, people will look back and say like, this is like somewhat off in one direction or another for whatever reason.

Um, and, and that's just how science goes and I've no problem with it.

**Sean** [58:52]
What we do is we call-- we, uh, we call this the Morris constant three point six, right?

**Jack Morris** [58:58]
I would never.

**Sean** [58:59]
And then, and then we we, we set a like a, like a challenge, like a leaderboard of like, beat this, right? And like let, let people go.

**Jack Morris** [59:07]
Yeah. That assumes that we know the true constant ahead of time and we can measure the error rate.

**Sean** [59:12]
It's, uh, it's doable. You, you laid it out here.

**Jack Morris** [59:14]
Yeah. Yeah, yeah, yeah. That makes sense.

**Sean** [59:17]
One minor doubt I have is like the goal actually isn't memorization, it's generalization, right?

**Jack Morris** [59:23]
Mm-hmm.

**Sean** [59:24]
The best memorizer model may not be the best generalizer model and you like this incentivizing people to, to max this number might actually just be fruitless in terms of actual intelligence.

**Jack Morris** [59:35]
I see.

**Sean** [59:35]
Like you just get the best actual compressor. Like you're just gonna get Gzip.

**Jack Morris** [59:39]
That's totally true. And there's this pattern in research, you know, time after time it's like someone poses a question and then people answer it over and over and over again. But it's, it's often much more fruitful to just ask a new question.

Maybe it just doesn't matter how much GPT models can store and you should just like work on something else.

**Sean** [59:59]
We'll figure that out. Um, did you wanna, uh, dwell on this, this side at all?

**Jack Morris** [1:00:04]
Yeah, let's just talk about it real quick. Uh, definitely not the algorithm itself.

**Sean** [1:00:08]
By the way, what are your tools for doing these kinds of charts, uh, and these kinds of diagrams? Like I, I just I'm kinda curious of behind the scenes on the tools.

**Jack Morris** [1:00:15]
I think like visualization has definitely been fun hobby of mine during grad school. This one actually, Oscar, my co-author, made this one. Maybe I gave some like prompting, but he made it. I, I think most the last few papers have all been in diagrams, Google diagrams.

I was using Figma for a while and Illustrator. I think Illustrator actually is, is the best tool.

**Sean** [1:00:41]
Oh, did you know the Transformers, Transformers diagram was in Adobe Illustrator?

**Jack Morris** [1:00:45]
Oh, yeah, yeah, I did know that actually. Yeah, because that's the only way you can get arrows that sort of like curve like that. Um, and they, they have good shadows. Diagrams is like the least robust, but it's the most accessible.

And honestly if you, if you're good, you can make pretty good stuff. Excalidraw is, is nice too if it's not going in a, a paper.

**Sean** [1:01:06]
Yeah. It's just too rough for a paper, but you need something professional looking, you know. It helps, like if you're gonna publish your work, you need to make it look nice and professional and like official, right? So this is what it is.

**Jack Morris** [1:01:17]
Yeah, yeah. And I think there's something worthwhile about saying like, okay, if, if I'm gonna put my name behind this, like I wanna spend time making all the references perfect and, and all the diagrams professional, all the captions are correct.

And I think it's like important to put that level of detail into your work. That's a little cheugy, but let's finish this off. So okay, we're talking about bits, information theory, what, what information sort embeddings. We were talking about language model capacity.

I think a much more quest- more practical question is, uh, maybe so-- maybe this is more analogous to the vector database hacking, embedding threat model we discussed, is like if you have access to a set of model weights, what can you learn about the data?

### Data Inversion

**Jack Morris** [1:01:58]
So like you were just mentioning, Gemma 3B came out yesterday, and you can download it, and it takes up a certain amount of space on disk, and it was trained on some data, but we have no insight into what the data was.

I mean, it's probably English. It's probably some distribution of web text. I would guess there's a lot of code, and we seem to have a lot of information about the model, right? You have this file, and there's like many ones and zeros, which means something, but it's kind of like a very highly compressed version of the training data.

But I, I, I would be extremely surprised if they do any type of like private training. Like there are these mechanisms for doing like differentially private language model training or, um, even just anonymization in the pre-training pipeline. I bet they don't do any of that.

They just sort of like train on the data, and then they kind of know that we don't have the right tools to decrypt the model weights. And so that's like my dream is we can come up with some way of, of translating model weights back into text data sets.

And so in the, the most recent kind of drop, uh, paper drop is that paper approximating language model training data from weights, and it turns out to be a really hard problem. Like trying to go from model weights to text is, is really hard, and we do something a lot simpler, which is like...

Well, there's, there's two ways we make it simpler. The first thing is we assume access to two checkpoints, which I think is, is probably not the case in Gemma. But in, in the case of DeepSeek, if you download the four hundred billion- ...

parameter model weights, it's this giant file, and you can actually get two of them. You can get the base model weights and the fine-tune model weights. So the way we put this, we-- you have this kind of like difference in parameter space telling you what DeepSeek fine-tuned on, and it's very controversial.

I mean, there's sort of like geopolitical... Definitely at the, at the corporation level, they're really interested in the implications of like what did DeepSeek train on, and they've released this kind of treasure trove of information of what they trained on, which is the actual model weights, but we have no tool for like interpreting or kind of decrypting this weight difference.

And so we started with something really simple, which is instead of even just trying to like regenerate the training data, we take just a web corpus and try to do selection of training data that kind of like looks like the true training data and gives us performance that's as close as possible to the true training data.

So there's this complicated method, but it's something like you just sort of like look at the data point gradient and see if it points in the direction in weight space of the fine-tune, and then you take like the top data set.

There are some tricks to it, but it's basically just like gradient-based selection based on this weight difference, and it seems to be okay. Like it can get us pretty good training data. So I guess i- if you actually wanted to use this, it would be like your competitor releases a base model and a fine-tune, and you're trying to recreate their data set.

So you can take this weight difference and take a giant web data set. Like if I was doing this at a company, I'd probably try to scale it up to trillions of tokens and then select the exact data points that try to produce the model.

And, and it turns out you, you can train a pretty good model with that. It-- we don't get to quite the performance of the original model, but it does seem to be like trending in that direction.

### Datasets Thesis

**Sean** [1:05:24]
This is like very creative. I don't know what the, the, the, the use of it exactly is.

**Jack Morris** [1:05:30]
Yeah. Like when would you be in this exact situation?

**Sean** [1:05:34]
Decently often for the open model labs. Like even DeepSeek are, are one, like has released an update. Mistral does it pretty frequently. Llama does it frequently. It's not impossible, but like I, I think it's like, it's like-- I think that I really like the creativity in using quote-unquote synthetic checkpoints to do this.

**Jack Morris** [1:05:50]
Mm-hmm.

**Sean** [1:05:51]
Which is... I don't think I've heard this from any other place, so I, I don't know if you, you came up with the idea.

**Jack Morris** [1:05:58]
It's like linear interpolation in, in weight space.

**Sean** [1:06:01]
Okay. Um, that's a bunch of the recent work. Uh, I wanted to sort of cap things off with the data sets question if, if that's-- is, is that a good-

**Jack Morris** [1:06:08]
You can ask me whatever you want.

**Sean** [1:06:10]
Well, it's not, it's not like a... It's not a ask. It's just like I think this is a very good thesis. Uh, I think it's a hot take. I almost invited you to speak based on just this alone, but it was, it was a little bit late, uh, to talk.

**Jack Morris** [1:06:22]
Oh, for the, for the conference.

**Sean** [1:06:23]
Yes. When I look for conference keynotes, I look for something that has a broad overview that kind of put the last few years in perspective or is an insight that you can reasonably rely on to like last for a while, so you can get some mileage out of it.

You know, and I think a lot of ideas in AI come and go. But like things that are trend... These things are scaling laws, things that are trend lines, things are, things that are like, um, there's, you know, there's no new ideas in AI, that I pay attention to.

So maybe you want to recap, like what's, what's the backstory if, if there was one?

**Jack Morris** [1:06:52]
Yeah, yeah, sure. So the, the meta backstory is I've sort of started writing on Substack- ... and this is a post that I wrote a, a few months ago.

**Sean** [1:07:01]
The highest art form of humanity.

**Jack Morris** [1:07:04]
Yeah. Yeah. Yeah, publishing papers wasn't doing it for me anymore- ... and, and I moved to Substack, and this is the name of the post, There Are No New Ideas in AI, Only New Datasets. One, one guy pledged me, but then I found out he was like my former student from a class I was teaching.

So I, I don't think it really counts. But-

**Sean** [1:07:25]
It counts. He's a friend. He's your first supporter.

**Jack Morris** [1:07:27]
A pledge is a pledge, man. I'll take whatever I could get. So, so the underlying thesis is that whenever Maybe I'll, I'll lay out this framework first. So there's this, uh, this book called The Structure of Scientific Revolutions by Thomas Kuhn that I read near the beginning of my PhD, which suggests that science kind of moves in these cycles where not very often there's something he calls a paradigm shift, which is like a-- you could think of it as a zero to one innovation where everything changes.

And then it's followed by a rapid period of small innovations, a lot of like reapplication of previous techniques, pre-paradigm shift techniques to the new era, and then things sort of slow down as we wait for a new paradigm shift.

And I was kind of asking myself what's unique to the paradigm shifts that we've seen in, in AI. And, and by the way, to me, AI and language models are somewhat synonymous at this point, like at least for the foreseeable future.

I'm, I'm certain that will change. But basically, everything that's pushed the boundary to whatever we have now that resembles like intelligence has come from language models. And so those breakthroughs came in, in a few steps.

**Sean** [1:08:44]
Yep.

**Jack Morris** [1:08:44]
So I think the idea is also like a meta commentary on the research community, because what everyone wants as a researcher is some kind of like cute new method that no one has thought of before that just works on the existing data better than the previous methods.

That's like, for whatever reason, like the kind of most glamorous thing people think you can do as a researcher. Like Mamba, it's like, it's like a transformer, but it's like more efficient and works better. So that's what a good idea looks like.

And I think everyone wants to like, find something like that. But if you look at what's actually borne out in practice, it's never been like that. I think like all of the things that I would consider paradigm shifts in the Kuhnian sense came from a new technique but trained on new data, and I think the new data is super, super important.

So I wrote it as a series of four paradigm shifts. The first is the emergence of deep neural networks with AlexNet, which I think was like twenty ten to twenty twelve era, where we just started training on ImageNet, which is like a scale no one had ever seen before of millions of images.

And then the second thing was transformers and BERT, and this Attention Is All You Need paper, twenty seventeen, the first GPTs, twenty eighteen, which is web scale pre-training. Like no one had ever done that before. No one had ever tried to scrape all the text off the internet and then tokenize it and feed it into models.

Like it's a crazy idea, and I think like we should be honest. I mean, transformers are incredible and like their staying power is never going to cease to amaze me. They're like much more optimal than I think anyone ever knew.

And I don't know if we'll ever beat them, but the real innovation is web scale pre-training. And I think like we honestly probably could have gotten this with RNNs. I know like the Scaling Laws paper shows that RNNs have worse curves for scaling, but probably people would have been-- like I bet you could have built ChatGPT with a very sophisticated RNN.

Like you didn't even need transformers. What you need is web scale pre-training and the third innovation, which is instruction tuning. And we thought it, it came with like reinforcement learning. But I think the big innovation of instruction tuning is actually the human preference data, which is like gathering positive and negative pairs of what looks good, like in terms of a chatbot interface.

And actually it turns out you can do supervised learning on that too. You can do DPO, which is a form of supervised learning. You don't even need the, the InstructGPT techniques. You just need the data. So like I'm sort of playing devil's advocate here, but I actually think this is true that like if we had the right datasets, we almost could have scaled like twenty fifteen era techniques and gotten something that looks like at least InstructGPT.

Reasoning models are, are a little different. Like they're-- I, I'm not sure if we could have that with RNNs or not. Like, I don't know if I'm in a position to comment on that with certainty, but they do fall into this framework, which is they really did emerge from a new data source.

In this case, it's something like a little different. It's like verification with symbolic systems like math, calculators, coding environments, unit tests, like things where we can provide numerical feedback to language model outputs. But we, we built a way to, to learn that and leverage it to get more intelligent systems.

And so whatever the fifth thing is, whether it's video or embodied AI or some kind of crazy innovation on reasoning models, uh, whatever comes next will probably be some type of new data source that we're not using yet.

**Sean** [1:12:18]
That's a really good thesis. I would say that the researchers I talked to would somewhat disagree. Yeah, obviously, this is like a hot take type of thing, and like-

**Jack Morris** [1:12:29]
Yeah

**Sean** [1:12:29]
... you already acknowledged that RNNs don't, don't scale to the same extent. Like they operate on the slope of the curve, whereas, you know, I, I guess like the amount of data or the type of data or the core insight just changes the order of magnitude of the x-axis, right, that, that we are mostly working on.

But like, uh, both are important. The way that I think someone put it to me was an improvement on compute or data efficiency is the equivalent of having a whole bunch more data that we, you know, that otherwise would be a lot more expensive to collect.

It's likely that, you know, the frontier models right now are just a collection of hundreds of these small little experiments that just stack up. You mentioned Mu- Muon in your, in your post, which seems to be the atom killer.

Curiously enough, like still none of the big models, um, use Muon, but like vibes are good.

**Jack Morris** [1:13:20]
Yeah, yeah. And the value of building better optimizers is really incredible. Like it's just a free lunch. You can just sort of like plug in a slightly better training mechanism, and then you save like a ton of compute and a ton of training time.

That's like hugely valuable.

**Sean** [1:13:35]
I think this is cool because I think, like, it's puts us in a mode of, like, if you were ever to ask what comes after reasoning, it has to be something on the order of this.

**Jack Morris** [1:13:44]
Mm-hmm. Yeah.

**Sean** [1:13:45]
And most ideas are not. Most ideas are not. And so this is cool in the sense of, like, it just jolts you out of incremental thinking into, like, what really is missing for the next paradigm. And I don't have an answer.

Do you have one? Do you have, do you have candidates?

**Jack Morris** [1:13:59]
Oh, I, I really haven't even considered that too much. I guess, like, scaling reasoning-

**Sean** [1:14:04]
You gotta do the autocomplete.

**Jack Morris** [1:14:06]
For step five.

**Sean** [1:14:07]
I mean, you got us all the way there, and you're like, you know- ... y- you sh- you gotta show us the way now.

**Jack Morris** [1:14:12]
We can say it's an exercise left to the reader. But, I mean, the reality is, like, predicting the future is too damn hard, you know? Like, I, I don't... Maybe it'll be obvious to me in hindsight in five years, but sitting here today, I, I really can't derive from first principles what the next wave of innovation will come from.

**Sean** [1:14:30]
Yeah. I think we have a few years left. Like, each of these, uh, f- phases lasted for a few years. Reasoning just started last year kind of. We got some, uh, some juice on this one. Cool. I think that is a, a, a broad overview.

Like, we've went way over time, but, like, I really enjoyed this. I guess my parting question for you is kind of a meta one. So I'm not an academic. I'm, like, kinda self-taught. I just read a bunch of papers, and, like, I talk to people all day as part of the podcast.

### Wrap-up

**Sean** [1:14:57]
How do I rate in terms of, like, my, my questions? As though, like, could I pass as a grad student? Like, or like h- what, what's my distribution like? Maybe I was maybe more industry-oriented than academics.

**Jack Morris** [1:15:10]
I think you gotta realize that, like, the only person that's an expert in your area as a grad student is you, and even, like, eventually your advisor defers to you for a small set of questions that fall within your very niche expertise.

So, like, I think you're clearly, like, a very good generalist and have, like, a huge amount of background on these topics and to the point where I would say you're passing the, the grad student Turing test, and- ...

I think if you went to a talk, like, people would just assume you have some weird research area of your own that they don't understand, you know?

**Sean** [1:15:45]
My research area is AI engineering. Like, I'm, I'm trying to, like, making it up as I go. Uh, but no, this is super helpful. Uh, okay. Well, I, I, that's about all we, we prepared. All the best in your search.

All the best in your PhD. I assume, like, uh, apparently the current PhD meta is you j- you do a bunch of small papers, you staple them together and, like, find, like, an overall theme. You do a defense, and that's it.

Like, that's the journey, which is kinda cool. Like, I, I would, I would love to do that. Uh, I, you know, I'm too old to do it, but, like, it's cool.

**Jack Morris** [1:16:13]
Yeah, yeah. It's, it's a great thing to do at, at any age.

**Sean** [1:16:17]
Well, it's better to do a Substack, right? And then you have- ... like, people, like, subscribing and pledging along the way and, like, getting validation and, like, yeah, that's, that's better than a PhD.

**Jack Morris** [1:16:27]
Substack.

**Sean** [1:16:30]
That's the title of the episode, like, "Substack Better Than PhD." Um- But no. Uh, thanks. Yeah, thanks for your time. This is, uh, really great. Where can people find you? What are you looking for really?

**Jack Morris** [1:16:42]
I'm online. You know, you can follow my Substack and Twitter. I tweet pretty consistently. And, you know, putting papers out. I guess, like, the most meaningful thing, to be honest, is to engage with the research and send me an email if you really care.

Uh, that, that's amazing, and, like, I love having those kinds of discussions. And you mean, like, what I'm looking for in a job or out of life?

**Sean** [1:17:05]
Your research direction. Like, what interests you over anything else that's, like, if, if there's someone out there looking who has a problem and is looking for someone to help them on it, like, you are the guy for underscore.

**Jack Morris** [1:17:17]
Oh, yeah.

**Sean** [1:17:18]
Yeah.

**Jack Morris** [1:17:18]
Uh, hopefully if you listen this long, like, I think, like, my research is a lot more well connected than some people's PhD research in that it all falls into, like, a very small manifold of, like, all possible problems.

And so if you, if you wanna work on anything within that space or that's sort of, like, adjacent to the problems that we discussed in terms of, like, language model, uh, maybe not even language model, but model weight and activation information, I think anything that can be described as that is very interesting to me, and I would love to talk.

**Sean** [1:17:55]
Awesome. Well, uh, we'll put your contact info in the show notes, and thanks for your time.

**Jack Morris** [1:18:00]
Thank you.

---

This library is powered by PodHood (https://podhood.com), the podcast website platform.