Goodfire AI’s Bet: Interpretability as the Next Frontier of Model Design — Myra Deng & Mark Bissell

If you ask 50 people who, quote-unquote, "work in interp," like what is interpretability, you'll probably get 50 different answers. To some extent also, like where, where Goodfire sits in the space, we're an AI research company above all else and interpretability is a, is a set of methods that we think are really useful and worth kind of specializing in in order to accomplish the goals we want to accomplish.

But I think we also sort of see some of the goals as even more broader as, as almost like the science of deep learning and just taking a not black box approach to sort of internal representations and then bringing interpretability to training, which I don't think has been done all that much before.

So welcome to the Latent Space Pod. We're back in the studio with our special MechInterp co-host, Vibhu. Welcome. Mochi. Mochi- And- ... special co-host ... and Mochi, the- ... mechanistic doggo ... mechanistic interpretability doggo. Uh, we have with us Mark and Myra from Goodfire.

Welcome.

Thanks for having us on.

Maybe we can sort of introduce Goodfire and then, and then introduce you guys. How do you introduce Goodfire today?

Yeah. Uh, it's a great question. So Goodfire, we like to say, is an AI research lab that focuses on using interpretability to understand, learn from, and design AI models, and we really believe that interpretability will unlock the new generation, next frontier of safe and powerful AI models.

Um, that's our description right now, and I'm excited to dive more into the work we're doing to make that happen.

Yeah. And there's, there's always, like, the official description. Is there an, like an unofficial one that sort of resonates more with a different audience?

Well, being an AI research lab that's focused on interpretability, there's obviously a lot of... People have a lot that they think about when they think of interpretability, and I think we have a pretty broad definition of what that means and the types of places that can be applied, and in particular applying it in production scenarios, in high-stakes industries, and really taking it sort of from the research world into the real world, which, you know, it's a, it's a new field, so that hasn't been done all that much and we're excited about actually seeing that sort of put into, put into practice.

Yeah. I would say it, it's, it wasn't too long ago that Anthropic was, like, still putting out, like, toy models with superposition and that kind of stuff, and I wouldn't have pegged it to be this far along. Uh, when you and I talked at NeurIPS, you were talking a little bit about your production use cases and your customers.

And then not to bury the lead, today we're also announcing the fundraise, uh, your Series B, $150 mill- million at a 1.25B valuation. Congrats, you're a unicorn.

Thank you.

Thank you.

Very exciting.

Yeah. No, things move fast. We were talking to you in December and-

Yeah

... already some big updates since then.

Let's dive, I guess, into a bit of your backgrounds as well. Mark, you, you were at Palantir, uh, working on health stuff, which is, uh, really interesting because, uh, Goodfire has some interesting, like, health use cases. I don't know how related they are in practice.

They... Yeah, not, not super related-

Yeah

... but, um, I don't know, it was helpful context to know what it's like just to work with, uh, health systems and generally in that domain.

Yeah. And Myra, you were at Two Sigma, which a- actually I was also at, uh, Two Sigma-

Oh, really?

... back in the day.

Wow.

Yeah.

Nice. Did we overlap at all?

No.

Okay.

Uh, this is, this is when I was the- briefly a software engineer before I became a sort of developer relations person, and now you're head of product.

Yeah.

What are your sort of respective roles, just to introduce people to, like, what all gets done in Goodfire?

Yeah. Prior to Goodfire I was at Palantir for about three years, uh, as a, as a forward-deployed engineer, now a, now a hot term, uh, wasn't always that way, and as a technical lead on the healthcare team. And at Goodfire I'm a, a member of the technical staff, um, and honestly that I think is about as specific as- ...

like, as, as I could describe myself, 'cause I've worked on a range of things and, you know, it's, it's a fun time to be at a team that's still reasonably small. I think when I joined s- one of the first, like, 10 employees.

Now we're a- above 40, but still it looks like there's always a mix of research and engineering and product and all of the above that needs to get done, and I think everyone across the team is, is, you know, pretty, pretty switch hitter in the roles they do.

So I think you've seen some of the stuff that, that I worked on related to image models, which was sort of like a research demo. More recently I've been working on our scientific discovery team with some of our life sciences partners, but then also building out our, our core platform for more of, like, uh, flexing some of the kind of MLE and, and developer skills as well.

Very generalist, um, and you, you also had, like, a very, like, a founding engineer type role.

Yeah, yeah. So I, I also started as, I still am member of technical staff. Did a wide range of things from the very beginning, including, like, finding our office space and all of these-

Yeah

... like, nitty-gritty-

Beautiful

... we both, we both visited when you had that open house thing. It was really nice.

Oh, thank you. Thank you. Yeah-

Yeah

... plug to come visit our office. Um-

It, it-

Yeah, the work desk-

... looked like it was-

... very nice

... it was like 200 people. Uh, like, it has room for 200 people, but there were- you guys were, like, 10.

Yeah.

Yeah.

But growing fast.

For a while it was very empty.

Uh, but yeah, like, like Mark, I, I spend a lot of my time as, as head of product. I think product is a bit of a weird role these days, but a lot of it is thinking about, um, how do we take our frontier research and really apply it to the most important real world problems, and how does that then translate into a platform that's repeatable or a product, and working across, you know, the engineering and research teams to make that happen.

And also communicating to the world, like, what is interpretability? What is it used for? What is it good for? Um, why is it so important? All of these things are part of my day-to-day as well.

I love, like, what is things, because that's a very crisp, like, starting point for people, like, coming to a field. We will do a fun thing. Vibhu, why don't you want to try tackling what is interpretability, and then they can correct us?

Okay.

Great.

Um, so I think, like, one, just to kick off, it's a very interesting role to be head of product, right? Because you guys, at least as a lab, you're more of an applied interp lab, right? Which is pretty different than just normal interp, like a lot of background research, but you guys actually ship an API to try these things.

You have Embry, you have products around it, which not many do. Okay, what is interp? So basically you're trying to have an understanding of what's going on in model, like in the model, in the internals. So different approaches to do that.

You can do probing, SAEs, transcoders, all this stuff. But basically you have an hyp- you have a hypothesis, you have something that you wanna learn about what's happening in a model internals, and then you're trying to solve that.

From there you can do stuff like you can, you know, you can do activation mapping, you can try to do steering. There's a lot of stuff that you can do, but the key question is, you know, from input to output, we wanna have a better understanding of what's happening and, you know, how can we, how can we adjust what's happening on the model internals.

How'd I do?

That was really good.

I think that was great. I, I think it's also a... It's kind of a minefield of a... If you ask 50 people who, quote unquote, "work in interp," like what is interpretability, you'll probably get 50 different answers. And to some extent also, like where, where Goodfire sits in the space, I think that we're an AI research company above all else, and interpretability is a, is a set of methods that we think are really useful and worth kind of specializing in in order to accomplish the goals we want to accomplish.

But I think we also sort of see some of the goals as even more broader as, as almost like the science of deep learning and just taking a not black box approach to kind of any part of the, like, AI development life cycle, whether that means using interp for like data curation while you're training your model or for understanding what happened during post-training or for the, you know, understanding activations and sort of internal representations, what is in there semantically.

And then a lot of sort of exciting updates that we're... you know, are sort of also part of the, the fundraise around bringing interpretability to training, which I don't think has been done all that much before. A lot of this stuff is sort of post-talk poking at models as opposed to actually using this to intentionally design them.

Is this post-training or pre-training, or is, is that not a useful-

Primarily focused on post-training-

Yeah

... but there's no reason the techniques wouldn't also work in pre-training.

Yeah. It seems like it would be more act- applicable post-training because-

Mm-hmm

... basically I, I'm thinking like rollouts or like, um, you know, having different variations of a model that you can tweak with the-

Exactly

... with your steering.

Yeah. And I think in a lot of the news that you've seen in, in, on like Twitter or whatever, you've seen a lot of unintended side effects come out of post-training processes, you know, overly sycophantic models or models that exhibit-

Oh my God

... strange reward hacking behavior. I think these are like extreme examples. There's also, you know, very, uh, mundane, more mundane like enterprise use cases where, you know, they try to customize or post-train a model to do something and it learns some noise or it doesn't appropriately learn the target task.

And a big question that we've always had is like, how do you use your understanding of what the model knows and what it's doing to actually guide the learning process more effectively?

Yeah, I mean, uh, you know, just to anchor this for people, uh, la- one of the biggest controversies of last year was 4O Glazegate.

I've never heard Glazegate.

I didn't know that was what it was called.

The, the other one, they called it that on the blog post. And I was like, why did OpenAI call it, like officially use that term? And I'm like, that's funny. But like, yeah, I, I, I guess is, is the pitch that if they had worked with Goodfire they wouldn't have avoided it.

Like, you know, it's like

I, I think so. Yeah.

Yeah.

That's the pitch.

I, I think that's certainly one of the use cases. I think, and another reason why post-training is a place where this makes a lot of sense is a lot of what we're talking about is surgical edits. You know, you wanna be able to have expert feedback, very surgically change how your model is doing, whether that is, you know, removing a certain behavior that it has.

So, you know, one of the things that we've been looking at... Or is a, is another like common area where you would wanna make a somewhat surgical edit is some of the models that have, say, political bias. Like you look at Qwen or, um, R1 and they have sort of like this CCP bias in them and-

Is there a CCP vector?

Well, there's- ... there are certainly internal, yeah, parts of the representation space where-

Yeah, more or less

... you can sort of see where that lives. Yeah. Um, and you wanna kinda, you know, extract that piece out.

Well, obviously, you know, whenever you find a vector, a, a fun exercise is just like make it very negative-

Yeah.

... to see what's, what the opposite of CCP is.

The super America, bald eagles flying everywhere. But yeah, so i- in general, like lots of post-training tasks where you'd wanna be able to, to do that, w- whether it's unlearning a certain behavior or, you know, some of the other kinda cases where this comes up is...

Are you familiar with like the, the grokking behavior?

I mean, I know the machine learning term of grokking.

Yeah, sort of this like double descent idea of, of having a model that is able to learn-

Like aha

... a generalizing, a generalizing solution as opposed to e- even if memorization of some task would suffice, you want it to learn the more general way of doing a thing. And so, you know, another way that you can think about having surgical access to a model's internals would be learn from this data but learn in the right way, um, if there are many possible-

Hmm

... you know, ways to, to do that.

Can make interp solve the double descent problem?

Depends, I guess, on how you-

Okay. So I, I sh- I view dou- double descent as a problem.

Uh-huh.

Because then you're like, well, if the loss curves level out, then you're done but maybe you're not done.

Right. Right.

But like if you actually can interpret what is, uh, generalizing or what is, what is still changing even though the loss is not changing, then maybe you, you can actually not view it as a double descent problem and actually you're just kind of, uh, translating the space in which you view loss and y- like, and then you have a smooth curve.

Yeah. I, I think that's certainly, like, the domain of, of problems that we're-

Okay

... that we're looking to get at. Yeah.

To me, like, double descent is, like, the biggest thing to, like, ML research where, like, y- if you believe in scaling, then you don't... y- you need to know where to scale, and, uh, but if you believe in double descent, then you don't, you don't believe in anything where, like, anything levels off.

Like

Yeah.

I mean, also tangentially there's, like, okay, when you talk about the China vector, right? There's the subliminal learning work. It was from the Anthropic Fellows program, where basically you can have hidden biases in a model, and as you distill down or, you know, as you train on distilled data, those biases always show up, even if, like, you explicitly try to not train on them.

So, you know, it's just, like, another use case of, okay, if we can interpret what's happening in post-training, you know, can we clear some of this? Can we even determine what's there? Because, yeah, it's just, like, some worrying research that's out there that shows, you know, we really don't know what's going on.

That is... Yeah, I think that's the biggest sentiment that we're sort of hoping to tackle. Nobody knows what's going on, right? Like, subliminal learning is just an insane concept when you think about it, right? Train a model on not even the logits, literal, the, literally the output text of a bunch of random numbers, and now your model loves owls.

And you see behaviors like that that are just- ... they defy, they defy intuition. And, and there are mathematical explanations that you can get into, but-

I mean, there, there-

... it feels so early days

... objectively there are a sequence of numbers that are more owl-like than others. There, there should be.

Well, according, according to, according to certain models, right? It's interesting. I think it only applies to models that were initialized from the same starting seed.

Same-

Usually, yes.

Yeah, yeah.

But I mean, I think that's a, that's a cheat code because there's not enough compute, but, like, y- it sh- if you believe in, like, platonic representation, like, it probably will transfer across different models as well.

Oh, you think so? I think of it more as a statistical artifact of-

I see

... models initialized from the same seed, sort of, um, there's something that is, like, path dependent from that seed that might cause certain overlaps in the latent space, and then sort of doing this distillation, yeah, like, it, it pushes it towards, uh, having certain other tendencies.

Got it. Needs, needs more research.

I think there's, like, a bunch of these open-ended questions, right? Like, you can't train in new stuff during the RL phase, right? RL only reorganizes weights, and you can only do stuff that's somewhat there in your base model.

You're not learning new stuff, you're just reordering chains and stuff. But okay, my broader question is, when you guys work at an interp lab, how do you decide what to work on, and what's kind of the thought process, right?

'Cause we can ramble for hours, "Okay, I wanna know this, I wanna know that," but, like, how do you concretely... Like, you know, what's the workflow? Okay, there's, like, approaches towards solving a problem, right? I can try prompting, I can look at chain of thought, I can train probes, SAEs, but how do you determine, you know, like, okay, is this going anywhere?

Like, do we have set stuff? Just, you know, if you could talk about that.

Yeah. It's a really good question. I feel like we've always, um, at the very beginning of the company, thought about, like, let's go and try to learn what isn't working in machine learning today, whether that's talking to customers or talking to researchers at other labs, trying to understand, um, both where the frontier is going and where things are, are really not falling apart today, and then developing a perspective on how we can push the frontier using interpretability methods.

Um, and so, you know, even our chief scientist, Tom, spends a lot of time talking to customers and trying to understand what real-world problems are, and then taking that back and trying to apply the current state-of-the-art in interp to those problems, and then seeing where they fall down, basically.

And then, uh, using that, those failures or those shortcomings to understand what hills to climb when it comes to interpretability, uh, research. So, like, on the fundamental side, for instance, when we have done some work applying SAEs and, and probes, we've encountered, you know, some shortcomings in, in SAEs that we found a little bit surprising, and so have gone back to the drawing board and, and done work on, um, better foundational interpreter models, and a lot of our team's research is focused on what is the next evolution beyond SAEs, for instance.

And then when it comes to, like, control and design of models, you know, we tried steering with our first API and realized that it still fell short of black box techniques like prompting or fine-tuning, and so went back to the drawing board and we're like, "How do we make that not the case, and how do we improve it beyond that?"

And one of our researchers, Ekdeep, who just joined, is... Actually, Ekdeep and Atticus are, like, steering experts and, and have spent a lot of time trying to figure out, like, what is the research that enables us to actually do this in a much more powerful, robust way.

So yeah, the answer is, like, look at real-world problems, try to translate that into a research agenda, and then, like, hill climb on, on both of those at the same time.

Yeah. Mark has the steering CLI demo queued up, which we're gonna go into in a sec, but I always wanna double-click on when you drop hints like, "We found some problems with SAEs." Okay, what are they? You know?

And, uh, let- let's, let's, uh, then we c- then we can go into the demo.

Yeah, I mean, I- I'm curious if you have more thoughts here as well, 'cause you've done it in, in the healthcare domain. Um, but I think, like, for instance, when we do things like trying to detect behaviors within models that are harmful or, like, behaviors that a user might not want to have in their model, so hallucinations, for instance, harmful intent, PII, all of these things, we first tried using SAE probes for a lot of these tasks.

So taking the feature activation space from SAEs and then training classifiers on top of that, and then seeing how well we can detect the properties that we might want to detect in model behavior. And we've seen m- in many cases that probes just trained on raw activations seem to perform better than SAE probes, which is a bit surprising if you think that SAEs are actually also capturing the concepts that you would want to capture cleanly and more surgically.

And so that is an interesting observation. I don't think that is, like... I'm not down on SAEs at all. I think there are many, many things they're useful for. But we have definitely run into cases where I think the concept space described by SAEs is not as clean and, uh, accurate as, as we would expect it to be for actual, like, real-world, uh, downstream Performance, um, metrics

Fair enough

Yeah

Yeah. It's the blessing and the curse of unsupervised methods, where-

I was thinking that

... you get to peek into the AI's mind, but sometimes you wish that you saw other things when you when you went inside there

Yeah

Although in the, in the PII instance, I think where into SAE, an SAE-based approach actually-

Mm-hmm

... did prove to be the most-

It did work well-

... generalizable

... in, in the case that we published with Rakuten, and I think a lot of the reasons it worked well was because we had a noisier dataset. And so actually, the blessing of unsupervised learning is that we actually got, uh, to get more meaningful generalizable signal from SAEs when the data was noisy.

But in other cases where we've had, like, good datasets, it hasn't been the case.

And just because you named dropped Rakuten, and I don't know when, if we'll get another chance, like, uh, what, what is the overall, like, what is Rakuten's, uh, usage, uh, or sort of production usage?

Yeah. So they are using us to essentially guardrail and inference time monitor their language model usage and their agent usage to detect things like PII so that they don't route, um, private user information to, um, downstream model providers.

And so that's, you know, going through all of their user queries, uh, every day, and that's something that we deployed with them a few months ago. And now we are actually exploring, um, very early partnerships n- not just with Rakuten, but with other people around how we can help with potentially training and customization use cases as well.

Yeah. And for those who don't know, like, it's, Rakuten is, like, I think number one or number two e-commerce-

Yeah

... store in Japan.

Yes. Yeah.

Yeah.

And I think that use case actually highlights a lot of, like, what it looks like to deploy things in practice that you don't always think about when you're doing sort of research tasks. So when you think about some of the stuff that came up there that's more complex than your idealized version of a problem, they were encountering things like synthetic to real transfer of methods, so they couldn't train probes, classifiers, things like that on actual customer data of PII.

Ah.

So what they had to do is use synthetic datasets and then hope that that transfers out of domain to real datasets. And so we could evaluate performance on the real datasets but not, not train on customer PII. So that right off the bat is, like, a big challenge.

You have multilingual requirements, so this needed to work for both English and Japanese text. Japanese text has all sorts of quirks, including tokenization behaviors that caused lots of bugs that caused us to be pulling our hair out. And then also a lot of tasks you'll see, you might make simplifying assumptions if you're sort of treating it as, like, the easiest version of the problem to just sort of get, like, general results where maybe you say you're classifying a sentence to say, "Does this contain PII?"

But the need that Rakuten had was token level classification so that you could precisely scrub out the PII. So as we learned more about the problem, and you're sort of speaking about what that looks like in practice, yeah, a lot of assumptions end up breaking, and that was just one instance where you...

A problem that seems simple right off the bat ends up being more complex as you keep diving into it.

Excellent.

One of the things that's also interesting with Interp is a lot of these methods are very efficient, right? So where you're just looking at a model's internals itself compared to a separate, like, guardrail, LLM-as-a-judge, a separate model. One, you have to host it.

Two, there's, like, a whole latency, so if you use, like, a big model, you have a second call. Some of the work around, like, self-detection of hallucination, it's also deployed for efficiency, right? So thinking of someone like Rakuten doing it in production live, you know, that's just another thing people should consider.

Yeah, and something like a probe is super lightweight.

Yeah.

Adds no extra latency really.

Excellent. Um, you have the steering demos, uh, lined up, so we'll just kinda see what you got. Um, I, I don't, I don't actually know if this is, like, the latest, latest or, like, alpha thing.

No.

Okay.

This is a pretty hacky demo-

Okay

... from, uh, from a presentation that someone else on the team recently gave. So this will give a sense for, for steering in action. Honestly, I think the biggest thing that this highlights is that as we've been growing as a company and taking on kind of more and more ambitious versions of interpretability related problems, a lot of that comes to scaling up in various different forms.

And so here you're gonna see steering on a one trillion parameter model. This is Kimi K2. Uh, and so it's sort of fun that in addition to the research challenges, there are engineering challenges that we're now tackling, 'cause for any of this to be sort of useful in production, you need to be thinking about what it looks like when you're using these methods on frontier models, um, as opposed to sort of like toy kind of model organisms.

So yeah, this was thrown together hastily, pretty fragile behind the scenes, but, uh, I think it's quite a fun demo. So screen sharing is-

Is on

... is on. So I've got two terminal sessions pulled up here. On the left is a forked version that we have of the Kimi, uh, CLI that we've got running to point at our custom hosted Kimi model, and then on the right is a, uh, setup that will allow us to steer on certain concepts.

So I should be able to chat with Kimi over here. Tell it hello.

This is running locally?

So the CLI is running locally, but the Kimi server is running back in the office. Well, sh- hopefully should be. Um-

That's too much to run on that Mac.

Yeah. I think it's, uh... It takes a full, like, H100 node. I think it's like you can run it on eight GPUs, H100s. So, so yeah. Kimi's running. We can ask it a prompt. It's got a forked version of our, uh, of the SGLang code base that we've been working on.

So I'm gonna tell it, "Hey, this SGLang code base is slow. I think there's a bug. Can you try to figure it out?" It's a big code base, so it'll, it'll spend some time doing this. And then on the right here, I'm going to initialize in real time some steering.

Let's see here. Continue Searching for any bugs. Looks like-

Feature ID 43205-

Yeah, yeah

... layers 20, 30, 40.

So let me, uh ... Th- this is basically a feature that we found that Inside Kimi seems to cause it to speak in Gen Z slang. And so on the left, it's still sort of thinking normally.

Yeah.

It might take, I don't know, 15 seconds for this to kick in. But then we're gonna start hopefully seeing it... Dude, this code base is massive for real. So we're gonna start seeing Kimi transition as the steering kicks in from normal Kimi to Gen Z Kimi.

And both in its chain of thought and its actual outputs.

And interestingly, you can see, you know, it's still able to call tools, uh, and stuff. It's, um, it's purely sort of its, its demeanor. And there are other features that we found for interesting things like concision, so that's more of a practical one.

You can make it more concise, um, the types of programs, uh, programming languages it uses. But yeah, as we're seeing it come in, pretty good outputs.

Scheduler code is actually wild. Um-

Sabotaging its own NGL.

I love some Yoda.

Yoda's code is actually insane, bruh.

Be cringe, NGL. What's the process of training an SAE on this or, you know, how do you label features? I know you guys put out a pretty cool blog post about, um, finding this, like, autonomous interp, um, something about how agents for interp is different than, like, coding agents.

I don't know, while this is spewing up, how, how do we find feature 43205?

Yeah. So in this case, um, we- our platform that we've been building out for a long time now supports all the sort of classic out-of-the-box interp techniques that you might wanna have, like SAE training, probing, things of that kind.

I'd say the techniques for, like, vanilla SAEs are pretty well established now, where you take your model that you're interpreting, run a whole bunch of data through it, gather activations, and then, yeah, pretty straightforward pipeline to train an SAE.

There are a lot of different varieties. There's top K SAEs, batch top K SAEs, um, normal ReLU SAEs. And then once you have your sparse features, to your point, assigning labels to them to actually understand that this is a Gen Z feature, uh, that's actually where a lot of the kind of magic happens.

And the most basic standard technique is look at all of your da- input dataset examples that cause this feature to fire most highly, and then you can usually pick out a pattern. So for this feature, if I've run a diverse enough dataset through my model, feature 43205 probably tends to fire on all the, uh-

Gen Z

... tokens that sound like Gen Z slang. And, um, so you know, you could have a human go b- through all 43,000 concepts and look at the pattern, but to automate that, you just kind of hand those examples off to a, a frontier LLM and ask it to identify that pattern.

And I've got to ask the basic question, you know, can we get examples where it hallucinates, pass it through, see what feature activates for hallucinations? Can I just, you know, turn hallucination down?

Oh, wow, you, you really, uh-

I solved it

... you really predicted some- ... a, a project we're already working on right now, which is, um, detecting hallucinations using interpretability techniques. And this is interesting 'cause hallucinations is something that's very hard to detect, and it's, like, a kind of a hairy problem and something that black box methods really struggle with.

Um, whereas, like Gen Z, you could always train a classifi- a simple classifier to, to detect that. Um, hallucinations is harder, but we've seen that models internally have some awareness of, of, like, uncertainty or some sort of, like, user-pleasing behavior that leads to hallucinatory behavior.

Um, and so yeah, we have a project that's trying to detect that accurately and then also working on mitigating the hallucinatory behavior in the model itself as well.

Yeah.

And-

I, I would say most people are still at the level of like, "Oh, I would just turn temperature to zero," and that turns off hallucination. And I'm like, "Well, that's a fundamental misunderstanding of-

Right

... how this works."

Yeah. Although I... So I... Part of what I like about that question is you c- there are SAE-based approaches that might, like, help you get at that. But oftentimes the beauty of SAEs and, like we said, the curse is that they're unsupervised.

So when you have a behavior that you deliberately would like to remove, and that's more of, like, a supervised task, often it is better to use something like probes and, and specifically target the thing that you're interested in reducing as opposed to sort of like hoping that when you fragment the latent space, one of the vectors that pops out will be the thing you're interested in.

And as much as we're training an autoencoder to be sparse, we're not, like, for sure certain that, you know, we will get something that just correlates to hallucination. Or you'll, you'll probably split that up into 20 other things, and w- who knows what they'll be.

Of course, right. Yeah, so there's known sort of problems with, like, feature splitting, um, and feature absorption. And then there's the off-target effects, right? Ideally, you would want to be very precise where if you reduce the hallucination feature, suddenly maybe your model can't write creatively anymore, and maybe you don't like that, but you want to still stop it from hallucinating facts and figures.

Good. Uh, so P- Vivo has a paper to recommend there that we'll put in the show notes. But yeah, I, I mean, I guess just because your demo is done, uh, any, any other things that you want to highlight or any other interesting features you want to show?

I don't think so. Yeah, like I said, this is a pretty small snippet. I think the main sort of point here that I think is exciting is that there's not a whole lot of interp being applied to models quite at, at this scale.

You know, Anthropic certainly has some, some research and, yeah, other, other teams as well. But it's, it's nice to see these techniques, you know, being put into practice. I think not that long ago, the idea of real-time steering of a trillion-parameter model would have sounded-

Yeah

... outrageous.

The fact that it's real time, like you, you started the thing and then you edited, uh, the steering vector, I think it's, it's an interesting one. Uh, TBD what the actual, like, production use case would be on that, like the real-time editing.

Um-

It's... Like, that's the fun part of the demo, right? You can kind of see how this could be served behind an API, right? Like-

Yes

... you're- you only have so many knobs, and you can just ... tweak it a bit more. And I don't know how it plays in. Like, people haven't done that much with, like, how does this work with or without prompting, right?

How does this work with fine-tuning? Like, there's a whole hype of continual learning, right? So there's just so much to see. Like, is this another parameter? Like, is it, like, parameter we just kind of leave it as a default, we don't use it?

So I don't know, maybe someone here wants to put out a guide on, like, how to use this with prompting, when to do what.

Oh, well, I have a paper recommendation that I think you would love from, uh, Ekdeep on our team, who, um, is an amazing researcher. Just can't say enough amazing things about Ekdeep. But he actually has a paper that a- as well as some, you know, others from the, from the team and elsewhere, that go into the essentially equivalence of activation steering and in-context learning, and how those are from a...

He, he thinks of everything in a cognitive neuroscience Bayesian framework. But basically how you can precisely show how prompting, in-context learning, and steering exhibit similar behaviors, and even, like, get quantitative about the, like, magnitude of steering you would need to do to induce a certain amount of behavior similar to certain prompting, uh, even for things like jailbreaks and stuff.

It's a really cool paper.

Are you saying steering is less powerful than prompting?

More like you can almost write a formula that tells you how to-

Like a one-to-one

... convert between the two-

Yeah

... of them. And so, like-

They should be formally equivalent, actually, in the, in the limit.

Right. So, like, one case study of this is for jailbreaks. They're... I don't know, have you seen the stuff where you can do, like, many shot jailbreaking? You, like-

Yeah

... flood the context with examples of the behavior?

When Taki put out that paper, a lot of people were like, "Yeah, we've been doing this, guys." Like- "We don't need a paper to tell-"

Yeah. Um, what's in this in-context learning and activation steering equivalence paper is you can, like, predict the number of examples that you will need to, to put in there in order to jailbreak the model-

That's cool

... by doing steering experiments and using this sort of like, um, equivalence mapping.

That's cool. That's really cool.

It's very neat.

Yeah, and I was gonna say, like, uh, you know, I can, like, back rationalize that this makes sense because, you know, what context is, is basically just, y- you know, it updates the KV cache kind of. A- and, like, a- and, and, and then every next token inference is still, like, you know, the, the, the, the sheer sum of everything all the way.

It's plus all the context up to date. And you could, I guess, theoretically steer that with, uh, you could probably, uh, replace that with your, um, steering. The only problem is steering typically is on one layer, maybe three layers, like, like you did.

So it's, like, not exactly equivalent.

Right. Right. There's sort of you need to, uh, get precise about, yeah, like, how you sort of define steering and, like, what... how, how you're modeling the setup. But yeah, I've got the paper pulled up here, Belief Dynamics Reveal the Dual Nature.

Yeah, the title is Belief Dynamics Reveal the Dual Nature of In-Context Learning and Activation Steering. So Eric Bigelow, um, Daniel Wergaft on the, um, who are, uh, doing fellowships at Goodfire. Ekdeep's the, the final author there.

I think actually to your, your question of, like, what is the production use case of steering, I think maybe if you just think, like, one level beyond steering as, as it is today. Like, imagine if you could adapt your model to be, you know, an expert legal reasoner, like, in almost real time, like, very efficiently using human feedback or using, like, your semantic understanding of what the model knows and where it knows that, uh, behavior.

I think that while it's not clear what the product is at the end of the day, it's clearly very valuable. Thinking about, like, what's the next interface for, for model customization and adaptation is a really interesting, um, problem for us.

Like, we have heard a lot of people actually interested in fine-tuning and RL for open weight models in, in production, and so people are using things like, um, Tinker or kind of like, uh, open source libraries to do that.

But it's still very difficult to get models fine-tuned in RL for exactly what you want them to do, unless you're an expert at, at model training. And so that's, like, something we're looking into.

Yeah. Um, I never thought... So, uh, Tinker from Thinking Machines famously uses, uh, Rank 1 LoRA. Is that basically the same as steering? Like, you know, what's the comparison there?

Well, so i- in that case, you are still applying updates to the parameters, right-

Yeah

... as opposed to being-

You're not, you're not touching a base model. You're, you're touching an adapter-

It's kind of-

A rank one. Yeah.

Right. But I, I guess it still is, like, more in parameter space than... I guess it's maybe, like, are you modifying the pipes, or are you modifying the water flowing through the pipes-

Okay

... to get at what you're after?

Yeah.

Just maybe one way.

I like that analogy.

That, that's my mental-

That's a good one

... map of it at least. But it gets at this idea of model design and intentional design, which is something that we're, that we're very focused on.

Yeah.

And just the fact that, like, I hope that we look back at how we're currently training models and post-training models and just think what a primitive way of doing that right now. Like, there's no intentionality really in-

It's just data, right? The only thing we can control is-

It's just data

... what data we feed in.

So, so Dan from Goodfire likes to use this analogy of, um, you know, he has a couple of young kids, and he talks about, like, "What if I could only teach my kids how to be good people by giving them cookies or, like, you know, giving them a slap on the wrist if they do something wrong?"

Like, not telling them why it was wrong or, like, what they should have done differently or something like that, just-

Yeah, figure it out

... they take action. Right. Exactly.

So that's RL.

Yeah. Right. And then, you know, it's sample inefficient. There's, you know, what do they say? It's like slurping, uh, feedback. It's like-

Supervision

... through a straw.

Through a straw. Yeah.

Slurping supervision through a straw.

The Andrej shadow there.

Right. And so we'd like to get to the point where you can have experts giving feedback to their models that are, uh, internalized-

High quality. Yeah

... and, and, you know, steering is An inference time way of sort of getting that idea, but ideally you're moving to a, a world where it is much more intentional design-

Interesting

... in perpetuity for these models.

Okay. Okay. This is one of the questions we asked Emmanuel from Anthropic on the podcast a few months ago. Basically, the question was, "You're at a research lab that does model training, foundation models, and you're on an interp team.

How does it tie back, right?" Like, does this... Do ideas come from the pre-training team? Do they go back? Um, you know, so for those interested, you can, you can watch that. There wasn't too much of a connect there, but it's still something.

You know, it's something they wanna push for down the line.

It can be useful for all of the above. Like, there are certainly post-hoc use cases where it doesn't need to touch that.

I think the other thing a lot of people forget is this stuff isn't too computationally expensive, right? Like, I would say if you're interested in getting into research, MechInterp is one of the most approachable fields, right? A lot of this training SAE, train a probe, this stuff, like the budget for this, one, there's already a lot done.

There's a lot of open source work. You guys have done some, too. Um, you know, you can- There's like, there's like notebooks from the Gemini team, from Neil Nanda-

Yeah

... who are like, "This is how you do it. Just stick to the notebook."

True.

Even if you're, like, not even technical with any of this, you can still make, like, progress there. You can look at different activations. But, uh, if you do wanna get into training, you know, training this stuff, correct me if I'm wrong, is, like, in the thousands of dollars, not even, like, it's not that high scale.

And then same with, like, you know, applying it, doing it for post-training RL. All this stuff is fairly cheap in scale of, "Okay, I wanna get into, like, model training. I don't have compute for, like, you know, pre-training stuff."

So it's, it's a very nice field to get into. And also there's a lot of, like, open questions, right? There's so many questions we have. Some of them have to go with, "Okay, I want a product. I wanna solve this."

Like, there's also just a lot of open-ended stuff that people could work on that's interesting, right? I don't know if you guys have any calls for, like, what's open questions, what's open work that you either open collaboration with or, like, you'd just like to see solved, or just, you know, for people listening that wanna get into MechInterp, 'cause people always talk about it, what are, what are things they should check out, start?

Of course, you know, join you guys as well. I'm sure you're hiring.

There's a paper, I think, from, was it Lee, uh, Sharkey? It's Open Problems in Interpretability, which I recommend everyone who's interested in the field read. Um, just, like, a really comprehensive overview of what are the things that experts in the field think are the most important problems to be solved.

I also think, to your point, it's been really, really inspiring to see, I think, a lot of young people getting interested in interpretability. Actually, not just young people, also, like, scientists who have been, you know, experts in physics for many years and in biology or things like this, um, transitioning into interp because the barrier to entry is, is, you know, in, in some ways low, and, and there's a lot of information out there, um, and ways to get started.

There's this anecdote of, like, professors at university saying that all of a sudden every incoming PhD student wants to study interpretability, um, which was not the case a few years ago. So it just goes to, to show how, I guess, like, exciting the field is, how fast it's moving, how quick it is to get started and things like that.

And also just a very welcoming community. You know, there's an open source MechInterp Slack channel. There are people are always posting questions, and just folks in the space are always responsive if you ask things on various forums and stuff.

Um, but yeah, the open paper, uh, open problems, uh, paper is, is a really good one.

For other people who wanna get started, I think, you know, MATS is a great program. What's, what's the acronym for? Machine Learning and Alignment Theory Scholars, I think.

Yeah. It's the schol- it's like the normally summer-

Yes

... internship style program.

Yeah, but they've been doing it year-round now.

Yeah.

And, um, actually a lot of our full-time staff have come through that program or, or gone through that program and, um, it's great for anyone who is transitioning into interpretability. Um, there's a couple other fellows programs. We do one as well as Anthropic, and so those are great places to get started if anyone is, is interested.

Also, I think been seen as a research field for a very long time, but I think engineers are s- sorely wanted for- ... for interpretability as well, um, especially at Goodfire, but, but elsewhere, uh-

Yes

... as, as it does scale up.

I should mention that, uh, Lee actually works with you guys, right? Like, in, in the London office, and, um, I'm adding our first ever MechInterp track in at AI Europe because I, I see this industry applications now emerging, and I'm pretty excited to, you know, help get, uh, push that along.

Yeah. I, I was-

It'll, it'll-

... thrilled to hear that

... effectively be the first industry MechInterp conference.

Yeah.

Um, so.

I'm, uh, I'm so glad you added that.

You know, it's, it's a lit- still a little bit of a bet. Like, there's, there's, it's not, like, that widespread.

Mm-hmm.

But, like, I, I can definitely see, like, this is the time to, like, really get into it.

Mm-hmm.

Like, you, you wanna be early on things.

Mm-hmm.

For sure. And I think the field is, it, it understands this, right? So, like, at ICML, the, I think, like, title of the MechInterp workshop this year was Actionable Interpretability, and there was- ... a lot of discussion around bringing it to, to various domains.

Um, and-

Everyone's, everyone's adding, like, pragmatic-

Pragmatic interp

... or actionable-

Right

... like, whatever. It looks like, okay, well, we weren't actionable before, I guess. I don't know.

And I mean, like, just, you know, being at Europes, you see the interp room. Uh, one, like, old school conferences, like, I think they had a very tiny room till they got lucky, and they got it doubled, but there's definitely a lot of interest.

A lot of niche, niche research, so you see a lot of research coming out of university students. We covered the paper, uh, last week. It's, like, two unknown authors, not many citations, but you know, you can make a lot of meaningful work there.

One thing I, I, I did want to call out because I think people haven't really, uh, mentioned this yet, is, uh, just interp for code, uh, I think is, is, like, an abnormally important field. We haven't mentioned this yet.

The conspiracy theory last two years ago was when the first SAE work came out of Anthropic was they would do, like, "Oh, we just used, uh, uh, SAEs to turn the bad code vector down." Uh, and then turn up the good-

And then quad code came out

... turn up the good code.

Yeah.

Uh, and, uh, I think, like-

Isn't that the, the dream? Like, you know, like, but basically, I, I guess maybe why is it funny? Like, it's, it's... If it was realistic, it would not be funny. It would be like, "No, actually, we should do this."

But it's funny because we know there's like, we feel there's some limitations to what steering can do, and I think a lot of the public image of, uh, steering is, like, the Gen Z stuff. Like, like, oh, you can make it really love the Golden Gate Bridge, or you can make it speak like Gen Z.

To, like, be a legal reasoner seems like a huge stretch.

Yeah.

And I don't know if that will get there this way.

Yeah. I think, um, I will say we are announcing something very soon that I will not speak-

Okay

... too much about. Um, but I think, yeah, this is, like, what we run into again and again is, like, we, we don't want to be in the world where steering is only useful for, like, stylistic-

St- stylistic things

... things. That's definitely not, not what we're, um, aiming for. But I think the types of interventions that you need to do to get to things like legal reasoning, um, are much more sophisticated and require breakthroughs in, in learning algorithms, and that's, um-

And is this an emergent property of scale as well?

I think so. Yeah, I mean, I think scale definitely helps. I think scale allows you to learn a lot of information and, and reduce noise across, you know, large amounts of data. But I also think we think that there's ways to do things much more effectively, um, even, even at scale, so, like, actually learning exactly what you want from the data and not learning things that you do, uh, that you don't want exhibited in the data.

So we're not, like, anti-scale, but we are also realizing that scale is not going to get us to the type of AI development that we want to be at in, in the future as these models get more powerful and get deployed in all these sorts of, like, mission-critical contexts.

Current life cycle of training and deploying and evaluations is, is to us, like, deeply broken and, and has opportunities to, to improve, so, um, more to come on that very, very soon.

And I think the, the SAE is basically maybe just, like, a proof point that these concepts do ex- like, if you can manipulate them in the precise best way, you can get the ideal combination of them that you desire.

And steering is maybe the most coarse-grained sort of peek at what that looks like.

Yeah.

But I think it's evocative of what you could do if you had total surgical control over-

Mm-hmm

... every concept and feature.

Every parameter, yeah.

Exactly.

There, there were, like, bad code features in there.

I've got it pulled up.

Yeah, yeah.

Just coincidentally as you guys are talking, so-

This, this was like, this is exactly why people would prod it, right?

Yeah, yeah, exactly.

There's, like, specifically a code error feature-

Yeah

... that activates-

So just turn it off

... and they shut it off.

Like turn it off.

It's not, it's not typo detection. It's like, it's, it's typos in code. It's not typical typos, and you know, you can, you can see-

Yeah

... it clearly activates where there's something wrong in code.

Yeah.

And they have, like, malicious code, code error. They have a whole bunch of sub, you know, sub broken-down little grain features.

Yeah.

Yeah, so, so the, the rough intuition for me, the why I talked about post-training was that, well, you just, you know, have a few different rollouts with all these things, uh, turned off and on and whatever, and then, you know, you can...

That's, that's synthetic data you can kind of post-train on.

Yeah.

And I think we make it sound easier than it is- ... just saying, you know, do... They, they do the real hard work.

I mean, you guys, you guys have the right idea, exactly. Yeah, we replicated a lot of these features in, in our Llama models as well. I remember there was, like-

And I think a lot of this stuff is open, right?

Yeah.

Like, yeah, you guys opened yours. Um, DeepMind has opened a lot of essays on, um, Gemma. Even Anthropic has opened a lot of this. There's, there's a lot of resources that, you know, we can probably share-

Yeah

... of people that wanna get involved.

Yeah.

And special shout-out to, like, Neuronpedia as well.

Mm-hmm.

Yes.

Like, uh, yeah, amazing piece of work-

Mm-hmm

... to visualize those things.

Yeah, exactly.

I guess I wanted to pivot a little bit on, onto the healthcare side, uh, because I, I think that's a big use case for you guys. Uh, we haven't really talked about it yet. It is a bit of a crossover for me because we are se- we are, we do have a separate science pod that we're starting up for-

Oh.

No way

... for AI, for AI for science.

Wow, okay.

Uh, just because, like, it's such a huge investment category.

Yes.

And also I'm, like, less qualified to do it. We actually have bio PhDs to cover that, which is great. But I mean, to just kind of recover- recap your work maybe on the EVL 2 stuff, but then, and then building forward.

Yeah, for sure. And maybe to frame up the conversation, I think another kind of interesting just lens on interpretability in general is a lot of the techniques that we're describing are ways to solve the AI-human interface problem, and it's sort of like bi-directional communication is the goal there.

So what we've been talking about with intentional design of models and, you know, steering, but also more advanced techniques, is having humans impart our desires and control into models and over models. And the reverse is also very interesting, especially as you get to superhuman models, whether that's narrow super intelligence, like these scientific models that work on genomics data, medical imaging, things like that.

But down the line, you know, super intelligence of other forms as well. What knowledge can the AIs teach us as sort of that, the, the other direction in that? And so some of our life science work to date has been getting at exactly that question, which is, well, some of it does look like debugging these various life sciences models, understanding if they're actually performing well on tasks or if they're picking up on spurious correlations.

For instance, genomics models, you would like to know whether they are sort of focusing on the biologically relevant things that you care about or if it's using some simpler correlate like the ancestry of the person that it's looking at.

But then also in the instances where they are superhuman and maybe they are understanding elements of the human genome that we don't have names for or- ... um, specific, you know, yeah, discoveries that they've made that, that we don't know about, that's, that's a big goal.

And so we're already seeing that, right? We are partnered with organizations like Mayo Clinic- Leading research health system in, in the United States. Our institute as well as, uh, a startup called Premomenta, which focuses on neurodegenerative disease. And in our partnership with them, we've used foundation models they've been training and applied our interpretability techniques to find novel biomarkers for, um, Alzheimer's disease.

So I think this is just the tip of the iceberg, but it's, uh... That's like a flavor of some of the things that we're working on.

Yeah. I think that's, uh, really fantastic. Uh, we- obviously we did the Chad Zuckerberg, uh, pod, uh, last year as well, and, like, there's gonna be a plethora of these models coming out because there's so much, um, potential in research.

Um, and it's, like, very interesting how it's basically the same as language models but just with a different f- underlying dataset, but, like, it's the same exact techniques. Like, there's no change basically.

Yeah. Well, and even in, like, other domains, right? Like, I-I, you know, robotics I know, like, a lot of the companies just use Gemma as, like, the, the, like, backbone, and then they-

Oh, yeah

... like, make it into a VLA that, like, takes these actions. It's, it's, it's Transformers all the way down.

I mean-

So yeah, all those techniques

... re- like we j- we have Med-Gemma now, right?

Right.

Like, this week even there was Med-Gemma 1.5. Uh, and they're, they're training it on this stuff, like 3D scans, medical domain knowledge, and all that stuff too. So there's a push from both sides, but I think the thing that, you know, one of the things about MechInterp is, like, you're a little bit more cautious in some domains, right?

So healthcare mainly being one. Like, guardrails understanding, you know, we're, we're more risk adverse to something going wrong there. So even just from a basic understanding, like, if we're trusting these systems to make claims, we wanna know why and what's going on.

Yeah. I think there's totally a kind of like deployment bottleneck to actually using foundation models for real patient usage or things like that. Like, say, you're using a model for rare disease prediction, you probably want some explanation as to why your model predicted a certain outcome and an interpretable explanation at that.

So that's definitely a use case. But I also think, like, being able to extract scientific information that no human knows to accelerate drug discovery and disease treatment and things like that actually is a really, really big unlock for sci- like scientific discovery, and you've seen a lot of startups, like, say that they're gonna accelerate scientific discovery, and I feel like we actually are doing that through our Interp techniques and kind of like almost by accident.

Like, I think we got reached out to very, very early on from these healthcare institutions, and none of us had healthcare backgrounds.

How did they even hear of you? Like-

A podcast.

Oh, okay.

Yeah.

Yeah, podcasts are- Okay, well, now is that time, you know, for-

I'm like, "Everyone, everyone can call us up."

Podcasts are the most important thing. Everyone should listen to podcasts, and everyone should come onto podcasts.

Yeah, and they reached out. They were like, "You know, we have these really smart models that we've trained, and we wanna know what they're doing." Um, and we were, like, really early at that time, like three months old, and it was a few of us, and we were like-

Yeah, what the hell?

... "Oh, my God. Let's, like- ... We've never used these models. Let's figure it out." Um, but it's also, like, great proof that Interp techniques scale pretty well across, across domains. We didn't really have to learn too much about-

Yeah. Interp is a, a machine learning technique. Machine learning scales everywhere, right?

Yeah.

Like, and then it's obviously-

Exactly

... it's, it's just like a general, uh, insight.

Yeah.

Um, probably to finance too, I think, which would be fun-

Mm-hmm

... for our history.

Mm-hmm.

Um, I don't know if you have anything to say there for finance.

Yeah. Well, just, uh, across the sci- Like, we've also-

Yeah

... done work on material science. Yeah, it, it really runs the gamut.

Yeah. Awesome. And, you know, for those that should reach out, like, you're obviously experts in this, but, like, is there a call-out for people that you're looking to partner with, design partners, people to use your stuff? Outside of just, you know, the general developer that wants to plug and play steering stuff, like, on the research side more so, like, are there ideal design partners, customers, stuff like that, that should reach out?

Yeah, I can talk about maybe non-life sciences, and then I'm curious to hear from you on the life sciences side. But, um, we're looking for design partners across many domains. Language mo- anyone who's customizing language models or trying to push the frontier of code or reasoning models is really interesting to us, and then also interested in the frontier of models that work in, like, pixel space, as we call it.

So if you're doing world models, video models, um, even robotics, um, where there's not a very clean natural language interface to interact with, um, I think we think that Interp can really help and are looking for a few partners in, in that space.

Just 'cause you, you mentioned the keyword world models, um, is that a big part of your thinking? Do you have a definition that I can use? Because everyone's asking me about it.

Uh, about world models?

There's quite a few definitions, I'd say.

I don't feel equipped to be a, an expert on world model definitions, but the reason we're interested in them is because they give you, like, you know, with language models, when you get, uh, features, you still have to do auto-Interp and things like that to actually get an understanding of, of what this concept is.

But in image and video and world, it's, like, extremely easy to, to grok what the concept is because you can see it and you can visualize it, and, um, this makes the feedback cycle extremely fast, um, for us, and also for things like, I don't know.

If you take-- think about probes in language model context and then take it to world models, like, what if you wanted to detect harmful actors in world model scenes? Like, you can't actually, like, go and label all of that data feasibly.

But maybe you could, um, synthetically generate, you know, I don't know, world, like, harmful actor, uh, data using SAE feature activations or whatever and then actually train a probe that was able to detect that much more scalably. So I just think, like, video and image and world has always been something we've explored and are continuing to explore.

Um, Mark's demo was probably the first moment we really, like- We're like, "Oh, wow, like this is really gonna... This could really, like, change the world of-"

The, the steering demo?

Yeah. No, the image demo.

The, the, the diffusion one, yeah.

Yeah, yeah, exactly.

Yeah. We should probably show that. And, and you demoed it at World's Fair, so-

Yeah.

We can just link that.

Nice. Yeah.

Yeah, yeah.

People can play with it, right?

Yes.

Yeah, it's still up.

paint.goodfire.ai. Yeah.

Yeah. I think for, for me, one way in which I, I think about world models is just, like, this, like, having this consistent model of the world where everything i- that you generate operates within the rules of that world.

And imagine it would be a bigger deal for science, uh, or, like, math or anything that where, like, you have verifiable rules. Whereas I guess in natural language, maybe there's less rules, and so it's not, not that important.

Yeah. And, and which makes the debugging of the model's internal representations, uh, or its internal world model, to the extent you can make that legible and explicit and have control over that, I think it makes it all the more important.

'Cause in language, it's sort of a fuzzy enough domain that-

Yeah

... if its world model isn't fully like ours, it can still sort of, like, pass the Turing test, so to speak. But I know there have been papers that have looked at, like, even if you train certain astrophysics models, it does not learn F equals ma.

Like, the same way that you can, you know, to have a model do well for modular arithmetic-

Mm-hmm

... but it doesn't really, like, learn what... how, how we think of modular arithmetic. It learns some crazy heuristic that is, like, essentially functionally equivalent, but it's probably not the sort of grokked solution that you would hope for.

It's how an alien would do it.

Right.

I don't know.

Right.

You know?

Right. Exactly.

Uh, but, but no, no, I think that's probably, I think, a, a function of our learning being bad rather than the... well, the, that approach probably not being real because it's, it's how we humans learn.

Right.

Yeah, right.

Well, it's just, it's the problem of induction, right? All of ML is based on induction and, and it's impossible to say I have a physics model. You might have a physics model that works all the time except when there is a character wearing a blue shirt and green shoes, and, like, you, you can't disprove that that's the case unless you test every particular situation your model might be in.

Yeah.

Um, so we know that the laws of physics apply no matter where you are, what scenario it is. But from a model's perspective, maybe something that's out of distribution, it just never needed to learn that the same laws of physics apply there.

Yeah. You were very excited because I, I read Ted Chiang over the holidays, um, and I was very inspired by this, uh, short story called Understand, uh, which apparently is, like, pretty old. You, you must be familiar with it.

To me, it was like, it's, it's this fictional story. It's like the inverse of Flowers for Algernon-

Yeah

... where, uh, you had someone, like, uh, get really smart but then also try to outsmart the tester. And the story just read like the chain of thought of a, of a super intelligence.

Yeah.

Right? Where they're like, "Oh, I realize I'm being tested, therefore..." And then, okay, what's the consequence of being tested? "Oh, they're testing me, and if I score well, they will use me for things that I don't wanna do.

Therefore, I will score badly, and I, and, like, but not too badly that they will raise alarm." So -

Yeah

... this is like, uh, so model sandbagging is a thing that people have, have explored. But I just think, like, uh, Ted Chiang's work just in general seems to be something that inspires you. I just wanted to prompt you to talk about it.

I think... So Ted Chiang has two... I- is a, is a sci-fi author who writes amazing short stories and-

His, his other claim to fame is, uh, Stories of Our Lives, which became, uh, the movie Arrival.

Exactly, yeah. So, so two books of short stories that I'm aware of. He also actually also has a great, um, just online blog post. I think he's the one who coined the term of LLMs as, like, a blurry JPEG of the internet.

Oh.

I should fact-check that, but he- It's a good, it's a good post. But I think almost every one of his short stories has some lesson to bear on thinking about AI and thinking about AI research. So, you know, you- you've been talking about alien intelligence, right, and this AI human communication translation problem.

That's, you know, exactly sort of what's going on in Arrival, in Story of, um, Story of Your Life, and just the fact that other beings will think and operate and communicate in ways that are not just challenging for us to understand but just fundamentally different in ways that we might not even be able to expect.

And then the one that's just super relevant for interpretability is the other short book of short stories he has is called Exhalation.

Yes.

And that is literally about a robot doing interpretability on its own mind.

Oh, okay.

Uh, so I just think that that, you know, you don't, you don't even have to squint to make the analogies there.

Well, I, I actually take, uh, Exhalation as a discussion about entropy and-

Yes

... order. Uh, but yes, uh, there's this, there's a scene in Exhalation where basically the, the, uh... So everyone is a robot. Uh, so the, the, the guy realizes he can set up a mirror to work on the back of his own head, and then starts doing operations like that and looking in the mirror and doing, doing this.

Yeah. And I think Ted Chiang has written about, like, the inspiration for that story. It was, like, half inspired by some of the things he had been doing on entropy. There's apparently some other short story that is similar where a character goes to the doctor and opens up his chest, and there's, like, a, like, a ticker tape going along.

He... It's like he basically realizes he's, like, a Turing machine. Um, and I don't know. I, I think especially as it comes to using agents for interp, that story always sticks in my mind.

I find the, uh, brain surgery or, like, surgery analogies a little bit, a little bit morbid, but it is very apt. Um, and when we talk to a lot of computational neuroscientists, they moved to interp because they were like, "Look, we have unfettered access to this artificial intelligent mind.

It's so mu- You have access to everything. You can run as many ablations and experiments as you want. It's an amazing bed for science." And, um, you know, human brains, obviously, we can't just go and do whatever we want to them.

Um, and, and I think, uh, it is really just, like, a moment in time where we have intelligent systems that can really, like- Do things better than humans in many ways, and, um, it's time, I think, for us to, to do the science on it.

I'll ask a, a brief, like, safety question. You know, uh, MechInterp was kind of born out of the alignment and safety conversation. I, I... Safety is on your website. It's not, like, something that you, you, like, de-prioritize. But, uh, like, there's, like, a sort of very militant safety arm that, like, wants to blow up data centers and- ...

like, stop AI. And, and then there's this, like, sort of middle ground, and, like, is, is this, like, a conversation in your part of the world? Do you go out to Berkeley and Light Haven- ... and, like, talk to those guys?

Or are they, like... You know, there's, like, a brief, like, civil war going on right now?

I think, I think a, a, a good amount of us have spent some time in Berkeley. Um, and then there are researchers there that we really admire and respect. Um, I think for us it's, like, we have a very grounded view of alignment and, and safety in that we want to make sure that we can build models that do what we want them to do and that we have scalable oversight into what these models are doing.

And, and we think that that is the key to a lot of these, like, technical alignment challenges, and I think that is our opinion. That's our research direction. Um, we, of course, are going to do safety-related research to make sure that our techniques also work on, you know, things like reward hacking and, and other, like, more concrete safety issues that we've seen in the wild.

But we want to be kind of, like, grounded in solving the technical challenges we see to having humans be... Humans play a big role in, in the deployment of, of these in- super intelligent agents of the future.

Yeah. I've, I've found the community to actually be remarkably cohesive, whether it's talking about academia or the interpretability work being done at the frontier labs or some of the independent programs like maths and stuff. I think we're all shooting for the same goal.

I don't know that there's anyone who doesn't want our understanding of models to increase. I, I think everyone, regardless of where they're coming from or the use cases that they're thinking-

Hmm

... whether it's alignment as the premier thing they're focused on or someone who's coming in purely from the angle of scientific discovery, I think we would all hope that models can be more reliably and robustly controlled and understood.

It seems like a pretty unambiguous-

Yeah

... g- goal.

I'll maybe phrase it in terms of, like, there's maybe, like, a U curve of, uh, of this where, like, if you're extremely doomer, you don't want any research whatsoever. If you're, like, mildly doomer, you're like, okay, there's this, like, high-agency doomer is like, "Well, the default path is we're, we're all dead, but, like, we can do something about it."

Whereas there's, there's other people who are like, "No, just, like, don't ever do anything," um, you know.

Yeah, yeah.

There's also the other side. Like, there is the super alignment, like, people that are like, "Okay, weak to strong generalization. We're gonna get there. We're gonna have models smarter than us and use those to train even smarter models.

How do we do that safely?" That's... You know, there's the camp there too that's trying to solve it, but yeah, there's, there's a lot of doomers too.

Yeah. Well, and I, and I think there's a lot to be learned from taking a very, um... Like, even regardless of the problems that you're applying this to, also just, like, the notion of, like, scalable oversight as a method of saying, let's take super intelligent or, or current frontier models and help use them to understand other models, is another case where I think it's just, like, a good lesson that everyone is aligned on of ideally you are setting up your research so that as super intelligence arrives, that is a tailwind that's also bolstering our ability to, like, understand the models.

'Cause otherwise you're fighting a losing battle if it's like the systems are getting more and more capable and our methods are sort of linearly growing at, like, human pace.

Yeah. Uh, he- uh, Vivo got- did call out something like, you know, I, I, I do think a consistent part of the MechInterp field is consistently strong to weak, meaning that we, we train weaker models to understand stronger models, something like that.

Um, or maybe I got it the other way around.

It's the other way, weak to strong.

The other way. Yeah, yeah. The question that Ilya and Yann LeCun posed was, well, is that gonna scale? Because eventually these are gonna be stronger than us, right? So I, I don't know if you have a perspective on that, 'cause I...

That is something I still haven't got over even after seeing that.

There, there's a good paper, there's a good paper from OpenAI, but it's somewhat old. I think it's, like, '23, '24. It's literally weak to strong generalization paper.

Yeah, yeah.

But the thing is that most of OpenAI super alignment team has-

They're gone

... they're gone.

But, like, I think the idea, the idea is solid.

Yeah, yeah.

Well, no, it's, it's ba- There's now more-

They're still back.

Well, yeah, yeah. I think, I think there's some new blog, uh, posts coming out of OpenAI.

I know. And just, you know, check the Thinking Machines, uh, website to see who's back.

They're still back. Um, there's more coming.

You know, you know what I mean? Like, weak to strong-

Yeah

... seemed like a very different direction. And when, when it first came out, I was like, oh my God, this is, like, this is what we have to do.

Mm-hmm.

Uh, and, like, it may be completely different than everything, all, all the techniques that we have today. I, I-

Yeah. My understanding of that is it's, that's more, like, w- weak to strong when you, when you trust the weak model and you're uncertain whether you can trust the strong model that's, that's being developed. I'm sort of speaking out of my depth on some of these topics.

Yeah.

But I think right now we're in a regime where even the strong models we, uh, trust is reasonably aligned, and so they can be good co-scientists on a lot of the problems that we've been, we've been tackling, which is a nice, a nice state to be in.

Hmm. Yeah. Any last thoughts? Close the action?

I don't think so. As we mentioned, actively hiring MLEs, research scientists. Um, you can check out the careers page at Goodfire. Um-

Where are you guys based?

San Francisco. We're in, um, Levi's Plaza, like, by Quirk Tower. That's where our office is, so come hang out. Um, we're also looking for design partners across, um, people working in, in reasoning models, um, world models, robotics, and then also, of course, people who are working on building super intelligent science models or looking at drug discovery or disease treatment.

We would love to partner as well.

Yeah. Maybe the way I'll phrase it is, like, you know, maybe you have a use case where LLMs are almost good enough, but you need one magical knob to tune so that it is good enough. You, you guys make the knob.

Yeah.

Yeah. Or foundation models, uh, in, in other domains as well. The, the... Some of those are the, um, especially opaque ones because you can't, you can't chat with them. So, so, like, gen-

What do, what do you do if you can't chat with them?

Oh, well, like, thinking about, like, a genomics model or a-

I see

... material science model. So like a-

Yeah

... yeah, a, a-

They label, they predict

... a narrow foundation model.

Yeah, yeah, they predict. Yeah. Got it. Got it.

I was gonna say, I thought the diffusion work you guys did earlier was pretty, you know, pretty fun. Like, you could see it directly applied to images, but we don't see as much interp in diffusion or images, right?

Like, I see genomics and-

Oh, it's, it's gonna be huge. Like, look at these video models. They're so expensive to produce and, like... I mean, basically a Midjourney SREF is kind of a feature, right?

The what?

Midjourney SREF.

Oh, yeah.

Like the, the, the string of numbers that you-

Right. Right, right, right. Yeah.

The style reference, I guess.

Yeah. No, I, I mean, I think we're starting to see more of it. And I'll say, like, the, the research preview of our diffusion model, kind of like a creative use case and the steering demo you saw, I, I think of those much more as, as, as demos than, um, a lot of the sort of core platform features that, that we're working with partners are unfortunately sort of under NDA and less demoable.

But I will, you know, hope that you're gonna see interp pervading a lot of what gets done, even if it is behind the scenes like that. So some of the, yeah, some of the public-facing demos might not always be representative of, like, the...

It's, it's just the tip of the iceberg, I guess, is one way to put it.

Okay. Excellent.

Right.

Thanks for coming on.

Thanks for having us. This was a great time.