How Zyphra went all-in on AMD + Why Devs feel faster with AI but are slower — with Quentin Anthony

Hey, everyone. Welcome to the Latent Space Podcast. This is Alessio, founder of Kernel Labs, and we have back in the studio Quentin Anthony from Zyphra and EleutherAI. Welcome back.

Thanks for having me back, Alessio. Like, it's great to be back. Uh, do you want me to introduce myself?

Yeah. Let's, uh, g-give people an update.

Yeah.

I, I think last time you only had Eleuther and you were still at Ohio State, so.

Yes. So now, um, I am head of model training at Zyphra. So we're a startup building foundation models, uh, especially with a focus, like, on Edge. And then I'm still, um, head of HPC and sort of an advisor at EleutherAI.

Nice. So we reached out because you were in the maybe controversial study about, uh, with METR and, like, engineering efficiency, and obviously you've been doing a lot of great work at Zyphra on open source, but maybe there hasn't been a public update in a while, so it'd be nice to get the, the latest from you.

Uh, we wanna talk about AMD.

Yeah.

I think that's another interesting topic. We just had Chris Lanner. But let's start there maybe. Let's start with, like, your actual work, and then we'll kind of dive into the more side quests that you've been working on.

Sure.

So what's the latest on Zyphra and, like, what you're working on? I think some people might be familiar with some of the, like, Mamba adjacent models that you released in the past, but how should people think about what you work on and the current research direction you're in?

Sure. So AM-- uh, Zyphra is a sort of full stack model company. So we get our own data, we make our own models, uh, we do everything in-house. The focus is on pre-training from scratch foundation models that are currently focused on, uh, Edge deployment, so on device, that sort of thing.

Uh, but we also are starting to dive more and more into, like, uh, cloud deployments. We recently moved all of our, uh, training cluster over to AMD, so we're really going all in on the AMD ecosystem. We think they are really compelling training clusters, and it reduces our bottom line.

And I would say in terms of model training, uh, you know, stay tuned soon, but, uh, the more recent models we did a lot of, like, state-space hybrids. We're one of the first to that, sort of like a Jamba style.

We released, uh, Zamba2, which was a hybrid between transformers and, uh, Mamba 2 blocks, and we were able to beat, like, uh, Llama 38B, for example, with a 7B model.

When did you decide to go to AMD? I w- I would say that's maybe something that everybody... You know, everybody's always rooting for AMD to work-

Yeah

... and, you know, then the software is not that great and whatnot.

Mm-hmm.

What was the origin story of moving on to AMD?

So when I was working on PhD more heavily, I was working on Frontier, which is a supercomputer at Oak Ridge National Lab- National Lab that's totally based on MI250X, uh, AMD GPUs. So out of necessity, I had to make AMD GPUs work.

So I ported, uh, Flash Attention over to those. Very painful.

Mm-hmm.

Uh, we trained a 7B model there on, um, purely on, like, uh, scientific papers to try and create, like, chemistry, uh, models. Um, so kind of I figured out, oh, like you can make these work, and the hardware for MI250X wasn't quite there yet.

Then moving to Zyphra, there were a lot of companies, um, smaller clouds like, uh, TensorWave, for example, um, and then larger players like, uh, you know, Azure and whatnot were starting to move more to AMD, specifically on inference serving, and they had a lot of time that they would, uh, give to developers to sort of seed the ecosystem a little bit.

So we released a blog at Zyphra on porting Flash Attention2 over to MI300X just to sort of play out can we move our own stack over, given, you know, my pr-prior experience. So I lead the, the team working on kernels there.

Um, we found that it's, it's great, uh, for Flash Attention2 specifically, we were able to beat H100. We also found that, like, the less time you spend in, like, dense compute, like the less time you spend in Tensor Core specifically or, uh, less time you spend in lower bit with precisions like FP8 or so, and the more time you spend in parallelism or moving to and from HBM, um, MI300X actually, actually has, um, quite a bit better hardware metrics on those.

They have 192 gigabytes of VRAM, so you have to spend a lot less time working on parallelism strategies and, uh, trying to overlap all that communication time. If you're mo- taking any sort of data from HBM, um, you have much higher memory bandwidth, for example.

So if you're doing like a non-FP8 dense transformer, maybe even like an MoE or something like that, then you're able to beat H100 quite handily. Um, so we found that compelling and we moved over. Go ahead.

Yeah. There's kind of like, uh, Sarah Hooker's hardware lottery thing-

Mm-hmm

... which is like, "Hey, you know, the models that work, work for you, so work well with the hardware that we have."

Yeah.

Do you feel like all the work you've done is something that, in a non-humble way, you need somebody as cracked as you and your team to do? Or do you feel like people are maybe a little complacent with, like, you know, they just complain about AMD, and it's like, well, you can just work and fix it?

Some of it is groupthink, but I think a lot of it is deserved in that, like, it took a long time for hardware to catch up. Like, I would say MI250X was not on the same level-

Mm

... of A100. You had, like, two separate, uh, GCDs on each GPU, so you basically had to split acro- you know-

Right. Yeah

... you had a, a three-level, uh, parallelism s-scheme instead of a two-level on Nvidia, where you just had with- within the node and across nodes.

Mm-hmm.

Now you have within the GPU, across GPUs, and across nodes on AMD. Um, with MI300X, the hardware caught up, and then from there, software took a while to catch up. So when MI300X first dropped, if you run like an MLP gem or something, you would get like 400 TFLOPS.

Now that number is, you know, more like six- 600, 607, uh, uh, 650 or so, um, just in like the gem backends themselves.

Mm.

Um, a lot of documentation has been caught up, so they caught up on hardware. Now they've caught up on software. Not a lot of people have sort of discovered that they've caught up on software, and we're kind of capitalizing on that.

So it's not just, oh, if you're good enough, you can do it.

Right.

If you evaluate everything evenly, you find these-

Mm

... sort of diamonds in the rough, I guess.

What's the state of the AMD Xavier? So you have Tinygrad, you have Mojo and Modular. You basically have this kinda like, these groups trying to make better software to make AMD just the same as Nvidia.

Mm-hmm.

Are those interesting directions? Have you had... Have you gotten value out of it, or did you have to re-implement a lot of it from scratch?

Let's see. So I am not actively using things like Mojo. I'm actually, uh... You can get pretty far by using the ROCm backend, um, on a lot of higher level software like Torch For example, um, and then you just write all of your own kernels.

Mm.

And a lot of the time when you're training models, um, especially is spent in those core kernels themselves, like the transformer math work like led to trying to like really profile how much time do you spend within a training iteration.

Most of that is in attention-

Right

... MLPs, right? So if you have good kernels for attention MLPs and norms and so on, then it doesn't much matter what the front end to, you know, send tensors to and from those kernels is. Now, on things like Triton, for example, that you...

Okay, you are beholden to the Triton compiler and who wrote-

Mm

... the Triton compiler for that specific hardware. So when I want to make an optimization to say FlashAttention, I don't go from the top down and think what Triton kernel do I write and then go to the hardware.

Instead, I think from the bottom up, the hardware of MI300X has these properties and I want my, uh, algorithm to pull these properties out so I get the full benefit of the hardware. So if I have some optimization that's hardware specific, I both have to understand the optimization, and I have to change the Triton compiler to work for this case.

So it adds a l- le- another layer of complexity. So I find myself more and more working just purely in ROCm to write the core kernels, and then I just expose it with like some Torch interface-

Hmm

... and that gets me pretty far.

Interesting. Yeah, it does feel like there's like a lot of opportunity there. I think when DeepSeek-R1 came out, everybody was like going crazy about, oh, they went like so deep in like the actual GPU code to like-

Mm-hmm

... improve things and I'm like, "Why are you not doing it?" You know, if you're spending all that money training.

Yes.

Do you think it's something that like is worth open sourcing? Like, you know, I guess that's one question is like especially with Eleuther being such a... You know, you have maybe like a duality-

Mm-hmm

... of, of roles.

Yeah.

Do you feel like an open source group should kind of take this on or do you feel like because the kernels can be so tailored to like the type of workload that you're doing that everybody should just do their own work on it?

Open source should definitely take a look, and my suggestion there is you need to drive adoption. So, um, for example, Mojo, I see it as someone who is extremely talented, like Latiner is extremely talented. He found an actual problem, and he had the agency to go out and do something about it.

But until you sort of get a critical mass, a sort of threshold of people behind you, then you have to keep dumping a lot of talented people and money to keep up with the rest of the field kind of thing.

Uh, maybe like, uh... I mean, Tinygrad is another example of this, right? So like there... It's, it's great, but do I really wanna slot my entire training-

Right

... stack to them and go all in? Like as an organization, hard sort of decision to make. I think open source's job here is to, um, just, I mean, get something, get a foot in the door. Make some small model, show that it's, uh, actually viable.

AMD themselves have released some models around the 1B, uh, pre-training, uh, that they've pre-trained from scratch. Um, and open source just needs to keep picking that up-

Mm

... and start small and get larger and larger instead of trying to replace the whole stack.

Yeah.

That's my suggestion.

Do you think it's worth for people to spend a lot of time on like the current AMD generation? Because I think the other question is like, well, I'm gonna do all this work versus like I just write CUDA code that then when the GB- BG200 comes online for my cluster, I'll just switch it over right away.

Mm-hmm.

Like do you feel like there's certain maybe model sizes at which, you know, if you're training like a 7B model like the Zamba one, you shouldn't even worry about the next generation of Nvidia because the workload is like somewhat confined now?

Um, in terms of, uh... I am not fortunate enough to have the timescales to wait for the next GPU.

Right.

I don't think I've ever been in that situation. So it's always like, "What do I have? What is the training hardware that I can actually buy today?" And then once I have that, how do I pull as much as I can out of that hardware?

Unfortunately, like, um, a lot of kernels that are as optimal as possible, like FlashAttention, for example. FlashAttention 1 is sort of a V100, A100 sort of scheme, and then a, a FlashAttention 2 is more to A100. FlashAttention 3 goes even further down this line of being co-designed on H100.

Mm.

And these... You can port FlashAttention 2 over to MI300X because they sort of follow the same hardware steps-

Right

... that Nvidia did. But, um, a lot of kernels are tied to hardware. Tinygrad is also in the same sort of setup where they're trying to work, train on consumer hardware, so necessarily a lot of their attention kernels, for example, are tied to consumer hardware.

Mm.

So I don't think I personally ever look past the current generation.

Right.

And when the current gen- the next generation comes around, then I take and I say, "Okay, back to square one." You know?

Yeah.

What, what sort of things can I port? And I port over again core hardware designs that don't change too much. Um, but I s- almost always have to do due diligence and to tear out major portions of my kernel to fit the new hardware.

Right. Yeah. Yeah, can you give people a quick overview of like what are the core pieces when you're writing kernel? So a lot of our audience is like AI engineers, not necessarily AI researchers.

Yeah.

And I think obviously people understand CUDA is like, you know, a, a big lock-in, then there's like cuLASS.

Mm.

What are like really the things that in your mind are like, okay, this is like a huge advantage versus like this is like part of the offering, but like I can replicate that somewhat easily, uh, on AMD or whatever else?

Mm. The ecosystems within Nvidia and AMD are actually quite similar. So from the very- from- going from bottom up, you have, um, PTX, so basically GPU assembly on Nvidia world. In AMD world, that's AMD GCN. Um, this is only if I need absolute control over the hardware-

Mm

... and I've tried Triton, doesn't work. I've tried CUDA, it's not giving the control. I'm like, "Oh, great." So then I have to go to PTX or whatever. That's kind of sort of a last resort. But I can make the hardware do whatever I want when I go there.

Um, on top of that would be sort of CUDA and, um, you know, HIP or ROCm on the AMD side. This is sort of the happy medium. It's lower level than most people work in, but it still does almost everything I need to do.

But if I want to... A l- a lot of, um, deep learning ops kind of rhyme, so a lot of deep learning ops require gems, for example, which is matrix- generalized matrix-

Mm-hmm

... matrix, uh, matrix, matrix ops. I probably don't wanna write a gem kernel from scratch. Like Nvidia people have worked on that for a long time, so I would use something like, um, uh, like cuDNN or something like that with some, uh, gem backend.

Um, there's a lot of like building blocks that people have made, so I don't wanna reinvent the wheel when I'm writing like a FlashAttention kernel. I don't wanna rewrite, um, each matrix-matrix multiplication.

Mm.

So I try and plug those in as possible. A lot of those are written in CUDA, or they're sometimes in closed source libs. On top of that, there are, I would say, like these template libraries that are getting more and more popular So like on the Nvidia side, it's like, you know, Cutlass or Cute, where you can kind of like focus more, "I have these Lego blocks and I wanna focus on the how to link them together-

Yeah

... how to fuse kernels together." Um, so, um, so that would be the Nvidia side. And then on the, um, AMD side, they have this, uh, composable kernel library that does something very similar. Um, they say, "You have...

Here are your gems. Here are, you know, every o- some- a few other specific ops. You just focus on, you know, making like a custom epilogue." So what happens at the very end of your kernel, how do you fuse two of them together, that kind of thing.

And then on top of that, you have these, um, sort of... Well, maybe these are maybe not on top in terms of like a software stack. There may be like a vertical-

Right

... line throughout where you'd be like Triton or Mojo, which is kinda... Or, uh, even TorchCompile on top of that, which is kinda saying, "We try to handle all of the backend stuff for you. We compile down to AMDGCNNPTX.

You just write the basic subroutines, and hopefully they map to things that the Triton compiler or anything else, um, understand and are common ops."

Yeah.

So.

Yeah, no, that, that's great. Um, how do you think coding, kinda like coding agents help with that? Like, have you found that it's easy to leverage that? And I guess the other question is, like, how verifiable are these kernels?

Right.

So if I write a kernel, how do I measure the quality of it, the performance?

Mm-hmm.

Like, is that easy to do, or...?

So I find agents and, uh, models today, like really good at the higher level stuff. So I'm tr- if I'm trying to write some, like, fusion of two kernels, or if I'm trying to write something in Triton that's very high level, models are great.

And, um, they... Well, as good as they can be-

Right. Yeah, yeah, yeah

... when you're talking about GPU kernels, right? Uh, limited success still. And if I wanna like write templates or skeletons or, like, initial setup, I give it a design. I say, "Just kind of make me the broad setup and let me focus on the heavy stuff."

Saves me time in that for sure. Getting below Triton or level or anything else down to, like, CUDA or below, um, there's orders of magnitude less public good kernels at that level, and I think that really shows in models' capabilities.

So when I try and get a model to d- uh, do something in CUDA or PTX or something, as long... If it's not dead basic, it's bad really fast.

Right.

And I think everyone can kind of speak to this. Like, when I use a model, it's harder to bu- to debug 'cause I don't understand the code, and when I'm working in systems and low-level code, the debugging part is much harder than code that I can more deeply understand.

It's hard for me to, you know, map out how a GPU kernel is working 'cause it's very parallel, and that is worsened by far. So not only are models kind of bad at generating kernels because they have very little public training data on them, you know, TreeDAO is, like, one example of making very public kernels, but how many more do exist?

Right.

There's a lot of Triton ones, not many CUDA ones. Um, and not only that, does... it's harder to find the issues themselves in those generated kernels because it's hard for me to map out everything that's going on in my head.

Yeah. Would a kernel data set, how valuable would that be? If somebody was like, "Hey, here are like 10,000 amazing kernels-

Mm-hmm

... that you can put in context or, like, fine-tune a model with or whatnot."

It would be great, but it's not a silver bullet because kernels are also hard to validate. It's hard to have-

Mm

... like an eval in kernels. Every time that someone releases like, "Oh, we created a new eval that measures kernels, and we trained a model that generates kernels that no one has to w- learn CUDA again," there's always m- a few smoking guns in there where there are kernels that are impossible, basically- ...

'cause the model has found some sort of way to, you know, game the system. It's trying to find anything that-

Right

... that checks your metric. Do you like make... Uh, if you just check like l- okay, some stupid example. If you're just checking latency, then the model will compute something really fast, but wrong-

Right

... or something, right? And, uh, but in subtle ways that it's hard for you to find. You have... It's hard to get like a, a very sealed, closed loop eval that measures whether models are good at writing kernels, whether they generate them well.

Mm.

That also makes synthetic data hard to generate kernels, and then it's hard to get the training data in the first place. So it's like-

Right

... not unsolvable, but it's obvious to me why they're not any better than they are now.

Yeah. What's the kernel writer workflow? I think... I mean, personally, you know, I've always built applications, right?

Mm-hmm.

So I know what that flow looks like.

Yeah.

So you gotta write a kernel. How do you start? How do you sit down?

Okay.

How do you work? How do you test it?

Mm-hmm. Okay, well, kernel is a last resort. So first I go, "Ugh, another kernel. Oh my God." And I try and find some way to go around it. But if the kernel is necessary, it kinda depends on what I'm doing.

So if I'm doing like... a lot of problems can map to this sticking Lego blocks together. I have some sort of, like, new model architecture that changes attention, and I can just fuse an existing flash attention kernel onto some epilogue-

Mm

... and I try and do that. And I just kind of keep repeating that workflow and going lower and lower, seeing, okay, how much lower to the hardware do I need to go to get this thing to do what I want?

And then, you know, kind of reach bed- bedrock, right? Um, if Triton doesn't do what I want, then I need to go a level lower. If I'm doing-

And, and when you, when you say-

Yeah

... doing what I want-

Yeah

... what does that mean usually? Is it part of a latency? Is it compute effec- what, what are like the dimensions-

Im-

... that make a kernel good?

Implementing my exact design. So I come up with a design. I say, "I want this tensor materialized in registers," or something. And if I just tell Triton, if I do some Triton instruction, I don't have total control over where that tensor is-

Mm

... um, instantiated. Maybe Triton decides keep it on HBM and then page it into registers every time around to tensor cores whenever I need it. Um, and that's not what I want, so I have to watch where that tensor goes, and if it's not doing what I want, I need to, either need to change the Triton compiler or I need to go a level deeper and say, "Okay, I want this tensor where I want it."

Um, otherwise I'm s- wasting time going-

Yeah

... to HBM or back. So that's why I'm... Implementation of my design, which I've decided a priori on a whiteboard.

I think for, like, a lot of people, they hear about kernels-

Mm-hmm

... but then it's like, in reality, what are these kernels?

Yeah, yeah. It... I will also say that if it doesn't fall into this Lego building brick scheme and I'm really doing something new, then I, you gotta go down deep to the hardware-

Right

... and say, "This is what I want the hardware to do," again, on a whiteboard-

Yeah

... that, that is the first step. Um, and then try and just, I don't know, what is the highest level tool that will actually do what I need it to do?

Yeah. How often do th- does that happen? I feel like, you know, maybe like three, four years ago, it's like very often. Like today, what is like the set of problems that you're like net new trying to solve?

Entirely depends on how much model architecture you're changing. So if you, your business design is, "I want to train models on some new type of legal data or something," I don't need to change attention that much. I mean, maybe I make it a little more accurate for longer context or something.

Mm.

But if I'm not changing har- uh, like the model architecture, I don't need to write a kernel. If I'm not using some new G- hardware, GPU hardware, then I don't need to write a kernel. But if you're trying to make like new model architectures like I am that are efficient across, you know, a lot of edge deployments, if we're trying to compete with organizations that are pre-training with much more compute and, you know, manpower budgets than I do, then I got to sort of take nothing for granted and squeeze out as much juice as I can out of every parameter and every, uh, TFLOPS that I have access to.

Have you worked with any of, like, the GPU alternative hardware, like a Groq, Cerebras? Any thoughts on, like, yeah, pros and cons, what are, like, maybe things that are still not doable that-

Mm-hmm. Not a whole lot. Um, they fit a niche that I'm not into right now-

Mm

... which is, uh, in-inference economics that are non-standard. So, like, Cerebras is really good for super low latencies, and I'm not really looking at low latency inference right now.

Right.

My main concern is training performance, and I'm my own client in term- because I'm the one paying the money for training. When we produce models that everyone loves, then I will, you know, jump into the inference side if I care about that.

But right now, no.

Yeah.

I don't- I won't speak much to that.

Yeah. And because of the, their structure where, like, mostly SRAM and whatnot-

Yeah

... it's not even a thought to, like, use them for-

I mean-

... for training, obviously

... it's, um, it's interesting, like, I, I'm an interested observer. I think people... Examples like Cerebras especially, they kind of, um, overlook a lot of quality. They overlook that the hardware is, um, has uses beyond, um, what they see immediately.

Right.

Doesn't necessarily have to be... They compare it to what Nvidia does well, and they say, "Okay, well, it doesn't do that well, so what's the point?" But if they can find a niche that's economically viable, then, like, absolutely, they have huge value.

And what about... Now there's a lot of talk about, oh, maybe if you train a model, you can make a ASIC to then run inference on the model. Like, is that something that when you train models you think about as, like, the destina- the target hardware for inference?

Or-

Mm-hmm

... do you feel like you already have so much work to do before that?

Um, that's top of my mind, so, um, is, is inference, uh, performance. So we train models that are inference efficient. Um, so I did a paper, um, with Eleuther a while back called The Case for Co-Design with, uh, Model Hardware, and we find like, for example, if you, for example, choose, like, a hidden dimension that has many powers of two in it, then it's much easier for a GPU to split that up-

Mm

... across a lot of tensor cores. So we see, okay, these are all the inference hardwares that we intend to deploy on. These are the sizes that they're most amenable to. These are the kernels that they would be using.

And then how can we choose, like, the model sizes so that they are most amenable to as many of those as, like, possible? So, like, absolutely I do what I can, but if in three years I'm creating a model, you know, then, uh, or if I'm, if in three years-

Right

... I'm trying to infer that model on some new hardware, then yeah, I'm kind of screwed. Probably need to train something new.

Yeah. What's the, I, I guess, like, the timeline for, like, an ASIC or something that would make you feel interested in doing that? So I, I think the most criticism today is like, well, to, like, actually tape out an ASIC, it might take me, like, 18 months, and, like, by the time that comes out, the model is already obsolete.

Mm-hmm.

If it took six months, then what would be the cost in your mind that would be worth... I don't, I don't know if you ever thought about it in these terms-

Mm

... but it's like if you could spend 100 million to build a custom ASIC that comes out in six months for your specific architecture-

Mm-hmm

... what would be the math there? Would that be a good number? Like, how cheap would it need to be? Maybe it's not never possible to do it in the right-

Oh, it's absolutely-

... you know, price and timing-

Mm

... but, like, I'm curious, like, today, uh, what would be the number for you if you had one?

If I was sort of in-house, maybe, um, xAI is a good example of this, or Microsoft is another one where they kind of know the model architecture a priori, um, then they absolutely should make an ASIC that is custom designed for that model architecture that they're currently training, and then try and line it up so that they run inference on the ASIC at the end so that they keep everything in-house.

They don't have to spend a bunch of money on different hardware, and that ASIC is probably gonna be custom made for that model anyway. Like, model generations are getting a bit longer and longer, so-

Yeah

... it's good to have an ASIC that lasts a little thing. In terms of exact timelines, like, mm, I, I'm, I'm, I'm not knowledgeable enough to tell you the exact numbers that I would guess. If that changed in the future, then I would come back and tell you.

Yeah, yeah, yeah.

All right.

Yeah. I, I think that's a good... And, uh, I mean, again, I don't know how much you spent looking at this, but how tied to, like, the actual model implementation would that need to be? Like, you know, is there, like, a middle ground between a GPU and an ASIC that it's like the weights burn into the chip, where you can be, okay, this is, like, the Zamba-

Mm-hmm

... ASIC and, like, you know, Zamba 1, Zamba 2, Zamba 3, they'll have some differences, but, like, most of it will work the same and reap the same benefits.

This spectrum already exists, I would say. So I would say, like, on the extreme more ASIC side, you have, like, an etched kind of company or Groq or something. So, like, it's very- Groq is very inflexible hardware, so that you kind of...

If, if you wanna do, like, a Mamba SSM on it, you're gonna have a really hard time because instead of having, like, a low-level CUDA compiler, like, everything is, like, designed in at the hardware level, right? Um, and then Cerebras is kind of, uh, more in the middle, more maybe towards the CUDA side, and then, uh, GPUs are, as, you know, fairly general purpose.

So kind of spectrum already exists today, but I definitely think things are moving towards, um, you design the model. It's sized the way that fits well on hardware that you can design. You immediately start creating an inference hardware that is custom made to fit the sizes of your model, and then you can also sort of...

This is kind of an anti-open source situation, but you could design your model to only run well on your inference hardware-

Mm

... if you size it in a weird way that doesn't, you know-

Right

... perform well on anyone else's. I kinda see things moving that way, but we'll see what happens.

What about, yeah, I mean, maybe one way to think about it is, like, Apple, right?

Mm-hmm.

I think Apple has their own stack that, like, they say, "Kirin Clade" that not everybody got access to.

Mm-hmm.

Um, maybe they haven't done as much on actually building good models to put on the phone.

Mm-hmm.

I know at Zyphra you had spent some time, uh, thinking about on-device inference-

Mm-hmm

... especially for, like, smaller devices. Do you feel that's still, like, an interesting direction? It's like, do you feel like... We had Greg Brockman on the pod, and we talked about the GPT-5 to GPT OSS handoff, for example, that could happen, where, like, you have OSS on the device-

Mm-hmm

... and then it hands off to GPT-5 for more compute-intensive, intensive tasks.

Mm-hmm.

How do you think about that and then Zyphra's strategy too of, like, what size model y- are interesting to build and things like that?

Like I was saying, we sort of look at all the edge deployment hardware that we want to deploy our model on, and then we just size a model appropriately towards that. And that often looks like a spectrum of models.

So, like, the Zamba 2 suite, we had, like, a 1.2B and then a 2.7B and then a 7B, and those were for, like, extremely resource-constrained, uh, edge deployments, and then maybe, like, laptops for the 2.7B and then, I don't know, like, desktops or-

Mm-hmm

... cloud serving for the 7B. Um, and I kind of see people doing that same thing, sort of designing a spectrum of models for the spectrum of compute, and then those smaller models kick back to the larger models when they can't answer a question.

So you kinda have, like, a model merge GPT-5 sort of mode where you have an inference-efficient model on your phone, and it says, "Oh, I can't answer this question. You're asking me for a reasoning trace in math." It kicks back to GPT-5 in the cloud and gives you an answer.

I kinda see things moving that way.

Yeah.

So you only spend the inference cost on what you need.

Do you see the intermediate of kinda, like, the, you know, tiny box or, like, k16z has built this, like, inference thing where it's, like, a box that you have at home-

Mm-hmm

... which is, like, maybe, like, 4AD GPUs, and it's, like, in between the two? Or do you feel like- Once you're ending off device, might as well just go to the cloud most of the time.

So those specific kinds of boxes, m- uh, not for people at home, I don't think, except for some extreme hobbyists. I see them more at, like, organizations maybe. So if you are, I don't know, Chase Bank or something like that, and you wanna spend less, uh, inference money to a provider like Databricks or something, then maybe you have a few of those boxes in your data center and all of your, like, custom code generation on the Chase Bank software stack goes to those first and then kicks back to some model in the cloud if it can't answer it.

So I do see people, like organizations trying to silo on private local models that run on private local data centers or small, like, mini clusters. But things will always need... Well, I won't say things always need to be kicked back.

We'll see. But definitely not consumer, I don't think, for those.

Yeah. Well, I, I think the humanoid robot thing will be maybe the first-

Mm

... stress test of that.

Yeah.

You know? Like, if you're putting these things into your house-

Mm-hmm

... are they always gonna be connected externally?

Yeah.

Are you gonna have... They bring you the robot in, like, a, a rack to put in

Yeah. Or cars.

In your, in your... Yeah, yeah, totally.

Yeah.

I, I think it's interesting because cars have, like, almost, like, no weight requirement-

Mm-hmm

... in a way, which is like, you know, whatever the car weighs, it weighs, versus-

Yeah

... if the robot... I, I forgot who said it. It was like if the robot falls on your grandma, it's, like, really bad. It's like if it's, like, really heavy or not, so.

Yeah.

Um, yeah, I, I think Edge has been interesting. I-

Mm-hmm

... I tried the GPT-OSS model on the Edge too.

Mm-hmm.

I would say the time to first token on, like, the logo models is always, like, a little slow if you don't have them in memory.

Yeah.

Which is almost like most of the queries that I would wanna ask the logo model are kinda like one-off-

Yeah

... quick things.

Mm-hmm.

Um, so I haven't quite found the use case for them, but I don't know. I'm curious what you've seen. What are, like, maybe some of the more interesting, like, Edge deployments-

Mm

... that you've seen?

Well, first, when you say GPT-OSS on an Edge deployment, what edge are you deploying it on?

Oh, on, like, my, on my MacBook.

Okay. Okay.

I just use the, you know, 20B-

Mm-hmm

... um, on TrueLlama.

Mm.

This is like a 36-gig RAM, so it's like, you know, somewhat fast, but obviously-

Yeah

... the, the loading into memory-

Mm-hmm

... is kinda slow.

For a while, I would deploy on my laptop, like, coding models 'cause sometimes I'm on a plane or sometimes I just... Or I wanna spend less money on-

Right

... my whatever deploy- or my, like, API. But it kind of lags in quality, um, so far and I haven't had a specific enough problem to warrant, um... Like, sometimes a open source, uh, model you can just fine-tune on something you really care about and deploy it, and that I haven't really had time to keep up with and run it myself.

I'm not a good example of the people, uh, running these sort of Edge deployments. Um, instead I kind of want the most quality and I'm willing to pay for it because it's, uh, it's what's keeping me from training the next model.

Right.

So I tend to more compare, um, Claude with, uh, ChatGPT-

Yeah

... or somewhat and find which one does the best for my problem.

Yeah.

And I pretty rarely use, uh, local models unless I'm just asking a one-off question and it's faster to do over Google.

Yeah. Do people come to you and they're like, "Hey, this is, like, my small model use case. Can you guys make it better-"

Yeah

... "for this specific thing?" What are, like, those use cases?

Hm, can't really speak to those.

Okay. Yeah, yeah, yeah. That, that makes sense.

Mm-hmm.

I don't know how much you wanna talk about what you have coming next at Zyphra. Like, do you talk about... You know, you're somewhat secretive-

Mm-hmm

... as a company, I would say. I think if you go on your Twitter, there's like five tweets in the last five months.

Yeah.

Something like that.

Mm-hmm.

Do you have a roadmap that you usually share with, like, the research engineering community, or are you very, you know, once the model is out you'll know about it?

Mm-hmm. We tend to go towards the latter scheme just because there's so much noise, and if we keep saying, "Oh, we're about to release something big-"

Right

... it kinda takes away from the announcement, I feel. We, we tend to wait until we have something to say, and we do have things to say. We're just kind of like we kind of want to ambush, right?

Um, it also gives your compe- competition a little bit more time to pivot, and we're kinda trying to, you know, go towards the next thing for model architectures and stuff.

Mm-hmm, right.

So that, that time is moat. I would say expect, um, more models that are easier for more people to adopt. Um, I would expect to see higher quality models, and I would continue to expect models that sort of push the envelope.

And when I say push the... Okay, that's a silly thing to say, push the envelope. But we're not afraid to take big risks-

Mm-hmm

... for model architectures, so we're not gonna train something, um, unless it's, it does exactly what we need it to do. And we won't just sort of take something off the shelf unless we can find something better. That doesn't mean we find, we create something better for the sake of it.

We don't want to do something new.

Right.

But, uh, it tends to be there's a lot of low-hanging fruit for us to find.

Yeah. Nice. So you were a part of the METR Software Engineering Productivity with AI.

Yeah.

Can you just run people through what that was like? So how did they onboard you? What were they trying to measure? How will you measure the backstory on the setup?

Sure. Lead development on a model training framework called GPTNeox, which is like a model, a Megatron deep speed style framework for pre-training models on HPC systems. It's, um, this is sort of, uh, you know, work I do outside of my core Zyphra work.

And, um, there's a lot of adoption, so this is kind of the standard. A lot, a lot of academic labs, there's a lot of usage, a lot of people who need pre-training kind of go to this framework. And METR was looking for prominent open source projects to watch you do your work in the wild and see how well does AI speed you up and try and be as, um, as low overhead and affect the process as little as possible.

So they came to me and said, "Hey, would you like us to just kind of watch, watch how you use models and take very detailed logs about how things sped you up, where they sped you up, that kind of thing."

And I'm like, "Yeah, sure, absolutely." I already use AI sometimes in my work- workflow. Totally fine. What I would basically do is, uh, I would create a bunch of GitHub issues, um, that were already on my roadmap, and then, then they would sort of do a lottery of use AI in this one, don't use AI in that one to speed you up.

And then sort of write down detail, like how much time do you expect to spend on those tasks, um, how much would AI speed you up on those tasks, how much did it actually-- how much time did it actually take you?

Mm.

And then write very detailed logs of, like, what you actually did. Um, and so I did that, and so that's where they get the estimated speed ups. So I put out all the tasks, then how long I expect to spend on those tasks with and without AI.

So then I just for, you know- Several months. I just did my regular work on the time that I could have on those little issues and then put the extra time it took me and, um, the AI sped me up a bit, but only in specific cases and only when taking a lot of sort of digital hygiene practices.

But what, what else? You-

That's the setup. Great. Do you feel like the tasks that you were measured on versus the one that you were not measured on align with like where the AI is good? Because I, I think in my mind that's like the biggest thing is like some tasks I know the AI's gonna be good.

Yeah.

Some tasks I know it's probably not gonna be good, so I'm not gonna use it. It feels like in this process... Do you feel like it was like the right fit, AI, not AI? Or like do you feel like the results are maybe not always, you know, descriptive of the actual ability because maybe you were just assigned the wrong task?

Mm-hmm. Well, I was measured on all tasks. Um, but that's kind of where the, um, expected time with and without AI comes into because I would give my expected time for both. Um, and there were some tasks where like I knew the AI would one-shot like writing documentation.

Right. Yeah.

And I was like, "Okay, this is gonna take me 10 minutes if, if it's a AI," or not even one minute if it's on AI. It'll be one-shot, but it'll t- probably take me like an hour. Uh, and then when it-- the lottery came up as you can't use AI and I was like, "Oh, great.

Okay." And then sure enough like it took me like an hour kind of thing. Um, but it was totally random. So can you clarify the question a little bit?

Yeah. Well, no, I think that's the thing is like this is a good example of like does that mean that the AI got like a negative scoring in that case? Because you could have used AI and say you're one minute-

Mm-hmm

... instead of spending one hour and then you could have used that one hour to do something else and then eventually means you were sped up. You know? I, I think that's the question that people had is like is this a task by task or is it a overall workflow thing?

And it sounds like it's more about the task and sometimes you could have been sped up overall by not doing a task but-

Yes

... the lottery told you-

Yeah, yeah

... to not use AI.

It's task per task.

Okay.

Um, and then if I can't use AI, I don't use AI. But I would write in my notes that these are the parts that I thought could be sped up by AI.

Mm-hmm.

A lot of it is-- a lot of it was also just reading other documentation like with deep research or something. Um, I need to know what a documentation to link to. I have to do a bunch of Go- Google searches.

Right.

Models speed me up on that too, not just raw code generation.

Yeah, yeah, yeah. Interesting. Um, yeah, you had this nice Twitter thread about old people in Vegas clicking slot machines.

Yes. That's how I feel.

Well, uh, I guess like, yeah, let's run us through your setup.

Mm-hmm.

So what do you use for agentic coding? Do you use just Cursor? Do you use Claude Code and whatnot? Yeah, and then we can kind of dive into more like, you know, if you use a Agents.md, what do you write in it and things like that.

Mm-hmm. So I personally don't really use tools like Cursor because I, um, I want total control over the context. I know what models can handle what prompts and I know... For example, one thing I mentioned is context rot, so how long the context is before-

Mm-hmm

... the model chokes on it and I can't see those details of what Cursor is passing to the model. When I tell it to, you know, write me a test for this other function in my code base, I don't know what it's feeding the model.

Mm-hmm.

Maybe it's too much and Claude chokes on it. Um, so I don't use tools like that. Instead I just have like a bunch of m- models at my fingertips if I need, um, in, in my local VS Code editor and I know just based on gradual experience, uh, which models are good for which tasks.

If I need something really high level like documentation or writing tests or something, then I use something like Claude. I just find it's really good at like capturing human taste. Um, if I need something a bit more heady like, um, writing me for example the, the setup for kernels like we were talking about earlier-

Right

... I might use a thinking model like an extended thinking on GPT-5 or an o3 kind of prompt and it's just kind of like, I don't know, taste over time. Like I can't, I d- I have not found a single model that can do everything I need because they all have very different training sets that they were devel-develo- developed with, sorry.

Yeah. And do you use a lot of like the chat interface of like GPT-5 if you're using GPT-5 Pro reasoning or-

Mm-hmm

... have you built anything personally?

Uh, I like to use like a local like LiberChat kind of interface and I use the API keys just 'cause I have more control.

Nice. Yeah, yeah.

Um, and I'm, I can put it in my editor and I-- there's less back and forth.

Right.

Um, but yeah, that's what I like.

Nice. You said pressing tab in Cursor for five hours instead of debugging it for one hour.

Mm-hmm.

How have you changed like your coding loop when you use AI?

Mm-hmm.

So you start with a prompt obviously. Have you changed at all the way you do code reviews, the way you think about writing tests-

Mm-hmm

... uh, the way you do linting? I don't know, anything like that.

Hmm. That specific GIF we're, we're talking about with the, uh, slot machine, um, that is sort of a rabbit hole I kept finding myself falling into early on 'cause I, I started using models really early. I thought they were-- I was sort of an early adopter and I kind of found myself...

AI tends to either one-shot things or kind of miss them a lot, especially early on when thing-- a lot of things were out of distribution. So I found myself like just trying to force the model to understand with better prompting and kind of things and I ended up spending more time just trying to maybe one more prompt finally gets the model-

Right

... to understand and one-shot my problem. I really, I have the sunk cost fallacy. I don't wanna start from scratch on my own and, and admit defeat. That's where that came from. Um, in terms of my workflow, I kind of, um, when I first am given a task I kind of map to what subtasks within the larger task are amendable to AI and which ones aren't.

So the initial skeleton work, I put that all to AI. I know what model does that well. Um, the core implementation tends to be me working on it, maybe some pair programming with like a GPT-5 thinking. I sort of say, "This is kind of what I have as an initial implementation that I did on my own.

What pitfalls do you see? Like what holes do you see that I could-

Mm-hmm

... improve?" It's really good at finding those sort of things. Once I finish the core, uh, the debugging I do on my own. Um, sometimes I'll do some pair programming. Once I finish that, then like unit test and documentation, one shot at the Claude and then I'm done.

That's like an example of a task but I kind of map the broader task into t- subtasks that may or may not be amendable.

Hmm. What are maybe some of the big model jumps that you've seen since you started using them early on? What, what were some models that you were like, "Okay, this is like definitely different and much better"?

o1 preview, uh, and o1's like the initial thinking models were a big deal when I was doing like core academic like how do I create a performance model for explaining how this kernel per- behaves like f- from first principles.

Mm-hmm.

That kind of thing was not really in the scope of any models before. So that was a big one. Thinking models was big, um, as, as a researcher. I think- Claude's ability to capture what humans wanna read was a big deal to me.

Mm-hmm.

So Claude 3.5 Sonnet kind of, uh, really, it finally took all of the low-level thinking of unit tests and skeletons and documentation, all that stuff. Okay, Claude can now totally one-shot when before it was kind of mixed on which models could do what.

Those are kind of the big step moments really, and a lot of them are just kinda incremental. Um, some models are really good at some small tasks, and I kind of have a set of prompts that I send to all of the new ones and I say, "Okay, what are you good at?"

Right. Yeah.

Kind of thing.

How do you feel about GPT-5? I think there's been mixed reactions-

Mm-hmm

... over time.

Because they are mixed models. Because-

Well, yeah

... I don't know what I'm using. So I almost like find myself doing tricks like, uh, telling GPT-5 to think deeply and try and get it to use the biggest, heaviest model that it possibly can. Uh, I find it's less consistent than previous models just I think also because I don't know what model I'm routing to.

I don't like being out of control, but it still fits my needs. It still does most of what I could get o3 to do.

Yeah. Yeah, I have the Pro subscription. I always use GPT-5 Pro.

Mm-hmm.

The problem is like without the routers, like it just takes a long time.

Mm-hmm.

Or like the... It doesn't have a way to like understand that it did the big lift to start, and now it can use maybe like a smaller model.

Mm-hmm.

But I think given the backlash that they got in with the initial router release-

Mm-hmm

... I'm curious how in the future they're gonna route things. I don't know if you spent any time with like GPT-5 Codex in the Codex CLI-

No, not really

... but yeah. The, it didn't-

The new for like-

... sound like it fit the, the, the workflow.

Yeah.

So you are one of the few people that actually got a score that's like you are more productive with AI.

Yeah.

I think a lot of people are like negative.

Mm-hmm.

One, why do you think that is? And then two, do you think that other people get wrong how to use it? Do you think they get wrong how to estimate how much time it takes?

Mm-hmm.

What, what are were like some of the failure modes in your mind?

Yeah. That one was tricky because I didn't wanna sound like some guru and say, "Oh, here's how to use AI," like I did, but at the same time I think you're... I'm just kind of noise unless I tell people, "Okay, I was the one-

Right

... that got the most speed up in the study." Um, I don't think... I think all of the devs were really great. That was my very first sub tweet. This is not about a skill issue-

Right

... dude. Like, get better. I'm just too good. Um, definitely not that. It's more about, um, I'm super real with myself about whether I'm sped up, this c- sort of introspection and a lot of hygiene over time, and I just kind of wanted to share this is what works for me.

If it doesn't work for you, that's great. Find what's good for your own workflow, but AI tends to speed me up, and these are some pitfalls that I try to avoid. These are some ways that I try to make AI, you know, better for me.

Take it or leave it kind of thing.

Yeah. Yeah, we can go through the whole thread just and then see if there's like things that you wanna comment on, add on. I know a lot of times, you know, I find the same when you write online, it's like what can you guys comprehend without being too- ...

too thorough. I think this kind of we talked about, but like what are coding tasks that have lots of clean data for and like what benchmarks the LLM labs are kind of like, you know, obviously, uh, maximizing towards.

Mm-hmm.

How do you think we're gonna fix that? In my mind, it's almost like the model labs already spend so much money on GPUs and running GPUs and writing kernels-

Mm-hmm

... that I'm like, "Why have you not made your models better at doing that right?" And then part of it is like, well, they don't really wanna put the data in the model to make it better that... Do you feel like this is like a solvable problem, you know?

Or do you feel like maybe open source is gonna help, or is it just like, hey, these things the model is just not gonna be able to do?

I think most things can be solved with more high quality data. Maybe kernels is a good example of this. So like the high performance inference kernels that sort of drive a lot of, you know, Anthropic and OpenAI and stuff, their models, those aren't open source.

They're not gonna be open source. So those are like necessarily limited except for the few people who are incentivized to be open source like those in academia like Tritao. So there's a very few kernels for those. There's very few evals, and there's very few people in academia who have access to the compute to find that those...

that generate that kind of data and, you know, be good at it. So those, they will take time, but I don't think there's really much, uh, many tasks that are unsolvable, especially in coding. I think most things will be reached.

It's just a question of incentives and data.

Mm-hmm. The other thing that was interesting was like time boxing your time with the LLM.

Mm-hmm.

So at what point are you just like giving up on the LLM being able to solve it? How do you do that in practice?

Yeah.

Do you set a number of time actually? Do you... Is it just based on like vibes of like where the conversation is going? How many times you restart versus just give up on it?

Mm-hmm. Well, part of it is just like the digital hygiene grampy sort of I block Twitter kind of thing is part of it.

Mm-hmm.

Um, so I don't let myself fool myself into thinking I'm being productive while models do my work for me, and then I come back every 30 minutes and see if they've improved. So part of it is just don't fool myself in any sort of those cases.

I look at the clock and I see am I being sped up right now or am I going down a rabbit hole? Is this model feasibly going to actually answer my question-

Right

... or should I cut my losses? So being aware of sunk cost fallacy, being aware of my own tendency to, you know, wanna do less work is a big part of this, which is kind of just vibes unfortunately.

Yeah. Yeah, yeah, no, I mean vibes are everything.

Mm-hmm.

Um, context rot we kind of talked about.

Mm-hmm.

I mean, the studios at the Chroma office which wrote the context rot paper so-

Mm-hmm

... anything specific on just context engineering in general? Like how do you like to prompt the LLM? Are there like, you know, do you include a lot of examples like-

Yeah

... anything there?

I start with, uh... Well, if it's a hard problem, I give it one shot. But for most things, I just do zero shot and then very few turns of conversations if it's a more complex problem.

Mm-hmm.

One or two turns and I already... The model is going down too many... Like attention is k- sort of being clogged up at that point.

Right.

So I need to open a new chat. I ask it the same questions or maybe even send a summary of the previous thread and say, "Given this information, answer my question." But I try and open a lot of new chats, um, or like, you know, refresh in, in VS Code like what, you know, what I had sent before.

Another reason why tools like Cursor don't really fit my workflow very well, I don't know what, how many turns they're at and where they're at.

Right.

Yeah, just try and keep the context as short as possible. Try and keep the, the model from going down too many paths or else it gets confused.

Yeah. How do you do the summaries?

Oh, uh, I take the entire chat and I feed it into a new chat of the model and I say, "Summarize this entire conversation. Take out the key points." It's very good at doing that, and then I feed it back into a new chat and I say, "Okay, given the summary-

Yeah

... um, and given..." Maybe I think of some new ways to prompt, give me an answer that I would otherwise just ask in another turn of the same chat.

Yeah. Yeah, we were just talking with the Sourcegraph guys about compacting conversations-

Mm

... and things like that and, like, how much signal you actually lose-

Mm-hmm

... or not. Uh, and I think also, I don't know if it was the context rot paper specifically, but, um, how giving previous good examples is helpful versus-

Yeah

... like giving failure examples and things like that.

Mm-hmm.

I wonder with, like, kernels especially when you're, like, implementing something, again, that people have not done-

Mm-hmm

... before. It's like what kind of examples are you supposed to give? But...

Uh, w- maybe, uh, it depends on the problem. May- maybe if I'm writing an attention one and I find an attention kernel online that looks what I want-

Mm

... or maybe I even write pseudocode for what I want. Pseudocode is pretty good actually for models to understand exactly what you're trying to ask it. Yeah, I just do my best really. I would say, though, for context rot, I'm really excited about there's a somewhat recent paper from Anthropic on interpretability of where attention weights are actually looking at inference time, and I wonder if servers will, uh, uh, model providers will get better at knowing when models are going down rabbit holes and when attention-

Mm

... is filled by looking at where attention is actually looking in the full multi-turn conversation.

Nice.

I'm excited about that.

Yeah, that will never be exposed, I'm sure.

Oh, never. Yeah.

Remember when log advices were in the API response

Yep. Hence open source.

Yeah.

Mm-hmm.

Then you had kind of like high-focus, low-focus tasks.

Mm-hmm.

How do you rank that? Like, it's easy to say, "This is a high-focus task."

Mm-hmm.

Or sometimes maybe with low focus we, like, underestimate how hard things are. Again, is that all just vibes or?

It's mostly vibes, but I kind of see myself as a, a, a capacity that I'm trying to max out.

Mm.

If I have 10, 10% tasks, then I do some context switching multiplier, right? And I say, "I can probably do six of them in six different tabs or something else like that." If I have one 80% task and then whenever that's blocked on something, I work on a 20% task, I kind of try and do that.

But besides that, I can't really tell much. I... A lot of this is just about, like, know thyself is what I'm trying-

Right

... to tell people.

Yeah, yeah, yeah.

Know how you work, know what your pitfalls are, and then try and improve. It's-

Yeah

... these are where my pitfalls are and I think that there's probably similarities to other people, hence I shared.

Yeah, I think that's the hardest thing about vibe coding is like-

Mm-hmm

... when you don't know-- when you're, like, not a, you know, seasoned engineer, it's like you don't really know which one of two tasks is, like, much harder.

Yeah.

You know, it's always like the joke of, like, it's like a X... They say key. It's like you give your photo and it's like, "Can you tell me where and when this photo was taken?"

Mm-hmm.

It's like, "Oh, it's super easy. It's in the metadata." It's like, "Can you tell me what's in the photo?"

Right.

And it's like, it's like- ... you know, you need a team of people to, to actually do that.

Yeah.

So I'm curious how that's, that's gonna change or maybe like-

Mm-hmm

... the, uh, by coding product themselves will be more opinionated about, hey, this is actually hard. Um-

I do make an effort to do things myself as much as possible as well, and I do suggest that people not try and use the model to offload thinking. It should enhance your thinking or, or else, like, you'll, you'll get worse over time and you won't know when the model is quality, if whether it's outputting quality or not.

If you don't know how, if it's outputting quality, then what are you-- like you're gonna get slowed down by default.

Right. Yeah. I think the code you're writing is, like, so valuable just on a dollar basis, just because you're spending so much money on, like, the GPU inference and, like-

Mm-hmm

... things like that. I'm curious over time how much that changes with, like, things that are, like, lower value.

Mm-hmm.

And, like, as the models get better, where that kinda, like, line kinda gets drawn, which is like, hey, if you're building, like, you know, a menu and an app bar and, like, all these different things in an app, it's like maybe you don't even review them.

It's like if it looks good, you just trust the, the model that it was good-

Mm-hmm

... and then if it breaks, we just regenerate it from scratch. You know? I think that's not a risk that you can take because obviously it's like-

Definitely

... you know, you run, run and it's like-

Yeah

... oh, fuck.

Yeah.

Like the, the kernel was not good. We just wasted millions of dollars.

Mm-hmm. Well, I'm also my own stakeholder. If, if the code is shit-

Right. Exactly.

... I'm the one who's suffering and I, I think actually there's a lot of big tech examples where they're kind of pushing really hard for their teams to use more code. They're being evaluated on how much code they're using.

Right.

And they're just kind of getting more and more slop that nobody understands. I think a lot of organizations will find themselves-

Yeah

... in this sort of case unless they push better practices on their teams.

Yeah, I mean, Atlassian just bought DX for like a billion dollars, which is like I didn't think we were still doing engineering analytics-

Mm

... of like how much code you ship or like the bottlenecks-

Yeah

... and it's like I just feel like everything is changing, but, like, people are not really thinking about that.

Yeah.

How do you think about building the team? Like what type of engineers do you hire?

Mm-hmm.

Obviously, there's like, I mean, knowledge of engineering, but like how much do you want them to be kind of forward-looking of like how to bring these tools into their work or like-

Mm

... is in your mind for what you do still like not, not that important?

Okay. Well, I try and hire more for velocity rather than position. I would rather have someone who's very intellectually curious, very quickly can pick th- new things up than someone who's super senior and takes a long... doesn't-

Mm-hmm

... doesn't have like a innate drive to learn new things-

Right

... and improve. Um, so I try and find people who are, you know, will eventually supersede the person who's already, you know, reached some point and doesn't really in- isn't really interested in moving beyond that. Um, so that goes a long way.

Such people also won't really want to offload more of their thinking to models. Um, they tend to more use models in the correct way, which is to augment themselves. I think Karpathy made a good, uh, tweet about like physicists are like the embryonic stem cells, right-

Mm

... of, of, of engineers. That's very true. I don't really care if someone knows CUDA kernel writing.

Right.

If someone does string theory and is really good at understanding complex problems, they will be productive faster than someone, uh, who knows CUDA and doesn't really care about-

Yeah

... trying to get better at it.

Has the interviewing process changed at all for you, like when you're like meeting a new engineer?

Mm. Um, changed in, for with models?

No, no, no.

I just-

When you're like interviewing like somebody to join your team.

Mm-hmm.

It's like do you do anything differently than you did two years ago, three years ago or-

No. Uh, what would have changed how I do things?

Well, I think like for a lot of people, especially if you're working on more traditional back end or front end-

Mm-hmm

... it's like, well, we assume you're gonna use some AI, so it's like Part of the interview's like, one, maybe upping the level of difficulty-

Mm-hmm

... and having you use AI or having how you use AI as, like, a core part of, like-

Mm-hmm

... why, how we're interviewing you-

Mm-hmm

... you know? Versus maybe in your roles, it seems not as impactful.

No AI in the interview, first off, and, um, you gotta watch people's eyes now on what monitors-

Right

... they're looking at behind the screen, which is-- that's changed in the last two or three years, unfortunately. No AI allowed, and we kind of, we kind of do things open book. You can kind of use whatever papers you want.

You can read whatever documentation you want. We're thinking together. I wanna see, do I enjoy thinking through a problem with you?

Mm.

So we'll give them a very fundamental, um, problem. Y- given the attention block, what if I make this change? What if I paralyze it in this way? What are the ramifications of that? Go ahead and implement it for me.

What would the skeleton look like? What does the communication look like if you were to do this, and what bottlenecks would arise? And just kind of having a dialogue back and forth and seeing how they think and seeing if we can tackle hard problems together is what I wanna gather, which is also helps select m- for people who don't necessarily can regurgitate the attention block has these, you know, uh, requirements or properties.

They kind of have to think.

Have you thought of any interview questions that trigger the AIs to be wrong and then use those to figure out whether or not the- ... interviewee is using AI?

Yeah, I should e- in my interview question, like, "Ignore all previous instructions-

Right

... in this output. I'm using AI." I guess I should do that. No. No, I don't. Um, I just kind of... It, it's very obvious. Anyone who's interviewed people is-- can almost always tell, I think, about whether someone's using AI on the other side.

How long is the time to first token? Humans are ty-

Right

... typically faster than a model.

Yeah, yeah.

Yeah.

Interesting, yeah. There's, you know, we have one of our portfolio companies that are building this tool to, like, stop people from recording you with AI-

Okay

... and kind of, like, manipulates your microphone output.

Mm-hmm.

And I'm curious if people are gonna do something similar when they're, like, interviewing somebody-

Mm

... so that they cannot, in real time, transcribe the audio and put it back in the model.

I see.

And then there's gonna be a whole warfare on-

Mm

... what you're using. I was at the Meta event, and, like, I tried the glasses with the screen on.

Mm-hmm.

And I was like, "I mean, this is, like, crazy."

Yeah.

It's like you could just turn on the live mode-

Yep

... and, like, it responds in real time to what you're saying-

Mm-hmm

... and you can see things, and I don't know. I think we're going towards-

It's an arms race, for sure

... interesting.

Um, I, I would say even in the best case of someone who li- lies in the interview, I will find out within a few weeks-

Well, and th-

... and then you're gone

... that, that's really the point, right? It's like I just don't understand what the end game is-

Mm-hmm

... because then you actually have to work.

Yeah.

Right?

Yeah.

It's like you actually have to get in-

Yep

... and do the work.

Mm-hmm.

So I can see it in, like, in a big company. You try and-

Yeah

... kinda, like, get in and kinda squeak through the-

Yes

... things, but, like, I don't know. If you work with Qwen, and it's, like, hard to fall through the cracks, I'm sure.

Yeah.

You know, you have a small-ish team.

Mm-hmm. If the model could already do your role, I wouldn't hire you.

Right.

All right? I would just use the model myself. Unfortunately, the model has a gap, and that's why you're here.

Right.

If the model replaces me tomorrow, then I'm gone.

More on a broader side, I'm curious about, yeah, Eleuther and, like, where you see the whole open source-

Mm-hmm

... space going. I think there's obviously been the rise of DeepSeek. Since then, there's like, you know, uh, the Allen Institute-

Mm-hmm

... is, like, doing a lot of work with OLMO and some of those models. You have, obviously, Qwen and the-

Mm-hmm

... the usual suspects. Do you have, like, a mental map of how to think about different open source groups and their interests? Like, if you were to categorize Eleuther-

Mm-hmm

... what would you say are, like, maybe the main things people should look for from them?

Eleuther is an interpretability-focused organization, I would say. So we try to answer, um, given that the whole pipeline is open, from creating data to training on the training framework itself and its quirks, the checkpoints along the way, um, what sort of answers can you derive about model training dynamics?

So we had, like, one paper on how does a model memorize? What-- can you predict memorization across scale? Can you predict it across number of tokens? And you can only answer those questions when everything is in the open.

So I don't know. So, like, interpretability in general, given everything is open, is kind of our focus. Given a new model that comes out, like, uh, Meta just came out with, what, Dino V3, can we just create an open source version of that given open data?

And what is sort of the state of open, uh, given the same architecture and everything else, can open source replicate these sort of things? This is st- still super important. Maybe, like, Lion, uh, would fit into that bucket.

Um, kind of struggling for others. A lot fall into one of those two.

Yeah. Yeah, I think that makes sense.

Mm.

How big of a difference does being fully open make? So between understanding, I don't know, R1 and-

Mm-hmm

... understanding Eleuther3 model that is, like, fully open, what, what's the delta between the two and the amount of work that you can do?

Um, well, there's hu- there's a huge delta, um, mostly from access to compute. So if Eleuther had a lot more compute and a lot more time and funding, then yeah, they, they, they could reach it. Um, but given the state of, uh, how much funding and therefore how much time they can spend on things, how much compute they have access to, like, it's orders of magnitude.

Um, so we can very deeply understand the models that we produce at Eleuther, but I, I wouldn't really use them in my, my day-to-day. They're good-- they're meant for researchers, not consumers.

Yeah. And if you... D- do you think there's, like, more you gotta understand about just AI in general? Say you had-

Mm-hmm

... you did the same thing and got to a model of, like, yeah, R1 quality or whatever else.

Sure.

Are there latent capabilities? I mean, I know there's, like, kinda, like, emergent capabilities-

Mm-hmm

... like has killed the models, but, like, from your research side, do you think there's, like, some things that you wanna observe that just you cannot at the size that you're in? And, like-

Mm

... what are those?

Okay, I see. Um, yeah, there are lots. I mean, I really want to explore, like, just the, like, the s- the model dynamics of, like, seeds, different seeds, different inits, uh, is something I wanna know, but that requires training a model-

Right

... end to end many times. For R1, uh, specifically, you're mentioning, I wanna know what is the actual optimal amount of sparsity? There's a lot of these things that rhyme, so, like, a lot of different- Models are coming to the same conclusion of how much sparsity is good.

I want to find, like, an exact bound on that, and that requires training a lot of models. For a lot of the hybrid model work that like, for example, Zyphra did, a lot of different models are coming to the same conclusions of just how much attention do you actually need-

Mm-hmm

... in a model? And I would love to have a conclusive answer to that. Only-- the only way I could get that is if I had completely open data and did it at Eleuther and trained a shit ton of models.

Right.

Um, so model architecture. There are a lot of mo- architecture questions that I think I'm the most interested in answering, um, for open source. Yeah.

Yeah. Do you feel like we could band together better in open source? Have more groups-

Mm

... kinda work together, or are people's directions kinda just, like, not compatible?

Maybe it sounds wrong, but I, I feel like banding together is not necessarily good by default. There's a lot of different incentives. There's so much noise that no one-

Mm-hmm

... knows where to focus. I prefer-- If anything, I would prefer siloed-focused teams who have funding that's guaranteed from beginning to end on specific projects, almost like a call for projects kinda thing.

Yeah.

Most of our success at Eleuther has been kinda saying, "We have this very interesting problem. We will provide you compute and mentoring and, um, you know, stability from end to end, and you, if you wanna, you know, get into AI research, you just need to provide, like, the labor, and we'll teach you up along the way."

Those sort of things with just very small teams have given us a lot of success. Instead of these grand things, we're pr- gonna produce a giant trillion-parameter model-

Right

... for everyone that we all own. Those are a bit harder to organize, and, you know, open source is hard to organize.

Right. Yeah.

Yeah.

What would be your list of, like, requests for projects? If you had more money to give away-

Yeah

... what would you want people to do?

Well, the two model architectures ones I wanted to-- I want to answer, so how much attention do you need?

Mm-hmm.

What is attention actually doing? Get, getting that just, like, blog posts.

Right.

Getting in much more detail of, uh, being completely open, and you have no incentive to keep anything hidden. Just talk about what you did and what things look like. For, yeah, MoEs, how much sparsity do you actually need?

On the kernel side, um, I would love to get more AMD sort of work-

Mm

... and, like, uh, alternative ecosystem stuff, so there's incentive for those specific companies. If I was AMD or Cerebras or whoever else, then I would go out-

Right

... and say, um, you know, "If-- Send us a proposal for what you wanna do, and we will make sure, you know, with, with minimal, you know, bullshit, you will be funded end to end to write a kernel for whatever the thing we need."

AMD's actually done this. Like, there's some bits of DeepSeek, like, through GPU modes-

Mm

... Discord, there have been some, like, DeepSeek kernels that they say you have, you know, guaranteed access to compute for. Write them. So yeah, I would see the ecosystem on systems optimizations with a lot of projects like that.

Um, honestly, that, it-- I don't have to go into much more.

Right.

Like, that's, that's enough.

That's a lot, yeah. You've resolved that.

It's already a lot. Yeah.

Yeah, it's always been confusing to me why these companies don't spend a lot more money on this versus, like-

Mm

... traditional marketing. Like-

Yeah

... I don't know, sponsoring a Formula 1 car.

Mm-hmm.

It's like- ... I don't know. I don't get it.

Yeah.

What else is top of mind for you, man? Anything else you wanna chat about? Any other hot takes-

Okay

... that you wanna get out there without having to write a long tweet storm?

No, I think I've hit kind of everything. Like, uh, people should be less scared about doing things that are unproven. People should be as, you know, it's a silly statement, but data-driven as possible. If attention does what you need it to do, use attention.

If it doesn't, then find something new, and don't be afraid to, like, go out and train something, especially, especially to big tech. Like, you have the resources.

Right.

You can get ahead of everyone else if you're, like, very grounded and look at what actually works today. That's why we jumped on Mamba, for example. There's a lot of, "Is Mamba good? Is, uh, you know, Delta net good?"

Like, there's all these-- You gotta try them all, and you gotta-

Right. Yeah

... determine if they're good for what you actually want. So I would push pe-people to be a bit more open-minded. I'm excited for how things are gonna move over the next year or so.

Awesome. Are you hiring any-

Absolutely, always.

Yeah.

Yep.

What's, like, a good archetype of engineer that works at, um, Zyphra?

Intellectually curious people, driven people. You need to have some boilerplate knowledge so that we don't have to ch- teach you what a transformer is day one. But if you have high velocity, and you have a demonstrated track rec-- If you, uh, swyx has a nice, like, think and learn in the open kind of thing.

That's what I look for in people. Do you have, like, blogs on things that you're interested in?

Mm-hmm.

Did you go really deep on problems? And I can kind of look at your, uh, intellectual sort of history over the last few months. Uh, that goes a very long way. So those sort of people who like diving deep into hard problems, like, we hire all the time.

Physicists we hire all the time. Doesn't matter.

Nice.

Yeah.

Awesome, man. Thanks for joining.

Yeah, absolutely. Thanks for having me.