# ⚡️ Beyond Transformers with Power Retention Latent Space · 2025-09-23 Jacob Buckman, CEO of Manifest AI, discusses their solution to AI's expensive computational bottleneck: the ever-growing KV cache for long context inference. Power Retention replaces attention in transformers with a fixed-size memory, achieving 10X training speedup at 64K tokens and 100X inference speedup over FlashAttention. Manifest releases Vidrial, a just-in-time CUDA kernel framework that finds optimal configurations, and Metamorphosis, a process to convert existing transformers like StarCoder-3B into Power Retention models in just two hours of mid-training. Buckman demonstrates PowerCoder, a 3B code model that matches baseline loss after 10K steps and handles contexts up to 32K and beyond. The open-source release includes kernels and tools, aiming to build community trust and encourage adoption by inference providers. He also highlights the need for genuinely long-context datasets beyond internet text, such as human trajectories. ## Questions this episode answers ### What is Power Retention and how does it improve on transformer attention? Jacob Buckman, CEO of Manifest AI, explains that Power Retention replaces the transformer’s attention mechanism, which requires a perpetually growing KV cache, with a fixed-size memory. Each new token is compressed into this memory, preventing the quadratic cost growth. The memory size can be scaled based on compute budget, similar to model parameters, enabling 10× training and 100× inference speedups at 64K context lengths. [4:06](https://addtry.com/44a8d23b-7530-487d-8d7b-7352b774bc69?t=246000) ### How does the Vidrial framework optimize CUDA kernels for AI models? Jacob Buckman describes Vidrial as a CUDA kernel framework that uses just-in-time sweeping to find the optimal configuration for any given operation and hardware. Instead of a kernel designer hardcoding decisions, Vidrial writes a generic kernel that explores all possible implementation choices—like tile shapes and pipelining—then empirically tests them to select the fastest for each problem shape, often yielding 20–30% speedups over standard kernels like FlashAttention on non-standard inputs. [8:38](https://addtry.com/44a8d23b-7530-487d-8d7b-7352b774bc69?t=518000) ### What is metamorphosis in the context of Power Retention, and how much training does it require? Metamorphosis, as Jacob Buckman explains, is the process of converting a pre-trained transformer into a Power Retention model by swapping the attention mechanism and fine-tuning all parameters. He demonstrated with StarCoder-3B: after replacing FlashAttention with Power Retention, loss initially spiked, but just 10,000 training steps (around two hours) recovered the original loss, and additional training at longer contexts pushed loss down further, enabling efficient extension to 16K tokens. [14:45](https://addtry.com/44a8d23b-7530-487d-8d7b-7352b774bc69?t=885000) ## Key moments - **[0:00] Manifest's Mission** - **[1:20] KV Cache Problem** - [1:40] Jacob Buckman identifies the growing KV cache as the fundamental bottleneck for scaling long contexts. - [2:54] Jacob Buckman reveals that 256K+ context models use windowed attention, discarding past data and causing middle-context degradation. - **[4:06] Power Retention** - [4:21] Power Retention: a fixed-size memory that compresses tokens instead of a growing KV cache. - **[5:27] Inference Benefits** - [6:10] Power Retention’s fixed memory size simplifies inference by partitioning GPUs into static slots, eliminating KV cache scheduling overhead. - [7:21] Power Retention delivers 10x training speedup and 100x inference speedup over FlashAttention at 64K context length. - **[8:14] Vidrial** - [9:58] Vidrial, Manifest’s CUDA kernel framework, uses just-in-time sweeping to find the optimal GPU configuration for each input shape. - [12:16] Vidrial outperforms FlashAttention by 20-30% on non-standard input shapes by empirically tuning kernel parameters. - [13:40] Power Retention is a drop-in replacement for FlashAttention: pip install retention and change one line of code. - **[14:27] Metamorphosis** - [15:00] Jacob Buckman: 'You start as a slow crawling transformer caterpillar, and you end up as a beautiful lightning fast on long context power retention butterfly.' - [16:16] Jacob Buckman shows that metamorphosis from StarCoder to PowerCoder recovers full loss in just 2 hours of training (10k steps). - [17:13] PowerCoder achieves 35% HumanEval accuracy (up from StarCoder’s 30%) after metamorphosis and continued training. - **[19:35] PowerCoder Demo** - [20:31] Manifesto demo: PowerCoder repairs a broken layer norm class in the NanoGPT repository after analyzing the codebase. - **[22:43] Open Source Strategy** - [23:32] Jacob Buckman hopes the open-source community will create retention variants of popular transformers, building an ecosystem similar to Hugging Face. - **[26:15] Scaling Roadmap** - [28:21] Jacob Buckman predicts that within 6-12 months, large foundation models will be trained with Power Retention, following a similar community adoption curve to transformers. - [28:38] Power Retention trains 5x faster at 32K context: 4.8 vs 23 seconds per iteration on a 3B model. - [29:36] Manifest is hiring deep learning and CUDA engineers for foundational architecture breakthroughs beyond tweaking transformers. - **[30:09] Data Challenge** - [30:09] Jacob Buckman says long context models only shine on data with long-range structure, not stitched short documents, and seeks datasets like human computer-use trajectories. ## Speakers - **Alessio** (host) - **Jacob Buckman** (guest) ## Topics Language Models, Inference ## Mentioned Fireworks (company), Manifest AI (company), SF Compute (company), Together (company), FlashAttention (product), Log Cabin (product), Manifesto (product), Metamorphosis (product), NanoGPT (product), Power Retention (product), PowerCoder (product), QuenCoder (product), StarCoder (product), Triton (product), Vidrial (product) ## Transcript ### Manifest's Mission **Alessio** [0:04] Hey, everyone. Welcome to the Latent Space Podcast. This is Alessio, founder of Kernel Labs, and I'm joined by Jacob Buckman of Manifest AI. Welcome. **Jacob Buckman** [0:12] Thanks for having me, Alessio. **Alessio** [0:13] Of course. You know, we at Decibel, we invested in Manifest right at the start. That's about two and a half years ago. You had just left Google Brain. Carlos had just left, uh, OpenAI, and I think you set out on this mission of unlimited context window for models. And, you know, to set the context, when you started, the context window of these models was, like, eight thousand tokens, you know. So it was very, very limited. So maybe just do a quick background on, like, the origin of the company, you know, some of the beliefs behind it. And also, you've done a great job of kind of, like, staying focused on your beliefs instead of just chasing what the mainstream discussion is. So yeah, just maybe give people a quick history of Manifest. **Jacob Buckman** [0:54] Yeah, absolutely. Manifest came from a collaboration between me and my co-founder, Carlos, basically thinking about what the real breakthroughs to get to the next sort of phase of this technology would be. So sometime in, you know, early 2020, 2021, the writing was sort of on the wall for people who were working in the space that by just basically scaling up what we had, we could unlock some really crazy potential. ### KV Cache Problem **Jacob Buckman** [1:20] And the question was: How much potential? Can it take us all the way to, you know, AGI, like proper human-level intelligence on, on every task? And the conclusion we came to was not quite. We have a lot of pieces in place that could get us there, but there's one technical bottleneck that we felt was actually going to be a blocker, and it is basically the problem of large inputs. And in the, uh, sort of like frame of language models, large inputs means long contexts. It's a question of how much content can be synthesized all at once by the model. And we know how to scale up the models to make them smarter by increasing the parameters, and we can do that very efficiently and at scale on large clusters. And also, we know how to sort of scale up the total amount of knowledge ingested by the model, training them on larger and larger datasets. But we do not have good, like, cost-efficient ways to scale up the context. And this is something that might be surprising given how far everybody has pushed their context lengths in recent years. But actually, what a lot of people are doing is not solving the core underlying problem behind the, like, very expensive cost of long context. It's the fact that the KV cache is always growing. That's sort of the, the fundamental bottleneck, is that every single token that you see, you have to remember in the KV cache. And this makes every subsequent computation, like every additional token you predict, get more and more expensive. And this growing cost just quickly becomes the bottleneck of the whole thing, and it winds up intractable. And rather than sort of like solve this at its core and rethink attention, people instead thought up various quite useful, but ultimately workarounds, right? Band-Aid solutions, things like windowed attention. So this is very common. Anybody who says they're using a transformer with a context length of, you know, two hundred and fifty-six thousand or more, they're not using a true transformer. What they're using is a windowed transformer that essentially throws out a huge amount of its information at various layers within the network. And this does kind of work, but it doesn't really work. We've tested these models extensively, and they're actually just much worse at processing information from further and further back in the past. And this is why you see the funny phenomena of long context models sometimes being really good at using things at the very beginning of their context and really good at using things at the very end of the context, but sort of messing up and getting fuzzy on everything in the middle. A lot of this is just purely architectural. These architectures just aren't designed to do uniform computation across the whole context length, and so they end up with these weird idiosyncrasies once you train them. And a lot of this we at Manifest only realized along the journey, you know. It's not like we had the full picture laid out at the beginning. But we did recognize the importance and sort of like the fundamentalness of scaling up the input as a problem. And so we founded the company to essentially tackle this at the deepest level. ### Power Retention **Jacob Buckman** [4:06] And what we ended up coming up with is a different family of architectures. Replacing transformers entirely ended up needing to be the way to go. So we call it retention-based architectures. Power Retention is the specific variant that, uh, we're about to release. And it basically works by instead of the memory constantly growing, this, this constantly growing KV cache, you have a memory that is a fixed size, and each new token, each, each new token simply gets compressed into this fixed size memory, so it doesn't grow. But crucially, the memory itself can be scaled up or down based on the difficulty of the problem and the compute budget. So in this regard, our architecture has a state, like a memory, that is very similar to how we oftentimes think about parameters in other models. Just like for a really big training run, you want to use a ton of parameters, similarly, for a really big training run, you want to use a giant memory. That's going to help you. But if you're very, uh, inference latency sensitive, for example, if you're going to be using a model to sample many, many tokens and you want to get the speed up and the price down, then you don't want to be using a large number of parameters. And similarly, you don't want to be using a large memory. And you certainly don't want to be using a constantly growing memory that makes each token more and more expensive. So this is sort of the, the space of ideas where Power Retention fits in. And yeah, I'm excited to see what people do with it. ### Inference Benefits **Alessio** [5:27] Yeah. That's awesome. Are there any fundamental differences? So you mentioned the KV cache, right? The KV cache does allow us inference providers to offer, you know, cache prices on tokens. So if you kind of keep the prefix the same, then, like, you only get incremental costs on the new tokens added. Are there big difference, so now you run inference with a Power Retention model versus a normal attention model? Or is it gonna be for most, um, you know, for most models, like a pretty easy switch? **Jacob Buckman** [5:56] It's actually gonna be way cleaner on the engineering side to do inference with Power Retention and similar models. The KV cache, in addition to the fact that it's big, one of the real challenges just infrastructurally is that the size is dynamic, right? The more a user is querying the model or the more tokens they're querying it for, the larger the KV cache gets. And so allocating a GPU to multiple users becomes very tricky because each user has this constantly growing, and when you wanna like remove a user from that GPU, there's a whole scheduling problem to decide which new users get their caches loaded in. You wanna pick ones that fit, but not that are too close to the limit because then you're gonna have to rotate them out pretty soon, but not ones that are too far from the limit because having a ton of extra memory is very wasteful. It's just a really challenging engineering problem to efficiently use a whole fleet of GPUs to serve inference for many users when you have these dynamically growing KV caches. But with Power Retention, this completely goes away. The retention architecture just comes with this fixed size memory, and so every GPU can just be partitioned into, say, eight pieces. This GPU has enough memory to serve eight users, and that's it. They never grow. They never shrink. You're never wasteful. When you have seven users, you have one extra slot, and when the eighth user joins, you put them there no matter who they are, no matter what they're querying, no matter how long they've been using it before. It just doesn't matter. So it actually is going to meaningfully reduce inference like complexity as well as costs. And just to give you some sense of how much, at training time, the sort of speed up that we get purely from the flop reductions and efficient hardware-level implementation of flash atten- of, of, of Power Retention over flash attention at a context like, like 6K tokens. Obviously, our, our speed ups actually grow the longer the context is. And at 64K tokens, we get something like a 10X speed up at training. But at inference time, because you're not only saving flops at inference time, but also paging in and out of memory of the KV cache, you actually get 100X speed ups from Power Retention over like flash infer at context lengths at around 64K and only growing from there. So these are real material differences. It's going to be a massive, massive reduction in inference costs and complexity, um, as a result. ### Vidrial **Alessio** [8:14] Yeah. And so you're also releasing, um, just to go from the bottom up, uh, Vidrial, which is a framework for CUDA kernels. So I'm, I'm assuming you're gonna write-- you're gonna release all the kernels for like Power Retention. Um, and then I also know you rewrote the flash attention 2 kernels, uh, and actually faster than the ones that Trida originally implemented. So talk about that level of, of the stack. **Jacob Buckman** [8:38] Yeah. So the Power Retention architecture is very flop efficient, but this is not actually useful if it's not hardware efficient too, right? If the GPU is getting wildly underutilized, then it doesn't matter that you're doing many fewer computations, it's gonna take longer and therefore ultimately be more expensive, right? So one of the key goals we had at Manifest was to release an architecture that isn't just theoretically faster when you count the flops, but is actually practically faster in terms of wall clock time. And that required a huge investment into getting really, really good at writing ultra-efficient hardware-friendly kernels. And Power Retention is a very tricky thing to get right. It's actually in many ways significantly more involved than flash attention or a lot of the other kernels that people, uh, like to write. And so we actually couldn't use something like Triton or Palace. That's one of these more higher level Pythonic frameworks. Actually, our initial version of Power Retention was written in Triton, but we realized quickly that it just didn't offer the flexibility to really squeeze the performance that we wanted out of the GPU. So we took a step back and dove into CUDA, and before we knew it, in order to write the kernels that we wanted to write for Power Retention, we actually had to write an entire fully general framework for writing CUDA kernels, and that's what we call Vidrial. And sort of the key intuition behind Vidrial is the idea of just-in-time sweeping. So when you're designing a kernel to do a particular operation, mathematically it's going to do one thing, right? You always want it to produce the same output. But computationally, there's a huge number of ways for most kernels to actually implement. The GPU has all sorts of different hardware features and the choice of basically like which cores to use to copy data, where to copy it to, when to send it to the tensor cores, all sorts of things like tile shapes and pipelining that really can completely change the performance of the actual kernel even if the outputs are always the same. And for most kernels, most CUDA kernels, you basically as a kernel designer just figure it out. You just think really hard and try a lot of stuff, and you write a config that is-- you write a kernel that is configured to be in general making the right decisions. And then you release it, and everyone uses it and gets pretty good performance and is happy. But we realized that this can be done much better because on any particular user's hardware, any particular problem shape, you know, like the shape of the input, the memory layout, et cetera, there is an optimal configuration, and it might not be one that the kernel designer thought of. But rather than having the kern- putting it all on the kernel designer, we could just write the entire space of possible ways to execute any given mathematical operation on a GPU and let the user sweep over all the different ways to compile this kernel and empirically decide which is the best one. And this is a little expensive. It takes a few minutes. But ultimately, if it takes a few minutes to shave off twenty percent a sort of like call to this kernel on a kernel that's going to be called billions of times on a huge cluster over a three-week training run, then it is unbelievably worth it, right? And so Vidrial is all about writing the most generic possible kernel. Essentially, you're never just writing one kernel. You're writing every possible kernel that could implement this operation. And when you're actually executing it, you're sweeping over all the different ones, measuring the time for each one, and always choosing the best. And the result of this is that if you take something like FlashAttention and implement it in Vidrial, then on the problems where FlashAttention is really well optimized, we end up basically matching performance, one to two percent faster sometimes. But on the problems where FlashAttention is not well optimized, which is a huge number of problem shapes just because there's so many possible problems, we can get twenty or thirty percent faster, literally just by making really intelligent decisions about how to use the GPU hardware by empirically testing different ways of setting things up. **Alessio** [12:34] And the, the inputs to that are the model plus the GPU. What, what are like the-- like, you know, once you do this, does the same sweep kernel work for most people? Or is it very specific to, like, the number of GPUs, the size of the cluster, the type of run that you're doing, blah, blah, blah? **Jacob Buckman** [12:51] Yeah. We find it's, it's pretty robust to different setups. So yeah, we're also, like, releasing some precompiled, pre-optimized kernels. So if you're doing something very normal, right? Like if you're doing, uh, FlashAttention with a batch size of eight and a head dimension of sixty-four and eight heads on a sequence length of a thousand and twenty-four, like this is a totally bog standard thing. If you just run NanoGPT out of the box, you're gonna be calling the kernel with this exact sort of input shape. And if you're doing that, you won't have to sweep yourself. We'll be releasing a, a, a pre-swept kernel that'll do it for you. But if you're straying from that, and you wanna use a sequence length of like one thousand two hundred and forty-three for some reason, now we're going to be way faster than FlashAttention because we're actually going to find the right way to configure the kernel to call it on that specific input shape. **Alessio** [13:40] Great. And so people now can just do pip install retention basically, um, and that's it. It's, uh, it's pretty, pretty plug and play. **Jacob Buckman** [13:48] Yeah. It's completely a drop-in replacement for FlashAttention. So yeah, you... From the retention package, you can access our FlashAttention kernel written in Vidrial, and you can also access Power Retention, which is the sub-quadratic architecture that I was talking about earlier, which not only is extremely hardware efficient 'cause it's written in Vidrial, but also has these huge flop savings because it's genuinely compressing the memory instead of memorizing every single thing that it's seen. Import and a one-line change, and it'll... You're off to the races. You'll see huge speed ups if you're doing any sort of training or inference on long contexts. ### Metamorphosis **Alessio** [14:27] Yeah. And, you know, the other thing is this is not a change that requires pre-training. So you can take a pre-trained model with some architecture and use this to kind of like, like you called it Metamorphosis, turn it into like, uh, the new power retention architecture, basically. **Jacob Buckman** [14:45] Yeah. So it's not something that requires pre-training at like a real pre-training scale. It does typically require a little bit of what people are now calling mid-training, basically fine-tuning but on all the parameters, right? It doesn't require very much though, which is why we... This process of metamorphosis, right? You start as a slow crawling transformer caterpillar, and you end up as a beautiful lightning fast on long context power retention butterfly. And it actually, um, I can actually show you some training curves if you're interested. Takes surprisingly little compute there. So, um, yeah, maybe I can share my screen here. Uh, so yeah. This... There we go. So here's an example. This is our, uh, in-house sort of run visualization platform. I call it Log Cabin. And this is basically showing... This blue line here is a model called, uh, StarCoder-3b. This was released by, uh, the BigCode project, sort of Hugging Face affiliated, small-ish but real meaningful scale, three billion parameter coding model. And, uh, this is the baseline level of loss that it achieves on like a very standard, they call it the stack dataset and specialized to Python. And what happens when you switch it from FlashAttention to Power Retention? Well, you get from this blue loss right up to this orange loss. You take a big hit. Like, immediately the model is just not making the right predictions basically anywhere. But after just ten thousand steps of training, which this training one took about two hours, this orange curve, you see that it fully matches the original loss. And from there, you can start pushing the context length. So we did more training at 4K and then even more training at 16K context. And you can see that you can really keep pushing the loss down. And not only that, but the long context training will actually be fast, where if you tried to train base StarCoder on 16, it would take way, way longer to do this. And here we have a downstream evaluation. Maybe I can zoom in here a little bit. **Alessio** [16:44] Yeah. **Jacob Buckman** [16:46] And- **Alessio** [16:46] And the, the base model is, um, eight thousand, right? Eight thousand a hundred and ninety-two token sub context window? **Jacob Buckman** [16:54] I think it's just four thousand and ninety-six actually. **Alessio** [16:56] Four th-- Okay. Yep. **Jacob Buckman** [16:57] But they did a little bit of longer, uh, context post-training, but using windowed attention. So they didn't actually extend the attention to 16K. They kept it at 4K, but just used the sliding window approach to train on longer sequences. And yeah, so this is, this is a downstream evaluation. This is the, um, human evaluation, like HumanEval likelihood and basically corresponds to the performance. So, um, we're basically perfectly able to match after just two hours of training the downstream performance on these tasks, and then continued training allows it to get even better. In the end, this converges to, I believe, about thirty-five percent accuracy on HumanEval, whereas the StarCoder baseline was about thirty percent. And yeah, so this is just like Very manageable, very achievable, uh, for, for a lot of different groups. And when you get out at the end is this, uh, metamorphosized model that is just as good as Starcoder, but with power retention, so we call it PowerCoder, and it has way cheaper inference time, inference costs as well at long context. **Alessio** [18:00] What's the cap? So you have 16K context. Could it go to 32, 64K? Like, how much can you expand the context size? **Jacob Buckman** [18:11] Um, we actually have not found a limit. It just comes down to resources. Like, um, each subsequent training costs more than the previous. So this c- costs about two hours to train at 1K, and then this represents another about six hours of training, and then this represents another 16 hours of training. Um, so it does get, like, you know, you're training on more tokens, right? So it gets more expensive as you train on long context just because you're, you're training on more tokens. So, uh, yeah. But, um, we don't pay a quadratic cost. If you were looking at the Starcoder baseline, it would get even more, more expensive way more quickly. Um, so it's actually these 16K runs become very tractable, uh, for, for a lot of groups. But yeah, we could push it to 32K. We could push it to 64K. We could probably push it beyond that, although, uh, that would then require doing some, like, tensor parallel or sequence parallel, so that just would be, uh, a little more implementation effort that we, we haven't done yet. Would be very doable. **Alessio** [19:03] Um, and you ran this on SF Compute, I saw on your, uh, on your announcement. **Jacob Buckman** [19:09] They're, they're one of the main compute providers we used for the research, and this act- one actually was not trained on that cluster, but yeah, they're, they're great. **Alessio** [19:18] Nice. Yeah, we had them on, on the pod before, so I always try and shout out previous guests. So you now have also built a PowerCoder, like obv- I mean, obviously you've shown the, the loss, the loss curves. Uh, but you're, like, actually releasing the PowerCode Python, uh, model, right? ### PowerCoder Demo **Jacob Buckman** [19:35] Yeah, I can show you a demo of that too. I have it over here. Um, actually, before I do that, I just wanted to show off real quick the difference in training speeds. So this is two runs. Uh, these are both at 32K context, and the only difference between these two is this is the Starcoder 2 baseline at 32K, and this is the PowerCoder at 32K as well. And so I'm just gonna kick these both off, really give this guy a little bit of a head start, and, uh, we can come back in a bit and see how much progress each of them has made. But yeah, so here is the PowerCoder model in action. We decided to go for a slightly different interface. Everyone does a little, like, you know, chatbot, whatever, so we wanted to do something a little bit different. Um, we made a tool called Manifesto, where basically you can point it at any repo, including a very large repo, and it will basically fix any mist- in that repo or, or do its best to, of course. It's only a three billion parameter model, so, you know, it has its, it has its limitations, but it can do some very cool stuff. So here I just set up that this is like a, the Nano-GPT repo, and we can just go look at, you know, um, like the model.py file, for example, right? This is, like, completely standard model.py. Um, and you can see the, there's, there's lots of other files in here too, just like regular Nano-GPT setup. **Alessio** [20:58] Can you make it slightly bigger just so it's easier? **Jacob Buckman** [21:01] For sure. **Alessio** [21:03] Just zoom in a little bit. Yeah, that's good. **Jacob Buckman** [21:05] There we go. Okay, so yeah, let me show that again. We just have like, uh, as you can see it here, right? Nano-GPT, just like with normal training files in it, and now, um, if we go into model.py, you can see this is just like a very ordinary Karpathy-style setup. And so let's say we just, like, deleted, uh, this class and replaced it with this comment, and then maybe we can do one more here. So now these just not unimplemented functions. And now, uh, we can basically see what this does, and it will basically just take a look at the repo, read all the code in the repo, and propose some repairs. So you see here, model.py, the one we messed up, and it starts repairing layer norm. And yeah, it's, uh, basically just giving, "This seems like a valid layer norm class to me." Honestly, this probably, like Nano-GPT probably appeared in its training corpus at some point. **Alessio** [22:07] Right. **Jacob Buckman** [22:08] Yeah. Um, but yeah, this is the idea, and it's, uh, generally pretty good on Python repos. The distribution was weighted towards repos that have a majority of files ending in .py. So it is mostly good at that, and again, that was just like a computational, uh, sort of efficiency thing. We just decided to target something smaller scope because we're a pretty small lab and, and a little resource-constrained. But yeah, and look, it also made one more change. It added a, a log or two. So that... And these will be released on Hugging Face for people to play with along with a command line tool. ### Open Source Strategy **Alessio** [22:43] Do you see more of these models being built for different languages? Do you think that's like a... Now that you have this new architecture, it's like, is that a good direction for people to work on? Do PowerCoder TypeScript, PowerCoder Rust, blah, blah, blah. **Jacob Buckman** [22:57] Yeah. I mean, ultimately, I would just like a PowerCoder everything, right? Like, you should just be able to train one sort of foundation scale model that does it all. And, um, we are building up to that. That's something we plan on doing at the, like, 30 billion scale in the coming months. This is sort of just like a first demo, you know, just a, a taste of what's to come. And something that I'm really excited to see is since we're going to be completely open sourcing all of the pieces that you need to do this metamorphosis yourself, I think there's gonna be a lot of groups who have, uh, sort of different needs, and they wanna metamorphose the different base models. Maybe someone wants to do QuenCoder instead of Starcoder, or someone just wants to do, uh, a text model or a question answering model or an instructor tune model instead of a code model. And we'll release all the tools needed for them to do that. So they can just get a handful of GPUs- It basically pip install retention to get our kernels, modify the architecture. We'll be releasing guides and tutorials on exactly what the modifications need to look like, but they're honestly very, very straightforward. And they'll be off. They'll come out with a nice, shiny new, uh, power retention architecture that has the same performance on whatever dataset they want as the original base model did. And my hope is that people will start to open source these. So just like right now, there's a large ecosystem of open source transformers on Hugging Face and other places that lots of people use and benefit from. I'd really like to see the same thing happen, like each of these models be transformed into a retention variant and exist alongside the original. And I think this would be really great, especially as people start thinking about things like, uh, chain of thought, where you're sampling tons and tons of tokens at inference time and inference costs really start to add up. I, I would really love to see, uh, people basically switch over to retention for those use cases, because you're going to be able to save an enormous amount of both money and time, um, on those, those long context inference use cases. **Alessio** [24:54] So if you are Together or Fireworks or any of these, like, inference providers, should you just be doing this for every single model? Like what are, what are like maybe the downsides? Otherwise, it just sounds like it's... Does it make sense to not do this? **Jacob Buckman** [25:09] No. I think it only makes sense to do this. And yeah, uh, I love the guys at Together. I think they're, they're, they're really great and very, like, open source friendly as well, which I really appreciate. So they're one of the people I think I would love to partner with. You know, I think they stand to benefit a lot basically from, you know, like once a model is in wide distribution enough in the open source space that people, like, trust it and like it, then it becomes appealing for them to serve. And so I think they're a very natural partner here. Maybe we, we could work together to train some of these big power retention metamorphosizes- metamorphoses that they can then serve to their customers at much lower prices and much faster latencies than any attention model at long context. **Alessio** [25:57] Cool. Um, I know some of them are listening, so- **Jacob Buckman** [26:01] Hey **Alessio** [26:01] ... maybe they'll, they'll reach out. What else is on your roadmap, I guess? You know, so you started with this idea of like, you know, longer context should not come with a quadratic, kind of like a penalty, so to speak. Do you feel that from here you know what's next to like start scaling this? You know, what, like what-- how far do you think you are from being able to do GPT-5 on power retention? You know, kind of like when the frontier models do that. ### Scaling Roadmap **Alessio** [26:29] Is it a matter of having access? Is it a matter of... I know obviously some of them are like MoE models. They're not necessarily like dense transformers. Yeah, what, what, what's kind of coming up for you? **Jacob Buckman** [26:39] Yeah. I think ultimately, it's a matter of building community trust around this architecture, and this is something that is really important to us and one of the main reasons for open sourcing as well, is because we've run tons of experiments on this architecture. We've seen what it can do, but we're just a small lab. We need everyone to see what it can do, and that means just putting it in people's hands for them to play with it themselves. If you think about the history of the transformer itself, right? The transformer didn't drop and then immediately get scaled up to, you know, like the seven hundred billion parameter scale. Instead, it was something that the whole community used, played with. There was this gradual realization of how strong it was on lots of different tasks. There's things like flash attention from Tri Dao, which produced these massive, massive improvements in performance, and it was just basically like a big community effort for people to realize this is the architecture we want to be training at scale. And honestly, I wouldn't expect the big foundation model companies to want to overnight drop everything they're doing and pick up power attention. That wouldn't, that wouldn't make sense. But I do think the momentum is, is, has already started to build. A lot of people are interested, and over the next couple months, maybe year, I think we'll start to see some really big successful models trained in this way. And, um, you know, there is admittedly a lot of, I guess I would call it institutional momentum at places that have invested massively in training a particular architecture like the transformer at massive scale. Investment in terms of engineering resources, scientific effort like plotting scaling laws, even just pre-trained weights that they already are confident in. And I do think the burden of proof is high in order to shift that equilibrium. But I think it's going to happen. And so yeah, the first step is for everyone to just like do these metamorphoses, play with the resulting models and let everyone see how powerful it is. And after that, I think, you know, probably within six months to a year, we're going to start to see the really big foundation models being trained in this way. **Alessio** [28:38] Awesome. Um, anything else we missed? Anything you want to shout out? **Jacob Buckman** [28:43] So yeah, the, uh, training speed difference you can see, I don't know, maybe it's a little too small here, but, um, it's about like four point eight seconds per iteration for power retention at thirty-two K. This is the three billion parameter StarCoder model, and it's about twenty-three seconds per iteration. So this is gonna be about a x speed up at thirty-two K. And you can see it's like, you know, way, way more advanced. This one's at step twenty, this one's at step ninety. And yeah, so there you go. That's like the, the training speed difference. And as I mentioned, the inference speed difference, uh, I don't have a demo of, of it here, but it's, it's even more massive. It's like another like x faster at inference, relative speed. **Alessio** [29:24] Nice. So if you're a inference provider that wants to metamorphosis their model, please reach out. Um, are you bringing on anybody else on the team or like any request for collaboration that you have? **Jacob Buckman** [29:36] Yeah. So the team is, uh, right now small but mighty. We're definitely looking to add anyone who's interested in any of the stuff we've talked about today. Architecture research, pushing the limits of long context, making foundational architectural breakthroughs instead of just, you know, tweaking what already exists. If you are excited by those ideas, especially of a strong background in deep learning, uh, CUDA programming or anything mathematically adjacent to those ideas, please reach out. We'd love to talk to you. Um, also another big thing that we're looking for is the right datasets. So this is actually maybe a, a interesting tangent, but The value of long context is deeply and intrinsically tied to the structure of the data. It's really easy to construct really long context datasets where the context is useless. ### Data Challenge **Jacob Buckman** [30:25] And actually, in some ways, this is how most pre-training datasets are constructed today. If you take, uh, like any old, like the pile or, well, one of the more modern ones, basically any common crawl-derived scrape, open web text, any of these things, you're going to find that almost all documents are very short. There's some long context documents, but the vast majority of them, I think last time I measured it, it was something like ninety percent are of context length two K or below. So if you're doing really long context training runs on that data, typically how people do it, they call it packing, right? They just sort of like stitch together different short documents until you get that long sequence. And sometimes you get a long single document subsequence, so you do learn some long context stuff. But by and large, most of your long context compute is in some sense wasted because there is no long context structure in the data. And this just seems to be a property of internet text. In general, most internet text has mostly short-term structure. There's just not that much value in capturing long-term structure. And so compute optimal models on internet text actually don't have that long contexts. And so of course, you're perfectly fine using transformers for that sort of situation. But internet text is not the only dataset that there is, although it is probably the most popular and most accessible. And in addition to internet text and code and all the other things that people like to use, we'd like to look for other datasets. So if you're somebody who has a interesting dataset and a unique long context dataset where you feel like the context actually matters, it's not just a matter of, you know, thinking more about a short context window to make the prediction, but you actually need to incorporate information from very far in the past, I would love to talk to you as well, looking for the most exciting datasets out there to really showcase the power of this long context unlock. In particular, anything involving, um, human trajectories over really long periods of time. For example, a, a human completing some task, be it a, a administrative task or a coding task, a computer use task. Those sorts of trajectories we think are really, really exciting to work with. And if you are somebody who is in possession of a lot of them and wants to make the most of them, we'd love to collaborate. **Alessio** [32:37] Awesome, man. Thanks so much for taking the time. I will put the links to the model, um, Vidrial, Power Retention, and, and all of that. Yeah. Thank you. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.