⚡️Mercury: Ultra-Fast Diffusion LLMs — Estefano Ermon, CEO Inception Labs

Hey, everyone. Welcome back to another Latent Space Lightning Pod. This is Alessio, partner and CTO at Decibel, and I'm joined by Squeaks, founder of Small AI.

Hey, today we are doing another lightning pod, uh, but this time I think our first diffusion language model pod with Stefano Ermon from Inception. Welcome.

Thank you. Yeah, excited to be here.

So Inception, you've been making a little bit of noise with, like, diffusion models, uh, you know, over the last, uh, few, few months. Um, but I think the, the big one is now that, you know, you, you are actually competitive on coding tasks, um, and obviously the, the tokens per second is, is really impressive.

So basically, like, could you tell us the, the overall story of Inception and maybe, like, why you're betting on diffusion language models?

Sure, yeah. So the story started, uh, back in twenty nineteen actually. At that time, uh, you know, I'd been doing research in generative models, uh, since, uh, basically two thousand and fourteen, two thousand and fifteen when I started my faculty position at Stanford.

Been working on a variety of different kinds of generative models. But yeah, in twenty nineteen, uh, I was working on image generation, and back then the best generative models were GANs, generative adversarial networks. And, uh, we were pretty unhappy with GANs and how hard they were to train.

And so we started pushing this idea of, uh, building a different kind of generative model with my student, uh, Yang Song. And, uh, we kind of, like, showed that there was a better way of building image generative models and that you could train this diffusion kind of model that, uh, instead of generating images in one shot, uh, it kind of like generates them through this iterative refinement procedure.

And so we kind of like pushed out the, the... some of the first papers on, on, uh, diffusion models for image generation using deep neural networks. We used to call them score-based generative models. And that kind of like pick-- uh, was picked up very quickly by industry, by folks at Stability, by Midjourney, and then kind of like became the state-of-the-art for image generation, video generation.

Basically, any kind of continuous modality, you know, diffusion models and variants of diffusion models are now the, the, kind of like the best approach that, that exists. And since basically two thousand and twenty, two thousand twenty-one, I've been trying to get diffusion models to work on, uh, di-discrete data, so text generation, code generation, DNA generation.

Um, and it's been really hard. But, um, after many years of trying, we were able to show that, uh, diffusion models-- We figured out a way to train basically some kind of discrete diffusion model that was competitive with autoregressive models at small scales.

And so we published a paper at ICML, uh, last year that won the best paper award, uh, showing that for the first time, basically, diffuse-- discrete diffusion models were competitive on language generation with autoregressive models up to the GPT two kind of scale.

And I got very excited about those results, and I really wanted to see what happens if you train bigger models on more data and you put all the kind of engineering efforts needed to really get to state-of-the-art, uh, results on tasks that matter.

And so that's how Ince-Inception came along. And, uh, so I, I recruited some of my best, uh, former PhD students, a lot of Stanford folks, and then we started the company last year, last summer. And yeah, we've been working on, uh, basically improving those, uh, research ideas that we had.

And, uh, we've been successful in training the first commercial scale diffusion language models. Uh, we call this model Mercury. We released the first one, Mercury Coder, back in February, and, uh, we just announced, uh, a couple of weeks ago, the first, uh, diffusion language models for more like chat, uh, kind of like text use cases, so summarization, translation, reply prediction.

And yeah, these models are really exciting. They are much faster than autoregressive models. They have some very interesting new capabilities, and we are excited about the future of this technology. I can see a world where in the future all language models will be diffusion based.

I think it's a superior kind of like approach. And so I'm, I'm very excited to be at, uh, at Inception working on this.

Can you maybe do a quick intro on diffusion models for discrete text? Because I think people have an understanding of image generation, which is you add the noise, you remove noise, but it's kind of hard to grasp how you actually train a diffusion text model.

Yeah. Uh, it is actually a very similar kind of idea. Um, typical lang-lang-language models or LLMs are autoregressive, meaning that they generate objects left to right, typically one token at a time, one word at a time. Um, and you can do the same thing for image generation, like you can generate images one pixel at a time.

Uh, it's just not a very good idea to do that. Diffusion models, on the other hand, they work by generating objects, kind of like in a coarse to fine way. You start with a rough guess what the answer should be, and then you refine it, again, using a neural network, typically.

And diffusion language models work the same way. You kind of like start with a guess of what the answer should be, and then you refine the answer using a neural network by modifying essentially multiple tokens effectively in parallel.

That's kind of like the reason diffusion, diffusion language models are much faster compared to autoregressive models, is that each neural network evaluation doesn't just give you one token, like in the typical autoregressive world, but it's able to output essentially or modify multiple tokens.

And so that's where you get the speed ups, the fact that you're able to, to modify many things basically in parallel.

And so the-- when you're creating a training data set for a diffusion model, what, what does it look like? Is it just a normal pre-training corpus and then instead of mask-masking a token, you're like, I don't know, what are you

Yeah. So there is a lot of secret sauce involved, but roughly, uh, yeah, the, the approach is similar. So you, you can start with the usual training data that you would use for training an autoregressive model. You have to specify a neural network, which could be the same kind of architecture that you would use for an autoregressive LLM, so it could be a transformer if you want.

The difference is that instead of training it to predict the next token, you train it to denoise a piece of text or a piece of code. And there is different ways of adding noise or removing noise. In the case of images, you would typically add Gaussian noise and you would train the model to, to remove the noise, although other ways are possible.

In the case of text, uh, again, there is different mechanisms of adding noise. You could imagine masking out some com- some parts of the, the sentence and then training the model to infill, to guess basically what the missing tokens were.

Or you could imagine a different kind of noise models where maybe you flip tokens, which is more similar to the Gaussian kind of case, where you, you kind of like randomly flip some of the tokens and then you train your neural network to basically fix these mistakes and, and correct the, the, this noisy data point and map it back to, to the clean data that you started from.

Yeah. Um, I noticed that you, uh... So that you did publish a paper recently, I think yesterday, that, or something, where you, you confirmed that you're actually using transformers for at least the Mercury models, yeah? Um, and basically, does it have to be trained end to end, or can you just take any existing open model backbone and, uh, uh, and, you know, do this process?

Like, do, do you have to-

So-

... initialize from zero basically? Or can you just, can you lean, can you sort of leverage off of existing, uh, other, other language models?

There have been, uh, attempts in the, the literature, um, where people have tried basically using, um, existing, uh, open source models and then kind of like fine-tune them. The challenge is that, yeah, the training objective is quite different because you are-

Yeah

... training, uh, based on denoising as opposed to next token prediction. Diffusion models are not causal, and that is also kind of problematic. Uh, I mean, it's a big advantage of diffusion models that you don't have to be causal.

You can actually look at the whole context to the left and to the right as you decide kind of like the edits that you wanna do, which is giving, you know... It, it's a, it's a very powerful thing if you think about how you generate objects, but that makes it quite different, quite difficult to adapt kind of like models that have been trained with, with causal masking to, to this, to this, to this, to this new task.

And so there have been attempts in the literature, annealing the, the, the masks, uh, the attention mask. Like, there have been various tricks and attempts at doing that, but it's not straightforward.

Yeah. And then on the sort of the equivalent of contrastive loss or guidance or anything like that, does that require any innovations or is it kind of, does it transfer over nicely from the image side?

It transfer, transfer over pretty nicely. You can do, yeah, you can do basically classifier guidance.

You have to train... Yeah. Basically, I'm, I'm just trying to see where you can shortcut, you know, leverage off the existing open model ecosystem, but it sounds like you can't, you have to do everything from scratch, like there's nothing to, to reuse.

Well, you can use architectures. Uh, I think, uh, that at least the, the shapes, uh, that, that can be, that can be leveraged, uh, so you don't have to reinvent necessarily completely different neural network architectures. Can-- A lot of the data can be used.

Like, uh, I think perhaps there are optimizations and you can think about what kind of data sets work best for diffusion models, but it's pretty much compatible in terms of like the types of data that you would need to do pre-training, mid-training, post-training.

It's all, it's all quite compatible. But you have to change, you have to innovate on the, on the training losses, on the objectives, which is actually exciting because you can do things that are not possible without regressive models, and suddenly they become possible with the diffusion models.

Yeah. Yeah. Okay. So yeah, you know, broa- broadly stepping back a little bit, why do you think diffusion models, diffusion language models are working now? Um, you know, I think it's been talked about for, for a while, for the last maybe three, four years.

But all the examples I've seen were always like very kind of, uh, yeah, not, not like... You know, is there a why now? Obviously, you have some secret sauce, but like I'm just trying to figure out like is, is there anything fundamental?

I mean, depends how you define fundamental. I think there was some math- ... uh, that, that developed, uh, in terms of like thinking about... I like to think of diffusion models in terms of like learning score functions and, and kind of like gradients basically of probability density functions, which are not really well defined in the discrete world, but there are kind of like mathematical and analytical objects that you can kind of train using denoising like objective, like score matching like objective.

So there was some pretty fundamental math that had to be developed. Then there's also a lot of engineering and, and, and tricks that to, to build on top. And then I would also add the fact that, you know, once somebody shows that it's possible, you know, suddenly other people are much more incentivized to, to get their own thing to work and, and I think everybody progresses more rapidly once there is kind of like a proof of concept that, that something is possible.

And so, yeah, we've seen, you, you know, the, the, the Google guys also have a, have a model. So it, it's exciting to see that the, the field is really starting to, to pick up speed.

Yeah. Amazing. Yeah. Any thoughts on the Google side? Uh, you know, obviously you were working on this for a while. Was it a surprise?

I mean, I know they've been trying, uh, for, for, for a while too. It, it was great to see that, uh... My understanding was that, you know, I don't know exactly how they do it. I thought they were, um, um, you know...

I don't know exactly what's their recipe, what's their architecture, what's their model, but it was, uh... It seems to perform pretty well. It was, it was exciting to see that at least some other player in the space was able to come up with a, independently come up with a solution that actually works, um, because I'm pretty sure it's quite different from what we do at Inception.

So yeah, it's, it's, it's exciting I think.

Yeah. Like to, to my knowledge, Gemini Diffusion hasn't been widely released yet. But like I'm curious how to test it. Like if you got Gemini Diffusion today, would you run anything? Is there, is there any use case that you'd be very interested in?

Any prompts or... I, I don't know how to evaluate as of now.

So there is a playground. Um, I think you, you-

Oh, for Gemini Diffusion? Oh, I didn't know that.

Yeah, yeah. There's a playground. Yeah, yeah. So you can play with it.

Okay.

It, it's, uh-

Okay

... it's pretty fun. It does a lot of, uh, infilling, uh, tasks, uh, uh, pretty well, which is also kind of like what we've, we are seeing. Diffusion models, not surprisingly, they work pretty well at infilling, where you really need to be able to use context to the left and to the right.

Um- Or auto-complete is one of the main use cases also for, for Mercury Coder. And it seems like their model is also doing pretty well at kind of editing, inpainting kind of tasks. To me, they've released some benchmark numbers.

Uh, they seem to be pretty close to, to the numbers that we were getting with the Mercury Coder back in, uh, some... you know, back in early this year. So, yeah, it's, it's, it's, uh, it's exciting that we see that, uh, they've also been able to get this technology to work.

And to wrap on maybe the pre-training phase, is there also post-training in the same way that people do with normal LLMs? Well, not normal, but you know what I mean. With like instruction following and things like that. Does that work the same as those models, or is it different in diffusion?

So our, uh, pipelines, yeah, we have something similar in the sense that we also use, uh, we, we can do basically SFT and we can also do, uh, we, we do reinforcement learning, RLHF. We do reinforcement learning from human feedback.

Again, some of the algorithms have to change. For example, we have a DPO algorithm, uh, specialized for diffusion language models. I was one of the original authors of the DPO paper that was originally, uh, designed for LLMs, autoregressive LLMs.

I worked on a paper extending DPO for diffusion models, more continuous diffusion models like image generation, video generation, and that's also been quite impactful. Like if you look at some of the papers that have been released on some of the leading open source image generative models based on diffusion, you can see that the DPO piece using human feedback to really align the models to human preferences is really important if you want to improve quality.

And we've extended a variant of DPO to diffusion language models. So we're actually also able to use preference data to, to fine-tune the models. And, uh, but yeah, basically we have pipelines that are compatible with the existing SFT and RLHF datasets, and we, we do that with customers.

Like we often work with customers that have, uh, their own internal datasets. They want us to fine-tune the models on their proprietary data, and we can easily do that. It's all, uh, pretty easy once they share the data with us.

Ticker muted.

Ticker muted. Maybe give people also an idea of on the compute side when it comes to pre-training, how different the scale is, and then as you move to inference also what that looks like, both in terms of throughput, compute requirements and also like context size.

I don't know if there's much to say there, but if there's anything.

Yeah. So we're, we're... unfortunately, I'm not able to share much on the training side in terms of like the amount of compute or the number of tokens that we're using. Uh, on the inference side, what we're seeing is that, uh, diffusion models are much more efficient.

We're actually able to Pareto dominate autoregressive models. If you think about the, the typical trade-off between, uh, um, throughput versus latency, which you kind of like cannot... can, can, can get by changing the inference parameters like the batch size and things like that.

What we're seeing is that we're actually able to Pareto dominate autoregressive models. For the same throughput, we can get better latency, or for the same latency, we can get better throughput. And so throughput basically means cost effectively. So for the same cost, we can make the models go faster or for, um, yeah, the other way around.

Is that the killer use case of diffusion or are the mo-model also just better?

Right now, that's, uh, one of the real differentiating factors, uh, where we're seeing a lot of traction, like a lot of our users care about latency. So we're seeing a lot of use cases where-- that are latency sensitive, and so people really care about being able to give answers to their users quickly.

Lots of applications where there is a real time, kind of like component voice agents, where you're interacting with a person in real time or IDEs or like vibe coding kind of things where you really want to be able to see results quickly, like maybe you're vibe coding a website or a UI.

So lots of these use cases where speed matters, we're seeing a lot of traction and, uh, you know, it can mean two things. It can mean either just giving even faster results to the users to improve the, the experience or sometimes the value proposition can be people are currently using small autoregressive models to meet the latency constraints.

What you can do is you can put in a bigger diffusion language model, um, that is going to be faster because it's a diffusion model and it's going to be higher quality because it's, uh, it's, you know, it's a bigger model.

It has just more parameters. It's been trained on more data. And so that's also been, uh, a very valuable kind of like use case also of diffusion language model.

How should people think about the Pareto frontier today as far as, um, capability model? I know you had-- you released some numbers on the coder, uh, model, which is roughly the same as 40 Mini, Mistral Small 3. Anything else on non-coding?

Uh, or maybe I missed it in all the announcements.

Yeah, it's, it's actually, it's been released recently. We had our generalist model evaluated by Artificial Analysis and the intelligence score from AA, Artificial Analysis, around forty. So it's, it's comparable to GPT 4.1 Nano, Claude Haiku, uh, kind of like closed source speed optimized models.

It's roughly comparable in terms of intelligence, but significantly faster, five, ten X faster compared to, to this kind of speed optimized autoregressive models. We're working on next generation models that are going to be even more intelligent, but that's not currently publicly available.

Is there something... So if you think about use cases, is there something that people should not be using diffusion models for or do... This is just kind of in your mind, it's like a superior, just swap everything over to these and then as we scale them.

It's funny because you're seeing image generation going towards autoregressive models, and then you're seeing the text models go to diffusion. How do you kind of see the, the model use case match?

So the API is the same. It's like text in, text out. So in principle, any use cases, uh, that is supported by an autoregressive model is also supported by a diffusion language model. In terms of capabilities, yeah, as I mentioned, like we're not frontier level, and so there's certainly use cases where you probably wouldn't want to use a diffusion language model, or at least not the current, uh, existing diffusion language models.

And so, yeah, I think at the moment it's more a function of kind of like what's the intelligence level, how much do we care about latency or cost? So there is definitely a bunch of use cases where people-- where diffusion language models are already competitive, they're already the best solution, uh, for those use cases.

Uh, there are others that are still yet to come, and I think that will require, uh, scaling up and, and making further improvements to, to the, to the training and the data and various kinds of things. I'm pretty optimistic about a future where, um, diffusion models can become the dominant solution.

I've seen it happen before with GANs a, uh, a few years ago, so, uh, I wouldn't be surprised if that's the case also here. Um, I really like this idea of using context to the left and to the right, uh, the fact that you have error correction that is built in.

Uh, so you think about an autoregressive model, once you output something, you can never take it back. And so if you wanna do... you know, if you wanna fix mistakes, maybe you can do a reasoning chain, maybe you can do it that way.

Um, but it's kind of unnatural, it's kind of clunky. Uh, diffusion model has this nice thing of kind of like fixing mistakes as you go. So, um, I think it's gonna be pretty exciting once we have next generation models maybe with reasoning built in.

I think we're gonna see some, uh, some very interesting capabilities emerge there.

Yeah. Yeah. So in your mind, it's just a scaling question left, not a, you know, missing capabilities from the architecture, like limitations of, of it.

Right. Yeah. No, and, and I think there is gonna be potentially new improvements that can be... I mean, I'm sure that we're still pretty far from a local optimum. Like, if you think about autoregressive LLMs, people have been trying all kinds of things, and so maybe there's not that many low-hanging fruits.

But I'm exci- what is exciting to me as a researcher and as an engineer, if I think about diffusion language models, it's, it's all very new. Uh, we have something that already works pretty well, but I'm sure that there is a lot of design choices that were suboptimal.

Uh, there is still a lot of ideas to be discovered, and so there could be big leaps, big improvements possible just by doing R&D on the way we train the models, the way we do inference. You have a lot more knobs.

It's a different design space, so there is a lot of exciting R&D to be, to be done. And we're seeing it even at the company, like the improvements that we're seeing week over week are pretty substantial. And so it's, uh, it's, it's...

I, I'm personally very excited about this.

Yeah. Awesome. Um, one, one more thing about on the inference side. I think a lot of people are very cognizant of the serving requirements between autoregressive and diffusion. A lot of people rely on the batching characteristics in, in production.

You would not have that, right? Are you aiming to like, uh, you know, are you aiming to be a... to abstract that problem away for people, or are you just gonna release the models open, in open ways, and then people will deal with it however they want?

So we don't have a plan at the moment to, to release models or to open source any model. And in fact, it's a little bit tricky even to... even if we wanted to release, let's say a small, maybe less capable version of the models, we'd have to release the inference code, which is also s- proprietary.

Unlike an autoregressive model where the inference code is kind of like straightforward, there is not that many ways you can do inference in an autoregressive model. You can tweak a little bit the sampling, but it's, it's pretty straightforward.

A diff- in a diffusion language model, there is a lot more knobs and a lot more innovation that can happen on the inference side. And so it's very hard for us to open source a model because we'd have to release the inference code.

And by the way, we have our own inference engine. So just like you would normally serve an LLM using a vLLM or SGLang or TensorRT-LLM, we have built our own inference engine, and so we are supporting production traffic already with our own inference engine.

We support continuous batch and quantization. Like, we support a lot of the features that are supported in these serving engines. Caching, like prefix caching, like a lot of that stuff is already handled by our own serving engine. But yeah, it, it gets a little bit tricky if we wanted to, to open source because a lot of the IP is around on the inference side as well.

You know, we have a bunch of inference nerds listening to us and who might be interested in joining you. Like what, what's like, you know, one underappreciated problem or something interesting that has come out of this production work?

There is a lot of, uh, just like the workloads, uh, look different, and there is different opportunities for optimizing, uh, aspects of the serving, uh, the way we serve models. So yeah, we have a bunch of smart people that are kind of like working, like MLCs, I would call them.

I think, you know, I've been working in this space for a while with, with students at Stanford. I was on the Flash Attention paper, so I've been focusing on these kind of issues for, uh, for, for a long time.

And the things you see once you start using additional language models are quite different. So there is new opportunities for optimizing the models, the, the, the inference algorithms, the way we implement them, and we write kernels to really make them go fast.

So there is a lot of exciting opportunities for, uh, for doing work. And yeah, if you're interested, reach out to us. We're hiring at all levels. So I'll use this opportunity-

I'm sure

... to, to, to, to do this plug for applying to, to, to our jobs.

Yeah, absolutely. I mean, this is, you know, if we can get people a job, if we can get you an imp- uh, an engineer, I think that's a win for us doing this, right? Okay, so I wanna move on a little bit in terms of like we, we...

let's say we have production and high quality LLMs. You made an interesting controversial statement earlier that you said most frontier models or most language models will use diffusion in the future. I don't think that's consensus. Uh, could you just elaborate why?

I think that there is a chance. I mean, I, I, I don't know if that's gonna happen, but I feel that that is... there is a world where that could happen. Um, and, and I think it could be driven...

If it happens, it's gonna be driven by efficiency. Like, we're all constrained by essentially power. Uh, and, uh, and if you have... I mean, at the end of the day, it's all an inference game, right? Okay, training is expensive, but then the, the thing that matters is being able to serve these models cheaply and without consuming more energy that we have access to, and you only have so many gigawatts of power and so many data centers, and there is a growing demand for tokens that you need to be able to serve to users and all kinds of apps that are being built.

And so there is a real need to find a solution that can give us maybe a 10X improvement on the, on the efficiency side, and there's a really good chance that diffusion models are the thing. So if indeed it's possible to match the quality of frontier models in terms of quality, if you have a win on the, on the inference side, that is gonna be the, the solution that will be, uh, that will be deployed, right?

Um, because-

Yeah

... um, it's, uh, at the end of the day, it matters. Is a, there is a real resource constraint, and so, um, the better solution will, will be deployed and will win and will be scaled up. Yeah.

Uh, I would just note that, uh, it's interesting that you guys on your paper, you noted that your context length goes up to 128k tokens, and you use normal training recipes. That is the one thing, if I think about autoregressive versus diffusion, that you, you don't have a cache that you can prefill, but I'm sure you figured it out somehow, right?

Like, that, you know, that... But I think that's, that's kind of cool that, that, like, that just translates over.

Yeah. There are interesting caching strategies that can be implemented, including, uh, some that have been in, in papers. There is some growing literature on, uh, KV cache-like mechanisms for, uh, diffusion language models, and so you can find something on- in the literature as well from NVIDIA, from academic groups.

So there is definitely a lot of optimizations that can be done.

Yeah. Yeah, yeah. Uh, so I would say, like, quality, it's anyone's game. It's really gonna be amount of compute, data, all that. Not necessarily driven by, by just the algorithm itself, but, like, I think you unlock a lot of different use cases, which is very cool.

And it, it's interesting that you released the Chat Tune model because actually maybe the, the, the f- the focus sh- should be everything but chat. Like, chat, fine, autoregressive, right? Back and forth, like, sequential, okay. But, like, everything else, like, nobody's releasing any base models, and actually you can, you can offer, uh, something very different there.

That, that would be my, my pitch to you in terms of, like, you don't have to fight with, uh, autoregressive language models head-to-head on quality. Actually, you could just do everything else because they just don't h- they cannot do it.

Yeah, yeah, for sure. Yeah. I think, uh, uh, that's, uh, we're definitely, you know, exploring the market, working with customers, figuring out where we have an edge, uh, w- are there use cases where we're particularly good compared to autoregressive models, capabilities that are just not, like, for example, around controllability, like, or, or think about hallucinations, error control.

Like, there is all kinds of interesting things that are in theory possible with, with diffusion language models, and so we're also excited to, to explore that. It's, it's a lot easier to provide an OpenAI-compatible endpoint that anybody can build on top of.

Yeah. Yeah, yeah. That's the, that's the industry standard. I mean, I think there's a demand from, for new form factors and new capabilities, and that would be your relative strength, you know?

For sure.

So I think, uh, you mentioned customers. Yeah, I think that the, the last thing I wanted to touch on was just what kind of customers are you looking for and that you're ideal for? Like, if someone says, "I have this problem," they should come to you.

What, what is that problem?

Yeah. Whenever they have applications where they are not able to get the level of quality they need, uh, from existing models, typically because they, they need to use one of these closed source speed optimized models or they're fine-tuning a small autoregressive model.

So any kind of applications that is latency constrained where customers are not able to get the quality that they need, uh, diffusion language models can be a great alternative. We can, uh, increase the quality of the answers by plugging in a bigger model, or we can make the existing kind of form factors go much faster and increase the user exper- and improve the user experience.

So any kind of, like, latency sensitive applications of LLMs, uh, diffusion models right now have a, have a real edge.

Yeah. Amazing. Stefano, thanks for coming on.

Yeah, thanks for having me. Yeah. It was really good to meet you.