🔬Max Welling: Materials Underlie Everything

I want to think of it as what I would call a, a sort of a Physics Processing Unit, like a PPU, right? Which is you have digital processing units, and then you have physics processing units. So it's basically nature doing computations for you.

It's the fastest computer known, as-

Yeah

... possible even. Uh, it's a bit hard to program because you have to do all these experiments. Also quite, quite bulky. It's like a very large sort of thing-

Yeah

... you have to do. But in a way, it is a computation, and that's the way I wanna see it. You can do computations in a data center, and then you can ask nature to do some computations, right?

Your interface with nature is a bit more complicated, but then these things will have to seamlessly work together to get to a, you know, a new material that you're interested in.

Yeah. It's a pleasure to have Max Welling as a guest today. Max has done so much over his career that I've been so excited about. If you're in the deep learning community, you probably know Max for, um, his work on variational autoencoders, which has literally stood the test of time or officially stood the test of time.

If you are a scientist, you probably know him for his, um, like, pioneering work on graph neural networks, on equivariance. And if you're in material science, you probably know him about his new, uh, startup, Cu- CuspAI.

Mm.

Max has a long history doing lots of cool problems. Um, you started in quantum gravity, which is, I think, very different than all of these other things you've worked on. The first question for AI engineers and for scientists, what is the thread in how you think about problems?

What is the thread in the type of things which excite you, and how do you, um, decide what is the next big thing you wanna work on?

So it has actually evolved a lot. Um, in my young days, let's put it, I would just follow what I would find, like, super interesting. I have kind of this sensor I think many people have, but maybe not really, uh, sort of used very much, which is, like, you get this feeling about getting about very excited about some problem, right?

Like, it could be, you know, what's inside of a black hole or what's, you know, at the boundary of the universe or, you know, what a- what is quantum mechanics actually all about. And so I followed that basically throughout my career, but I have to say that as you get older, this changes a little bit-

Mm-hmm

... in the sense that there's a new dimension coming to it and this is impact.

Mm-hmm.

Working in two-dimensional quantum gravity, you're pretty much guaranteed there's gonna be no impact in what you do relative-- you know, maybe a few papers, but not in the, in this world, uh, at this energy scale. As I get closer to retirement, which is fortunately still, uh, you know, ten years away or so, I do wanna kind of make a positive impact in the world and, um, I got pretty worried about climate change.

Um, and I think we... And, um, I think we sh- you know, and, uh, politics seems to have a hard time solving it, es- especially these days. And so I thought, better work on it from the technology side, and that's why we started CuspAI.

But there's also a lot of really interesting science problems in, you know, material science. Um, and so I-- it's kind of combining both the impact you can make with it as well as the interesting science. So it's sort of these two dimensions, like working on things which you feel is like, oh, there's something very deep going on here, and on the other hand, trying to build tools that can actually make a real impact in the world.

So the, the thread that when I look back, look at the different things you've worked out, some of them seem pretty connected, like the physics to, to equivariance and-

Yeah

... and, uh, graph neural networks maybe. And that, that seems to be somewhat related to Cusp. Do you have a, a thread through there?

Yeah. I think physics is the thread. So having done, you know, spent a lot of time in theoretical physics, I think there is f- first very fundamental and exciting questions, like things that haven't actually been figured out in quantum gravity, so that is really the frontier.

There's also a lot of mathematical tools that you can use, right? In, for instance, in particle physics, but also in general relativity, sort of symmetry space play an enormously important role, and this goes all the way to, to gauge symmetries as well.

And so applying these kinds of symmetries to, uh, machine learning was actually, you know... I thought of it as a very deep and interesting mathematical problem, right? I did this with Taco Cohan, and, uh, you know, Taco Cohan was the main driver behind this.

Went all the way from s- just simple, like, rotational symmetries all the way to gauge symmetries on spheres and stuff like that. So, and, uh, Maurice Weiler, who's also here, um, when he was a PhD student with me, you know, he wrote a, an entire book, which I can really recommend, about the role of symmetries in, uh, in AI and machine learning.

So I find it's a very deep and interesting problem. So more recently, so I've taken a sort of different path, which is the relationship between diffusion models and, and a field called stochastic thermodynamics. This is basically the thermodynamics, which is a theory of equilibrium, so, but then formulated for out of equilibrium systems.

And it turns out that the mathematics that we use for diffusion models, but even al- for reinforcement learning, for Schrödinger bridges, for MCMC sampling, all has the same mathematics as this theor- this physical theory of non-equilibrium systems. And that got me very excited.

And actually, uh, when I taught a course in, um, Muizenberg, uh, it is South Africa, close to Cape Town at the African Institute for Mathematical Sciences, AIMS, and I turned that into a book. So two years later, the book is finished.

I've sent it to the publisher, and this is about the deep relationship between free energy, s- diffusion models, basically generative AI and s- s- stochastic thermodynamics. So it's always a f- some kind of... I don't know. I find physics very deep.

I also think a lot about quantum mechanics, and-

Yeah

... it's, it's, it's a completely weird theory that actually nobody really understands.

Mm-hmm.

And there's a very interesting story, which i- is m- maybe good to tell, to connect sort of my, my PhD back to where I am now. So, um, I did my PhD with a Nobel laureate, Gerard 't Hooft.

He's just the most brilliant man I've ever met. He was never wrong about anything as long as I've seen him. And now he says quantum mechanics is wrong, and he has a new theory of quantum mechanics. Nobody understands what he's saying, even though what he's writing down is not mathematically very complex.

But he's trying to address this understandability, let's say, of quantum mechanics head-on. I find it very courageous and am completely fascinated by it. So I'm also trying to think about, okay, can I actually understand quantum mechanics in a more mundane way, sort of, you know, without all the weird, uh, multiverses and collapses and stuff like that.

So the physics has always been the threat, and I'm trying to apply the physics to the machine learning, uh, to, to build better algorithms.

You are still very involved in understanding, in understanding physics and the world-

Yeah

... even beyond just, like, applications to machine learning or, uh, introducing new formalism- formalisms. That's really cool. Yeah.

Yes, I would say I'm, I'm not, I'm not contributing much to physics, but I'm contributing to the interface between physics and science, and that's called AI for science or science for AI.

Yeah.

It's kind of a sup- It's actually a new discipline that's emerging.

Yeah, yeah.

Um, and it's not just emerging, it's, it's exploding, I would say.

Yeah.

That's the better term. Because I know you go from investments into, uh, like in the hundreds of millions now in the billions. So there's now actually a startup by Jeff Bezos that, you know, is at six point two billion C round, right?

It's, like, insane. Like, it's just the largest, you know, startup ever, I think, right? And that's in this field, AI for science, right? It tells you something that we are creating a new bubble here.

Yeah.

Right.

So, so why do you think it is? What has changed that has, like, motivated people to start working on AI for science type problems?

So there's two, two reasons, actually. The one, one is that people have been applying sort of the tool, the new tools from AI to the sciences-

Yeah

... which is quite natural. And there's, of course, I think there's two big examples. You know, protein folding is, or is a big one, and the other one is, uh, machine learning force fields or sometimes called machine learning interatomic potentials.

Mm-hmm.

Both of them have been actually very successful. Both also have something to do with symmetries, which is also cool.

Yeah.

And sort of people in the AI sciences saw an opportunity to apply the tools that they had developed beyond advertise placement-

Mm-hmm

... right, or m- you know, multimedia applications, um, into something that could actually make a very positive impact in society, like health, drug development, materials for the energy transition-

Yeah

... carbon capture. These are all really cool, you know, impactful applications.

Yeah.

Beside that, the science and the kind of the... is also very interesting and sort of the, um, I would say the, um, the fact that these sort of, these two fields are coming together and that we're now at the point that we can actually model these things effectively and move the needle on some of these, uh, sort of science, uh, sort of, uh, methodologies, is also a very unique moment, I would say.

And people recognize that, okay, n-now some- we're at the cusp of something new. Where it... which is also what our... is we're also what the company is called after. We're at the, we're at the cusp of something new, and of course, that always creates a lot of energy.

It's like, okay, there's something... It's like sort of virgin field, right? It's like nobody's... green field. Nobody's been there, you know. Um, I can rush in and I can sort of start harvesting there, right?

Yeah. Yeah.

And, uh, and I think that's also what's causing a lot of, uh, sort of enthusiasm in the fields.

If you're an AI engineer, basically if the people that listen to this podcast will be... and you maybe don't have a strong science background, how does... But are excited. Most, I would say most AI practitioners, be it engineers or scientists, would consider themselves scientists, and they have some background, a little bit of physics, a little bit of chemistry, college, maybe even graduate school, that have been working or are starting out.

How does, how does somebody who is not a scientist on a day-to-day basis, how do they get involved?

Well, they can read my book once it's out. But, um, this is basically saying that there is more... We should create curricula that are on this interface. So I'm not sure there is possibly already at some universities actual courses you can take, maybe online courses you can take.

These workshops where we are now are actually very good as well, and, uh, we should probably have more tutorials before the workshop starts. Actually, we've-- I've kind of proposed this at some point. It's like maybe first have an hour of, of a tutorial so that people can get new into the field.

But yeah, there's a lot out there. Um, most of it is, of course, inaccessible. But I would say, um, we will create much more books and other content-

Yeah

... that is more accessible, including this podcast, I would say, right? So-

Yeah

... um, I think, y-you know, it, it will come and, you know, these days you can watch videos and things. There's a huge amount of content you can, you can go and, uh, and, and see.

So maybe a, a follow-up to that. How do people learn and get involved, but, but why should they get involved? I mean, do we have a lot of people who are... of our audience will be interested in AI engineering, but they may be looking for bigger impacts in the world.

Yeah.

What opportunities does AI for science provide them to make an impact, to, you know, change the world that working in this, the world of pure bits would not?

So, so my view is that, um, underlying almost everything is a material.

Yeah.

So we're, we're focusing a lot on LLMs now-

Yeah, yeah

... which is kind of the software layer. But I would say if you think very hard, underlying everything is a material. So-

Yeah

... underlying an LLM is a GPU, and underlying a GPU is a wafer on which we would-- we will have to deposit materials. Do we wanna wait a little bit?

Underlying everything is a material. So I was saying, you know, there's the LLM. Underlying the LLM is a GPU on which it runs. And then in order to make that GPU, uh, you have to Put materials down on a wafer and sort of s- shine on it with a sort of, uh, EUV light in order to etch kind of the structures in.

But that's now an actual material problem because more or less we've reached the limits of, you know, scaling things down, and now we are trying to improve further by new materials. So that's the fundamental materials problem. We need to get through the energy transition fast if we don't wanna kind of mess up this world.

Mm-hmm.

And so there is, for instance, batteries. That's a complete materials problem, right? There's fuel cells. Um, there is solar panels, so they can now make solar panels with new perovskite layers on top of the silicon layers that can capture, you know, theoretically up to 50% of the light, where now we're at, I don't know, maybe 22 or something.

Yeah.

Right? So, so these are huge changes all by material innovation. And, um, yeah, I think wherever you go, you know, I can probably dig deep enough and then tell you, well, actually the very foundation of what you're doing is a material problem.

And so I think it's just, you're very nice to work on this very, very foundation. And ex- also because I think this is maybe also something that's happening now, is we, we can start to search through this material space.

This is, this, this has never been the case, right? It's like scientists w- the, the, the normal way of working is you read papers and then you come up with a hypothesis, you do an experiment, and you learn, et cetera.

So that's a very slow process. Now we can treat this as a s- as a search engine, like we search the internet, we now search the space of all possible molecules, not just the ones that people have made or that are in the universe, but all of them.

Yeah.

Right? And, and, and we can make this kind of fully automated. That's the hope, right?

Yeah.

You can just type-- It becomes a tool where you type what you want and something starts spinning and some experiments get going.

Yeah.

Right? And then, you know, out come a list of materials, and then you look at it and say, "Maybe not," and then you refine your query a little bit.

Yeah.

And you kind of do research with this search engine where a huge amount of computation is and, and experimentation is happening, you know, somewhere far away in some lab or some data center or something like this. I find this a very, very promising view of how we can sort of come, you know, build a much better sort of materials layer, um, underneath almost everything, and also more sustainable-

Mm-hmm

... materials. Our plastics are polluting the planet. If you can come up with a plastic that kind of destroys itself, you know, after, I don't know, a few weeks, right? And actually becomes a fertilizer. These are, these are things that are not impossible at all.

These, these things can be done, right? And we should do it.

Can you tell us what-- a little bit just generally about CuspAI, and then we-- I have a ton of questions about it.

Yeah. So CuspAI started about 20 months ago, and it was because, um, I was worried about-- I'm still worried about climate change.

Mm-hmm.

And so I realized that in order to get, you know, to, to stay within two degrees, let's say-

Mm-hmm

... we would not only have to reduce our emissions to zero by 2050, but then, you know, another half century or even a century of removing carbon dioxide from the atmosphere, not by reducing your emissions, but actually removing it at a rate that's about half the rate that we now emit it.

And that is a unsolved problem. But-- And if we don't solve it, two degrees is not gonna happen, right? It's gonna be much more. And I don't think people quite understand how bad that can be. Uh, uh, like four degrees, like very bad.

So, um, so this technology needs to be developed. And so this was, uh, my and my co-founder, Chad Edwards', um, motivation to start this startup. And also because, you know, we saw the technology was ready, which is also very good, so if you're-- you know, the time is right to do it.

And, uh, yeah, so we, we now-- In, in the meanwhile, we've grown to about 40 people. We've kind of collected a hundred and thirty million investment, uh, into, into the company, which is for a European company is quite a lot.

I would say it's interesting that right after that, you know, other, uh, startups got even more. So that's kind of tells you how fast this, this, this is growing. But yeah, we are, we are now at the-- So we, we've built the platform, of course, but, uh, it's, it's for a s- a series of material classes, and it needs to be constantly expanded to new material classes, and it can be more automated because, you know, we're now putting LLMs in, and so the whole thing gets more and more automated.

And now we're moving to sort of high throughput experimentation, so connecting the actual platform, which is computational, to the experiments so that you can get-- also get fast feedback from experiments. And I, I kinda think of experiments as something you do at the end, although that's what we've been doing so far.

I want to think of it as what I would call a, a sort of a Physics Processing Unit, like a PPU.

Yeah.

Right? Which is you have digital processing units, and then you have physics processing units. So it's basically nature doing computations for you. It's the fastest computer known-

Yeah

... if possible even. Uh, it's a bit hard to program because you have to do all these experiments. Also quite, quite bulky. It's like a very large sort of thing you have to do.

Yeah.

But in a way, it is a computation, and that's the way I wanna see it. So I want us-- You can do computations in a data center, and then you can ask nature to do some computations, right? Your interface with nature is a bit more complicated.

But then these things will have to seamlessly work together to get to a, you know, a new material that you're interested in. And that's, that's the vision we have. We don't say super intelligence because I don't quite know what it means.

And I don't wanna oversell it, but I do want to automate this process and give a very powerful tool in the hands of the chemists and the material scientists.

That's actually, uh-- Brings up a question I wanted to ask you. First of all, can you talk about your platform to, like, whatever degree, like, explain kind of how it works and, like, what you-- your thought processes was in developing it?

Yeah. Actually, it's been surprisingly-- It's not rocket science, I would say. It's not rocket science in the sense of the design. And basically, the design that, you know, I wrote down at the very beginning is still more or less the design, although you add, you add things.

Like, I, I wasn't thinking very much about multi-scale models, and I was-- come on now, rated that actually multi-scale is very important. In the beginning, I wasn't tr- thinking very much about self-driving labs, but now I think, you know, we-- that's-- we are now at the stage we should be adding that.

And so there is sort of bits and details that we're adding. But more or less, it's what you see in the slide decks here as well, which is there's a generative component that you have to train to generate candidates, and then there is a digital twin, multi-scale, multi-fidelity-

Mm-hmm

... digital twin, uh, which you walk through the steps of the ladder. You know, the, they do the cheap things first. You weed out everything that's obviously unuseful, and then you go to more and more expensive things later.

Yeah.

Um, and so you narrow things down to a small number. Those go into an experiment, you know, do the experiment, get feedback, et cetera. Now, things that also have been more recently added is, uh, sort of more agentic, uh, sort of parts.

You know, we have agents that search the literature and come up with, you know, actually the chemical literature, and come up with, you know, chemical suggestions or for doing experiments. We have agents which, uh, sort of autonomously orchestrate all of the computations and the experiments that need to be done.

You know, they're in various stages of maturity, and they can be continuously improved, I would say. And so that's basically-- I don't think that part is rocket science, but, you know, the, the, the design of that thing-

Mm-hmm

... is not, like, surprising. What is, it's surprising hard to actually build it, right? So that's, that's the thing that is-

Yeah

... where the mo- the moat is in the data that you can get your hands on and the, and actually building the platform.

Yeah.

And, and I would say there's two people in particular I want to call out, which is Felix Hanke, who is actually, you know, building the scientific part of the platform, and Alessandro de Maria, who is building the, the m- sort of the sca- the, the s- kind of the s- uh, the MLOps part of the platform.

Yeah. And so-- And, and recently, uh, we also added, um, sort of Aaron Walsh to our team, who is a, a very accomplished scientist, uh, from Imperial College. We're very happy about that. He's gonna be our chief science officer.

And, uh, we also have a partnerships team that sort of seeks out all the customers because I think this is one thing I find very important. In prin- it's so complex to d- to actually bring a material to the real world that you must do this, you know, in collaboration with, uh, sort of the domain experts, which are the companies typically.

So we always-- we only start to invest in a direction if we find a good, uh, industrial partner to go on that journey with us.

Makes a lot of sense. Over the evolution of the platform, did you find that y- that human intervention, human, um... I guess you could start out with a pure-- You could, you could imagine two directions. One, you start out making everything purely automatic, um, automated, agentic, so on.

Mm.

And then later on you, like, find that you need to have more human input and feedback at different steps. Or maybe did you start out with having human feedback-

Yeah

... in lots of steps and then like kind of-

Yeah

... uh, figure out ways to remove, you know-

That-- It's a, it's the second one. So you build tools.

Yeah.

So you-- So it's much more modular than you think.

Yeah.

But it's like we need these tools for this application, we need these tools. So you build all these tools, and then you go through a workflow.

Yeah.

Actually ma- in the beginning, just manually. So you put them, okay, now first this tool, then run this tool, then run this one, et cetera. So, so you put them in a, in a, in a workflow.

Mm-hmm.

And then you figure out, oh, actually, you know, this, this porous material that we, that we're trying to make actually collapses if you shake it a bit. Okay, then you add a new tool that says test for, you know, stability, right?

Yeah.

And so there's more and more tools, and then you build the agent, which could be a Bayesian optimizer, or it could be an actual LLM, you know, maybe trained to be a good chemist, that will then start to use all these tools in the right way, in the right order.

Yeah.

Right? But in the beginning, it's like you as a chemist are putting the workflow together.

Yeah, yeah.

And then you think about, okay, how am I going to automate this, right?

Yeah.

One very easy question you can ask yourself is, you know, every time somebody who is not a super expert in DFT-

Yeah

... and he wants to do a calculation, has to go to somebody who knows DFT.

Yeah.

And so could you start to automate that away? Which is like, okay, make it so user-friendly so that you actually do the right DFT for the right problem and for the right length of time, and you can actually assess whether it's a good outcome, et cetera.

And so you, you start to automate smaller, small pieces and more bigger pieces, et cetera. And, and the end, the whole thing is automated.

So, so your philosophy is you want to provide a set of specific tools that make it so that the scientists making decisions are better informed and, uh-

Yeah

... less so trying to create a, an automated process.

I think it's, this is sort of the same what you're saying because, yes, we want to automate.

Yeah.

But we don't see something very soon where the chemist and the domain expert is out of the loop.

Yeah.

But it, but it's a retreat, right? It's like, okay, so first you needed an expert to tell you precisely how to set the parameters of the DFT, DFT calculation.

Yeah.

Okay, maybe we can take that out.

Yeah.

We can maybe automate it, right? And so increasingly more of these things are going to be removed.

Yeah.

Um, in the end, the vision is it will be a search engine where you, where somebody, a chemist, will type things and will get list candidates, but the chemist will still decide what is a good material and what is not a good material out of that list, right?

And so the vision of a completely dark lab-

Uh-huh

... where you can close the door and you-

Yeah

... and you just say just, you know, find something interesting, and then it will k- it will, it will just figure out what's interesting, and it will figure out, you know, and say, "Oh, I found this new material to blah, blah, blah, blah," right?

That's not the vision I have.

Yeah.

At least n-not for, you know, I don't know, a long time. So for me, it's really empowering the domain experts that are sitting in the companies and in the universities to be much faster in de- in, in developing their materials.

Yeah. Mm-hmm.

And I should say it's also good to be a little humble at times. Because it is very complicated-

Uh-huh. Yeah

... you know, to bring the-- to make it and to bring it into the real world. And there are people that are doing this for their entire lives.

Yeah.

Right? And it's like, I wonder if they scratch their head and say: Well, you know, how are, how, how are you gonna completely automate that away, like, in, in, in the next five years? I don't think that's gonna happen at all.

Um, yeah. So, so to me, it's a increasingly powerful tool in the hands of the chemists.

I have a question. You've talked before about getting people interested based on having, you know, sort of a big breakthrough in materials-

Yes

... versus incremental change. I'm curious what you think about the platform you have now and are sort of stepping towards, and how-- Are you chasing the big change or is this, like, incremental or is there-- They're not mutually exclusive obviously, but-

Yeah

... what do you think about that?

We follow a mixed strategy. So we are definitely going after a big material. Again, we do this with a partner. I'm not gonna disclose precisely what it is, but we have our own kind of long-term goal. You could call a lighthouse or, you know, a, um, sort of moonshot or whatever.

But, um, it is going to be a, you know, a really impactful material that we want to develop as a proof point that it can be done, and it, and it will make it into the re- into the real world, and that AI was essential in actually making it happen.

Yeah.

At the same time, we also are quite happy to work with companies that have more modest goals.

Mm-hmm.

Like, I would say one is a very deep partnership where you go on a journey with a company.

Yeah.

And that's a long-term commitment together. And the other one is, like, somebody says, "I need, I need a force field. Can you help me train this force field and then maybe analyze this particular problem for me?"

Yeah.

"Um, and I'll pay you a bunch of money for, for that, and then maybe after that we'll see." And that's fine, too, right? But we prefer, you know, the deep partnerships-

Yeah

... where we can really change something for the good.

Yeah. And do you feel like from a platform standpoint, you're ready for that? Or what are the things that... A-and, and again, not-

Mm

... asking you to disclose proprietary secret sauce, but-

Yeah

... what are the things, generally speaking, that need to happen from where we are to where-- to get those big breakthroughs, I guess?

What I find interesting about this field is that every time you build something, it's actually immediately useful.

Mm-hmm.

Right? And so unlike, uh, quantum computing-

Mm-hmm

... which-- or nuclear fusion, so you work for, I don't know, twenty, thirty, forty years and nothing, nothing, nothing, nothing, and then it has to happen.

Yeah.

Right? And when it happens, it's huge.

Mm-hmm.

So it's, uh, it's quite different here because every time you introduce... So you go to a customer and you say, "So what do you need?" Right? Okay. So we work, uh, let's say, on a, on a problem like, uh, water filtration.

We wanna remove PFAS from water.

Yeah.

Right? So we do this with a company, Kamira. So they are a deep partner-

Yeah

... for us, right? So we're on a journey together. I think that the breakthrough will happen with a lot of human in the loop because-

Yeah

... there is the chemists who have a whole lot more knowledge of their field, and it's us who, who will, you know, help them with AI, training AI and new methods. And in that kind of this interface, these interactions, something beautiful will happen.

And that, that will have to happen first before this field will really take off, I think. And so in the sense that it's not a bubble, let's put it that way.

Yeah.

So as people see that it's actual real what's happening. So in the beginning, it will be very, you know, with a lot of humans in the loop-

Yeah

... I would say, and I would, I would hope we will have this new sort of breakthrough material before, you know, everything is completely automated, because that will take a while. And also it is very vertical specific. So it's like completely automating something for m- problem A, you know, and you can probably achieve it.

Mm-hmm.

But then you'll sort of have to start over again for problem B because, you know, your experimental setup looks very different. You know, the machines that you characterize your materials look very different. Even the models in your platform will have to be retrained and fine-tuned to the new class.

So every time, you know, you, you, you have a lot of learnings to transfer, but also, you know, the problems are actually different.

Yeah. Yeah.

And so, yes, I, I would want that breakthrough material before it's completely automated.

Yeah.

Which I think is kind of a long-term vision. And I would say every time you move to something new, you'll have to start retraining, and humans will have to come in again and sort of, "Okay, so what does this problem look like?"

And now-

Yeah

... sort of, you know, point the, the, the machine again, you know, in the new direction and, and then, and then use it again.

For the non-scientists amongst, me included, a bit of a scientist, there's a lot of terminology. You mentioned DFT, uh, you me- equivariant, we've talked about.

Mm.

Can you sort of explain in, you know, engineering terms or at the level of sophistication in engineering, well, how-- what is equivariant?

So equivariant is the infusion of symmetry in neural networks. So if I build a neural network, let's say, that needs to recognize this bottle, right, and then I rotate the bottle, it will then actually have to completely start again because it has no idea that the rotated bottle-- Well, actually, the input that represents a rotated bottle is actually a rotated bottle.

It just doesn't understand that. Where if you build equivariant in, basically, once you've trained it in one orientation, it will understand it in any other orientation. So that means you need a lot less data to train these models.

And these are constraints on the weights of the model. So the-- So basically, you have to constrain the weight such that it understand it. And you can build it in, you can hardcode it in. And yeah, this-- the symmetry groups can be, you know, translations, rotations, but also permutations.

Like in graph neural network, there, there are permutations. And in physics, of course, there's many more of these groups.

To play devil's advocate, why not just use data augmentation by-

Yeah

... your bottle is in all the different orientations?

As an option. It's just not exact. It's like, uh, why would you go through the work of doing all that where you would really need an infinite number of augmentations to get it completely right-

Yeah

... um, where you can also hardcode it in. Now, I, I have to say, sometimes actually data augmentation works even better than hardcoding the equivariant in. And this is something to do with the fact that if you constrain the optimization, the weights before the optimization starts, the optimization surface or objective becomes more complicated, and so it's harder to find good minima.

So there is also a complicated interplay, I think, between the optimization process and, and these constraints you put in your network. And so, yeah, you'll, you'll hear kind of contradicting claims in this field. Like, some people And for certain applications, it's, it works just better than not doing it.

And sometimes you hear other people, if you have a lot of data and you can do data augmentation, then actually it's easier to optimize them, and it actually works better than putting the equivalence in, so.

Do you think there's kind of a bitter lesson for, um-

Hmm

... mathematically founded, uh, models and str- strategies for doing deep learning?

Yeah. Ultimately, it's a trade-off between data and, uh, and inductive bias.

Yes.

So if your ind- if your inductive bias is not perfectly correct, you have to be careful because you, you, you put a ceiling to what you can do. But if you know-

Mm-hmm

... you know, the symmetry is there, it's hard to imagine there sh- there isn't a way to actually leverage it. But yeah, so there is a bitter lesson. The, the... And one of the bitter lessons is, you should always make sure your architecture scale, unless you have a tiny data set, uh, in which case it doesn't matter.

Um, but if you, uh, you know... The, the same bitter lessons or lessons that you can draw on LLM space are eventually going to be true in this space as well, I think.

Yeah.

Yeah.

Can you talk a little bit about your upcoming book and tell the listeners, like, what's exciting about it?

Yeah.

Why they should read it.

So this book is about... So it's called, uh, Generative AI and, and Stochastic Thermodynamics. It basically lays bare the fact that the mathematics that goes into both generative AI, which is the technology to generate images and videos, and this field of non-equilibrium statistical mechanics, which are systems of molecules that are just, you know, moving around and, uh, you know, relaxing to their ground state or w- that you can control to have certain, you know, be in a certain state.

The mathematics of these two is actually identical, and so that's fa- fascinating. And, in fact, what's interesting is that Geoff Hinton and Radford Neal already wrote down the variational free energy for, uh, machine learning long time ago. And there's also Karl Friston's work on free energy principle and active inference.

But now we've related it to this very new field in physics, which is called stochastic thermodynamics or non-equilibrium thermodynamics, which has its own very interesting theorems, like fluctuation theorems, which are-- which we don't typically talk about, but we can learn a lot from.

And I think it's just, it can sort of now start to cross-fertilize. When, when we see that these things are actually the same, we can, like we did for symmetries, we can now look at this new theory that's out there developed by these very smart physicists and say, "Okay, what can we take from here that will make our algorithms better?"

At the same time, we can use our models to now help the scientists, you know, do better science, right? And so it becomes a beautiful cross-fertilization between these two fields. You know, and the book is rather technical, I would say, and it takes all sorts of things that have been done in sto- stochastic thermodynamics and all sorts of models that have been done in, in the machine learning literature, and it basically equates them to each other.

And I think hopefully that sense of unification will be revealing to people.

Yeah. Wait, and when is it out?

Well, it depends on the publisher now.

Okay.

But, uh, I w- I hope in April. Um, I'm gonna give a keynote at ICLR, and I-- it would be very nice if I have this book in my hand, but, you know, it's hard to control, uh, these kind of timelines.

Yeah. I'm, I'm looking forward to it.

Great.

Likewise.

Thank you very much.