# The Future of AI Infra: from Kubernetes to Agent Sandboxes — Akshat Bubna, Modal CTO Latent Space · 2026-07-08 Modal CTO Akshat Bubna explains how the cloud platform evolved from a serverless runtime to an AI cloud built for elastic inference, agent sandboxes, and post-training workloads. He argues that Kubernetes was never designed for bursty GPU-heavy AI workloads, so Modal built a decorator-based infrastructure that co-locates compute requirements with code. The company added GPUs a year before ChatGPT and now powers inference for custom models at Suno, Runway, and robotics firms, using GPU snapshotting to slash cold starts. Modal's new Auto Endpoints incorporate open-source DeFlash speculative decoding for frontier-level performance. For reinforcement learning rollouts, Modal provides up to 100,000 sandboxes simultaneously. The platform spans 17 cloud providers, features private IPv6 networking via eBPF, and supports serverless multi-node training with RDMA. Bubna also reveals Modal's shift from developer experience to agent experience, noting that agents benefit from the same minimalist SDK and observability tools that humans use. ## Questions this episode answers ### What is Modal's 100,000 sandbox problem? Akshat Bubna, Modal's CTO, explains that reinforcement learning (RL) rollouts can require spinning up 100,000 sandboxes for just seconds, creating extreme burstiness. Modal's sandbox infrastructure is built for this, offering rapid scaling and snapshotting to handle ephemeral, compute‑heavy agent environments that traditional clouds cannot support efficiently. [15:39](https://addtry.com/ee951c76-20c7-429a-82c7-b1635801d8b4?t=939000) ### Why is Modal focusing on agent experience instead of developer experience? Akshat Bubna says Modal has reoriented its SDK team from developer experience (DX) to agent experience (AX). Since AI agents now write infrastructure code, it is better to let them modify concise decorator‑based definitions than navigate hundreds of untyped Kubernetes YAML files. This gives agents a self‑provisioning runtime and faster iteration, mirroring the advantages Modal gave human developers. [6:09](https://addtry.com/ee951c76-20c7-429a-82c7-b1635801d8b4?t=369000) ### How does Modal handle elastic inference for custom AI models? Akshat Bubna states that Modal’s largest use case is elastic inference for custom models, serving companies like Suno for audio and Runway for video. Modal auto‑scales GPU resources across unpredictable traffic patterns using GPU snapshotting to cut cold starts. Its capacity pool across 17 cloud providers and regional routing enable deploying many models with elastic, burstable scaling. [13:27](https://addtry.com/ee951c76-20c7-429a-82c7-b1635801d8b4?t=807000) ## Key moments - **[0:00] Intro** - [0:00] Modal shifts from developer experience to agent experience, says CTO Akshat Bubna. - **[1:54] Origin Story** - [1:54] Modal CTO Akshat Bubna met CEO Eric through an investor; Eric was already building a runtime to replace Kubernetes for bursty workloads. - [3:34] Modal added GPUs a year before ChatGPT, not foreseeing the AI boom. - [5:38] Akshat Bubna states Modal users own their code despite the proprietary SDK, eliminating lock-in fears. - [6:03] Agent experience on Modal is faster than writing Kubernetes YAML because the self-provisioning runtime co-locates infrastructure with code. - **[6:09] Agent Experience** - [6:59] Observability dashboards are now more important than reading code when agents write it, Akshat Bubna argues. - [8:01] Modal is a cloud platform built from scratch for AI, covering inference, training, batch, and sandboxes, but not always-on web servers. - [10:28] Modal built sandboxes in May 2023, before the concept gained traction, and used it to let agents iterate on themselves. - [13:04] Elastic inference for custom models from Suno, Runway, robotics, and comp bio is Modal's biggest use case. - **[13:27] Elastic Inference** - [14:54] GPU snapshotting dramatically reduces cold start times for Modal's inference auto scaling. - [15:44] RL rollouts can require 100,000 sandboxes with bursty scaling, Akshat Bubna reveals. - [17:06] Modal open-sourced DeFlash, a block-based speculative decoding speculator that achieves frontier-level inference speed. - **[17:53] Speculative Decoding** - [19:58] Modal's upcoming Auto Endpoints will shadow traffic to continuously train custom speculators, making optimized inference hands-free. - **[20:11] Auto Endpoints** - [24:03] Ramp's accounting agent uses Modal's sandboxes, snapshotting, and fast scaling for a reactive background agent. - [25:16] AI workloads are shifting from 8:1 to 1:1 GPU-to-CPU ratio because agents frequently call out to CPU-heavy tasks. - **[25:22] Supercloud Capacity** - [25:40] Modal's capacity pool spans 17 cloud providers, with a reliability layer to use NeoClouds without owning data centers. - **[28:05] Networking** - [29:53] Modal offers serverless multi-node training with RDMA for post-training small runs, bypassing TCP for faster transfer. - **[35:05] Auto Research** - [35:54] Modal's internal 'auto inference' harness uses an agent to spin up GPU sweeps and tweak configs to optimize performance. - [39:00] Modal's growth demanded proactive capacity planning; the compute strategy team models capacity like an airline hedging fuel. - [41:00] Modal is building a batch tier for non-urgent compute, offering cheaper pricing with 24-hour completion, targeting bio and non-LLM jobs. - [47:50] Akshat Bubna: 'Infrastructure has become exciting again because of the immense scale AI workloads demand.' - [48:40] Modal sees significant growth in computational biology and robotics, with active robot deployments and drug discovery. - [49:49] Akshat Bubna's postmortem on Replicate: model API market serves hobbyists and lacks stickiness, while Modal focuses on flexible, code-level products. - **[53:07] Ecosystem Takes** - **[57:09] Agent Future** - [58:00] Modal built Modal Bench to pinpoint where AI agents struggle, using agent hallucinations as product feedback for new CLI features. ## Speakers - **Swyx** (host) - **Vibhu** (host) - **Akshat Bubna** (guest) ## Topics Inference, Compute, Agent Infrastructure ## Mentioned Anthropic (company), Cognition (company), Decagon (company), Gitpod (company), Hugging Face (company), Luma (company), Modal (company), Ngram (company), OpenAI (company), Ramp (company), Replicate (company), Runway (company), Suno (company), Auto Endpoints (product), DeFlash (product), Kubernetes (product), SGLang (product), vLLM (product) ## Transcript ### Intro **Swyx** [0:00] We've actually changed our SDK team to think about agent experience instead of developer experience, and we think that the same benefits that apply for DX also actually apply for AX, which is: why would you have an agent read through hundreds of Kubernetes files and write YAML that's not even typed, when it can basically make a couple of changes in a decorator, and it gets this sort of self-provisioning runtime of being able to see its changes live in action. **Vibhu** [0:29] Before we get into today's episode, I just have a small message for listeners. Thank you. We would not be able to bring you the AI engineering, science, and entertainment content that you so clearly want if you didn't choose to also click in and tune into our content. We've been approached by sponsors on an almost daily basis, but fortunately, enough of you actually subscribe to us to keep all this sustainable without ads, and we want to keep it that way. But I just have one favor to ask all of you: the single most powerful, completely free thing you can do is to click that subscribe button. It's the only thing I'll ever ask of you, and it means absolutely everything to me and my team that works so hard to bring Latent Space to you each and every week. If you do it, I promise you we'll never stop working to make this show even better. Now let's get into it. **Swyx** [1:19] We're here with Akshat of Modal, CTO of Modal, together with Vibhu. Congrats on your Series C. **Akshat Bubna** [1:25] Thank you. **Swyx** [1:26] Your party yesterday was amazing. **Akshat Bubna** [1:29] Yeah. **Swyx** [1:30] All the photos and all the swag. **Akshat Bubna** [1:32] We had a bunch of art installations, which is kind of fun seeing our products on pedestals next to Rodin. **Swyx** [1:41] Very nice, very nice. When you started, it was not the GPU inference company. I mean, maybe it was in your mind? Take us back to the origin story. ### Origin Story **Akshat Bubna** [1:54] I actually first met Eric who's the CEO through an investor, and back then Eric was already thinking about building a new kind of runtime, and he got there thinking through why our workflow orchestration product is so hard to use. It's because you've run them on Kubernetes, Kubernetes is hard to manage, it's not built for burstiness and custom images, and it has a terrible developer experience. **Swyx** [2:20] And I'll interject for listeners who are new, we interviewed Eric two years ago, and there's a bit more of the story there from Spotify and all those things. And I actually came across Eric through Data Council because he did that talk on the sort of serverless container stack that you guys did, which was like, that was my first like, okay, I need to take Modal's very seriously moment. But I was still very unclear, like, do I actually need all this for just my data pipelines? **Akshat Bubna** [2:48] Yeah, I mean, initially what we were thinking about was if we build a better runtime. It's a very useful primitive in itself. There's a lot of things that get solved by serverless functions, like you can do ETL stuff, you can do job queues, you can do all this bursty processing, which it turns out every company needs more. But then we also were thinking about this as like, this is a primitive that we can build a whole collection of products on, which are very work-wise. So perhaps data engineering would have been the first one, but we were thinking about inference. Back then it was more classical inference, like computer vision stuff and running XGBoost and whatnot. But we added GPUs to the product a year before ChatGPT came out. **Swyx** [3:34] Nice. **Akshat Bubna** [3:34] We just didn't think it would be that big of a deal. **Swyx** [3:36] Yeah, just like add A100. Was there any early key problem that really sparked off why you built it? **Akshat Bubna** [3:43] Yeah, primarily it's just none of the tooling that was out there was built for, one, a really great developer experience. And also there's a general trend of a lot of the workloads that we were seeing were very, this is, I wish there's a better word for it, but compute heavy. Like they need, one, like need a lot more resources, so you need to burst up and down a lot, versus like Kubernetes designed for like slow scaling and more for like web server use cases. And also there's just a lot more specialization in like what kinds of environments these workloads run in, like we kind of sometimes need accelerators, sometimes we need different kinds of images. And this is just like a consistent thing that we saw across a lot of companies. That would be the next step. **Swyx** [4:28] Yeah, that'd be nice. **Akshat Bubna** [4:29] I don't know how much this factored into the early story, but I wrote a post when I was at Temporo about infrastructure, software-defined infrastructure, something like that. **Swyx** [4:37] Yeah, the self-provisioning workload. **Akshat Bubna** [4:38] Self-provisioning, yeah, yeah, yeah. I can't even remember my own post. And then you put me on the landing page. **Swyx** [4:43] Yeah, we really liked the term, and so we stole it. **Akshat Bubna** [4:47] Because you had the insight that everything can just be in decorators next co-located with the code,right? Was that a big part of the original story or was it just like a DX layer? **Swyx** [4:56] That was really important because we really didn't want people to spend so much time writing YAML, and it seemed like you could really condense the surface area of what you're doing, put it in code so you can actually operate on it, just like you can operate on other code and build stuff that's more expressive and dynamic. And so yeah, that was always a very important part. **Akshat Bubna** [5:19] Then the pushback is, this is a DSO. It's your closed source. I am locked in to Modal. **Swyx** [5:26] Yeah, we never really got pushback for that because the nice thing about Modal is you can bring whatever code you have. And sure, the DSO is at the sort of configuration layer for what hardware you're using, how you're scaling things up, but you still own the code. And that's been an important part of our story, even as we do inference now. **Akshat Bubna** [5:47] Yeah. **Swyx** [5:47] How much of it do you think still stays the same today? Like if you were to build something today, DevX obviously very important, but I feel like a lot of this has kind of been changed with just hooking up to an agent, half cloud code, half CodeX, implement a tool. There's very agent-native primitives that are kind of different than if I'm doing this myself,right? **Akshat Bubna** [6:09] We've actually changed our SDK team to think about agent experience instead of developer experience. And we think that the same benefits that apply for DX also actually apply for AX, which is why would you have an agent read through hundreds of Kubernetes files and write YAML that's not even typed when it can basically make a couple of changes in a decorator, and it gets this sort of self-provisioning runtime of being able to see its changes live in action. ### Agent Experience **Akshat Bubna** [6:38] Yeah, it just seems from the customers we talk to, they actually find the model is way faster to use for agents versus operating on a different subgrid. **Swyx** [6:49] Yeah, because again, you co-locate the infrastructure requirements to the code that runs it. **Akshat Bubna** [6:52] Yeah. **Swyx** [6:53] Well, the negative thesis now is that nobody's looking at their code anymore, so there's no point. **Akshat Bubna** [6:59] Yeah, I mean, people aren't looking at code. One thing we actually still see is really important is observability. Like how good is your dashboard? And of course, like we have, we've pushed a lot of it to the CLI so that agents can do their own investigation, but you still need humans to go and interpret what's going on and make judgment calls and whatnot. And that's, I feel like, maybe more important now than looking at the code itself. **Swyx** [7:26] Yes, because like, you know, you can try to treat the code as a black box and then use sort of see the observable action that comes out of it and then just prompt a change. **Akshat Bubna** [7:35] Yeah. **Swyx** [7:36] So I think actually it takes a bit of restraint to not specialize, to say, I want to ship a new primitive and then just be general purpose. People ask you, what are you for? You're like, I don't know, we can do this, we can do that. **Akshat Bubna** [7:51] I mean, I'd be curious to see, you know, like, okay, if we were to ask you, like, what is Modal for even at a high level? There's a lot you guys do, sandboxes, GPUs, everything. How do you answer? **Swyx** [8:01] Modal is a cloud platform that's built for, where we've built the primitives from scratch for AI applications. Andright now it basically covers inference, training, batch processing, and sandbox workloads. But we're building a lot more and. **Akshat Bubna** [8:17] I noticed you didn't say web server. So there is still a role for like the always-on, large-scale Kubernetes type of things? **Swyx** [8:23] Yeah, absolutely. We're not trying to compete with the renders of the world because, yeah, we think the differentiator for us is our other workloads that need specialized compute, need to scale up and down a lot. Yeah, they're just shaped differently. **Akshat Bubna** [8:41] I think you're building a lot of it alongside the startups,right? They're innovating quite a bit. Even in your latest blog post, even in the Series C, the customers that you mentioned here, the Cognitions, Decagons, RAMPs, and whatnot, they're innovating with you,right? And that's not something AWS is doing directly with them. **Swyx** [8:59] Yeah, absolutely. I think, I mean, this is again classic. We're a small team, we can move really fast. Our engineers are working with their customers and figuring it out. **Akshat Bubna** [9:09] So my first week at Cognition, I walked in, there was someone wearing a Modal shirt. I was like, what are you doing here? They were like, yeah, I just, I am embedded inside of Cog. **Swyx** [9:19] Yeah, I think that was Payton. We sent him over because the latency of communication was too high otherwise. Yeah, you know, distributed node, you have to place one and co-locate. So actually, so I had a direct personal experience,right? So I worked on Small Developer three years ago. It was inspired by Claude 1. I think you onboarded me at some point, like just before, and I was like, oh, like I need some bursty compute. Like I was just going to try using Modal. And it was a pretty pleasant experience. Apparently I showed up in the board meeting, like the analytics. **Akshat Bubna** [9:54] Yeah, you blew up on Hacker News and we got a big traffic spike. I actually, I think the way you used Small Developer was Modal functions for running stuff, which was, like that was a good use case. **Swyx** [10:08] Yeah, so to me that was proto-Cognition. **Akshat Bubna** [10:10] Right. **Swyx** [10:10] If only I had like stuck to it. Like that was like, if you just like draw the tech tree out, you're just like, yeah, like probably this will happen. **Akshat Bubna** [10:17] Yeah, like you were so close. **Swyx** [10:18] I was literally like, but the funny story there is, at the same time, we were talking to a bunch of customers who needed something like sandboxing. This is like 2023. **Akshat Bubna** [10:30] Yeah, yeah, yeah. **Swyx** [10:31] So we built. **Akshat Bubna** [10:32] You should use a new APIright after that. **Swyx** [10:33] Yeah. **Akshat Bubna** [10:33] Yes. **Swyx** [10:34] Like we built sandboxes in May of 2023 before anyone was even knew this was going to be a thing. And the first example we published was, we took Small Developer and put it in a loop so the agent can iterate on itself. **Akshat Bubna** [10:47] Loops are hard these days. **Swyx** [10:49] Yeah. **Akshat Bubna** [10:51] Loops in, when was this, 2023? **Swyx** [10:53] Yeah. **Akshat Bubna** [10:54] A small chick. **Swyx** [10:54] It's like mid-2023. **Akshat Bubna** [10:56] I mean, so I mean, those sort of, obviously for listeners, like the problem was the models are not built for any of this,right? Like you're trying to like, they're not post-trained to understand, like, you know, looping and like self-correction and tool calling was there, but like also not that great. **Swyx** [11:10] Yeah, I don't remember if you used tool calling in this one, but yeah, the models would just diverge after like 10 iterations and not produce anything meaningful. **Akshat Bubna** [11:18] Yeah. But like then, so, I mean, okay, like now talking to myself three years ago, the answer is. **Swyx** [11:23] Of course they will get. **Akshat Bubna** [11:24] Collect all the failures, build benchmark, and then collect all the, you know, examples, build an RL environment, sell it for like $10 billion to Meta, and then also train a model and then sell that for $60 billion to Elon. And it's like, yeah, put in the money machine. Like it's like, it's actually about their heart. **Swyx** [11:43] I mean, it's hard to have that kind of inherent conviction that the stuff will get that much better. **Akshat Bubna** [11:48] Even retrospect, it's so fucking obvious. Like. **Swyx** [11:51] Fair enough. **Akshat Bubna** [11:52] Like what else were we doing back then? I don't know. Anyway, yeah, so, I mean, that was the start of your sandboxing journey,right? I feel like it didn't blow up, blow up until like last year. **Swyx** [12:04] Yeah. **Akshat Bubna** [12:05] So there was like a couple of years of quietness. **Swyx** [12:07] Exactly, yeah. **Akshat Bubna** [12:08] We were very underrated product value. Like my experience with Modal, Charles, before he had joined Modal, met this guy at a hackathon and he really insisted we wanted to run some small model, you know, not hosted anywhere. And he's like, ah, there's this cool company Modal. They'll like spin up a GPU sandbox, we can throw it on there. They'll take a Hugging Face link and, you know, like there's so much value justright there,right? Like instant hosting, spin it up, spin it down. It'll stay cold, but, you know, we run the demo a few days later, it'll come back up. And like all this stuff in retrospect, like it's still what we needed like today. **Swyx** [12:41] Yeah, I mean, it's still needed today. Obviously, workload shapes have changed a lot as we run stuff for people with really massive production scale. And there, it's not about scaling from zero to one, but it's how do we scale really elastically from like 1,000 to 1,500 GPUs very quickly in a given region. It's the same shape problem. **Akshat Bubna** [13:05] Okay, so you look at, say, Cursor Composer,right? They had a, we'll do RL on a model every couple hours. You guys have a whole version of RL inference gym and whatnot. When you look at workloads like that, you're basically doing train runs where you need to scale up, scale down every hour, thousands of GPUs,right? That's the example for we do need it,right? **Swyx** [13:27] Well, so actually, I'll take a step back and maybe talk about like how people use Modal today. Because our biggest use case actually is Elastic inference. And the thing we first found product-market fit with was inference for custom models. ### Elastic Inference **Swyx** [13:43] So we kind of stayed away from the LLM space. And we were serving companies like Suno for audio, Runway for video, Robotics, CompBio companies that train their own model elsewhere, but Modal is the best black box that for deployment, scaling to however many GPUs you need as your traffic pattern changes. And we saw all of them actually like have a very unpredictable traffic pattern. It's like diurnal. It's some days like the company will build launch and, you know, they'll need like way more. And it's not just one model that they deploy. All these companies deploy lots of different models in different regions. And so the auto scaling problem becomes even harder because then you have to scale within a certain region. And those cycles sort of are offset. So at different times, you need to scale up in different regions. So that's like our sort of. **Akshat Bubna** [14:37] And that, you know, that in and of itself is a huge category. There's a bunch of inference providers which, you know, provide this fireworks, does this as a service together, whatnot, base 10. That's kind of carved into its own niche for language models, at leastright now. **Swyx** [14:51] Yeah, I mean, the thing that we have actually specialized in is the auto scaling aspect. Because we found that it's not universally true that everyone else can auto scale. And we've gone deeper into it on the tech side, but we've incorporated GPU snapshotting into the product so we can actually take the GPU state, like you torch compile a model, snapshot it, and the next cold starts way faster. And so going back to your question, it's why you need a lot of burstiness for inference. But then people also do a lot of on-demand training. Like for RL stuff, your rollouts are bursty, as you said. People also do a lot of batch jobs. So we'll see a lot of companies, before they have a training run, they'll need thousands of GPUs to run encoding or something like that. And I think those things are much more bursty than I agree that agents are not that bursty. Sandboxes are, except when you're doing RL. RL is insanely bursty. **Akshat Bubna** [15:44] Yeah. **Swyx** [15:44] Yeah, like when you're doing rollouts, you sometimes need 100,000 sandboxes and then you're in the seconds. **Akshat Bubna** [15:52] Yeah. I'm curious if you've seen early sparks of continual learning. There's some people, like our friends at Ngram recently announced this. They're trying to do training. That also seems like a different workload,right? If you're doing training 24/7, per se, there's a very weird dynamic of how you're using GPUs between people and whatnot. But it seems like something you guys would work for. **Swyx** [16:15] As you said, we're fortunate to work with a number of customers at the frontier at Ngram, some of our customers. And they are taking the primitives we have and trying to use them in very interesting ways, like continual learning. It's possible as the stuff gets better, some of that will be part of our offering as well if more people need it. But we're participating to see how it shakes out. **Akshat Bubna** [16:39] Is there a primitive that you added after sandboxing that was the next step in the story? **Swyx** [16:46] I guess we've been going much deeper into LLM inference because we realized that some of the advantages we have with like auto scaling, again, especially in different regions and whatnot, are not present elsewhere. And the place where we had a gap was we weren't working on the model there itself. Like we were a black box. And we realized that we actually can get to frontier-level model performance by having great people who work on all of this. And we've actually been open sourcing a lot of our work in terms of recently we shared our work on DeFlash, which is a block-based speculator. And we've open sourced all of it. So you can get, by using open source DeFlash, you can get the same performance as you would with one of the proprietary providers. And the next thing we're thinking about here. **Akshat Bubna** [17:38] I thought this was actually an interesting block question as well,right? Like, I think in here you make a claim that, you know, not a claim, just that how effective speculative decoding really just gets you. **Swyx** [17:48] Yeah. **Akshat Bubna** [17:48] Anything you want to point out from this around, you know, what people should know? ### Speculative Decoding **Swyx** [17:53] Yeah, absolutely. I mean, the high-level summaries, it would help to describe what speculative decoding is. **Akshat Bubna** [17:59] Yes. **Swyx** [18:00] I think, like, so we've covered like EGO and all those, like Hydra and all those things, but it was like two years ago. **Akshat Bubna** [18:06] I think it doesn't hurt,right? **Swyx** [18:07] The speculative decoding is you have a smaller model called a draft model, predict tokens ahead of the bigger model. And then you have the bigger model verify all of this, all the tokens are predicted. And the reason it's faster is if you're predicting one token at once, you're kind of bound by memory bandwidth. But if you can batch the verification of the draft model, then you're much more efficient using compute and it's faster. And as long as your draft model is producing a lot of tokens that can get accepted, which is called the accept length, you can get speed up that's multiple times of, you know, the original model speed. And that's what we highlight here. It's like people talk a lot about we made these kernels faster and whatnot, but improving kernel only gives you like a few percentage points of improvement. And increasing accept length literally is a multiplicative decrease in. **Akshat Bubna** [19:01] Like two to four X. **Swyx** [19:02] Yeah, exactly. **Akshat Bubna** [19:03] Without much head-on performance. **Swyx** [19:05] Yeah. **Akshat Bubna** [19:05] I think it may be, I mean, you are running a second model,right? So it maybe is like more expensive in the compute, but. **Swyx** [19:12] I meant quality performance. But yeah, I mean, I think. **Akshat Bubna** [19:14] So there's no drop in quality performance because you're always, you're never accepting a token that does big models. **Swyx** [19:20] It should be better or it's same. **Akshat Bubna** [19:21] Exactly,right? Yeah. **Swyx** [19:23] And so we've been working a bunch on DeFlash, which is a block-based speculator. So it's, instead of predicting one token at a time, it's predicting a block. And we've been open sourcing our work with it. The next thing for us here is for helping people train speculators and custom models. It's something that traditionally is very FDE-driven, support, deployed, engineer-driven, like you work with customers and help them do that. And our vision for, this is why we launched Auto Endpoints is we want to make frontier-level performance available to everyone. And so we've mentioned this announcement, we kind of teased it. The next thing we're launching is basically as you run an Auto Endpoint, we shadow traffic and. **Akshat Bubna** [20:09] Do you want to explain what Auto Endpoints are? ### Auto Endpoints **Swyx** [20:11] Yeah. **Akshat Bubna** [20:12] I love it. **Swyx** [20:13] Yeah. So this is, I guess, going back to your modal is you touch the code, but sometimes people actually don't want to touch the code and they want to get started with an endpoint that works and has all the great performance and scalability that modal has. So we've made that easier with basically a way to create an endpoint from our UI, from the CLI that has all of our optimizations that we talked about, like the DeFlash stuff already baked in. And there's full transparency. So we give you the code, you can go run it yourself, and if you want, you can eject out into the full modal experience, which we see as people get sophisticated, they do want to tweak the models, they want to fine-tune stuff. You can still do all of that. It's not a black box. And yeah, the next thing, as we tease later in the post, is how do we give you value even beyond this in terms of having your draft models evolve as your data distribution evolves, again, without having to talk to a person. And yeah. **Akshat Bubna** [21:14] I guess just to kind of understand it directly, I mean, you know, obviously you have the GPUs, you have an endpoint that's compatible, you serve open model. If someone was to do this themselves, what's the delta that you guys provide? So you do a lot of open source great work on effective inference. How does it compare to say, I take the same model, GLM 5.2 FP8, take off the shelf inference engine, VLLM, SGLang, you know, get compute of similar capacity, similar cost. What's the kind of delta that plugging into something like this offers outside of the benefit of, you know, scaling? **Swyx** [21:49] It's kind of interesting because we've taken the approach of open sourcing our contributions and upstreaming them. We work closely with the SGLang team. We actually want the improvements that our team comes up with to be there in open source for others to use, even outside of modal. The benefit to us is we have a team that has significant expertise in terms of if you do have something that is not there, our team can help you get that performance first. The other thing is with these endpoints, we are way more elastic, as you said, than anyone else. And you have true scaling to zero, you have true burstiness. And in practice, that matters a lot more to people than just finding the GPU and running model code. **Akshat Bubna** [22:35] Yeah. And I will say it's actually not that straightforward. Just like what I said is easier said than done,right? **Swyx** [22:41] Yeah. **Akshat Bubna** [22:42] It's, I think still for the average person, still hard to just gut check using different, there's quite a bit of combinations you can make there. The trade-offs aren't really known at face value. **Swyx** [22:55] Yeah, I mean, it's not just that. I think it's that running production-grade inference is a hard infer problem. Even if you subtract out the auto scaling, it's controlling things like tail latency and making sure every request is delivered at least once and whatnot. **Akshat Bubna** [23:14] There's a lot of innovation that you can do here. I think it's very interesting that you're starting to encroach on, like as you become a full cloud, you're starting to encroach on other people's turf. What will you not do? **Swyx** [23:27] Well, we want to follow our users and make sure they get like a platform that has everything that works well together. Soright now we're kind of focused on the model lifecycle and the agent lifecycle. So both like going from data prep to training to inference, and then also if I want to deploy a background agent, let's say, you know, sandbox, you put some storage, a whole bunch of other stuff. **Akshat Bubna** [23:53] We talked to a coal who did Open Inspect, yeah. And obviously Real Inspect also is on modal. **Swyx** [23:59] Yeah, so Ramp Inspect was a great example of a background agent that was really successful because they were able to use some of the primitives like snapshotting and fast scaling to just have something that feels really reactive and works well. **Akshat Bubna** [24:17] Yeah. That's the new CTO of Rampright there. **Swyx** [24:20] Yeah, Rahul. It was really, really fun. Yeah, I mean, you know, I think all very bullish. Like, you know, one of my reflections was also I did not originally, obviously when I met you guys, you weren't that much in the GPU game and now you're all about inference. And one of the points that I hinged on for Jensen's keynote at GTC this year was what we're calling like the inference inflection,right? That, let's say in AI workloads or machine learning workloads, it used to be like, let's call it eight to one, GPU to CPU, and now it's more like one to one, which is like an interesting, like because of how much agents basically are blocked or call out to CPU-heavy stuff, the actual like limiting factor that swings back and forth from GPU to CPU a lot more than it used to be all GPU and then occasional CPU. **Akshat Bubna** [25:16] Yeah. **Swyx** [25:16] GPU, CPU. And now it's like just constantly. And you just have to colocate everything. ### Supercloud Capacity **Akshat Bubna** [25:22] Yeah. And that's one of the things that actually, again, we see as something appealing about modal, which is we've built this capacity pool that spans 17 cloud providers. So we're very good at running on various kinds of cloud capacity across the world. **Swyx** [25:39] You don't have your own data centers. **Akshat Bubna** [25:40] We don't have our own data centers. We just run across a lot of NeoClouds and providers. **Swyx** [25:46] Yeah, yeah, yeah. **Akshat Bubna** [25:47] You're running the math and you're like, what's the cutover point where you're like? **Swyx** [25:51] Yeah, it's a good question. I mean, part of it is we see our differentiator in the software layer and being capped to light and focusing on the software helps us move really fast. So far it's worked out well because there are so many other people building data centers that we're able to work effectively with them and again, focus on what makes us special. **Akshat Bubna** [26:10] Yeah. 17 gets you into like the local providers sometimes. Like, this was the most interesting one. **Swyx** [26:17] There are actually a lot more NeoClouds than you expect. And they all have various degrees of various levels of reliability. And that's why it's something we've invested a lot of time in is actually building our own reliability layer on top. So if the GPU falls off the bus or something happens, user workloads are not affected. And that actually lets us use a lot more capacity than you as a user would be able to. **Akshat Bubna** [26:47] It's a useful thing to have because now everyone knows like what layer you are and like you sort of optimize for being the supercloud of all clouds. **Swyx** [26:56] Yeah, that's the idea. And so I guess when you mentioned colocation, that's another interesting thing where one thing we've seen is people come to us when they want very specifically located CPUs or GPUs like they want. **Akshat Bubna** [27:12] Oh, to pin it in like EU? **Swyx** [27:14] Exactly. Or EU, EU West, data, Australia. **Akshat Bubna** [27:17] It's either locality thing or performance or what? **Swyx** [27:19] It's either data locality or latency. Like you want your, they're running sandboxes in modal, they want them to beright next to your GPU. **Akshat Bubna** [27:26] Yeah, that's easy. That's it. You know. That is important in all those things. And so like you've kind of accidentally, I don't know if it's accidental, but like you've built the perfect primitive for agents to express themselves. And then, you know, like it's almost very funny how every extra development just involves more file system, just involves more CPU. **Swyx** [27:45] Yeah. **Akshat Bubna** [27:46] Just like the things that you already have. I don't know much about if there's any like networking usages that are interesting, but you've also done some good work on networking. **Swyx** [27:55] Yeah, I mean, that's exactlyright. Like we're sort of just taking compute through the networking and building stuff on that layer for, again, the stuff people need. **Akshat Bubna** [28:04] Yeah. **Swyx** [28:05] We see a few interesting network things coming up. One is people actually want networked sandboxes. So we have. ### Networking **Akshat Bubna** [28:12] For like a Docker cluster type thing. Sorry, Docker Swarm. What is it called? **Swyx** [28:17] Compose. **Akshat Bubna** [28:18] Compose type thing. **Swyx** [28:19] Yeah. So actually, if you want Docker Compose, our sandboxes now support this thing called sidecars. So you can, a sandbox is actually a pod of containers and you can run multiple containers in sandbox. Also useful because going back to networking, people want a lot of control over outbound networking from a sandbox. Like they might want to run a man-in-the-middle proxy for like maybe logging stuff for RL or controlling how egress can happen to a domain, inspecting credentials. And yeah, so we've kind of had to build a lot of that stuff ourselves. **Akshat Bubna** [28:53] Yeah. **Swyx** [28:54] But then also sometimes people actually want sandboxes spanning multiple nodes to talk to each other, which is an emerging thing we're seeing. We have support for that for a different reason. And yeah, we'll see if that becomes the same. **Akshat Bubna** [29:07] Like just an open socket. This is directly like mTLS. **Swyx** [29:11] We do support that, which is you can expose a tunnel inside a sandbox. **Akshat Bubna** [29:15] Yeah. **Swyx** [29:16] And then you can either expose it to the public internet or it can be, and you can add like an HTTP auth layer above it. But we have this thing called I6PN, which we haven't talked about, which is this like overlay network using IPv6 addresses. So if modal containers within the same workspace, when this is enabled, can actually address each other using this private IPv6 address and no one else can. It's like sort of private networking for containers. We actually built it because we needed it as a primitive for our distributed training product. So we have this other feature, which is you can add a decorator to a function and you get a cluster of GPUs and they have RDMA networking. So you can run a distributed training job that's truly serverless. And we did the overlay network for that. But then we've seen that people are using it for other reasons and I'm kind of intrigued to see, yeah, what would people do with it. **Akshat Bubna** [30:14] Build primitives and people figure it out,right? **Swyx** [30:16] Yeah, exactly. **Akshat Bubna** [30:17] Yeah, but that's pretty interesting. **Swyx** [30:18] They're like, they read the docs, we think, let me use that for something they never intended to. This was literally not even in our docs page. People somehow found it and they're using it. **Akshat Bubna** [30:28] I mean, the way you portrayed it with like RDMA versus TCP, like very well laid out, but just the transfer speed change at scale for RL, like yeah, you have it, you have it built in. I'm sure someone found it, found it to be a lot more efficient before you actually made a thing out of it,right? **Swyx** [30:47] Yeah. And not to split hairs, I guess the overlay network actually is the TCP overlay network. The reason we have that is you need that to do the key exchange for RDMA before you set up the RDMA network on top of that. But then people found the TCP part. **Akshat Bubna** [31:03] Can I tell you, this is like a big aha moment for me because, so I, you know, I review 2,200 submissions for the World's Fair. **Swyx** [31:11] Yeah. **Akshat Bubna** [31:11] And then I got this from John Osterholt, who I don't know if do you know John Osterholt? **Swyx** [31:15] The name sounds familiar. **Akshat Bubna** [31:17] He published a, he's a well-known professor, published a lot of interesting software design books. And this is the talk he chose to submit. It's on RDMA and inference. And I'm like, you wouldn't think that this guy who is like kind of operating systems guy would care about RDMA. **Swyx** [31:35] I mean, it makes sense to me because. **Akshat Bubna** [31:38] This is Google Cloud,right? **Swyx** [31:40] Like the way you move around your KV cache and how efficiently you can do it, how efficiently you move your weights from your training GPUs to your inference GPUs in RL is, there's a lot of degrees of freedom and it is basically a systems problem of moving memory around and scheduling. **Akshat Bubna** [31:58] This shows you how primitive my understanding of networking stuff is. Is this like the domain of WireGuard as well? **Swyx** [32:05] Not quite. **Akshat Bubna** [32:05] So adjacent? **Swyx** [32:07] So explain everything. **Akshat Bubna** [32:09] Sure, sure. How do we move memory around GPUs? **Swyx** [32:13] No, sorry. Yeah, that is memory. Sorry, I was talking more and maybe I was talking like five minutes back about the private IPv6 addressing that you've set up. It's like it's basically a VPN. **Akshat Bubna** [32:25] Yeah, it's sort of like a VPN. And yeah, WireGuard is, yeah, you'reright. It is. **Swyx** [32:31] Yeah, you already moved on to new topics. **Akshat Bubna** [32:32] A similar, in the same space, WireGuard is encrypted and this is. **Swyx** [32:37] You don't need encryption. **Akshat Bubna** [32:38] Yes, it's not encrypted. That's no interference. This is TCP and we have eBPF programs that will reject or allow the TCP connection based on whether you're allowed to do it. **Swyx** [32:50] Used to involve a full sidecar, but now you have eBPF in a Linux kernel. **Akshat Bubna** [32:54] Yeah. **Swyx** [32:54] Yeah. **Akshat Bubna** [32:55] I don't know if this is a natural follow-on to the topic of like my skepticism on distributed training is that, well, like people spend a lot of money on like cables to hook up GPUs and even that is not like fast enough and that's the bottleneck. Is your networking fast enough? **Swyx** [33:14] Yeah, so I guess you're talking about sort of fully distributed training like a Dialog or something, which is like cross data center. **Akshat Bubna** [33:21] That would be, yes, that's the extreme. **Swyx** [33:23] Yeah, you're kind of in the middle and then other people would have like the Milanox cables up in like their actual data center. When you run multi-node training on modal, RDMA, I think Milanox is, or Infiniband is like, also uses RDMA. But basically it's a way to bypass the TCP networking stack and transfer stuff much faster between one node to the other. And we have, I think like three terabit per second internal networking, which is the standard that's needed. **Akshat Bubna** [33:56] Okay. So I misunderstood what part of the stack you were. **Swyx** [33:59] 15 nodes over TCP is going to over RDMA. **Akshat Bubna** [34:01] Okay. **Swyx** [34:04] Yeah, I mean, very impressive. **Akshat Bubna** [34:07] So effectively you're extending sort of like the model philosophy to the training cluster. Like, yeah. **Swyx** [34:14] Yeah, and we're not going for obviously like large scale pre-training runs. The thing that we've built multi-node training for is we see a lot of smaller scale post-training. Like people are post-training like medium size quant models so they can get higher quality on inference. This is a perfect fit for something like that. **Akshat Bubna** [34:36] Yeah. That is my impression of how a lot of these labs explore branches in post-training and then eventually merge whatever they find in. **Swyx** [34:46] Yeah, the other use case we've seen for multi-node training is even if you have a big cluster, your researchers are still doing small runs and. **Akshat Bubna** [34:53] Yes. **Swyx** [34:53] You're having elasticity there matters a lot more. **Akshat Bubna** [34:56] Yeah. Like this is actually like the current limiting factor for auto research, which is like you basically need to give your model some GPUs. ### Auto Research **Swyx** [35:06] We have a blog post on auto research and modal is, yeah, like turns out to be a pretty good substrate for that. **Akshat Bubna** [35:14] So my impression is auto research means many things, like anything that Andre coins. Right now still science fair,right? Like not actually like, I don't know how many people are actually doing this. **Swyx** [35:24] I thought the same thing. **Akshat Bubna** [35:25] Yeah, you would know. **Swyx** [35:26] We, like our internal both training and inference teams actually use this sort of the general shape of this quite a bit. Like we have this one internal repo called auto inference, which is essentially we've automated our own FDE efforts using this harness, which is the agent will just spin up a sweep of different things. It'll even run like an NVIDIA inside profiler and it'll like tweak configs and it'll arrive theright thing. It'll change your GPUs from H200 to B200 and actually works really well. **Akshat Bubna** [36:01] Nice. **Swyx** [36:02] So. **Akshat Bubna** [36:02] By the way, I enjoy that your FDE is so technical that you have to do these things. It's very different from FDE from other people. **Swyx** [36:09] Yeah, yeah. Our FDE team is essentially they're like applied inference researchers or applied training researchers. **Akshat Bubna** [36:17] Someone told me like they have to be able to build, but they also have to be able to sell. Do they have to sell or are they like, they're good? It's just like post-sale type of thing. **Swyx** [36:24] It does. Being able to talk to a customer and engage effectively with them matters a lot. **Akshat Bubna** [36:29] They all want the same thing. **Swyx** [36:30] Yeah. But it's not really a sort of sales thing. We pair them with, we have solution architects as well that are more on the pre-sales side. **Akshat Bubna** [36:38] Okay, let's spend a bit more time on auto research. This is a big focus for me for this year. Where does this go? You know, like have people explored enough, like, you know, there's all these beautiful charts of like improve Linux to the level of a bit and then you find the next thing. Is this basically sort of one abstraction up from normal training? Is that how we think about it? Or do you think about it differently? Like model level training versus basically like AI driven hyperparameter search. Some people call it like neural architecture search or whatever,right? **Swyx** [37:09] Yeah, I mean, so the stuff I've seen people do with it is lower on the architecture level. It's pretty much tweaking parameters, but it's basically a hyperparameter sweep that's guided by some sort of model intuition. So it's like much more efficient than whatever other sweeper you would have. **Akshat Bubna** [37:27] Yeah, I mean, you know, it's just a question of where you want to spend your compute, you know? **Swyx** [37:31] Right. **Akshat Bubna** [37:31] Because yeah, you can just throw infinite amounts of money on this and somehow you'll bang out Shakespeare, you know? **Swyx** [37:37] Yeah, infinite monkey. **Akshat Bubna** [37:39] Yeah, I mean, so like very good for modal. And I think it's also very important that agents can spin up other agents. They can spin up their own infrastructure. Like very good for you. How good are LLMs at generating model code? Like, you know, the benefit of existing pre-LLMs is that you are in the data. **Swyx** [37:57] Yeah, they're actually surprisingly good. I think like pre-Claude 4 they were not, and then now they're able to one-shot stuff out of the box. We're playing around with releasing like a Modal Bench for like the harder things that the LLMs cannot do yet. And maybe. **Akshat Bubna** [38:14] What's an example of that? **Swyx** [38:15] I think the things that sometimes agents struggle with withoutright guidance and a skill is how to use the rest of our observability. Like how to, if something is failing, like how do you look at the logs and then update theright thing? It's sort of reasoning about that. **Akshat Bubna** [38:36] But they're able to. **Swyx** [38:37] One-shot, like. **Akshat Bubna** [38:38] Yeah. You can just add a skill to it. **Swyx** [38:41] Yeah. So we have a model skill now that, which is kind of actually why we built this Modal Bench, it's to find things like that so we can address them in our. **Akshat Bubna** [38:49] It's your new skill. Yeah. **Swyx** [38:51] No, no, I mean, it's good. Are you facing any shortages? You know, like we talk a lot about GPU shortages, but also CPU, also memory. Yeah. **Akshat Bubna** [39:00] We have had a lot of growth, which means that there's a, we've had to be much better about proactive capacity planning. **Swyx** [39:10] Yeah. **Akshat Bubna** [39:10] So we have. **Swyx** [39:12] Which by the way, like it's like an MBA's like dream job is like just planning this stuff. I think last time you and I talked about something about this. **Akshat Bubna** [39:18] Yeah, I mean, we have a really competent team of people that we call, the roles call it compute strategy. So yeah, if anyone listening here wants to work on that. **Swyx** [39:28] Compute strategy? **Akshat Bubna** [39:28] Yeah, yeah. **Swyx** [39:29] I mean, the normie is called it FP&A or something. **Akshat Bubna** [39:33] Well, it's not FP&A. There's a lot of interesting financial questions of like, what is the blend between one year and three year reservations? How do we forecast our own capacity? How do we, basically, especially since our capacity is very fungible across different GPU types and different regions, like you basically have to model a lot of it. And you also have to have an opinion on how the supply chain is going to evolve. And then you have to like take bets based on that. **Swyx** [40:04] Yeah, tokenomics. **Akshat Bubna** [40:05] This is like probably not a real point, but I was trying to think about like what other industries, I was trying to think about like, you know, we cannot be first to like these kinds of problems. **Swyx** [40:14] Yeah. **Akshat Bubna** [40:14] And what other industries have had this? And I was like airlines with fuel. And like they have to hedge their fuel. And like I think for a long time Southwest, because they made it like a hero fuel bet, they like were like super low cost because compared to everyone else. **Swyx** [40:29] Yeah, I haven't thought about that. **Akshat Bubna** [40:31] We're at a fun time too, you know? **Swyx** [40:32] Yeah, it's a lot of the compute business in general for us is also about being very good about capacity management. That is how you have great unit economics, but also over time it's how you can unlock more value for customers. Like one of the things we're building now is like a way for customers to get, if they don't care about latency, like get much cheaper pricing and they'll get results back in like the next 24 hours or something. Like a batch tier essentially. And those are levers we have because we control the whole stack and scheduling and whatnot to give people sufficient. **Akshat Bubna** [41:08] Yeah, I feel like they're not as popular. Like those, like the frontier labs have all those APIs. They're not as popular as they should be. **Swyx** [41:15] The demand that we see for something like that is actually not for LLMs. Although sometimes people want to run evals and do synthetic data prep and there it makes sense. But it's from a lot of non-LLM companies, like people who are doing computational bio, like they haven't run really big batch jobs and they don't care about when they get it back. **Akshat Bubna** [41:37] Yeah, and like they have a reasonable, it's also like a cousin to the stopping problem of like, will this finish in time? **Swyx** [41:46] Yeah, you can bound it. **Akshat Bubna** [41:47] Yeah, like you can give people SLAs on it. **Swyx** [41:50] Yeah, yeah. **Akshat Bubna** [41:50] I think what's interesting is like the next phase of modal. Like what, you know, do people expect from you now that you're sort of established and you're like a well-known compute player among all these leading companies? You had an inference launch week and we talked a little bit about the launches. Like what else? What else should people know? **Swyx** [42:10] We are building primitives that make our user's lives much easier. So I think, for example, LLM inference, thousands more companies are going to post-train their own models and deploy open source models for inference. So we're thinking a lot about what is the best product shape for that. And that involves everything from our training gym to then endpoints that get frontier level performance. You know, again, but I haven't talked to anyone. It looks somewhat different on other verticals. Like we're also seeing a lot of real-time audio video stuff in there, which is why like we're working on things like regional routing with fallbacks. So you can get sort of GPUs that are as close to your users as possible. So you get like low latency for video streaming and whatnot. And then on the agent side, it's we're still sort of working very closely with our customers because stuff is changing so fast in terms of what they need. And I think beyond sandboxes and persistent file systems, there's a lot of other things people need from this agent stack as they build production agents. So yeah, we're thinking about those other things that fit in there. **Akshat Bubna** [43:27] I want to ask what the other things are. **Swyx** [43:30] Yeah, I probably should shareright now. **Akshat Bubna** [43:32] I think, okay, so I do think a lot about the principal components of cloud and you do talk about compute storage networking. Because so far for me it's fine. So far for, I mean, the first couple generations of cloud it's fine. What's different, qualitatively different about agents that you need some new permission level. Like a lot of people, you know, obviously, okay, and I'll just kind of spew tokens at you until it like hopefully sparks something. **Swyx** [43:58] Yeah. **Akshat Bubna** [43:59] Like the new level now is whatever cloud code does, which is dangerously skip permissions or like allow list by command or like whatever,right? And sometimes they're like, okay, we have like this adaptive thinking mode where like just trust me, bro, I will make the calls for you. Is that it? You know, like basically like sort of LLM mediated permissions. **Swyx** [44:18] Right, looping it with a goal and letting it run. **Akshat Bubna** [44:21] Yeah, I mean, I'm skeptical of LLM mediated permission for stuff that is at the sandbox level because you do want hard boundaries. Otherwise, obviously someone can exfiltrate stuff. **Swyx** [44:34] But like maybe that's old school thinking. Maybe we're at the dinosaurs. Maybe the AI OS or the LLM OS is really, the kernel is a goddamn LLM. Like it makes you feel uncomfortable. **Akshat Bubna** [44:46] Yeah, I'm in the world. **Swyx** [44:47] But that's what trusting the LLM is. Like imagine a spiritual cow, perfect LLM. **Akshat Bubna** [44:51] Right. **Swyx** [44:52] Let it. **Akshat Bubna** [44:53] Maybe, maybe. **Swyx** [44:56] I want to test the boundaries,right? Like obviously I don't believe that, but I want to see where I'm wrong. Because that's the non-consensus. **Akshat Bubna** [45:03] Yeah, I mean, I think you always need hard guardrails when you want, and you can pair those with software guardrails,right? And that's going to be a lot of mediated. **Swyx** [45:15] I want to end with a couple of your commentary on like the ecosystem outside of modal. Managed agents, everyone has one. Gemini, OpenAI, Claude, very useful for you. But also like it is their way of starting to edge into your space. **Akshat Bubna** [45:31] Yeah. **Swyx** [45:32] What's going on? **Akshat Bubna** [45:33] Yeah, I mean, we're very excited to partner with Anthropic and some of the other foundation labs, Bull on Name Who we're also working with. The way we see it is the managed agent thing is a great place to start if you're starting out building an agent. But then when you get to building something more production grade, like your company that's like RAP that's building their own, RAP also runs their accounting agent on us, so their external facing agent. You need a lot more control over your compute primitive on things like what sort of, how do you persist different files that the agent has access to and how do you snapshot and restore? How do you control the networking? Maybe you want GPUs. When you get to that point, you kind of want a specialized sandbox provider that gives you those things. And that's the role that we are trying to play. We don't really have an opinion on the harness, whether it runs in, it's a cloud managed agent and you hook it up to Modal Sandbox, we run the harness in Modal Sandbox. We'll see where people converge with that. **Swyx** [46:40] Yeah. **Akshat Bubna** [46:41] Do you have any opinions on like the meta harnesses? It's just another layer on top of these things. **Swyx** [46:46] You mean like the OpenPie and. **Akshat Bubna** [46:48] OpenPie is one. I think Vercel had one, which I can't remember the name ofright now. Fred Schott had one. And then to me, most recently was Databricks that had OmniGen. All these are sort of meta harness, like kind of pseudo agent cloud type things. **Swyx** [47:05] I personally have not played around with them and built agents with them. **Akshat Bubna** [47:09] I mean, everything's bullish modal as long as it consumes more infra. **Swyx** [47:12] That's why we're focusing on the infra layer. It's somewhere where our relative competences and also it's a hard problem solving. **Akshat Bubna** [47:21] Yeah. I mean, I will say like just generally reflecting on, I don't know if there's other topics on modal, but like just generally reflecting as an infra person, not as intense as you, but in that field, this has like been the most exciting time in infra. Like it was boring actually for a while. And you couldn't really get people excited about data infrastructure. Like Eric would get on data council, everyone just watched the video. And like it's like, look at how many sandboxes I can spin up and no one gave a crap. **Swyx** [47:54] Yeah. **Akshat Bubna** [47:55] And like now everyone gives a crap. **Swyx** [47:57] That's true. It is a very exciting time. And I think a lot of that's driven by just the amount of scale all of this stuff needs. **Akshat Bubna** [48:05] I think like a lot of your initiatives, a lot of your like product directions make sense in retrospect, which is like the best kind. But I wouldn't necessarily have thought about it myself. **Swyx** [48:16] We need the predictions, you know? **Akshat Bubna** [48:17] I mean, I think there's a lot that you just don't even see,right? Like you have the batch, you have the voice, you have the multi-model, but what else, you know? **Swyx** [48:24] What else is coming up for us? **Akshat Bubna** [48:26] Yeah, where do you see things going? **Swyx** [48:28] Yeah, I mean, in general, it's clear that there's a, obviously it's a huge shift happening. I think one thing that's not as obvious to people because LLM inference gets talked about so much and is also, we work with a lot of companies that are doing things like drug discovery and computational bio, like the chat discovery world. Big things are probably going to happen there. We work with a lot of robotics companies that are actually putting robots in like active deployments and getting good results out of them. **Akshat Bubna** [49:00] Is there air-gapped modal? Is there a version that is like on-prem, air-gapped, whatever? **Swyx** [49:04] No, we don't. **Akshat Bubna** [49:05] You should cloud only. **Swyx** [49:07] Yeah. **Akshat Bubna** [49:07] Yeah. Okay. But yeah, I mean, so what you're saying is like because you're focused on primitives and they're good primitives, you find use cases and all these kinds of things. Actually probably diversifies you a little bit away from LLMs all the time. **Swyx** [49:20] Yeah, absolutely. Our goal isn't to only serve the LLM inference market. **Akshat Bubna** [49:25] You know, not just on the website, the audio, the. **Swyx** [49:27] Yeah, we said both on the. **Akshat Bubna** [49:29] Computational bio, images. Yeah, I mean, there's a lot here. There's Qty TTS, customizing, oh, Chatterbox. You know, there was a customizing Whisper. **Swyx** [49:39] Yeah, yeah, yeah. **Akshat Bubna** [49:40] This screen reminds me of a fallen competitor, which Replicate. What's your postmortem on what happened? **Swyx** [49:49] This is one thing we've kind of stayed away from is providing an API for models. Because I think providing model APIs is, some of it ends up serving like a really hobbyist market, which is much less sticky. And we've always wanted to build for companies that are building sort of products and need sort of more flexibility that's not just an API. **Akshat Bubna** [50:12] Which you can build an API for a model and this is clearly what it is. But you can, but you're saying you can wrap it into a more fully functioning backend that you run. **Swyx** [50:21] Yeah, so actually all of our examples, it's not that spin up this model, here's an API token, use it. They're actually all code. And so the point is that this is an example. **Akshat Bubna** [50:31] Starter code. **Swyx** [50:32] Yeah, but you can tweak it however you want. And if you're like a company building a product, like computational bio and whatnot. Yeah. **Akshat Bubna** [50:40] I guess I'm trying to tease out for listeners. When does it stop becoming, oh, you're just an API call and you're just a wrapper on an API to becoming what you call a product,right? Like what is that layer? Like what, like, you know, obviously more lines of code, but like beyond that, what is the substance that people add that qualifies it to be something more? **Swyx** [51:01] I think there's a little bit of like a selection effect of like a lot of the companies who do want to get deeper into that level are probably building something that's more differentiated. And I think an example is like we, with LLM inference, originally we worked with companies that were building their own post-training frameworks or they were, RAMP actually early in the day was training their own tokenizer and like swapping out the tokenizer in Llama and whatnot. I'm not saying that's successful in that case. A better example is like, let's say Suno. Because Suno does not use Modal for training. **Akshat Bubna** [51:41] My hand upon, yeah. **Swyx** [51:42] But they use Modal for all their inference. And that's because they have like a custom, they have completely custom model architecture. And that means that they have to be at the code level and tweak things that are not, you know, it's an API. **Akshat Bubna** [51:56] It's interesting as well, like we had Ethan most recently on the XAI Grok team make a prediction that actually like the next tier in video gen is not a better video model. It's a better model or agent that orchestrates video models. **Swyx** [52:11] Oh, interesting. **Akshat Bubna** [52:11] Language model backbone that can use tools and write code. **Swyx** [52:15] Like, yes, I can make my 6 second video or my 10 second video from Grok, but actually I want my 6 minute video and I'm not going there through normal video gen. **Akshat Bubna** [52:24] Yeah, that's interesting. I actually, so we have GPU sandboxes and recently have seen a few companies doing sort of agents that do video manipulation or. **Swyx** [52:36] Yeah, give it FFmpeg and just run FFmpeg. **Akshat Bubna** [52:39] Like that, that's not enough. You need to give it Adobe. **Swyx** [52:42] Yeah, I hadn't put it together with like, it would actually be a video production thing. In my mind, these things were going more towards editing and. **Akshat Bubna** [52:50] Yeah, I think about this a lot. **Swyx** [52:53] Obviously. **Akshat Bubna** [52:54] Sorry. Luma, Luma agent is a version of this for video production, but you know, it's a one-off. I was going to get your quick takes on some other stuff that happens in recent news and see if you have anything interesting. ### Ecosystem Takes **Akshat Bubna** [53:07] Gitpod? Very like, somewhat like, you know, different market. They're in like sort of like the CICD market, but actually technically very impressive. I don't know if you've like taken a real look at them. **Swyx** [53:18] Yeah, we've, people on our team have talked to the Gitpod team and they've, they're technically very strong. **Akshat Bubna** [53:24] Yeah. **Swyx** [53:25] I actually am, we're very bullish in Modal and the CI market as well because there's more agents, coding agents. They're going to run a lot more CI and the primitives there can be much better. **Akshat Bubna** [53:38] I think there's a lot of wasted CI. **Swyx** [53:40] Yeah. **Akshat Bubna** [53:40] So is it just like, let's filter? Like what is the highest order bid here in improving CI for agents? **Swyx** [53:47] Well, there's a lot of wasted time in CI on like preparing your artifacts and like, you know, getting you to the, basically preparing your dependencies and whatnot. And obviously like build systems help with that, but like if you have primitives that are like memory snapshot and restore, can you just run CI more efficiently? **Akshat Bubna** [54:10] Oh, oh, okay. Okay, okay. Interesting. Yeah. I mean, another form of like, you know, on-demand compute. **Swyx** [54:17] Yeah, exactly. **Akshat Bubna** [54:18] Yeah, yeah. **Swyx** [54:18] It needs the same, again, platform. **Akshat Bubna** [54:20] Yeah. So, so for those who don't know, Gitpod rebranded to Ona. It was like this whole thing. I actually like sort of semi-sounded the alarm at Cognition. I was like, you should take these guys seriously because their infra is very good. **Swyx** [54:32] Yeah. **Akshat Bubna** [54:33] And, but, you know, then they joined OpenAI and presumably we'll see Codex Cloud from the Owner team, like which I think would be very, very strong. To me, like teams like that, that can set up the networking and like the secure boundaries for like your like agents to have their own cloud each effectively is what you're doing kind of. And I'm just trying to draw the analogy or the differences. If you have studied them, like what is the philosophical difference, you know? **Swyx** [55:02] My sense is maybe they didn't go off to theright market at theright time because we, I guess I'll just go lucky with like agentic use cases really taking off and needing like more of like a sandbox shaped thing than like mind setting is, yeah. **Akshat Bubna** [55:21] Sandboxes, like CICD is sandboxes. It's just like build time sandboxes versus runtime sandboxes. And actually it turned out runtime was better. **Swyx** [55:29] Right. And the difference there is runtime sandboxes have a different configuration surface of like how you configure images, how you like attach like storage and yeah, it's fascinating. **Akshat Bubna** [55:41] Other people, Astral, also OpenAI, also like Python tooling ecosystem people. Are you still sort of bullish building on top of Python? Also recently Modular also got bought by Qualcomm. Just any of your takes there. **Swyx** [55:57] Yeah, I mean, we had Python as our first SDK language because that was the language that people did data and ML in. Actually now have gone TypeScript SDKs as well. And our runtime is completely language-compatible. It's written in Rust, but it's not tied to Python by any means. We haven't seen, I think with like inference and training stuff, people are still very Python. And the interesting thing with like the agent stuff is people use our TypeScript SDK a lot more because they're not actually doing anything that needs ML. I don't think we'll have to go beyond that super soon because Python and TypeScript are just still dominant. **Akshat Bubna** [56:34] The last two languages in the world. **Swyx** [56:36] Yeah. **Akshat Bubna** [56:36] That's it. **Swyx** [56:37] Well, English and prompting. **Akshat Bubna** [56:39] English and prompting. I occasionally talk to people who try to build new languages. They're like, even, what's his name, Bret Taylor, who's chairman of OpenAI, was like, we need a new language for LLMs. No one has come across one. And I keep looking, you know, Python and TypeScript, you have a lot of data plus, but then also they are very imperfect as just as languages themselves. Then my close is, I think Modal used to be a big bet on developer experience and you've pivoted the team to agent experience. ### Agent Future **Akshat Bubna** [57:09] Is it like the way now, like do you do, can entire companies and unicorns, multi-unicorns be built on just having better agent experience? Do you need something else? **Swyx** [57:20] It's a big part of our identity. It's not just, you know, like the very tactical, how does an agent use the CLI, but it's also how easy is it to spin something up? Like what is your iteration time when you want to spin up a new service and you want to get something going in prod? In practice, that matters a lot to people. And I think it will continue to matter. Like people are building stuff even faster and if you give them ways to do it quickly and not have overhead, then. **Akshat Bubna** [57:51] I think the debate for me has been, do you do anything differently that is like very fundamentally different for developer experience versus agent experience? You seem to be on the side of, they're like this, they're like. **Swyx** [58:03] Yeah, we actually will set up a blog post on that. **Akshat Bubna** [58:05] Cosine similarity on like 0.9 or whatever. **Swyx** [58:08] Yeah, I mean, pretty much it's the main shift for us has been, as I said, like we built this benchmark, Modal Bench, to see where agents are lacking and yeah, actually literally add surface areas to a product if they're reaching for something, like maybe this should just be a CLI. **Akshat Bubna** [58:24] They hallucinate, yeah, they hallucinate their own features. **Swyx** [58:26] Yeah, and sometimes it makes sense. Like if they're reaching for this thing, it's product feedback, like give it to them. And then, yeah, actually moving, we used to only have like logs and metrics in our UI, just moving all those things to CLI as well so they're accessible in that form. **Akshat Bubna** [58:41] Simple as that. Cool. Thank you so much. Yeah, this is great. **Swyx** [58:44] Yeah, thank you. **Akshat Bubna** [58:45] This is a great update and I can see why you guys have succeeded so much. It is really focused, but also really good execution. **Swyx** [58:54] Thanks. I mean, we have a long way to go. **Akshat Bubna** [58:56] Allright. Thank you. **Swyx** [58:56] Cool. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.