# DeepSeek V3, SGLang, and the state of Open Model Inference in 2025 (Quantization, MoEs, Pricing) Latent Space · 2025-01-19 Amir Haghighat and Yineng Zhang from Baseten explain how they serve DeepSeek V3, a 671-billion-parameter MoE model requiring H200 clusters with FP8 support, making it the top open-weights model. They detail Baseten’s dedicated inference model (no shared endpoints) and its reliance on SGLang for performance gains like Radix prefix caching and MLA attention support. Yineng highlights SGLang’s edge over vLLM and TensorRT-LLM in usability and customization, especially for large models. Amir outlines three pillars for mission-critical inference: model-level performance (via frameworks like SGLang), horizontal scaling across regions and clouds, and low-latency multi-model workflows. The episode also covers quantization trends, MoE architecture, and speculative decoding as key enablers for open model deployment. ## Questions this episode answers ### Why does DeepSeek V3 require special hardware like H200s to serve, and what are the specific memory requirements? Yineng Zhang explains that DeepSeek V3 is a 671-billion-parameter MoE model. Even using FP8 precision, the weights alone need 671 GB of memory, and additional memory is needed for the KV cache. Standard H100 GPUs (640 GB total for eight cards) cannot fit the model, so Baseten uses eight H200 GPUs (each with 141 GB, totaling over 1,100 GB) or multi-node H100 setups to run inference. [2:21](https://addtry.com/3ab0438a-7ad8-4725-af03-5d2716c6746d?t=141000) ### What are the key techniques in SGLang that make it faster than vLLM for LLM inference? Yineng Zhang highlights RadixAttention, SGLang's prefix caching technique using block size 1, which increases cache hit rates compared to vLLM's block size 32. SGLang also implements constrained decoding with jump forward acceleration using X Grammar, allowing it to skip tokens when the output must follow a schema, and a zero-overhead CPU scheduler. These optimizations contributed to a reported 3x throughput improvement over vLLM at the time of its benchmark blog post. [36:02](https://addtry.com/3ab0438a-7ad8-4725-af03-5d2716c6746d?t=2162000) ### What does Amir Haghighat identify as the three pillars for running mission-critical inference workloads in production? Amir Haghighat outlines three necessary pillars: first, performance at the model level, which involves choosing the right serving framework and techniques like speculative decoding. Second, horizontal scaling infrastructure that goes beyond Kubernetes to replicate models across multiple regions and clouds to avoid capacity limits and ensure low latency. Third, developer experience for building multi-step, multi-model workflows with low latency, such as Truss Chains, which enable models to communicate directly without network overhead. [28:36](https://addtry.com/3ab0438a-7ad8-4725-af03-5d2716c6746d?t=1716000) ## Key moments - **[0:00] Intro** - **[1:22] DeepSeek V3** - [1:22] DeepSeek V3, a 671B MoE model, is the leading open-source LLM on benchmarks and ChatArena, says Yineng Zhang. - [2:21] DeepSeek V3 cannot be served on a single H100 node; H200 or multi‑node setups are required due to 671 GB FP8 weights. - [3:16] DeepSeek V3 released FP8 weights, requiring custom kernel support that TensorRT‑LLM lacks by default. - [5:44] Q: Why are customers migrating from Claude to DeepSeek V3? A: Rate limits, cost, latency, and need for full model control, says Amir Haghighat. - [7:01] Baseten customers specify latency, throughput, and cost requirements, not particular GPU SKUs, explains Amir Haghighat. - **[8:00] FP8 Training** - [8:43] Baseten does not offer shared inference endpoints; customers get dedicated resources to avoid noisy neighbor problems. - [10:23] "Our customers would not be okay with us touching the weights," says Amir Haghighat on Baseten’s quantization approach. - [11:10] DeepSeek is the first team to train a large model natively with FP8 mixed precision, says Yineng Zhang. - **[13:10] MoE & Pricing** - [13:31] Yineng Zhang predicts MoE inference optimization will be essential in 2025 as fine‑grained MoE architectures gain adoption. - [14:52] Llama attempted to train an MoE model but failed, which is why it released only dense models, explains Yineng Zhang. - [15:41] Baseten charges based on resource consumption, not per token, because customers run custom models on dedicated infrastructure. - **[18:09] Serving Infrastructure** - [18:23] Truss, Baseten’s open‑source model packaging library, supports TensorRT‑LLM and SGLang, offering flexibility for custom inference engines. - [22:40] To enable sub‑400 ms AI phone calls, Baseten created Truss Chains for low‑latency multi‑model workflows that stream data between models. - **[26:30] Frameworks** - [26:56] Yineng Zhang states SGLang’s throughput is up to 3× higher than vLLM for common workloads, while its usability surpasses TensorRT‑LLM. - **[32:39] SGLang Techniques** - [32:57] SGLang began as an LLM programming front‑end in 2023 and became a full inference engine by July 2024, achieving 3× vLLM throughput. - [35:59] SGLang’s Radix attention uses block size 1 for prefix caching, achieving higher KV cache hit rates than block size 32 approaches. - [37:33] Baseten built KV‑cache‑aware load balancing atop Radix attention to route requests to replicas with relevant prefixes, cutting latency. - [38:33] SGLang maps structured output schemas to finite state machines, enabling jump‑forward decoding that skips forced token sequences. - [40:34] Q: Which structured‑output engine is better? A: X Grammar outperforms Outlines and is now the backend for TensorRT‑LLM’s constraint decoding. - [42:50] X.ai uses SGLang because its creators, Lianmin and Yin, are members of X.ai’s technical staff, confirms Yineng Zhang. - **[46:44] Fine-tuning Future** - [48:02] Despite reasoning models, Amir Haghighat sees no decline in demand for traditional fine‑tuning, especially in specialized domains like healthcare. - **[50:02] Inference Pillars** - [50:02] Cursor’s team reached out to Baseten to discuss their DeepSeek V3 implementation, signaling developer demand for fast coding inference. - [50:58] Amir Haghighat outlines three pillars of mission‑critical inference: model‑level performance, horizontal scaling across clouds, and DevEx for workflows. - **[55:34] Outro** - [55:54] Baseten sponsored the first Latent Space hackathon with 100 croissants, an event Alessio calls a turning point for the AI community. ## Speakers - **Alessio** (host) - **Swyx** (host) - **Amir Haghighat** (guest) - **Yineng Zhang** (guest) ## Topics Inference, Language Models, Cloud ## Mentioned Baseten (company), Blend AI (company), Fireworks (company), OpenAI (company), Replicate (company), Together (company), Claude (product), Cursor (product), DeepSeek (product), Gemini (product), H100 (product), H200 (product), Llama (product), Outlines (product), SGLang (product), TensorRT-LLM (product), Truss (product), Whisper (product), X Grammar (product), vLLM (product) ## Transcript ### Intro **Alessio** [0:05] Hey, everyone. Welcome back to the Latent Space Podcast, our first recording of 2025. I'm Alessio, partner and CTO at Decibel Partners, and I'm joined by my co-host swyx, founder of Smol AI. **Swyx** [0:15] Hey, and today we are here with a special double guest episode with Amir. Oh, my God, I don't know your last, Haghighat? **Amir Haghighat** [0:24] That's close enough. Yeah, that is good. That was first try ago, that's really good. **Swyx** [0:30] And Yineng Zhang, uh, from Baseten. Welcome. **Amir Haghighat** [0:32] Thank you. **Yineng Zhang** [0:33] Thank you. **Swyx** [0:33] Amir, we've met before. You're, uh, co-founder of Baseten, which is, uh, you know, one of the leading sort of LLM inference platforms. I, I don't know how you-- so what do you consider yourself? **Amir Haghighat** [0:42] That sounds fine. **Swyx** [0:43] And, Yineng, you are lead software engineer on the model performance team, and you guys recently shipped DeepSeek V3 as one of the many models that you do host. You also are very involved in SGLang, and that, that was actually one of the reasons we're discussing an episode with you even before DeepSeek V3 dropped as a Christmas present to everybody. So w-we can take this a number of directions, but I think one thing we wanted to just get off the bat on was to start with, uh, DeepSeek and, and maybe-- and then we'll work our way backwards back to SGLang. But DeepSeek is, you know, more-- kinda more recent. Why are people so interested? What's the history of, like, I guess, like, DeepSeek in general from, from your perspective? **Yineng Zhang** [1:22] Yeah. Because, uh, DeepSeek V3, I think is currently considered the, the leading open source LLMs based on the benchmark results and the chat area results, and it, it's so big, you know. It, it, it's 671 billion parameter MoE, and I, I, I think it's a game changer for the open source AI, so everyone is interested in this model. ### DeepSeek V3 **Yineng Zhang** [1:48] Yeah. **Swyx** [1:49] One of the interesting things is, like, they, they are a bootstrapped, like, uh, you know, very private, small lab. Like, they have a lot, a lot less, fewer resources than, uh, than others. But it also-- it's also interesting, like, that it's just open weights. Like, um, for some reason, the Chinese labs are much better than the American labs at, uh, at sharing open weights, and that's obviously beneficial for Baseten. It is in your incentive to serve these models at, at all times. Like, you know, like, what, what are sort of the unique challenges that you face, like, offering something this, this large? **Yineng Zhang** [2:21] Yeah. I think, uh, because the model is very large, and if, if we use something like, uh, use H100, we cannot serve this model because, you know, even we, we use the H108 eight cards, it should be, uh, 640 gigabytes memory. The DeepSeek V3 model, it has, uh, 671 billion weights. So even use the FP8 precision, you need, I think, uh, 671 gigabytes for the weights, and you, you also need extra memory for the KV cache. So it's not possible to run that on H100. That's why we, we, we chose H, H200 to run that model or use multi-node to run that model. Yeah, it's very challenging. And I, I think another one is that DeepSeek V3, the s- the weights it released was the FP8 precision. So if you want to run it, you, you should, uh, support that kernel because I, I think the default is the BF16 and even larger. But if you want to run the FP8, you need to support the quantization. So I think currently even the TensorRT-LLM doesn't support the FP8. So if we want to implement that feature, we, we should, uh, do some feature development for that. The last challenging part is that, yeah, if you want to do some debugging or, or some performance benchmarking, it's very hard. Why? Because, uh, the model is so large, and the loading time is so long, yeah. So it make it more complicated for the developer to do some debug. **Swyx** [4:05] Is it complicated or just slow? I mean, you, you've just only mentioned loading time, but, like... **Yineng Zhang** [4:10] Yeah, loading time is slow. Yeah, you are right. **Swyx** [4:12] It's not complicated. It's just, you know, take-- just go for more coffee. **Yineng Zhang** [4:14] Oh, okay, okay. **Swyx** [4:15] Okay. **Alessio** [4:17] Uh, can you maybe just give people a quick rundown of all the models you support on Baseten, how it compares just on size, just to, you know, people hear 671 gigabytes, but, like, is that a lot more than other models? And then you mentioned the BF16, FP8. What's kinda like the usual that you see, and do you see any variation based on model size or anything like that? **Yineng Zhang** [4:39] I think at Baseten, something like Llama 70B is more common. I think Llama 3, uh, has released the 405 billion weights, but I think there are just a few users use that. So before the DeepSeek V3, I think we haven't encountered that issue for that so large weights. I think Deep- DeepSeek V3 is the first one so big model that we should, uh, use H200 or use H100 multi nodes. **Alessio** [5:12] And was that because of performance, or why did people not use the 4o 5B Llama? I, I think the, you know, what I hear from people is like, you know, the performance gains of, like, the 4o 5B at inference times are, like, not worth it, you know. So the 70B is kind of the sweet spot. Who are the people that use V3? Are people that were using maybe the Llama 70B model and just want better performance? Are people that are just experimenting? I think that's kind of the question that people always have. There's always a lot of excitement around open source models, but then maybe the question is like, what are they really good for? **Amir Haghighat** [5:44] I can answer this, uh, just ba- uh, observationally. The interest that we have seen, and some of this is running in production, some of it is just at the interest level, is generally not coming from folks who are trying to upgrade from a certain open source model to DeepSeek V3. It's, uh, we're seeing it, generally speaking, from folks who are coming from Claude And are doing so either because-- and here's-- I'm gonna give you a list of reasons, and generally the reasons are, are a combination, a, a certain combination of these in no particular order. Either it is they're being rate limited, or the price is too high, or they have certain, uh, latency requirements or time to first token requirements for their use case that Claude cannot hit. Or they want to have full control over the model, uh, as opposed to, uh, running it behind an API where the model underneath them can potentially change. And a couple other reasons, but generally it's, it's a, a combination of those. **Alessio** [6:46] You mentioned the speed and some of these things. Do customers wanna change the hardware also? Like, you're referring the H200 as kinda like the default thing. Like, do people come to you and it's like, "Hey, I'd rather use a smaller system and can I-- worse performance?" Or how do you work with customers on, on that? **Amir Haghighat** [7:01] Generally, people come with certain requirements around the latency, the throughput, the cost. Generally, they're not coming in saying, "I want this particular GPU SKU." At least like as, as, as we go up market and we're talking to, you know, foundation model companies for, for instance, the things that are top of mind for them are those, are those requirements, not a, not a particular GPU SKU. We're doing the different, uh, GPU SKUs, not because we wanna, you know, offer, "Oh, look, we have H200s. Look at us. We have Meg to the H100s. Look at us." It's not that. It's really because those are the tools to achieve, uh, a certain kind of time to first token for certain types of models or, or certain kind of throughput and, and scale, or a certain kind of price per million token or what have you, per million images, uh, depending on the modality. And that's the reason that, you know, we're, we're talking GPU SKUs. **Swyx** [8:00] I wanted to pick up a little bit on this FP8 thing. It seems like, you know, I think Noam Shazeer started talking about, um, sort of training natively quantized, and I think that's what DeepSeek seems to have done in-- as at least they said in their paper. ### FP8 Training **Swyx** [8:14] Is this a trend? Like, uh, is, is the community settling on one form, one sort of numeric that, that everyone knows about? Tell us more about what you're seeing here in, in terms of like the training trends, the, those sort of model trends and the, I guess... So I, I think a lot of, uh, companies as well, like, like Together, they'll also release like quantized versions of the Llama models, right? Like for like turbo or light inference or, you know, like just based on different levels of speed. Like, do you do anything there in terms of quantizing the models that you serve? **Amir Haghighat** [8:43] I'll let Yineng answer the, the, the sort of patterns around, you know, uh, using FP8 in training. But I wanna draw one, one distinction that, that I think gets to the, the, the latter part of your question, Shawn, which is that unlike companies, uh, like, you know, you mentioned Together or, you know, Fireworks or Replicate, Baseten d-doesn't provide a shared inference endpoint for the popular open source models. That is a product that, that we don't have on purpose. Those w-work really well, um, the shared open source, shared in-inference endpoints for open source models work really well for situations where, uh, the user is saying, "Hey, let me just call a certain popular model behind an API, pay by the token." That is not our average customer or the, the, the median customer. Our customers generally have their own custom models, very custom workflows, strict requirements around latency and, you know, time to first token, and can deal with noisy neighbor problems that like, oh, the, the API is slow because some other customer has been calling it a lot. Other requirements around infrastructure flexibility, around, a-around regions, uh, for, for latency reasons, for compliance reasons. That's the side of the inference market that, that we capture. And at Baseten, when you deploy a model, whether it's, uh, your own custom weights or, or, or an open source model, you get dedicated inference, dedicated resources, dedicated inference. And so when it comes to the quantization question, uh, where it matters is that we would never quantize the model behind, you know, the user's back and, oh, say, "Look at us, there's a faster Llama, uh, 70B that has been, you know, uh, somehow quantized to, to be faster and cheaper." Our customers are coming to us with those requirements that I mentioned, but in particular, when it comes to model quality, they have strict requirements. They would not be okay with us touching the weights, if you will. Uh, you know, we have done things like speculative decoding with a couple different ways, but, but all of those things, those methods guarantee the output is unchanged, uh, as opposed to quantization. So when it comes to quantization, we have built tooling that allows our users to quantize their models for the ones that we're working more hands-on, uh, through our forward deploy engineering team. We're working with them on, on evals as well to ensure that the quantized models are meeting their, uh, their requirements. However, this is all very much in conjunction with the engineers that are customers, as opposed to us doing it behind the scenes. **Yineng Zhang** [11:10] Oh, yeah, FP8 training is very interesting, and I think the DeepSeek team is the first one to use FP8 training for the large model. I think before that, maybe LingYi.ai... Uh, uh, sorry, zeroone.ai. So they used the- **Amir Haghighat** [11:29] LingYi. Yeah **Yineng Zhang** [11:29] ... E-Lightning. They used the F, uh, FP8 training. And the others, I, I think most of them used the BF16 training, and yeah, i-i-it's the game changer. And for us, because, uh, the FP8 kernel should be implement inference, it use the block-wise FP8. And currently, even you use something like, uh, CUDA or cuBLAS, you cannot support that. So you usually use something like a Triton to implement the, the kernel, or you usually use something like a Catalyst to implement that kernel. So I, I think that's the challenging part. **Swyx** [12:05] My theory is that the-- this will pick up in terms of the models that people release. Like, increasingly it'll not be F-- uh, uh, BF16. There was a bit of quantization. I was trying to look for the paper while you were speaking, but I couldn't find it. But there's a, there's sort of this like ablation of quantizations- Paper that was out last year that showed that there is benefits to quantizing and sort of natively training all the way until like, like six bit, and then, like even smaller than that m-might be going too far. Yeah, I'm not sure if you, you know what paper I'm talking about, but yeah, there's, there's an interesting trend for sure. **Yineng Zhang** [12:38] Yeah. I, I, I think even they use the FP8 quantization, the benchmark result is very good, such as something like QSM 8K, uh, the score is nearly ninety-four point six. It's so high, you know. I, I think it, it's higher than every other open source LLM, even the Llama 400, zero five billion. **Swyx** [13:00] So I'm gonna move on a little bit in terms of like one other notable detail, and then, uh, you know, we don't have to, uh, speak too much about DeepSeek, 'cause obviously we don't know, uh, that much unless, unless you work on the team. ### MoE & Pricing **Swyx** [13:10] But like, uh, you know, another trend that they have is the fine grain MoE. I think that there is this question about whether or not MoEs will be more of a thing. Basically, like, like this time last year, Mixed Draw was sort of kicking off a bit of a MoE trend with, uh, you know, eight by seventy, eight by twenty-two B. And then the rest of the year, not-- no MoEs, basically. So is this like discovery of fine grain MoE is going to be a relevant trend for this year? **Yineng Zhang** [13:41] Yeah, I, I, I think so because, uh, as far as I know, some companies such as Baidu or Baidu Dance, their internal, the dominant LLM, they use the MoE architecture, and their weights, I think, is similar to the DeepSeek MoE model. So I, I think, uh, at this n-new year, the MoE inference optimization will be very essential and, yeah, yeah, so important. **Swyx** [14:09] The, the-- and same time, like why hasn't, you know, the, the big labs done-- why haven't the big labs done it, right? Like I, I think, well, Llama 405B is dense. I think Groq is also a dense model, but you can correct me if I'm wrong. And yeah, it's, it's just a weird countertrend, I think. This time last year, I was like kind of writing my recap, and I was like, "All right, MoEs like seem like they're going to be trending," and then they did not trend. Anyway, so it's just, just a note that I would, uh, flag out there, but I generally agree, like it seems like fine grain MoE, um, is working out and, uh, I would definitely see-- definitely want to see more people adopting it. I went to Jeff Dean's session at NeurIPS, uh, and he also mentioned as well that I think one of the Gemini, I think Gemini one point five Pro is an MoE, which I don't think we knew before that. **Yineng Zhang** [14:52] Yeah. Yeah. So the reason why Llama, uh, open sourced, uh, the MoE model, because I, I think they, they tried to train our MoE model, but they failed. So that, that's why they didn't open source, uh, MoE model for Llama series. **Swyx** [15:07] What are the causes of failure? Like why, why do MoEs fail? Like I think this is a-another thing that people are talking about, right? Like the, the failures of 3.5 Opus, the failures of DeepSeek Five, like, you know, it's a, it's a thing that people are sort of rumoring. **Yineng Zhang** [15:18] Yeah, because if you want to train model, for the training staff, you need to some, some benchmark or some, yeah, score. But for the MoE model, the benchmark score is even lower than the dense model. So in that, in that way or in that case, they think of MoE model is worse than the dense model, so they didn't release that MoE model. **Swyx** [15:41] Okay. Well then one more thing, I guess, maybe more commercially relevant, um, DeepSeek's API pricing is very competitive. How do you decide pricing in this kind of landscape with open models? **Amir Haghighat** [15:52] Yeah. So it goes back to the, the, the use cases that, that we serve. Um, and again, going back to the fact that, you know, we don't have shared inference endpoints for the different models, and so our pricing is never, uh, per token. Customers, like I said, generally come with, with their own custom models or open source models, but with strict requirements around certain things, latency and time to first token requirements or, or security and compliance requirements or, you know, particular scale that they're looking for without running into noisy neighbor problems, things like that. And the way that, that we price has always been, uh, based on consumption, based on consumption of resources, and that takes one of two shapes. One is the shape where things are running inside of our infrastructure, and we're running, by the way, on top of multiple different public clouds and many different regions within those. Then we charge them based on the, the hardware that-- the resources that they're using. The second shape that it takes is that our customer brings their own cloud, and we're seeing this more and more where a customer has big committed resources inside of their AWS, VPC or, or GCP or, or, or what have you. And in that world, we also have a consumption model. Of course, the price is very different because they're using their own resources, but, uh, we are managing those resources for them. An example of that that, you know, we're, we're seeing more, more recently is the fact that we've had to build multi-cloud capabilities for us so that we can have a single model horizontally replicate across different regions and even different clouds. And more and more as we go up market, our customers have their own cloud commits. They are also multi-cloud in order to be able to get good prices and, and good capacity. And it's unreasonable to expect every one of them to build the same multi-cloud capabilities that we have built. Uh, and so then they take advantage of, of what we have built and, uh, use all of the different cloud resources that they have in a-- as a whole, holistic unit, uh, and have their models at inference time horizontally scale across those and even optionally overflow to our cloud when they start running out of committed resources. Um, all of that has a consumption pricing model to it. **Yineng Zhang** [18:09] Can we talk about what it takes to actually run your service? So we asked this-- we had episodes with, you know, Replicate, Model, FireWorks. We always like to ask this question. Obviously, since you're not the model maker, all the secret sauce is in how you actually run the model. ### Serving Infrastructure **Yineng Zhang** [18:23] I know you also have Truss, which is your more developer-led SDK. Can you maybe quickly run people through how do you go from taking the DeepSeek V3 weights to like actually run it, what goes on behind the scenes, and then we can talk about SGLang, um, in depth a little more. **Amir Haghighat** [18:39] Yeah, totally. So, uh, we have, like you said, we have Truss, which is our open source model packaging and, and deployment library. Truss works with, uh, different frameworks underneath it. It has very native and deep support for TensorRT-LLM. Somewhat as an accident of history, we, we happen to have access to TRT-LLM before it was announced, contributed back to it, um, and, and we still do pushed it to its limits and had to go beyond it in certain areas as well. So for example, if you know the, the, the Triton inference server, we've had to build our own version of that for, for performance and reliability reasons. But we invested in it heavily because, uh, it, it tended to be... For the use cases that we were seeing from our customers, it tended to be the best framework to handle the, the, the, the latency and throughput requirements that we were seeing. In particular, when it comes to the kernels that they come with, I'm, I'm yet to see folks do better than what NVIDIA can do, uh, when it comes to CUDA kernels. However, Truss is not tied to TensorRT-LLM. For example, for the DeepSeek example that, that you mentioned, it's working with SGLang, which is really cool to see and, and we will be investing more and more on SGLang, especially as the developer experience is just so much better than, than TensorRT-LLM. We've built a lot around TensorRT-LLM, productized them too, to make it easier to work with, but still SGLang has been a joy, uh, to work with. Another trend that is really promising, and I learned this from the SGLang folks, is that the TensorRT-LLM folks have promised to modularize a lot of TRT-LLM so that other frameworks li-like SGLang can grab certain parts of it and build on top of it. And so as a user, you don't have to go all in on one framework versus another. Uh, you can really pick and choose based on the requirements that, that you have. And that's really been our approach as well. We have customers on, on Baseten that are using TensorRT-LLM, we have ones that are using VLLM, we have a growing number that are using SGLang too. It's not about really tying yourself to one versus another. It's about using the, uh, the best of, of the bunch, depending on the requirements of the customer and for, for their inference workloads. **Alessio** [20:57] How did you think about designing the framework? So, uh, Replicate, also at Cog, uh, which was kinda more tied to, to Docker, what were maybe some of the design decisions that you had, and how do you think that that's changing, especially as the models change and, like, the runtimes change? **Amir Haghighat** [21:12] Yeah, totally. So, so we, you know, started Truss, gosh, like four or five years ago. And, uh, at the time, the, the sort of principle that we had in mind, uh, was, let's make sure that easy things are easy, but hard things are possible. And so y- an example of easy things being easy is that, you know, think of it as a very simple... You know, you have a model, what, what do you need to do to serve it? Well, you need to load it up, and then you need to write the code for the, for the inference path. And, you know, Truss actually had, you know, hooks for these two things. And, and so you could just, you know, write two functions and, and voila, your, your model was, was, was being served at least as a single unit. We can talk about the horizontal scaling part, uh, separately. That's a whole different topic. A-and so we did well when it comes to easy things being easy. I think we struggled with, with the hard things being possible in the early days. Hard things... Example of hard things are, are cases where we're seeing where more and more our customers have their own custom models, custom models that, uh, sometimes they've, they've fine-tuned, sometimes they've pre-trained. You know, we, we now have six or seven foundation model companies as customers who are sophisticated enough to pre-train their, their, their own models, and they're trusting us with the inference layer. That's not a situation of, "Hey, here's two functions, good luck." Have to have much deeper integrations with them. And, and so that's where we started rethinking some of the abstractions of, of Truss over time to allow for, uh, for those custom use cases, and, and, and that has been successful. Another place where, where we didn't think about at first, but became important, was seeing more and more use cases where the customer was saying, "I can serve my models on Baseten using Truss fine, but my use case is not just call the model, get the response, and run with it. I actually have a multi-step inference workload." So, uh, an example of that is the company Blend AI with, uh, their AI phone calls. To make an AI phone call happen, you need to transcribe what the human said, a couple of LLM calls to figure out what to say back, and then text to speech to, to actually have the, the end-to-end, uh, workflow working. Uh, now you can have these, these three separate models, three separate deployments, but think about what happens is that you have to call the first model, wait for the response, call the second model, get... Wait for the response. All of that network back and forth is killing you. The latency is, is ki- uh, is becoming too high. That's not something that, that we had designed for initially. Uh, and so that's when we came out with Truss Chains, which is the DevX for building these multi-step, multi-model inference workloads, but doing so in a very low latency way, so that instead of you orchestrating all of these calls and, and in-incuring all of that network latency, uh, you're actually making one call, uh, and these models are actually talking to each other. They run independently on their own hardware, on their-- with their own auto-scaling behavior, but the data from one to the next step is being actually streamed. And that way, going back to the AI phone call use case, you can get sub four hundred millisecond latency AI phone calls that actually feel very realistic. **Alessio** [24:16] And those are all models hosted on Baseten, or do you also do chains? **Amir Haghighat** [24:20] Those happen-- Those have to be models hosted on Baseten. If one of those steps is not hosted on Baseten, then- **Alessio** [24:26] Right. Yeah, yeah **Amir Haghighat** [24:27] ... you're still incurring a massive latency on, on the network side. **Alessio** [24:30] Yeah, and then just to maybe tie this into SGLang, how do you kinda think about the hidden magic, you know? Should people know that you use SGLang? Like, should people care? Like, especially for the people building the models, you know. Like, does it matter to them they use a, a certain model runtime, or do they not care, everything just goes through the Baseten platform the same? **Amir Haghighat** [24:50] Yeah. Should we talk about it? Yes, hundred percent. We want to be the transparent provider. I don't want to say, "Oh, just give us your model, and voila, magic, and trust our magic." Uh, I, I want that magic to be, uh, very transparent, uh, to our customers. That has worked really well for us. And you really need that, especially when you're onboarding, you know, foundation model companies. And, uh, you know, they're, they're not gonna, you know, just turn a blind eye on how things are run, uh, underneath the hood. When it comes to customers caring about what's happening underneath, they do, but more than caring about this framework versus that, they care about how the final output. In other words, is the quality the same or somehow something has changed underneath the hood and the model isn't actually producing the same quality? Uh, how is the latency? And especially for certain use cases, what is the time to first token, and is that sustained? What is the P95 of that? What is the P99 of that? How well does it handle throughput? When you start getting a massive burst of traffic, uh, does it still sustain those P95s and P99s? How do I make sure that the security of, of the data being sent, uh, into the model is, is guaranteed? How do I make sure compliance is guaranteed for HIPAA use cases? And how do I make sure that the data rema-remains within a certain geo for compliance reasons or for latency reasons? And so those are the things that-- those are the concerns that the customers are coming to us with. Less so about, "Hey, here's my model. Make sure you run it with TRTLM or make sure we run with SGLang." ### Frameworks **Alessio** [26:30] Yep. Can you maybe give us an overview of all the different frameworks that people might use? So you have SGLang, TRTLM, VLLM. Those are kinda like maybe the open source research ones, and then some of the other commercial companies are building the, some of their own stuff. But what's the state-of-the-art today? Maybe like the top three most popular, and then we can talk about why SGLang came to be and what makes it different and some of the performance boosts that you get. **Yineng Zhang** [26:56] Okay. Yeah. I think for the common use case, maybe not, not the DeepSeek V3. For the common use case, I think SGLang's performance is better than Theom, and its usability is better than TensorRT-LLM. So when user care about the performance and the usability or, or the... I think they will choose SGLang. And for the DeepSeek V3 case, because we do a lot of optimization in SGLang, something like, uh, DeepSeek v2, they proposed attention variant named MLA, Multi-Latent Attention. And I think SGLang is the only framework support that. Maybe LLaMA and TRTLM al-also support, but Theom doesn't support. And also, in DeepSeek-- Oh, sorry. In the SGLang version 0.4, we also support the DP attention for DeepSeek. And in the latest, uh, SGLang release, we also support the block-wise FP8 kernel, and that kernel was adopted and copied by Theom later. So I, I think we have done a lot of, uh, optimization for DeepSeek. That's why SGLang is the recommended, uh, uh, LLM engine by the DeepSeek team. **Amir Haghighat** [28:12] And maybe one thing to point out, and I think this is, this is important, is that the framework that you choose is part of the equation for running mission-critical inference workloads, but it's only a part of it. So may-maybe I can draw this out just based on the experience, uh, based on what I've seen in the market, uh, as to what it takes to run mission-critical inference workloads in production. I think it takes three things, and each of them individually is, is necessary but not sufficient. Uh, one is performance at the model level. Uh, so, uh, in this case, how fast are you running this one model running on a single GPU, let's say? The framework that you use there can, can matter. The techniques that you use there can matter. The MLA technique, for example, that, that Yineng mentioned, or the, the CUDA kernels that, that are being used. But there's also techniques, uh, being used at a higher level, things like, uh, speculative decoding with draft models or with Medusa heads. And, and these are implemented in the different frameworks, uh, or you can even implement it yourself, but they're not necessarily tied to a single framework. But using speculative decoding gives you massive upside, uh, when it comes to being able to handle high, high throughput. But that's not enough. Invariably, that one model running on a single GPU, let's say, is gonna get too much traffic that it cannot handle. And at that point, you need to horizontally scale it. Uh, that's not an ML problem. That's not a PyTorch problem. That's an infrastructure problem. How quickly do you go from a single, uh, replica of that model to five to ten to 100? And so that's the second, that's the second pillar that is necessary for running these mission-critical inference workloads. And what does it take to do that? It takes... Some people are like, "Oh, you just need Kubernetes, and Kubernetes has an autoscaler, and, and that just works." That doesn't work for, for these kinds of mission-critical inference workloads, and you end up catching yourself wanting to bit by bit rebuild those infrastructure pieces from scratch. This has been our experience. And then going, uh, even a layer beyond that, Kubernetes runs i-in a single cluster. It's, it's, it's a single cluster, and it's a single region, tied to a single region. And when it comes to inference workloads and needing GPUs, more and more, you know, we're seeing this, that you cannot meet the demand inside of a single region, a single cloud's single region. In other words, a single model might want to horizontally scale up to 200 replicas, each of which is, let's say, two H100s or four H100s, or even a full node, and you run into limits, uh, of the capacity inside of that one region. And what we had to build to get around that was the ability to have a single model have replicas across different regions. So, you know, there, there are models on Baseten today that have fifty replicas in GCP East and, uh, eighty replicas in, in, uh, AWS West and, and Oracle in, in London, uh, um, et cetera. And that was a big investment, uh, that, that we had to make. Uh, the final one is wrapping the power of the first two pillars, uh, in a very good developer experience to be able to, uh, afford certain workflows like the ones that I mentioned around, you know, multi-step, you know, multi-model inference workloads. Because more and more we're seeing that the, the, the market is moving towards those, that the needs are, are generally in these sort of more complex workflows. So these are the third-- the, the three pillars that it takes to, to run mission-critical inference workloads. And the choice of the framework, the serving framework, is really a part of the first pillar. And, and that's something that I'm seeing in the market that, like, people who are somewhat new to it, they're like, "Well, VLLM equals, equals production." That's what it takes to run inference workloads, and that's... In practice, that is not true, and I wanted to call that out. **Yineng Zhang** [32:13] I agree with Amir, because I, I think it's open source libraries such as Phi-2 LM, SGLang, LLaMA or TensorRT-LLM. They only provide a library. They don't provide a product, uh, solution. **Alessio** [32:25] Yeah. Can we maybe talk about some of the SGLang unique things? So I've read through the paper. Sounds like, um, some of the main use cases, like when you have very large batches, which makes sense for, for your use case, and also kind of longer, longer context. ### SGLang Techniques **Alessio** [32:39] You know, what was the decision behind creating the framework, which I think is, like, around one year old. I think the paper came out December twenty twenty-three, something like that. So it's still- **Yineng Zhang** [32:48] Yeah. Yeah. Yeah **Alessio** [32:48] ... still fairly new compared to some of the, the other ones. And then maybe what were some things that you had to change, you know, a-as you built it or any, any fun stories? **Yineng Zhang** [32:57] Yeah, yeah, yeah. I think last year... Oh, no, not last year, sorry. Uh, at twenty twenty-three, maybe August. At that time, Lianmin and Ying want to create the SGLang, maybe for the front end, something like LLM, uh, program. They want to solve that problem. And, uh, at twenty twenty-four January, they support something like Radix cache. It's a prefix caching technology. I think SGLang is the first framework that support prefix cache. And at February, they also support constraint decoding and support some jum-jump forward. So at that time, it's, uh, known for the language generator, not the inference backend. And at twenty twenty-four July, or Ju-June or July, we want to make SGLang a fully functionality LLM inference engine. It's just equal with-- equivalent with the Phi-2 LM or with the TensorRT-LLM. So at that time, we published a blog compared with other frameworks, and its performance is, is amazing, yeah. At that time, I, I think, uh, its performance is maybe three times... Its throughput is three times than Phi-2 LM. So after that, Phi-2 LM also do some refactor to make it faster. And at, uh, uh, September and December, we continue to release new versions for SGLang. Yeah, we, we support some, uh, DeepSeek optimization, such as MLA optimization, DPR attention optimization, and we also support the zero overhead CPU schedule. Also, we, we support something like SGLang router for the cache aware load balance. Yeah, we, we deliver so many features. We just build and ship. And I think why Lianmin and Ying want to create a new framework rather than use the existing solutions such as Phi-2 LM or TensorRT, because at that time, you know, at that time, maybe, uh, for the Phi-2 LM, I think, uh, it's easy to use, but its performance, maybe it's not good. Some design, I, I think is not okay. Yeah, maybe the, the code is a little messy, and if you want to extend some new feature on top of that, it's a little hard. And TensorRT-LLM, I, I think it's blazing fast. Its performance is so good, but it's not easy to do some secondary development. If you want to add some new feature, it's a little hard. So they just think about, "Oh, how can we create a new framework? It can achieve the good performance. Also, it's easy to develop, to maintain." So that's why they, they create the SGLang project. **Alessio** [35:43] Let's run through maybe the three main techniques behind SGLang. So the first one is, uh, Radix attention, which focuses on KV cache. And when you think about a model that is, you know, as large as DeepSeek V3 especially, uh, like having better KV cache reutilization is great. Can you just talk a bit about that performance impact? **Yineng Zhang** [36:02] Uh, yeah. Radix cache, I, I think it, it's the technology of the prefix caching, and it is a special case for something like block size is one. You know, for Phi-2 LM or for other frameworks, they use something block size thirty-two, and SGLang use the block size one. I think if you use the block size one, you can make the cache hit rate higher than other frameworks. I think that's the mainly, uh, benefits. **Alessio** [36:30] And for your case specifically, how does that change when you have like a Baseten-type use case where you don't-- do not have a shared endpoint versus like, you know, is this less helpful for GPU clouds to do one model for, like, many people that have, like, very different use cases versus, like, when you have just one endpoint for one customer? I'm sure they have a system prompt that a lot of models share and things like that. Anything you want to mention there? **Amir Haghighat** [36:55] Yeah. We've seen this be, uh, massively helpful in-- for the reason that you mentioned. There is a certain sort of finite number of, uh, prompts or at least prompt prefixes that are being used per customer. And **Swyx** [37:12] What we've seen is that previous caching, uh, and, and the different techniques to make that better has been massively helpful. However, we still had to build on top of that. The example there is that you have a, a model with, you know, dozens of replicas, uh, each of which has its own state of KV cache. A new request comes in, uh, and what we used to do back in the day was that, uh, that request would be randomly assigned to one of these replicas. But the better way to do it is that knowing the state of KV cache in these different replicas, trying to decide, uh, which one it should go to. One of the, um, parameters that you need to consider, there are other parameters to consider around the size of the queue at, at each of the replicas and, uh, the location of each replicas, depending on how geo, where you want to be. But adding that additional consideration around KV cache-aware load balancing, well, was something that we saw improve latency quite a bit for our customers. **Alessio** [38:17] And then the second part, which was maybe the harder, uh, one to, to understand, uh, as a practitioner, which is the, this idea of, like, turning some of the, the coding process into a finite state machine instead of a more open-ended when you're using, especially with structure outputs. Can you maybe explain what that means? And I would love to learn, too, so, uh, maybe this is, uh, an opportunity for everybody to better understand how you think about going from a normal kind of like token-by-token decoding to having a more, I wouldn't say pre-compiled, but like pre-understanding of what the paths are gonna be. **Yineng Zhang** [38:51] I think SGLang support constituted decoding, and it also support the jump forward, and we use something like outline or the X grammar to do the, something like change the comfort of the schema fr-from, from JSON to the, uh, fi- FSM, the state machine, and we can use the state machine to control the output. Something like, uh, the output may be to the JSON mode or something like... Yeah, it, it should be, obey some rule. So in that case, because the output should, should, uh, obey some rule, so you can skip some token, something like, uh, you, you should decode four times, but you should obey the rule so... Or you can get that token in advance. You can just use one preview to replace the, uh, four decoding, yeah, for example. So that's why you can jump forward. **Alessio** [39:45] I guess the question is, like, why doesn't everybody do that? When I was reading, I was like, "This just sounds better," especially both for accuracy, you know, you're kind of constraining for structure output as well. You can do faster decoding. Are there downsides to it as well? **Yineng Zhang** [39:58] I think maintain jump forward is a little hard, yeah. At later, we, we support something like CPU overlap. I think in the overlap model, we even make it compatible with the jump forwarding, because just if you want to maintain the jump forward with other features, it will be more complicated. So I think we, we only use it default setting. We disable it by default, but if you want to enable it, you can just, yeah, use some arguments to enable that. But it, it's a little hard to maintain, especially compatible with other optimization features. **Swyx** [40:34] Just as a side note, you mentioned X Grammar, which I had never heard of, uh, about, and I looked up the GitHub repo. It's actually from MLC, which, uh, we talked to TQ- **Yineng Zhang** [40:42] Yeah, yeah, yeah **Swyx** [40:43] ... uh, I think a while ago. Uh, any comparisons between X Grammar and Outlines? Is there trends in these, in this world, or is it mostly settled science? **Yineng Zhang** [40:52] To be honest, I, I prefer or X Grammar. You know, uh- **Swyx** [40:56] Okay. Yeah, tell us more. **Yineng Zhang** [40:57] Yeah. M- MLC AI is funded by, by Tian, Tianqi. Both Tianqi and X Grammar's other, Yuxin Dong, they were the, uh, students graduated from Shanghai Jiao Tong University, and the creator of SGLang, Lian Min and Yin, they also graduated from Sh- Shanghai Jiao Tong University. **Swyx** [41:18] Oh, my God. Is it the Berkeley of China? **Yineng Zhang** [41:20] Yeah, you're right. And I, I think X Grammar's performance is better than the Outlines, and also in the TensorRT LLM, the latest release, TensorRT LLM also integrate X Grammar as the backend for the constructed decoding. **Swyx** [41:39] Okay. This is new to us. We had, uh, Remy from Outlines, uh, speak at my past conference, but, uh, I did not, uh, even- **Yineng Zhang** [41:46] Yeah **Swyx** [41:46] ... I wasn't even aware of X Grammar being, uh, being a thing. But yeah, I mean, structured output is something that a lot of people care about. We had, uh, OpenAI talked about their structured output implement- implementation, and there's a lot of interest in, in making sure that there are no trade-offs. I think there's a little bit of FUD, uh, around how maybe the models are dumber when you use structured output instead of the sort of base, sort of next token generation. But I don't think it's significant, uh, that much. **Yineng Zhang** [42:11] Yeah. **Swyx** [42:12] We can talk about, uh, the last one, uh, which I don't know if it's, uh, as relevant for Baseten, which is the third technique of SGLang, which is API speculative execution, which seems to be only for API-only models. **Yineng Zhang** [42:23] Oh, yeah. I, I think it's the front-end feature. It's not the backend. Yeah, something like you have some control flow for the LLM task. Yeah. Such as, uh, you have the, uh, one request to get a result, and you, uh, just continue to another call. And for, for this case, you can use the SGLang front-end language to describe the control flow. It, it will make it easier to control that pattern. **Swyx** [42:50] Okay. Awesome. Tracing this human path, I- I'm pretty sure I know the answer, but is there a reason, like, big projects like, uh, Groq, you know, like X.ai also use SGLang? **Yineng Zhang** [43:00] Yeah, yeah, yeah. You are right. **Swyx** [43:02] Is it, is it just p- the same people? **Yineng Zhang** [43:04] Yeah, yeah, yeah. Right. Lian Min and Yin are the X.ai's, uh, member of technical staff. **Swyx** [43:10] I mean, it makes sense. I, I wonder if it, you know, w- what's the impetus for SGLang to kind of break containment- It seems like VLLM obviously has the, the-- it's one year older, it's, it has more community pull. I wonder how this will shake out. I, I don't really know, but, uh, you know. You said it's a li-- you said VLLM's library of SGLang's much, uh, much more comprehensive. I mean, do people care? Maybe it's, uh, like, when you're serving models at scale, then, then you start really prioritizing the sort of performance that SGLang, SGLang offers. **Yineng Zhang** [43:43] I think i-if you care about the performance, maybe TensorRT-LM is the best solution for now, especially for the latency sensitive scenery, TensorRT-LM doing well. But if you also want to implement, uh, some, some feature by yourself or do some optimization by yourself, you, you want to the-- customize the framework, I think SGLang is a good option. And the VLLM, I, I think, uh, VLLM's community support is very nice because it was used by so many users, and it, it has so many GitHub stars. And, you know, SGLang, when I participated in the SGLang team at July, it has only two thousand stars, and right now it has more than seven stars. Yeah, I, I think it also grows so fast. **Swyx** [44:31] Anything that people should look forward to, uh, that's on the roadmap for SGLang that people should be aware about? **Yineng Zhang** [44:36] Yeah, yeah, yeah. And we, we post the, uh, roadmap in the issue, and we pin that issue. Also, we have biweekly meeting to sync with the community about our progress, our plan, which feature do we want to implement in this quarter, something like that. And we also co-host some meetups, something like, uh, the first meetup we co-host with the MLC LM and FlashInfer, and we also, yeah, participate at the-- some hackathon, something like Camel AI Hackathon. We, we do the, uh, presentation about SGLang. Yeah. **Swyx** [45:12] I just saw it now. It, it sounds like, uh, actually there, there's, you know, you mentioned Eagle, you mentioned Medusa. I think, uh, Amir mentioned Medusa, but Eagle is also part of that cabal, the speculative decoding techniques. Uh, it looks like now you support it. **Yineng Zhang** [45:24] Yeah, we, we already support it. And I, I think in the open source implementation, something like, uh, VLLM, uh, SGLang, and other frameworks, I, I think it's the SOTA performance. And currently, even, uh, use the TensorRT-LM, it only support the Eagle one, not Eagle two. Yeah. **Amir Haghighat** [45:42] And one thing to note about speculative decoding, different versions of it, is that the framework supporting it is one thing, but you will have to do the job of the training of the draft models or the additional heads, and a lot of the benefits will come from how good you are at the training aspect, uh, in terms of the data that you use to train the, the draft model to essentially distill the, the, the target model or mimic its, its behavior so that you can have a very high rate of acceptance. It's, uh, the kind of throughput improvement that you get are ultimately dependent on how good of a job you do at training the, the draft model in the case of the draft target model mechanism. So that's another thing that's like, hey, does the framework support it, or you can just turn on speculative decoding with, uh, with a flag. That's not the case. There's more that, that goes to it. ### Fine-tuning Future **Swyx** [46:44] One more side thing on training. I also noticed that with OpenAI offering fine-tuning for o1 and all these things, I think people are also very interested in sort of RL trainers is what they-- what you have here. Uh, it looks like you're supporting Hugging Face TRL and OpenRL HF. Do you think that this will become something that a lot of people are demanding, like the sort of general feel of RL for LLMs, um, relatively abandons, I think, up till like the end of last year, basically? **Yineng Zhang** [47:13] Yeah. I, I, I think so. **Swyx** [47:15] I don't know. It's like it's one of those things where like maybe people have to wait for a base model that is-- that has some layer looping or some other sort of friendly architecture for reasoning instead of just pure RL on LLMs, because I think so far people have not really exploited RLHF, uh, as much in the wild. Uh, I mean, correct me if I'm wrong. **Amir Haghighat** [47:38] Yeah. So I, I, I can, I can give you some examples of when we've seen it work. Again, this is generally done by our customers before they come to us for inference, but there's examples like in the healthcare world, fine-tuning models for understanding medical jargon, like for Whisper, for instance, you know, a version of Whisper that can actually understand medical jargon. So that's not-- that's non-LLM use case. In the LLM use case, staying in the healthcare space, models that can do medical document extractions and do a very good job compared to even, you know, state-of-the arts, um, because of the data that, uh, the company had gathered, uh, through human in the loop. Are those gonna go away? The need for those are gonna go away because there's a model that can do reasoning and, and do a very good job at it? I don't know. My intuition says yes. Will it be cost effective? That's the question that, that I have. Um, and in the short term, no, in the long term, maybe. But I haven't seen the need for the more traditional fine-tuning actually go down. Uh, in fact, that-- we see that quite a bit right now in the market. The question for us is, hey, do we want to address that market knowing that the entire market might go away one day? And my general answer to that is, let's solve today's problems. Uh, even if they're not around in two years, you will learn a lot, uh, along the way by onboarding customers that have today's problems, and you learn from them about tomorrow's problems, and you will build ahead for it. **Swyx** [49:16] What-- So hang on. Why, why do you think fine-tuning might go away? **Amir Haghighat** [49:19] Because like you said, there's gonna be models with, uh, complex reasoning capabilities that can actually figure it out in a, you know, few shot kind of way without needing a large data set to fine-tune the model with. That's what some people are saying. **Swyx** [49:36] Uh, I really have a trouble believing that, that that'll be the case. Much more believe that it's just easier to change your prompts rather than actually f- do full fine tunes or e-even, uh, param-parameter efficient fine tunes. **Amir Haghighat** [49:51] For sure. **Swyx** [49:51] Is there anything else that we haven't touched on that, you know, you wish the people ask you more about because, you know, it, it is something that is very interesting from, from your point of view of, like, what you're seeing among your community? **Yineng Zhang** [50:02] Uh, yeah. When we released the DeepSeek V3 support, we have some community user, something like, uh, Cursor. Do you know Cursor? I, I think it, it's very popular. ### Inference Pillars **Swyx** [50:13] Of course. Of course. I use them every day. When I type codes- code dot inside of my IDE, uh, my terminal, it actually opens Cursor instead of VS Code. I feel very bad- **Yineng Zhang** [50:23] Yeah, yeah, yeah **Swyx** [50:23] ... for VS Code. **Yineng Zhang** [50:25] And when, when we released the DeepSeek V3 support, uh, some employee from the Cursor team also very interested our implementation and ask, uh, re-reach out and ask some question from us. So I, I think as SGLang grows faster and the, the, the features optimization, we, we iterate, uh, so fast and I, I think there will be more users from different companies, from different teams to, to use it. **Amir Haghighat** [50:55] Honestly, I would go back to what I emphasized earlier, which was that I wish more people asked about what it takes to run mission-critical inference workloads. Because I see this in the market sometimes that they're like, "Well, I can just use vLLM, and that puts my model behind an API and that is, that is production." But really it takes three pillars that all need to be there. One is, uh, performance at the model level. That is where, you know, frameworks that we talked about today really, uh, help you with, but you still have to guide them, like when it comes to, you know, speculative decoding. Yes, they support it, but, you know, who's going to train or, or fine tune the, the draft model or the Medusa heads or who's going to ensure that the, the reliability of, of the vLLM server that's, you know, you see it in production like, you know, there, there are crashes, you know, h-how do you, how do you recover from those without affecting production traffic? But, but by itself, that's not enough, uh, because invariably that one model running on, uh, a set of hardware, uh, is gonna get too much traffic that it cannot handle and at that point you need to horizontally scale it and that's not an ML problem, that's not a PyTorch problem, that is, uh, an infrastructure problem to ensure that you can horizontally scale up your model extremely fast to meet your, you know, P90, P99, uh, latency requirements and to ensure that you're not running out of capacity in a single region that that model lives, you end up having to scale that model across different regions, uh, and across different clouds even to ensure that that model is not being starved of resources in the one place that, that it lives. So that's, that's an area of in-investment that we started investing in, uh, some, some time ago and like really paid off this past year. And the third pillar is enablement of workflows. Workflows such as the sort of AI phone call example that I told you about, the, the ones that require multi-step, multi-model inference, but in a very low latency way. That's the third pillar that really allows the developers to be able to use the power of the first two pillars and then be able to combine them when especially you need multiple models for, for your workflows, and doing so in a, in a reliable way, repeatable way, and a low latency way. And that is... tho-those are the three pillars honestly that, that we have been investing, uh, a lot on. You know, some of which we started investing in three years ago and it really started paying off, you know, a year ago. So it, it, it takes, it takes quite a bit of build to get it to the point where you're truly running folks', uh, customers' mission-critical inference workloads. What do I mean by mission-critical inference workloads? Inference where if inference is slow or down, the, the main product of our customer is slow or down, so they really care about it. They have strict requirements around latency, around being able to support, uh, large throughput, about being able to do so in a way that, you know, other customers' usage doesn't affect the SLAs that they are getting in dealing with noisy neighbor problems. And inference done in a compliance way, whether it's, it's HIPAA or, or, or certain SOC, uh, requirements. And also inference done in a geo aware kind of way, both for compliance reasons and also for latency reasons, uh, where when you-- where you forward the traffic has an impact on latency in situations where, uh, fifty milliseconds really matter, hundred milliseconds really matter and we're seeing more and more of those use cases. **Swyx** [54:41] Well, one way I would recommend doing that is kind of a, sort of a manifesto type of thing and I'm sure you know Heroku's Twelve-Factor App. **Amir Haghighat** [54:48] I've seen that, yes, yes. That's a good idea actually. Yeah. **Swyx** [54:51] Yeah. Maybe even separate, put on a separate property than Baseten and just go like, "Here's what we think, you know, mission-critical AI application should be," and, you know, have some thought leadership there and, and flesh it out and, you know, see if the, the market takes it on as a, as a mission and obviously you will be best prepared to, to serve that market, um, as well. I've also seen this done very well with enterpriseready.io. I think, uh, it used to be done by, I think it was called Gravitational or Replicated- **Alessio** [55:18] Yeah, yeah, Grandma **Swyx** [55:18] ... one of those. **Alessio** [55:19] From, uh, Replicated. **Swyx** [55:20] Yeah, yeah. These kinds of things, when you have a list of requirements when like y-you're like, "Look, everybody needs this." Uh, okay, like write them up and then, s- you know, put a little bit of marketing on it, spit it out from the main company brand, uh, that, that tends to work very well. **Amir Haghighat** [55:33] Yeah. Good idea. ### Outro **Swyx** [55:34] Cool. Well, thanks, thanks so much for your time. I think this is a really good dive into both Baseten and SGLang and a little bit of DeepSeek V3 which, uh, people are very interested in. I'm trying to talk to them as well because obviously they're, uh, they're, they're a f- a fascinating lab and, uh, but, you know, I, I think you guys, you guys are doing a, a lot to, uh, make it accessible for everyone, so thank you so much. **Alessio** [55:54] And as a, just to give Baseten some street cred, you know, uh, they were one of the first sponsors for Latent Space events, and Amir brought a hundred croissants at our Latent Space hackathon in 2023. So, uh- **Amir Haghighat** [56:08] Ah. **Alessio** [56:08] Yeah, just wanna bring that up how we saw, I saw Phil and Ed at AWS re:Invent, uh, and I told them that was one of the, the first events that we really did and one of the turning points of this industry as far as community goes in my mind. You know, everybody, everybody, everybody was there. **Amir Haghighat** [56:24] The croissants? **Alessio** [56:24] No, no, not the croissants. The, the event itself. **Amir Haghighat** [56:27] Yeah, entire companies launched- **Alessio** [56:28] Yeah, you were... I, I mean, you know, like, uh, Nader from Brev was there and did the Jo- with Joseph from Roboflow, they did like the prompt battle thing, like Harrison was a judge and it's like Jerry from Llama Index was there, like kinda like everybody that, uh, is kind of now breaking out. Uh, if you look at the, the graph- **Amir Haghighat** [56:45] Yeah **Alessio** [56:45] ... that, uh, Jensen put on, on the screen at CES with some of the companies they work with, a lot of them were at that event. So yeah, thanks for staying, staying involved with us, Amir, and I'm sure we'll, we'll do more together. Uh, and thank you guys. **Amir Haghighat** [56:59] Many more years to come for sure. **Alessio** [57:01] Thank you for taking the time today. **Swyx** [57:03] Good to see you both. Alessio, Sean --- This library is powered by PodHood (https://podhood.com), the podcast website platform.