DeepSeek V3, SGLang, and the state of Open Model Inference in 2025 (Quantization, MoEs, Pricing)
Jan 19, 2025 · 57:18
Amir Haghighat and Yineng Zhang from Baseten explain how they serve DeepSeek V3, a 671-billion-parameter MoE model requiring H200 clusters with FP8 support, making it the top open-weights model. They detail Baseten’s dedicated inference model (no shared endpoints) and its reliance on SGLang for performance gains like Radix prefix caching and MLA attention support. Yineng highlights SGLang’s edge over vLLM and TensorRT-LLM in usability and customization, especially for large models. Amir outlines three pillars for mission-critical inference: model-level performance (via frameworks like SGLang), horizontal scaling across regions and clouds, and low-latency multi-model workflows. The episode also covers quantization trends, MoE architecture, and speculative decoding as key enablers for open model deployment.
