Truly Serverless Infra for AI Engineers - with Erik Bernhardsson of Modal
Feb 19, 2024 · 1:08:58
Erik Bernhardsson, creator of Annoy and Luigi, founded Modal to build a 'postmodern data stack' with a self-revisioning runtime that eliminates container cold starts and lets developers attach GPUs with a Python decorator. His serverless platform can fan out to thousands of GPUs within seconds, becoming a natural fit for GenAI workloads like Stable Diffusion and fine-tuning (e.g., RAMP fine-tuning 100 models in parallel). Modal differentiates from Replicate by targeting custom models and workflows, and from Modular by offering a managed cloud service rather than licensed software. Erik argues that buying hardware is inefficient for startups and that infrastructure founders must be willing to spend years on load balancing, page faults, and DNS. He shares how his IOI Gold Medal background shapes Modal's talent culture, hiring competitive programmers for complex scheduling and bin-packing problems.
