Why every AI Engineer needs an AI Gateway (ft Portkey.ai CEO)
Feb 5, 2025 · 31:18
Rohit Agarwal, CEO of Portkey, explains why every AI engineer needs an AI gateway: an operational platform that connects to LLMs more efficiently, handling routing, observability, guardrails, and cost management. He notes that 90% of production use cases don't use automatic routing, but reasoning models and agents are driving new routing demands. Portkey's open-source gateway minimizes latency with a 21 MB memory footprint using JSON transformers. Key jobs include routing, observability (monitoring price, performance, accuracy), guardrails (e.g., regex-based, empty output detection, PII redaction), and human feedback tied to business metrics like video download rates. Rohit also highlights MCP adoption for simplifying agent-to-service connections, with bidirectional communication via RPC.
