This blog sketches the AI stack as it’s emerging from real‑world practice and invites the community to refine it. The goal is to create a shared, peer‑validated blueprint for anyone building AI systems, shaped by real experiences, not marketing claims.

When enterprises build AI agent platforms, selecting the right gateway is critical for routing models, models tools (like MCP servers), and securing traffic. While LiteLLM is widely recognized for its extensive provider catalog and unified OpenAI-compatible API, it often falls short in enterprise evaluations because crucial security and platform controls—such as SSO beyond five users, secret manager integrations, scoped guardrails, and audit logging—are locked behind a proprietary enterprise license. Conversely, agentgateway (an Apache 2.0 open-source project hosted by the Linux Foundation) serves as a robust, general-purpose data plane for HTTP, gRPC, and LLM traffic, providing enterprise-grade security features without embedded license gates. Furthermore, agentgateway boasts a stronger public security record, avoiding the critical vulnerabilities that have plagued LiteLLM. For teams that need a true microservice gateway for AI services rather than just an expansive SDK, agentgateway presents a more secure, open, and scalable architectural choice.

In this episode of Agentic Conversations, we sit down with Ambud, Principal Engineer at Pinterest responsible for general technology efficiency, fresh off delivering a controversial keynote on AI infrastructure optimization at scale. Ambud walks us through his Five Layer Cake framework - a structured approach to driving efficiency across every level of the AI stack, from silicon and hardware procurement to model selection, inference engine design, and governance. We explore how decisions compound across layers to unlock real business growth, and how the wrong choices can lock you into expensive commitments for years.