When enterprises build AI agent platforms, selecting the right gateway is critical for routing models, models tools (like MCP servers), and securing traffic. While LiteLLM is widely recognized for its extensive provider catalog and unified OpenAI-compatible API, it often falls short in enterprise evaluations because crucial security and platform controls—such as SSO beyond five users, secret manager integrations, scoped guardrails, and audit logging—are locked behind a proprietary enterprise license. Conversely, agentgateway (an Apache 2.0 open-source project hosted by the Linux Foundation) serves as a robust, general-purpose data plane for HTTP, gRPC, and LLM traffic, providing enterprise-grade security features without embedded license gates. Furthermore, agentgateway boasts a stronger public security record, avoiding the critical vulnerabilities that have plagued LiteLLM. For teams that need a true microservice gateway for AI services rather than just an expansive SDK, agentgateway presents a more secure, open, and scalable architectural choice.

In this episode of Agentic Conversations, we sit down with Ambud, Principal Engineer at Pinterest responsible for general technology efficiency, fresh off delivering a controversial keynote on AI infrastructure optimization at scale. Ambud walks us through his Five Layer Cake framework - a structured approach to driving efficiency across every level of the AI stack, from silicon and hardware procurement to model selection, inference engine design, and governance. We explore how decisions compound across layers to unlock real business growth, and how the wrong choices can lock you into expensive commitments for years.

As Large Language Models evolve, the real challenge isn't just generating text—it's remembering context, maintaining state, and managing historical data effectively. How do you provide LLMs with persistent, long-term memory without overwhelming context limits or skyrocketing latency and costs? ​Whether you're building stateful AI agents, implementing advanced Retrieval-Augmented Generation (RAG), or managing enterprise-grade vector and relational data, this meetup covers the practical architectures and trade-offs behind modern LLM memory systems.