The way we handle API traffic is broken for GenAI. We've spent years optimizing for microservices—fast, stateless, and lightweight API calls. But GenAI changes everything. Requests are slower, heavier, and more complex, requiring long-lived connections, massive payloads, and streaming responses. Suddenly, traditional API gateways are struggling—timeout limits are too short, rate limiting models don’t fit, and payload constraints are blocking innovation. In this episode, we unpack the new challenges of GenAI traffic and why infrastructure must evolve—again. We look back at previous API shifts, from the C10K problem to the monolith-to-microservices revolution, and how they reshaped networking. Now, AI-driven workloads demand a new kind of API gateway—one that handles token-based rate limiting, cost-aware request shaping, and scalable AI inference traffic.

Rohit Krishnan, Chief Product Officer at Bodo.AI, joins Demetrios to discuss AI's evolving landscape. They explore interactive reasoning models, AI's impact on jobs, scalability challenges, and the path to AGI. Rohit also shares insights on Bodo.AI’s open-source move and its impact on data science.

Alexa shares her journey into software engineering, from early struggles with Airflow and Kubernetes to leading open-source projects like the Envoy AI Gateway. She and Demetrios discuss AI model deployment, tooling differences across tech roles, and the importance of abstraction. They highlight aligning technical work with business goals and improving cross-team communication, offering key insights into MLOps and AI infrastructure.

Key lessons for founders that are thinking about or starting their companies. 15 years of inception stage investing from how data science companies like Yhat went to market in 2013-14 and how that's evolved, to building companies around OSS frameworks like CrewAI; Eliot share's key learnings and questions for founders starting out.

There seems to be a shift from workflow languages to code - mostly annotation pythons - happening and getting us. It is a symptom of how complex workflow orchestration has gotten. Is it a dominant trend or will we cycle back to “DAG specifications”? At Stitchfix, we had our own DSL that “compiled” into airflow DAGs and at MicroByre, we used a external workflow langauge. Both had a batch task executor on K8s but at MicroByre, we had human and robot in the loop workflows.

In this MLOps Community Podcast episode, Willem Pienaar, CTO of Cleric, breaks down how they built an autonomous AI SRE that helps engineering teams diagnose production issues. We explore how Cleric builds knowledge graphs for system understanding, and uses existing tools/systems during investigations. We also get into some gnarly challenges around memory, tool integration, and evaluation frameworks, and some lessons learned from deploying to engineering teams.

Recent rapid advancements in Artificial Intelligence (AI) have made it widely applicable across various domains, from autonomous systems to multimodal content generation. However, these models remain susceptible to significant security and safety vulnerabilities. Such weaknesses can enable attackers to jailbreak systems, allowing them to perform harmful tasks or leak sensitive information. As AI becomes increasingly integrated into critical applications like autonomous robotics and healthcare, the importance of ensuring AI safety is growing. Understanding the vulnerabilities in today’s AI systems is crucial to addressing these concerns.

Alex Strick van Linschoten, a machine learning engineer at ZenML, joins the MLOps Community podcast to discuss his comprehensive database of real-world LLM use cases. Drawing inspiration from Evidently AI, Alex created the database to organize fragmented information on LLM usage, covering everything from common chatbot implementations to innovative applications across sectors. They discuss the technical challenges and successes in deploying LLMs, emphasizing the importance of foundational MLOps practices. The episode concludes with a call for community contributions to further enrich the database and collective knowledge of LLM applications.