MLOps Community Podcast

Anurag Beniwal (Member of Technical Staff at ElevenLabs) breaks down the real-world challenges of building voice agents—from latency, transcription accuracy, and turn-taking to the tradeoffs between cascaded systems and end-to-end speech models. The conversation explores why production systems rely on “constellations” of models, how to design for non-technical users (especially in customer support), and why voice unlocks richer context—but introduces far more complexity than chat. Ultimately, it’s a deep dive into making voice AI practical, reliable, and usable at scale.

Jesse Vincent breaks down how modern “agentic” software development is shifting from writing code to managing intelligent systems. He shares how his Superpowers toolkit uses structured workflows, skills, and subagents to turn vague ideas into executable plans—emphasizing that clarity of intent matters more than coding itself. The conversation explores how AI agents can be guided using psychology, why separating roles (planner, implementer, reviewer) leads to better outcomes, and how iteration—not perfection—builds powerful workflows. Ultimately, the future of software isn’t code—it’s specs, judgment, and orchestrating agents to do the work.

Most teams treat evals like a last-minute checkbox—ship first, panic later—but that’s exactly backwards. The real edge comes from treating evals as a continuous, evolving system from day one, not a static test suite. Because here’s the uncomfortable truth: LLMs don’t fail cleanly or consistently, and neither do your users. If you’re not constantly adapting how you evaluate, you’re basically flying blind—just with more features to hide it.

Scaling LLMs in production requires balancing cost, latency, and performance. Through techniques like dynamic GPU scaling and TensorRT optimization, latency was reduced by up to 70%, while iterative learning and tight alignment with business goals ensured strong ROI.

Most people cripple coding agents by micromanaging them—reviewing every step and becoming the bottleneck. The shift isn’t to better supervise agents, but to design systems where they work well on their own: parallelized, self-validating, and guided by strong processes. Done right, you don’t lose control—you gain leverage. Like paving roads for cars, the real unlock is reshaping the environment so AI can move fast.

Kashish zooms out to discuss a universal industry pattern: how infrastructure—specifically data loading—is almost always the hidden constraint for ML scaling. The conversation dives deep into a recent architectural war story. Kashish walks through the full-stack profiling and detective work required to solve a massive GPU starvation bottleneck. By redesigning the Petastorm caching layer to bypass CPU transformation walls and uncovering hidden distributed race conditions, his team boosted GPU utilization to 60%+ and cut training time by 80%. Kashish also shares his philosophy on the fundamental trade-offs between latency and efficiency in GPU serving.

AI agents are shifting the role of developers from writing code to defining intent. This conversation explores why specs are becoming more important than implementation, what breaks in real-world systems, and how engineering teams need to rethink workflows in an agent-driven world.