What makes a good AI benchmark? Greg Kamradt joins Demetrios to break it down—from human-easy, AI-hard puzzles to wild new games that test how fast models can truly learn. They talk hidden datasets, compute tradeoffs, and why benchmarks might be our best bet for tracking progress toward AGI. It’s nerdy, strategic, and surprisingly philosophical.

Greg Kamradt: Benchmarking Intelligence | ARC Prize

I’m sure the MLOps community is probably aware – it's tough to make AI work in enterprises for many reasons, from data silos, data privacy and security concerns, to going from POCs to production applications. But one of the biggest challenges facing businesses today, that I particularly care about, is how to unlock the true potential of AI by leveraging a company’s operational business data. At Snow Leopard, we aim to bridge the gap between AI systems and critical business data that is locked away in databases, data warehouses, and other API-based systems, so enterprises can use live business data from any data source – whether it's database, warehouse, or APIs – in real time and on demand, natively. In this interview, I'd like to cover Snow Leopard’s intelligent data retrieval approach that can leverage business data directly and on-demand to make AI work.

Bridging the Gap Between AI and Business Data

The creator of FastAPI is back with a new chapter—FastAPI Cloud. From building one of the most loved dev tools to launching a company, Sebastián Ramírez shares how open source, developer experience, and a dash of humor are shaping the future of APIs.

The Creator of FastAPI’s Next Chapter

Willem Pienaar and Shreya Shankar discuss the challenge of evaluating agents in production where "ground truth" is ambiguous and subjective user feedback isn't enough to improve performance.

The discussion breaks down the three "gulfs" of human-AI interaction—Specification, Generalization, and Comprehension—and their impact on agent success.

Willem and Shreya cover the necessity of moving the human "out of the loop" for feedback, creating faster learning cycles through implicit signals rather than direct, manual review.
The conversation details practical evaluation techniques, including analyzing task failures with heat maps and the trade-offs of using simulated environments for testing.

Willem and Shreya address the reality of a "performance ceiling" for AI and the importance of categorizing problems your agent can, can learn to, or will likely never be able to solve.

Everything Hard About Building AI Agents Today

Your AI shouldn’t just work in the demo. It should work in production.Join us for a high-impact, no-fluff Mini Summit focused on building AI applications that can withstand the real world — where APIs fail, users take days to respond, and servers crash without warning. Whether you’re deep into building with LLMs or just starting to explore how to operationalize your AI systems, this event is packed with practical insights, real-world strategies, and modern tooling to help you ship reliable AI that doesn’t break. What’s on the agenda?🔹 Process Calling: Agentic Tools Need State Function calling gave LLMs a way to "do" things — but it’s not enough. When you’re building agents for customer-facing use cases, stateless abstractions fall short fast. Learn why the future of agentic tooling is process-based, not function-based, and what it means to build agents that remember, recover, and reliably finish what they start. 🔹 Building Reliable AI Applications with Durable Workflows Chaining functions together is easy. Keeping AI workflows running when things go sideways? That’s the hard part. This talk introduces durable workflows — systems that checkpoint state, recover automatically, and gracefully handle everything from human delays to API flakiness. You’ll see real examples of AI pipelines that stay resilient in production. 🔹 No YAML? No Problem: Orchestrate Kubernetes Workflows the Easy Way with Python Sick of writing orchestration logic in YAML? You’re not alone. Discover how Hera, the Python SDK for Argo Workflows, lets you express complex Kubernetes workflows using clean, testable Python code. Keep your business logic and orchestration logic in one place — no indentation nightmares required. Who should attend? AI engineers, MLOps professionals, infra leads, and builders who care about more than just flashy demos. If you're looking to make your AI actually work at scale, this one's for you. RSVP now and start building AI that doesn’t break.

Building AI That Doesn’t Break

<h1>AI Agents Are Already Working — Let’s Talk About It</h1>Agents are no longer just experiments. From e-commerce to customer support to analytics workflows, they’re quietly getting real work done in production.On July 17, join the MLOps Community for Part 2 of Agents in Production — a virtual event focused on the messy, practical side of building and deploying AI agents.What’s on deck?<ol><li data-list="bullet">Taming complexity: agent memory, behavior control, latency vs. response tradeoffs</li><li data-list="bullet">Stories from the field: How companies are actually using agents in live environments</li><li data-list="bullet">Tooling that works: routing, evaluation, UX, and cost performance in the wild</li></ol><h2>It’s free, it’s global, and it’s going to be packed. Full agenda coming soon</h2>

Agents in Production 2025

<h1>🔥✨ GenAI in Games, 3D Animation &amp; VFX ✨🔥</h1><h3>🗓 June 10th @ 10:00 -11:30 AM PST Pacific West-Coast Time 19:00 - 20:30 CEST Central Eastern Time</h3>🔥Hot Topics 🔥:<ol><li data-list="bullet">AI in Game Development: Friction or Fruition?</li><li data-list="bullet">Demo GenAI game tools usage in pre-production</li><li data-list="bullet">Beyond the Prompt: Using AI as a tool to accelerate execution, not replace creative intent</li><li data-list="bullet">Legal Playbook for AI Creativity</li></ol>Calling all creative visionaries—producers, PMs, creative directors, and pipeline innovators. Join us for a future-forward session exploring how Generative AI is transforming entertainment studios—from previs to post-production. Whether you're navigating massive asset libraries, complex feedback loops, or tightening delivery pipelines, this is your chance to rethink workflows from the ground up—technically, creatively, and legally.Hear from a diverse lineup of presenters, each bringing their unique perspective—from real-time GenAI integration and creative collaboration tools to pipeline automation, IP and legal frameworks, and AI-assisted worldbuilding.Discover how studios are using GenAI to unlock smarter systems, efficient iteration, and more reflective space for creativity, all while addressing the evolving ethical and legal challenges shaping the future of our industry.If you're crafting cinematic universes or building next-gen experiences at scale, this is where innovation meets responsible storytelling power.

GenAi in Games, 3D Animation and VFX

<h1>ALL THINGS AI AGENTS</h1>​Come to learn and see the freshest of fresh.<h2>May 28th at The Hibernia</h2>We’re bringing together the builders, researchers, and doers working on&nbsp;AI Agents in Production&nbsp;in the most lo-fi way possible.<h2>30+ campy booths with practitioners.</h2>Only engineers. No marketers.​We already got a great line-up including&nbsp;Boundary ML, Cleric, Saphira, Happy Robot, Pipeshift, Anthropic, Arcade, Falconer, Prem AI, Tadata and more.<img src="https://d2xo500swnpgl1.cloudfront.net/uploads/mlops/some-file-39e282e0-812a-45ac-90da-fb200b01117e-1746830172336.png"><h2>​3 lightning talks&nbsp;by Experts from:</h2><ol><li data-list="bullet">​Databricks</li><li data-list="bullet">​Orkes</li><li data-list="bullet">​Tonic</li></ol><img alt="some-file-76254893-ce3e-4de6-81f9-2053ea2f5a65" src="https://d2xo500swnpgl1.cloudfront.net/uploads/mlops/some-file-76254893-ce3e-4de6-81f9-2053ea2f5a65-1746830202607.png"><h2>Shout-out to our Sponsors for Supporting the Event</h2><img alt="some-file-7410ae16-2833-4744-b975-ebcd71b6abc5" src="https://d2xo500swnpgl1.cloudfront.net/uploads/mlops/some-file-7410ae16-2833-4744-b975-ebcd71b6abc5-1746830310394.png">

AI Agent Builders Summit: World Tour Kickoff

🔥 Snowflake Mini Summit — Let’s Build for Real 🔥Mark your calendars: 🗓 Wednesday, April 23 🕔 5 PM UK timeWe’re bringing the heat with a Mini Summit you won’t want to miss. If you're building anything with ML, LLMs, agents, or just trying to wrangle data into something useful—this one’s for you.We’re diving into:<ol><li data-list="bullet">Real-world MLOps workflows (no BS, just stuff that works)</li><li data-list="bullet">How teams are using Snowflake, ZenML &amp; Featureform to scale fast</li><li data-list="bullet">What it takes to ship AI products in 2025</li></ol>Expect sharp insights, real talk, and zero fluff. Let’s get tactical, let’s get nerdy, and let’s level up!

Iceberg, MCP, and MLOps: Bridging the gaps for Enterprise

The MLOps Community is where machine learning practitioners come together to define and implement MLOps.
Our global community is the default hub for MLOps practitioners to meet other MLOps industry professionals, share their real-world experience and challenges, learn skills and best practices, and collaborate on projects and employment opportunities. We are the world's largest community dedicated to addressing the unique technical and operational challenges of production machine learning systems.

MLOps Community

Job Title

Login to the MLOps Community community to connect with others, attend community events, and more!

Login or Sign Up for even more

To unlock all parts of the community and get the best experience, complete your profile.

MLOps Community

Events

Content