How AI covered a human’s paternity leave: getting analytics agents to actually work in production // Quinten Rosseel

When your head of data goes on paternity leave, you learn whether your AI agent actually works. For a logistics SaaS company with a 2.5-person data team, our AI analyst ""Wobby"" became the unexpected backup, handling 60% of incoming data questions from the business. This talk shares the hard-won lessons from taking an AI agent from concept to daily use. You'll learn why we abandoned our web UI for Slack, why BIRD benchmark scores meant nothing for our actual success, and how we built an eval system that caught real failure modes instead of synthetic ones. We'll cover the technical decisions that mattered: context engineering, metadata design, and latency optimization. We'll also cover the non-technical ones that mattered more: channel design, user onboarding, and building trust with skeptical business users. This is a practitioner's guide to agent deployment. What worked, what failed spectacularly, and what we'd do differently next time.