Agents in Production 2024

If you think AGI is just around the corner, try building an agent. Today's frontier models often feel like idiot children, capable of brilliant superhuman replies and at the same time making absurd logical errors, while lacking any common sense. Even worse, those errors pile up, making them unreliable. AI Agents hold the incredible potential to change how we work, learn and play. But any company working in agents quickly realizes they're incredibly hard to build, to make reliable and to generalize. Instead, most teams have shifted to building narrow agents that they can scope to a particular problem with lots of glue code, heuristics and prompt engineering. In this talk we'll take a look at why agents are so challenging to build, what we can do about it and whether just when they'll live up to the hype of doing complex, open ended tasks in the real world.

At Faber Labs, we have built and patented the first-of-their-kind scalable specialized Agents that autonomously seek to maximize Conversion Rate (CR), Average Order Value (AOV), Return on Ad Spend Optimization (ROAS) With rapidly evolving variations on Agent-based RAG systems, the demand for highly efficient and goal-oriented retrieval systems is increasing. GORA (Goal-Oriented Retrieval Agents) is a pioneering system designed by Faber Labs to first-of-their-kind scalable specialized Agents that autonomously seek to maximize Conversion Rate (CR), Average Order Value (AOV), Return on Ad Spend Optimization (ROAS). We will explore how GORA distinguishes itself by embedding the overall system goal into its pipeline, ensuring that each component not only performs optimally but also cohesively works towards a unified business objective. Key aspects of GORA include its interactive feedback mechanism, which transitions from traditional static queries to dynamic conversational interactions, allowing for a more natural and effective user engagement. We will discuss the system’s ability to maintain low latency despite its complex architecture, emphasizing how efficient integration of components is crucial for user efficacy.

Cleric is building an AI-powered Site Reliability Engineer capable of autonomously monitoring, diagnosing, and fixing issues in complex distributed systems. This talk will explore how Cleric's AI agent can handle novel situations, learn continuously from its environment, and respond to issues at speeds far beyond human capabilities. It will cover the technical approach behind creating an AI that can understand system architectures, reason about failures, and take appropriate actions without human intervention. The discussion will also examine how autonomous SRE agents could reshape the future of software operations and reliability.

The State of AI Report analyses the most interesting developments in AI. We aim to trigger an informed conversation about AI's current state and its implications for the future.

The internet and its business models are changing. For the last 16 years, Stack Overflow has been at the forefront, helping to shape the future of the web. In this time of disruption comes a necessary time to reflect, change, and break norms. With the expansion of generative AI, many LLM providers are not allowing the world to operate as it has previously. Instead, a new paradigm has emerged: content created by thousands of creators across the Open Web is now being used to train models without respect or attribution for the original creator.Consequently, in the current transformation, human-centered sources of knowledge are obscured but a key component of the AI stack. Companies and organizations lucky enough to host these engines of knowledge production are at a decision point; how do they continue to grow and invest in their communities when the technological landscapes have changed. A change in strategy will allow these companies, AI tools, and the communities and data sets that power them to thrive.

Why can't my agent perform actions on my behalf? How is tool calling different from retrieval? How do I test my LLM tools? How do I evaluate different model's ability to call tools? These and many more questions were some of the main points we've been figuring out as we've been building Arcade AI. This talk will focus on the approaches we tried and the approaches we found that worked. If you want to or are building an Agent that can call tools, this will be a good talk for you.

This panel speaks about the diverse landscape of AI agents, focusing on how they integrate voice interfaces, GUIs, and small language models to enhance user experiences. They'll also examine the roles of these agents in various industries, highlighting their impact on productivity, creativity, and user experience and how these empower developers to build better solutions while addressing challenges like ensuring consistent performance and reliability across different modalities when deploying AI agents in production.

In this segment, the Panel will dive into the evolving landscape of AI, where large language models (LLMs) power the next wave of intelligent agents. In this engaging panel, leading investors Meera (Redpoint), George (Sequoia), and Sandeep (Prosus Ventures) discuss the promise and pitfalls of AI in production. From transformative industry applications to the challenges of scalability, costs, and shifting business models, this session unpacks the metrics and insights shaping GenAI's future. Whether you're excited about AI's potential or wary of its complexities, this is a must-watch for anyone exploring the cutting edge of tech investment.

A presentation from Euro and Paul about Agents, and what they have learnt from their 15,000 Colleagues who Use Ai-Agents Every Day

A fireside chat between Euro and Thomas about Agents, their challenges and their future

The field of agentic system design is an exciting and quickly developing area with significant implications for software use across various sectors. These systems differ from conventional software by employing decision-making processes that are non-deterministic and often unpredictable. By drawing insights from diverse fields such as robotics, cybernetics, and biology, we can begin to cultivate an understanding of how to construct systems that are more dependable than their individual probabilistic components would suggest.

The rapid development of Large Language Models (LLMs) has led to a surge in applications that facilitate collaboration among multiple agents, assisting humans in their daily tasks. However, a significant gap remains in assessing to what extent LLM-powered applications genuinely enhance user experience and task execution efficiency. This highlights the need to verify utility of LLM-powered applications, particularly by ensuring alignment between the application's functionality and end-user needs. We introduce AgentEval, a novel framework designed to simplify the utility verification process by automatically proposing a set of criteria tailored to the unique purpose of any given application. This allows for a comprehensive assessment, quantifying the utility of an application against the suggested criteria.

Generative AI Agents represent the current frontier of LLM technology, enabling dynamic interactions and intelligent workflow automation. Building and deploying production-ready agents, however, requires navigating various complexities and understanding key lessons learned in the field. This session distills over two years of practical experience into actionable best practices for successful agent development. We'll dive into techniques like meta-prompting for prompt optimization, establishing robust safety and guardrails, and implementing comprehensive evaluation frameworks. By addressing these critical areas, you'll be well-equipped to build more effective and reliable Generative AI Agents. Check out Patrick's recently authored whitepaper published on Agents: https://www.kaggle.com/whitepaper-agents

In our journey from concept to production, we focused on delivering consistent behaviors to build user trust. This talk will cover the design and refinement of AI systems using agentic frameworks and deterministic components, emphasizing the integration of continuous learning and human oversight.

OLX has a catalog of millions of unique items across a very diverse range of categories. While existing technologies for navigating this assortment are reasonably effective, LLMs offer the potential for a much improved user experience. In this session, we will talk about how we utilized the Agentic flow to transform a customer’s standard search on OLX into something highly personal, fully reflecting the customer’s demands and feelings. Aside from Generative AI, we will also share how we redesigned the search pipeline to work well with agentic architecture. And lastly, we will talk about some lessons learned along the way!

Software Engineering can serve as a diverse, rich testbed for evaluating the next generation of language models and driving their development. This talk introduces a line of three works that has established the potential of this research direction and guided industry advancements towards autonomous software engineers. First, SWE-bench is a benchmark that evaluates an Al system's capability to resolve real world GitHub issues, featuring 2294 task instances collected from 12 distinct Python repositories. Second, SWE-agent is an autonomous system that uses a language model to interact with a computer to solve software engineering tasks, setting a state-of-the-art 12.5% resolved rate on the SWE-bench test set. Lastly, SWE-bench Multimodal, a new dataset of 617 task instances from 17 JavaScript repositories, shows how many existing coding agents are overfit to Python, raising the property of “generalizability as an overlooked but desirable trait of AI systems.

Integrating Large Language Models (LLMs) into production-level data workflows presents both significant challenges and opportunities. In this talk, we'll introduce Numbers Station, a platform that automates data analytics workflows using LLMs, Retrieval Augmented Generation (RAG) over a Knowledge Layer, and a customizable multi-agent architecture. We'll start by discussing practical use cases for analytics, such as dashboard search, query generation, or automatically adding analysis summaries to slide deck presentations. We'll then delve into the methodologies for deploying LLMs within data analytics workflows, focusing on a detailed case study to build a SQL agent from the ground up. We will cover the architectural considerations necessary to support agent-based analytics, including the role of dynamic control flows and the importance of incorporating business context through a unified Knowledge Layer. This session aims to provide a deep technical insight into transforming theoretical AI frameworks into practical, scalable solutions that advance organizational data capabilities.

In this talk, I will discuss practical issues and technical fundamentals of coding agents. This will include both the promise of what coding agents may be able to do if they are implemented properly, what tasks they can do well now, some challenges in using them appropriately, and some of the challenges that we need to solve to make them more effective -- file localization, file editing, planning, error recovery, and "good taste". I will also give a few demos of our open-source OpenHands toolkit (https://github.com/All-Hands-AI/OpenHands) to illustrate each of the points.

In a world dominated by Python-centric AI frameworks, JavaScript developers have often found themselves at a disadvantage when it comes to building intelligent, multi-agent systems. KaibanJS aims to change that. In this talk, we’ll explore how this first-of-its-kind, JavaScript-native framework enables developers to create and integrate AI agents effortlessly into their projects. We’ll break down real-world use cases, dig into the technical features, and demonstrate how KaibanJS is helping to bridge the gap for JavaScript developers in the AI space.

The Creative Singularity is here! Welcome to the age of transformable media, where audiences can generate and remix their entertainment at will – and AI systems will regenerate the content in return. This talk will take you behind the scenes on several of Transitional Forms' dynamic media projects, designed to use machine intelligence as a real-time creative collaborator to generate the content itself.

This talk explores how AI can automate the invisible processes in healthcare, like insurance calls, to reduce administrative burdens. I'll share our journey with HeyRevia and how we create AI solutions to help providers focus on patient care.

AI agents change what we can achieve with software; but also change how we have to think about building software systems. In this talk, I'll share some of the lessons we've learned while building a powerful AI agent for complex support settings.

Few companies have shipped an agent at the scale of Replit, but with scale comes challenges. This talk explores how Replit built and scaled its agent, and how it adapted from a small engineering team to involving almost half of its engineering organization almost overnight.

As conversational AI systems become increasingly autonomous and mission-critical, ensuring their reliability presents novel challenges that parallel those faced in self-driving vehicle development. This talk explores how simulation-based testing and evaluation strategies from autonomous vehicles can be adapted to build more robust AI agents. We'll examine how traditional software engineering practices are evolving in response to AI systems, where deterministic unit tests give way to probabilistic performance metrics and behavioral analysis. Drawing from real-world examples, we'll demonstrate how comprehensive telemetry — both in pre-production simulation and production environments — provides crucial insights beyond simple pass/fail metrics. The presentation will delve into the critical balance between computational cost, latency requirements, and signal quality in AI system evaluation. We'll introduce a framework for developing evaluation strategies based on reliability requirements across different use cases, from bug detection tools where any true positive provides value, to medical assistance systems that demand near-perfect accuracy. Attendees will learn practical approaches to implementing simulation-based testing pipelines, techniques for meaningful telemetry collection and analysis, and strategies for defining appropriate reliability thresholds based on application context. This session will benefit ML engineers, software architects, and engineering leaders working on production AI systems.

In an era where knowledge workers are inundated with manual tasks and corporate memory systems, agents hold massive potential to streamline and document their workflows. Agents also hold the potential to unlock a vast amount of skills and knowledge from the front-line employees of global businesses. This talk explores how dynamic agents can capture work patterns, from inboxes to task sequences, using clustering and process mining techniques. These methods ground agentic planning and provide more robust, company-specific workflow orchestration. We’ll also discuss how knowledge graphs power effective memory management, and how Interloom’s innovative MemoryRank™ system leverages reflective agents to create and serve agentic memories, optimizing task management for both human workers and specialized agents. Attendees will gain insights into how agentic memory systems can drive efficiency and productivity for end-to-end processes and workflows.