Using Agents in Production: Past Present and Future // Euro Beinat

Euro Beinat [00:00:05]: Thanks everyone. So it's always a pleasure to be back here. I'm going to share some of the things that we learned in the last about 12 months about deploying agents in production at scale. And the number 30,000 here is just not a random round number. It's exactly the number of agents that we expect to have in production across process by the end of March. So in the next few months I'll share what we learned in that process, what worked, what didn't work, and what we expect is going to happen next. But as a way to give some context to all this, Prosus is a technology company. We focus on E commerce.

Euro Beinat [00:00:46]: We are present in 100 countries across the globe, South America, Europe and Asia predominantly. And everything that we do is to connect those that want to buy things, those that want to sell things. We have platforms in between, from food delivery to traditional E commerce to jobs to real estate. So the entire spectrum of what we call E commerce, the focus in terms of technology at this moment that we have this big technology shift is mainly two things in using AI agents and large commerce models to do two particular things. On the one hand, to change the way E commerce works, make it more agentic, personal, understand, intent and ecosystemic. At the same time, to use the same technology to create an AI agentic workforce. So to enable all our colleagues working across the group to do better, to do more, to be more agile and so on. I'll focus most on the second one, but just one note on the first one.

Euro Beinat [00:01:47]: So the next E commerce would love to talk a lot about that. So this wave of technology change is going to alter completely the way that we shop, the way that we purchase things, the way we use the Internet in general. And we believe it's going to be agentic, personalized and ecosystemic. There are already dozens of applications here around, let's say our group. So here are three examples. IFood, for instance, is food delivery company in South America, one of the fastest growing food delivery companies globally. And in Ifood we developed ilo, which is an agent that supports the entire shopping experience on WhatsApp or on the app from beginning to the end. But it's personalized in the sense that everything you see there is just for you.

Euro Beinat [00:02:32]: It understands the intent and it guides you through the process. And so the same for Despegar and Easyco, where Despegar for travel and experience in Easygo for finance and purchases and many, many others. So this is the way we implement agents in our customer facing applications. But today, given also the context of this conference, I'm going to focus mostly on this, on the AI workforce, the AI agentic workforce and the way we think about it is a transition from where we are now, where we use everybody of us definitely do AI assistants that offer occasional help and they can be each one of the brands that you know there, and they are like interns, which, you know, they know a lot of things, they're very easy and eager to help, but. But they are not really trustworthy all the times and sometimes they hallucinate. So what we want to do is to go from that position to the position which we have at our availability. Senior colleagues, senior agent colleagues, which can be trusted. They know the domain, they know your organization, and they can execute tasks end to end.

Euro Beinat [00:03:43]: That can help. So the way we see this is that every colleague across the group, and I guess every employee globally, will have 1, 2, 10, maybe 20 of these agents helping them to do better. Where better means many things. Of course it is productivity. We can do more with the same resources, but it's also quality. So it means that not only we do more, we can do better in different metrics, because different use cases have different metrics and these things are usually measurable. So you can measure productivity, you can measure quality improvement, say customer support, type of grades and so on. But then also very important is independent, so you can work and do things without asking a colleague.

Euro Beinat [00:04:31]: And agility, so you can work outside of your comfort zone. Like developers that use languages which have perhaps less experience with or even marketing or legal colleagues that develop small applications to do their work better, so they can do that without being dependent on others. So that's the focus of, of this deployment of agents. When I talk about agents, I have something very specific in mind. I'll just go through an example which is going to clarify the language that we use. And hopefully it's the same language. But in any case, I think it's a good idea, it's a good example to showcase what we have in mind. So this is a real use case and has to do with some of our partners, which can be sellers, can be restaurants, can be car dealers, can be real estate agencies that use our platform to reach to customers.

Euro Beinat [00:05:27]: There are hundreds of thousands of these partners globally and of course we provide them with support. And that support, it's, let's say, many dimensions in this particular case. Here is to answer one question, and the question is from a partner seller in this case that wants to know why there has been a recent drop in sales. The typical way that this would work is that the partner, the seller would contact us, in this case through WhatsApp, goes through a CRM and the question would be picked up by a partner manager. So that would be the person that in normal situations would be the interface with. And this can be a restaurant or car dealer and so on. Normally that person would go inside the organization, ask our data analyst how to answer this question. That would be people looking at the data, coming back with a report back and forth when the question has been properly answered.

Euro Beinat [00:06:25]: That would be, let's say, communicate to the seller. And the reason for the drop in sales is because another restaurant down the road opened let's just two weeks ago and it has a similar menu item, something like that. This is a different way of doing things, which has the same result, has better result, but it's based on agents. So the seller still goes through the WhatsApp channel, but then the next step goes to an AI Account Manager. This is a tool that is trained to answer these kind of questions. This agent asks the question he recognizes about data, asks the question to an AI data analyst, which is another agent, and that agent is capable of talking to databricks of Tableau or whatever other data sources you have, collect the data, package it, give it back to the AI account manager, reviews the information back and forth again, and when satisfied, sends the answer to the seller and sends a copy of that to our partner manager. The outcome is the same, but it has some differences. First of all, the partner manager now, in a normal way might take an hour, 15 minutes, an hour or maybe a day to give the answer, depending on whatever other things happens.

Euro Beinat [00:07:44]: This is instantaneous. The other thing is that instead of covering or supporting 20% or 30% of the restaurant partners is going to cover all of them. So there's a big difference. And by the way, this is already in production, is one of the 20,000 agents that we have in production. Now, how do we look at this? So the way we have been thinking a lot about which tools we could use to enable everybody in the organization, and I think at a very high level there are two classes of tools. The first one, those that apply to well structured frequent use cases here, each one of them has a very large addressable total, addressable market. It can be technology coding, it can be customer support, can be sales. Each one of these areas is similar, independent of the organization, independent of the industry.

Euro Beinat [00:08:40]: Therefore, entire businesses have been created to support them. And then there's another class which is a long tail of sparse specific use cases. These Are thousands of tiny total addressable markets of hyper specific applications. It can be one person, a team of 10 people and so on. There is no particular tool set that enables them. So the tools have to be created, the agents have to be created. But if you sum them all up, it's a massive business or is a massive value which is created for the organization. I'm going to focus in particular on this second class.

Euro Beinat [00:09:22]: And since we didn't have tools that we could use at scale, we developed the tools ourselves. So tocan, it's our builder of AI agents. It has been created in house, it's not for sale, it's just available to us. But many of the things that we learn are most likely replicable. So we decided to do this. This, by the way, it is the history of this tool. Started 2019, released just before ChatGPT, and then became an agent based tool in 24. And we started supporting our employees, our colleagues with tools to create agents about a year ago.

Euro Beinat [00:10:02]: The goal now is to make sure that everybody can create this agent. So 30,000 of these around our group. It is the reason why we build it is because of safety and privacy reasons, is because we can optimize for our own use cases, is because we can connect to our own specific integrations or MCPs and guarantee safety. And also for cost effectiveness, it is less expensive. This is where we are now. We released the tool to our colleagues in December last year. You can see there was a few of our colleagues started building agents. And then you can see at this point now, in October, we had about 8,000 of these agents in production which are used every week.

Euro Beinat [00:10:48]: But about 20,000 of these agents in total have been created. Maybe they are used less frequently. You can see the progressions here. There's a slower progression the first months and then there is an uptick. I'll talk about that later. Because one of the things that we learn is about how to get this into an exponential growth. Another thing that we notice here is that these agents have a sort of grading, a seniority. And they range from interns to senior Interns are agents that work with documents, but they don't do very complicated things.

Euro Beinat [00:11:22]: The juniors have access to systems, so we give them credentials so we can go into systems. Intermediates have not only access to system, but they can also put back data to database. So now you need to start trusting them, not only to give you the right thing, but also not to pollute your systems. And finally, seniors are agents that orchestrate. Agents are more complex in general, there Is a sort of relationship between. There is a very strong relationship between seniority and two attributes. The first one, the number of tools that these agents have available. The more tools, the more complicated things you can do, the more use cases you can implement.

Euro Beinat [00:12:06]: But also the number of integrations, how many systems you have access to. Here you can see some of the systems which are typical operational system like email, calendar and so on. But it can also be systems of record, can also be help desk, can also be, let's say all the other system an organization have, plus a few which are really specific for our own companies. And there are many of those in our own companies. Obviously these tools can also go out on the web, which is the last block to the right. And I think there is a lot of interesting applications between internal and external systems. But there are also a number of potential issues that happen when these two things are mixed. And I hope there will be discussions about that in the rest the day.

Euro Beinat [00:12:56]: Let me give you an example of a few of these agents. So the first one on the left, it's a senior agent. We call it a restaurant account executive. It is used by our account managers in food delivery. So the counterparts are restaurants. And what these account managers want to do, they want to meet those restaurants on a regular basis and give them insights about their performance, help them perform better. This job of creating the material before going to the meetings has been done traditionally manually. Part automatic in this case is completely automatic.

Euro Beinat [00:13:36]: The tool goes out, collects everything from multiple systems, assembles the information, creates a small web page. This web page is the basis of to go and talk to our partners restaurants. In this case it's used by about 200 of our colleagues at this moment and it does the work of about 30 full time employees. Obviously in this case the impact is productivity, but it's also quality of the advice and it's also coverage. So they can serve many more restaurants than otherwise. The one in the middle is the data analyst. Is one of the most common agents that we have worked on in the last year. Is one of the hardest to get right.

Euro Beinat [00:14:16]: Still, after a long time it looks at giving everybody access to data in language, in standard language. So you can ask data questions in English or Portuguese or your language and goes down, creates the query, gets something back for you. It is intermediate in this case because it uses one or few databases with a very, very precise type of workflow. But it does have a big impact on productivity. It gives independence. It means that all people that cannot write SQL still have access to database without asking to the data analyst and it gives agility for the same reason. The last one is a junior agent that has only one user and it's me. I use it to read newsletters.

Euro Beinat [00:15:04]: I subscribe, like many of you, to dozens and dozens of newsletters which I can't read for reasons of time. So this agent goes into my email every Saturday, creates an overview of all the topics picks across all the newsletters. The topics which are mentioned by all of them, which is a sort of voting, seems important, but also sifts for things which are important for me. And it has all the things that I instructed it to do. I mentioned this one because it's an example of junior agent, but also an interesting one because it doesn't have a productivity dimension. It doesn't save me time, it saves me theoretical time, but not time that I would spend otherwise. But it does give me coverage of information. It does give me independence because I can do sifting through without asking others and I can personalize it.

Euro Beinat [00:15:53]: So this still is very important. Let's go back to this diagram here because it's one of the things that we learned and we care most about. So you can see this progression of creation of agents. Remember, these are agents that have been created by individuals across the organization, not by engineers, by our colleagues in sales, in marketing, in hr, in logistics, everywhere. You can see that up to July of this year the growth was continuous, but it was not exponential. This growth is closely correlated to additional features that we produce, to additional integration that supported, to MCPS that we made available. So the question was, how do we start this exponential progression? We have to come to terms that after a certain point. AI adoption is never a technical problem, it's always an organizational problem.

Euro Beinat [00:16:53]: This adoption here started because we reduced the barrier. One of the simplest possible barrier was very important and very impactful, in fact, was that a lot of people thought that creating agents is something that engineers can do, software people can do. So we had to remove that. How did we do it? By showing how to do it. We get together with the salespeople, with HR people and so on, and we create very quickly agents. We demonstrate how simple that can be, demonstrate how you can iterate until you get what you want, eliminate the barrier. So we made agents uncool. Again, there's nothing special about them.

Euro Beinat [00:17:34]: It's something that can make work for everybody, but had to be done. It was not a natural way for people to understand these agents. The second, obviously you need to do upscaling at scale with all the opportunities, all the Features that we have and then a number of other tools that enable people to get into this mood of I have this job to be done and I can think of an agent to solve that job to be done. That mindset doesn't come by itself, has to be facilitated. One of the ways to facilitate that is through competitions. It's one of the many interventions that we have been testing and doing in the last year. Let me show you an example of this one here. This is a competition which we call Prosus Got AI talent and it started now, is going to be completed in March.

Euro Beinat [00:18:30]: And it's an internal competition inspired by Shark Tank. Teams across the entire group can be in as many teams as they like, compete to solve real business challenges and compete with each other for the best interest agent that has the most valuable and impactful solution. They compete for a prize but also to participate to the final event in March. So you can imagine now hundreds of teams that are building things every month. There's going to be a selection. This selection goes up to a point in which we have the final event in March. It is designed to ignite innovation across the ecosystem. The ultimate goal is to make to create the organization on muscle for AI across process.

Euro Beinat [00:19:17]: And this is one of the instruments we really believe that this is going to create. Visibility, is going to give recognition to those that do a good piece of work here. And it's going to make agent creation just one of the normal things that everybody does. If that happens, I'm 100% sure that we as an organization we will have a competitive edge. So summing up all this, there are a few learnings that I want to underline. So the first one is that beyond a certain point when technical tools are available and so on, the adoption here becomes an organization and culture problem. So we need to shift from features, techniques and so on to how do we get people bottom up to create these tools? Because that's what these tools they help themselves. These are tools that help every individual to automate some jobs to be done.

Euro Beinat [00:20:16]: The second thing that I personally believe sincerely about is that this is a bottom up process. It is a process of collective discovery. In spite of the fact that we've been around with developing these tools for some time, this is still new for everybody. And the best way to learn is by doing it. So we provide tools for everybody to experiment and do it. And experimentation at scale is going to become a distinct advantage of this organization. The final thing which is obvious for those of you that are technical in this context here, these Tools work when we can surface and encode domain knowledge. A lot of the work here is to make sure that this domain knowledge implicit in the way they work it is surfaced so that agents can take advantage of that.

Euro Beinat [00:21:11]: With that one, I want to flow into the, let's say, the organization of today. There are going to be three stages and we are going to talk about all the things that I mentioned here in much more depth. I hope that you will have tons of questions and contribution. I really hope we can all learn from you. Stage one is going to be about E commerce. Stage two is going to be AI workforce. And stage three is going to be about mpc, NCPS and protocols. With that one, thank you very much and I look forward to interacting with you during the event.

Demetrios Brinkmann [00:21:45]: Euro Incredible.

Euro Beinat [00:21:47]: Oh boy.

Euro Beinat [00:21:51]: I heard there was some iFood people in here.

Euro Beinat [00:21:54]: Great to see you again.

Demetrios Brinkmann [00:21:56]: You like this?

Euro Beinat [00:21:58]: I love it. I love it.

Demetrios Brinkmann [00:21:58]: We got to keep it fresh for the crowd. That was awesome. This has been really cool. We have one minute for some questions. € don't escape too soon. There's so many agents that you talked about. 30,000. That's a huge number.

Demetrios Brinkmann [00:22:13]: How do you evaluate all those agents?

Euro Beinat [00:22:15]: Oh, great question. One by one. So I mean there's an entire framework for evaluating these agents and to be totally honest, we are developing as we go. In some cases it is relatively easy because you can ab test without the agents. So you know, for each agent. Exactly. What's the impact there? For other cases there's no. Let's say there's only before and after.

Euro Beinat [00:22:39]: So you have to rely more on information that the users provide you. But usually there are four directions though. Dimension. The first one is productivity, whatever productivity means, right? Because it's different from different things. The second one is quality. The third one, agility. So you can do more than different things that you were able to do before. And the fourth one is independence.

Euro Beinat [00:23:00]: You do not depend on others to do what you need to do. These are four dimensions and how it's implemented is really case specific.