Nishikant Dhanuka [00:00:08]: Hi everyone. Welcome to our talk. I'm here with my colleague Beatriz Ferreira, she's joining us from Portugal. Beatriz, are you here? Hi, Beatriz.

Beatriz Ferreira [00:00:19]: Hello? Yeah, can you hear me? Hello.

Nishikant Dhanuka [00:00:22]: Good. So we are going to talk about how we are using AI agents to transform e commerce experience at olx. So we developed a product which is called OLX Magic. And through this talk we will share some of our journey and some of the learnings that happened along the way. Let's go. Online shopping is changing and we see that happening all around us. And there are certain themes which is emerging new interface. So currently we interact with these e commerce platforms through apps.

Nishikant Dhanuka [00:00:55]: What if I could interact with the platform by doing different things, touching, showing examples, pointing with voice commands. Maybe the same way that I would interact with a smart human who knows how to assist me. So that's new interface, experiential. So currently we buy things first and we experience them later. What if in the new wave we could experience things first and then we decide based on that experience if it's the right fit for us. Ecosystem. Ecosystem plays a big role. So if you buy a product or a service, what if the platform knows what other products or services that goes along with it? And I think travel industry is a great example that you book a flight and there are other things that you want along with it, maybe some kids activity, your taxi to the hotel and so on.

Nishikant Dhanuka [00:01:49]: So those things could come together. Hyper personalization. So personalization is already a thing, but users in the next wave of e commerce, users expect hyper personalization where I want results, which is uniquely, which is specific to me and it's very different from the other person who's giving, who's asking for the same thing everywhere. So e commerce is everywhere, so it doesn't matter whether. So you don't need to go to an E commerce platform. Every digital platform that you touch becomes an E commerce platform. So you go to. And we see that happening already, right? So people are already shopping on Insta, on TikTok, TikTok has TikTok shops and so on.

Nishikant Dhanuka [00:02:30]: So that's also kind of a theme which will become more prevalent in next wave of e commerce. And finally agents. So the conference is about agents in production. So agents and assistants, they, they are becoming guides that can execute, that can advise us, they can even execute a lot of tasks on our behalf, help us navigate the web, maybe even negotiate prices and so on. And so you outsource your chores to agents, but you keep the experience because you still want to experience things. But maybe agents can take over the routine work. So online shopping is changing and the next wave of e commerce is AI first. So olx.

Nishikant Dhanuka [00:03:14]: So for some of you who don't know, who doesn't know olex. So OLX is a global leader in online classifieds. It's a secondhand marketplace where buyers and sellers, they come together to exchange a wide variety of goods and services ranging from small things like toys to very high value items like cars, real estate and so on. So we built a product, so we took a step in the direction of this next wave of e commerce. We built a product, OlxMagic, which is basically a conversational experience to buy products on olx. And so this product has a lot of features but you know, there are three important things that I want to highlight. First, the idea is that when you buy things it's, you have a lot of questions and you know, it's never in a single shot. So you're buying shoes or maybe you, you're not sure.

Nishikant Dhanuka [00:04:05]: You say I want size 41, you want to change it to size 42. Maybe initially when you started you have a different budget in mind, but now when you see options, so you refine your search a lot. So with olxmagic, one of the main features is that it helps you refine search through multi turn conversations and it also helps you along the way by even advising you what you should focus on, even suggesting some follow up questions. Second. So generative AI has made it much easier for us to express ourselves not in just text, but also images, video and so on. So Alex, Magic can handle multiple modalities. So what that means is you can upload an image. So you like a product that you see, you upload an image and you ask Magic to show you similar items.

Nishikant Dhanuka [00:04:50]: And finally the technology. So of course we are using LLM, but underneath we are using Agentic Flows and we are also combining with the most advanced search on OLX catalog. So let me show you some of these features through a demo. So this is the main chat screen and we start with some onboarding cards to help the users. I have a dog, I have a golden retriever. So I want a vacuum cleaner that helps with dog hair. But I don't know what are the, you know, I don't know which vacuum cleaner to buy. So my request is a bit broad right now, I've not decided and that information is not there in OLEX product catalog.

Nishikant Dhanuka [00:05:33]: So what you see happening here. So this information is pulled from web, right? And we get some sources at the bottom and Then I also have some follow up questions being suggested to me, so I can go for that. But there's a particular model that I like, so maybe I say that, okay, show me Dyson V15. So that was one of the options that the previous results suggested. So now when I do that. So behind the scene it's an agentic architecture which is actually going through Olix catalog and it's searching. So a lot of things are happening. And I get these 25 results presented to me which I can scroll.

Nishikant Dhanuka [00:06:15]: And some of these are also, you know, we also diversify the results. So some of these are also accessories. And there's also a nice summary. So it tells me that the price ranges between this and this and these are the different items. So I get a quick glimpse and maybe I like two of these options so I can select them and I press Smart Compare and now I get my results and the result comes in. So it, it makes a nice table with a side by side comparison. And I also get some textual recommendation at the end that if I consider this, then this product, otherwise a different product. So that's an example of refining the search through OLxMagic.

Nishikant Dhanuka [00:06:56]: Now let's take another example with an image. So let's say I want to buy. So I saw these sneakers and I like them and I want something similar on olexmagic. So I could just go to my gallery, attach the image and send it to olxmagic. And again what you see here is a different pipeline gets triggered. So we do visual search this time and you see we get similar results. And since this time my request is very specific, I even get a final recommendation which is kind of in the direction of hyper personalization. Another scenario is let's say, so we spoke about some requests which are specific or something in between.

Nishikant Dhanuka [00:07:36]: But let's say my request is very broad. So I'm going for a camping trip and I want, and I don't know what I want to buy, but I know that, you know, I want to buy a couple of things. So because the idea is that people can come to Magic with a broad range of requests and Magic should be able to handle, you know, both specific and broad requests. So in this case it asked me a couple of clarifying questions and then it would actually, since my request is broad, it would actually suggest me a couple of different options. So you know, there's a row which is about tents, there's another row which is about a different camping gear and so on. So then it helps me with my broad Search. So that's kind of, you know, an idea behind what we built with Ox Magic, but that's not it. So we learned a couple of lessons along the way.

Nishikant Dhanuka [00:08:34]: So we built it, we put it in front of the customers, we see how customers are using our product. In fact, we interviewed a lot of these people and there are a couple of things that we learned. First is conversational interface. So you know, you don't need a chatgpt for everything. So your conversational interface might not be suitable. So though it's powerful because, you know, it helps you refine your search, but it might not be the best choice for everything. And you know. So for example, in user interviews we saw that, you know, people, sometimes you type in a yellow jersey and you want to see a yellow jersey immediately.

Nishikant Dhanuka [00:09:05]: You don't want any follow up questions being asked and everything. So there are it suits in some environments, but not everywhere. Then second is so you should design your product for non tech users because most of the users who would be using the product is non tech. And we saw many examples of that. So for example, initially we had these onboarding cards and for people who are using ChatGPT and Gemini, it's very common to see these onboarding cards and they use it. But we see a lot of, very few of our customers are using onboarding cards. And then we shifted to a different strategy which was more clear and we saw a higher kind of uptake. And then finally, don't make your users think too much because if you have a chat interface, then that chat interface can do a lot of things and a user needs to discover.

Nishikant Dhanuka [00:09:58]: So an example is Smart Compare. So you saw this example with Smart Compare that you can actually in the message type it to compare, but people don't know about it. If there's a button for Smart Compare, then it's in front of them. So based on all this we created, so we curated and we created a second version of Magic where the idea is that we bring the experience closer to what users already understand and they're familiar with. So again, let me walk you through a quick demo. So it starts now with a search bar. With a search bar which is much more powerful, but it's also familiar to the users. And let's say in this case I want to buy a smartwatch, so look for that.

Nishikant Dhanuka [00:10:41]: And a couple of things are happening on the screen. So there's something called Magic Helper. So it's asking me questions to refine my results. Then responses with LLM and agents takes time. So I Have something. I have a section there called instant results. So this also creates engagement to the users who are still waiting for, let's say, more intelligent responses. And then I get my responses, which I can scroll down vertically, but at the same time I have things that I'm familiar with.

Nishikant Dhanuka [00:11:16]: So there's a sorting button, so I can sort by price, I can sort by relevance, and so on. And I can also explore. So there's something called smart groups. So it's a balance between exploration and exploitation. So there are these smart groups, which is showing me complementary items as well. So in this case it's also showing me a charger and I can click on that and it takes me in a different direction on a new search, which is about charges. For smartwatch, there's another thing that we built which again uses LLM is contextual tags. So we highlight.

Nishikant Dhanuka [00:11:48]: So we try to understand what features are important for the user and we try to, without even clicking on that, we try to make it. Then there's a feature which is refine. So for example, here you see on the screen that now I'm trying to refine the search, but instead of using traditional filters, I just type in that I want to see only smartwatches in Warsaw and only under 500 zloty. And it makes it feel like traditional filters, which gets shown there. And now you get these results. And if you look closely, so these listings are from Warsaw. And if I sort them, I can see. If I sort them by price, I can see the highest is 500.

Nishikant Dhanuka [00:12:32]: That's not it, because sometimes you also need filters which are, let's say, unconventional, which are not defined as filters. So maybe in this case I want. It's important that I want a smartwatch with a heart rate monitor or ECG function. And again, to remind you, this is a secondhand marketplace. So these things are not always so these are real users who are uploading and people don't fill all the information. So maybe this information is not defined as a filter, but it's present somewhere, either in the description or images, so you get the results. And again, the idea is that the magic makes it look like a traditional filter, which you can unselect at any point and you can go back to your previous results. So that's one scenario.

Nishikant Dhanuka [00:13:18]: Now let's take another example, again with image. So it's a search bar, but you can also upload an image. So again, in this case, I go to my photo library. There's this nice toy car that I like for my son and I Want to buy something similar, but so I like this, but I want it in a different color. I don't like yellow, I like red. So in this case it's a visual search but with a modification and that triggers. So again there's an agentic flow happening behind the scenes, but this triggers a different pipeline and I get the results in red and I can keep refining my query. Right.

Nishikant Dhanuka [00:13:59]: So in this case maybe I am only interested in Jeeps. So that particular car of. So I say Jeep only and then it defines the results and so on. So that's kind of the second iteration and here's a sneak peek. So this product will launch it next week and we'll a b test it. But yeah, so this is the second iteration and now for the next part, I want to invite Beatriz again on the stage and to explain a little bit what's happening under the hood. Beatriz.

Beatriz Ferreira [00:14:35]: Glad to be here. Talking about magic and what's under the hood of all of these. So yeah, if we go to the next slide, we show kind of the high level architecture of our agent, which is really at the core of magic. Are we showing the next slide? Okay, thank you so much. As any AI agent, our agent is able to interact with the user in natural language and understand the user's intent and requests, then autonomously perform tasks to be able to perform according to these requests and then finally get back to the user. So in our case, our brain has access to several models through a model router and we are using both a combination of both best in class commercial LLMs and also some internally fine tuned LLMs according to the use case that we are tackling. And then this agent is able to call and has access to several tools according to the generated plan. Within these tools we can see web search as Nishi showed in the demo, text search and visual search and for instance URL parsing among others.

Beatriz Ferreira [00:15:57]: Within these visual search and text search we include our retrieval and re ranker flows. Our agent is also able to deal with very different kinds of requests as again Nishi showed in the demos, ranging from very broad to very specific. Another example of a broad request is like find me a gift for my mother's birthday. And so in this case our agent will generate a plan in which it will ask clarifying questions in order to iteratively refine the search and understand the user intent. And when it reaches a point in it's okay, we can find and show display relevant items to our user, it will trigger a parallel, so calling parallel. So do several calls to Our text search or visual search tools in order to then present to the user a lot of results in a complete way and showing different options. We also highlight our steps of guardrails and lycoge detection. We had to iterate a bit on these.

Beatriz Ferreira [00:17:12]: For instance, for guardrails, in the beginning we were a bit more permissive, but then we found users that were using magic to fulfill other requests and just extract, exploring and buying on olx. So there's always some work needed to be there and some iterations there. In the end, we stream back all these responses back to the user and we also do this in a streaming way to try to alleviate the latency issue that we get from passing through all these steps and calls. Finally, another very important topic that we explored is evaluation. And of course we did this with specifically for our search system. And we use techniques like LLM as a judge and human in the loop and they were really important to help us gain very interesting insights about our search system and also to guide the development of the tool. If we go to the next slide, we will show you a bit of the secret sauce of our search. So we should remind ourselves that we are talking about the shopping assistant.

Beatriz Ferreira [00:18:26]: And I really like this quote from a colleague from the project that says your Gen AI shopping assistant is only as good as your search. And this is really true. So this search is really a critical component of our system. And a key enabler of our search system is using hybrid search. So this is really important to keep the user engaged throughout the journey and then to make sure that we find precisely what the user is looking for so that we show to the user the products he's looking for. So hybrid search is a combination of both keyword search and semantic search. And it really is the best of both worlds because both can have some limitations. And we show here a very nice example.

Beatriz Ferreira [00:19:16]: And these are really real examples from our catalog showing. For instance, if I query for Echo Marine blouse in traditional keyword matching search, I only get one result because only this product has exactly these keywords. But if I use semantic, it will understand the meaning of the query and it will get blouses of similar blue colors. I will get all these results. With hybrid we can combine both. This is a key enabler for search in Magic. Then another very innovative feature of us is visual search. So we have this multimodality.

Beatriz Ferreira [00:20:02]: We can either search via text, visual and also voice. So in visual search, imagine that I saw this dress online and I'm looking to find something similar to this on Olx I could enter a dress description. So like this. So these are again real screenshots from Magic. And imagine I describe it like this. This is a blue dress with the V neck and white sleeves. And this would be the results that Magic would show or a search system would show. And these look very different from what I was looking for.

Beatriz Ferreira [00:20:37]: And as we say, image is worth more than a thousand words. So we have our visual search and we can upload an image and find visually similar products to this image. So if I upload the image of a dress, of this dress in specific, you could see the results there with five identical. We'll see very visually similar results shown, but we also have a different flavor. As Nechi also showed that I can add a modifier to this search. For instance, imagine that I don't want this in blue, but I want this in red. This goes beyond color. We can also change other attributes, for instance the shape or like the sleeve length or even for other contain different attributes.

Beatriz Ferreira [00:21:27]: Our agent will understand and will make a search that is grounded on the image, but we'll apply the modifier on top and so we have the relevant results that we can show to the user. Yeah, back to you Nishi for the go live and challenges that we face. Thanks again.

Nishikant Dhanuka [00:21:44]: Thanks Beatriz. Yeah, Slides. Yeah. So the product is live. So we are currently a B testing it. And the way we a B test is so on the left hand side you see that the user is on OLX platform and they are searching. It's your regular Olix search. And then kind of a button appears at the bottom and if a user is interested they click on that button and it actually takes their entire context from Olix, feeds it to Magic and it already pre prompts Magic for results which might already be relevant for the user.

Nishikant Dhanuka [00:22:29]: So this helps also in the cold start problem that we saw earlier that we described with the onboarding cards. So yeah, and these are some of the key learnings. I'll go through them quickly. So we see, you know, one of the big things that we see is that it's also an adoption of a new technology from users. So users shift their behavior. So if you compare their behavior on search bar versus in a chatbot, we start seeing that let's say 1/3 of people are starting to write, give more context and write in natural language. But 2/3 of them are already using it, are still using it like a keyword search and not giving it context. And so the journey continues.

Nishikant Dhanuka [00:23:12]: So we are building the product side, but it's Also the adoption from the users and then finally wrapping up. So these are some challenges that we still need to solve. Agents. So there's a lot which is happening under the hood, many calls. So it takes time. So there's a latency of responses, 5 to 10 seconds, which for example, in a typical search you would expect something like 300 millisecond, 500 milliseconds. So I need to work on that cost and ROI. So agents.

Nishikant Dhanuka [00:23:40]: So though LLM calls are getting cheaper, but agents make a lot of these calls so that's expensive scalability. So all the tools that the agent called, they need to be scalable to support the entire traffic and that's a hard engineering problem to solve. Batries already spoke about some of the guardrails that we are implementing for security. And then finally monetization. Right. So these new experiences, they also demand a new model for monetization. The previous model doesn't work, so we are still actively working on all that. Thank you.

Nishikant Dhanuka [00:24:10]: That's it.

Demetrios Brinkmann [00:24:12]: All right. Well Nishi, there are a lot of questions for you in the chat so I want to get right to them. We are going to go with the one that has the most upvotes. First, what was the most powerful gen AI addition to traditional search that led to increase in business KPIs like number of TRX or so.

Nishikant Dhanuka [00:24:34]: So what was the most powerful addition feature of Gen AI which got added to traditional search, upped the KPIs. Yeah. So that led to an increase in conversion. I would say that. So we track all the different features and we see different conversion for different categories for each of these features. But for example, I'll give one example. So smart comparison, that's really powerful. So it's a simple idea that you know and it's quite easy to do with an LLM call.

Nishikant Dhanuka [00:25:05]: Right. You don't need a very sophisticated agent. But that simple idea where people want to compare a couple of products and you get a natural language response. We received a lot of positive feedback on that.

Demetrios Brinkmann [00:25:16]: Do you use just LLMs or also traditional ML models?

Nishikant Dhanuka [00:25:20]: Both. So we combine. We combine. So. So OLEX already has a search architecture, right. So there are these models which is defined and genai we use. So the way I think about gen AI is it's a presentation layer. So both at a presentation layer.

Nishikant Dhanuka [00:25:37]: So understanding user intent and presenting the results back to the user and what Bitri spoke, for example, hybrid search. But aside from that you still need your regular components, LTR framework, LTR ranking and so on. So we kind of yeah, it's a combination.

Demetrios Brinkmann [00:25:53]: So another one for guardrailing. And maybe this is for you, Beatriz. Are you using any framework services or self hosted models?

Beatriz Ferreira [00:26:02]: Yeah, so we have our own abstraction. So our own frameworks. And yeah, we are using best in class models for guardrails. Of course, lots of that. It's in the prompting itself, so making it right to the context of our agent to olx. So yeah, it's about. It's like the context of the prompting and then the model and our abstractions and framework.

Demetrios Brinkmann [00:26:25]: Okay, so the majority of the heavy lifting is done in the prompt, not after, when it's outputting.

Beatriz Ferreira [00:26:33]: Yeah, yeah.

Demetrios Brinkmann [00:26:35]: And then how are you going back and evaluating that it's not giving any information that it shouldn't be?

Beatriz Ferreira [00:26:43]: Yeah, so we have several ways to do that. So for instance, we do that offline afterwards we revisit our conversations and evaluate what went there, right and wrong. But yeah, of course before putting anything live, we do lots of testings and we heavily test the response of the LLM. But right now, yeah, for like those specific use cases, we have that very well framed. And yeah, a simple example, it's working well. Go ahead.

Nishikant Dhanuka [00:27:15]: Just building up on what a simple example of that is. So sometimes guardrails gives false positives. So we saw that someone uploads an image and then they don't write anything, then the guardrail blocks it. And a lot of people do that. And in our testing when we realized that then we could change the prompt accordingly, then we started evaluating. So we keep discovering these cases, we keep adding that to our test scenario and then we have a kind of an evaluation pipeline and we keep fixing things.

Demetrios Brinkmann [00:27:43]: Yeah, the long tail, it's almost like that is going to take up a majority of the work because as they come up, you have to add them in. And add them in.

Beatriz Ferreira [00:27:51]: Exactly. You're never prepared for what the users are doing with your tool.

Demetrios Brinkmann [00:27:57]: So last one for you all. How does OLX Magic understand that there's sufficient info in the entire user request and no more clarifying questions are needed?

Beatriz Ferreira [00:28:11]: Yeah, I think again that's like part of how we have our smart way of our prompts and our system, almost a system flow. So there are a lot of components like this very sophisticated pipeline that goes. So receives the user input, goes through this pipeline and then it understands that, okay, I have sufficient information to provide relevant information to the user. And then of course we iterate on that. But again, I think it's like the system as a whole and our features, the way they are connected that is able to make everything work starting from the clarifying questions. Also refinement. We keep building this context for the user that then we reach a point that okay, by our experience this is good to send to our searching Alex.

Nishikant Dhanuka [00:29:08]: But any feel free to add anything just one point. So for example, when we started we used to ask a lot of clarifying questions but we also kind of gathered that verbosity. Yeah feedback from the users that sometimes this clarifying questions comes in the way. So it's continuous optimization. You know, we try to balance not asking too many questions, but asking the right questions. And it's not always about just clarifying questions because maybe you can through an interface you can show a group of category to the user that user can select. And that tells you without asking a question that the user it's still broad user has not made up their mind. So yeah it's again a process of testing, learning continuously.

Demetrios Brinkmann [00:29:51]: Folks, this was very cool. It's time to go though. You got to get off this stage.

Nishikant Dhanuka [00:29:56]: Thank you everyone.