From Robotics to Recommender Systems

Miguel explains the limitations and considerations of applying ML in robotics, contrasting its use against traditional control methods that offer exactness, which ML approaches generally approximate. He discusses the integration of computer vision and machine learning in sports for player movement tracking and performance analysis, highlighting collaborations with European football clubs and the role of artificial intelligence in strategic game analysis, akin to a coach's perspective.

Join us at our first in-person conference on June 25 all about AI Quality: https://www.aiqualityconference.com/(url)

Miguel Fierro [00:00:00]: I'm Miguel Fuero. I'm a principal data scientist manager at Microsoft, and I like black coffee.

Demetrios [00:00:10]: Welcome back to the Mlops Community podcast. I am your host, Demetrios. And today, talking with Miguel, we actually had a bit of a moment after I stopped recording, and he told me that he changed the direction of his career because of the value that he saw in recommender systems. And this comes hot off of the tail of an episode that we recorded a few weeks back with Sankit on the Spotify recommender system. But he read a report from Amazon, and I think this is the Amazon shop, not AWS, and it mentioned how recommender systems were, 35% of revenue generated. So basically, just add a bead to that 35. And it was like $35 billion that recommender systems generated for Amazon. And when he saw that, he was in the computer vision and robotics field and he said, I got a pivot.

Demetrios [00:01:16]: This is too big and there is too much value that is happening with recommender systems for me to leave it alone. That was just one of the gems that he dropped in this conversation. So the last piece on that recommender systems example that he said that I am going to continue to replay in my head is how, at Amazon, just tally up the amount of data scientists that work at the company and think about the ROI that they drove. $35 billion worth of revenue generated from these, let's say maximum. There's 10,000 data scientists at Amazon, which I doubt there are. But you do the math. I don't do public math. I don't even do back of the napkin math.

Demetrios [00:02:11]: As always, if you enjoy this episode, just share it with one friend. Let's get into a conversation with Miguel. Let's dive into robotics. And I think, for me, one thing that I constantly think about when it comes to robotics, like deep learning use cases or just robotic computer vision systems in general, is how much of these use cases are similar to, quote unquote, traditional ML and how much are different. What are some things that you can take from the deep learning or computer vision use cases? The robotics use cases, autonomous systems use cases, all that. And they also work with traditional machine learning. And what are some things that are totally different? And you're like, nope, you cannot. That has nothing to do with the traditional ML.

Miguel Fierro [00:03:11]: That's a very interesting question. And I can start by saying that a lot of roboticists will say that in many use cases, you don't even need machine learning. They're not even talking about deep learning. So if you think about the. So Boston dynamics is arguably the best robotics company, right? And they've been running for several decades, and until very recently, they don't even have machine learning. Like, everything they had was physics from the 18th century. So it's Lagrange Newton. It's a very, very deep understanding of the equation of motion because that's all it is.

Miguel Fierro [00:04:03]: So it's the physics that people will study in first year of an engineering school or math or physics school, but a really, really deep understanding of that. And they use a technique called model predictive control that basically is a very simple technique that allows you to, in real time, you compute the movement dynamics of the robot, and it's all the time kind of doing a prediction and then getting information of the posture from the sensors and stabilizing the system. And that is done in c. Like, really, you need to recode all that in a very low level program. And that's what they've been doing for decades, actually. And then, of course, if you start thinking about computer vision, not only the force and the control and the stability, which is actually one of the biggest problems in robotics, maybe the most important problem in robotics, the stability, the postural control. Actually, my thesis was precisely on that, on post trial control. Right? And so if you start thinking about the computer vision, that's a completely different thing, to the point that in many research centers, typically where people do a PhD, typically go to a university right now, I think nowadays you can even might be able to do some business in companies.

Miguel Fierro [00:05:56]: Traditionally, it's universities. So interestingly, most labs, they decouple these problems. So you have a lab where you have a human robot, or you have what is called a mobile manipulator, which is a robot that maybe has hands, but they have a platform with wheels that moves, so they don't have the problem of legs, the stability. They might use the same platform, but if they use computer vision, it's a completely different team. And maybe the thesis or the research line is more about AI, more about machine learning, deep learning and that kind of things. But for example, when I. So I did my PC between university in Spain called Carlos Delfero and King's College in London, and in both cases, the teams were specialized. What people do typically is they kind of decouple the problem.

Miguel Fierro [00:07:03]: So that's why, for example, that's the reason. That's the main reason why you probably hear that Boston dynamics, they kind of started machine learning very recently, because the big problem that they had to solve is the stability problem. And, you know, they solved it amazingly and just to finish this point, something very interesting that is happening is these kind of. There are two school of thoughts in robotics, particularly humans, which is kind of what I spend a lot of time researching, which is the type of actuators. You can have electrical actuators, but you can have hydraulic actuators. And until I think, two months ago or something like atlas and all the robots, they had hydraulic actuators. I think the dog also had some electrical. But, you know, there is this kind of difference.

Miguel Fierro [00:08:10]: Right. And it all has to do with the torque and the ability to control. Right. Like the benefit of hydraulic aggregators is that they can produce a higher torque. So if you are in a situation of unstability, you're not stable like everybody have seen all these people kicking the robots. So a hydraulic actuator is very useful in that case because you need to be able to stabilize very quickly. And that stabilization, actually, it creates a lot of forces and a lot of inertial forces that you need to compensate. And doing the same thing with an electrical actuator is much more difficult.

Miguel Fierro [00:09:09]: And if you've seen the, actually, if you've seen some of the demos of the new robot, the new Hamari robot of Boston dynamics, I was impressed that they don't use electrical, they don't use hydraulic actuators, they use electric actuators, but you haven't with this new robot, they are not doing the back flips, they are not doing all that stuff. So I wonder, and I'm curious to see if they will be able to do, to have the same stability skill with these electrical actuators that they are cheaper. It's easier to fit these, to power these actuators. So, yeah, it's a very, very interesting difference.

Demetrios [00:10:04]: So if I'm understanding this correctly, you go with the electric ones because they're cheaper, because they're easier to implement, but they don't have that torque.

Miguel Fierro [00:10:12]: Yeah, exactly. It's not that easy to, it's not that easy to do, particularly when you have a very, a very strong movement. Right. Like that is where the electrical actuator is more difficult. It's much more difficult to don't try.

Demetrios [00:10:35]: And so this is also fascinating because you're mentioning Boston dynamics just started throwing some machine learning into what they're doing. It is that almost repetitive mantra that we hear on this podcast quite a bit, where if you don't need machine learning, there's no need to try and use it because you're going to make your life a lot easier. If you can do it, without it. So it's almost like putting machine learning on your robot is just about as useful as programming in string theory physics to the robot. It's like, no, we just need this 18 hundreds newtonian type physics that's going to help us with force and help us with movement.

Miguel Fierro [00:11:18]: Yeah, I can give you an example.

Demetrios [00:11:22]: Right?

Miguel Fierro [00:11:22]: Like, if you ask the traditional, like, old school roboticists, like the control engineering roboticists, they will say that they don't need to use that. And for example, I remember when I was doing my PhD, I was researching in post trial control. So I had a human robot, and I needed to do postures, different posters. And one of the things I tried was to use a neural network to compute a movement and to do what is called the inversion direct kinematics, which is basically how you move the actuators to produce a specific trajectory in the hand or in the leg. Right. And what happened to me, I started looking into that, and I realized that it made no sense because we know the analytical equations to solve that problem. So we know the exact equations and we can do the control loop. So a machine learning solution for that problem, it will be just an approximation.

Miguel Fierro [00:12:38]: It's like, you already have the formula, right? Like machine learning, what it does is just. Is kind of doing an approximation of the formula. But if you do have the formula, why are you going to need that? But the thing is, you know, obviously, you know, like, understanding the physics and, you know, like, the formulation for something as complex as a human robot is challenging. I find it very challenging, and I know it is challenging for everybody because very few people actually solve that problem. So that's why a lot of people don't use machine learning. And even to the point that instability and control for robotics, there are two kind of models. What is, what is called the compressed mass model, say something called zero moment point. But basically what it means is that you create a model of the robot that is like an inverted pendulum.

Miguel Fierro [00:13:41]: Like, imagine you assume that all the mass of the robot is in the center of gravity. And then when the robot moves, you use control algorithms to actually make the stability. And the other approach is the distributed mode. That actually is, you take all the different joints, all the different masses of the robot, and you model that in equations. So you have many different questions, as many as the masses and joints you want to put into the robot. And then you solve the equation of motion that actually, there are two formulations. You have the formulation of Newton, which is the famous force equals mass time acceleration, the famous formula of Newton. There is another formulation of Lagrange based on energy, like potential energy and kinematic energy.

Miguel Fierro [00:14:37]: So basically you have these two formulations, but at the end, it's something that is solved in the 18th century or something like that, but it's really so complex that a lot of people use this compressed way of modeling the robot, which is much easier to model in math, but is less accurate.

Demetrios [00:14:59]: It reminds me of basically like a mono repo versus microservices.

Miguel Fierro [00:15:03]: Yeah, exactly. It's the mono repo. Yeah, the famous Monorepo.

Demetrios [00:15:09]: Totally. So excellent. I really appreciate you breaking that down, because I had no idea, and I do know that there are a lot of robotics companies out there that aren't using ML, but then there are quite a few that are incorporating a lot of different ML and computer vision use cases into their mix. And so it's interesting to see how complex things are. Like you were saying, yeah, we could use some ML in this use case, but if we already know it and we can use it, we can map it out properly with a heuristic, then let's just do that, because otherwise the ML is going to be an approximation and it may or may not get it right. So that's cool to think about. And a nice little thought experiment on whether or not it's totally used or needed. I do know that you also messed around when you were doing your little robotic stuff with a computer vision football use case.

Demetrios [00:16:11]: Can you tell me more about that?

Miguel Fierro [00:16:13]: Yeah, so I'm a football fan. You know, in the states, people call it soccer. Like in Europe, we college football, and that was actually, that was. That was another case for. It wasn't during my PC, it was later, it was when I did Microsoft. And, you know, there is this big trend of sport analytics, and I think sports are going to professionalize in the sense of being more data driven. So basically, the idea was to track the different movements of the different teams and the ball to basically get some analytics. And I think at Microsoft, there has been some very interesting work around that area.

Miguel Fierro [00:17:07]: We partnered with some football clubs in Europe. I know Google also, they partner. There's a paper by DeepMind, actually, with Liverpool, Liverpool in the UK, they are doing this kind of analytics. And even Real Madrid, I know they have a CTO that is super into, into these analytics, and that includes a lot, a lot of computer visual object detection, object tracking, and I think a lot of these sports, not only football, but tennis or basketball or many other footballs, they are going to start professionalizing more, and maybe we'll see in the next ten or 20 years that they have these professional clubs, they have a lot of funding. They will have their own data science team and they will get analytics. And this ideas will be shared with the coaches and specialists in the team for improving the, the performance on the team. And also, another thing that I think is very trendy is data science or machine learning for the health monitoring of the athletes. Again, typically these people, they have the best doctors, and if you hurt your knee, normal people will stay in bed like six months.

Miguel Fierro [00:18:53]: So these people is like, no, no, no. These people is maybe one month or something like that. Right. So I think, you know, getting this combination of really advanced medicine with new technologies like data analytics or AI, I think is really, really powerful.

Demetrios [00:19:15]: And what are some of these use cases with the object detection and object tracking? I know I've seen a few that are really cool. And we have a few people in the mlops community that also work with basketball and making sure that you can get a proper readout of the players, how far they're running, if they're passing, if they're making mistakes, that type of thing. But it almost feels like by saying, okay, let's add some object detection to the videos, it's starting with the tech and not really starting with the problem. So what are, are the problems there?

Miguel Fierro [00:19:53]: Yeah, so I think it's mostly strategy. I remember there is a show that it was the spanish national team when they won the World cup. They were actually talking about one specific goal and one of the defenders said to, he was a corner, right? So he basically said to the other guy, he said, you know, I notice that they, you know, the other team, they are not defending in the middle of the, in front of the goalkeeper. Right. So if you put it to the center and you try to pretend there. And what I'm going to do is I'm going to go from back, I'm going to run and I'm going to jam and I'm going to try to get a goal with my head. Right? Yeah. And then, and then actually the guy, the guy did it and they scored.

Miguel Fierro [00:20:56]: So, like, the way I think about it is, you know, obviously these guys are professionals. They, they've been playing football all their lives and they, they are the elite people, right. And they don't, they are not only. It's not that they are really skilled in playing, but they understand the game as kind of like as a coach. Right. So I can imagine that that's the kind of information that an artificial intelligence can also understand. Somehow, and I don't think we are close to be able to understand that. I also don't think that there is enough funding to reach that level.

Miguel Fierro [00:21:39]: Maybe some clubs start investing in that and in ten years we have an advisor that advise the coach and can do some strategies, but in a similar way that these people kind of saw a hole in the strategy of the other team, I can imagine an AI can do something similar.

Demetrios [00:22:04]: Those real time analytics. Yeah, it's incredible to think about and it's also really cool to recognize that we're still very, very early in the abilities and the capabilities. And I know with american football they're also doing a ton. And even when you watch american football games on like Amazon these days, you can see that they'll break it down and you get those real time analytics just piped onto your screen and you get a bit of a taste of what I imagine the coaches are seeing. And so that's awesome.

Demetrios [00:22:43]: All right, real quick, some words from our sponsor Zliz, and then we'll be right back into the program. Are you struggling with building your retrieval augmented generation, aka rag applications? You are not alone. We all know tracking unstructured data conversion and retrieval is no small feat. But there's a solution available now introducing Zliz cloud pipelines from the team behind the open source vector database Milvis. This robust tool transforms unstructured data into searchable vectors in an easy manner. It's designed for performance. Zilliz cloud platforms pipelines Zilliz cloud pipelines also offer scalability and reliability. They simplify your workflow, eliminating the need for complex customizations or infrastructure changes.

Demetrios [00:23:38]: Elevate your rag with high quality vector pipelines. Try Zilliz Cloud pipelines for [email protected]. you can find the link in the description. Let's get back into this podcast.

Demetrios [00:23:50]: Alright, so then let's go into what you are most known for now. And I think at Microsoft your nickname is Mister recommenders. Is that true?

Miguel Fierro [00:24:01]: Well, I mean, like, I get a lot of, I don't think they call me that, but I definitely get a lot of requests to support teams that are doing recommendations both internally and externally.

Demetrios [00:24:18]: And what are some of these patterns? Because we, as I mentioned before, we hit record. We literally just had sankeet on here from the Spotify recommender systems team and he was talking about how they had set up their vector stores and their embeddings to be able to constantly retrain you as a user and your embeddings to make sure that it's the most up to date and gives the best music suggestions. But then it also is powering many different teams. So it's like the vector store. Anybody can grab those embeddings of the user and then plug them into whatever machine learning model they are doing. So whether it's search or it's discover weekly, or it is the actual next song that you're going to play, all of that is quite useful because they can transfer that learning. And there's one central spot where they have those embeddings. Are you seeing common system design when it comes to recommender systems? Because I also know that there's like feature stores for very real time recommendations.

Demetrios [00:25:27]: And it feels like the vector store and the feature store, depending on the use case, can be very complimentary or you may only need one of them. What's your gist on all this? Because I'm sure you've seen a ton.

Miguel Fierro [00:25:40]: So I think the cool thing of being at Microsoft, in comparison to being in a place like Spotify or Netflix or TikTok, which they have amazing recommendation systems, is that in Microsoft at least I've seen many different use cases. I've seen, and I worked on like a team that they have a recommender team. So they have many different recommenders and they've been involved in that over the years. And I work with companies, for example, that they are starting to integrate maybe their first recommender or the first few recommenders. And you know, it is like the use cases for. So these two kind of teams are completely different. And basically it all has to do with the maturity of the team, the number of people, the amount of funding each of these teams have. What I would say is that in general there are three types of architectures in recommendation systems.

Miguel Fierro [00:26:54]: The first one is what we call the batch architecture. You get your data, you train your machine learning model, and instead of taking the model, deploying the model and score in real time, what you're going to do is you're going to score for each user and you're going to store these recommendations in a database. So you can think of database of the first column is the user id and then you have all the recommendations. And then you retrain this every day, for example, or depending on the company in the use case. But imagine it's every day. Now, what is the benefit of that? The benefit is that from your front end you are just querying a database. This is very easy to get a request time, very, very low request time below ten milliseconds in some use cases. For example, in e commerces or it is very, very important to create APIs that have these requests on.

Miguel Fierro [00:28:06]: What is the problem? The problem in that case is that you're not understanding the needs in real time. I always put the example of the Microsoft store. So if you go to Microsoft has a store, Microsoft.com and you can buy from office like Microsoft 365, but you can also buy Xbox, you can also buy games. So what could happen is that I've been buying software or I've been buying, I don't know, maybe computers and mouse. So what the machine learning, what is going to do is if I train it is going to recommend things that are related to that. But what happens if I go to the website and I start looking at an Xbox with the batch architecture, we won't be able to understand that. Miguel, even though Miguel bought a computer and software, Miguel right now is interested in an Xbox. So you, you should recommend, you know, maybe, maybe games or something like that, right? So that's the, that's the what is called the real time architecture.

Miguel Fierro [00:29:28]: So in the real time architecture what you do is you train your model and deploy your model. So you need some kind of cluster that is able to hold this model in real time. If you are using some kind of deep learning model like LLM, obviously your request time is not going to be as good as if you are just querying a database. So you'll need to start doing some gymnastics to be able to reduce this request time. And then you have the mixture of the batch in real time that it has different names. Some people call it the hybrid architecture, some people call it the two step recommender architecture. Recall, re rank architecture. But the idea is that what you're going to do is you're going to do an initial training of a model and you're going to generate some candidates.

Miguel Fierro [00:30:27]: And maybe these candidates are 1000 candidates for each person and you're going to store these candidates in a database. And then in real time what you're going to do is you're going to create another model that what it does is re rank the candidates for each person. So in real time what you're doing is you're taking one user with a list of items that maybe are thousand items. These are the candidates. And then what you're doing to do is based on the real time information, you're going to reorder that list and show them maybe the top ten or the top five or the top one, something like that.

Demetrios [00:31:11]: Yeah, that's kind of what I was doing.

Miguel Fierro [00:31:14]: Hopefully from the point of view of developing this architecture is much more difficult, but it's actually, it has the best of both worlds. Right. It's faster than the real time, but it's not, you can get a really, really good solution in that architecture.

Demetrios [00:31:35]: Yeah, it's funny you mentioned that. And that's exactly what Spotify was doing. And Sankit was saying how most of his time goes into making sure that those candidates are accurate. Yeah, because that's where those candidates and that evaluation of candidates, and we all, I'm sure, know this now too, because of rags and how popular that has become. That retrieval step is so important. And for a company like Spotify, I think he mentioned they had 100 million items that they could recommend at any given time. And so for him it was like, okay, out of all these hundred million, which ones are we going to make sure that we recommend or which one? How can we make sure that it is the most pertinent for the person that is looking at it? And, yeah, so this is fascinating. And it also makes me think about features and feature stores, feature platforms and also just the embeddings and how when you're looking at these different ways of doing it, you have batch, you have the real time, or you have the hybrid.

Demetrios [00:32:44]: Have you seen common ways of setting up your vector stores and setting up your feature stores if you have it, or feature platforms? And what does that look like?

Miguel Fierro [00:32:56]: So yeah, for us, basically everything I worked on was on Azure, right? So Azure has evolved from the way they do their vector stores. Now we have the Azure ML feature store and vector store. That is pretty advanced. I think these guys, they have been doing databases for decades, the famous SQL server. I think basically all the team around these databases, what they've been doing is they've been evolving the database for the new use case. And basically the new use case, the most important use case right now is AI. So I think there is this component of vector stores and feature stores and all the stuff that it helps on the accuracy. But at the end of the day, one particularity of recommendation system that I think is very different to other areas, you know, like NLP or computer vision, is that unfortunately in recommendation systems there is not a clear, better solution.

Miguel Fierro [00:34:25]: And that is very, that that's not good, right? Because you know, if you go, if you go to computer vision, you know, everybody's doing right now either convolution networks or vision transformer and that's it, right. In the old days, you know, some people might remember there is this single sift key points, right? And some other, and some other structures. Like, nobody's using that anymore. In, for example, in NLP, you have LLMs, right? Yeah. You know, like, right now, nobody, I haven't seen anybody using convolution networks for NLP or very few people doing lstms. What happens in these areas is that you see that new algorithms make the older algorithms obsolete. Basically, you just can try a few things and basically you can focus on optimizing the few options that you have and go from there in terms of recommendation systems. That's not true.

Miguel Fierro [00:35:43]: In recommendation systems, depending on the use case, depending on the data, you can use linear models and simple models. You can use even. No machine learning at all.

Demetrios [00:35:56]: Sure.

Miguel Fierro [00:35:57]: From there, too, the super advanced llns. Right. So it's very, it is quite difficult to kind of fine tune your model to make sure that you are providing value. Right. And another of the, and that's a very big difficulty.

Demetrios [00:36:20]: Right.

Miguel Fierro [00:36:21]: From, of recommendation systems. So this is a problem, and again, this is a problem that, as I mentioned at the beginning, if you are a very mature team, you've tried so many things that you kind of know, like, maybe you have, like companies like Spotify or Netflix or Tickler, they have, you know, different recommenders for different kind of content. But if you're starting from, if you're an e commerce, for example, or if you, if you are any other company that you're starting, you need to go through all these different tests and different algorithms to figure out which ones work for you. So, you know, the same, for example, a company like Nexus that you mentioned, they have millions of songs to an e commerce that they, they may have in the thousands of items, right? Yeah. Algorithm completely changes. So one of the problems I've seen recommenders is being able to select the right algorithm. And then the other big problem is that at the end of day, when you are using recommendation systems, what you're trying to do is you're trying to improve the user experience, increase the active use usage, increased revenue, increase conversion, stuff like that. So you need to do an A B test.

Miguel Fierro [00:37:52]: So basically, the processes, the data engineers prepare data, the data scientists create a model that you have either data scientists or ML engineers or different, different companies use different names, but somehow deploy this model in any of the architecture that we mentioned. And then there is this process of a b testing. And then again, like the A B testing, you need a platform as a platform to be able to take this model. And the A B tests, you know, you need some time for the A B test to be significant for the statistics to be sound and to be accurate. Right. So maybe you need one week or two weeks for each experiment, right? So if you think about it, every time you do an experiment is very expensive in terms of time. So maybe you need to wait two weeks, right? Yeah. And one thing that happens in experimentation and people who is in a B test in experimentation know is like the average of industry is that 80% of the experiments fail.

Miguel Fierro [00:39:06]: So you're going to try a new recommender, and on average, for most industries, 80% of the time you're going to fail. So it is a challenging, it is a challenging situation. And I believe that of AI, of the AI solutions, the personalization idea is one of the most powerful solutions and probably one of the ones that have the highest return of investment. But getting to this return of investment, it is not easy. So one problem that I've seen in many companies is that they don't fund enough, the data science team of recommenders, because it is something that is not just, you just get a few trials and then suddenly you're doubling your revenue. So you need to spend some time doing this. And if you're a company and the decision makers, they maybe don't understand AI very well, and maybe they don't understand recommendations on personalization. They don't see the value.

Miguel Fierro [00:40:27]: But you look at, you look at, for example, companies like Spotify or TikTok or Netflix or Amazon that they know they spend resources in these teams and the return of investment is crazy. Yeah. Just to give you some numbers, the 35% of the revenue of Amazon.com comes from recommendation systems. So if, so think about, think about that, right? Like if last time I looked at, I think they were doing over 100 billion in revenue. So 35 billion is basically a team of people like us, right? Like, if you think about it, and I always put, when I pitch this idea of recommendation systems at Microsoft, one of the things I do is I go to the tree of the VP. So you have Satya SEO, then you have the different vps and different products and you have the sales organization. It's like a tree, right? And then I ask, okay, so who of these vps make 35% of the revenue? They are not that of Microsoft, right? There are not that many. There are not that many.

Miguel Fierro [00:41:53]: And then it's like the question is, okay, so if you take the two or three people who make, you know, this number of billions, how many people are there? And the numbers in the tens of thousands of people in general. So to generate 35 billion, 40 billion you need in the tens of thousands of people. Well, I don't know exactly the number of people working in recommendations on Amazon, but I'm sure it's not in the tens of thousands of people. So the ROI that you can get from recommendation system is insane.

Demetrios [00:42:33]: Yeah. Wow. I had never thought about it like that. And this brings me to another point which I think you brought up, and it's how expensive experiments can be, just in terms of not expensive as far as headcount goes, but expensive in terms of time. And also the fact that 80% of these experiments fail, it makes it really hard to justify doing another experiment.

Miguel Fierro [00:43:01]: Exactly.

Demetrios [00:43:01]: So how do you as, because I know you're leading teams right now at Microsoft. How do you get across to leadership that it's worth making another attempt or trying another experiment and getting that buy in to have leadership understand that and then also not see you as just a cost center?

Miguel Fierro [00:43:25]: Yeah. So that's a very good question because that's precisely a big part of my job is basically talk to the leaders of my organization and even all these organizations in Microsoft to actually help them see the potential and the opportunity. And the only, like, at least what I've found that is more useful is when you come with results. So basically my pizza. Hey, I'm Miguel. My team does recommendation systems and I've done this number of million dollars with recommendation system. So this pitch and this is my presentation. So suddenly if you come with, with results, actually my technique is very simple, right? Like I go, I say, hi, my name is Miguel Fierro and I've been working on this problem and we made this amount of money.

Demetrios [00:44:30]: And then I pause, then you drop the mic.

Miguel Fierro [00:44:34]: I wait, then I wait for them to ask. Because you cannot go to a person that doesn't understand AI or recommendation systems and say, you know, AI is the future, or recommendation system is the future, right? Because, you know, first of all, maybe it's not because, and also they have like a thousand different problems in their head, right? But, you know, if you go to these people and say, look, I've got these results. So I'm not saying, I'm not telling you that AI is the future. I'm not telling you that recommendation system is important. I'm just telling you that I got the results, the ROI is 40. 40,000%. So what do you want to do? Right?

Demetrios [00:45:30]: Balls in your court.

Miguel Fierro [00:45:32]: Yeah, exactly. Now is what do you want to do, right? And I found these very, very useful.

Demetrios [00:45:39]: And how do you calculate that Roi and how do you calculate how much money that you've made.

Miguel Fierro [00:45:44]: So the typical way of doing this is you do an a B test. And in the A B test, you need to measure some key metrics. Depending on the business, it's different, right? Like you have revenue leave. Like average revenue per user, for example, is a metric that is important in other companies. A conversion rate is important. Monthly active usage like Mao, or depending on the engagement, depending on the business, you have different metrics. But what you do is you do the experiment and you say, okay, so we increase this metric 3% or whatever number. And then you say, well, based on the, for example, amount of revenue or amount of average revenue per user or conversion that we had last year, that equates.

Miguel Fierro [00:46:47]: Like if we use this model, that equates to these million dollars. And I think another thing that I found very useful is to talk in terms of dollars, right? You're asking. It's kind of. I'm asking for headcount. I'm asking for dollars. You give a result. In terms of dollars, it's very, very easy for. Because it's kind of very easy for them to understand.

Demetrios [00:47:16]: Man, that's fascinating. And I find it so cool how you think about the incredible ROI when you're looking at it as like, hey, we saved or we increased the average sale price or whatever, the average value per customer by this percent and it equals this many dollars. And then you say, and I did that with a team of 20 people or 50 people. And so the, the ROI on that is 40,000 times whatever you put in the cost of those 20 people was well worth it.

Miguel Fierro [00:47:58]: I think it's very interesting, right? Because what I've seen, particularly working in Microsoft, is that now thanks to OpenAI, there is a lot of companies that are really understanding that we are in a new industrial revolution, which is the revolution of AI. But I've been there, I've been selling to customers where when AI was like this magical thing that nobody really understood. So it's very interesting how the, how important it is that the key decision makers, they have an a strategy. And actually, you know, like, I don't, I don't work with them, with the sales organization in, in Spain, in Microsoft. But I, you know, I know many people. So if you ask these, these people, like, what is the, like, what is the difference? Like what, why? Two years ago, like most companies were not interested in AI, and suddenly now many companies are really thinking about AI and having a strategy. And it's all about trust. It's all about that two years ago, the CEO's of these companies were not convinced.

Miguel Fierro [00:49:30]: They were not convinced of AI, right? And I think all this, and something very interesting is that all these CEO's, typically they are all enough to have seen the Internet revolution because the Internet revolution was 20 years ago. So it's not that, it's not like 100 years ago safety. No, no, it's 20 years ago. So many of us are old enough to have seen this and you've seen companies that they didn't get into the Internet and many of them failed. Right. But you know, it happened the same thing a hundred years ago with a steam engine. With the steam engine. Like a lot of companies, they were the steam engine and electrification came and all these companies went away.

Miguel Fierro [00:50:18]: But people haven't lived that, but we have lived the Internet. So now, only 20 years later, we have another revolution, which is AI. So I think now people are like, okay, I'm not going to make the same mistake that people do 20 years ago with the Internet. Now I'm going to get into AI. And I think it was actually, it was thanks to chat GPT, thanks to chat GPT, people really started to see AI as a useful solution and that actually was what convinced the decision makers. And then when you have the buy in from the CEO, from the decision makers, then top down is much easier to actually start implementing these AI solutions.

Demetrios [00:51:13]: Yeah. Then the budgets all of a sudden open up and we got to get involved in this and find out what's our AI strategy. I really like that. Well, man, this has been great. I want to end with one question because I know that we're talking about LLMs and we're talking about chat GPT and tying that into recommender systems. Have you seen many people using LLMs with recommender systems or is that just over engineering it, making it too complex? I know there's a lot of papers that come out that try and incorporate the two. And I just wonder, like out in the wild, is it being used? Have you seen it? Have you tried to use it?

Miguel Fierro [00:51:55]: Yeah, yeah. So, yes, absolutely. So I think there are many ways to combine reco and LLN. So you can think of llns as just another machine learning algorithm. I'm one of the authors of the recommenders open source library that actually we donated to the News foundation recently from Microsoft. And in a similar way that we have gradient boost entries or linear models, we also have transformer based architectures or LNC, that you can use with recommendation systems so they can be used just to power these algorithms. Another approach is to have a chatbot that is like a personalized chatbot. So basically, imagine when you go into something like Netflix or even an e commerce, for example, you're looking for movies, you're looking for products that you could chat via voice or via text and say, I would like to buy some product.

Miguel Fierro [00:53:18]: And then based on this conversation you can actually have, and based on your history of purchases and your preferences, you can have the answer. Some recommendations by this recommender that actually is an LLM combined with some other techniques, but the other days is basically just another chatbot that is personalized. And the thing is that the interface is different. If you think about something like Netflix, for example, every raw that you see in Netflix is a recommender. So there is no conversation. But for example, I've seen a lot of use cases on search that the search experience is being changed to some kind of chatbot in a similar way that Google or Bing is kind of being moved to a chatbot experience. The search experience in an e commerce probably is going to be moved to some kind of chatbot experience as well.

Demetrios [00:54:30]: Fascinating to think about that. Yeah. And it does feel like those use cases are more prevalent when you don't need that super fast rendering time and you can kind of fudge it because those LLMs aren't particularly fast. They aren't known for being fast, at least not these days. So, man, I know we wanted to talk all about you donating recommenders to the Linux Foundation. I think we're going to have to sit. Save that for the next conversation. Yeah, it's been great.

Demetrios [00:55:01]: It's super cool to talk with you about robotics and how if you can avoid using ML in robotics, you probably should also the football use cases of your past life and using those real time analytics and how we're seeing more ML in that, and then the different styles of recommender systems, the batch real time hybrid, and of course the ROI on recommender systems, which I think is probably the most important piece here. I'm going to be quoting that one for a while. So thanks.

Miguel Fierro [00:55:37]: Yeah, thank you. Thank you. Thank you for the. What is.