A Playground for AI Engineers

Paulo Vasconcellos [00:00:00]: Amazing, dude. Thanks for the opportunity.

Demetrios Brinkmann [00:00:02]: Of course, bro.

Paulo Vasconcellos [00:00:04]: Hotmart is a playground for data scientists because all the challenges, opportunities that we see in data science and machine learning we are tackled, we are tackling in a hotmart for such specific problem. We still use some classic machine learning models. Just to give some context for the audience that might not know hotmart. But basically hotmart is the leading company, the creator economy. So we offer a platform and a whole ecosystem for digital creators to live from their passion. So imagine online courses, E tickets, ebooks, basically anything that can be distributed online, you can use hotmart for that, E tickets for digital events and things like that too. And I like to say that hotmart is a playground for data scientists because all the challenges opportunities that we see in data science and machine learning we are tackling in hotmart. So we are basically, if you think in the education industry we are trying to solve problems like spin detection, fraud detection.

Paulo Vasconcellos [00:01:22]: When we're talking about fraud detection, we think more always about financial people trying to make fraudulent purchases like that. But there is a lot of people that are creating courses, trying to teach things that are not appropriate. One good example is people that create courses on how you can buy fire guns in Brazil. That's something that's not allowed. Or how we commit drugs at home. So these kind of things are all the time in hot model and we need to create intelligent systems to detect this beforehand and block them. So for that we use a lot of machine learning models and AI as well. But this is just the tip of the iceberg.

Paulo Vasconcellos [00:02:12]: So we created recommendation systems, we use forecast models and we've been doing that for a long time. And I think I would say that our first machine learning model that went to production was like 2015, something like that. And with the hype of the generative AI, we saw a lot of opportunities as well.

Demetrios Brinkmann [00:02:38]: You just told me that you have replaced a lot if not 90% of your tasks with LLMs, but you're still using Spacy for some.

Paulo Vasconcellos [00:02:48]: Yeah, yeah, definitely. I think LLMs is here for stay. Everybody is using the cookie on the block, but we still use our machine learning model, classical NLP for some tasks that even with LLMs, even they are so powerful, are so smart. But for some specific tasks we still use our own models, training with open source and like spacy, other frameworks like the good old scikit, learning things like that. So yeah, I think that that's a good thing to have started to work with machine learning models before the boon of Genai, because you can see where Is the opportunity to use LLMs and where you still need to use things like classical and LP approaches.

Demetrios Brinkmann [00:03:42]: But what is it that the LLMs can't do that a spacing model can do?

Paulo Vasconcellos [00:03:49]: Actually if you think like prompt engineering, trying to tune the model, few short examples and things like that, you can use LLMs for that, but sometimes you need a very fast solution for a specific problem. I think the thing that comes to my mind is HT extraction because we trained in a model for HTT extraction that we still use along with LLMs but it's way faster and cheaper for us. Imagine I'm talking here, but I'm not saying the user case, but at hotmart we have AI agent for customer success to solve customer tickets. I have a problem with my account, I can't pay things like that, I lost my password and things like that. And can you imagine like thousands and thousands of these tickets arriving each day and we need to classify them. And even in our scale using an LLM is like an overkill. You are using an expensive solution because you pay for the tokens to imagine customer tickets, that there's a lot of text, a lot of context that they give. And for such specific approach, for such such specific problem, we still use some classic machine learning models, but for the user facing solution we prefer to use LLMs because it got more human touch for that.

Demetrios Brinkmann [00:05:22]: So is it just that behind the agent these spacy models are a tool that the LLM can call or is it specifically a workflow that happens?

Paulo Vasconcellos [00:05:32]: Yeah, it's although it's in the same context like customer success, but it's two different problems that we are trying to solve. One of them is solving the customer problem. Like I have a problem with my account, with my course, I can access my course. But also we need some very, very powerful reports and insights for the customer success. So we use AI for that, we use machine learning models for that. We can now talk to the customer success team and say this is the most common calls people are getting in touch with Hotmart to solve. So we don't need to use LLMs for that because we have entity extraction, you can use text classification and things like that. So if you have a huge amount of volume like customer tickets, maybe you can use classical approaches or even small language models.

Paulo Vasconcellos [00:06:33]: That's something that's getting in the track. But that's a bad point for us at hotmart because small language models are really good, but when it comes to multilingual multi language they suffer a lot. And for those who don't know. Hotmart is basic. We are based all around the world. We have offices in Colombia, in Spain, we have offices in Asia, North America and things like that. But mostly our operations in Brazil. So most of our customer needs are in Portuguese.

Paulo Vasconcellos [00:07:08]: And for that this small language model suffers a lot.

Demetrios Brinkmann [00:07:12]: Yeah, especially compared to your baseline which is a very advanced spacing model.

Paulo Vasconcellos [00:07:18]: It's like, okay, yeah, that's true. Then this model was around before the LLMs. And honestly one of the two things that we made when LLMs became a thing was can we replace this classical machine learning with LLMs? Because we have some benefits from that. Like it's cheaper, right? It's very fast because we can make inference using some classical approaches. But what's the problem that we have continuous training. New things arises, new subjects, new topics, people talking of different things. So we need a good workflow, a good pipeline to keep training this model. We need to evaluate this model.

Paulo Vasconcellos [00:08:01]: We need to make sure that there is not a data drift or concept drift. So at the end of the day is about trade offs. Like I have this technological effort that I need to do to create this solution, but I got a better quality, a better, more accuracy. Instead of using an LLM or even a small language model, are you hosting.

Demetrios Brinkmann [00:08:29]: The classical ML model on GPUs or do you not even need that?

Paulo Vasconcellos [00:08:35]: No, we don't need that.

Demetrios Brinkmann [00:08:36]: So that's another consideration. It's like maybe with the small language model you could load it onto a cpu.

Paulo Vasconcellos [00:08:41]: But yeah, but we lost. When we need to scale the solution, you can do it for 1, 2 data points, but when you think like an operation of thousands of tickets per day, it makes them feasible.

Demetrios Brinkmann [00:08:57]: Exactly.

Paulo Vasconcellos [00:08:59]: So we basically are using classical approaches like as I mentioned, the good old Scikit learning, logistical regression, multi class approaches and they are running our own infrastructure, so they are running basically in kubernetes and microservices. So it's really cheap to create the solution. As I mentioned, there is an effort to keep training this model, but when you put in the balance, when you make the trade off, it still pays off.

Demetrios Brinkmann [00:09:28]: Yeah, and then the entity extraction or all of these insights from these models are being routed to some kind of a dashboard so you can tell when things are spiking. Like I imagine if all of a sudden everyone's calling in about the same thing. You're getting alerted for that.

Paulo Vasconcellos [00:09:46]: Yeah, we have some insights, I would say in an offline basis and online basis. Like we have this monitoring for the teams when something is getting out of hand. But mostly I think it's for generating insights intelligence for their strategy. Like imagine that I said to you that hey Demetrius, we have a lot of problem with our payment system. So that's an insight for you. For the next quarter you can get in touch with the payment solution team. Say hey, there is a spike in this. This problem relates to our payment project, to payment platform and to solve it.

Paulo Vasconcellos [00:10:32]: Because one of the business metric that we are using a lot we keep tracking is nps. So the net promoter score, we need to keep track that the satisfaction of the customer is good enough for the company when we scale our business. When you create new solutions and things like that.

Demetrios Brinkmann [00:10:53]: Yeah, because at the end of the day your customer has a very loud microphone because your customer is in some way shape or form an influencer, big or small, right?

Paulo Vasconcellos [00:11:04]: Yeah, that's true. And as you mentioned, it can bring some risk for the company, like especially branding risk. So imagine that to create a solution that impacted negatively the the customer complains and we don't know that there is a pain in there that we need to solve. So yeah, that's important of such simple approaches but powerful outcomes that the solution brings to the table.

Demetrios Brinkmann [00:11:38]: So what other kind of use cases are you guys doing? I love this one because it's so fascinating to me how you're leveraging both the LLMs and you tried to kill the spacey model but you couldn't. It was like no, it still is better. And the consideration of hey, Portuguese is a really big factor, Speed is a really big factor and costs are huge. Makes a ton of sense to me. What are some other use cases that you're putting into production?

Paulo Vasconcellos [00:12:09]: Talking specifically about agents. One great use case that we have, hotmart is what we call hotmart tutorial that's basically a digital teacher for your students. So imagine you are digital creator like Dimitrios decided to launch courses on how to grow an amazing hair and things like that.

Demetrios Brinkmann [00:12:36]: I mean it's not a bad idea. Honestly, now that you're saying it, I should do that.

Paulo Vasconcellos [00:12:40]: But you're losing opportunity there.

Demetrios Brinkmann [00:12:42]: Yeah, exactly. I'm leaving money on the table. Continue. You have my attention.

Paulo Vasconcellos [00:12:49]: So imagine that you created this course and people start buying. Sometimes people might have some specific questions that your lessons or things like that are not explicitly teaching them or they weren't paying attention to the classes when we are talking in the video. So we make this AI tutorial that basically 24, 7 online teacher that we have that you can ask questions about the course, about Dimitrios course and this model, it's basically LLMs with rag and other things like vector storage that we use fallbacks and things like that guard as well. And this AI agent will use only your content to answer the question for your student. So you don't have any chances of this model getting the information elsewhere or something that you didn't say in answering the question. So this has been a massive success at hotmart because people are getting a lot of support for fraud they are purchasing. Because when you are a digital creative, some people are a business of one person and you have like thousands and thousands of students, you can help all of them at once. So when you use an AI system like that, you can delegate a lot of these questions that a machine can answer based on the content that you already created and for the creator.

Paulo Vasconcellos [00:14:28]: What we create, what we deliver, is not only this experience to delegate this task to an AI agent, but also we deliver a lot of insights of what people are asking for the AI tutor.

Demetrios Brinkmann [00:14:45]: So you're analyzing those conversations that are happening just like with the support bot. Hey, it looks like everybody's asking things about the payment system.

Paulo Vasconcellos [00:14:55]: Yeah. Or there are some funny interactions. Interactions that we see in this, in this AI tour. So we have tour experience. The experience for the student asking anything they want about the course and insights for the creator. And you can imagine things like an online course for, I don't know, personal life, like relationships. It happens a lot. Like people use the AI tutor as his therapists.

Paulo Vasconcellos [00:15:30]: You know, like, oh, I don't know what I'm doing with my life, I don't know if my wife still loves me. Something like that.

Demetrios Brinkmann [00:15:39]: I believe it.

Paulo Vasconcellos [00:15:40]: And there's a lot of risk in there too, because that's why you need Guadalupe, because you can imagine like an AI agent interacting with this person. Not a romantic relationship, but in something like a sentimental interaction. It's really important to have good graduates to make sure that AI say, hey, I can help you with that.

Demetrios Brinkmann [00:16:04]: But it's not part of the course, dude.

Paulo Vasconcellos [00:16:06]: Yeah, it's not part of the course. Actually, it can be part of the course. Yeah, imagine like relationships online course. I will help you to be a better version, a better man in my relationship. Yeah, in your relationship.

Demetrios Brinkmann [00:16:20]: I also need that one.

Paulo Vasconcellos [00:16:21]: Yes.

Demetrios Brinkmann [00:16:23]: There's a lot of courses that I need. I hopefully can make enough money from my hair course to buy these other courses anyway.

Paulo Vasconcellos [00:16:32]: But that's it. You have the context that is relationship courses. But even in that context, the AI agent can answer something like that. This AI agent can use your Course, if you told something related to how to become a better man for your life, something like that, the AI agent can use. But if it's something out of the context, we have some strict guadarrails to make him don't talk about it. But this is an edge case. Most of the use cases that we have are courses related to personal development. Like, I want to learn this new skill, learn a new language.

Paulo Vasconcellos [00:17:17]: I want to learn how to become a developer, something like that. But the devil is in these edge cases, and we need to be very careful for that.

Demetrios Brinkmann [00:17:29]: Yeah, yeah, I believe it. And so when you're analyzing this stuff, then you give it to the creator and you give these creator tools in a way and say, look, lots of people are asking this question. You should probably create a new module that adds to the course and explains this better. Or rerecord this section because people get to video 3 and they ask the same question.

Paulo Vasconcellos [00:17:52]: Exactly. That's exactly what we. The kind of intelligence that we want to deliver for the creator. Because sometimes they don't have this visibility. What they know about the students is like, oh, okay, Dimitris made a purchase of my course. I'll deliver my course. But I don't know how the experience has been for this person. Now they know because of the interaction they make with aitutor.

Demetrios Brinkmann [00:18:19]: Yeah. And me as the person who bought that course, I get to see. I get to interact with the course in a way that it's not like I feel like I'm bugging the creator.

Paulo Vasconcellos [00:18:31]: Yeah.

Demetrios Brinkmann [00:18:32]: I imagine that I wouldn't ask the same questions directly to the creator as I would ask to the AI agent creator.

Paulo Vasconcellos [00:18:40]: That's an interesting point of view because. Yeah, that's true. Imagine for a person that chooses shy a person that doesn't have.

Demetrios Brinkmann [00:18:47]: So I wouldn't want to make them feel bad, but if I know it's an agent, I'm going to be like, what the hell's going on here? You know?

Paulo Vasconcellos [00:18:55]: Yeah, there's no judgment for my question. You ask for a chatgpt something, they say, hey, great question.

Demetrios Brinkmann [00:19:01]: You know, you're absolutely right. Okay, real quick.

Demetrios Brinkmann [00:19:05]: If you happen to find yourself in the South Bay on March 3rd, 3rd, we're going to be taking over the Computer History Museum for our Coding Agents conference. That's right. We're organizing another conference. Despite me telling myself so many times after we did our AI Quality Conference that I would never do another conference.

Demetrios Brinkmann [00:19:25]: Because of the stress that it caused.

Demetrios Brinkmann [00:19:26]: Somehow I seem to have broken my vows because the pull was too Strong. I have been falling in love with all of these new ways to code and use coding agents. So I wanted to organize a place where we can learn from the best in the game at how they are getting the most out of them. Some of the notable speakers that we have already announced are Sid, friend of the POD and co creator of Claude Code Harrison Chase, the founder of LangChain Thomas Reimers, the founder of Graphite who just sold it to Cursor, good old Dex, the founder of Human Layer and the dude who popularized the term context engineering and harness engineering. And last but not least, someone I consider a very close friend, Michael Eric is going to be doing a workshop. He's a Stanford lecturer and actually he was a past co host of this here podcast that you're listening to. You know, before he got all famous and stuff. Come join us.

Demetrios Brinkmann [00:20:30]: It's intimate by design. There's only 450 people that we're going to let in to the room to try and keep the signal as high as possible. We've got early bird pricing happening until February 1st, so I'll see you there.

Paulo Vasconcellos [00:20:44]: Something that we've been working from the past two years. We launched the Hotmart Turo in 2023. So if you remember like ChatGPT was launched in November 2022 if I'm not wrong, November 2022 and we launched less than a year. The AI tutor. I think it's most important when we think about an AI agent. You need to think not only about the hype. There is a lot of hype and pressure to create AI systems, create AI features. But you need to pay attention to how you measure suspects of your AI agents in production so how success looks like.

Paulo Vasconcellos [00:21:28]: When we think about AI Tutor Hotmatutor that we created, something that's really important for us is direct feedback from the user. So each question that the tutor answer there's a thumbs up, thumbs down so people can interact with that and it's amazingly positive. Their exception because they consuming a course. I think it's about context because I'm here learning this course, learning this lesson. I make a question and my question get answered correctly. So that gives me some satisfaction of okay, that course is really good for me. Right? And also something that's really important for us is like the engagement of the solution. Not only engagement, but okay, you talked once to, to the Hotmart tutor tomorrow, are you going to talk to them tomorrow as well? Like how people are repetitive asking things to tutor.

Paulo Vasconcellos [00:22:33]: So one metric that we like to use is how much People are interacting on a weekly basis with the AI tutor. If these people are coming back and asking another question, because you can have people that like 100% of your user base ask one question, but the answer is so bad that they don't come back, they don't answer the question anymore. So these are some of the metrics that we pay close attention when we deliver user facing solutions like the AI tutor that has a huge impact on the student experience and need to pay attention to make sure that this new solution creates a better experience for them, but also aligns with the strategy of.

Demetrios Brinkmann [00:23:15]: The company in a way. That metric on if a user comes back and how much they're using it throughout the week or the day is a proxy for how much value they're getting.

Paulo Vasconcellos [00:23:26]: Yeah, that's exactly because you can understand one problem that we try to solve with the AI tool as well is like there is a drop in engagement and not only for Heartmart but any digital platform. You can think of YouTube like that. People you know that you are, you have a YouTube channel, you, you see that there is a lot of engagement in the beginning of the, of the video and they drops a lot. So one thing that we need, we want to make sure is that is the attention, the retention bigger for person for courses that use the Hortmart tutorial. And we measured that like let's compare a course that doesn't use tutor and a course that uses tutor for their students. Nice. And the difference was absolutely immense. Like people that use the Hotmart tutor, they come back and keep their retention higher than people that don't use.

Paulo Vasconcellos [00:24:33]: Because I try to understand why this effect and I think it's something because we have a human touch like than just looking at a screen and just fancy recording.

Demetrios Brinkmann [00:24:46]: Yeah, maybe you get lost and everybody learns differently. I also have heard stats where the majority of people who buy courses never finish them.

Paulo Vasconcellos [00:24:58]: Yeah, that's true. I think not only for a digital course, but you can think of college, you can think of master's degree doctorates and things like that. And yeah, that's a problem that we are trying to keep in their attention to make sure that the courses is delivering value for their journey, for their learning journey. I think it's solutions like AI agents can bring some. Imagine when you are learning something, when you are watching a video, it's. It's a passive interaction. I'm just there consuming that video. Once I have an interface that I can interact with, I move from this passive interaction to an active interaction.

Paulo Vasconcellos [00:25:46]: I'm making Questions I need to think of the question that I'm going to make. I need to have at least some kind of critical thinking on the question that I need to formulate for that. I think that improves a lot, I can tell you, because this is the way that I like to learn.

Demetrios Brinkmann [00:26:04]: Me too.

Paulo Vasconcellos [00:26:05]: I was one of those guys in the classes that asked questions to the teacher that has raised her hands to make some questions. I think we can have a little bit of that with AI agents in an educational field by interacting with questions that can be answered using the context of the whole courses that I make.

Demetrios Brinkmann [00:26:27]: Yeah, it shows that you're trying to grok the greater theme or the greater concept that is being shared. It's not just like sitting back, seeing how much you understand and then ultimately doing nothing with it.

Paulo Vasconcellos [00:26:42]: Yeah, that's true. And the good thing that we extract from this user case is that we see that for the educational industry, AI agents can make a major role in the experience, in the student experience throughout this journey. So we basically made more investments in AI agents, not only AI agents, but AI as a whole, to create new features for our users to create new features of how they can create their own agents. I think three months ago we launched Agent as a product that's basically just like you can create an online course, you can create an e book, sell them at hotmart. We also offer the opportunity to you to create your own agent at Hotmart and start selling this. This agent. So the good part is that you can use your own knowledge base. Like you.

Paulo Vasconcellos [00:27:52]: As I mentioned, you have an online courses on how to deal with your hair and things like that. There is a lot of knowledge in there that we can use.

Demetrios Brinkmann [00:28:00]: Lots of podcasts talking about my hair routine.

Paulo Vasconcellos [00:28:02]: Of course that's true. So why not create an agent using that knowledge base? It's not like a generic ChatGPT agent that you can asking anything. It's Dimitrios agents. They put his knowledge in this agent that they are using to create better answer. Helping me with something. So we have people that are creating agents. Imagine like nutritionists, things like that. They are creating agents to help you with your diet, create some, some recipes for you based on what you learned from the course.

Paulo Vasconcellos [00:28:44]: So the agent as a product has been used as an opportunity for upselling. Like you buy the course and you get an AI agent that you can use on Hotmart with WhatsApp and things like that. There is a lot of opportunity there. We are seeing some great results from some major creators that we had in our industry and we see that in 2026 we are going to invest that more because this is something that the world is going. This is the way the world is going. It's not only about online courses and things like that, but also using agents to create new products and new experience.

Demetrios Brinkmann [00:29:27]: Yeah, that interactivity makes a lot of sense. How have you seen the difference in ecosystems or in platforms from the ML platform that's supporting the spacy models? And you're just retraining and you have the evaluation there that's pretty. I'm not going to say it's totally understood how evaluation works there to then going to these AI products. And maybe you're just hitting an API, but you're trying to evaluate by looking at the different turns in the conversations that the users are having or the questions that folks are asking.

Paulo Vasconcellos [00:30:02]: I think the biggest lesson that we get from that is that you need to. When you work with AI agent platforms, you need to be ready to change everything that created so far because the industry is moving so fast. I like to give you the example of chain of thoughts or reasoning models. As I mentioned, ChatGPT was launched in November and they launched the API in March 2023. And at the time we created this AI agent for customer success and we need something that later became reasoning models. Like we need a model to think about what they are going to do. At the time even structured output was so hard it was almost impossible to make an LLM to just spill out JSON for our application. So this one was one of the lessons and why I'm telling you that I mentioned that we are using machine learning models since 2015 and at the time we had a very strong machine learning platform with ML Ops team and things like that.

Paulo Vasconcellos [00:31:16]: So we were solving problems related to data extraction like feature stores. We were using experiment platform like MLflow to create experiments to deal with the whole lifecycles of these models. Continuous training pipeline using, I don't know, serverless GPUs and things like that. These were problems that a machine learning platform needed to solve. Then Genai comes out. So then Genai appears and we saw everything that we made so far. Yes, we can use some of them, but we need to create a lot of new things that we weren't planning. I think the biggest example are vector databases.

Paulo Vasconcellos [00:32:03]: Nobody was looking for vector database at the time. If you are not on search engine team, search Recommendation team, you were not paying attention to that. So we needed to evolve this machine learning exclusively platform for machine learning plus Genai platform as well as so the first experience was something that said it was an API that people call use some components of this API. Oh, I have this mod, this prompt that I want to use rag and so you can call this API to use as LLM layer Vector database. But we need to keep it involved because we showed value from this solution and we need to evolve, creating new features, improving the experience. So we need to create components for graduating, we need to create components for fallback. Like if my vendor, there's a problem with my vendor, I need to use another vendor so the user doesn't have any problem. And now we are seeing some problems even with that infrastructure.

Paulo Vasconcellos [00:33:25]: We are reviewing this infrastructure to make it better for internal solutions that we are creating, like AI Tutor, the Hotmart tutor, but also for the external clients, like allowing digital creators to create their own agents using the same platform. So we don't have two ecosystem that we need to take care of. But I think it's almost impossible to plan for this environment. And we didn't see it come because even the industry was not ready for that. So this is something that we need to pay attention. When you're creating a platform is you need to make every part of your platform, every component, very independent of each other. At beginning we use it. Some of these vector database vendors imagine like Pinecoin or something like that, but we needed to move to our own infrastructure.

Paulo Vasconcellos [00:34:26]: At the end of the day, it was just to change the endpoint in our AI platform because the vector storage was not attached to the platform itself, it was just a component. So I think that's some learnings that we had throughout this experience.

Demetrios Brinkmann [00:34:41]: You got to be ready to burn it all down, basically.

Paulo Vasconcellos [00:34:43]: Yeah. Or at least make some major changes in these. In this infrastructure.

Demetrios Brinkmann [00:34:49]: Well, yeah. I even know a lot of people who now are like, we don't really care so much about vector databases because we just want to give our agents the search capabilities.

Paulo Vasconcellos [00:35:02]: I see and let.

Demetrios Brinkmann [00:35:03]: So the agents can call a tool which is search and then they figure out how to get what we need. Instead of like everything is now going to be run through the Vector DB or it's going to be vectorized and then thrown into the Vector DB and all of that stuff.

Paulo Vasconcellos [00:35:19]: Exactly. Vector storage. It's a good amount of effort that you need to put to maintain an infrastructure like that. Even if you're using something like Postgres or Open Search, there is an infrastructure that you need to take care of. And I get the people that say like we don't need regular database, but when we think like when we think in a case such as I mentioned Hotmart Tutor that uses the creator's knowledge to answer a question. You can just use a search tool for that because that information is not publicly available. It's something that I created for the course that I'm selling, the community that I'm creating. So I figured the answer that you need to have both.

Paulo Vasconcellos [00:36:06]: You need to have both actually you need to create some good tools for this platform and be ready to when things change. I think the best example was mcps. It's not even one year they launched an MCP and everything now is mcp. Like you need to create an mcp.

Demetrios Brinkmann [00:36:26]: So much talk about it. That's so true.

Paulo Vasconcellos [00:36:28]: Yeah. If you think about agent to agent protocol, Google launched in April, April or May, something like that. So it's really, really new. What will be in the next year. You need to be ready to adapt your platform for new needs that you need.

Demetrios Brinkmann [00:36:44]: Yeah. It's almost like don't let the systems around the AI products get too entrenched because you got to be nimble.

Paulo Vasconcellos [00:36:54]: Yeah. I think the main point is that you need to isolate the problem. What's the problem that we are trying to solve with this solution? So we don't fit the tool in the problem. We need to find what's the best tool, the best solution for that. The problem that we are facing, it's a scalability problem. It's low latency problem. These are things that we really notice when we are creating these systems. So for a person asking a question for hotmarturo latency might be not a problem.

Paulo Vasconcellos [00:37:31]: Imagine we interact with LLMs. I use perplexity a lot and I don't use the search component anymore. I just use the research because I don't care if my answer takes one minute or two, if it's the right answer, I don't care. But imagine that you're creating a sales agent for example that need to read some leads, some code leads and things like that for them. Maybe latency is really important because.

Demetrios Brinkmann [00:37:57]: Yeah. Where it's coaching you in the call and it's suggesting things for you to say. It has to be real time.

Paulo Vasconcellos [00:38:02]: Yeah. Or even a text based like imagine AI agent contacting you through WhatsApp or Telegram. Do you want a sales agent to spend like two minutes thinking the next message for a person that is not probably doesn't want to talk to you. Like I why is there a salesperson sales agent talking to me? I don't want this product anymore like.

Demetrios Brinkmann [00:38:27]: That Something funny happened to me six months ago. I asked Deep Research for a report.

Demetrios Brinkmann [00:38:32]: On different GPU providers and it was absolutely shit. I couldn't figure out what each Neo.

Demetrios Brinkmann [00:38:39]: Cloud value prop was.

Demetrios Brinkmann [00:38:41]: So that set me off on my latest side quest of creating a practitioner's guide to choosing GPUs.

Demetrios Brinkmann [00:38:49]: I'm happy to announce this guide is now ready to see the light of.

Demetrios Brinkmann [00:38:52]: Day and you can download it for free right now by clicking the link in the show notes. We've already got some community members feedback of what they wish they would have known before signing a gigantic contract. And I would love to hear from you if this provides any value or you have some things that the rest of the community should be thinking about when they're on the market for GPUs. Go ahead, download that resource right now, completely free. It's in the link in the show notes.

Paulo Vasconcellos [00:39:25]: I think this is the beauty of this. AI agents. Each use case is a different challenge that you need to face.

Demetrios Brinkmann [00:39:31]: Yeah. And recognizing which challenge is the most important and which pieces are the most important for you. It feels like you have spent many cycles on that. You are looking at, okay, here's the most important metrics that we want to focus on and here is how we're going to think about them and how we're going to isolate them and try and move towards moving the needle on them.

Paulo Vasconcellos [00:39:58]: Exactly. And I think one of the business skills that a data professional, an AI professional developer can have nowadays is a good discernment in everything that's going on. Because you know, well that it's really easy to get hyped for everything that they announced online. Oh, this new tool is so amazing. This changes everything now they are so doomed. We are back. Things like that. I think the last one that I noticed that was with Toon, you know, Tune, they want to replace Jason.

Paulo Vasconcellos [00:40:36]: Things like that.

Demetrios Brinkmann [00:40:36]: Somebody asked yesterday in the event, like, are you guys using Tune? And it was just the speaker just.

Paulo Vasconcellos [00:40:43]: Said, no, no, everybody's talking about it, but what's the point? Does it really solve a problem for you? Like is really the.

Demetrios Brinkmann [00:40:54]: Yeah. Do you have that problem bad enough to where you need to spend the effort to figure out how to use Tune?

Paulo Vasconcellos [00:41:01]: Yeah, I can make sure. I would bet money for saying that Nobody's using Toon right now in AI.

Demetrios Brinkmann [00:41:08]: Except for Twitter.

Paulo Vasconcellos [00:41:10]: Except for Twitter and LinkedIn, everybody's using toon like that. But the real problem that we are trying to solve are way bigger than the structured output that a model responds.

Demetrios Brinkmann [00:41:24]: Talk to me about some of the other use cases you have I mentioned.

Paulo Vasconcellos [00:41:28]: The AI sales agents that we have, what problem we are trying to solve. There's a lot of. Before that, hotmart owns its own payment platform. We have Hotpay, that's our payment system specifically for the creator economy. People might ask why do we need a specific payment system for the creator economy? The truth is that there is a lot of problems that we are trying to solve that other players doesn't want to solve. Because when you are using checkout tool for your community, you want to make sure that you can track the needs for the person that is going to the checkout and things like that. You want to have control of why people are dropping out that say, abandoned. Why they abandoned the.

Paulo Vasconcellos [00:42:23]: The car. The car and things like that. And well, we developed this platform a long time ago and one problem that we had was people sometimes got into the sales page, learn everything from the course. Like we know that what this course is teaching. Okay, he's teaching this technique. We use this approach. This is the price, this is the time I will have to use this content, go check out, but I don't finish the purchase. As you can see, that impacts the approval rate of this product.

Paulo Vasconcellos [00:43:07]: And a problem that we're trying to solve is just trying to recover these people that abandoned their charge. So we developed this AI sales agent. Actually, we have some interactions before even using human agents to reach each one of them. Like each person. Yeah, no, it's a person.

Demetrios Brinkmann [00:43:31]: It was an actual human.

Paulo Vasconcellos [00:43:32]: It was an actual human that was.

Demetrios Brinkmann [00:43:34]: Talking to the other human.

Paulo Vasconcellos [00:43:36]: Yes, exactly. Through WhatsApp in this case. That's a major messaging platform that we use in Brazil. The good thing is that we could use some baseline for the solution that we need to solve. We know what was the response time for humans, what was the compression rate? So we had this baseline. So can we create an AI agent that will do the same job and having a better approval rate in that sense? So we created this AI sales agent. That's the only problem that we had at the time was the latency. As I mentioned, it's really important when we try to close a sale.

Paulo Vasconcellos [00:44:25]: It's really important. Timing is very important because I want to reach you and maybe you are going to reply just two hours later. And I can't just answer your question two hours later. I need to be very.

Demetrios Brinkmann [00:44:40]: On it.

Paulo Vasconcellos [00:44:40]: Yeah, on it.

Demetrios Brinkmann [00:44:41]: Because when you're online, you got to catch them exactly in that moment.

Paulo Vasconcellos [00:44:45]: So I think that was a good benefit of this AI agent. And we deployed this Model in production I think in July. And it improved the approval rate of the customer by a lot. Honestly, I can't share specific numbers of the solution yet, but we saw some of these creators improving their revenue from the sales that the AI agents make. Because what this AI sales agent is basically using, we are using context from the course. We know what the course is teaching, we know the payment methods they offer, warranties, support, things like that. And we created a workflow that can answer any question that the user might have, any objection the user might have before they close the sale. So an experience sounds like this.

Paulo Vasconcellos [00:45:51]: You receive a message in WhatsApp from our sales agent and saying something like that, hey, Dimitrios, I know your name. I even know your name because you are logging the platform. So I tell you I saw that you try to buy this course but you didn't finish your purchase. What went wrong? Do you need help with anything? And we have been tracking some of these interactions. Something is really good things, interesting things happens. Some people say, oh, I will wait for my son because they know everything about computers and he can help me to finish the sale.

Demetrios Brinkmann [00:46:32]: Because they're afraid of fraud or something.

Paulo Vasconcellos [00:46:33]: Yeah, probably something like that.

Demetrios Brinkmann [00:46:35]: They don't want to put their credit card in.

Paulo Vasconcellos [00:46:37]: Exactly. And the sales age needs to break this objection. Like, okay, it's okay to wait, for example, but what's your worry? Do you don't know how to use a credit card? Do you have a question about the course? Oh, I don't know if this course is true. I don't know the platform, I don't know hotmart. And this model starts breaking this objection. Like, oh, hotmart is the leading company. We offer a total refund for you if you don't like the courses. And so people get more interested in the product and some of them finish the purchase after this direction.

Demetrios Brinkmann [00:47:23]: But can they purchase right there in WhatsApp?

Paulo Vasconcellos [00:47:25]: Exactly. Really they can. We have a feature called Fast Buy. If a person already bought made a purchase at hotmart, the second time we. The second time this person tried to make a sale, this person doesn't need to fill the forms with his name, email, things like that. We just send a code for your email and we say, hey, do you want to finish this purchase? Just tell me the code. And we transfer this experience to the WhatsApp. So do you want to close the sale now? Do you want to buy the course? And the person says, yeah, I want.

Paulo Vasconcellos [00:48:07]: So we have in this workflow something like the person want to buy some signals like person want to buy. And then we see that this person already made a purchase. And then the next interaction with this agent is he saying to the clients like, oh, I saw that you already bought at hotmart, you don't need to fill out all of the details. I just sent you a code to our email, just tell me what it is. And I finished the say for you right now. So this kind of experience we want to deliver for digital creators so they can just improve their own content. Like I don't want to think on strategies on how to recover clients, on how to try to close sales with some leads. I just want to focus on my content to create better community, to create the best courses.

Paulo Vasconcellos [00:49:04]: And with solutions like AI sales agents, AI tutors, AI customer success agents. These type of agents can delegate these tasks that a human usually would do for AI. So AI can do and this digital creator, they can focus on. The part that I think that an AI can't do that is creating the best course, creating, sharing his knowledge with their students.

Demetrios Brinkmann [00:49:34]: How does it work with the agent so that it can have access to the system to know that someone's bought already or to the tools to be able to send the email. All of that is it. The agent has tools like get user information check, like how do you do that?

Paulo Vasconcellos [00:49:57]: Some of these informations are populated in a low latency database. Like some information that doesn't change, that we already know, like your name, we know your email, what course you try to buy and this information doesn't change. But some of these other context needs to be built like on the fly. If I make a question about the content, I need to look for this information in our vector databases, in our other knowledge bases to fill out the context so the AI can use. So yeah, it's on workflow that uses agents with tooling, but most of them are not. I think the right term is context engineering, like creating the whole context for that interaction to make more fluid, to make more accurate with what the user needs 100%.

Demetrios Brinkmann [00:51:04]: I want to say that I got to give you huge props because thanks to you I say it all the time, but I can't say it enough.

Paulo Vasconcellos [00:51:14]: Dude.

Demetrios Brinkmann [00:51:15]: When I was starting the mlops community, I found the data hackers community and it was my guiding light for the mlops community. It was already an established community, it was very active. There was so much value being created in there. And it was not salespeople in there just driving the conversations. It was real practitioners that were sharing their insights and talking about things, sharing their opinions, showing their projects that they've worked on. And for me, that was the coolest thing ever. And so I just. I distinctly remember some of those first times that I would drop into the data hacker Slack and just be like, damn, there's so many cool channels in here.

Demetrios Brinkmann [00:52:05]: I can't keep up with all of them. I'm going to keep tabs on a few of them. And one of them was like, the memes and random channel type thing.

Paulo Vasconcellos [00:52:13]: One of the most important.

Demetrios Brinkmann [00:52:14]: Exactly. And it was so cool for me because I got to learn about data science, data engineering, machine learning engineering, and simultaneously practice my Portuguese.

Paulo Vasconcellos [00:52:28]: Oh, that's part of the truth.

Demetrios Brinkmann [00:52:30]: And I will say this, man, I can't thank you enough because I remember some of the first times that we were doing virtual meetups in the mlops community, which it wasn't even a community then. It was basically just like we were getting on zoom calls, and it was very, very nascent still. I would post like, hey, I'm gonna be doing this in case anybody wants to come by. And you came into one of those posts and you wrote in the thread like, dude, what you're doing is so cool. Keep it up. This is awesome. That, like, this.

Paulo Vasconcellos [00:53:07]: That's true. That's true.

Demetrios Brinkmann [00:53:09]: And I still, like, I vividly remember that, like, oh, dude, this guy who's like the moderator, he like, gave me the stamp of approval. Like, this is cool. I was like, all right, I guess I'm gonna keep going, you know? Yes.

Paulo Vasconcellos [00:53:20]: I mean, focus on one context. That is Brazilian ecosystem. But we are reaching a lot of people around the world, and I think that gives a lot of different connections. We see different approaches for old problems. So, yeah, thank you for the words. And we are learning together from each other.

Demetrios Brinkmann [00:53:40]: Yes. If anybody wants to practice their Portuguese and data skills at the same time.

Paulo Vasconcellos [00:53:47]: Exactly.

Demetrios Brinkmann [00:53:48]: It's the data hackers community. You guys got a podcast, you got a newsletter. Subscribe to all of it.

Paulo Vasconcellos [00:53:53]: Yeah, the podcast. We have some Instagram pages as well. We are on linkagent. We have a medium blog that is collaborative with the community. So people in the community write the blog posts. And if they want to know, we made with Demetrios the first AI translated episode. Data hackers podcast.

Demetrios Brinkmann [00:54:19]: That was so good. Yeah, because my Portuguese was a little rusty.

Paulo Vasconcellos [00:54:23]: No, it was good. It was good. It was a good baseline. So we moved to the AI translator. You can see how good AI is.

Demetrios Brinkmann [00:54:32]: It was a lot of fun and I appreciate you guys being so patient with me on that.

Paulo Vasconcellos [00:54:36]: No problem at all.

Demetrios Brinkmann [00:54:38]: Awesome. Well, yeah. We'll end it there. That was great.