Sign in or Join the community to continue
Cut One AI Cost and Two More Grow Back
Posted Nov 05, 2026 | Views 13
Share
Speakers
Ioannis Zempekakis
Director AI & Data Science @ Prosus Group | ai-by-design.com
Data Science Leader focuses on delivering data science products that maximize positive impact on all stakeholders. Empower and assemble teams to start with the customer first and solve pain points responsibly and ethically. Passionate also about presenting our AI-by-Design framework for developing human-centered and resilient AI. Public speaker and co-founder of www.AI-by-design.com.
Currently, also helping the amazing founding team of TheyDo on starting their journey on AI.
Personally, I am interested in discussing and learning about (A)Intelligence, the human mind, system design thinking, and what it even means to be human.
+ Read More
Demetrios Brinkmann
Chief Happiness Engineer @ MLOps Community
At the moment Demetrios is immersing himself in Machine Learning by interviewing experts from around the world in the weekly MLOps.community meetups. Demetrios is constantly learning and engaging in new activities to get uncomfortable and learn from his mistakes. He tries to bring creativity into every aspect of his life, whether that be analyzing the best paths forward, overcoming obstacles, or building lego houses with his daughter.
+ Read More
SUMMARY
What does AI really cost once thousands of people inside a company start using it? Ioannis Zempekakis, Director of AI at Prosus, has been running agents in production since 2024. We put that question to him on a tandem bike ride from a favorite Amsterdam cafe to Vondelpark.
Ioannis explains the decision to stop relying purely on cloud APIs, and why that approach breaks down at the scale of a large company. He covers why agent harnesses and sandboxes are as greedy as software gets, and why agents scheduled to run every five minutes repeat a lesson the data world already learned about freshness. He also explains how the habit of throwing away a codebase and prompting again quietly adds up. His bigger point is that even if model progress froze tomorrow, there would still be about two years of value left to squeeze out. The cheapest wins come from controlling context and running a basic model router, while ten new features a month keep sprouting like hydra heads.
+ Read More
CONTENT & TRANSCRIPT
Demetrios (00:00) It also there to be shared state between the client and the server. So certain features like I'm not going to be too long because watching this keynote but recently I caught up with Yonis and we talked about the total cost of ownership in AI on a bike. Enjoy. Beautiful. One, two, three. Whoa. Why are you going that way? Way harder than I thought it was going to be. I'll see you on the other side. You're good at it, man.
I don't do anything, man. We're going to talk about your million-dollar decisions. Are you ready for that? You going to be able to tell me about this million dollar decision that you did? What was your whole revelation?
Ioannis Zempekakis (01:00) There there are a couple of things, right? So the you get lots of um peace of mind if you go and hit an endpoint from GCP and OpenAI and cloud.
It's just that doesn't scale. Yeah. Right. Too expensive. It's too expensive. And you're talking doesn't scale to,000 10,000 person companies. Yeah. It doesn't scale to uh I think uh in the in the end it doesn't matter to 5 10,000. You can be one person company.
It's on the other side, right? Who who consumes who consume who consumes your product, right? So the moment that you go into a real consumption, especially if you have harness, it's as greedy as it gets in terms of tokens, right? Uh because actually you give your your harness a whole computer.
Demetrios (01:48) Yeah. With the sandbox, it's like, oh, hey, we got this sandbox that we're can do whatever it wants.
Ioannis Zempekakis (01:54) Yeah. So the sandbox itself as a cost is not very expensive. It's just a storage if you think about. But the agent now can store lots of things in the store and then you can connect it can do 10 times more integrations than it used to be in the past because now it can run its old skills and it can run its own integrations.
So it's five times smarter. the models are smarter and then suddenly you need to pull out information from Slack, email, Google doc, granola, notion and check whether uh what we have discussed today has been properly translated into a PR or in into our linear linear and then that can happen 100 times and then the cool thing is that it feels it's free but actually it's not free
Demetrios (02:44) Cuz it's free in terms of human consumption.
Ioannis Zempekakis (02:48) It's free in terms of human consumption and it's free in terms of um we've been consuming one way or another this type of technology highly subsidized. Demetrios (02:59) Yeah. You're on the highly subsidized narrative.
Ioannis Zempekakis (03:04) You're thinking that I think I think we are still on the high subsidized. If you put down the numbers and the unit economics, I think we are still not yet.
And then and then what's happening is okay you build this product you give it to the people and then it's new so everyone can try to do all the crazy things that they can think of right so they build apps they create documents they create presentations they schedule things that probably they don't need to take place every 5 minutes right so there's also the misuse right so and and and and admittedly we we were let's say first you release a technology and then you fix the experience, right? We are in this kind of idea at this moment because it's also very important to validate the technology and it's also changing very fast. So yeah, sometimes you release it and then you don't make it clear um you know you don't make clear prompts. Do you really need this to run every week? Do you really need this to every every five minutes? Right? And do you really need to run every Saturday? Why it runs every Saturday?
Demetrios (03:04) When it’s the same in the data world, right? You didn't need data to be fresh a lot of times and a lot of ways that you could save money was 100% just by exactly reduce the the frequency right I agree it's just much more expensive and easier to set up in the data world it was only the data folks that were and they knew and that was their responsibility and someone was looking at them right so now you have given this powerful tool to everyone to everyone which by the way they lot of them they still to put down lots of effort into what can it do? Yeah. How can it help me? And they can make the same up 10 times because the first time they didn't know. So the second time they try and then this is the cool thing. They try to do very cool things that you we didn't even think about.
Demetrios (05:01) So you're pushing that down and that's cool. But then you're also getting this repetitive nature because maybe the first time it doesn't come out as right and then they just throw away the whole code base
Ioannis Zempekakis (05:12) And then you do it again and then you do it again and then again. So there's this type of and that gets expensive and that can get expensive very quickly because the context is getting expensive and then you need of course to go into context management. You need to go how do you how do you do semantic search on your skills right? So how do you feed it? In the end, what works best is um the simplest, right? So, give the the agent as less instruction as it require um because then you are not over overcrowded with lots of context and it works better even. So we went into that path and we went into that path first with cloud and open AI and then we find out very quickly that um the the the costs are growing exponential day after day. Um,
Demetrios (6:06) And this is according to you with the subsidies. So you're expecting that eventually the subsidies will run out and it's going to get even more expensive. Exactly. You recognize that's not feasible.
Ioannis Zempekakis (06:25) Yes, that is not feasible because lots of applications during this transformation are actually becoming viable. It's a in the end it's a unit economic right. So if it cost me $10 to you know summarize my emails I can live without right but if it cost me 0.00001 then that can become a big pipeline of 10 things that in the end it can make my life 10 times faster or better. So then you need to bring the unit economics in place and then from…
Demetrios (06:51) How are you deciding what use cases are worth doing and which ones aren't? Is that where the hard work comes in? Is that why it's so hard?
Ioannis Zempekakis (07:00) Yes, it's very hard. Um
Demetrios (07:03) Because at the end of the day you can't have a judge looking at every use case and saying this one is worthy, this one is not.
Ioannis Zempekakis (07:09) No, there are two ways. The one the traditional one you just keep human in the loop. Someone is the So you make someone responsible for the budget.
Demetrios (07:23) CFO or somebody around there. Yeah.
Ioannis Zempekakis (07:25) Someone needs to pay. Exactly. Someone needs to pay the bill, right? And that person you tell him you have hundreds of something of tokens, right? And then he's the one responsible to you know distribute it accordingly. And then what is that? That's one way. And then there you can make everyone's life easier. You can explain to the users how much something costs. You can make a little bit of educational.
Yeah, there's a little bit of education. There's lots of underground uh you know, you can become more informative. For example, hey, you know, you don't have to rebuild this. Someone else has already built it.
For that in mind, we are building now and it's live a marketplace that you can select different apps that others have built, different agents that others have built, different skills that others have built.
Demetrios (08:09) I've always had a lot of hate against that. And I'll tell you why. Nobody actually wants to comb through the marketplace to see if they potentially can find their use case and then tweak a little bit of it. Maybe they send their agent out to do that and that that can happen.
(08:32) But if it is so easy to just reproduce it on my end, why wouldn't I just have my agent reproduce it in the exact way that I want it?
Ioannis Zempekakis (08:37) Yeah, I agree. So I think the first objective of marketplace is educational, right? So it shows what is possible, right? Okay. Yeah. So lots of people don't know that. So that's my first objective.
The second on the on the product point of view, you need to be able to surface this work in the chat and not the user will. So the user shouldn't go to the marketplace and start searching. If I ask something um build build me a to-do list to track my the teams the velocity of my team on the PRs and the productivity durometrics right so then it can immediately say hey someone else has built it boom you get it ready that would possibly work right that would possible work right and because then sometimes you know judging all these 100 things per dashboard it also takes effort right so in the past you had the person who's doing that and only that now I need to and then I have the space and Dora and then I need to explain to the agent how I you know and all these things you don't want man if someone has done it I don't want to invest creativity on there I prefer to invest creativity somewhere else right so so that serves that goal right on the marketplace so we are also going that to avoid you know rebuilding lots of things but the biggest change was we build very good evidence we have talked about this a year ago so much I think the first discussion that we had about having this discussion was on reels. I remember we scheduled this call and you never show up.
Demetrios (10:13) Me not show up to a party. And you know who you're talking to. Um and we were going to talk all about evals back in 2024 and then again in 2025. Now we're in 2026. Exactly. We've had so many and then it's evolves.
Ioannis Zempekakis (10:30) And then it evals, evals, evals, in the end. What you want is that you want to find a way to make your engineering work more professional. Yeah. And and evals is a is good evals is the way to make it more professional. So if you build good evals, you can run and benchmark all the models per different tasks, right? You you have cluster your evals and then you can say for this task I need this model. For this task, I need this model.
Demetrios (11:02) Um and this is where we moved now, right? So, we're really getting granular though on a the task and b the evals with all the different models. Sounds like a headache.
Ioannis Zempekakis (11:12) Yes, that's we go back. It's hard, right? So, you get lots of things for free if you rely on just hit the API. Just hit the API. The most expensive model
Demetrios (11:23) And ignorance is bliss.
Ioannis Zempekakis (11:24) And ignorance is bliss. And by the way, their SLAs's are working quite good because any other NeoCloud today, they are still trying to figure out and optimize and it's it's it's not GCP, right? So, if there is one thing for sure, it's not GCP, right? GCP's had time to figure this out. Exactly. They had time to figure this out.
Demetrios (11:46) Should we go on the tandem? Yeah. Let's go ride some bikes. Do we start? Yeah, we're starting. We're start. Dude, I you're much more maneuverable than Dex. Yeah, man.
Ioannis Zempekakis (11:59) Cuz with him, you have to commit fully. He was trying to lead, bro. If he has decided the the path is right, then you just can't go.
Demetrios (12:08) I couldn't choose. Even with the the steering wheel up here, I could not go. So, but anyway, you know what I was going to talk to you about? You know what I think what you're saying reminds me of is this idea of total cost of ownership. Have you heard that term thrown around? Yep. Yep. Couple of years ago. Yeah. Exactly. And now it feels like what you're saying is we thought AI had this total cost of ownership, which was we would just ping an API and figure it out, which in in a way it does kind of have that, but then you start to realize we're not prepared to pay that cost. We looked at the total cost and we said I don't want that.
Ioannis Zempekakis (12:59) Yes. And also I think there are lots of applications for intelligence that we are still scratching right. Yeah. So oops that wasn't me.
Demetrios (13:10) Nice on the fly. What do you mean by that? like we're still scratching the surface on certain things that
Ioannis Zempekakis (13:18) I mean we are still we are still in the the ROI has not been maximized yet.
Demetrios (13:25) And you said this to me before. ROI is a funny one because I've heard that the finance folks of public companies don't like to talk about ROI as much these days since they have to go and then say that on public earnings calls. So they avoid that term at all costs. And when you say ROI has not been maximized or found, you're just kind of saying like we don't know if what we're doing with AI is actually providing value. Is that what I'm understanding?
Ioannis Zempekakis (14:02) No, no, no. for the same cost if you have completely optimized your way of working you can extract much more value. Yeah. Right. So think it like that. Think it like um even if it stops tomorrow the development of the models there's still been for next two years with the same level of intelligence we can do optimizations. Yeah, right.
That's what I mean, right? There's a lot of stuff that you can squeeze out of it. Yeah, there's lots of stuff that we can squeeze out of it. There are lots of processes that you can remove and there are lots of process that you can add, right? There are lots of things that you can do on on user experience to reduce waste and and and and lots of engineering optimizations, right? So, we can we went go a little bit left here. Yeah. All right. There's some glasses. Oh, thought I understood. Left, left. Uh, all right. Don't worry. Just going with the flow. You don't have an option, bro. I'm controlling this Yeah. Um, so that's the that's and then the the the value then comes when you really the easiest optimization is actually controlling the context. Yeah. And uh then the models, right? So how you can find out Mhm.
Uh which model do you need for which tasks? Yeah. Basic router. Basic router. Not the smartest router on the world. Just a basic a basic router. Um and that'll get you pretty far. And that's and that's where we are at the moment. Right. So in with this approach we have reduced our cost to 10x. Mhm. And we are just starting with this.
Right. So there's going to be much more optimization going forward. But then at the same time we are also building 10 new features every month right so you so the pace is out running the optimization yeah that's I I think we are in this phase right so I think we are in the phase where you know you you you cut to one and then you see two popping out right isn't that your Greek history isn't that what happened exactly Yeah, exactly. So, we need Hercules. Yeah.
Are you that Hercules we're looking for? I'm not Hercules yet. Not yet. Not yet. Not yet. And then what is cool is you can start start training your own model as we are also start doing with LCM where then you are start plugging in into specific tasks and that's all for this this whole idea of ROI. Yes. Don't stop paddling or you stop paddling on purpose. On purpose because Yeah. Exactly.
Should I go down? I think so. I'll see you on the other side. Yeah. We We don't have to get any Oh, I almost hit that car. Wait. Run. Jump on. No, no, no, no, no. You don't want that. All right. Ready? The three. One. One, two, three. This is nice. Full thrust. You just got to go with me, huh? Yeah.
So um and then you start fine-tuning smaller models because lots of experience um it also requires a much smaller model in also in the sense of latency right?
Demetrios (17:46) Yeah. But you are then having to go and recognize which use cases are going to be done repetitively. Yes. and how can we optimize that so that it becomes a workflow and we can use the smallest model to get us the end state that we're looking for. Exactly. And a lot of these use cases are one-offs. Yes. Are they?
Ioannis Zempekakis (18:12) Yes and no. So if you if you take an organization Mhm. that has a logistics then it would probably pay off to start doing some uh post training or fine-tuning on some of these logistic traces. Right. Okay. Um and then you expect to get uh similar performance maybe a little bit better at 20 30x the cost at 10x the latency or 5x the latency. simply because the model is is much smaller, right? So there's no smaller, faster, cheaper.
Yeah. Exactly. Exactly. Um and then you can't do this for everything, but for example, you can probably train a model and that's something that we are discussing for doing specific data analytics, right? Uhhuh. Because this is I mean yes, it's never the question is probably never the same, but everything around is exactly the same. I think we're clear.
Where are we going? Clear. We're going straight and then we're going left. You're saying the question isn't the same, but the process is the same. Yeah. The process the same. The data stays the same. Um the the metadata, your meta data, the databases, the databases, right? So you can you can have probably a track of 100,000 questions that have been already asked, right?
Demetrios (19:33) But does it need it? That's the question I have. Like does the new We're going right here, right? Pedal. Yep. Yeah. Big turn. All right. All right. Right. We did it. And then we're going to do another another one eventually because are you not just doing hard engineering problems for the sake of hard engineering at that point? You know what I mean? Like does the Is the juice worth the squeeze or is it just a whole lot of squeezing? Um, do you understand what I'm saying? Yes, I understand. Training a model or rlinging a model, going back to that total cost of ownership, there's a whole lot of things that you need to do. Yeah. To make that successful.
Ioannis Zempekakis (20:30) Yeah. I I I think the answer lies on the it's always related to scale, right? Yeah. So if you do it for five people, then no, it doesn't. If you do it for 5,000 data analysts or 5,000 engineers, yeah, that they hit this every day, then it does. Now, have you can you potentially run this exercise to the cost of I don't know uh the the engineers spend also 3 hours doing this instead of no, but you can you can make some good you know you can run some good numbers and then you can see okay if we go that way mhm… with that type of extrapolation then it will pay off right so that's the first thing and then you're also building the muscle right so you are training your team and also there are lots of byproducts of this for example the building a model router right y so building a model router is one thing and you can buy it let's say yeah there's a ton of them out there yes but you need also for a model router you need also to have an amazing eval suite right and then you you go going to buy products which sometimes
Demetrios (21:48) Yeah, the steps removed of what the main thing you think you need and then you realize wow if we're going to have success for this thing then you have to look a little bit deeper.
Ioannis Zempekakis (21:58) Exactly, exactly, but then yes it is engineering and and I don't think so that GenAI has been removed the hardcore engineering uhhuh I think I think contrary because now you can build so fast. You need the basis to be very solid. Uhhuh. Right. So, you need good infra. You need good observability. Where do we go? Right or left? Left. Left. Left. And then we're going to left. We're going to cross here. All right. See if we do our right crossing. Ready for that? Ready for this? Oh. Oh. And then we go right or left. Left. Left. Left. Left. Left. Okay. Left. Into the park. I'm trying to go into the park. Ah, get a little But there's stairs there, so I don't Okay, so we go right here. We go right there. No, there are stairs there. I know that's not going to work.
Demetrios (22:53) Okay. Okay. Look at the lion, though. Lion in a bear park. This doesn't look like a lion, but I understand. No, that's totally a lion. No, that's definitely not a lion. Oh my god. You're just Except if it just had a haircut. Can we stop pedaling? All right. All right, coast. Okay, so going back to it, the muscles you build are valuable and it's much more beyond like the basic hey
Ioannis Zempekakis (23:29) 100%. And then what it happens there is a byproduct is that these muscles and these capabilities can actually be then launched at scale to the whole organization. Right? So the moment that you find out how to good evals, yeah, then you do them for everything. Then you do it for everything because every agent will require some evals one way or another, right? So then you can package it and then and then and then enable everyone to do it, right? And and then that that depends on the it's always on the scale in a scale of 30,000 employees like process ecosystem has at the moment. This is incredible. So it can it can completely change how we do engineering and develop
Demetrios (24:15) Thinking there, huh? On that turn, you got a little scared. I heard it in your voice. Oh, we're going left. And so, okay, there's the cost benefit analysis that you're doing too to decide, does this use case warrant us to go and try and do our thing on our own or try and create our own version of this internally, right?
Ioannis Zempekakis (24:44) Yes. But also there's another thing when the technology change so fast, you want to build some technology in house.
Demetrios (24:52) Oh. Oh, he said no. Keep paddling. Okay, I can do that. Keep paddling. No bikes. Well, we'll do a lap. Ah, but we didn't know that. Yeah. So, the Dutch person knows, right? He does. He knows the rules. He has done it. For some pull laps. Yeah, man. Let's do some uh keeping, right? Yeah. I know, dude. This would have been a lot easier. So, um so anyway you were telling…
Ioannis Zempekakis (25:28) So, yeah I remember so I was telling that because technology is changing so fast you also don't want to lock yourself in in something that you don't control. Mhm. Right. So there's also a value of optionality. Right. And those are the muscles that you're building. Yeah. So then that's one of the advantages if you build imagine now you commit into a there is no clink clink clink there is but
Demetrios (25:56) I don't I don't want to do that to the kids on their shoulders scare them. So the um imagine now if you commit to OpenAI or claude and then you're locked in because you built everything on it. Yeah. Exactly. it's not so easily swappable.
Ioannis Zempekakis (26:10) and and and I mean OpenAI is probably let's say the safest bet but there are also lots of open source libraries out there. Yeah. Imagine if you have commit to uh Open Claw. Uhhuh. In December. Yeah. Uh probably around March to scratch down everything and then follow her. Y right. And now open code or pi code. Right. So and then and then these projects can also divert uh so then you always need to think okay is it then do I converge back right so there's the open source there are there are other small companies that they can pivot any time also right so the fact that you are a customer doesn't mean that they don't decide tomorrow to do something else and then what do you do with with the whole stack right well Demetrios (27:00) Yeah, even Amazon or GCP can do that too yeah exactly yeah yeah yeah I mean there's so many products that they've released that eventually just died. Yeah, I agree.
Ioannis Zempekakis (27:18) So you so in that sense in that sense having a let's say being a not being the first that you adopt something it might also be if you want to scale it's sometimes good option right or you build it yourself because then you can quickly pivot so if you think about account we start building a first chat Mhm. GPT assistant on Slack in Slack. Yep. Yeah. And it was just a way to plug in ChatGPT to Slack.
Yes. As a way to to talk to GPT3 in 2022 or 2021. One month before Chad GPT. Yeah. Um and then we pivot from there into an let's say a web application where you you had to select different roles. Yeah.
Um, and then you went from this to an agent where the agent could select the tools and now you weren't into a harness, right? And this is only realistically possible if you control at least a big part of the stack.
Demetrios (28:24) Yeah. Um because otherwise what you're just at the liberty of OpenAI doing that for you.
Ioannis Zempekakis (28:34) Yeah. Or or if you if you if you if you use or if you use another startup or any other let's say open source library there's always there is that there's the other thing around which is better faster cooler and then you either need to commit to what you have and make a pull request and then pray that the all the forces will align to get accepted or you need to making changes to your own to to your to your to your stack but then you don't you cannot ensure easily backward compatibility. Uhhuh.
Demetrios (29:13) Well, I guess what I'm thinking as you're talking about this is there's a certain amount of use cases that you're trying to figure out where to invest your time in to optimize and you want them to be at a scale, but you also want them to almost be like generalizable in a way. So, you were talking about the data analyst and it's not that you're learning a one data analyst workflow. You're not optimizing that one workflow. You're trying to optimize as much as you can in the data analyst like complete workflow. Yeah. The suite. It's not just that one thing that the data analyst does, right? And so there it's it's more generalizable than it is use case by use case. Yeah. How do you go through and you look at these and you recognize all right this one feels like somewhere where we should be investing our time and effort.
Ioannis Zempekakis (30:13) Yeah, good question. There are some operations that they are critical for business, right? Uhhuh. And those are probably those that they've been measured over the time. They've been there data metadata. Do we step out here? Yes. Right. They're stepping out. Don't fall. All right. One, two, two, three, send it. So the metadata Yeah. So there there are there are already some operations that um actually there are operations that you have investing already effort before AI. Yeah. Right. So customer support, data analytics, um account management, optimizing your marketing budget, optimizing your logistics, right? So you know, right? So the business didn't change, right? So then you know what are the use cases that you need to invest and make them more scalable and then at the same time what you need to do which is also our our strategy is that you want to give you want to unlock all the other new things that are now possible right and the way to do it is by giving a tool like I don't know tokan where you can build apps um and then everybody can build apps now right and then you can see people that before they need to have an engineering team Now they can they can literally just go and build something themselves and then you can see probably over time how many other people are using it. Uhhuh. Right. Uh how transforming the business growth. Yes. And then you are start getting new insights. Okay. Now this is how we'll shift. Right. So that's that's a more bottom up approach. Right. Of course there's then the the top down which is okay where the company wants to go. Right. So, where the company wants to invest, um, where do you want to optimize and innovate? Whoop, that was on me. That was me in my Yeah, it's crazy. I just do a little bit left and then it feels like you go complete left. We're going right here. Yeah. Did I answer your question?
Demetrios (32:20) Yeah. So, the other piece to this that I think is fascinating that we've talked about before and I don't know if you really have a clear answer on yet is who are you hiring and what kind of skills are folks expected to have because AI engineer means so many things now? Yeah. Right. Like what does an AI engineer even mean?
Ioannis Zempekakis (32:45) So, you don't hire for for sure you don't hire for titles. You hire for specialization in a specific area. Yeah. Of your product. Right. So what do I mean by that? It's probably extremely hard to find a person who has s experience. So he he knows his his his infra and platform and AI and evals. Right. So the whole and the whole stack, right? Yeah. So what serving optimizing serving and GPU all of but there are some things that you cannot discount and this is for me good software engineering design thinking around this right so how you can design good lean scalable solutions okay because the coding now more or less has been abstracted but how do you design your solution has not been right and how are you testing for that with the traditional old design system uh solution. So you do a whiteboard uh design uh system and maybe you do more than one. You might need to do two. Yeah. Uh just to really get a feel. Yeah. Where are we going? We're going to go left. We'll go up here and then go left. Which one? There. Yeah. To the crosswalk. All right. See if we can You tell me. See if we can survive this intersection. Oh boy.
Demetrios (34:13) Woo! Now we're on, wait, relaxation mode. All right. All right. One, two, go. Three. Let's go. Sure. Yeah. Sorry. No. Did I go too hard on the No pedal? No. Okay. So, you've got the traditional whiteboard interviews.
Ioannis Zempekakis (34:34) Yes. That needs to happen, especially now because coding itself doesn't say anything. You need to understand if you understand how to build things that they scale things that they can you know they are not overengineering now overengineering is quite simple because you just ask but then maintaining is hard you like it or not it's smelling the overengineered code that Fable tries to give you exactly and then if you ask what else I can do to improve it just keep on coming and keep on coming right so it never says I think I've done enough uh let me go back yeah congratulations Exactly. Um and then that's the first one. Then second, he needs to know AI, right? And and and and for me, this is a very broad thing, right? What do you mean by know? Yeah, exactly. So he doesn't necessarily need to know how to train a cat boost, not even a neural network, I would say. Um not for what we do. Yeah. Um there's another team in Prosus that they are responsible more for training the models. Uhhuh. Uh so that's not the thing but they need to have a good intuition and a good knowledge about how to set up an agent. Uh what are the existing limitations and when you say agent what do you mean? You can pick up Hermes for example right? Okay. So can can I pick up something around why for example Hermes mess has been a preferred agent over open claw right okay and have you tried and have you tested and what have you find out right so why open close stale right so what is the reason for it how do you build evals right so there there's a little bit understanding all of that is knowing AI exactly that's my kind of health mushies and then he doesn't necessarily need to you know go build evals.
Uhhuh. But he needs to know going forward the more successful products in terms of engineering and scalability is going to be those that they have a very strong foundation but then there's lots of work that's being done from the agents. Yeah. Right. So he needs he he doesn't need to engineer things if the agent can take care of it. Uhhuh.
Right. So he you need to understand whether this person can realize the the tradeoffs. Do I build this how I used to build it five years ago? Yeah. Or do I completely delegate this to be a decision of the agent? Mhm. Right. So do I build for example um a static dashboard with front end to show some numbers or do I allow the agent to you know develop a custom canvas style right dashboarding that it can change over time right so because the whole software is going much more into personalized software yep so if you build a very hard tightened product it's going to be very hard then to decouple it.
Demetrios (37:35) Yeah. So recognizing that product sense. Yeah. The almost expecting someone to understand that just as much as they understand the software side of things.
Ioannis Zempekakis (37:47) Yes. And I think the hardest part in hiring Yeah. is to find people that they know both because people that they are very very good at software engineering uh they have they are already having some years of experience. Uhhuh. And if you think about it, AI was there before. So then the question is okay, where I find a person who is good at software engineering, but then at the same time he So which means that he has experience in software engineering, but then at the same time he's in love with these new technology that he wants to try everything, right? This is…
Demetrios (38:22) But I'm sure that there's easier by the day. Yeah. Because everyone has test. We do the one, two, three. Yeah. One, two, three. Hey. All right. I'm going to give you some force. You gave me a little push. Yeah. To accelerate. Acceleration. There's your car. Huh? Nice. There's your car. Oh, that's my car. That's your car.
Ioannis Zempekakis (38:45) We've made it in one piece. I didn't expect that. Oh my god. You didn't expect that? I thought I would need to say sometime, hey, Nina, my wife, I love you. Please don't. You're probably not going to see me again. Where are you going?
Demetrios (39:00) I think we can park it here. All right. All right. All right. So, okay. We stop. We're stopping. Stop. I don't see you stopping. All right. You want to finish that thought? Uh, yes. Uh, and then we can finish the part.
Ioannis Zempekakis (39:15) Finish it. It's getting easier and easier. But I don't know if it's getting easier to find someone now. I think it's harder to find someone with AI engineering and then train him how to be software engineer because that requires a lot of grinding and with a with coding agents now it's much harder then finding uh software engineers that they are very open-minded I can understand and then yeah and then you plug them into a very heavy AI team and then they can start uh they can start you know smoking AI 24/7 smoking out you
Demetrios (39:51) But what I found is it feels like it is a lot of noise in what the title is or what like someone who says, "Yeah, I understand AI." Does that mean they understand how to code with AI? They understand how to build agents. They're using Hermes and they're they understand like the context window, they understand the serving side, they understand the infra side. What does that even mean? And so the lack of understanding in what or the shared concepts I think is the lack of shared concepts makes it much more difficult.
Ioannis Zempekakis (40:24) They are not job titles. Job description. So your job description really needs to be these are the five things that we mean when we say AI engineer. Let's define our terms. Yes. Exactly. Let's go back. Okay. AI engineer doesn't mean that you build an agent using Lancraft. Yeah. Right. And you know how to use cloud code. Yeah. Right. That's what I mean. Okay. And then and sometimes this is a interview, right? Yeah. So we spend we invest lots of time and talking lot to lots of candidates to find out who can be happy, right? Yeah. Because it's a it's a it's a different role than probably what they used to, right? Or if they've done it, then they know. So that's also a case, right? So you can you can also hire people that they are building similar products one way or another. They know. Yeah. Yeah. Then you need to check other things.
Demetrios (41:15) But the eval dude, the eval are what'll get you. Like, are they ready to grind out all these evals and that hard part, man? This has been awesome. I'll let you go. I'm glad we finally did it. It was a long time coming.
Ioannis Zempekakis (41:27) Thank you. Let's do it again. Talk about evals. We can always talk about evals.
Demetrios (41:32) As long as you're ready to get back on this tricycle, this tandem with me, do it.
Nice. Nice.
+ Read More
Watch More
Cost Optimization and Performance
Posted Apr 27, 2023 | Views 1.3K
# LLM in Production
# LLM
# Cost Optimization
# Cost Performance
# Rungalileo.io
# Snorkel.ai
# Wandb.ai
# Tecton.ai
# Petuum.com
# Mckinsey.com/quantumblack
# Wallaroo.ai
# Union.ai
# Redis.com
# Alphasignal.ai
# Bigbraindaily.com
# Turningpost.com
Do More with Less: Large Model Training and Inference with DeepSpeed
Posted Jun 20, 2023 | Views 1.7K
# LLMs
# LLM in Production
# DeepSpeed
# Redis.io
# Gantry.io
# Predibase.com
# Humanloop.com
# Anyscale.com
# Zilliz.com
# Arize.com
# Nvidia.com
# TrueFoundry.com
# Premai.io
# Continual.ai
# Argilla.io
# Genesiscloud.com
# Rungalileo.io
Putting the AI back in Medallion Lake Design // Simon Whiteley // DE4AI
Posted Sep 18, 2024 | Views 676