Packaging MLOps Tech Neatly for Engineers and Non-engineers

Jukka Remes [00:00:00]: You can put the, we can extend on this later but basically you can put the expectations for things on our platform and then you can take it to the ML operations side and then you can also follow up on how those expectations are met. So I'm Jukka Remes, I'm a senior lecturer of ICT and AI at Hagaheli University of Applied Sciences and also the founder and CTO of a two wave AI. And yeah, mostly I drink tea.

Demetrios [00:00:39]: We were just talking about open source platforms you mentioned that is very, very top of mind for you. Almost passionately top of mind. Why is that?

Jukka Remes [00:00:52]: Well yeah actually I'm a quite late entry to the like actually contributing to the open source. I've been mostly using those but then we like found this new life for this platform that we created at Silo AI for envelopes, a certain type of open source platform and now I've been working since Silo in academic setting and then I've been in the research world before and I really, when I started things there weren't these kind of like stacks available. So I'm kind of coming the full circle back to the research world which would benefit from MLOPs and then this like having the open source platform which we can develop for that purpose specifically then it's now one of my contexts that I'm living out.

Demetrios [00:01:56]: Can you explain to the listeners what was Silo doing and why did you need to create a MLOps platform?

Jukka Remes [00:02:04]: Oh yeah, so I actually yeah I joined Silo originally as a. Because Silo was like now it's acquired by AMD and the realities might be different but back when I joined it in 2020 just after the COVID time started it was a consultancy largely and I joined as a solutions architect. So I did a client cases and big and small and many of those actually that I ended up working with Envelope. So there was like this demand at that time to build practices for the envelopes as well as then the platform settings and then we also like then launched as one phase of Silo to have this like technology areas and I also took up on leading the cloud AI and mlops area that we had at Silo and it entailed like both competence development because we had like people with very various backgrounds at silo, especially coming from academia. So very strong deep research based AI and kind of data science type of skills but varying levels of engineering related to AI and mln. Then partially we started putting effort into the competence development, having information sharing sessions, workshops, workshops like all kinds of things, visiting lectures and such. But we also started developing that because we had a certain amount of internal R and D activities at SILO even at that time. So we started developing in my team then this platform based on open source components.

Jukka Remes [00:04:10]: And it was that it's quite like loosely, it's kind of integrated but quite loosely. So we are basically collecting those kind of like open source technologies that made most sense and we're going to extend that and make the platform for certain kind of client cases so that we can they can for those clients that completely for example outsource the ML development that we would have like a platform to do the development on and deploy the models and such. But then also regarding this kind of like cases where we have clients who are bit more mature or even really mature and they want to build their own platforms or parts of the like MLOps setting, then we can leverage for example even just part of the platform setup that we have for those purposes. So that was kind of the origin of the platform work in itself. Yeah.

Demetrios [00:05:18]: So classic story. You have AI researchers and these ML researchers that don't have as strong of engineering chops and they probably want to see their models and their work get into production. So you create a platform to help that process. You mentioned something there that I want to hit on, which is you not only created the platform, but you created more than the platform. The processes around that. Can you explain what those were too?

Jukka Remes [00:05:50]: Well, actually not internally that much. We didn't like standardize internally that much. We pretty much focused on the, on the platform development itself. But of course like in the client cases there was like what would be the workflows and these kind of things for various clients.

Demetrios [00:06:14]: Take me back to that moment when, especially internally, when you're saying okay, we need to start building an ML platform. Where do you even start? What do you go with and why is that the place to start and then fan out from?

Jukka Remes [00:06:31]: Well, it was like based on the kind of the demand that we were facing. So it was very practical in that sense that it had to do with the like what we thought that we would need when engaging the SILO clients at that time. So we would need the technical platform specifically?

Demetrios [00:06:46]: Yeah. Which piece of the technical platform?

Jukka Remes [00:06:48]: I mean is it like end to end? Like. So we, we basically incorporated, which is still the extent that we got at that point at Silo. We got the pipeline orchestration and then the model management. So supporting both training and development and the model and metadata management and for the experiments and then the deployment, but also included some components for the monitoring part so that was the kind of the intent and we covered the whole extent end to end, all the way from training to deployment and operating the models.

Demetrios [00:07:35]: Now I'm curious, did you also do stuff with the data?

Jukka Remes [00:07:41]: No, we didn't actually go that way then ultimately. So yeah, that was one idea that we would extend to that direction, but never got to that. So the kind of the reality at Silo, as in any this sort of fast evolving and fast growing company, was that the direction changed in somewhat like how we do things. And so new things came along and then we kind of developed a platform forward in this R and D collaboration that we had with international consortium a bit. But then we didn't that much yet anymore like put the R and D efforts there. But there was actually a separate like R and D entity within SILO established. And so most of the R and D went kind of like a bit to other direction. So.

Jukka Remes [00:08:42]: And ultimately this also led to the possibility and the idea that let's open source the platform because within that consortium like University of Helsinki, there was like interest for that also Fraunhofer Institute in Germany and some of the companies involved with that.

Demetrios [00:09:05]: And so if I'm understanding this correctly, you saw for the external clients of Silo, you needed to create some way that these folks could standardize and get these models into production. And so you stitch together a few open source tools to make sure that you were landing on these key pieces within the ML development like life cycle. And after you had done that you recognized, well, since this is pretty much a full platform in itself, why don't we take what we've built and put it out there into the world?

Jukka Remes [00:09:49]: Yeah, exactly. And then of course like continue with that because then we could involve. The University of Helsinki has continued developing certain things on top of that actually in the research like context because they have been making a setup, extended setup with that platform. We are soon like also merging that effort into the, into the main repo of the platform. But they have included support for this high performance computing or supercomputing in the centers that are available in Finland for research use. And we are still actually continuing to extend on that. And then I have now continued in the university context myself with the students to work on the, on the, on the project.

Demetrios [00:10:43]: So tell me what use cases this supports because I imagine it is certain types of models that are being built or it is certain types of, of use cases that you're supporting or is it across the board you're trying to help support all different models.

Jukka Remes [00:11:06]: I haven't yet myself looked that much into how the components keep up with the large model context. So have they as far as I understood from the universe, the Helsinki that the. Because we have basically Kubeflow and then we have mlflow and then we have Kaiser from the Kubeflow ecosystem. So they are pretty generic. Right. So you can basically do anything with those. But then how practical it is or how efficient it's to do the trainings or do deployments with those. Regarding the large models, that's for example one thing that it seems to have been tested by University of Helsinki.

Jukka Remes [00:11:54]: But that's one new thing that I and I'm looking into with them. But yeah but in general it's pretty much meant for like to cover all kinds of cases. Of course often if I go present to some especially certain fields of research as they bring up some more. I don't know if they are exotic or edge, but perhaps a bit more rare cases. Like what about the federated learning here and this kind of like setup. So but the basis, I mean that's very generic. But then of course like it needs like extra, extra work to perhaps cover certain kind of cases.

Demetrios [00:12:40]: Since we're talking about researchers getting a bit more platform chops or using an ML platform, have you pondered the value that an ML platform brings to a researcher?

Jukka Remes [00:12:59]: Yeah, so like I said, actually I need to comment still on the previous one that you can of course run any bigger workloads as well with any GPU requiring ones. And of course with the kind of the additions that we are now doing, there's a super computer environment that has a lot to do with leveraging the GPU superclusters that are available like the AI factories even though like the clusters that are within the scope of the AI factories that European Union just funded and which are being launched at the moment. So yeah, so the intent is to go to that direction that we can also train large models also with this platform. But yeah so the value of the platform. So in research. So like I said I was working pretty early on like before the whole even not just before the deep learning times started but before the data science kind of like era started as well. I was at University Hospital of Oulu in Northern Finland and it was like that time analytics and ML. I was running certain pretty ready made algorithms through all like data from brain imaging through those and there was a lot of like within that research, a lot of like variation as to like what kind of like parameters do you use with those analytics tools and what kind of filterings of data or other data pre processings you need to have there.

Jukka Remes [00:14:50]: And then when I had myself, for example, that kind of setup where you have these different options and you need to somehow orchestrate running a very big set of different kind of experiments basically or analytics jobs with different kind of pre processings than handling all those combinations and the results therein, or even creating the running of those. Then I ended up doing that without any of the tools that I had so far. So that's largely kind of the basis of my appreciation for the current stacks because it was really tedious and I see a lot of opportunity, I think. I'm not sure if you are familiar with knime. I guess that's. Yeah, I guess it has like found its place like in bioinformatics at least and in some. So KNIME was kind of like the, I think the first one emerging at that time when I was doing this and after then came a lot of like different things. But yeah, even the Python, like of course it was used a bit, but it wasn't like in that big of a use, especially in those fields that I was in yet at that time.

Jukka Remes [00:16:10]: So this bringing the consistency to complicated research processes like increasing the quality, we are supposed to trust the information, the scientific results that we publish from there, right. So they are supposed to be the creme de la creme of all the information that we have in the world. Even though I'm not sure like how much fake science there is also nowadays with. But still like the quality criteria are very high. Right. So if you are doing the complex stuff, then you need something to deal with the. Even executing that complex stuff, but then making sense of the complex stuff and the results.

Demetrios [00:16:54]: This is a fascinating statement how the reproducibility of researchers work potentially comes into question. If you do not have a standardized way, or at least a way to actually go and put that model into production. If it's just research that's being done in the lab, then you write a paper on it, that's great. But there could be some problems when you go down the line and you say, all right, now let's actually make this something that the industry can use.

Jukka Remes [00:17:34]: Yeah. And even, even like, even if we stay in just the science, then like the kind of how well can we assess like this goes back to the, again like more generic things than rather ML alone. But it reflects on the ML as well because there are reviewers that need to be able to review the results. And then if it's much more credible and it's much more Efficient to review if you can share everything. So you can like share nowadays of course, like there's more and more like you are able to share at least the experiment setup code and parameters and these kind of things. The data is a bit tricky given what kind of constraints might be related to that use. But it's even better if you can share it in exactly the same way as you run so you can have the full pipelines and the full experiments as such and somebody can replicate that, then it's much more accessible in some sense. But yeah, like you said, also the kind of like taking the research results into any practical use, then if there's that sort of applied research angle that the next step would be the commercialization or the productization, then there's a huge gap if you cannot take it.

Jukka Remes [00:19:03]: I'm always myself reflecting. For example, I was working at Nokia Technologies for a while and then I saw there and I saw at IBM where at which I was working after the Nokia that okay, yeah, there are both of these big corporations, they have the research centers, but the research is actually a lot in many cases like even there in that sort of corporate setting, disconnected from the kind of the product development. So they are kind of like different all the different stages. So perhaps they are like lifted and sifted in certain manner. The results then to the product side and taken further there. But there could be much smoother transition to actually commercialize. And I think now with the advent of the. Or even it's not even an advent anymore, but the coming of the regulation like AI act and these kind of things, then it's.

Jukka Remes [00:20:01]: If you need to have all this probability of that you have done your new things when developing the model. So how can you prove that if you don't start that from the research stage, if you try to catch up with that in the product development phase.

Demetrios [00:20:23]: Yeah, I don't know how it was at IBM, but I know that a friend of mine, Danny, was working at Google from I think it was 2015 to 2020. And his whole job was to figure out how to take the research that was being done in the ML sphere and plug that into products that Google had. And so he was working closely with the researchers and anything that was coming out that seemed promising, he would go around to different teams and say, you think we could use this? You think we could maybe plug this in here? Would it be useful if. And then he would sit with the PMs and try to figure out if there was products there.

Jukka Remes [00:21:08]: Yeah, that's also like one perhaps Aspect like how, what is the driver? I'm really for this sort of unlimited innovation, if that's possible, and kind of researching something that is not constrained by the, by the business. But then you have this kind of a bit like tech first kind of approach there. Right. So that it's not necessarily like addressing. You're kind of going it the other way around because the results might not be based on the real world problems. Right. And then they are not addressing exactly some problem.

Demetrios [00:21:50]: Yeah. And sometimes you can create something very cool that people don't figure out is actually valuable until five years later, like the transformer models. And it's not even you who figures out really how to productize that and bring it to market. So it's a fascinating one. And he was also the same guy that told me when Chat GPT came out, he was like, yeah, I remember when Transformers were at Google and I was trying to figure out ways to put them into different like Google sheets and figure out if we could add them to Google Docs and, or the Translate app that they had. So it's a fascinating one to think about. Now I, I also wanted to touch on this platform that you're creating a bit because of what you have under the hood. You mentioned that you have Kubeflow, you've got some MLflow and then you've got Kserve.

Jukka Remes [00:22:50]: Yeah. And then Prometheus and Grafana. Basically those are the kind of the five things we were supposed to like extend on. Then it's completely possible because it's pretty simple, but it's packaged together. I guess that's the kind of the beef of the whole thing in terms of making it easy to use.

Demetrios [00:23:08]: And so is it another abstraction layer that sits on top of these different tools or is it that these tools now can just really easily be deployed as one cohesive unit?

Jukka Remes [00:23:24]: That's kind of like the one major thing because I think that like playing around with the platforms and trying to build them like some separate set of like instructions, like according to some separate set of instructions and then manually that's like perhaps automating text, but in a custom way. It doesn't make sense like for many users. So first of all, it might be like tedious even for some people that deal with the infra and deal with the platform. If it's a new thing to them, then let alone let's say that we have that university student or researcher from any field that just wants to run the ML. So if they want to take this kind of setup into use and they don't have access to some cloud service as the, for example, universities not often have and not in the companies even have 10. This kind of helps with that. But the kind of the philosophy originally also around this would be that then you would develop this kind of like more of pipelines on top of that. And of course when we are using kubeflow as a basis, then of course then if you have only kubeflow pipelines or kubeflow there, then you end up defining them in the way that you define things for QFlow with the DSL and that kind of things.

Jukka Remes [00:25:04]: But that was the kind of the idea that we and 1 could provide like these pipeline templates in a sense that that would be the kind of. Then the, in a sense the more integrating things that you would have like the end to end pipeline for all the different stages and using all the different parts of that. And you could aggregate this like library of pipelines. So it's a bit similar as what you can have for example with AWS, SageMaker and Azure ML. Right. You can define pipelines there and then the whole team or community or whomever can share them and use them as a basis easily to launch their own things on top of those platforms. So that was kind of the philosophy here that we have pretty loosely coupled services. So those services are not that much yet at least like integrated within the platform.

Jukka Remes [00:25:56]: What is more is the kind of like integrated package in a sense that you can install them easily to put in place. Of course, like this work then that my students at my university have been doing on top of that, like with Git repositories and CICD pipelines to go with this or any other platform. That's kind of like extra layer also there.

Demetrios [00:26:27]: Well, you've given me a lot of confidence in the youth when I hear about AI researchers using Git and cicd. That is amazing.

Jukka Remes [00:26:37]: First of all, yeah, I'm not sure like how many actually use that. I would guess that many use some notebooks just as a. From the corner of their own laptop's hard drive.

Demetrios [00:26:52]: Yeah, but this idea also is something that makes a lot of sense to me in that I can as a researcher go and grab a pipeline off the shelf and not have to worry about standing up these different tools and taking the time to figure out why did installing kubeflow just crash my whole system. You know, like it is a much easier type of workflow if I can now go and grab these pipelines off the shelf. They come with everything that they need and I don't know how. How it works at Chicken or the egg. Is it that I get the pipeline and by using the pipeline it installs everything. Maybe it terraforms out everything that I need as part of that pipeline. Or do I do I get everything I need and then I can start using the pipelines?

Jukka Remes [00:27:49]: Yeah, it's more like the latter one. Yeah, currently. So it's not like that it would provision it any place. That would be kind of like extra thing to do as well. But the kind of the installer that we have for the platform is in the open source like we have from the Silo times the Google Cloud like manage kubernetes like how to deploy it there. But the main kind of installer with the open source version of this platform is such that you can. It's basically running with the kind so the kubernetes on Docker or in Docker and so then you can basically provision the platform in your own computer as we are doing. Actually there's a lighter weight with just the qflow pipelines there or you can do it to basically any machine.

Jukka Remes [00:28:45]: So we are for example using the virtual machines that are available in the Supercomputing center or this idea center of Science at Finland for that. But you could basically put them anyway. So that kind of like need to make them pretty generic in that sense. That has like being the approach that we have been using so far. But then yeah, after that then with this git related stuff that the students had been doing there, we are kind of like providing tooling with which you can set up like in a sense like configuration related, like git repositories and then ML project related. So you can establish this configuration repository for. It doesn't perhaps make that much sense for one person, but you could establish that for the whole team of researchers or data scientists or whomever. And then the tool, with the tool you can set up the projects for the individual people or per project.

Jukka Remes [00:29:58]: So those are the kind of the ML repos and those link back to the configuration repo. So whenever you have like some configuration that you need to update, everybody can just basically pull much of the changes to their own repositories. For example, where do the common platform instances reside? So that if the endpoint addresses are changing or these kind of things are changing. So you can. You can pull that updates to all the repositories that the actual ML developers are working on. And then the tool also set up the CI CD pipeline so you can kind of like hide away kind of this sort of like platform interaction or platform API interaction stuff behind those CI CDS pipelines. So it like you can configure what kind of like CI CD workflows and what kind of like file structure, like a directory structure you want in the repo. But after you have that and we have like a default of course like to go with that, then it just set ups that for you and then you can start introducing your Python code or whatever to the different steps of the, for example the ML pipeline.

Jukka Remes [00:31:17]: And then when you introduce that and if you commit to certain branch, it like compiles the ML pipelines within the CI CD pipelines and then deploys that to wherever your platform instances are that are, that are connected to that branch in the git. So this is kind of the way.

Demetrios [00:31:35]: It'S basically like you're like saying this is a self serve productionization tool.

Jukka Remes [00:31:47]: Yeah, yeah, but of course like you can use that for, like I said, also do the research. So after that you would have in the research context also or in company context, if you don't have some cloud platform to use, or even if you do, you can set up the platform easily to some environment or environments, different instances of that. And then you can work with the, as usual with the MLKit repo, but like focusing on the kind of the contributions that you want to do to the ML part and not to worry too much about the, to the engineering.

Demetrios [00:32:30]: And what are some of the pipelines that you've been seeing folks set up? It's also training pipelines.

Jukka Remes [00:32:38]: Yeah, yeah, of course like with the, we have only like the example pipelines at the moment. So yeah, so those cover the training parts, but also in those you, you can have also like as part of the same like qflow pipelines you can have the deployments. But the, the idea like especially originally was that you, you could like launch whatever so you can launch just separately the training pipelines, you can do the deployments as a separate things and so forth. So.

Demetrios [00:33:17]: Now I understand why you were saying it's also the ML part, not just the productionization, which makes a lot of sense. Okay, so now there is a bit of a juxtaposition that we talked about before we hit record where you mentioned teams desperately need ML ops, but it's not until there's this pain that they feel. And so you, you in a way are hanging out in a world where you can be accumulating tech debt and accumulating tech debt and not necessarily have to bring on certain practices or platforms to help you get rid of that technical debt until you hit a certain level and then you're crossing the chasm in a way. And I was just talking to a friend of mine on another podcast how AI built this about this very same thing, how there's no ops is very hard as a practice and so it inevitably will turn people away from it because it is so much work. But you get to a point where you can't deny that work and you absolutely need to do it. And so if you are a company in the beginning stage and you have not hit the pain where you need to figure your shit out, you can kind of live ignorant and blissfully.

Jukka Remes [00:35:08]: Yeah, I guess like, I think like a nice comparison that I heard was to like software DevOps like just recently that now it's kind of like a pretty, pretty common stuff in the software development itself. It still wasn't like 10 years ago or so, but with the envelopes, even though there has been like several years already with that. I think it's the. Even the awareness, let alone then the kind of the adoption they are pretty still like not perhaps in infancy anymore, but at low level. And well, as you said, one can go about your business or go about your things ignorantly and such, and then you at certain point end up catching up or needing to catch up with that debt. And I guess in research actually that's a bit like perpetual thing because there is certain kind of like perhaps like threshold that there's a kind of this, like we discussed like this standing chasm between the research and then the kind of the utilization of the research results in some product context.

Demetrios [00:36:39]: Yeah. And the product hitting a certain scale. And then once it's at that scale, then you start to have to make the product velocity go faster. And so to get that velocity that you need, you are figuring out, all right, well, let's implement some processes around this.

Jukka Remes [00:36:57]: But also like I think just for example, if we consider the lineages and the tracking of the results, then I think, oh, I suspect that the now the AI act might actually force to consider this that we talked about previously that how can you actually. How can the utilizer of the for example models done in the research, if they are utilizing those in a commercial setting? How can you really comply with the regulation necessarily unless you have the kind of the whole story behind those models? I'm actually a bit skeptical about some of the foundation models, like how will that turn out in terms of the AI act because you have model cards and you have certain information there. But does that ultimately suffice? So that might be this kind of traceability of how things have been done. One factor that will actually lead to more mlops adoption in the research stages as well. But I don't know what kind of incentive that is to say academic research. Then perhaps there will be some funding instruments guiding ultimately that if you want to get the funding for the academic research, then you should also have some sort of like you should somehow deal with these things as well. I don't know.

Demetrios [00:38:31]: Well, yeah, you already see it with, at the time of recording this. Llama4 just came out this week and I think there's a asterisk in the whole release notes that say, oh yeah, this is freely available for anybody except for if you're in Europe, then you really have to be careful with it and we're not going to take any. Or we just almost want to step away from that whole shitstorm and having to put our model out there open source and then potentially get sued or have the regulations beat down on us. So in that regard, yeah, it's, it's a little bit of you need the practices, but you also need the transparency to show what you're doing. If you really want to put it out there with the EU AI Act, I think a lot of people are unclear on exactly what you need to do because it is written in language and it is not written in code. And so the engineers that are creating the actual ML and AI or the researchers that are creating it are sitting around thinking, okay, how does, how does this apply to what I am actually making?

Jukka Remes [00:39:50]: Yeah, and I, I think it's, there's still a lot of like as far as I understand it now, there's many things up to both interpretation as well as pending definitions or even some parts of the certain things related to AI act. They are coming along only after a certain while. I have led myself to understand, from people that understand the, the relationships of standards to the regulations also that probably certain standards will be actually defining many of the technical details ultimately for those. But that's also like a longer process in a sense because like yeah, but there are many of those standards already so those will be probably like being leaned on when these things get actually interpreted or more set in stone. I have been working with the regulations in the medical field because that's the, besides the software and mlt third kind of like or more like domain specific part of my whole career, but not only to a certain degree regarding the models in that context because it was like earlier but now I'm working with the AI actor Because that's the partiality context of the company that I have been founding lately.

Demetrios [00:41:38]: What are you working on these days?

Jukka Remes [00:41:41]: Yeah, so besides these like mlops things that I'm. Those are like largely like this kind of like basic MLOps related and for example envelopes related DevOps things that those I'm doing in the academic context. But then I have now been founding this new company, 8wave AI. So there we are putting something more on top of the mlops. So we are basically like connecting the, the AI operations both from the training as well as the deployment side to the how do you guide and lead the AI project so that they actually provide both value as well as then are compliant with the regulation. So yeah, that's, that's what we are building product for. So.

Demetrios [00:42:44]: So and how do you build a product that fills that need? Because it feels like it is inherently a, a service, not necessarily a product.

Jukka Remes [00:42:57]: Well, it's a. Yeah, we are, well we are building basically a digital service but it's a product development. So we are not, it's not consulting. Right.

Demetrios [00:43:08]: So and so you're helping folks that are using AI and ML understand how the AI act in the EU has actual repercussions for them, is that it?

Jukka Remes [00:43:27]: It's more like that you can, you can put the, we can extend on this later but basically you can put the expectations for things on our platform and then you can take it to the ML operations side and then you can also follow up on how those expectations are met. So we are providing in a sense certain type of missing management layer on top of the envelope. So connecting the kind of the needs that for which the AI features or AI for processes should be built for. So, and it doesn't concern only the AI act, but the also the broader business needs so that you actually have some return of investment from the AI project. This is actually something that we, we saw both at Silo with people and as well as then before that with some of the people that are involved with the company that often the projects failed because of that. So they are a bit disconnected from the.

Demetrios [00:44:43]: So it feels to me like what you've created is almost a connector between the PRD or it is in a way it's a very strong and battle tested PRD for the ML efforts because you're helping spec out what is needed legally and for compliance reasons and then for business reasons. And so you were saying that the reason that a lot of different projects or, or ML products failed was because they didn't have that insight into building out everything that needed to be built out on the compliance and different stakeholders.

Jukka Remes [00:45:34]: Yeah, perhaps not before because of the compliance, but before because of that. Like they are not like necessarily building exactly towards what is needed by the business. So there's a broken phone. So everybody are working in their own corners of the organization and we are trying to change that.

Demetrios [00:45:57]: Yeah, I've seen that. I've heard that many times on this podcast. It's a broken phone or you're not really. Each, each expert is trying to optimize their little silo and they're not looking at it as a bigger product. And so you have someone who is trying to optimize the accuracy score of the model and later they realize, actually I'm building a model that doesn't do what needs to be done.

Jukka Remes [00:46:30]: Exactly. Yeah.

Demetrios [00:46:32]: And so you're, you're coming at it from a different angle, I think, than I've seen and heard other folks talk about coming at it from which is you're building this middleware in a way that is helping the business folks, basically every stakeholder, because there are so many stakeholders in this process. You're helping them all add their issues or their requirements to the document or to this software that you've created so that at the end of the day, when a product pops out the other side after that birthing journey, you have the right product. Not something that has been optimized in a cave without any feedback for. For many years.

Jukka Remes [00:47:22]: And you know exactly what has been happening there. So you also like keep track of all the things kept happen. So you have the like constantly the understanding like how it has come to be. So you're also tracking the kind of the operations there. And then you have the confidence to report for audit purposes or you have also the confidence to actually take it to production.

Demetrios [00:47:49]: Right.

Jukka Remes [00:47:49]: Because the business decision makers don't need to be afraid anymore because they can see that. Okay, yeah, we are here. We have done this. It fulfills disk and this kind of things. So yeah, we're good to go now.

Demetrios [00:48:03]: Isn't this what most project management software is trying to do? And why do you feel like it falls flat like a Jira or a ClickUp is.

Jukka Remes [00:48:12]: I think they are like applied on a certain level. We have a broader ambition there to really cover all the stakeholders in a way that makes sense to them. Of course, like you could like it's again like a. Same thing as with the envelopes. Yeah. You could build your own platform. Yeah. You could use some tool or perhaps like you could do like use Some individual tool from some place.

Jukka Remes [00:48:44]: You can, for example, crack experiment information in QFlow as well. But then often people want to use MLflow more than that, even when they have the QFlow. So everybody could build whatever ways of applying things. But what we are trying to do is productize that so that you wouldn't need to do that. Of course, that's the whole value proposition of the productization there to begin with, that you don't need to have these hacks or like custom things that you end up like also then needing to that much maintain.

Demetrios [00:49:28]: I see. I. Okay, so the, the whole thing here is that because I was trying to figure out and play it back in my head, why is JIRA not good enough for this? Or why is ClickUp not good enough for this? But you're saying not only are you gathering requirements, which is what you could potentially be doing in Jira or ClickUp, you're plugging into the platform so that it is tracking the experiments and it is helping document each step of the journey in this whole ML product life cycle, so that it's not like you have to go into JIRA and try and figure out a way. Are you uploading a CSV file to your Sprint to showcase, hey, here were all my experiments and here's the one that we're going with.

Jukka Remes [00:50:17]: Exactly.

Demetrios [00:50:18]: Yeah, you're. You're natively building it into MLflow and you're saying, we've got this connection with MLflow, so if you want to go and then figure out what the experiments were, you can click into it and see all of that inside of mlflow.

Jukka Remes [00:50:36]: Well, you can like see the, like, I think it's still different. Like, use that if you want to have like ML flow for hosting the models and such, that that's like as part of the envelopes, that's a different thing. But with our platform, then it's a kind of additional thing to which you integrate and then you put those kind of things there, that kind of like metadata there that the different stakeholders need to see there in terms of like lineages and then the performances and hitting the targets and this kind of thing.

Demetrios [00:51:16]: So it took me a minute to fully wrap my head around it. But I love that idea of.

Jukka Remes [00:51:27]: You.

Demetrios [00:51:27]: Have this buddy that is taking the journey with you to help make sure that you're documenting everything that you're doing. But before you even get that far, you have all of the stakeholders gather requirements and plug it in so that you're building the right thing. And then when you do take that journey you are automating a lot of this documentation life cycle and making sure that that metadata gets put in the right place and then is not lost. Or you have to go back and try and sift out two years later and figure out why exactly did we make that decision and have that model out there in 2023, et cetera, et cetera.

Jukka Remes [00:52:12]: Exactly. Yeah, yeah, yeah.

Demetrios [00:52:15]: Cool. Is there anything else that you want to hit on that we didn't talk about?

Jukka Remes [00:52:21]: I'm not sure. Perhaps this covers pretty much what I've been at and I just want to say that the of course like with the. There are other options like to the open source platform that I was talking about. I think as far as I've looked at Flyte it covers certain bits perhaps even a bit more like approachable manner. It doesn't necessarily cover everything as far as I understand. So not perhaps the deployment side, but I guess if you are interested in the platform, please come try it out. If you are even more interested, come make a contribution, join the project. We are pretty early on with the open source one but I think there's a good stuff going on there.

Jukka Remes [00:53:14]: And for example the software DevOps related part that's agnostic so even though we have this kubernetes in Docker kind of setup for the kubeflow and multiflow and so forth, then I think the DevOps part where you can set up the git based CI CD operations for your ML projects that's applicable even more widely. So there's different kinds of things and of course if you happen to be in in Finland or interested in the or needing to use the AI factories and the supercomputing we are extending to Tant like said. So now in Finland and later on perhaps in Europe you can utilize this to launch jobs into the supercomputers. It was actually interesting to hear about one of the first like AI factory launches that was in Finland because the they claim that in this European HPC consortium, at least with the SLUMI supercomputer that we have in Finland as part of that the capacity now going forward in relation to these AI factories it should be free for all small and medium size like companies in the union. Which is quite a tall promise I think. But I think it's very like there's for the kind of the product development part there's capacity apparently available when you need to train the models. So perhaps in the future you can utilize that capacity through this platform.

Demetrios [00:55:07]: Free GPUs for everyone.

Jukka Remes [00:55:09]: That's what we like to hear and big clusters of them, so you can really do the large model trainings as well, so.