MLOps Community
+00:00 GMT
Sign in or Join the community to continue

Learning From Our Past Mistakes, or How to Containerize the AI Pipeline

Posted Aug 08, 2024 | Views 172
# AI Pipeline
# GPTScript
# Dagger
Share
speaker
avatar
Solomon Hykes
Co-founder & CEO @ Dagger.io

Solomon Hykes is the Co-founder and CEO of Dagger.io, the first programmable CI/CD engine. Before that, he was the co-founder of Docker, where he served for 10 years as CEO then CTO, and a founding member of the CNCF Technical Oversight Committee. Solomon grew up in France and now lives in San Francisco.

+ Read More
SUMMARY

Former Docker Founder Solomon Hykes' talk brings together DevOps and AI, showcasing the new Dagger open source project that replaces CI YAML with clean code. Technical and code-first, this talk will showcase how the combination of new primitives and AI capabilities can dramatically change the way we write and test software. After briefly explaining the Dagger project the talk will transition to a demo showing a glimpse of the future combining CI as Code with AI to enable dynamic pipeline generation. He will show how an AI model can automatically assemble the perfect pipeline using Dagger functions based on a set of declarative instructions leveraging GPTScript.

+ Read More
TRANSCRIPT

Solomon Hykes [00:00:10]: Hello, my name's Solomon. I'm the co founder of Dagger, and before that I was the co founder of Docker. Anyone heard of Docker before? Yes. Okay. They told me to ask because in AI conferences, you never know. That's what they told me. So I heard Docker usage is down around these parts. So let's see.

Solomon Hykes [00:00:29]: So I want to talk about a little bit of history, but not too much, and how much we can learn from the past to try and make the future less painful. So eleven years ago, we launched Docker. That's me with more hair. And the problem we were trying to solve at the time was I actually found some old slides for this. This is an original slide from the Docker presentation. You can see my sense of style. So the problem we were trying to solve was the problem of shipping software from a to b. It was very painful, but now it's all solved, right? Problem solved? Of course not.

Solomon Hykes [00:01:08]: That's the topic. This is also an original slide. I love this because you can see the version numbers, and this is a snapshot from the typical stack in 2013. But the gist of it is you've got a lot of infrastructure and a lot of software that needs to run on that infrastructure, and you've got a big mess. Different components need to run in different ways in different places, and that's just really hard. And so 2013, really painful to just deploy stuff. Just really painful. Does anyone remember developing and deploying applications before containers? Before Docker? Okay.

Solomon Hykes [00:01:46]: Okay, cool. Okay. So, of course, we proposed this idea of a standard container, a way to package stuff inside the container so you can abstract away what's inside from what's outside. Right. So you get separation of concerns. And we're copying here the idea of the shipping industry that figured that out before us. You know, you can ship all sorts of stuff inside the box, and once the box is sealed, you don't have to worry about what's inside, and you can move it across all this cool infrastructure. So you get this separation of concerns.

Solomon Hykes [00:02:16]: And we just applied that to software. Right. Any application running on any server, pretty much give or take. And that helped a lot. So fast forward to two years ago. We got the band back together, and we started a new company called Dagger. And this time we're working on improving CI CD pipelines. And so really it's sort of a phase two for us.

Solomon Hykes [00:02:42]: Instead of worrying about the box and what's inside the box and where it goes, now we're worrying about what comes just before the box. Where does it come from? How does it produce? What does that factory look like? That's cranking out containers. And this is where maybe if you're starting in the industry, you'll ask, well, what's wrong with CI CD pipelines? And then turn to your left or right. To a more experienced person, they're looking like this right now. Well, where do I start? Because CI CD pipelines are a mess. The first problem is what we call push and pray. By the way, this is my first time trying the more, more more thing with chat GPD. So let's see if it works.

Solomon Hykes [00:03:21]: Push and pray. That means your pipeline is out there. You can't run it locally, so if you want to change it in any way, you gotta just wait a few minutes for the change to go into effect. Anybody have experienced push and pray here? Okay. All right, well, she's still with me then. As you grow, you have different teams. Each team has their own tool chain. The tools are different, the languages are different, and it's all shell scripts to glue it all together.

Solomon Hykes [00:03:44]: Right? Because at some point it's got to integrate. By the way, the I for CI is for integration. Don't forget that. Then you have the pretty, the classic. It's working in dev. Why is CI failing? Well, I don't know. Figure it out. So things kind of snowball from there.

Solomon Hykes [00:04:00]: Then your team grows. You're now a successful company. CI CD pipeline problems fixed? Actually the opposite. It's getting worse. Now you're talking about CI migrations, rewriting custom configuration files, and you just kind of give up halfway. And just now you have two cis, and then you just keep going. And so now you're this super successful mega corporation selling AI all over the place. And now you've got these big meetings with platform teams that argue overdose.

Solomon Hykes [00:04:28]: Three different build tools, four different test systems. You got four cis now because you bought another company and the integration is still not done. Just getting worse and worse. I asked for a few fires in there. I think it looks pretty nice. So that's CI CD. That's the problem we're trying to solve and the way we're solving it. Oh, I forgot to start my timer.

Solomon Hykes [00:04:47]: Ha ha. The way we're solving it is by connecting. A few insights. The first is, well, this looks very familiar. This CI CD problem that everyone has today looks a lot like the old problem that we used to have with applications before containers existed. Which leads to insight number two. Hey, what if we thought of these really complicated CI CD pipelines as applications? They're complicated and we got to improve them and test them and collaborate on them. It kind of makes sense.

Solomon Hykes [00:05:19]: So maybe that means we should run those in containers also. What a great idea. Some people are trying to do that, by the way. I'm not going to say it's a best practice. I'm going to say it's a practice. It's in progress. Okay, insight number three. Why? Why have we not actually done this successfully? And the reason we think is that pipelines or applications, sure, but they're a specialized application, right? They're kind of a different kind of application, which means they need a specialized container engine because the good old container engine that we know hasn't changed that much in its general shape since ten years ago.

Solomon Hykes [00:05:58]: And it doesn't know about pipelines. So it can't help with your pipeline as much as it can help with a generic application. So what we're proposing is to specialize the container engine for pipelines. Three specializations. One, you gotta make orchestration programmable. You need to make the artifacts programmable, and then you need cross language linking. So just real quick, let me talk about those three. The first one is programmable orchestration.

Solomon Hykes [00:06:23]: When you run a web app in the container, it's just running in the container, doing its thing. It's happy. It doesn't actually need to know about the container engine running it. When you're running a pipeline in a container, actually what you're doing is you're running each step of the pipeline in its own container. Otherwise it's not going to work. But then that step running in a container usually needs to know about the container engine because it needs to run more containers. So you've got a lot of containers. Running containers.

Solomon Hykes [00:06:53]: Running containers. Who here has ever run a container with a docker socket mounted inside the container? Okay, so you know what we're talking about. It's really bad, but, you know, it shouldn't be because when you're running a process in Unix, can that process run more processes? Yes, that's a normal thing to do. So it should be possible in containers. Number two, the artifacts, you got steps. Great. What do they do? They run, sometimes they fail. Then you look at the logs.

Solomon Hykes [00:07:22]: That's not actually a pipeline. That's half of a pipeline because the other half is the artifacts flowing through the steps. Right. Does the container engine know about the artifacts? Nope. Actually, does the higher level CI, that's kind of a band aid on top of the container engine being too dumb know about the artifacts? Even that doesn't know about the artifacts so, you know, what are we doing? We think your container engine should just know about the inputs and outputs of the steps that it's running. And there's a lot of engineering you need to do that. You need core types so you can reason about things like files and directories, etcetera. Then you need a way, you know, you need a modern type system so you can kind of build from there and say, this is my artifact.

Solomon Hykes [00:08:04]: It's got this file, it's got this metadata, whatever, it doesn't matter if it's weights, a container image, a config file, a test report, it's all artifacts, right? And of course you need to be able to do that in multiple languages. So you need native bindings for, you know, the team over there doing that in python, another team doing it in Java, whatever. And all of that needs to be cached. Why not, if you don't cache it? It was pretty bad before, but now that everyone's moving these multi ten gigabyte files around for AI applications, then if you don't have caching, then forget it. And lastly, cross language linking. This is where someone usually says, but I do everything in Python, I don't need any of this. It's fine. It's not fine.

Solomon Hykes [00:08:46]: It will not be fine. Because believe me when I say I had people ten years ago saying exactly the same thing. Oh, my web app is all Python, I don't need this. That's the part that I find funny, that we're just kind of back where we started. But yeah, your favorite language is not going to win. Your favorite build tool is not going to win. This requires a cross language ecosystem, which means you need a way for everything I talked about before to somehow work across languages. This step written in Python has to connect with this step written in Ruby, typescript, whatever, and that's it.

Solomon Hykes [00:09:24]: The fourth insight to kind of connect this to this beautiful conference and audience is that this architecture that's emerging of AI applications makes everything I just said even more relevant, because we're actually now blurring the lines between the application and the pipeline. A web application has pipelines. An AI application kind of is a pipeline, or it's a collection of pipelines. So everything I just said, I think is just even more needed. And that's it. This is where I show a demo, but I don't think I have time left for a demo. But we do have a booth, and I can do a demo on the other side of this curtain if anyone's interested. The demo was actually my favorite part but yeah, all right, fine, fine, fine.

Solomon Hykes [00:10:12]: I did not plan for that. I'll try. Just tell me when I have to go. And I'm used to getting kicked off stage. Okay, wait, this is too small. So. Sorry, sorry, sorry. I'll just do this.

Solomon Hykes [00:10:30]: Is this fine? Size is okay. Just a little bit more. Okay, go. SDK. I apologize. Not the most popular language here maybe, but I'm too old to learn new languages. Old languages. Okay, so I'm going to initialize this and I'm going to do, I'm the DevOps guy.

Solomon Hykes [00:10:52]: I want to put together integration tests for this rag pipeline thing I heard about in the next team over. And so the AI team was kind enough to publish a module, a dagger module, with withdeze all the steps I need. So I'm going to go to this daggerverse search engine that shows me all this is the cross language ecosystem we were talking about every possible step you can think of. This one's called gptools. It's all open source. You can see the repo here. I have a generated API cross language here and it says, ok, install like this, like so. So I'm going to dagger install the thing.

Solomon Hykes [00:11:30]: And now I've got a bunch of containerized steps I can use. And now I can do something like let me take a URL here. And Marcos, my teammate, said, okay, if you call this step and you give it your open API key, your OpenAI key, the URL of a YouTube video and a question, it's going to run whisper and then it's going to get a transcript and then going to send that transcript and do some rag and that's going to use OpenAI to answer questions. Cool. I ran this before. So it's cached, right? So now it's telling me about the video. So this is me. Oh, it's really loud.

Solomon Hykes [00:12:15]: This is me basically running little pipelines on the fly, right? If I want to look at the source code of this module here, I can go look at it, but I don't have to know what language it's written in. And the last part, and then I promise I'm done. If I want to start gluing this into my own pipeline, then what I'm going to do is let me open this in my editor. I'm going to switch from calling pipelines from the command line to call them from code. So here I got my little basic file. I got these two functions. We call them functions. They're basically steps.

Solomon Hykes [00:12:57]: Let me remove this. So let's say I want to just query. So I want to query this URL and return the answer. So here, this is the magic part. I have automatic, I have generated bindings to the modules. I just installed cross language, right? So I get this little pipeline assembly thing, gptools, dot, what's the API again? Oh, yt chat. YouTube chat, right, yt chat. Oh, let me do this something simpler, just because I don't want to take too much time.

Solomon Hykes [00:13:44]: Let me get the audio file, the audio file from that URL. Let me get the transcript from that audio file, and then let me just create a directory, an empty directory, let me add some files to it, and at the root of that file I'm going to add the audio and the transcript, something simple like that. So like a little build function. And so I'm going to, because it's a build, it's going to return a directory, okay? So I'm just playing with the inputs and outputs of my pipeline like it's a regular function. These are the core types we talked about. And then I'm just going to return this. And I just developed a pipeline. So now if I go back to the shell, whoops, I list available functions.

Solomon Hykes [00:14:48]: Got my little query here, query a YouTube video. So here you're seeing a little bit of the cross language linking I was talking about. I'm writing a function here in go, and now here I can call it connected into a pipeline from the command line. So let me just pass that URL from before. Notice that here it's running whisper against the, it's downloading that video, running whisper against it, generating the transcript, but it's instant because it's all cached from before. And here I'm just going to say output it here, give me the data here. Anyway, this is what I got for now. I feel bad taking too much time, but here I have my audio file, my transcript, and I'm just composing these little steps.

Solomon Hykes [00:15:45]: I could just keep building up little functions. So we basically took the development of CI CD pipelines, doesn't matter if it's build test deployment, data processing, inference, fine tuning, whatever, and we're breaking it up into these little functions. You write them in the language you want, dagger then loads them all, runs them in containers, and then takes care of passing the artifacts around with cross language linking and caching across the board. So that's the idea. And then you run nci the end. Thank you, Solomon. Thank you. Yep.

+ Read More
Sign in or Join the community

Create an account

Change email
e.g. https://www.linkedin.com/in/xxx or https://xx.linkedin.com/in/xxx
I agree to MLOps Community’s Code of Conduct and Privacy Policy.

Watch More

Common Mistakes in the AI Development Process
Posted May 26, 2021 | Views 703
# Presentation
Productionizing AI: How to Think From the End
Posted Mar 04, 2024 | Views 405
# Productionizing AI
# LLMs
# Bainbridge Capital
How To Move From Barely Doing BI to Doing AI - Building A Solid Data Foundation
Posted Dec 16, 2020 | Views 1.3K
# ternarydata.com