MLOps Community
+00:00 GMT
Sign in or Join the community to continue

4 Years of the MLOps Community

Posted Invalid date | Views 135
# Volunteers
# Quality beats quantity
# MLOps Community
Demetrios Brinkmann
Demetrios Brinkmann
Demetrios Brinkmann
Chief Happiness Engineer @ MLOps Community

At the moment Demetrios is immersing himself in Machine Learning by interviewing experts from around the world in the weekly meetups. Demetrios is constantly learning and engaging in new activities to get uncomfortable and learn from his mistakes. He tries to bring creativity into every aspect of his life, whether that be analyzing the best paths forward, overcoming obstacles, or building lego houses with his daughter.

+ Read More

At the moment Demetrios is immersing himself in Machine Learning by interviewing experts from around the world in the weekly meetups. Demetrios is constantly learning and engaging in new activities to get uncomfortable and learn from his mistakes. He tries to bring creativity into every aspect of his life, whether that be analyzing the best paths forward, overcoming obstacles, or building lego houses with his daughter.

+ Read More
Mihail Eric
Mihail Eric
Mihail Eric
Co-founder @ Storia AI

Mihail is a co-CEO of Storia AI, an early-stage startup building an AI-powered creative assistant for video production. He has over a decade of experience researching and engineering AI systems at scale. Previously he built the first deep-learning dialogue systems at the Stanford NLP group. He was also a founding member of Amazon Alexa’s first special projects team where he built the organization’s earliest large language models. Mihail is a serial entrepreneur who previously founded Confetti AI, a machine-learning education company that he led until its acquisition in 2022.

+ Read More

Mihail is a co-CEO of Storia AI, an early-stage startup building an AI-powered creative assistant for video production. He has over a decade of experience researching and engineering AI systems at scale. Previously he built the first deep-learning dialogue systems at the Stanford NLP group. He was also a founding member of Amazon Alexa’s first special projects team where he built the organization’s earliest large language models. Mihail is a serial entrepreneur who previously founded Confetti AI, a machine-learning education company that he led until its acquisition in 2022.

+ Read More

In this lively podcast episode, Mihail Eric hosts Demetrios Brinkmann, the founder of the MLOps Community, discussing its origin, structure, and challenges.

Demetrios shares amusing tales of job hunting on LinkedIn and building the community despite lacking technical expertise, emphasizing the value of sharing and humor.

They delve into the practicalities of hosting events, transitioning from self-funded to sponsorship-based, and tease upcoming activities with renowned speakers.

Mihail and Demetrios explore job dynamics, the importance of sustained relationships, and diverse engagement methods like newsletters and volunteering.

Demetrios reflects on his journey to Germany post-company closure, envisioning a global hub for AI learning, embodying the community's mission.

+ Read More

Demetrios Brinkmann 00:00:01: Hello, I'm Demetrios. I am the founder of the Mlops community, and I take my coffee in a chemex with a little bit of chaga mushroom powder and coconut oil. We're going to start. The last time that I talked to you, I asked you what you thought I was listening to. You want to do it again? Because, like, wait, I totally remember this. Is this, like, what you're actively listening.

Mihail Eric 00:00:28: To or what you were, like, listening to in the moment?

Demetrios Brinkmann 00:00:30: I forget now what I was listening to right before we hit record. Okay. Yes.

Mihail Eric 00:00:35: And it was, like, gregorian. I think I said gregorian chance or something like that.

Demetrios Brinkmann 00:00:40: So I'm not listening to Gregorian Enchant today, but I am listening to equally banger.

Mihail Eric 00:00:47: Equally banger, definitely. I don't think it's classical. I'm guessing, like, some sort of, like, a light flamenco guitar kind of thing.

Demetrios Brinkmann 00:00:56: Am into that. No, I went down the ASMr hole. So just the sounds of people, like.

Mihail Eric 00:01:04: Rubbing combs against, like, a mic.

Demetrios Brinkmann 00:01:07: Dude, I have been the most unproductive fucking person today because every five or 10 seconds, I just stop and zone out and listen to that. So if you want to get shit done, do not put on dance yet. Do not. That is not the high productivity, like, binaural beats.

Mihail Eric 00:01:28: Not at all. That's so weird to me. I find that so bizarre.

Demetrios Brinkmann 00:01:32: But okay. I did not guess that.

Mihail Eric 00:01:33: I would not have guessed that. Why don't we get started?

Demetrios Brinkmann 00:01:36: Yeah. Let's do this. Let's do it, dude.

Mihail Eric 00:01:44: I am here. I'm Mihail. I'm here with my very good friend that you all know. Demetrius Brinkman, the dawn of the mlops community.

Demetrios Brinkmann 00:01:54: Some of you love him. Some of you hate him.

Mihail Eric 00:01:56: Some of you wish he would stop playing guitar, stop making music in general. I think we all agree. I think we could all come together on that point.

Demetrios Brinkmann 00:02:04: But he's not stopping. The new song was a bit slaps. Some people said that song slaps, and I think that means that it's good. From what I've heard on TikTok and shit, I actually did agree with this.

Mihail Eric 00:02:16: And Shreya, even during the LLM production conference, was also like, damn, this is, like, kind of hot.

Demetrios Brinkmann 00:02:22: Yep, that's it. That's it. We got approval from Shreya. You know, game on. Good. You're good. It was a solid song. It was inspired by one of her papers.

Mihail Eric 00:02:31: Inspired. And she was like, oh, I love.

Demetrios Brinkmann 00:02:33: The moment where she said, hey, I've never had someone, like, write a song about my research. Whoa, whoa, whoa. Easy. That's right. Pump the brakes. Write it for you. About you. No, more like inspired.

Demetrios Brinkmann 00:02:49: We need to be very deliberate with our words here. Can't be going around spreading rumors.

Mihail Eric 00:02:54: So this episode of the podcast is about the history of the mlops community, and it has been a tremendously useful educational community and organization. Just honestly, and I'm not even exaggerating here, I think one of the top, if not the top collection of ML practitioners, data science people, people building ML systems effectively and practically, really out there, I can't think of know. Whenever anyone talks to me and asks me, like, hey, where are people that are big into ML? Where do they exist? Where do they go to? I say, it's the mlops community, hands down. So, Demetrius, it's his brainchild. It's been the thing he's been working on for four years. And to commemorate that four year anniversary, we're here to learn a bit about what it was like from early days. Where are we today? And then in the process, learn a little bit more about the mysterious man, the man, the mystery, the legend.

Demetrios Brinkmann 00:03:48: You make it so formal. I didn't realize I should have come with my suit and tie and shit. You clearly think it's formal. I do have notes. Let's show everybody the notes in case you ask those hard questions. I have my talking points. They're scribbles.

Mihail Eric 00:04:06: They're scribbles. I don't think those are notes. Just because you showed up a notebook doesn't mean anything. It's a lie. He didn't prepare at all. But I'm happy to be here. I'm really happy to be here. I'm really excited you asked me to do this, hands down.

Mihail Eric 00:04:19: Very happy to be here. So what I do want to start, though, here is we're going to talk about a lot of things. I want to start give us a sense for what is the community today? Like, where is the mlops community? What have we become? Give us some numbers here so we can textualize all this.

Demetrios Brinkmann 00:04:38: All right, so, first of all, I couldn't think of anybody else to do this but you. And actually, I called a few other people, but they weren't available.

Mihail Eric 00:04:50: Damn it.

Demetrios Brinkmann 00:04:52: You were the first choice after. The first choice after the. Wasn't exactly the first choice. So we'll just get that out there so we can be fully transparent in this retro, what do they call it? Blameless attribution or some shit like that. I don't know. But anyway, let's talk about the community for a minute and you are one of the, I think, 21,000 people in our slack right now. And so we've got, let me break down the shape of the community. We've probably got, like, two weekly newsletter editions that go out.

Demetrios Brinkmann 00:05:30: We've got the slack workspace where there's, like, 21,000 people. We've got local meetups that are happening in 37 cities from, like, Sydney and Melbourne, Australia, all the way to California to New York to Berlin. Amsterdam just happened last night, actually. And then we also have the virtual meetups, the virtual events in general, because we also have virtual conferences, which you so kindly did a little bit of a stand up act at, at the last one and booed off, dude. So I may have been booing you, but I was a little harsh because when we had the feedback form and we asked people to fill out what their favorite part was, we said, what was your favorite part of the conference? I was expecting people to talk about my songs and Demetrius the great host, and you know how arrogant I am or can be. And they were like, oh, it was Michael's comedy routine. And I was like, I cannot show him this. I cannot let him know about this.

Demetrios Brinkmann 00:06:32: So we may be inviting you back against.

Mihail Eric 00:06:36: My rates are going to go up. My rates are going up after that.

Demetrios Brinkmann 00:06:39: Just so you know. Yeah, exactly. I shouldn't have told you that. It would have been zero to zero to zero. Plus, between cheap and free, that's what we're looking at. But, yeah, we've got the podcast, obviously, that people are listening to that goes out twice a week. And sometimes we do, like, roundtables or panel sessions. And then the biggest thing and the blog, like the community source blog, so anyone can write for the blog, and then it gets vetted, and if it gets the approval, then it goes on to our blog.

Demetrios Brinkmann 00:07:13: And so that's another community piece. And then probably the scariest thing for me right now is that we literally just closed the location for our first in person conference, which is going to happen on June 25.

Mihail Eric 00:07:29: Wow.

Demetrios Brinkmann 00:07:29: Yeah. I'm kind of, like, shaking in my pants because it is scary, man. It's the first in person conference that we're doing, and so I'm going to focus on what I can control, which is get incredible speakers and then make sure that there is going to be an amazing experience for everyone that comes. But I'm not going to say that I'm not nervous about that. That's about it. As far as all the different stuff that we're doing and what the community looks like right now, I'm trying to think if there's other things that we have going on. I mean, we also have people that are full time employees in the community, which is huge. That's wild.

Demetrios Brinkmann 00:08:13: I think there's about seven direct employees of the community, and then there's like a ton of volunteers. So everyone that is leading a local meetup, I would consider them a volunteer like metric who's absolutely killing it, leading the blog posts and all of that charge volunteer, anyone that writes for the blog, I would say volunteers. So volunteers are harder to quantify, but there's a ton of them, too. Wow.

Mihail Eric 00:08:42: I mean, that's a lot. And look, what I want to just even highlight to the people that already know this. Maybe they're tuning in. But what I especially love about the community is not just that it's obviously the numbers are big, right? Tens of thousands of people. But it's the quality that I think has always stood out to me because there's so many discord channels, slack groups, I mean, you name it, of just data, people that are in data or whatever. And the quality there is just never particularly high. Whereas what you get in the Mlops community, that really stands out. And that seems to have always been a part of the thread of the community and a centerpiece was just the quality of the conversation and the quality of the people that are in it.

Mihail Eric 00:09:23: So that I think has been amazing.

Demetrios Brinkmann 00:09:27: We can get more into that, like things that reasons, I think that is. But the spoiler of that is, I'm not sure I have any idea why or how there is such high quality in the community. But I'm just as amazed as you are every day when we have these questions which are high quality questions, and then answers that are coming from all different directions. You have data scientists, you have data engineers, ML engineers, SREs, all answering it in different ways or different shapes and forms. But on some of the learnings, if you want, I can talk about some of the learnings that I've had over the years. Please do. Like what I think could have contributed to that. But I don't know the definite answer for why that is, besides banning people that are too spammy.

Mihail Eric 00:10:18: That makes a big difference for sure. Look, I'm happy to talk about learnings. Tell me about some learnings. One thing I just want to call out as well, and we should get into this at some point, is, look, you've made this whole global movement effectively in mlops, and that is incredibly impressive. And what I want to get to at some point in the conversation is, was that always part of the genesis? Did you have that in mind, or did this vision of a global mlops movement, did that evolve over time, or were you born to do this?

Demetrios Brinkmann 00:10:48: Dude, the thing is, so many times I've tried to sit down and be like, what's the vision? What's the mission? All that, like, the mission statement, the vision, we need to have a clear idea that people can rally behind. And at the end of the day, I haven't been able to put anything super clear down that isn't this, which is just like, I just want to have a place where I can meet cool people and learn about cool stuff related to putting ML into production or AI into production. So ML Ops, LLM ops, data engineering, data ops, whatever ops, flavor of the month it is, I want to have a spot that it's like, you have to put it out there, and then people can gravitate to it, but unless you are the one that is putting it out there, people don't know to gravitate to it. And so as far as, was that the vision all along the way that we started the community? And I've told this a million times, but I'll tell it again, and I'll give you kind of the short story in long. Yeah, it's basically like, I was in Spain right before the pandemic hit. I was working for this company called Dot Science, and Dot Science was doing data lineage. And data lineage in 2019 was a little early for the time that it was, because at that time, I don't know if you remember, in 2019, not a lot of people were actually seeing huge amounts of value in their ML efforts. And this is all, like, what I would call traditional ML or classical ML these days.

Demetrios Brinkmann 00:12:30: And what was happening is it was really hard to put machine learning models into production. And so these companies that were seeing really good returns or roi with machine learning, they were the gigantic companies, and they were, in general, building everything themselves in house. Like, now that I've talked to a lot of them, it's like the Facebooks. Yeah. And the pinterest even. They were doing this stuff since, like, 2014, 2016, whatever, because they had to. And they were already what we would consider mature, and maybe they would want data lineage, but they didn't want to go buy any data lineage tools off the shelf. And then the rest of the market was just like, I got Jupyter notebooks.

Demetrios Brinkmann 00:13:15: What's going on there? Dot science was floundering, and then COVID hit and boom. We could not get anybody on the phone for a sales call, because everybody was thinking, my company may go out of business in the next week. I don't want to think about bringing on a tool that is going to involve some kind of a sales process. I just want to focus on, am I going to have my job in a week or in two weeks? And so that was March 2020. And that's kind of the reason why I was like, oh, man, it's March 2024. It's been four years of the community, really, because right when that happened, we said, well, why don't we try and start a community or something? And I have to give a lot of credit to the CEO of Dot science, Luke Marsden. He came from the docker sphere, and so he was very heavily into the Docker movement and created Docker back in the day, and then was very early in the Kubernetes movement. And so he had open source in his blood, even though Dot science wasn't open source, funny enough, but he added in his blood.

Demetrios Brinkmann 00:14:35: And I talked to him one day, and it was like, yeah, we should start a community and we can let anybody in. And I was like, are you sure about that? We want to let our competitors into this community, and we're going to let them give talks on our virtual meetups. Because by that time, everything was virtual. If you remember, it was like this lockdown. And so he was just like, yeah, do it. The rising tide lifts all boats, and it doesn't matter if they're competitors. Doesn't matter. Like, just get the knowledge out there.

Demetrios Brinkmann 00:15:06: And so from that point on, Luke did the first virtual meetup, and then I was in charge of finding the speakers for the next virtual meetups. And after four virtual meetups, thought science went out of business. And that was the end of April 2020. And that was scary for me because I was like, oh, shit, I'm about to move to Germany. I had already paid off the movers and done the down payment and everything. So moving from Spain to Germany, I had my little one year old daughter. And Germany's cost of living is a lot higher than Spain. And I was expecting to have a paycheck.

Demetrios Brinkmann 00:15:47: And so then I was applying to every fucking job I could find on LinkedIn. And most of them were like, Zoom, because Zoom was hiring in 2020 and not a lot of other people were. And so then that started, and I remember talking to a friend who's not in software or he's not a developer at all. And I was like, man, there's this thing that is kind of gaining traction, and I feel like it might be something cool, but I'm not sure if I should focus on it 100% or if I should focus on trying to get a new job. And he was asking me the numbers, and I remember specifically telling him, and this was the middle of May, and we were planning to move to Germany in middle of June. And I was like, I think if by June, so by the end of May, we have 600 people in our slack channel, that will be, like, a huge signal for me. And then, yeah, we got that. And then it was like, oh, well, now maybe we can get, like, a thousand people in slack.

Demetrios Brinkmann 00:17:01: Oh, and now maybe we can do more stuff as opposed to just having slack. And so then it just spitballed from there. And so I was just focused on creating, being someone who can help foster initiatives. So people started coming with different ideas, and that was it. It was like, all right, cool, let's do it. Let's make it happen. Let's try and create more movement around this, because there's something here. Obviously, people are interested in it, and there's a tension that's happening.

Demetrios Brinkmann 00:17:41: So then it just kind of went from there, and there was a ton of things that, like, a ton of milestones that happened along the way. But I can tell you all about some of the learnings, too. I know you wanted to get to that, too.

Mihail Eric 00:17:54: Yeah, let's just go where it goes. You're already touching on it. I think the origin story coupled with the learnings, I suspect one of the milestones was the 600 mark, which is an interesting one to me. Did you have a very specific reasoning for that number, or was it just, hey, 600 sounds like a good, nice number.

Demetrios Brinkmann 00:18:14: It was just because it felt like that's way out of reach and especially when coming from sales. Like, if two people a week would talk to me, I was stoked. And now all of a sudden, it was like, there's a hundred people in this slack. That's insane. And then getting to talk to people, and it's so funny, too, man. I'm not ashamed to say this, but I find it hilarious. And I normally don't broadcast this one, but for those people that are listening, if you are ever looking for some entertainment, go back and listen to the second virtual meetup or the third virtual meetup because it was podcast format, but I had no idea what the fuck people were saying. No idea.

Demetrios Brinkmann 00:19:08: Like, they would start talking about monitoring and continuous training or data drift, and all of those concepts were foreign to me, and I would be looking at them and I had questions that I had sourced from my technical friends that I knew, and that was it. I had sourced those questions. There was ten questions. I would ask the question, I would get an answer, and then I would just kind of nod like, yeah, great answer. And then just ask the next question. There was no back and forth because I had no idea. Everything went over my head.

Mihail Eric 00:19:44: I think one of the biggest accomplishments of the MlOps community, in my perspective, is the fact that you've been able.

Demetrios Brinkmann 00:19:50: To lead people on for this long that you are, you've faked it till you've made it to one of the.

Mihail Eric 00:19:57: Leading communities in the world. And that is something very impressive to mean people, you know, take it as you.

Demetrios Brinkmann 00:20:05: But.

Mihail Eric 00:20:06: But Demetrius, who doesn't come from a traditional machine learning background, has been corralling some of the best minds in ML into his community, which is incredible.

Demetrios Brinkmann 00:20:15: There was one. So we did feedback know, and I would ask everyone in slack, like, once or twice a year to give a feedback form or to fill out feedback form on what we can be doing better in the community. And I remember one person was like, oh, maybe on the podcast. And this was probably like a year and a half into the community. They said it would be nice to have somebody who doesn't know, like a beginner in mlops to ask beginner questions. I was like, what the fuck do you think I am, man? You can't get more beginner than this shit. And then I started being like, maybe I kind of know. And that's when I started saying, like, I know enough buzzwords to be dangerous.

Demetrios Brinkmann 00:21:03: I don't really know what's going on.

Mihail Eric 00:21:05: That's awesome.

Demetrios Brinkmann 00:21:06: I still do the smile and nod every once in a while.

Mihail Eric 00:21:08: That's awesome. That is amazing. I've been witnessed to this where people ask you technical questions in, like, a forum or a LinkedIn thread, and they're.

Demetrios Brinkmann 00:21:17: Like, what do you think about is.

Mihail Eric 00:21:18: The best way to do this, this and this.

Demetrios Brinkmann 00:21:19: You're like, I ask in the community. Yeah, ask in the public chat. Generally these days, I do have more opinions, and I do feel like I can kind of give solid advice depending on who's asking and what the question is. But I still try to direct people to the community forum because that's one of the things that I'm a very big fan of, is that if one person has the question, I think others have the question, and it's worth putting it into a public channel so that everyone else can learn from it. And that's why I'm kind of not excited when somebody asks a question and then somebody generally, nine times out of ten, there's the person who's asking the question and then you have two or three people that are trying to clarify what's actually going on so they can give a good answer. And when you have these people clarifying and then you have the person that asked the question say something like, oh, I'll just DM you. It's like, no, don't go into DM, keep it out in public because we need this. Unless it's super sensitive or you can't share the clarification questions.

Demetrios Brinkmann 00:22:37: But for the most part keep it public so that everybody can learn from it. And that's the same with being in a meetup. It's great to ask questions after somebody gives a talk, right? And so, same stuff, but amazing. I digress.

Mihail Eric 00:22:52: No, Demetrius, tell us more about the origin story coupled with the learnings along the way. I actually have not heard of the full origin story, so this was new to me. But do tell us more about what's happened since, how that all unfolded from that point on. And then please sprinkle in the learnings that you think were the biggest ones over the course of the community.

Demetrios Brinkmann 00:23:11: Dude, I think one thing that stands out to me, which is a little bit sentimental, but I really enjoy meeting people and forming connections with people. But I also, now after it's only been four years, but four years is also a long time, kind of depending on how you're looking at it in dog years, it's a shit ton, right? And so people come and go and they float into the community orbit. And some people are really into this because it's very aligned with their career goals or with what they're doing at their job. And then inevitably people will float away. And so instead of trying to hold on to that, what I've started to do is just how can I keep contact with people but not feel like I need to hold on to people? Because there is very much this nature of like it's tied to the career. The community is very tied to what you're doing at your job. And if your challenges at your job change or you change jobs and all of a sudden you don't work on the ML platform, you work on the data platform, and the company isn't mature enough to be doing ML or AI or any of that sort, then you're probably not going to be hanging out in the ML ops community and you're not going to want to volunteer as a local champion or whatever so I'm grateful for everyone that has come into the orbit, and I think that's one thing that I've seen over the years, is that people come, they go, life happens. And it's so cool to have made friends and have this global network of friends, and it's also really cool to watch them grow, too.

Demetrios Brinkmann 00:25:09: You're a perfect example of that, right? Like, you're working on very ML specific stuff, and then it's like, oh, now you're going to go start your own company. And so I get to watch you grow and hopefully start a billion dollar company that you can make me an advisor to or whatever when you want. No problem, I'll be here and ready for it. But the other thing is that if I'm doing my job right, so I've kind of like, come to see that the community is not a community because of me. It's almost like in spite of me, so much shit, there shouldn't be the community. But the thing that is nice is that it's not like it's an influencer community type thing. It's not like, oh, people see my YouTube videos and then they come into the community. They come into the community because the value that the community has.

Demetrios Brinkmann 00:26:13: And then there's like, oh, yeah, and this random guy that kind of does shit in the community, he also has a podcast. All right, let's check that out. Or, oh, we came to a meetup and then we found there was other things that you could do, right? So it's not like it is you coming to the community because of me. And because of that, what I'm trying to do is make sure that in the community, I can give as many people a platform as possible. Because there's nothing more that I derive pleasure from and see joy on a daily basis than when I see people in the community that are winning in whatever way, shape or form that is. So if I can help them do that, that's like my true north, I guess you could say. So whether that is giving someone a chance to write on the blog, and then because of that, who knows? Maybe they get a job or somebody reaches out to them and they form a new connection. Or bringing somebody on the podcast, having like a shout out to an excellent blog that I read on the Internet in the newsletter, all of those ways.

Demetrios Brinkmann 00:27:23: And then just like making sure when there is quality work, and especially when it's somebody from the community that gets recognized. And so that's something that I think has been a huge learning for me. It's a little more personal. But I think that in community building is very valuable. And so it's just like, how can I give as many people a platform? How can I make the MLO ops community platform so big that we can fit more people on it? Right. That's the other, I guess, another way of looking at it. And that's probably why we have so many different manifestations of what the community is, whether it's the newsletter, podcast, or virtual events and in person events, all that stuff I was talking about. And then there's another one that I think is fascinating, which I've heard.

Demetrios Brinkmann 00:28:19: This one I did not coin. I got it from, I think the woman's name is Rosie and she's big in the community space, and she talks about monthly recurring relationships. And so how many people am I creating relationships with and then how often am I talking to them? How am I having recurring relationships? Not just like I get on a call with somebody and then I never see them or talk to them again. And maybe they're never actually in the community orbit. We could say. So those are a few. I mean, I've got a bunch, but maybe you want to chime in on some of these.

Mihail Eric 00:29:01: Yeah, well, I mean, there was so much gold in there that I could unpack a lot of things. But maybe I'll start by saying on your point around, in a sense that the community was bigger than you. It wasn't like your brand. That was the cornerstone of the community. I'll agree with this. You don't have enough Twitter followers for that to be true. There's no way that could have been 300.

Demetrios Brinkmann 00:29:25: I have 300 Twitter followers. Just to let you know, I'm trying to be an AI influencer, but it is not happening.

Mihail Eric 00:29:32: You're not even in the running, so.

Demetrios Brinkmann 00:29:34: That is not a concern.

Mihail Eric 00:29:36: But on a more serious note, I like this idea of building maybe what I would call, like, fault tolerance around the community. Right.

Demetrios Brinkmann 00:29:45: You set up in such a way.

Mihail Eric 00:29:46: That it's almost distributed, decentralized, that in a sense, it doesn't depend on any one person to continue existing.

Demetrios 00:29:54: All right, let's take a quick break from the episode and talk to you for a second about our sponsors of this show, Wandb, and what they've done with their newest LLM course. You can elevate your machine learning skills with weights and biases, free course on training and fine tuning, large language models, LLMs as they're called out in the wild. You can learn from industry experts like Jonathan Frank from good old Mosaic ML, friend of the pod, Weiwe Yang and Mark Seraphin. And while you're doing it, learning from them, you can deep dive into the LLM fundamentals, advanced training strategies like Laura and prompt tuning, all while getting your hands dirty and having some hands on applications. This engaging course offers 37 lessons and over 4 hours of videos and a certificate upon completion, enrolled now to master LLMs and advanced techniques. Check out the link in the description to start your journey and let's get back into the pod.

Mihail Eric 00:31:01: And I'll speak to this from a very personal note. I was running the SF meetup for a while and then got busy with startup stuff and kind of disappeared a little bit, became a little bit Mia for a while. But then I came back to it and it had grown. The same meetup in one part of the world had grown and there were people that had jumped in because they felt it was valuable enough to make it happen. And I think that speaks to exactly what you're describing, right? This idea of it's bigger than the sum of its parts in some sense. That's what's so amazing about this community. It's bigger than me, bigger than you. And people see so much value in this core piece of the relationship building you've described that people do jump in when they need to and they want to keep it going.

Demetrios Brinkmann 00:31:46: Exactly. And that's 100% what is for me, probably why when we go back to that, like what's the mission? What's the vision of the community? And it's just like, I'm just here to have a great time and meet cool people. That's what I'm trying to do. But everyone has different reasons for joining the community. And one thing that we're doing now, which is awesome, is we're trying to map out what are your reasons for joining the community so that we can give you what you're looking for. Because some people, they join the community because they're like, oh, well, I just started this open source project and I might get funding for it or I want to talk to people and see if it's actually valuable. And so there's one community user path or community member path. Some people are like, I have a very strong challenge that I'm trying to overcome at work and I need to debug this and I need to know how others have done this in the past.

Demetrios Brinkmann 00:32:45: So it's like stack overflow on steroids in a way, or very niche down. And then other people are like, I'm just looking for a new job. And so I'm trying to find if there's anybody that I can network with. And you have that person, you have others. Like, investors will join the community because they're like, I'm trying to find the new, hottest thing. And marketers will join the community because they're like, well, we want to show off our product. And so those people, I just say, talk directly to me and we can figure out how to make that happen. That's an easy one.

Demetrios Brinkmann 00:33:20: No, but we're trying to figure out all of these different journeys. Right. It's different paths and different pieces of the community are going to be valuable for these different Personas and what they're trying to get out of the community. So now that's something that I feel like is happening and is really a huge step forward because then you can know, all right, sweet. You're a data engineer, and you're looking to get into mlops. Here are the top four things, or three things that we would recommend you do, and we can have that set up as a trigger. When someone clicks on the form to join the community, they say, like, yeah, I identify as a data engineer, and I am looking to learn more about mlops. And then, boom, we can have email sequences saying, watch these meetups or join this event, blah, blah, blah.

Demetrios Brinkmann 00:34:26: And so that's something I'm super excited about.

Mihail Eric 00:34:32: That is such a deliberate and methodical way to think about community building. I can see you were a salesperson in a previous life.

Demetrios Brinkmann 00:34:39: I mean, just this idea of different demographics and segment user groups. I love it.

Mihail Eric 00:34:48: I'm a big fan. I think this is super cool. I want to dive into one thing more deeply because cool, relationship building, love, all that. There is this understanding that a lot of the reason why people come into the community is something career related, something related to their job, whether that's progress, growth, getting a new one. Can you tell us a little bit about what that looks like for people and maybe how successful you think you have been over the course of the community's lifetime in helping people level up in their careers? And maybe what were some of the milestones around that?

Demetrios Brinkmann 00:35:24: All right, Charlie Rose, coming with the hard hitting, huh? I see you. And you should sponsor the Mlops community.

Mihail Eric 00:35:33: By the way, whoever is listening, I think if you're looking to get talented.

Demetrios Brinkmann 00:35:38: People, strong talent, that's it. Strong talent. Well, actually, I used to search it out and really try and make sure that if there were jobs that were, how would I say this, being gotten from the community? Like, if people got a job not.

Mihail Eric 00:35:58: Grammatically correct, but, okay, yeah, let's go with that.

Demetrios Brinkmann 00:36:01: I'm not known for my strengths in grammar or public math, but I do it anyway. And so the thing is that I really wanted to highlight the fact that I've seen a lot of people get jobs because of their involvement in the community, and there are plenty of ways to do that. I still think that if you are active in any of these different channels or if you are volunteering in the community, that gets, like I said, I'm trying to create this gigantic platform and make it big enough that everyone can stand on it and we can lift everyone up. And one of the coolest things that I can think of, like, a tangible outcome, is that somebody starts volunteering for the community. And because of them being on this platform or doing things, maybe it's being a local champion. Now they're meeting people, and they're almost at, like, the center of the ML scene or AI scene in their city, and it makes them so much more likely to get a job. And I've heard about so many, or just like I've heard about people answering questions in slack, and then somebody will say, oh, cool, thanks for answering that question of mine. They bring a conversation to DM like, you want to have a coffee sometime or whatever, virtual coffee.

Demetrios Brinkmann 00:37:31: And then one thing leads to another and you got a new teammate. And so those experiences 100% happen. And it's really hard for me to quantify how much they happen because it is almost like, it's not like it's behind closed doors. It's just not like people are coming up to me and telling me every time it happens. But when I do find out about it, and I will say right now that if anyone has gotten a job because of activities in the mlops community, please tell me, because that makes my day. Like, literally, that is one of the other. That is absolutely incredible for me because it's just like this thing that started because I lost my job, right, is now helping others advance in their career. And so whether or not it's what they learn in the community is what is helping them advance in their career, or it's just straight up like, oh, I saw in jobs channel this job, and then I applied for that job, or I dm'd the person who posted it, and now I'm working for that company.

Demetrios Brinkmann 00:38:39: So there's a lot of that. I think that is so cool to see. And I get really stoked about that. I don't know if you can hear the emotion in my voice, but that gases me up man, so much.

Mihail Eric 00:38:51: Yeah, I mean, that's palpable. We can all see that. And then what's cool about it is you're not almost seeking it out, but it's happening naturally. And that seems like the best of both worlds, right, where you're giving people a platform to engage and actually have a community, which is something we all need as people. But then in the process, what's falling out of it is something that is very pragmatic, like very practical. It's a very tricky balance to get that thing in the middle because you go too hard in the career stuff. It becomes a very transactional sort of thing, and people are not very. It doesn't feel as authentic or organic.

Mihail Eric 00:39:29: You go too much on the other side, and we're just shooting the shit in slack and saying, like, hey, look at this meme. Look at that meme. And there's nothing valuable. But to hit that middle ground, I think, is incredibly hard and sounds like something that has been a big outcome and success story for the community.

Demetrios Brinkmann 00:39:46: Yeah, I love hearing about that, dude. I love it so much. That is one of my favorite things. And the other thing that I wanted to mention, there is one other kind of, like, takeaway that I have learned in the community building over the years that I think is worth noting. Before we jump to the next thing is that I've thought a lot about people that come to the community but don't necessarily actively contribute. Right. Some people might call them like, oh, lurkers in slack. And I am 100% encouraging people to do that if that's how they feel like they want to be part of the community.

Demetrios Brinkmann 00:40:40: And that mentality has changed from when the community started till now. And that is because, again, you can't quantify things that happen outside of your knowledge. But I have heard people talk about how they will take a screenshot of a question and threads that happen in slack and then share that on their company slack. And then that helps move the needle for some kind of a policy or some kind of design decision at their company. And so those things are like, okay, maybe you're not for some reason, you don't feel like you want to jump in on a thread or ask a question or whatever, but if you're still getting value out of it, that is huge. And so that's really cool. Or just coming in to whatever other way, like you just read the newsletter, and if that's how you want to engage, that's also great. Or you listen to the podcast, or you go to a local meetup or the in person conference, whatever it may be.

Demetrios Brinkmann 00:41:51: There's so many different ways to engage, and I don't necessarily think now that you have to be active in your engagement. Of course, it's nicer if you are, because nobody wants to join a community that's just a bunch of people that aren't active, really? Then there's not really a community. It's kind of like it's a dead zone almost, right? So some people have to be active, but at the end of the day, I still think that there's a lot of value in the people that aren't being super active. And the other piece to this is that kind of like dovetailing on that is the numbers. I know I said it was before we hit record. Like I was telling you how many numbers have come into the ecosystem. Right. But it's a lot, I would say.

Demetrios Brinkmann 00:42:44: And I think numbers mean nothing, man, because you can have a community of 30 people that are super engaged, and it's very valuable, and it can be the best thing for you at that point in your life. Or you can have a community of 30,000 people and it can be absolutely worthless. So numbers, at the end of the day, it's cool to say because especially if I'm trying to get like a big speaker for the conference, I usually will put out the big numbers, but then I think it all comes down to quality over quantity. Agreed?

Mihail Eric 00:43:27: Yeah, I've seen this. I'm sure you've seen this many people, I'm sure they're tuning in, have seen this. Those communities that have that claim, you see the numbers in the slack at the top right of how many people are in the channel. It's tens of thousands, if not more. And it's just promotion after promotion. No one's actually listening to anyone. You don't even see people upvoting because no one even tunes into this. So absolutely agree with this point.

Mihail Eric 00:43:50: Quality over quantity. Micro communities can be some of the most powerful ones out there. I want to touch a little bit on one of the things that I think has been one of the strongest pieces of the community, and maybe it's growth. Maybe you can speak to this. Just how much of this collection of volunteers you've been able to pull in to organize events, to contribute different pieces of their knowledge and their expertise, how did you foster that kind of a community that allowed for that to happen and encourage people to do that so that people want to do that? Because it seems like so much does depend on the volunteers. How did that happen? And can you tell us a little bit more about that aspect?

Demetrios Brinkmann 00:44:34: So, honestly, that was a huge milestone, and that happened fairly early on in the community journey, which was. And I remember it vividly, too. This is another one of those things where it's, like, hard to forget because it was so impactful. And probably in June. Remember, I was like, oh, if we hit 600 people in June, then that'll be a lot. And it was, like, in June or July, and it was in slack, and we could search it right now, I could probably tell you, but David Aponte was like, hey, man, who's, like, running this thing? You ever need any help? Because it seems like this is pretty cool. And I got some free time. And that had me being like, oh, shit, I don't have to do all this on my own.

Demetrios Brinkmann 00:45:28: There's other people that can help do this. That would be then. So then I was like, yeah, let's have a call. Let's do it. And from there, then more people started raising their hands. And I remember the second person was Vishnu, who is a longtime guest host on here, too. And I just remember that, okay, yeah, so what do you guys want to help with? And we just started jamming every week and having jam sessions. And in the beginning, David was who I leaned on really hard for the technical capabilities.

Demetrios Brinkmann 00:46:09: I was learning a lot from him because we would just have calls all the time, and he would be teaching me stuff, and I could just ask him anything. And so I could ask the stupid questions that I was afraid to ask the guests on the podcast. And then it got to a point where it was like, yo, dude, should we record these calls? And he was like, yeah, sure, let's do it. And so then he gave me a walkthrough of how cube flow works. I still had no idea what the hell was going on. He was going through the demo. I had no idea what he was saying. And I'm like, wow, this is awesome.

Demetrios Brinkmann 00:46:44: Yeah, great. All right. And just like, going on mute and being like, fuck, I don't know what he's saying. I don't understand any of this. That was really cool. And then it just kept growing. And what I realized now is that, again, going back to that, like, how can I create a platform? The best thing that I can do is make it as easy as possible for people to contribute. If it's super cumbersome, every time somebody reaches out and they're like, hey, I want to volunteer.

Demetrios Brinkmann 00:47:19: And then I'm like, all right, well, let's get on a call, and let's figure out what your strengths are and what you do, what you don't do. What now we do is we just have the docs, and I put those together, and I should probably update them probably pretty soon, because there's so much more that we can do. But we have the docs, and we have areas where we need help for people to volunteer. And we also have. If anyone says, I want to be a local champion, then we have so many resources that we can give people. And it's very clear now because we've done it 37 times. We know what people need, and we know where the biggest pain points are for somebody that is starting a local chapter. Right.

Demetrios Brinkmann 00:48:02: It's like help finding venues, help finding speakers, help finding sponsors, or just getting money for the food and beer and whatever drinks and all that. And help marketing so that people will come to the event. All of that is stuff that we tried to immortalize and put into documentation so that if you are someone who wants to do that, you can go and just read through all of this and gain all that knowledge, and then it's on you to act on it. Right. So, I think that's been huge, is how can we make it as frictionless as possible and also really help people when they want to contribute. Figure out, how can we add the most value in your contributions.

Mihail Eric 00:48:57: That's amazing. 37. I did not realize there were that many chapters. That's incredible. That's crazy.

Demetrios Brinkmann 00:49:02: And you want to know? 37. It almost bankrupted us, to be honest. I could tell you that story, too. Here's a learning that I had. Don't give 37 cities a fucking unlimited budget to do whatever the hell they want. But definitely not Sydney.

Mihail Eric 00:49:18: Don't give it to anyone in Australia.

Demetrios Brinkmann 00:49:19: Yeah, exactly. Okay. Sydney and Melbourne. Just actually Australia in general. Shit. We love that. No, the thing is that we got sponsorship for the in person, and it's funny because, like, sold sponsorship before we had in person stuff happening, but I was like, well, if we get sponsors, then that is going to be the catalyst to make me work towards creating in person events. And so we sold it to Neptune, and Jacob was awesome about super.

Demetrios Brinkmann 00:50:02: We were very aligned on, like. Yeah, we don't want to just go there and talk about our tool. We just want to make sure that we can send some Neptune branded beers to the event and also maybe some shirts and have one of our representatives go to the event and meet people. Because, let's be honest, the networking is half the fun. And so then we had that, and one of the things that we were trying to wrestle with was like, all right, cool. We got sponsorship, and we weren't super clear. As the first chapters, like, the first cities started cropping up, we weren't clear on what is the value that we can provide. And one of the things that we thought would be super valuable for people, and it turns out it's not even that valuable, but I can get to that in a minute, is, like, reimbursing these cities for food and drinks and then even videographers, too.

Demetrios Brinkmann 00:51:05: And so we said in the beginning, I think we said, you can expense the community up to $1,200. And so that was cool. When there was, like, five cities, and they were doing a meet up once a quarter, and then all of a sudden, there was, like, twelve cities, and they were doing a meet up once a month, and it was like, holy shit, we're paying, like, six or seven grand a month just for these meetups and the Neptune contract. That money's long gone, right? And it's hard to. Now we had to step back where we were telling everyone, we're going to help you. You don't have to worry about paying for food or drinks. We've got you covered. You can expense that to the community.

Demetrios Brinkmann 00:51:52: And then all of a sudden, we're like, actually, we got to talk about that, because the monthly bill and monthly spend is getting really high. And so that was a nice one. Luckily for us, dude, the community, like, this community, the members are so incredible. I don't know how. Again, I'm not sure how potentially it's because we just are, like, I'm not afraid to be weird. And so people know what they're getting when they're coming into this or they don't, because they don't come for me, like we said before. But as soon as they come in, they get the vibe pretty quick, I would imagine. And the people in the community were very supportive, and it was like, let's figure out how to transition from the community paying for food and drinks to, like, we help you find sponsors to pay for the food and drinks.

Demetrios Brinkmann 00:52:52: And so then it's not like they have to pay out of their pocket. It's just that they have to go and find some sponsors.

Mihail Eric 00:53:00: That's incredible. That's a cool story. I didn't even know that there was this communal effort to bring to keep Demetrius'bills afloat.

Demetrios Brinkmann 00:53:11: No, I'm not a stupid shit that I've done with sponsors money, man. Oh, my God.

Mihail Eric 00:53:19: There's so much good stuff that's fallen out of it. So that's really cool. I had no idea. I mean, funny story, but also the fact that the community rallied and came together, that's awesome. One of the last kind of big topics I want to touch on, and then I'll end with some more interesting questions.

Demetrios Brinkmann 00:53:35: Not to say these, I haven't been.

Mihail Eric 00:53:36: Interesting, but a little more out there.

Demetrios Brinkmann 00:53:39: Tell us a little bit about what's.

Mihail Eric 00:53:40: The future like, what does the future of the community look like? Where are we going, dude?

Demetrios Brinkmann 00:53:48: I mean, biggest thing on my mind is June 25, right?

Mihail Eric 00:53:52: Okay.

Demetrios Brinkmann 00:53:53: You hopefully are going to. I'll be there. Do your sequel to the stand up comedy routine. We are definitely not paying you to be there, just so you know. Actually, you have to pay to get in. That's one thing. I'm not giving you any discount or anything. All right.

Demetrios Brinkmann 00:54:12: But if you want, what I'm thinking is it would be really cool to have a stage or a section of the venue with a bunch of musical instruments. So anybody that plays music can go and jam whenever they want.

Mihail Eric 00:54:26: That's fun.

Demetrios Brinkmann 00:54:27: Or anybody that is a comedian can go and tell a few jokes. They can do a five minute set if they want. Tight five. That would be cool. That would be very cool. But yeah, it's exciting. We already have a few speakers that have confirmed, and I am so stoked. So this is another thing.

Demetrios Brinkmann 00:54:52: It warms my heart, like speakers, when I've reached out to them, they've been really excited about it. And so I can tell you, like Linus Lee, who I'm personally a super fan of, he works at notion. He's given a few talks for the virtual conferences. He's going to be there. Chip Hewen, who is famous in her own right. She's going to be giving a like seeing that. Also, I'm looking at the speaker lineup that we have, and this is buried at the end of this podcast. So I am imagining that not a lot of people are listening anyway.

Demetrios Brinkmann 00:55:33: So I'm going to say it because we have a whole marketing plan on how we're going to get this out. And if you listen to the podcast, you'll probably hear me say this a few times. Right. But we've got Fazan, who's been on the pod. He's AI product lead at LinkedIn. We've got Shreya Rajpal, who is the CEO of guardrails. She's in. We've also got Janit Doshi, who is a staff data scientist at Intuit.

Demetrios Brinkmann 00:56:02: Man, there's a lot of incredible people looking at Cameron Wolf, Sharon Zhu, some hard hitters. I mean, we're doing it in. I wanted to say in conjuncture, but that's not the word. In collaboration, it starts with a C. I knew it. Right again, grammar, not my strong suit, but in collaboration with Colena. So Colena does AI testing. Mohammed's been on the podcast if you want.

Demetrios Brinkmann 00:56:33: That was probably like two, three years ago, but that's what I'm focusing on. And again, I'm a little nervous because it's the first time, but it is cool. And honestly, I realized that.

Mihail Eric 00:56:49: I think.

Demetrios Brinkmann 00:56:49: We'Ve weathered the storm a little bit on the whole LLM ops craze and whether or not we need to change the name of the community. Look, even the balloons, they jumped up because that was a whole thing last year. And people saying, like, oh, mlops is so dead, and nobody does mlops anymore. And LLM ops is all anyone cares about. Or now it's the AI engineer. And so I think we've kind of weathered the storm, and we know what we're doing and where the strengths are and also what people in the community want to see. And so, speaking about the future, though, more, man, I think I'll be winning, or, like, I'll consider it a success if I can keep the light heartedness, I can keep meeting amazing people, I can keep connecting and learning and doing stupid shit and not taking anything too seriously. Right.

Demetrios Brinkmann 00:58:01: That is, for me, the key here, because if it ever becomes like, that feeling of just, like, then I think I've done something wrong and I've created a bit of what I would consider, like, a jail for myself. Right? So that's it. And as far as the future, man, I would love to hear from those that are still the three people that are still listening how we can create more value for you. What can we do in the community that would be valuable for you? And if that means more Michael comedy routines, we can make that happen.

Mihail Eric 00:58:50: I will start charging at that point, so don't let that be the first answer.

Demetrios Brinkmann 00:58:56: Oh, that's classic.

Mihail Eric 00:59:01: Amazing. Look, dude, I think I'm just speaking as one member of the community, and obviously it's been so awesome to be in it, excited about where it's going toward what's going to happen next. Super excited about this in person event. I mean, if you haven't checked out the materials and the talks from the virtual one a few weeks ago, you definitely should because it was pretty high quality. It was a whole thing. Thousands of people tuned in. So big deal there, if that's any indication. The in person one I'm sure will be amazing.

Demetrios Brinkmann 00:59:38: There we go. That's it.

Mihail Eric 00:59:40: I want to end up because the vibe of the community is more light hearted. One thing I did in prep for this is, and I want to ask.

Demetrios Brinkmann 00:59:49: Some crowdsourced questions from the community.

Mihail Eric 00:59:51: So these ones you'll have to field. But this just shows how lighthearted I think the community is. So one of the questions burn Burhan mentioned and is really curious about was becoming a caveman a deliberate choice for greater appeals of the community or an outcome of starting the community? Just a fair question, I think.

Demetrios Brinkmann 01:00:14: Just trick question. There is no wrong answer there. No. This is actually very tame. This is me, much more tame. You should have seen me ten years ago when I didn't have two daughters or any responsibilities in the world.

Mihail Eric 01:00:40: Fair enough. Fair enough. That's always been a part of his look and that will probably continue to be. So, another question from Jeremy McMinnis. What's a stronger hallucinogen, DMT or chat GPT these days?

Demetrios Brinkmann 01:00:56: I'm going to probably go with chat GPT. Got to. Yeah.

Mihail Eric 01:01:03: That's the right answer. That's the right answer.

Demetrios Brinkmann 01:01:04: Actually, you know what, dude? Oh, my God. You're going to love this. So again, a little bit of a tangent, but remember how I was like, we're going to keep doing stupid shit and you asked what the future holds? You ready for this? I don't think you're ready for this. And I almost don't want to say it to give away the fun, but again, I'm expecting there's only two people listening at this point. So here is what I plan to do potentially before the in person conference. If not, we'll do it over the summer and we'll do some summer of AI thing. We're going to take the shittiest base model you can get, so probably like stable lm and we're going to fine tune it so it only hallucinates. And we're going to call it mushroom GPT.

Demetrios Brinkmann 01:01:59: Mushroom GPT. That is what we got. So there you go, Jeremy. We'll see if you can handle that. That's amazing.

Mihail Eric 01:02:09: That is amazing.

Demetrios Brinkmann 01:02:10: I love. That's. That's cool. That's quirky. I thought you were going to say.

Mihail Eric 01:02:13: We'Re going to do like mlops branded mushrooms. But that's to hand out of the conference. I think this one is more legal.

Demetrios Brinkmann 01:02:22: Beware of the Kool Aid at the conference. Beware of the truth. Truth.

Mihail Eric 01:02:26: That's amazing. I look forward to that. Deujas pleasure as always. This was very cool. So cool to see and hear about the story of the genesis of the community and all the fun stuff that you've done and learned. And I expect that it will just continue. It's all uphill from here. So thank you again.

Mihail Eric 01:02:45: Thank you.

Demetrios Brinkmann 01:02:46: I appreciate that. Have a blast with you. Obviously, that's why I keep inviting you back on here.

Mihail Eric 01:02:53: That's my third tier interviewer.

Demetrios Brinkmann 01:02:56: That's why you have so much fun with me. I'm like, yeah, he's probably fourth or fifth string. Take it. As long as I'm on the team, I'll take it. Even a bench warmer gets a championship.

Mihail Eric 01:03:07: Trophy at the end of it, right?

Demetrios Brinkmann 01:03:09: Exactly. Let's look on the bright side. That is it. We are part of the first biggest polish and spanish speaking in an english country community based around AIML in the mid 2020s in the world. All right, so the largest in the world. I guess we'll claim it. What was they also like?

Mihail Eric 01:03:45: The number four lithuanian podcast on Spotify.

Demetrios Brinkmann 01:03:51: We are the number four tech podcast lithuanian podcast. Lithuanian English speaking, English speaking. No dubs. Exactly. That's going to change, though. Soon enough. We're going to be getting all of the podcasts dubbed with AI. That's AI generated voices and everything.

Mihail Eric 01:04:13: That's cool. Very cool, dude.

Demetrios Brinkmann 01:04:15: This was awesome. I always have a good laugh with you. Thank you for doing it and absolutely, man, humoring me on this. Pleasure.

Mihail Eric 01:04:21: Pleasure as always, dude.

Demetrios Brinkmann 01:04:23: All right, see you, man.

+ Read More

Watch More

Posted Nov 22, 2023 | Views 629
# LLM Fine-tuning
# Large Language Models
# Weights and Biases
# Virtual Meetup
Posted Nov 01, 2022 | Views 779
# Ray
# On Device
# Model Serving at Scale
Posted Jul 21, 2023 | Views 424
# LLM in Production
# LLM Stack
# Virta