Shipping LLMs: Buckle Up & Enjoy the Ride

AI in Production

Slides: https://drive.google.com/file/d/1mkhJ_7jzFq63ouwiToh3g3lKs-20ie4-/view?usp=drive_link

Adam Becker [00:00:05]: And our next speaker is Rex. Rex Harris. Okay, you're on the stage. How are you, man?

Rex Harris [00:00:12]: Can you hear me?

Adam Becker [00:00:13]: We can hear you excellently nice. Rex, you spent time, you did product at Pandora, at EA, at many other places, and most recently you're working nutrition, right? Precision nutrition using AI NML. We're excited to hear what you have for us today. I'm going to share your screen. It's right here. And I'll be back in just a bit. Take it away.

Rex Harris [00:00:42]: Yeah. Thank you, Adam. Glad to be here. Thanks, everybody. Okay, so we're going to talk today about shipping llms into production. Big surprise given the topic of this conference. But specifically, I want to talk about finding your why in terms of what value llms can bring to your product. Talk about what you're going to learn in that process if you haven't done this yet, and to remind everybody to have some fun.

Rex Harris [00:01:10]: So kind of go through and talk. This is going to be a little different than I think some of the other presentations you've seen today in last week. Part tech talk, part let's say motivational speech. And you'll see a little theme throughout this presentation on the buckle up and enjoy yourself with the racing theme. But there's a lot of complicated stuff, as you all know, in terms of what is necessary to ship into production. But I want to kind of remind everybody that there's also some real excitement to be had. So, as you know, I'm Rex, currently head of product AiML at Wisecode, doing precision nutrition through AI. Worked at a bunch of different places.

Rex Harris [00:01:59]: Also, as Adam mentioned, I've been doing product for a long time and using various forms of AI ML along the way. So kind of start off with just a little bit of what I would say is the sort of stages of driving proficiency with llms. I think everybody on this conference has had some experience with prompt engineering. You can kind of think of that as like you're driving along the streets, getting your bearings and starting to learn how to drive, building a proof of concept, much like taking those driving skills and going onto the highway, there's a bit more to manage and a bit more risk in terms of what you need to start understanding and putting into your application. And then production is really the racetrack. It's where there's a lot of things you really need to dial in before you get to that point. There's a lot of risk, but there's also a lot of reward. And so we hear a lot about this throughout, again, the conference and in general probably everybody has experienced this to some degree, whether in their own company or companies they know, which is there are these kind of LLM wrenches looking for nuts, kind of like the technology in search of a problem kind of adage.

Rex Harris [00:03:24]: And so llms are very powerful tools, we all know that, but they don't apply to everything. So what are those kind of nuts to look for? And these are some of the kind of key ones that I think everybody knows to some degree or has had some experience with. They're AI assistants. There's this kind of knowledge distillation or summarization and then content creation. And so a few examples here. I'm kind of talking my book on the first one, which is something I built at my last company where we built a kind of sleep coach for a smart mattress that we had built in the middle. Slack just announced their AI offerings, where you're able to kind of distill everything that's happening in your slack instance threads, channels, that kind of thing, and sort of distill down the bit of knowledge you need. And then gamma.

Rex Harris [00:04:15]: I'm throwing them some love here because I actually made this presentation using gamma, but using llms generative AI in general to create content. And so I love this quote from the CPO of Figma saying Figjam's user problems existed independently of AI. AI just happens to be an effective way to solve, you know, this is the find your why don't just apply llms? Because it's the latest technology find the reason for which you're going to put this into production. And once you do, then you are off to the races and it's a really exciting time to be in this space and as a PM specifically. But for others who are starting to build on this, you get these benefits and these kind of bits of fun that I wanted to make sure people were reminded of or introduced to, which is you get to now be a UX designer in so many ways. What you're building conversationally is something that you are designing for your users and you can iterate through that in a way that for many people, myself included, we didn't maybe get that opportunity to be so close to the design of a feature, which is really fun. Shout out to gradio here in the screenshot. But it's really easy with llms now and the tooling around it to rapidly test, learn from that iterate sort of typical product development lifecycles, but you can make really rapid changes and push that out to people internally or externally, get feedback and just constantly make changes, which is a wonderful thing.

Rex Harris [00:06:06]: Now, this one kind of funny for those of you who are like me. I put my son down for sleep for the night, and then I'm back hacking into the night sometimes when most of my teammates are no longer online. Well, with alums, you always have somebody to talk to. You're never working alone. I say that somewhat in Jess, but you do kind of feel that way sometimes, that you're kind of in this mode where you're getting sort of responses to the work that you're doing, which can be really fun. Of course, though part of the issue with that, as we all know, there's some non deterministic output that you need to trial and error through. This, of course, being a little throwback to the movie Ricky Bobby, but it can be very frustrating in some ways, but it can also be somewhat exciting in others because there's this kind of like variable reward you're getting from the output. So you got to kind of learn to embrace it.

Rex Harris [00:07:12]: This is one of the things that, especially for the kind of conversational AI applications llm that you might not think about is you can really get intimate with the customer needs. So you can see those logs and start to really see what your users are looking for. And not just in that feature, but in all features, because now they have a sort of chat experience that's really engaging and also is there when they're having moments of joy, but also moments of pain. And so this is context AI, great service where you can actually do kind of sentiment and insight analysis on your logs from your llms. And then, of course, as we all have experienced to some degree, this is a magical technology in so many ways. Shout out to Sora. I'm sure everybody talking about this and thinking about it in the last couple of weeks here, or at least the last week, but it really is something that as you're building and you're shipping, you get to really watch that magic happen and have kind of a front row seat. But of course, I couldn't do this presentation without at least highlighting some of the challenges that others have gone much more in depth around.

Rex Harris [00:08:28]: For shipping llms. There are new challenges you got to worry about. Prompt ejections, hallucinations, inference costs, privacy, latency evaluations. There's kind of a number of different new challenges to tackle with utilizing this technology. So definitely something to go in sort of with your eyes open and knowing that you're going to have to be challenged by this. But you do have a couple nitrous boosters if you will. To keep in mind, one is that models continue to get better, faster and cheaper. Shout out to the mistral folks, making sort of the open source community really vibrant on the front of super fast and much cheaper and very performant models.

Rex Harris [00:09:18]: And then don't discount the fact that, at least for now, because this technology is still so new, there is this kind of like hype train that your company and your team get to kind of ride. And I don't think that that's something that should be forgotten about. Really embrace that and use that to your advantage and to your team's advantage. So in summary, find your why. Make sure that you're using llms to provide real customer value for problems that exist already. Embrace learning new stuff. You're going to have opportunities but challenges alike, and then really have fun. Enjoy this to kind of take the moment and relish it and be grateful for kind of being here and learning again from this.

Rex Harris [00:10:10]: And I'll leave everybody with a quote from a pretty famous designer, Tom Ford. When you are having fun and creating something you love, really, that's what I want to convey to everybody here is, yes, there are a bunch of challenges, but make sure you're having fun. It will really come through in the product itself. On that this is me as Vin Diesel. You can find me on Twitter x at t reximus. If you have any questions, drop them in the chat. I'll stick around for a little while. I hope this was helpful.

Adam Becker [00:10:43]: Awesome, Rex, thank you very much. Helpful and it is inspirational and motivating, as you said it might be. It's very interesting. Not a lot of people sort of address the level of just like, the fact that you're thinking about the hype and riding that hype and fully just both enjoying it and actually leveraging it, this is not the kind of advice that you hear a lot. Why isn't it? And how did you come around to seeing it a little bit differently, do you think?

Rex Harris [00:11:15]: Yeah, good question. I think it's because when you hear the term hype or hype train or anything like that, it comes with a negative connotation. In a way. There's so much around generative AI and llms that could be misconstrued as not applicable or not something that people want to kind of get on the bandwagon for. But in reality, when you're building with it and you're starting to see the results and your team is starting to get excited about the fact that now in any application, you have this, again, kind of magical experience or feature that you can build for users if you have the right know. I think we got to really take a moment to appreciate. Yeah. And really embrace know, because it can generate just like anything in a company or a team momentum.

Rex Harris [00:12:22]: And, and it's important to try to embrace that.

Adam Becker [00:12:26]: That's profound, Rex, thank you very much. I, I think there is, I mean, it's so easy to sneer at everybody getting on the hype train, but I do think that just that quote, too, if you're actually enjoying it, it's going to show in the piano.