Machine Learning beyond GenAI - Quo Vadis?

AI in Production

Slides:

Demetrios [00:00:07]: And next up, we got Hannes. Where you at, Hannes?

Hannes Hapke [00:00:10]: I'm in Portland, Oregon. How are you, man?

Demetrios [00:00:12]: I'm good, dude. Did you laugh at all? You can tell the truth.

Hannes Hapke [00:00:19]: I chuckled.

Demetrios [00:00:20]: Hey, Mikhail will be happy to hear that. All right, dude, you got to talk for us, right? You got some lightning talks coming up, so I'm going to throw ten minutes on the clock and let you start rocking and rolling.

Hannes Hapke [00:00:34]: Cool. Let me share my screen and then we can get started.

Demetrios [00:00:37]: Let's do it. And while you're sharing your screen, I'm going to throw on here that in case anyone wants to come and listen to Hannes and I talk for a while. You came on the podcast, and so you can scan that QR code and join us on the podcast. Or if you like, shirts and stuff, we created some shirts that say my prompts are my special sauce. So go ahead and scan that QR code. Support the community while you're at it. All right, Hannes, it's all you, man. I'll see you in ten minutes.

Hannes Hapke [00:01:12]: Thanks for having me. Hi, everybody. My name is Hannes. I'm a machine learning engineer at digits, and we help accountants and business owners to streamline their books and basically automate all those mundane tasks around bookkeeping and accounting. There is a lot of machine learning involved, as you all can think of. And so last year, the entire community, we experienced this massive shift in machine learning. Everybody talks about large language models now, about LLM ops, all those different things like how do we consume open AI and things like this? And then there's always the question coming up of like, as a machine learning engineer, where should I focus? What is not commoditized tomorrow? Where can we build competencies? In this quick lightning talk, I want to give you four different ideas of what you potentially can do as a machine learning engineer to compete or not compete or build competencies alongside with model APIs. Number one, the key of machine learning, what we can do these days when all these basic models are commoditized, are basically finding the relationship between data points.

Hannes Hapke [00:02:31]: So any data we have is connected. You have unstructured text, you have structured tabular data, you have images, you have maybe voice data, and you want to figure out the relationships between those different data artifacts. And that is something we can do with machine learning, and that's also something which we can easily do with third party APIs like OpenAI or entropic. And so finding those relationships are really interesting, and they're also extremely domain specific and problem specific. So this is something where you can deep dive into. So here in the bottom image, I showed you like a little experiment we ran with graph neural networks and to find the relationships between different nodes in our data system or in our data, and we wanted to predict the edges. And then there's also another problem we talk about later is sort of like low throughput of llms. They work really great, but they're not really fast.

Hannes Hapke [00:03:28]: So if you have lots of data points you want to process really quickly, llms might not be the best one to go with. So how can you find those relationships in the data? Number one, you can try similarity based machine learning. We have written a number of blog posts on this, and also Dimitris and I had a conversation about this in the past. So go back to the MLops podcast where we talk about this. But this is something where efficient models, or you can generate efficient embeddings that will save you money, time. And what I say here lives, is because your embeddings might or will be more precise than generic model API embeddings. And as you can imagine, they're often domain specific. And so if you make your embeddings domain specific to find similar data points, you can shorten the dimensionality of those embeddings, and therefore their lookup later on will be much faster, less costly, and more precise.

Hannes Hapke [00:04:27]: So if you're interested in that, we wrote a blog post on our website developer digits.com, where we talk about similarity based machine learning, or go back to the Mlops podcast to listen to our show. The second option, what you can do is, and that's sort of a superset of similarity based machine learning. You can take a look into graph neural networks. This is basically, it's hard to come by labels. So in similarity based machine learning, you need labels which are sort of like similar data points, like basically buckets of how you define the similarity. Sometimes this is hard to combine, but your data is often a graph. So for example, we have different data points, we have transactions, they are connected to categories. The categories in your chart of accounts are connected to vendors or legal entities, the entities are connected to organizations and so on.

Hannes Hapke [00:05:26]: So as you can see there, you can build the graph out of the data pretty easily, and you can train neural networks to understand the graphs and the features of the individual nodes. And with those models, you can make really amazing things. Like you can, for example, predict similar edges, or you can predict similar nodes, or you can predict, you do link prediction between different nodes. And with those different tools, you can then find the relationships without specifically annotating or categorizing or bucketizing the data upfront. What we had to do with the simulator based machine learning, it is a little bit more of an upstart, but if you want to get started, highly recommend the UL here on this slide. It's a great introduction to graph neural networks. And then the other thing is, when you work with llms and model APIs very quickly, do you run into scenarios? If you have a lot of data, you want to process them? It gets either very costly or time consuming. So you can easily extract a few training samples using model APIs or llms, and then basically use some sort of a concept of teacher student forcing to build smaller models which execute faster, like the inferences faster.

Hannes Hapke [00:06:44]: And they're also cheaper to run and to host. So examples are named entity recognition here. This is something we do on a daily basis. We get banking transactions. We need to figure out what are the related entities in those banking transactions. And so we have dedicated models for those purposes and they have to be maintained and run. But that allows us to run multimillion transactions in a very short amount of time, because we don't have to run them through a 7 billion or 13 billion or even a 70 billion machine learning model. And then the last option.

Hannes Hapke [00:07:19]: What I would recommend here today is you can focus on fine tuning and hosting llms yourself. Most applications don't need gigantic models. They're often very domain specific, and that allows you to fine tune llms to your specific applications, whether it's directly fine tuning or running through things like lower and Qlore and things like this. The fine tuning methods have massively improved in the last twelve minutes, sorry, in the last twelve months. Probably also in the last twelve minutes as everything is changing so rapidly. And the quantitizing of those models got massively easier as well. And at the same time, hosting large language models is so much easier than it was like six or twelve months ago. Big shout out to the folks from Titan with them, you can basically host large language models internally in no time.

Hannes Hapke [00:08:14]: It takes you less than 30 seconds to run a large language model. Something would have taken you maybe a couple days to set it up in the past. So with that, those were four ideas of what you can do if you want to stay in touch. Connect with us on Twitter at digit or my ed, or find me on LinkedIn. I'm happy to chat.