Challenges and Opportunities in Building Data Science Solutions with LLMs

In this roundtable, we will share our experiences with LLMs across a number of real-world applications, including what it takes to build systems around LLMs in a rapidly changing landscape. We will discuss the challenges around productionsing LLM-based solutions, evaluation of the quality, as well as implications around risk & compliance.

Intro

Quantum Black is a, a center of excellence in AI and machine learning at, uh, McKinsey and Company. Um, we are focused on building, uh, machine learning models and leveraging them, uh, to really improve businesses and drive, uh, their performance. Um, so we have offices, uh, um, around the, the world with perhaps the major.

That's still in London, and this is exactly where, uh, Daniel and I are joining from. We will be joined by our colleague, um, based in, um, Brussels Paal. Um, and yeah, like today, uh, we wanted to share with you, uh, some of our learnings, uh, from the last few. In terms of challenges and opportunities, uh, that we faced when we're building data science solutions, uh, with lms.

So, um, like really excited to be here. Daniel, perhaps, uh, would you like to introduce yourself before we begin? Sure, of course. So, hi Daniel. Uh, data scientist at, in the London office, um, originally physics, PhD and. Really excited to work now on alarms and actually brings them to clients.

Wonderful. Just in time. Thanks for joining us. You're joining us from Brussels, right? I didn't lie, uh, Amsterdam. Yeah. Sorry, I was still in this zoom in the backstage area. Wonderful. No worries at all. Um, so yeah. Um, today we wanted to talk to, um, our lovely audience about the challenges and opportunities, um, that we faced when building data science solutions with LLMs.

And, um, we represent our strong community that is, of course, very excited about all the latest advances in, uh, large language models in general. Um, and of course, um, things are moving really fast and sometimes it's really hard to stay up to date with what's going on. And, um, things really, um, um, sometimes feel like they get out of hand.

Challenges and Opportunities in Building Data Science Solutions with LLMs

So we really wanted to focus today on. Um, things such as risk and compliance and all the important guard rails that need to be in place when we actually start, um, deploying these models and start bringing value to the end users. Um, so Pascal, uh, perhaps, uh, given your background and, uh, I would also, uh, love you to, um, say a few words about your, uh, yeah, your, your background before you joined QB and, uh, at qb.

Yeah, like we would want to hear a bit more about, um, how, uh, software engineering developments and innovations that happened, uh, in the last few months affected, um, the trends that we see in using lms, um, and in, in building data science solutions with LLMs. So, of course, you know, um, for a long time, LLMs, um, really had a very high entrance.

And, uh, we're really a matter of a few lucky ones. Uh, but now we see that, uh, the applications are, are really rising every single day and like it feels like they're growing exponentially. And of course, um, a lot of, um, this acceleration is really, uh, enabled by, uh, APIs that are now exposed. And, um, that would be really interesting to see your take on how.

Innovations are really, um, affecting this democratization of technology. Oh yeah, sure. Um, yeah, so maybe I'll, I'll just share a little bit of like how we approached it. So essentially, uh, a couple months back, I think the whole world got overrun by, uh, chat bt essentially, and it was in everybody's, uh, on everyone's mind.

Um, I think the moment when your parents-in-law start chatting with you through translations from chat and your fitness trainer, Um, just says, okay, today's class was based, um, you realize it's mainstream. And it kind of reminded me of this, um, a couple of these memes back in the, in the blockchain hype days, whereas like when your taxi driver is asking you what the latest is on blockchain, then you know, it's time to, to jump ship.

Um, and so I just realized, or we all realized, okay, we have to get involved here. Um, and um, then I. A little bit of my, my data engineering background, which like, I, I built two or three years of, of data platforms and, uh, have a lot of Kubernetes experience. Um, and we looked at what was out there on the, on the cloud providers.

So this is like Q4 last year. Um, and we very quickly hit the boundaries. So if you look at gcp, it wasn't possible to host these large models. They're just too large. So Vertex didn't support it. SageMaker, there wasn't really anything. Um, Azure also wasn't, it wasn't yet so clear how deeply integrated Microsoft and uh, and open AI are going to be.

So we realized, okay, there's definitely not, not anything out of the box. Um, and so let's supply what we know, which is let's use Kubernetes. Let's see how big the machines need to be. Um, And then let's try and containerize it and bring it into a world where we can apply our existing knowledge around machine, uh, ML ops and hosting, let's say normal machine learning models or smaller ones, um, and get them to work.

Uh, and so we looked at frameworks like, uh, Selden and um, McKinsey also has, uh, recently acquired iio. So we also looked at that. Um, but we. Again, hit this, this threshold of, well, it doesn't actually fit on a single instance, because some of these models were so large we couldn't fit them on a single Kubernetes node.

Um, and so while all of the tools were there, they weren't meant to handle. 300 gigabyte models because of course, uh, Daniel and I instantly wanted to load the largest blue model because why wouldn't you start with the largest one? Because then everything else, um, becomes easy afterwards. Uh, yeah. And so like we, and it actually ended up making everything work.

We used Ray in the end because Ray allowed us to, uh, cluster across multiple nodes and spread the, spread the model across different instances. And it also just felt. Mature because Ray comes from the same people that came up with Spark. And Spark just became so democratized or democratized big data so much it felt like a right move.

So we switched to Ray and got the blue model running there. Um, and then also looked at other models and got some of those running. And so now this is two and a half months ago and I feel like it's completely obsolete because somebody reported everything to c plus plus. And then colleagues send me pictures of them playing around with Bloom on their iPhone.

Um, and so like if I just take a step back now, I realize we probably don't need the complexity of Kubernetes anymore. Vertex just announced that they are gonna have l LM support. SageMaker didn't really make a big fuss of it, but they also support it now. Uh, Azure supports it now. Um, the models become much more tamable again because, uh, people are showing that seven to 9 billion models are also super useful.

And, um, this space is moving so quickly. That is actually really fun to see That code that you thought was a really good idea two months ago and soft infrastructure that you've built two months ago, uh, you can pretty much get rid of because you can stand on the shoulders of giants from Google and, uh, Amazon and not handle a Kubernetes cluster because ultimately that's just a lot of work for, for very low benefit.

So yeah, that was kind of our, our journey over the last three months. Um, you know, it was still very good to learn because we learned about all of. Infrastructure challenges that you have when you're trying to make these things work and you're trying to provide sufficiently large, uh, GPU work benches. Um, trying to bring these models in a way that you can scale them up and have auto scaling scale to zero is very difficult.

Um, but then I think taking a step back, again, realizing to build a product, it's not necessarily to pick the coolest technology, but just to get the job done and turns. Chances are the, uh, hyperscalers are gonna be able to, to help us a lot with these things. Thanks, Pascal, for your perspective. This is very helpful, Daniel.

Yeah. Given that, uh, you were also one of the, the newcomers do want to comment on, uh, your experience also, given that you're coming from a bit of a different background, um, uh, how was that for you? Um, after joining the last few months was, uh, super exciting. So I think one of the use cases that really stood out, um, on our side was the whole topic of, uh, document-based question answering.

So I've been starting to play with this, um, in second half of the last year. So just putting together on smart prototypes now a. In pretty much every industry. So, uh, from finance, sustainability, over healthcare, education, you name it, people do face the problems that they have to ingest, use amount of information, and usually they're looking for some very specific insights from, you know, a range of documents, data sources, you name it.

One of the tools that really stood out and were there was really exciting development happening in the last few months is Lang Chain. Um, by the way, congratulations if anyone is here on the call, you know, raising this 10 million seat round. And the parts that most people are getting really excited about is that it allows you to take these, but chat pity, which is already really, really good at answering questions from a wide range of general problems to really.

Tailor it to the specific problem domain. Also, another big advantage is that by providing resources, you would, you see hallucinations and on top of it you have the chance to, uh, validate some model outputs and cost check them. So that the topic that you also was excited about at the moment, uh, And Daniel, for you as a data scientist, um, did you feel that, um, all these new libraries that popped up and um, were like developed by the community really helped to democratize access to lms, um, and not only to the data practitioners, but also to the wider popul.

Um, I would say it definitely unlocks a lot of capabilities that, you know, most people wouldn't really have seen, uh, two years ago, I guess. Uh, people who were working already previously saw what's possible with GPTs. We got early access to the i p is, uh, we're aware of what can be done. But you're currently seeing it.

Yeah. It's mass market. Especially thanks to the hyper around chat g pt. Um, one minute. Um, do we maybe briefly go to the commands we are seeing here? So there's a question for Pascal. Uh, could you please share us a list of tools you used in the last few projects? And a second question about the infrastructure challenges.

Any specific ideas and tools helped you to fine tune. Beast, I guess such what mean biggest models? Um, yeah, I'll, I'll, so the first one, um, we tried to stay pretty cloud agnostic because, um, in general it felt like if you, if you can somehow get into the Kubernetes world, then it doesn't matter if you're on premise, uh, whether you have, um, a, a cluster in one of the hyperscalers and you can imagine.

Also a lot of the very large enterprises, uh, which are often McKinsey clients are, um, they tend to have still a fair bit amount of on-premise stuff. Um, and that's actually a super interesting world in which it's fun to, uh, to apply this technology because, um, when you have all of this knowledge that's kind of locked up in some on-premise systems, making that available, Of course it's super attractive.

But, um, what seems to be a bit of a pattern there is you bring the data to the model or do you bring the model to the data? And it, I think a lot of companies actually prefer bringing the model to their own data rather than sending all of their data to some other company. Uh, and so we went for Kubernetes and then, um, I think stayed that open, stayed with that open source stack.

So Kubernetes, Prometheus, Grafana, Loki for all of the, like, you know, non-functional. And then, um, let's say on the, on the l l m stack, it was really Kubernetes. Um, of course making sure the gp, uh, are available or the gps are available for the pods. And then, um, we had Ray, um, or Ray Cluster running on top of Kubernetes.

Um, because it then allows us to apply or to deploy a model across different modes. And we, uh, used ALPA in the end, which, um, was, I don't even know when Alpa was released, um, to get the very large models running. Mm. And then I think for the second part, like what the, um, what the challenges.

Okay. Um, for the, for the fine tooling of the beast models, and perhaps Daniel, you could also comment on that. Yeah. Daniel, do you, do you actually think what we used is still applicable? Because I feel like I just want to get into the new tools that were released over the last, like, deep speed I want to get involved in or I wanna try.

Um, yeah. Deep speed, uh, was a useful tool, so that made it really, really accessible. Also with the, uh, you know, uh, accelerated integration from hugging phase. I'm really, really excited about trying out a deep speed chat. Um, so I was just, uh, like yesterday or today's morning, looking at the performance figures since they gave on, you know, what kind of machines we can use to fine tune, uh, opt 30 B model.

And hey, you can do it on one instance, eight a 100. Spend a day or weekend on it. That sounded very, very promising. Yeah. Some, some more questions, uh, specifically on the Bloom project here from the chat. So any, any thoughts on that, Daniel? Um, that was of course one of the, uh, few large language model options, uh, generally available that you can run your own environment and fine tune to specific purposes.

So yes, we used it. I think, uh, fine tuned, it's, yeah, a lot of the world used, um, lama, right? Because that was a lot of, let's say a lot of the people that publish online, um, because they're just doing it for research. They all use lama and so that seem to, over the last month and a half or so, make the most progress.

Anything that's kind of a LAMA derivative. If you do anything commercial, you can't use that. All of the metamodels are untouchable, essentially. And so Bloom was the only one that was really feasible. And now I think Databricks came around with, uh, Dolly. So I'm also very keen to look into that because can we now use the innovations that we've seen over the last month and a half with Apaka, et cetera?

Apply that to, uh, apply that to Dolly. Can we maybe do a fine-tuned model that is specific to a, to a domain, like an insurance document specific model or something that, um, I'm, I'm pretty sure most people know, like the, the medical literature fine tune models. I think that's super interesting to see.

Mm-hmm. Yeah, indeed. So before we move to the next question from the chat, I wanted to address, um, another question to Pascal. So, uh, yeah, we see that in Indeed, as you mentioned, the space is moving so fast, uh, God knows what's gonna come out in the next week and how this is gonna change our ways of working.

Um, so perhaps, um, it would be interesting to get your view on the existing limitations that you still see in terms of the inf. Um, and tech architecture and perhaps, you know, like the big breakthrough that you believe could help us overcome these limitations. Um, I actually, I'm, I'm getting, getting really positive or becoming really positive about it because I feel like, um, we're getting to the point where if you just want inference, like I think we, we have like these three stages, right?

If you just want inference, Pretty simple at this point. You want to fine tune, that's harder, but still pretty doable. And then if you want to train from scratch, of course that's complete completely different beast. Um, but just the inference bit, we're getting back to the point where it's, it almost feels like just another model.

They're larger, but um, the platforms are adapting pretty quickly and they're making it feasible. So then I think the major part of work is not actually going to be the lm, but it's going to be all of the other engineer. That we already know for a while is just, it takes time. Um, if you want to do question answering like Daniel referred to, well, you need to have a vector store, and that vector store needs to be integrated and you need to have some form of, um, process flow.

Okay? The user asks a question. You want to enrich that question with additional context. Now you send it off to the l l m. You want to keep your, uh, risk and ethics checks, uh, involved there as well. So you need some. Um, process flow that organizes all of this. And that's classic software engineering requirements.

Um, which I think, we'll now, again, see that, you know, there's, there's the entire complexity of building a product and then the model is just one small, little piece. It's an old image that has been used for a while, but I think we're gonna come back to this where l l M is just a small model in this larger bucket.

Thinking about the product, thinking about your, um, the technology stack, where's the data? If you can't, if you, if you don't even have access to your own data because it sits in so many silos, then it's very difficult to bring it into this one vector store. Indeed, indeed. And it almost feels like a, you know, with, with, um, very, very similar to what happened to data.

Um, when lops became a thing, it's gonna, yeah, like the new advances in general are gonna disrupt, uh, the skillset that data scientists and missionary engineers need to have. So, uh, would you would like to share your perspective on, um, what you would, you would, um, recommend data scientists and data engineers to develop, uh, as a skillset, um, as we move forward?

Daniel, do you want to do the data science first? Um, I think we haven't called off engineers here. Please go ahead. Okay. Uh, yeah, so from my perspective, like data engineers need to be, again, more and more software engineers. Um, I, I often have this feeling like if you, if you talk to people, what is a data engineer?

You either have the. Former software engineers that now just do a lot of data work. And then you have the second camp, which is, I write SQL queries and um, spark code. And I think that second part that might not necessarily be the most significant, uh, skill set anymore, but the first part of bringing a, a large, complex software engineering, uh, project together.

Connecting to all of the different software sys, uh, data systems and bringing those data, um, assets into one place and making them available for the l m that's something I would focus on and definitely look into. Like vector databases are so, such a curious new kit on the block that I haven't learned about in university, and now it's so obvious, like why wasn't this always a clear database style or paradigm that everybody always talked?

Indeed. Thank you, Pascal. Yeah, uh, please Daniel. Go ahead. And I think this is also a nice segue to our next, uh, set of questions. Of course. Uh, so on the data science side, I have to say compared to traditional, uh, machine learning where it's released straightforward, okay, you have your square and now you're trying to optimize for it.

By adding features, you have a permitted units. Sometimes, uh, sea pump engineering, et cetera, feels much more like a soft science. So when you're only a consumer of these large language models, um, in general, the question of how to evaluate sea quality of the output in a structured way is a bit of a tweaky one.

And I haven't seen a good off the shelf solution for that yet. And other than that, I, I think. It's a member of New Field, so you have really a chance to be creative and come up with new use cases that get unlocked with large language models. So, yeah. Um, should I also pick up the, uh, value and cost of tokens question, or should you just, um, so yeah, let, let's perhaps leave it to the end.

I just really wanted us to talk a bit more about, um, an. Um, example of the application and, um, yeah, perhaps Daniel could share a use case, um, that we've, um, we've worked on, uh, at TB and what exactly takes, right. Like to build a solution end to end. So Pascal started describing the flow, but it would be very interesting to hear from your perspective on yeah, what exactly needs to be built to make this whole machinery work.

Yeah, of course. Um, let's maybe stick with this, um, link chain questioning example, um, the foundation. Uh, but the foundation model as the name ORs, um, in the next step needs to be enriched with, uh, the additional data stores as Pascal just outlined. So, so set up your database and just also relevant data, make sure the permissions, etc.

Are app. Potentially, um, depending on, you know, if you chose the API consumption of fho hosting wood, you might have to tailor this model a little bit. So for the problem domain, for the use case on top of it says a few other layers that you have to build. You need to build the whole infrastructure together, feedback from the user so you actually have a chance to, uh, intuitively improve your models.

You have to put in place actually a good amount of risk infrastructure. Both on the input side, so when users asking the questions so you don't have, or, so you would reducing the likelihood of, uh, do anything now moments. And also a review of the outputs to make sure that, uh, you know, especially it's, but available to a wider amount of people in the organization or externally.

Um, the output is again, filled out. And then, You have to build. Think about the whole UX perspective, UI perspective, especially if you are thinking about use case that are going beyond, Hey, I want to put a chat bot on my website just a little bit better. So, so if we double down on this question of validating and evaluating the answers, which of course in the case of, um, textual data becomes much trickier than just like, checking performance of your model, where it's just our squirt or uc or whatnot.

Um, can you share any thoughts about how that can be, um, improved, automated, accelerated, and Yeah, if, if you have any, Any learnings that you could share with the audience?

I would say there, uh, two or three potential avenues that you can take. Um, you can try to go with the just auto attribution and automatic validation. You can try to use the fact that the underlying models are probabilistic and not just generate a text, but what the likelihoods for different outputs.

Potentially, it's also possible to, uh, move towards more automated evaluation methods. So thinking about the approach that is taken in reinforcement, learning from human feedback. This is small critic model. Now of course, this is all still, you know, rapidly evolving at the moment. Ms. Carl, perhaps you, you could also share your perspective on, um, yeah, how you see that part of the.

Flow in this solution, be further automated and, uh, and improved. Um, yeah, but if you, if you don't mind, so Daniel just, or Daniel's, uh, description, particularly in regards to the risks. Um, made me think a little bit because I know that, or most of us, I guess, know that Open AI invested very heavily in aligning the model and making sure that it behaves in a certain way and not in other.

Whereas these role models, of course, have not gone through this kind of alignment process as much. And so in theory, um, as we have more and more of these open source models available, um, and deployed all over the place, we have to replicate all of these risk safeguards again and again and again. Um, and so I was just imagining all these different organizations having their own.

That can do all of the, or will do all of these things that we've seen with Bing initially, where Bing, um, just says, I'm tired of answering your questions and ask me something intelligent, please. Um, but then I also wonder is this just a phase where for a while we're all gonna have a lot of fun, um, breaking these models out of their, out of their.

But then at some point it just becomes boring because ultimately I think then we've all tried it once, just like we've sent memes on WhatsApp. Um, but then at some point it's just a tool and we all become a bit, a little bit more mature about it and just get on with our lives and, um, use it for the problem that we're trying to solve.

I don't know. What's your, you, you shake your head? Yeah. Let's see. I think, uh, human nature of trying to break things and getting unintentioned reaction is not gonna go away anytime soon. Yeah. I don't know if there's any good tools or frameworks in, in regards to risk and compliance that I would feel like.

Our drop in solution and just take care of it for you. That's probably something interesting to watch. I think this is very interesting cuz you know, like even before Geni Geni, um, was a thing, um, in traditional ML and ai, it took us years to come up with some frameworks. Like, not just the general framework, but some frameworks around risk governance and compliance and ethics, um, to monitor and like have some ideas on.

Yeah, indeed. You know, our models are prone to be biased and, uh, we need to actually do something about it. So I believe, yeah, given the, the, the pace of. At which this technology is evolving. Like we definitely need to live faster. But I have a feeling based on, um, how this been so far, it's gonna take us, you know, like months, if not years.

Mm-hmm. Um, But yeah, like, um, this is very interesting that you've mentioned that, of course. Like, as part of the, uh, validation part and definitely a layer of risk and compliance is, uh, extremely important. And especially when you start working in organizations with their internal data, this is becoming even more important.

Um, and, uh, yeah, perhaps just like to finish it. Um, Danielle, could you share some of your experiences with, um, with the organizations and how the approach, uh, how open they are at tb, this kind of systems? I would say it is a, a lot of enthusiasm at the moment, uh, to use, uh, you know, gen ai mainly inspired by one of the biggest challenges is to manage the expect.

Since it's cool to go to the website, you enter something, you get a response. But of course it still needs to be integrated as we just described. There's still a lot of steps to go through. Um, maybe also on the risk side of using these tools, I think at least in the beginning, it's a good idea to have humans somewhere in the loop, especially, uh, when it comes to, uh, the outputs that are being used downstream.

That might affect individuals. So I wouldn't completely go on across that automation yet. And yeah, I'm not sure which Do you want to go, Victoria? Um, actually believe that, and this is, yeah, we went full circle, uh, in the model and, uh, I would actually want to pick up some questions from the chat. Uh, so I see some of the questions already dropped here, um, specifically for Daniel.

So the one, I'm gonna read it out loud. I'm also playing with lung chain and also Baby Agi. Okay. Good start. I wonder how do you think it can be used for commercial? These kinds of tools require generating lots of tokens and can be really costly. So yeah, let's discuss the cost part of it. Um, actually that's an interesting point.

Yes, it costs a bit of money to use the large language models. In the beginning, I would've expected, okay, to get a good answer, um, you're gonna pay, let's say a dollar per question. Now, uh, few weeks later, there was an announcement. Yeah, we are having, uh, GT 3.5 turbo and the cost reduced by a factor 10. And, you know, suddenly you move to Tencent and you're like, oh, okay.

That's a good price with probably further reductions, uh, coming in the future. But more providers are coming once market with comparable models and ones become more efficient. Um, also, I would say commercially, yes, it costs you, I mean, worst case, let's say a dollar to answer a question, but if, to get to a similar answer, you would have a human spend half an hour going through multiple, multiple.

Control ing and then writing up the answer. Um, actually that might be a dollar of verifi spend. So it really depends on the use case that you're exploring. I would assume the economics would be very different if you're running a public facing website and you're trying to finance, uh, the service that you're providing only with add.

In that case, the cost of comput might of course play a bigger role. The adds question is gonna be an interesting one because, Ad business of the internet is currently being challenged, so you know, how much can you rely on an ad-based business model of a website to, to solve your problem if the technology that you're now leveraging is actually challenging that ad business model in the first place.

What do you think about that one?

I don't have a qualified opinion on that. Just time to form it. Um, thank you. I like, I think I like the point, like, um, never underestimate your, the value of your own time. If it saves you 10 minutes, that's 10 minutes. You can have quality light, you can go for a walk. I think that's probably worth a euro for most people.

Yeah, that's a great philosophical spin on it, Pascal. Um, so yes. Next question to you then.

I imagine that European companies, especially in finance or defense, are super hesitant to use US Cloud providers or US based d API services given the US Cloud Ag. Um, so I don't see the second half of the question, but yeah, I, I believe that there was, um, some requests for the commentary on that. Um, and you have perhaps you could share.

Your opinion on how that's gonna be, how that needs to be managed, um, for the European solutions? I think we just got the second part of the question. Oh, yes, indeed. How makes the competitive disadvantage of EU companies do to the, to this regard to LLMs and to what degree can we get something working maybe to anelli space within this reality, uh, in the eu?

So, I mean,

I don't think that this innovation only happens in us. Um, if there is a, I mean, there's, there's close to half a billion people in Europe, um, and that, that's more than enough of a market for, for companies to say, okay, let's, let's create a similar offering in Europe. If that is, if it's worth it, if there's, uh, like commercial value in this, but.

Um, I'll, I'll not go onto the European level, but I do think there's a huge value in creating a product that feels almost like a document QA in a box where you commoditize it to a degree where a smaller, medium enterprise can just take one of their virtual machines or their virtual appliances and.

Install it as a web interface and then you load a whole bunch of files in there. You just drop them all into a folder on on, on the Windows file system or on a Linux server, and then you have this document, QA in a Box. I think that is something that is super democratizing and I wouldn't be surprised if that comes out and not too long from now.

And that would resolve a lot of these arguments of, I don't wanna send data to wherever. It could be us. Could be China, could be Europe. The cloud. It could be to my competitor. Thank you, Bosco. So yeah, I believe, um, we're at time. Any final thoughts, remarks, comments? Um, I would say it's super exciting at the moment and I think there's a lot of interesting things coming up.

Once would also cost for training in. Additional open models coming out, hopefully as a whole reinforcement learning agents on top of it. Um, distillation of models and also structured data augmentation plus, uh, action transformers. So I think it's gonna be an exciting 2023.

Yeah, from my side, I feel like. Not forget that this is a, technology in theory should make our lives easier and better. So this is, I, I keep thinking about this. How can we do less of everything and not more? Now that we have geni, how can we do, we do less emails, less reading articles, less just trying to stay on top of everything and just doing a little bit more of the things that we actually want to do.

So I think that's just a reflection that keeps coming to my. I think it's a very beautiful place to end. Thank you both and yeah, thanks everyone for listening and for your wonderful questions.