Hard Learned Lessons from Over a Decade in AI

Mike Del Balso [00:00:00]: That feels like it was a five-minute convo, and we're a whole hour, man.

Demetrios [00:00:03]: Dude, it was a great one, huh? Action.

Mike Del Balso [00:00:07]: We did it. We're here. Cool.

Demetrios [00:00:12]: It feels like sometimes we get caught up in this like new gen AI world and, and we forget like half the Internet is still powered and half of the ML is still making like the majority of the money.

Mike Del Balso [00:00:24]: For folks, I would say most of the value that people get from like decisions that come from machines is not coming from LLMs right now.

Demetrios [00:00:35]: Not yet. And it's still debatable if it ever will.

Mike Del Balso [00:00:39]: For sure.

Demetrios [00:00:39]: You get to see from this spot where you're talking to customers all the time who are actually getting value from stuff.

Mike Del Balso [00:00:47]: Yeah.

Demetrios [00:00:47]: And they have to cross almost like this chasm.

Mike Del Balso [00:00:51]: So it's kind of interesting to think about right now. Like all the LLM stuff, where does it fit in the like adoption curve or sophisticated, you know, all these like you see a, the bell chart. The bell curve. Right. And technology adoption curve. And you know we've been around for a while at Tecton and we, we've had to think a lot about that adoption curve. So just for some background. Well, maybe I'll give some background for everybody.

Mike Del Balso [00:01:19]: Just kind of, kind of like how, who, who I am first. So. So I'm Mike, I'm one of the co-founders of Tecton. I'm our CEO. Before working at Tecton, I led the Michelangelo team at Uber, which is the the infrastructure that powers all of the the AI and machine learning and, and kind of like decisioning that happens at Uber. And so that's like real-time decisions, slow decisions, but very importantly, things that are in production. And before that I worked on the the ads decisioning system at Google. So, like which ads do you see when you go into, when you type a search.

Mike Del Balso [00:01:55]: Right. And also very like production oriented like it doesn't. Nothing matters if you don't actually put it in front of users.

Demetrios [00:02:02]: And that's how they print money, and yeah.

Mike Del Balso [00:02:04]: And it's like, and it's a really good example of this stuff really matters for some businesses. There's some businesses where their whole and we should come back to this but some businesses, their whole business model involves or requires really smart decisions to be made automatically. In the spirit of, hey, let's really figure out how to get as much as possible to production because that's where you kind of unlock the value. What was getting in the way to get to production at Uber? Well, there was a lot of things at first, we didn't have A way to train models. We didn't have a way to serve models, stuff like that. But I had, you know, my first, like, month or so there, I had a spreadsheet that I, that I built that tried to catalog, like, all the places where, you know, there's hundreds of data scientists, all the places where, like, people kind of wanted to use machine learning in some way. Right? And this is back in the like late teens, 20 teens. And what we found is that there was a lot of projects that were super, super valuable.

Mike Del Balso [00:02:59]: A lot of projects that were kind of just like random experiments that were like, I don't know if we should spend any time on this kind of thing. And then we started learning some other dimensions that we felt like were really important to, like, to catalog and categorize to allow us to figure out how to prioritize it. So, like, does this team have IT shit together? Like, do they even know what they're doing in the first place?

Demetrios [00:03:20]: That was the the official title.

Mike Del Balso [00:03:23]: That was the column header, right? Do they have their shit together? But also, like, do they have enough people to, you know, if we help them build this thing too. They have enough people to, like, take it and take over ownership of it afterwards? Because we were the central platform. Yeah, right. And so one of the things we found so. So, you know, we had this big list of all these projects. And so we went through this process of like, helping them out. We'd start with, you know, start with surge pricing, then work with the ETA team and the fraud detection team, stuff like that. And that was good.

Mike Del Balso [00:03:52]: We were, you know, knocking things out, getting things to production. When we'd find a problem, we'd fill the gap, add it to the platform, platform stuff like that. Along the way we, we saw, we got like the obvious stuff, built the model serving, model training. That's the stuff you hear about all the time. The thing that was surprisingly like, we didn't really know, people didn't talk about this that much at the time, but was a major blocker that we practically spent a ton of time on in every individual project, was just the data pipelines. Like, there's just data engineering problem, right? And back at this time, the industry was at this stage where it was like, we just spent all this time, this investment, we're figuring out. It was like a really big deal to figure out how do you get all your data together and like, record all the data is like just coming off the big data thing. How do you bring all the data together so you can do something with it, right? And now it was like, hey, let's do something with it.

Mike Del Balso [00:04:42]: And so what people were doing at the time was like, let's just have a dashboard. Let's, you know, look, let's, let's do descriptive things, diagnostic things. There's a problem, let's figure out like what happened. But now people were like, let's get more value. Let's go to more predictive, prescriptive things, right? And that's more like forward-looking. And so doing that requires, you know, a whole new set of technology, but also it requires that strong link to the data. And so we were building those data pipelines again and again and we, we realized, hey, we're doing the same thing in every single project. Let's, let's centralize this, let's automate it, just bring it into the central ML platform.

Mike Del Balso [00:05:20]: And that's what we called the feature store. So that was kind of like the genesis of actually Tecton because we built this feature store and then, you know, that wasn't really a term at the time, but that kind of became this inflection point in AI adoption at Uber rather. And it was kind of this Cambrian.

Demetrios [00:05:36]: Explosion of did it make it more self-serve?

Mike Del Balso [00:05:38]: Super self-serve. Because you didn't need us to build data pipelines for you, right? And give you like a way to configure which data do you want, transform to which way, and it'll be available to your model in real time, and you can build high-quality training sets so you can build your models, dude.

Demetrios [00:05:50]: And you know what, there is a few different blogs that I've recently read on the Uber ML and Data Engineering blog. And it is such a breath of fresh air because it's all about these predictive ML use cases, and it showcases how wild you can get when you have this baked into your culture. It's. First of all, I think with Uber, it's just in there, and folks are learning it. Even if you're not on the data team. You can, if you want, learn a whole ton about data, ML, AI, all of that stuff.

Mike Del Balso [00:06:24]: It's all about democratization there. That was our like mission on the team. And it was kind of weird because on my, literally my first week on the job, they were like, okay, you're going to go present the company's machine learning strategy to the CEO.

Demetrios [00:06:38]: Like, no pressure.

Mike Del Balso [00:06:39]: I don't even know anything about this company of like five. So that was kind of a, like a, a funny meeting that, you know, it went well, but it was like you guys didn't really set me up for success on this one, but. But yeah, bringing all that up because there's just like, you know, that was the stage where it was like, how do we do ML? Right?

Demetrios [00:06:56]: Yeah.

Mike Del Balso [00:06:56]: And Uber was definitely at the front of things where we were trying to figure out, like, not just how do we do it, but like, we actually want this business to be driven by it in some way. It's. This is not like a random side thing.

Demetrios [00:07:09]: Yeah.

Mike Del Balso [00:07:09]: This was like, we want to power pricing with this thing. We want to power fraud detection with this thing. There's a ton of fraud.

Demetrios [00:07:15]: And those feel like the big buckets, like rocks and. And then you have the sand, which is where. When I was reading these blog posts, it was so refreshing to see that even when someone signs on to Uber now, they get different flows on what you've been identified as. And so it's very customized on the flow. Just when you're onboarding or when you're opening the app, you're seeing a different thing. And that to me felt like it wouldn't be possible unless you have that democratization for the ML AI personalization piece of it, 100%.

Mike Del Balso [00:07:53]: And so that democratization was all about, like, let's make this possible for a lot of people. Right? And let's try to make it like kind of easy. It's not like it was trivial. And you know, anybody can go build the highest quality, like the world's best quality thing in life, like 20 20 minutes kind of thing. But it was like, if we have a way to get value, we put some people on it. We had smart people on the team. They could figure it out and build something pretty good. Right.

Demetrios [00:08:16]: And you did say something else that you recognize. And I think I read this in another one of their blogs, like the From Predictive to Generative blog post that they did. They talk about how different models and different use cases have different SLAs. And so if it's an experimental AI project, you're not going to get the same kind of love from the team if that shit hits the fan at 3 am as if it is the surge pricing model, which you know is driving so much value.

Mike Del Balso [00:08:45]: Absolutely. And we see that with our customers today, like they have. So. So in going through this journey, we just to like finish the thing, and I'll complete that thought is, you know, we recognize the value of all this data, like bridging the gap between the data you have and like, how do you use it in some way for automated decisions and. And at the Time, I was running an ML platform meetup in the city here in San Francisco. And it was like companies like Facebook and Twitter, and like those, like the obvious kind of companies, all the platform teams would get together and just show what they're working on. And everyone was kind of working on something like tangential, and it was obvious that this is like a thing a lot of people are going to need. So we started Tecton to basically build the best version of that and bring it to everybody.

Mike Del Balso [00:09:28]: Right. So we feel like we've, you know, we kind of created this category of feature store. It's kind of morphed into feature platform. And you know, we're, we're the leader of it today, and that's what we do. I say that like nobody thinks as much about feature problems, features, data pipelines for ML as I do.

Demetrios [00:09:45]: You've been doing it for the last what, seven, eight years, something like that.

Mike Del Balso [00:09:48]: And well, I started working on this stuff at at Google in 2013. So it's been over a decade. Yeah, it's been a long time. So this is why we could talk about these like adoption curves. Right? Like a lot of ML projects back in the day were very experimental. They were back in the day. I'm talking about like even 2018, 2019. People were like, we would love to find a way to figure out how to use machine learning in our business, but we don't really know like, what to do with this.

Mike Del Balso [00:10:13]: Can you help us today? It's like we know this is possible, and we just got to get this stuff into production. Here's, here's exactly what we need. And so that means that these types of projects, I have a much higher expectation of making it to production. Right. So the the like kind of perception and the ROI calculation for this stuff is pretty different. It's like, it's like a lower risk. It's like we're confident we can do this, and we kind of like know what the value is going to be. We know we can reduce fraud.

Mike Del Balso [00:10:42]: If we reduce fraud by 1%, it's going to give us $10 million a year, or you know, whatever the equivalent is. We get x percent more click throughs on the website, blah blah blah. Right? And then in the LLM world, it's a little bit different. It's like we don't really know if we can do this. It's kind of like a Skunk Works project. We don't even know if this technology can do the thing that we aspire for it to do. And it's with the reliability that we need, with the forget even about like the enterprise-like readiness part of it. It's just literally like we don't even know if the thing we want to do is possible.

Mike Del Balso [00:11:10]: So there's a lot of those kind of projects because people don't really understand what's fully possible.

Demetrios [00:11:13]: Well, it does feel like we've got a very clear understanding now of where predictive ML can add a ton of value. There's use cases now, whereas maybe in 2018 those use cases weren't as clear. We're like, could we do stuff?

Mike Del Balso [00:11:30]: And when I, like when we started Tecton, I remember in one of our like investor conversations, they go, hey, what use cases are you going to be good at? And then I was like, I don't know, like I don't even know all the things people use machine learning for. So I can't tell you even which ones are going to be the big ones. Like I can tell you what we did at Uber, but today it's like, it's very well understood where the value of like automated, high-quality, automated decisions comes from. And you know, I'm just saying that as a way to convey the general, the more broad concept of machine learning because sometimes that also includes like rules based decisioning to every one of these use cases goes through like a little bit of its own maturity journey where if you're just going to get started, the most basic thing is you start with an if statement. You know, if this, if this person is in this country, show them this. If this person's in this country, show us, right? And then you can like kind of make it a little bit more complex. Well, but if it's nighttime, do this, and you kind of grow this like business logic over time. Sometimes you'll adopt like a rules engine.

Mike Del Balso [00:12:32]: And this is what a lot of certain types of use cases, like in the financial world, they go really heavy on rules engines. And this is basically just like super fancy, like if statements and case statements and stuff like that. Then at some point, they go, okay, this thing's like just some really brittle mess. And there's better ways to do it. Let's just like train a model and put a model in there. And so, you know, the rules engine basically is a model. You're building a model, but it's like a hand-coded model in some sense. And so there's that like kind of maturity journey.

Mike Del Balso [00:13:03]: But the use cases where the value or the value is accumulated is there's, there's a couple of them right there's so we particularly focus on this real-time or fast decision world. But you can think of like, you know, a lot of companies are, are basically at their core, you can look at, you know, you can look at anything from different, like different lenses. But a lot of, especially in like the financial world, they're basically just decisioning businesses. Like, if you go to like any fintech company, everything related to that company is just about how do you make like I guess like four kinds of decisions really well, we need to figure out how to acquire customers well, so let's be really good at doing like marketing, like automated marketing. Yeah, Right. We need to estimate risk for the customer. So given that this person wants a loan, you know, how much credit should we give them, how should we give them a credit card, or how much should we underwrite their insurance, or whatever the financial product is that we're estimating risk for. The third category could be fraud detection.

Mike Del Balso [00:14:03]: Is this person who they say they are? Right. And so that's like a really huge area.

Demetrios [00:14:07]: And I hear about the fraud use case, I just am like the cartoon where dollar signs go into my eyes.

Mike Del Balso [00:14:14]: Yeah, I mean that's a, that's a really like large $sign 1 But. And it's one that anyone who touches money has to deal with. But it's like a very large category and not just the like, how much resources do I allocate to it, but how large of an impact does this set of decisions have on my actual business performance? Like, if you're the CEO of Coinbase, you care a lot about your fraud detection system. It's not like a random thing. Right. And so you have the acquisition, you have the risk estimation underwriting for your customers, you have fraud detection, and then you have something that every company kind of deals with, but they don't always think about it as automated decisions, which is just operational things. How do we make our customer support team like better enabled? How do we help them do support our customers faster? You know, there's like a 5B or maybe like a category 5, which is like personalization, but that's a little bit broken into like the acquisition stuff. Like, how do we make our products something that like customers like and want to use kind of thing.

Mike Del Balso [00:15:15]: And so a lot of companies really what they're, they have to get good at is building these decisioning systems like you want to have. If you're the CEO of some Neo bank kind of thing. Really, you're thinking about like I need to make my business and my team really good at delivering high-quality decisions reliably, quickly, like quickly, if that's like what it needs, right? That like that's what the product surface requires, and you want to do it accurately so you get the value. And accurately doesn't always mean like detective. For example, in the fraud detection, like let's catch as many fraudsters as possible to reject them. But the dis. Because the decision science behind it, like the ideal rate for fraud is not zero percent, because if you're rejecting every fraudster, you're rejecting too many good people.

Mike Del Balso [00:16:04]: Yeah, right. And so having a different like threshold there, where you let in a little bit more, a couple more fraudsters, but you let in a lot more good customers can actually make a really big impact to the bottom line. The bottom line of your business. Oh, I didn't even think this is something we hear from our customers all the time.

Demetrios [00:16:21]: Well, and again, going back to the maturity of now that we've been doing fraud. Well, it's not like all of a sudden we woke up and said we're doing fraud now. It's been a constant cat and mouse game for sure, but.

Mike Del Balso [00:16:36]: But it's doing become more sophisticated over time too. And doing it with ML, doing it with ML. But the fraudsters are also using ML. Right. And so we have customers who are confident that their opponents, the fraudsters, are actually training ML models to like estimate or kind of imitate their anti fraud models so they can work around it. Right. So it's like it is a constant cat and mouse game. But that's where when you talk about the use case maturity, which problems are people using machine learning for versus not using machine learning for? There's these categories of problems where it's getting very sophisticated and fraud is one of those.

Mike Del Balso [00:17:18]: And certainly how sophisticated a use case is correlates to how many dollars can be saved or gained by it. You know, how can it affect the business? So I would put like fraud in that, in that bucket, the risk stuff, everything from like credit decisioning, loan decisioning under insurance, underwriting, that's kind of all something, you know. Recommendations is a much broader topic. I'm sure you talk to a bunch of people who work on recommendations in one way. You saw there's a bunch of like really good blog posts that give like an overview of the recommendation, how to build a recommender. I think Eugene wrote a cool blog post about this, but it's like really broad. There's like a lot of different ways to do that. But if you go to like fraud, it's kind of like we're trying to do the same thing again and again.

Mike Del Balso [00:18:02]: We just need to do it really well. So you can go a lot deeper.

Demetrios [00:18:06]: Yeah. Because there's a lot of variety that you can get with the recommender system. And that ground truth with the recommender is debatable, I guess, because you don't know, if we didn't show them this, would they have still bought it or whatever the. That recommendation could be. With the fraud, it's probably not as debatable. You can tell.

Mike Del Balso [00:18:27]: You can figure out fraud. Yep. You find out. You don't always find out immediately because you find out was there a credit card rejection or chargeback later on, but you can. You get the data because you know, when you lost money.

Demetrios [00:18:38]: Yeah.

Mike Del Balso [00:18:39]: Um, so that's like that. That's a real thing. But people get to that point of, you know, because every company also has to make the decision of, like, which of these decisions do we need to own in house? Like, do we need to build a system for. Do we need to have a team that knows how to build these models, et cetera, or which of these things should we just, like, you know, buy some API and kind of, you know, like, just outsource this decision to someone else? And so there's different paths. Some companies say we're never, never going to buy any other, you know, rely on anyone else, and they just kind of slowly get their way to like, building their. More. Their thing being more sophisticated over time. And the people who do this are the folks for whom their proprietary data is particularly valuable.

Mike Del Balso [00:19:26]: Right. It's particularly predictive of fraud. Right. And then there's people who are like, look, I basically am doing the same thing as the next guy. Right. And I just want to, you know, there's another company that's like fraud detection as a service, and I'm just going to use that as my API. And, you know, like, if it. I'm going to, you know, it'll shoot me up to 80% good, but it won't get me to 100% good.

Mike Del Balso [00:19:48]: And what we're seeing now with the sophistication as you get farther in the bell curve is people are. Are mixing both of them. We're building in house, but we're using the APIs as. Or the external services as extra signals.

Demetrios [00:20:02]: Like a gut check?

Mike Del Balso [00:20:03]: Well, kind of like a gut. But just an input to the model. Like, what is the other model that. The outsourced model thing. Cool. That will give me like, the sense of. Think of it as like, there's A credit check model, credit worthiness model somewhere that someone built based, trained on, you know, all of the people across America. Right.

Mike Del Balso [00:20:20]: That's great. But you know, if you're fintech company maybe you certain, maybe you service a very specific socioeconomic like demographic and so the, the kind of like national level model is not even relevant for you. It's like an input but it's not exactly. And you have a bunch of data specific to those people. So you think, hey, I can't really use these other, these other things aren't as predictive as they could be because they don't have my, they're not tuned for my population, my customers. And I have all this like proprietary data that I've collected from my customers that I should be able to use. Right. So people are combining the best of both worlds and that's like what happens when you get even farther along in the maturity, in the maturity curve.

Mike Del Balso [00:21:02]: Right. So bringing it all, you know, back to like the very beginning of this, that's what we're seeing in predictive ML people are still moving forward and they're. And it's a cat and mouse game so you can't just like chill out and do nothing. There's no fraud team that just has their fraud model and then they're just like cool, that's done. Let's move on to the next.

Demetrios [00:21:19]: Yeah, we're going to go figure something else out.

Mike Del Balso [00:21:20]: Yeah, it's, it's like a, it's like a thing that, that the CEO reports in like earnings calls. It's a thing that is like a lot of people are being hired to work on this stuff because it's very valuable. Right. And there's a whole like category of these decisions but on the other side of the maturity curve it's this, these less problem and immediate dollar sign driven projects. Right. Doesn't mean there's not value there. But it's, but it's at the ROI may not be as, as obvious or well understood upfront. The impact may not be as immediately like observable.

Mike Del Balso [00:21:58]: Like, like sometimes projects are like we got, we want to run this, we think it'll be good but we're not going to be able to like see the impact. Yeah. And sometimes, and then the feasibility of the project could be like a question mark.

Demetrios [00:22:11]: Yeah.

Mike Del Balso [00:22:11]: And that's, you know, that's just categorizes or describes a lot of like newer LLM type projects. This is new cool technology and it's not even super, super new anymore. People are getting this stuff into Production but it's not at the like, you know, you're a banks fraud team level of sophistication. And so you need different things for different levels of like, of maturity through that journey.

Demetrios [00:22:35]: Empowering the business and how much it's powering the business. And going back to what we were talking about with the Uber and saying we've got almost these different buckets of models that we support because we know how valuable they are and we can say very clearly this model gets all the love and care possible because this is what the business is built on. 100 this model.

Mike Del Balso [00:23:03]: Yeah. We've had to get really good at learning which category does a different use case fit into and being able to talk to our customers and help them even figure that out because they don't know like Uber, you know, they're way out there in the, in the maturity curve. Right. And so they have all of this experience of like well what are these different categories and how should we operationalize different levels of support etc. But if you're like a bank, you're an E commerce company, you got three models, you don't really know you. Maybe everything is like mission critical. You think of it as mission critical. So one of the things we do at Tecton is you know, people come to us with these problems where they're like look, we're just trying to like figure it out and, and we help them figure out like what do you actually need? Right? You, you're building a recommender system.

Mike Del Balso [00:23:52]: What kind? Like how important is this to your business and how like critical is it? What are the SLA that you want to take on as a business and if, if the problem is one where well you know it, this actually doesn't matter if this thing fails and we can just go press retry like who cares? It's, we'll press retry. We're going to use these predictions like next week anyway. Or the predictions are just going to go into like a slide deck where we show like some forecast or something. Then you don't need to treat this like you guys don't want to build all the systems that are going to and the course and take on the corresponding cost and overhead that comes with treating it like it's a mission critical thing for your business. It's just like, just be fine with pressing like retry. The sets of use cases that we spend a lot of time on that intend to be like really correlated with value are when, when people have decisions that are automated at a, a velocity that humans can't be involved in and at a speed that humans can't be involved in. So like the fraud one's a good example. But also like recommendations, real time pricing, a lot of the real time decisions that happen in like the live customer flow where you can't actually like have a person there doing the thing.

Demetrios [00:25:03]: Yeah. Checking. Is this what it says it is? Yeah. Or should we be serving this to this person?

Mike Del Balso [00:25:09]: Yeah. And a lot of those use cases are like, as we were talking about before, they're like critical to the business.

Demetrios [00:25:15]: Yeah.

Mike Del Balso [00:25:15]: Like one of our customers is one of the biggest insurance companies in America and they use us to do all of the logic, all of the, the reasoning. When someone's signing up for insurance and they are super, super careful about reliability, they're like, you know, one of America's, you know, favorite companies kind of thing.

Demetrios [00:25:33]: They can't ruin their brain.

Mike Del Balso [00:25:34]: They can't go down. Right. And it's like a super high trust thing. And so when you look at how those, those teams operate, they go slower than other like, kind of cool technology Bay Area companies. But they do that intentionally. It's not like because they're not good at technology, but it's just that they're checking every single box along the way to have full reliability, disaster recovery, resilience, all of that kind of stuff. So they minimize the chance that something bad happens because they have to be there for their customers and it's that much more valuable.

Demetrios [00:26:04]: It's an enterprise use case.

Mike Del Balso [00:26:06]: Yes.

Demetrios [00:26:07]: You know what I was also going to ask you is how have you seen the best teams translate the value that they are doing on the data and ML teams to the business?

Mike Del Balso [00:26:20]: Oh, that's a good question. And you mean, do you mean with respect to like, how do they report it up within the company?

Demetrios [00:26:26]: Yeah. And you can always say like, hey, we made or saved 10 million bucks and that's great. But I think a lot of times it's much more nuanced than that. Right. And yeah, and sometimes you can't be that clear on it or if you are, you're fudging numbers.

Mike Del Balso [00:26:43]: You know what though? I think like, it tends to be the use cases that get funding are the ones where it's more clear. Because before you come to like a cool vendor who can help you do something, you have to know like, is this even being prioritized in the company? Back to the fraud example briefly. Like, that's always a priority. And it's never like, we're not sure if we should work on this kind of thing, right?

Demetrios [00:27:06]: Yeah.

Mike Del Balso [00:27:07]: But there's a lot of projects that are like, well, we think we could do this, but we don't really know like its value. How do they get scary? Those ones are, are actually scary because you're one reorg away from like that project not existing. Yeah, right. Or one layoff away, whatever it is. Right. And so, so there's a couple of ways that we kind of like help our customers. One thing we do, by the way is we have, we have just like a value framework that we work with them on. So it's like we come in and help you ask the right questions.

Mike Del Balso [00:27:36]: So we can write down like with you, put the business case together with you so you can show to like whoever's upstairs the right like value. And like, hey, this is why, this is why we should spend this money. This is why we should work on this project in the first place. Right. But there's a couple of like it all comes down to either like we're going to make more money, we're going to lose less money. Right. It's going to cost less or we're going to reduce risk, things are going to be less risky. The place that something like Tecton typically helps with is we help you go faster as a team.

Mike Del Balso [00:28:07]: Right. So you're a lot of, a lot of these use cases also really depends on or it's very important for them to be able to react to the new hacker or the new fraud vector, whatever. Right. But go from idea to in production much quicker that velocity. Yeah. And that helps with like that reactivity, that market reactivity I was just referring to. But also just like, you know, we try to take something that took six months and make it possible in six hours. And so how much, how many more things can you do in that year as a team? I think if you're spending in just this is like the, the logic that a data, a data leader should be thinking about is like I have, I'm spending just raw or fake numbers, a hundred thousand dollars for this person.

Mike Del Balso [00:28:46]: They can do two cycles this year or they can do, you know, one cycle every day. How much more value am I going to get out of that person? Right. So that's like the speed dimension. The second dimension is just accuracy. Like your models will get better. Right. One of the things we do is we help you build types of features, different signals into your models that you just weren't able to build before. And so you're getting more of the good information from your data going into your models and your Models are becoming more accurate.

Mike Del Balso [00:29:12]: Right. So it's like new signals. But it's also a lot of companies, they go, look, we have like all of this data, all of these cool signals in different places. But the way we architected this thing, we can only really use like you have to choose. We got to use. It's going to be a batch, but it's going to use only the real time. All these data don't come together. And so like how do we use it all together into.

Mike Del Balso [00:29:30]: In one thing? So we help people use all of their data in every decision. So that's the accuracy dimension and then a third is just like the reliability and scale and stuff like that. Like as people are going through that curve and these use cases become more important, you're moving away from being comfortable or okay with like having a hacked together duct tape thing thing. And you're moving to a world where you actually need that level of resilience. And we have a bunch of customers who've, who've either had major outages with their existing systems that literally cost them tens of millions of dollars. So there's a cost component to it or their ongoing cost from their system. It's just implemented in a really like, inefficient way. Like they recalculate every single thing every time they do a prediction or whatever.

Mike Del Balso [00:30:19]: And you don't need to do that like you should. The right way to do that is with incremental compute. And there's like nice ways to do that. We can help you.

Demetrios [00:30:25]: Actually, we had Rohit on here a month or two ago and he was talking about that like you can save a lot of money just by tuning your data preferences just a little bit, you know, and recognizing do we need to checkpoint this data? How fresh do we want this data? And if you are okay with it going from an hour to a day, that can potentially save you a ton of cash.

Mike Del Balso [00:30:48]: Yep, yep, 100%. And so we. So those are kind of like some dimensions of value. But you know, if you're working, if you're listening to this and you're working on some, you know, data science or machine learning project, the way it's been motivated to you is you probably heard of one of those things. Like we're doing this because we got to make the model more accurate. And so there's. There tends to be a primary bottleneck. But the underlying thing is actually we care about all of these things.

Mike Del Balso [00:31:16]: But you know, there's one thing that's more painful first.

Demetrios [00:31:18]: Well, and making the model More accurate is one piece of it. But then you're thinking, like, what does that enable? If the model is more accurate, what does that mean for sure?

Mike Del Balso [00:31:29]: And that depends on the use case. Right. And it's really hard for platform teams to do this at scale. So like, if you're the, you know, ML ops guy or the ML engineer on your company's like, platform, ML platform, there's, there's actually not. I haven't, I haven't found it. I would love to hear someone else has it, but I've never met a platform team and I struggled with this at Uber, who's got a really good way to collect automatically collect and like, and like track the impact of their help to all these, all of their internal customers. Right.

Demetrios [00:32:03]: Value they're bringing to those data scientists or whoever's using the platform. That is a great call.

Mike Del Balso [00:32:09]: And it's hard, right. And you, the way you would like to do it, say, well, what's the common currency across all these? It's dollars. So why don't we just ask all of our customers, the teams that depend on our stuff, like, how much money is the saving you? How much money did you make? How much money is the speed up worth to you? Cool. Let's just get a dollar value for all of these and then just like bring them together and you know, that's kind of the best you can do. But it's still shitty answer, right? So that's, that's why it's in, It's a shitty answer because, you know, things change and the people who you're talking to aren't even usually good at like, giving you those answers. And so you have a bunch of like, kind of half answers and then you got to aggregate and then they're also out of date and you got to kind of aggregate it around them or aggregate them together. And so it's not a very high quality signal that you can present.

Demetrios [00:32:54]: Also, you may have a great product that's bringing a ton of value to the data scientist. But if the data scientist is working on the wrong project.

Mike Del Balso [00:33:02]: Yeah, yeah.

Demetrios [00:33:03]: And they're not making money.

Mike Del Balso [00:33:05]: Yeah, but that's like, that's actually up to, you know, the, the, the business leader. There's, so there's someone who's the head of risk, there's someone who's the head of acquisitions, who has to make sure that their team is working on the right problems. And, and it's okay. You know, sometimes you work on experiments and like, they fail or whatever, but you hope you get it and it's okay to fail, but you hope you get it right, like on average, so you're net adding value. But you need those leaders to like get it so they can have the right mindset about investing the right amount. Not infinitely, but investing the right amount in this stuff to enable the team to be successful.

Demetrios [00:33:37]: I do like the idea of speed of iteration and being able to shorten that because that feels like something that you can always use as an anchor and say, look, this person, whatever we're paying them, it's like we have two of them now because of the platform.

Mike Del Balso [00:33:51]: Yep. Yeah. What some of the things we like learn about our customers along the way. We help them try to figure it out. It's. We try. We have a very like, hey, we're your partner, we're here to help you kind of approach is like, how many people are working on this thing today? And if you didn't have to do this stuff, like how many people would need to work on this? What else could they do? Let's talk about what's the before and after.

Demetrios [00:34:11]: What would this enable?

Mike Del Balso [00:34:13]: Yeah, exactly. We like write that down. Let's be, let's be really crisp about that. So. But it comes to like, it's. That's a thing that we see people thinking about more and more now because we've kind of gone through this like just actually as Tecton, we've gone through this journey of. In the early days, all this stuff was basically impossible. You could never use streaming data, real time data.

Mike Del Balso [00:34:36]: That was like the initial thing. Let's help people use their fast data in their decision. And so I think of like Tecton as having gone through these three phases, where the first phase is let's make this stuff possible for people. And so our first customers were people who were like, hey, we really just want to use this real time data. We have all the streaming events and we got to make our models better. We don't really know how to do it. It's impossible for us to do right now. Can we use your thing and like make it possible? And that was great.

Mike Del Balso [00:35:02]: And we can put in the work. The platform's excellent. We'll help you guys do stuff that was never possible before. You unlock a lot of value. The second kind of stage was cool. I know this is like technically possible, but like I work in a team and I'm in name your Fortune 100 company. Right? I'm in a company with a bunch of like special rules. We have compliance, we have weird politics, stakeholders, a lot of stakeholders.

Mike Del Balso [00:35:29]: We need to like, this needs to work for our organization. Like we need to be able to share these things across. We need to have, I need to be able to report up to my boss like how much money we're spending on this thing, have visibility controls, all of that kind of stuff. And that was really a thing that was a bottleneck for a lot of like larger companies and, and like teams that need to collaborate is preventing them from being able to use like modern ML tooling technology. So that's kind of like the second kind of set of things that we worked on as a company. I think we're just kind of like definitely coming out of that zone where we feel really good about like the top. If you're, if you're a Fortune 100 company, there's no reason you shouldn't be using Tecton for your streaming and real time decisioning. Right.

Mike Del Balso [00:36:13]: Then the third thing now is really interesting where if you think about like, okay now if you're someone, if you're someone who's working on these problems at any like big company, you should be not blocked technically you should be not blocked from like the organizational red tape perspective. So what's the gap between what you're doing today and what you could be doing? Right. Well, just think about, you're probably not the best machine learning engineer in the world. There's going to be someone who's a better machine learning engineer. What would that person be able to accomplish that you couldn't accomplish? Right. Maybe they go faster, maybe they're smarter and they come up with like better, you know, designs better features. They can do the thing in a different way. Right.

Mike Del Balso [00:36:55]: And our goal is to help every one of those people be the best ML engineer.

Demetrios [00:36:59]: Oh, now it makes sense because I saw what you guys released with helping with the features and so this feature creation, being able to almost consult AI, which is pretty meta in a way.

Mike Del Balso [00:37:11]: Yeah, yeah, it is. It's really cool. So we launched the AI copilot and that's for people who are building AI.

Demetrios [00:37:17]: Yeah, right.

Mike Del Balso [00:37:18]: So you know, you hear about how do I use AI to help me write code, how do I use AI to help me like write my essay or whatever. But, but a lot of these decisions are not, A lot of the decisions in like, like in an organization are not driven by an LLM. They have, they're still very, for whatever reason. There's many reasons that are, that make a lot of sense, but they're, they're driven by structured decisions, by predictive models. Right. But that's great that that's a predictive ML model, but does that mean that it needs to be hand tuned, hand built, stuff like that? What? Like, why would you expect that? Like, you are the best guy, you're going to build the best model, right? So what we're building is a system to like a copilot to help every ML engineer and every data scientist build the best possible models for, for their unique circumstance. And that can be as simple as like, hey, right now the features that you use in your model are things that like, how do you come up with the features? They're just, you guys just got to think about them, you just got to like invent them.

Demetrios [00:38:22]: Well, and I remember you telling me back in the day, like some of the best data scientists you knew always had this intimate understanding of the data.

Mike Del Balso [00:38:29]: Yeah.

Demetrios [00:38:30]: And they would have to spend so much time with the data to get that where it was almost second nature. And now that is a perfect use case for AI because it can just ingest all that data and give you those types of. Well, have you thought about this?

Mike Del Balso [00:38:44]: So yeah, there's two kinds of Personas here, right? There's two. Like by Persona I mean if you go to a bank or you go to like, you know, big company, there's, you're going to see two people, one per person A sitting in a chair, person B sitting beside them. And they have different skill sets. The first guy's really good at ML stuff, doesn't really know what's going on with the problem. He's not like, hey, I've been dealing with credit card default credit card chargebacks for 10 years, right. The second person is like, look, I've been doing credit card chargebacks for 10 years and I'm pretty good at SQL, but I'm not like an expert at machine learning. Some companies have like the person who can do both of those. But that's really hard to build a whole team, right.

Mike Del Balso [00:39:25]: And, and so they're not like completely impossible to find. But you know, it's really, it's really tough to find them. And even then they're not the best in the world at either of those things. Right. And so what we're trying to help people do is if you have subject matter expertise, allow that subject matter expert to impact your production system directly. Right. So they can convey their intuition to the system. So you know what, like new users who sign up from this channel, there's always something fishy about them.

Mike Del Balso [00:39:55]: I never really knew exactly like how to catch it, but like that's where you should go look and figure it out. And this system should help you figure out the right signals.

Demetrios [00:40:02]: Right. And sketchy ass channel.

Mike Del Balso [00:40:04]: That's so funny. They have the intuition about where's what, which alleys are sketchy and which aren't. Right.

Demetrios [00:40:10]: That's so true. And it's so in their head. It's not like they can document all they want, but nobody reads that documentation.

Mike Del Balso [00:40:16]: Right. And, and, and then you have like, people who are like, you know, to take the metaphor, continue. It's like they're really good at driving a car, but they don't know the city kind of thing. Right. They don't know anything about the alleys. And so every alley is like, I don't know, I'll go down it. Right. And that's also a waste of time.

Mike Del Balso [00:40:33]: And so like an AI co pilot can help automatically understand what's going on in your data so we can guide that person in the right direction. And so what this adds up to is like, we're going to help automatically come up with feature ideas, automatically author, like literally build those features. And in Tecton, when you write like a small snippet a feature, a small snippet of transformation code, that becomes a fully productionized feature from right from the beginning. So it's all about like production pipelines for important decisioning systems. And so that stuff's already. All the hard part is solved.

Demetrios [00:41:05]: Yeah.

Mike Del Balso [00:41:06]: Now let's get the AI to just help people like write the stuff, write the right things and write it quickly.

Demetrios [00:41:11]: You've got the ability to just say, yeah, cool, I want to create some productionized features with that. But how are you making sure that they actually do what they say they do? Yeah. And what you want them to do, there's some feedback loop there where you're evaluating.

Mike Del Balso [00:41:29]: We let people evaluate on their own. A lot of use cases have pretty complex systems to evaluate and they've been building it for years and stuff like that. So we can allow you. We'll give you the feature data back. And this is like, we're the best at making awesome fast point in time correct training data. So you can then figure it all out. And then you can say, hey, like, actually this feature sucked, let's just delete it. Right.

Demetrios [00:41:55]: Because you know what? I forgot that you plug into a predictive ML system. So all that evaluation is very mature as opposed to, oh, you're plugging into some chatbot and you have to figure out that evaluation.

Mike Del Balso [00:42:11]: Right, Exactly.

Demetrios [00:42:12]: You have to get the new.

Mike Del Balso [00:42:13]: Exactly. Like the eval systems Tend to be pretty, like you already got it kind of thing. But one of the things we are starting to do is allow you to report your labels back to Tecton. So we can, so we can feed that to the AI. So then the AI can go. It can. It's not going to be perfect because it may not have your specialized evaluation system. But if it knows like, hey, these charges were fraud and these ones weren't, then what it can do is at least let me like, you know, guide my search for a better feature.

Mike Del Balso [00:42:43]: So then I can find features that are like, at least look really predictive from the data that I have. Right. And then I can suggest, maybe an AI can suggest to you, say, hey, look, I found these like 10 features ideas. Check them out. They're pretty different than like the other features we have in the system. Tell me if you think they're cool or not. Maybe you can say, hey, like we don't like that because we never want to use that signal as a whatever as a feature. Or maybe say, cool, let me try all 10 of these and then I'll, I'll press like deploy to production for some of them.

Demetrios [00:43:13]: That goes back to the expert saying, no, I've been down those alleyways, get that out of here. Or I actually haven't thought about that. Let me try it. And how quick it feels like that would make a very quick iteration cycle too, because it suggests a bunch of features. Now you could potentially get on the other side of the coin where it's like, well, we've got a lot of noise with all these features that it's suggesting and how quickly can you run and see if the feature has value or not?

Mike Del Balso [00:43:43]: Yeah. So this is up to the, like your own eval system because you can run whatever you want in your own eval system. But you do raise a good point. Like a lot of our customers, one of the thing, one of the problems we tackled in the like stage two thing I was saying where we're like, make it work for teams, right? Is I got these different teams, they're kind of working on the same thing. And I'm sure they're just doing a bunch of duplicate work. And each of you know this, this domain, all of these use cases are pretty like data intensive, which means cost. Right. And so like I'm worried that I'm running the same pipeline on this team and this guy rebuild this pipeline and now I'm just paying double the cost in compute.

Demetrios [00:44:23]: That's true.

Mike Del Balso [00:44:24]: And like we have customers where like a single pipeline can cost tens of thousands of dollars a month, you know, depending on your scale or whatever CFO.

Demetrios [00:44:31]: Is cringing right now.

Mike Del Balso [00:44:33]: Yeah. And so you want, you got to be careful about this. It's like an important thing for them. And so we have stuff to help them, you know, find duplicates. Like, hey, heads up. Like these things look like they're basically doing the same thing. If you want to like kill that kind of thing.

Demetrios [00:44:44]: Yeah.

Mike Del Balso [00:44:45]: And we're gonna, we're gonna have a lot more launched here. I mean, and this, now is also a good time to plug. We're definitely hiring and engineering and we'd love. There we go. Some excellent people who are working on these problems. Working on. Well, basically, like if you're, if you're a hard worker and you're a humble person and you're curious, you know, we'd want to talk to you. Right.

Mike Del Balso [00:45:07]: In the usa, in, in the USA we have a couple people in LATAM also, but in the USA generally we've got offices in San Francisco and New York. So if you want to work in an in office environment, hit us up. Um, but yeah, like we're working on these really cool, like these are really cool projects. Right. And they're, they're things where like we spend time not with random teams who don't really understand. Like if you work on an app, on the app layer, your customer is someone who's like, like the, the person who is not working with technology where they're just, you're just solving their problem directly and they don't even appreciate what you do. And what we do is we work with like infrastructure teams, we work with platform teams. Like at a lot of the coolest companies in San Francisco, their top AI teams, you know, they are our customers.

Mike Del Balso [00:45:57]: So we work with them every day and we have shared slack channels with them. And so we have cool ideas, we run it by them and so we learn a lot. It's like a really cool like environment from that perspective. So we have a lot of like friends and you know what all the other different companies are doing in their. And their ML and AI infra.

Demetrios [00:46:13]: Yeah, I love it because you're marrying the two worlds and also you're looking at it as a production line or you're thinking this. What we're doing here on the data sense doesn't. Just because it is predictive ML, that we are powering very mission critical products of the company for it doesn't mean that we have to totally divert ourselves or disassociate ourselves from the lift that you can get from generative AI?

Mike Del Balso [00:46:43]: Absolutely. We see like generative AI is like super helpful for us in two ways. One is the whole thing of like we help our customers do a better job building their machine learning, right? So it's like, it's like gen AI for Tecton kind of. Right. But then, you know, people use Tecton for a lot of gen AI applications too, right? And so one, one really important kind of like, like pattern that we're seeing that a lot of people are doing, especially in the marketing side of things, is doing a hybrid of applications. So where they go, this decisioning system, we have a bunch of models that are predictive models and they feed into a gen application to do some generation, like some personalized texts. We have a bunch of models that are predicting like what is this person's gender, what is this person's, this thing. And they all have to kind of work in one coherent data system that's making an inference at one time, right? So it's like, like a hybrid.

Mike Del Balso [00:47:40]: So if you think of like predictive ML, then there's this hybrid where it's like you use both in the same application. And then you know, what we do in Tecton is we're delivering in any decision, it's just like context that's going into some decision making system, some model to do something. You can build embedding syntact on power, process unstructured data, or use all of those signals to drive a prompt that goes into a Genai model. And so a lot of our customers are doing that as well. And the value of doing that is you get all of your data that is going into any decisioning system governed through one central location and you get like the reuse of compute across it. So you really have like one platform to integrate all of your underlying data systems. You have one hub for that, for that data to go through before it goes. It fans out to various decisioning systems.

Mike Del Balso [00:48:26]: And that's really helpful if you have like compliance or something like that as well. Sam.