The Unbearable Lightness of Data

Rohit Krishnan, Chief Product Officer at Bodo.AI, joins Demetrios to discuss AI's evolving landscape. They explore interactive reasoning models, AI's impact on jobs, scalability challenges, and the path to AGI. Rohit also shares insights on Bodo.AI’s open-source move and its impact on data science.

Rohit Krishnan [00:00:00]: Name is Rohit Krishnan. Company is Bodo. Bodo. AI title as Chief Product Officer. And I usually go for a latte, although these days I'm having a lot more decaf.

Demetrios [00:00:12]: All right. A little hat tip to my man Rohit, because he was the first person to turn me on to the idea of when a model is thinking, why don't we have the ability to click into the specific, specific steps that it is going over so that we can help guide it better? I'm Demetrios, your host for the ML Ops community podcast today, and this conversation was fascinating. I got to tell you, I'm a huge fan of Strange Loop Cannon, your substack where you've been writing. I feel like you are a original thinker in this world of parrots. And so it's nice to get a splash of differentiation whenever you drop a newsletter, I guess we could call it, or blog post. Right?

Rohit Krishnan [00:01:12]: That. That's very kind. Thank you very much. It's a labor of love.

Demetrios [00:01:17]: Yeah. Well, we're going to talk about a few of these different blog posts that have really captured my imagination. But the first thing that I want to dive into was your insight that I saw on Twitter. You are like one of the main reasons that I still check my Twitter.

Rohit Krishnan [00:01:35]: That's very kind.

Demetrios [00:01:36]: The thing that you put on Twitter that I absolutely loved was, I think it was talking about Deep Seek. Right. Or it was a Quinn motto, and it was the reasoning behind it. And you mentioned the UI on this could be better if we could click into each one of these steps, these reasoning steps, or each one of these passes that it's doing, and then adjust the prompt or direct it in a different way. And I just felt like that was something so in front of my face, you know? But at the same time, I had not ever heard anyone put it like that.

Rohit Krishnan [00:02:22]: That's very kind. Yeah. I think it was about deepseek, the R1, if I'm not wrong.

Demetrios [00:02:28]: Before, it was cool. It was way before.

Rohit Krishnan [00:02:31]: Yeah. I've been a Deep Seat fanboy for a little while. The way I think about reasoning models is that. Why is that interesting? The interesting part is like when we started out in the pure chatbot world, right? You ask a question, it's an autoregressive model. It gives you back an answer. In some sense. That's what you do, right? It gives you an answer. You don't like it, you kind of poke it, prod it, push it somewhere else, and it kind of goes and does that thing for you.

Rohit Krishnan [00:03:01]: Then we started Introducing tools that they could use. And then you could even say, like, remember there was a little While with like, ChatGPT when it started introducing the tools, but, like, it just wouldn't do the right tool. So you had to kind of stop it from searching the web. Or you had to say like, no, no, no, go use Python. Like, write a script. Don't just do the math in your head. I feel like with reasoning models, we are at the first part of that curve a little bit, right? Because clearly, if you're reasoning in the sense that you can see the thought process flow through, you know, where it's getting closer to sort of something interesting now, because ultimately it is a model still, and it's not like, you know, able to understand the entire context that you have in your mind. Sometimes you do want to just stop it and be like, no, no, just wait right there.

Rohit Krishnan [00:03:50]: I want to pick it up from there. Because that is the truly interesting part that I want to go off of now, in, like, mechanical, not mechanical, in like, scientific queries, that might be equivalent to finding that and saying, like, I want to take that and research it. I want to take that and go search for it. I want to take that and write a piece of code to test that assumption out, what have you. But it's the same if you want to do stuff for literature, right? Like, it found, whatever, an interesting illusion about something, something Shakespeare. And you're like, that is an interesting. That is a cool thought. I want to pick up from that point and extrapolate, because quite often, otherwise it might just go back or go sideways or go somewhere else.

Rohit Krishnan [00:04:33]: And ultimately, we are consumers of these models. So I think the more control we have of figuring out how we can take it for a spin, the better it is for us. The flip side is all of these models get far better at figuring out what we want as users because that's what they're being trained for. And as they're getting better at figuring out what we want as users, the need to do this perhaps reduces a little bit for the normal models. But for reasoning models, where fundamentally we ask really detailed questions and the answer back is not like a message but like a report, then the need to do that increases dramatically. Because I don't know about you, but every single time, whatever at work, whatever, you get a report back from somebody, even if it's extraordinarily well done, you have questions, you're like, I want to understand that a little bit more. I want to poke more here. I want to increase the sort of Magnification that you put on this, this part is not that interesting.

Rohit Krishnan [00:05:33]: So let's, whatever. I think that dialogue is really important and that dialogue is really hard to do with like a 10,000 word, you know, output that deep research throws at you. It is much easier for you to say like that's an interesting reasoning point. I want to give you insight as to what is useful here while you're thinking things through. We'll get there. I'm a hundred percent confident we'll get there.

Demetrios [00:05:56]: Really like making it a collaborator is something that is cool to fantasize about. Because when you're doing it as it is, spitting out these reasoning points or the steps that it's going through and you're able to. I think the way that I visualized it was you're able to click on one of those and then add more context or as you said, spur multiple different actions from that one piece of reasoning.

Rohit Krishnan [00:06:31]: That ultimately these are chains of thought. Right. It is, it's, it's rolled out chains of thought. And if it is having a chain of thought, then we should be able to pause it or poke it and kind of take things from there. I think that just that is the correct mode of interaction with that particular thought process. We can ask it to think more from that point. You can add more context. You can just say like that is really interesting.

Rohit Krishnan [00:06:58]: Why don't you go search against that and come back with some sort of insights like, because not everything needs to be equal. Valence.

Demetrios [00:07:07]: Yep. It's almost like checkpointing in a way. Or you're giving it more feedback as to what is important or where the signal is versus where the noise is correct.

Rohit Krishnan [00:07:21]: You do want the model to do the best job. But also like fundamentally all of these models, to a large extent they're blind as to their context. I mean you have to provide. That's why like models like O1 or whatever do so much better if you give it more information. Because like it just how would it know? There's no way for it to know. So like you just have to continually provide a huge amount of background before it gets really good at providing those answers. And the way to provide a large amount of background might be deposit in the middle and kind of add more information. I think that would be a phenomenal way for us to figure out how to steer them in the direction that makes most sense for us.

Rohit Krishnan [00:08:01]: Given a question, given an answer. Right. It also means like you might be able to side play with them together, saying you have a report generation that is A fundamental co creation aspect that is being built on. But there's also the part where it's not just the report creation. You actually have a conversational agent where you have a conversation simultaneously so that you know what things to put in the report. I don't know whether you tried out the new deep research feature from OpenAI, but like, it provides you a. It has a couple of questions it asks in the beginning and then refines the plan and then it goes up and just does its work for like whatever, 10, 15, 30 minutes. Now the output is amazing.

Rohit Krishnan [00:08:44]: However, most of the time the interesting stuff that it surfaces is somewhere deep inside that process of that 30 minutes. And if we had better steerability, would that be better or worse? I think it'd be better because you can actually get a lot more. Like you can ask it to go deeper into spaces that are most interesting to you within a report, as opposed to sort of asking you to sort of generally create a report and then having this like, you know, 5,000, 8,000 page output, the 8,000 word output that you then need to refine by saying, oh, guess What? In section 4.5, why don't you. That's a very bad way of analyzing it. At least that's what my hypothesis is here.

Demetrios [00:09:24]: It feels like there is that potential product that you have, the report and you can just go and drop comments in and then it will spur more progress or more action from the model and try and refine these different parts. Or there's this idea where you have the transparency as things are happening under the hood and you get to see it as it's happening and then like you said, steer it in the way or, or try to figure out if this is valuable or if this is not. I guess the hard thing in my mind would be it will probably take you a while to recognize what's important. And so as it's creating this report, it will probably spit out the report and then you'll want to retroactively go under the hood and say like, it's really right here on step seven that we should take a left turn instead of a right.

Rohit Krishnan [00:10:28]: Precisely. Because ultimately like you do want to have that input into things that you think are important. Because you're the one asking the question, usually to take an action, to learn something, to use it somewhere else, whatever it might be. The goal is for you to utilize that information for something, right? Even if it's purely for edification, just because you like it. And that requires more interactivity with the output that is coming. Your way. And the cool thing about R1 is that it's the first reasoning model where we can literally see its entire chain of thought. And it is just like a fascinating thing to read through on its own, right? Because the number of times that it kind of goes like, oh, but wait, or like, let me backtrack partially because of how it is trained, it's a very human friendly way of reading and understanding what it's thinking.

Rohit Krishnan [00:11:23]: So it's almost like you're peeking into the mind of somebody who's doing the work for you. And if that's the case, then you want to kind of steer it a bit better, to kind of guide it in the right direction. You don't want it to fall into a cul de sac or like something that you're not excited about or interested in. Right. So like the more input you can give it, the better. And why should all the input be front loaded as opposed to sort of stuff that you can give it along the way, so to speak. Especially if it's acting as an agent, it should be possible.

Demetrios [00:11:53]: I think a lot about the idea of cognitive load and how if this is going out to users and you as a user have to front load all of this information and get it right and then wait 15 minutes and see if the output is correct and then refine it and wait another 15 minutes, et cetera, et cetera. The way that I see this manifesting that is one potential future is that just like when you use notion AI. I don't know if you've used that much, but I've used it. It's very much like clicking. Yeah. And you don't have to prompt much unless you want to. And so there's probably you click into one of these chain of thoughts and then you have some basic things that can be done. The most common things where it's like go deeper here or search the web for and then you can add whatever you want.

Demetrios [00:12:50]: So there's a lot of cool product manifestations of this. I do wonder about the technical feasibility on the model side. Like, do you feel like it would be that hard for a model to pause and just cut that context? Because I know that with something like the Streaming API for Voice, you do have that capability built into the model and you can have that cutoff and then the model understands. Okay, I cut off here. I didn't say everything that I wanted to say. So I'm not going to assume that the person understands all of that stuff that I was about to say.

Rohit Krishnan [00:13:31]: Well, I think it's feasible. And the reason is primarily that ultimately the multiple rollouts of the chains of thought in order to select the right path that it should go down is relatively interruptible, so to speak. What has happened before or the things that it has rolled out doesn't necessarily just vanish, right? As long as we can sort of capture it somewhere. If you want to kind of reuse it or start from that, that's possible. And if you think about the smartest models that exist at the moment, they. You can, you can choose your reasoning effort to be high, low, medium, whatever, which to me indicates one that like, not just to me, which shows that you can just decide to not push it to the edge if you wanted to. Like once you thought it through, at some point, it kind of gives you an answer. It's just not the best answer considering the question that you asked, which effectively is a way of saying, like, at that point, at every point you have the choice of like, searching more, which means rolling out more and more, or taking in more input.

Rohit Krishnan [00:14:29]: And there is no reason taking in more input should fundamentally be, you know, architecturally difficult or super difficult. Right? I would be extraordinarily surprised if that's not being worked on, especially because it is being like people are trying to introduce tool use into these rollouts anyway. And if you introduce tool use into these rollouts, then almost by definition you are introducing a way for the model to do something. Now, tool use can be internal, like writing a piece of code and verifying it or whatever, but it can also be external, right? It can be doing rag over a corpus of documents, or it can be going and searching or going and finding or like, you know, whatever. Those are bringing extra external pieces of information back in. So, you know, you could even do a dirty version of at this point, if you're unsure because of whatever reason, or like if a user interrupts you take that input in and then roll it out from there. Totally feasible.

Demetrios [00:15:27]: I love this topic. I'm very excited for the future of how we interact with LLMs for this very reason. Because like you said, I would be very surprised if there aren't folks already working on this. And if they aren't like, get on it because it's. We're asking for it and it's not. If we could see it, then I'm sure the top research labs can also see it.

Rohit Krishnan [00:15:53]: Correct?

Demetrios [00:15:54]: What other wild stuff do you have that you ponder about daily or weekly?

Rohit Krishnan [00:16:04]: I mean, one thing that I keep thinking about is, especially since O1 Pro, I would say that like we've hit the point where the models actually are ridiculously good. I think we've, we've hit that point now. Something I think is like, the models will keep getting sort of better and keep getting faster, smarter, tool use, et cetera. Like all of that stuff will happen, which effectively means that at some point in the not too distant future, we will start being able to actually use these things for real life work more and more because we can already use them, but they're kind of like right now it's more jobsy and bicycles for the mind usage as opposed to, you know, autonomous usage. But I feel like we are getting to a point where we can kind of see a path towards doing that. Right. People would say they've already always seen it, which might also be true. But to me it's like, you know, understanding what's going on inside a code base and contributing a PR is the kind of thing that like you can visualize a model being able to do given the right kind of toolkit for it to get the information from that and be able to kind of go into that in such a scenario.

Rohit Krishnan [00:17:22]: I kind of keep thinking like, you know, I have a, almost a socioeconomic question of what does that world actually look like? Because a large number of existing normal white collar jobs will transform quite dramatically. And if it transforms quite dramatically, I don't know what the second order implications of that are likely to be for the economy. Like, you know, yes, sure, I'm not a doomer. So like, I feel like we'll come up with new ways to actually utilize the extra manpower and insight that we have lying around. And that's great, but is that it or like, you know, do we have a fundamental realignment in terms of what types of jobs basically just go away? Like the, you know, I had this post at some point about, you know, computers used to be a job, right? It used to be a job description for people, but it's no longer. Now it's a machine. And similarly, you can kind of envision maybe analysts or researchers being a job description for people today. That kind of goes away entirely, in which case you have to have a very different format of interaction with them, with the AI, such that you're able to capture the information or utilize it in some fashion.

Rohit Krishnan [00:18:33]: I don't know what that looks like. Right. I mean, would you end up having a lot more orchestration necessities as the primary kind of way of interacting with the models? Unclear, but I think that is an interesting thing that I spend a bit of Time thinking about.

Demetrios [00:18:47]: So I do love in one of your blogs how you talked about some jobs that could spur off of this are high quality data labelers that come through. And we know that. All right, we're going to. I forget how you put it, but we know that this is eventually going to be a job that will be done by a machine, but we have to get a large amount of data before that can happen. So we're going to recruit labelers in this specific field and you're going to be almost like temp workers for a bit.

Rohit Krishnan [00:19:24]: One of the things that I think of is that it's not even data labelers per se, right. I think it's almost like if these transformer based models are ridiculously good at identifying patterns and utilizing them in order to create sort of the output, if that is true, which it seems true, then it stands to reason that any ongoing process or any ongoing piece of work, given sufficient data, should be made more automated. Correct? Because like you have that data and if that data actually gets captured, especially if the process data gets captured, you should be able to automate it. And if that's the case, then effectively no matter what you do for work, part of your job ends up automatically entailing the requirement for you to be the training wheel for the AI that will come in the future. Like, I don't see how we get away from that first, like extremely simple tasks, maybe you can just get data labelers, that's fine. But you know, if you're doing sort of high value tasks that can't realistically be done by an AI, then what does that actually mean? I mean, does that mean that you kind of just do your job and there's data capture happening simultaneously and at some point, two years, three years down the line, or whenever they get actual amounts of data, then you actually get converted into the AI world? It's feasible, right? I mean, this is kind of what I mean by what is the economic reality of 2040 kind of look like assuming we don't do recursive self improvement and everything gets crazy, which is possible, but then that's very hard to kind of predict or understand. But to me, like this seems, this seems possible that like, I don't know, name a job, right? I mean, they all have to go through a certain set of pieces of work and every single person doing that job almost inherently ends up being a data labeler for some models that'll get created at some point in the future. I don't see what I mean, I think that's feasible.

Demetrios [00:21:27]: One thing that I think about is how much noise the models create right now and how much the output makes information cheap in a way. And because of that, going back to the like, no cognitive load, or a lot of cognitive load, if you're expected to read a 8,000 word report for everything and then have to tweak it and have to go back and forth with it, that type of stuff can get very noisy. AI is trusted to do a lot of these things that we're talking about, like in, in any job or field. But let's, let's take this example. I really like the one of understanding the code base and then submitting a PR. Sure, the person who has to review all these PRs is going to go fucking nuts because there's going to be a lot of shitty PRs.

Rohit Krishnan [00:22:29]: Correct.

Demetrios [00:22:30]: And so maybe you say, well, we'll get another AI to review the shitty PRs and then only the good ones will come up. And then you think, well, is that going to get rid of all of the noise? You know what I mean?

Rohit Krishnan [00:22:47]: I see what you're trying to say. I mean, I don't think of. I don't know if noise is the right framing. I like the framing, but I don't know, I think the. Well, first of all, of course you will get it to review the PR as well as submit. Like it just makes sense because like in some way reviewing the PR is probably easier than writing the PR in the first place. And it's also true that like many of them might not, especially as we are on the curve, many of them just might not be great. It's a fact.

Rohit Krishnan [00:23:22]: However, what we have done in that world is that we have effectively commoditized PRs, right? Like PRs are no longer a big thing. It's just like whatever, it's like a tweeting, you know, anybody can kind of do it. It's like it's a, it's, it's not a high effort, intensive, complicated research activity. It's just something that anybody can actually contribute to. And if that is the world, then noise is fine, right? Because like a lot of noise on Twitter, it's still fine because like we use it for some reason effectively because we are able to gather the signal from that somehow. And the work that goes in, just because it has higher volume and variance doesn't really matter because net Net stuff just works out. And PRs are become like in some ways, like you break it, you just do another one and you just break it. You do like you're not, you're not scared anymore that like you're just going to like break stuff because suddenly the velocity of you being able to do that just goes up.

Rohit Krishnan [00:24:19]: And I think that's true of a lot of different types of pieces of work. I think the place where it becomes hardest is things like academia or something where there is inherent roadblocks in terms of how much you can actually create. So, you know, highly productive academics, like, whatever. Asimilglu in economics, he writes like an extraordinarily large number of very highly rated papers in a very short period of time. I think that was a while that he was writing, I think like one a month or something that kind of goes into one of the top journals and you know, co authored, etc. And I'm like, okay, but today if you're able to speed that up and say like you can do a top journal entry once a week, what does that do to the entire system? It just kind of breaks it down, right? Because suddenly one a week means like no other part of the system can capture you. Like nobody can review it fast enough. Journals don't publish that fast.

Rohit Krishnan [00:25:17]: Like all you got left is like throw it up on archive, right? And at which point you have effectively, you know, broken through the existing hierarchies in terms of sort of how this can work in a very similar way to like whatever Agile, et cetera, kind of broke through the six month, one year release cycle hierarchy that used to exist. Because if it was one, six months, one year, then like there's only so much you could do. But now if you can push something every day, every week, every two weeks, suddenly like, you know, the way you do the work itself changes and some types of work just goes away completely. I think that is going to hit multiple different other parts of the economy. So I think noise is a, to me seen in this fashion, a certain level of noise acts as a feature, not a bug. Because it demonstrates that you have a sufficient velocity that we are willing to kind of take the hit of that noise, if that makes sense.

Demetrios [00:26:11]: Yeah, it really makes me ponder the idea of the human on the other side that is meant to be ingesting all of this information. Like if we're publishing these very in depth research reports once a week now as opposed to once a month, the attention of the folks who are reading that is so valuable because there's going to be, we think there's a lot of information right now and there's a lot of that we can spend our time on and our attention on, but it's going to get exponentially worse or better depending on how you're looking at it.

Rohit Krishnan [00:26:56]: I think one of the funny things is like, I am a fairly firm believer in, in the fact that like market mechanisms work remarkably well and market mechanisms doesn't need an actual financial, whatever, dollar market. It just means, like checks and balances are phenomenal. It just works, right? Like, I remember sort of for the last like decade, I've been hearing about the rise of deepfakes in some sense. Yeah. And it used to be, you know, it's AI now. It used to be Photoshop, whatever. Like, there's all of, there's always a new technology that makes it easier. And the same for like cyber or whatever, right? And you know what? They're all right.

Rohit Krishnan [00:27:39]: Like deepfakes are insanely easy to do today. It's just a fact. However, it's also true, like, we are all, we are not all like kind of walking on eggshells trying to figure out, we kind of figured out a way around it, right? There's enough signal in the noise that if I see something weird on Twitter or whatever, I have this, at least this thing saying, like, is that real? And that seems to kind of help because then you have a bit more logical or common sense to kind of capture those things. And to me that's like a kind of check and balance. I think the same thing applies for most of these large scale interventions that might come from AI changing the way that the world works for, for people. Like, AI makes writing research reports super easy. Okay? That means the value you place on a research report might go up or down great. But it also means like, you suddenly have very different yardsticks and benchmarks and heuristics for figuring out if one of these things makes sense for you to do or not.

Rohit Krishnan [00:28:46]: Right? Like, I mean, you kind of choose your battle, so to speak, a little bit. And as a, you know, in a weird way, as a human, what information should you actually look to ingest is a non trivial question, right? Like, what is your job? Is your job to read all the reports that AI is writing? That seems pointless? Like, why would you do that? Today I read sort of stuff that people write because I read the things that like my team writes, because reading that is how I get the information about what they want. But in the future, if like their AI writes something and my AI reads something and summarizes the things I need to know, like in a bullet point list, that's fine as well, right? Like, I mean, it changes the velocity, but the way I consume it will fundamentally differ in terms of sort of from, from the way that it works today. I don't, I'm not saying that's the way it should be or will be, but I can easily see it moving up a layer of aggregation, sort of curate the information coming towards you.

Demetrios [00:29:52]: There is another thread that I wanted to pull on which is in the blog post that you wrote about AGI and like what that actually would look like, because I feel like it is kind of playing on what we're talking about right now. Right. And it's, it's thinking about what, what does it look like assuming things continue to get better and get better and we are able to leverage the benefits that we already see. And one of these, you kind of break down different vectors on how it could get better or how it could maybe not get better. And I, like at the end of the article you mentioned, it's probably going to be a little mix of all of these. Can you explain what these vectors are in your head and then how they can get better or worse?

Rohit Krishnan [00:30:51]: Yeah, I mean, I think the, I should say the primary reason I ended up writing that essay was I think it's the one that said like, what would the world with AGI look like? I think that's the one.

Demetrios [00:31:02]: Yeah, yeah, yeah.

Rohit Krishnan [00:31:03]: Is that a large amount of the conversations around this future world that we have revolve around extreme assumptions. And extreme assumptions means that almost every extrapolation that you can make kind of just goes up in smoke. Like once you assume that, you know, AI will make creating better AI super easy to the point where we are not realistically going to be bound by energy or resource constraints, then kind of we don't need to like, what's the point? Right? I mean, you can say anything at that point and there's like no way to kind of prove or justify or back solve it at all. So my question was like, there are some fundamental constraints we kind of know, barring crazy breakthroughs which will happen. But like, at least for a base case, we should be able to look at this and go like, how much power is going to be required? How much, you know, how many chips are going to be built? And using those things, can we at least back solve to say like, even if, you know, 1h100 was all that's needed to run like a human level agent continuously, what does that mean? Is that good? Is that like, what is the change in the labor market or what's the change in the world that was my like, setup for the question, just to like do a base case scenario. And the thing is like, you know, people got to talk about trillions of AI agents floating around. And that has to be like, there's no AI agent in the ether, right? They have to run on some substrate. So like, if they're running on H1 hundreds, then like H1 hundreds need to be created and we have better chips now.

Rohit Krishnan [00:32:42]: Fine. H100 has some average power consumption that happens today. It has a service life which is actually not that large. I think it's like three years or something running at like reasonable utilization. If you're running it at 90, 100% utilization, the life falls even further because there's failure rates, right? I mean, llama paper talks about a very high failure rate for these GPUs when you connect them together, which is fair enough, because like, everything in the world has depreciation. Why should this be one of the few things that doesn't have. So now you have the number of GPUs, you have the amount of power that is created, you have some failure rate, which means you can have a rough trend of like either from a CapEx or an OpEx point of view, how much does this cost for you to kind of run it? And if you run it 24 7. How many agents are you kind of talking about roughly? That's the kind of math that I'm trying to.

Rohit Krishnan [00:33:32]: That I was trying to do. And I think the number that I got to was something like, I don't know, 40, 45 million full time agents, given all of these assumptions. And the point of that is not to kind of say that's too high or too low or even too accurate. But the point was to say that like, look, these are the assumptions that need to be broken if that 45 million needs to become 45 billion, right? Like, if an AI agent suddenly becomes as efficient to run as humans are, like a few watts, whatever, then it'll break. But then guess what? We need to get there. That is a fundamental constraint that needs to be broken. Or we need to say that like a substantial proportion of the world economy basically becomes chips and energy. It just has to.

Rohit Krishnan [00:34:14]: There's no way out, right? Like, I mean, think about it. If you added an extra, like if you double the labor workforce across the entire world, that's a major shift. But we have doubled the entire workforce of the whole world before, and that's been fine. It doesn't mean all of us are kind of jobless, right? It doesn't mean that at all. It means we have found new jobs. If you triple it still we'll find new jobs because like we. There's enough stuff for everybody to do that tripling the workforce doesn't necessarily mean that we end up in a dystopia. So I think all of the conclusion therefore is some version of the fact that we are bottlenecked by certain very clear, very strong physical capex investments.

Rohit Krishnan [00:34:59]: For lack of a better word, some of these things are getting better. I had some numbers about li I think I assumed thousand x improvement in some of these numbers from like O3 to like running an AGI on H100 or whatever. But these are the kinds of extrapolations you will need to make if you want to figure out what the world actually will look like as opposed to sort of assuming the conclusion which is to say what if chips get a million times more powerful and a million times cheaper? Like, okay, but like you gotta push that through. Right. Like in the sense of making a chip today requires ASML to create certain lithography machines and there's only so many they can make and if they can make it and TSMC has a certain number that they can manufacture. Like the world of atoms is not infinitely elastic. You kind of have to pick and choose where it comes from and figure out where it leads. And also answer the question of like, hey, is, you know, a substantial proportion of the world GDP going towards just creating semiconductors and energy a sustainable equilibrium? And I'm not sure it is like what the.

Rohit Krishnan [00:36:12]: Like we are all happy with Nvidia stock prices going up, but you know, it's at 3 trillion. Are we happy when it's 10? I don't know. Are we happy when it's 20? There is some breaking point where we kind of go like that is too large a percentage of the economy that is going into one company or one industry. Right. I mean like that's just the way that all S curves eventually bend. And I just wanted to kind of draw a line a little bit around what that. What bending cycle is.

Demetrios [00:36:39]: Yeah. And then you mentioned how there's, there's these different pieces. Yeah, it can get better here or it. The assumption that it's going to run on H1 hundreds can be one assumption. Maybe we drop that down and it all just runs on our CPUs. But we still need these other. We still need to think through the bottlenecks and where we're going to have different bottlenecks if we're trying to game out these scenarios.

Rohit Krishnan [00:37:05]: Correct. Bottlenecks don't disappear just because we want them to. I think they exist, they move. But, like, they 100% exist. And the story of the world economy has been one where we keep discovering new bottlenecks and breaking through those and going to the next one, which is why we effectively have somewhat of a linear growth curve, despite exponential effort going into each individual part of it.

Demetrios [00:37:28]: Yeah. So right now, you hear a lot of talk about the bottlenecks being the energy and chips and just GPUs. I've also heard some people come on here and they say it's not necessarily GPUs that are the bottleneck. It's more the memory on the GPUs that are the bottlenecks. And so folks are trying to figure out clever ways to disrupt that. Assuming that we get, as you said, this expansion in GPUs, we have to look at, like, what steps would we need to take for that to happen? I know there's a ton of investment that is going into this right now, but these kind of things are going to play out over decades.

Rohit Krishnan [00:38:17]: Yeah. I think the investment kind of tells you a little bit about what is required for us to kind of push that boundary forward a little bit. Like the amount of effort that's required for one extra data center, one new nuclear power plant, 100,000 new chips, which just shows you that when you're playing at the margins, the numbers get very large, very fast. Yeah.

Demetrios [00:38:38]: And then how do you. I really like the idea of how do you justify when enough is too much or too much is. Is too much. What does that look like where you say, all right, do we really want to invest half of the country's GDP into this one big bet?

Rohit Krishnan [00:39:01]: Yeah. I mean, I don't think we kind of. This is one of those things where we don't. We won't know until we hit it. But I'd ultimately, like, you know, right now, everything is kind of going up and to the right. Like, the investments are bearing fruit, and we are clearly being able to utilize these tools to get amazing results. Even if we extrapolate that thing out, there is this limitation on the amount of investment that we can sustainably take on board versus the amount of output we can sustainably utilize. And, like, it's a constant tussle between those things.

Rohit Krishnan [00:39:31]: Right. In order for us to get to the future, which I completely believe we will, but it's just, you know, if you believe something else, that is totally fine. But, like, we would need to justify it with some Similar assumption saying like this is what I expect, it's likely to happen and then you got to justify that and the justification can't just be. Or maybe it is that like hey, no, AI will make everything, you know, a million times cheaper. Which is like if that's a justification then let's talk about it. But like to me that's kind of hand waving the issue away. Right. You're assuming the conclusion and it's so.

Demetrios [00:40:09]: Vague and it is so ungrounded because I don't know about you, but it's not like AI has made much in my life cheaper.

Rohit Krishnan [00:40:22]: Right.

Demetrios [00:40:22]: I'm actually spending more money on subscriptions now than I've ever spent before.

Rohit Krishnan [00:40:26]: There's a lot of AI subscriptions to spend money on.

Demetrios [00:40:30]: So yeah, it's, it's funny to just like you said, be a bit hand wavy and think that just because we have AI we are going to see A a ton more productivity and B it's going to make our professional lives cheaper or it's going to make things that we need to get done in our professional life cheaper.

Rohit Krishnan [00:40:55]: Correct. I think like, you know, we have to have, we have to work towards doing that. I mean one way to think about the current kind of deep seek brew, haha, is that it's the first time, at least that I've seen that. Maybe not the first time, but it's like the first major time that like efficiency was a, even came up as a KPI as opposed to performance.

Demetrios [00:41:17]: Yeah, yeah, because it's, it's like we've been throwing everything we need at it.

Rohit Krishnan [00:41:24]: Correct.

Demetrios [00:41:25]: To get the performance but now we're looking at like oh, can we get that performance without a blank check?

Rohit Krishnan [00:41:32]: Correct, correct. And that has implications, right? Whether if you have the ability to do that, that has certain implications on what we might expect to get in.

Demetrios [00:41:42]: The future and how we can again tune those different vectors that you talk about and, or at least just be aware of them.

Rohit Krishnan [00:41:50]: Precisely. And look, things getting more efficient is part of this, is part of this anyway. But like GPT4 level models, when GPT4 came out was much more expensive than it is today. Right. However, there is a natural reduction in cost due to further and further advances when we learn how to do these things better and that you have to kind of account for that versus like if you think that trend is going to get substantially broken and there will be a discontinuity, then that needs to be assessed or analyzed. Now fact of the matter is even the current trend of decreasing cost that happened over the last few years, if you extrapolate that out to like the next 10 years, you could start seeing some pretty crazy numbers. I don't know if that's even feasible. But like, that is the crux of the question effectively, when is the breaking point for some of these things, like Moore's Law held over I don't even know how many orders of magnitude.

Rohit Krishnan [00:42:51]: So it kind of held for a very long time, right? And fall in solar cell prices kind of held over a very large order of mag. Multiple orders of magnitude. Is this one of those things that I can hold there as well? Maybe, but then we can go back to the question of like, can we run an O3 level model on a CPU and kind of work backwards in order to get some of those answers?

Demetrios [00:43:12]: Well, tell me about what you're doing for your day to day, besides being a complete philosopher. I know you've got a company, I would love to hear a bit more about it.

Rohit Krishnan [00:43:25]: Happy to. So the day job, like what I spend my time on. I'm the cpo, the chief Product officer of a company called Bodo Boto AI, which won't surprise anyone. Boto is basically the best way to do data engineering and data analytics. If you want to work in Python, that's the easiest way to say it. So you have large scale data sets that you want to do any kind of analytics or engineering on, transformations, processing, pre processing, what have you. Then rather than needing to use anything external like you know, Spark or Pyspark or whatever, or Dask, you can just do it in Python and we make that happen. So we are much, much faster, much more efficient, much easier to use than pretty much anything else on the market, which we recently went open source, which was one of the like big, big hurdles that was in front of us.

Rohit Krishnan [00:44:18]: So pretty excited about that.

Demetrios [00:44:20]: Nice congrats and thank you.

Rohit Krishnan [00:44:23]: Yeah, that's been a little while coming. So like we're finally glad that we managed to push it out there. And the goal is we want to get as many people as possible to try and play with it, use it, you know, get an understanding from the examples that we have already put out there, see the benchmarks that we put out there, knock on it, figure out where it breaks, basically make, make the world of Python great again, you know, because current like Python's came up as the lingua franca in the last several years for pretty much everything, especially AI. But if you want to do kind of any hardcore data stuff with Python, you kind of need to end up stepping out of it and going and using something else. We just wanted to kind of break that cycle and keep it inside Python, because that also means a much larger group of people can contribute and work on it.

Demetrios [00:45:09]: And so you're talking basically like data frames.

Rohit Krishnan [00:45:13]: Correct. So we, if you work in, I don't know, Pandas, NumPy, whatever SciKit learnt, it gets dramatically faster. You can see some of the benchmarks in our repo, but the numbers are pretty, pretty absurd. It's like 20 to 250x faster, depending on sort of what you're comparing it against.

Demetrios [00:45:36]: Wow.

Rohit Krishnan [00:45:37]: And that's important because, like, ultimately, if you want to do, I mean, if you want to train an AI model, right, you start with a bunch of usually raw data lying around somewhere and then that needs to be processed, cleaned, trained, tokenized, whatever. Like there's a set of, you know, deduped. There's a set of things that you need to do to that data before you are getting ready to the point where you shove it into Pytorch and click train py. That is the part that is the most cumbersome, takes the most effort, takes the most sort of. It's the most painful. And if you can just make that part faster, then I think it's going to be pretty dramatically useful, not just for us or for AI, but like for data engineering across the world. I think that's the goal.

Demetrios [00:46:23]: Agreed. And it's funny that you mentioned that, because I've heard actually in one of our most recent podcasts with a friend of mine, Floris, he was talking about how he identifies as an AI engineer these days, but he comes from the data science ML engineering world. And I was like, what's the difference? What do you have in your mind that makes an AI engineer different than an ML engineer? And he said, well, now I do a lot of software engineering and I have prompting sprinkled on it, and I have a little bit of eval sprinkled on it, but it's mainly software engineering. Back when I would consider myself an ML engineer, it was a lot of the data stuff. So that data processing, those data pipelines, that was his day in, day out.

Rohit Krishnan [00:47:15]: I mean, and you, even once you are in the AI engineering world, you still have to do it. It just like sometimes it is abstracted away for you by someone else who's kind of done it. Like if you think about the, you know, what was it, a couple of years red pajama data set, kind of like there's a bunch of these things that exist in the world. Right. Or all of these are effectively things that you still have to put in the effort for to get to a point where you can actually use it for AI training. So we have an example that I think is pretty neat to try and use the pile, the large open source and just to show how you can pre process it in different ways. And I think we're doing a benchmark at the moment to compare that against other existing large kind of competitors and other things that exist to kind of do data engineering work. And like that's, that's the crux of the problem that your friend is talking.

Demetrios [00:48:10]: About now of course, I can only imagine you being the product guy. How many times a day do you think about how you can add AI to the product?

Rohit Krishnan [00:48:22]: Don't even get me started. A lot. A lot. Across the board. From using it by ourselves, making us incorporate it far closer to the various kind of AI use cases across the board as much as humanly possible. Right. Because it's like it just supercharges everything if you use it for yourself. And I'm a believer in the world as you know, so like it's kind of, it's not going anywhere.

Demetrios [00:48:50]: There are different ways that you could plug it in. And I like this idea of just bringing it to that world. Or maybe you add a feature that is an AI feature potentially where you're helping people dedupe or you're helping people do what they need to do inside of the product. So how do you, how do you look at that?

Rohit Krishnan [00:49:14]: The core question for us is like there's a set of use cases where we can directly go and help those things out. So that's kind of the classic one. We're kind of doing that, you know, like I said, sort of pre processing of the data. We can even speed up or parallelize LLM inferencing, which is something intriguing that we discovered because it's Bota is built on HPC like, you know, MPI technology. So effectively we can parallelize the inference calls, which is kind of like a neat thing that we discovered. But the flip side of that is like, are there aspects of how people can actually utilize our product or our repo that we can make it easier from an AI perspective? So one example that I'll tell you here is that we're kind of working on is can we create like a cheat sheet that we can give to any AI that we speak with or talk to to say, can you make this code border compatible? And like it's something that we're working on. Because effectively, if you believe that the future will involve a lot more AIs actually working, then guess what? The importance of readmes and those kinds of documents go up rather dramatically. Yeah, that would be a great.

Rohit Krishnan [00:50:20]: That's an example of that.

Demetrios [00:50:22]: Yeah. I was just talking to my buddy Fausto about this and he was saying how hopefully now or very soon PDFs come with a LLM readable metadata piece.

Rohit Krishnan [00:50:41]: Because that would be lovely, man.

Demetrios [00:50:43]: PDFs are the bane of so many people's existence and especially when you're trying to ingest them or you're trying to make them capable of parsing PDFs is.

Rohit Krishnan [00:50:55]: Surprisingly one of the still largest problems that exist in the world. Which is weird, but it's true.

Demetrios [00:51:00]: It's. It's so weird and it's so difficult. That is the thing like parsing them and then con. Keeping the context or if there's forms. I know a lot of people who are in the finance realm and they say they really try hard to use LLMs, but the PDFs are the hardest area and if they could just get around that parsing PDFs piece, then everything would be much smoother.

Rohit Krishnan [00:51:28]: And so I completely agree with that.

Demetrios [00:51:30]: You feel like there's gotta be a new standard. Hopefully there's a way that we can add that metadata to PDFs from now on so that it's easier for an LLM to understand.

Rohit Krishnan [00:51:44]: Yeah, I think that's the. I mean, I think we will get there. I think Jeremy Howard started this thing to add LLMs txt to every website so that it's easier to scrape. I feel like similar things are going to happen to a lot more of these different normal existing formats as well. Because ultimately, like, if that's how you read information, then you got to make it easier for the LLMs, man. I mean, like it's. It's really hard otherwise, if you want.

Demetrios [00:52:10]: An LLM to read it, because that's the other philosophical point that is like, do I want to let the LLM have all of my SKUs and the categories and everything that I've worked so hard as a platform, this E commerce platform to aggregate, then if I just give it to an LLM, is that going to make my moat? Not.

Rohit Krishnan [00:52:38]: I think it's a classic question, but I feel like even if you don't want it to get out, you might still want to do it so that your internal LLM can read it and make your life easier. So in some sense, like, I don't think there is a Luddite defense here that works scalably, right? I mean, you kind of need. The train's only going in one direction.

Demetrios [00:52:58]: Yeah, well, we had a whole panel discussion on this because what does that world look like? Where is it a Google agent then calls your platform of whatever E commerce agent? Or is it that the Google agent goes from end to end? Or is it that it's even before you go to Google and you've got your own agent on your laptop?

Rohit Krishnan [00:53:24]: I mean, it could be any of like. In some ways you can easily see it being any of the above. I feel. Right now, I feel agents are still not strong enough to be able to kind of get autonomously deployed. But if you replace agent with like situationally aware function call, which is which, then 100% that's going to happen now, right? Any piece of information that is coming onto my laptop or any piece of information, if you're Google, that is coming onto your servers, you can have a function like you can have a context aware call to kind of figure out what you want to do with it, or proactively as well as reactively. I think that'll change the way that we actually deal with these things.