The Five-Layer Cake Approach to Scaling AI Without Wasting Money

Alex Salkever: [00:00:00] Efficiency is money you don't spend.

Ambud Sharma: Correct. But efficiency is money that allows you to grow your business more.

Alex Salkever: Pretend that I'm your friend, a CTO of a company that just got a $500 million check from A16, and Andresen just told us, "Okay, just floor it. So you're gonna be the biggest company ever." Take this five layer model and help me think about what should I be doing at each of these layers to build that company?

Alex Salkever: When you change models, lots of things break.

Ambud Sharma: That's true. And when you change quantization, lots of things break. But when you change quantization, a lot of performance gets delivered as well.

Alex Salkever: Hi, I'm here at FinOps X with Ambu Sharma, who is principal engineer at Pinterest in charge of general technology efficiency. Is that correct?

Ambud Sharma: That's correct.

Alex Salkever: So tell me a little bit about, uh, you gave a, a [00:01:00] controversial keynote this morning, uh, that scared a lot of people a little bit, uh, about, and you're specifically talking about the five layers of, um, uh, of infrastructure and all the different components for optimizing AI.

Ambud Sharma: Yeah. So, I mean, uh, efficiency in general is important. And the way I look at it, efficiency is, um, that, um, efficiency allows you to buy headroom for brewing the business. So that's how, like, I've looked at it, that's how Pinterest looks at it as like, it's how do we reinvest those dollars that we get, uh, when we save resources, how can we re- re- reinvest that and make our business better, um, and, uh, help our pinners, help our advertisers.

Alex Salkever: Efficiency is money you don't spend.

Ambud Sharma: Correct. But efficiency is money that allows you to grow your business more.

Alex Salkever: Beautiful.

Ambud Sharma: So talk about, talk about, uh, the five layer cake. Um, the five layer cake is just a, an easy way to digest the information about trying to weave in all of the [00:02:00] problems and solutions that you have to engineer across the various layers of the stack to drive efficiency in the AI token landscape.

Alex Salkever: And the five layers, I think this Jensen also has, talks about this too. Yes,

Ambud Sharma: he does. Yeah. He does. And then that's, that's part of the reason why I kind of picked on it because - Yeah, because people know it. It's, it's, it's easier for people to remember. Sure.

Alex Salkever: Yeah.

Ambud Sharma: Um, and the core part is that things compound acro - uh, along the layers, and then they influence things across the layers as well.

Ambud Sharma: So an example is you have to decide what la - what hardware are you going to run. And that's a decision that you're gonna make. And whether you do that in the cloud, um, as in, um, you're getting some capacity in the cloud and then you'll buy some savings plans in the cloud, uh, but then you'll be contractually locked in for, for the duration that you select, let's say three years.

Ambud Sharma: So it's an important decision. You can't retract that decision. And if you ended up buying the wrong hardware, um, it will end up costing you a lot of money because then you will have to buy the right hardware to power the use cases that you're looking for. So you have to have an understanding of what is, what, [00:03:00] what's the right decision to do.

Ambud Sharma: Uh, the same thing applies for capacity as well. Again, these are physical commodities, this fundamental layer, physical layer of the infrastructure that, that can't be changed once it's in place, uh, or to change it, it's rather expensive. Uh, right? So, so the, as we talk about the five layers, uh, capacity, um, is layer two.

Ambud Sharma: Um, layer one was silicon and hardware. Layer three is the inference stack, which is what engine are you running? Uh, layer four is the model, um, that's deciding what's, what's the AI model that is running. And then layer five is the governance and the routing around it. Um, so that's the five layer cake that we talked about.

Ambud Sharma: Um, and the important aspect there is how do you make decisions across these layers and how do you drive incremental efficiency? Because efficiency is then, is always an, an, uh, an evergreen project. We actually call it our evergreen project. And, and, uh, the reason is because we will, we will never achieve 100% efficiency.

Ambud Sharma: 100% efficiency means zero cost. Uh, and that, [00:04:00] that's just impractical, right? So we will find new ways of solving these problems, and as we do, uh, we make incremental progression across, uh, at, at each layer, um, and we compound the gains that we get at each layer and deliver returns to the business.

Alex Salkever: So you've had a privileged position working at a, you know, a very large successful company with a very sophisticated technology, uh, ecosystem and infrastructure.

Alex Salkever: I would like you to take this five layer cake and pretend that I am your friend, a CTO of a company that just got a $500 million check from A16. Uh, we're an AI company and Andresen just told us, "Okay, just floor it. Now build. So you're gonna be the biggest company ever." Take this five layer model and help me think about how, what should I be doing at each of these layers to build that company?

Ambud Sharma: For sure. So l- let's start with first understanding what's the [00:05:00] business and the use case. Because if you don't kinda nail that in, uh, ahead of time, every decision that we make here is, is subject to change based on that holistic vision direction that you're gonna drive. Um, that includes things like, as we talked about, model selection, right?

Ambud Sharma: So what is it that you're trying to do? And I'll give you a very simple example. If you're doing a coding use case, um, and you're, you're doing, um, code generation, um, or agentic coding, you're extremely prefer bound. And what that means is you need a lot of context and that context is gonna send again and again to the LLM.

Ambud Sharma: Um, and you want some smart and very intelligent decode tokens or the decode phase that is coming out, the output tokens that are coming out. And so you have to think about your hardware in accordance with that. Right. Um, and just to give you an example, uh, what that means is, well, you might not care about a lot of bandwidth, but you really care about high performance prefills.

Ambud Sharma: Um, and you need the latest and greatest hardware to be able to go drive that because you care about [00:06:00] extreme amounts of compute density. You do care about things like, uh, things like memory bandwidth, but just enough memory bandwidth to, to keep your prefills at a high performance rate. And I, I kind of went dive deeper into, uh, to the, the technology aspect of it, but that's just to correlate an example of, uh, how that decision affected the LayerCake.

Ambud Sharma: So let's, let's come back around to the high level.

Alex Salkever: How do you - Yeah. Or if you, if you want, I can give you two business type models. Sure. Let's say we're building the next cursor. Sure. We're gonna kill cursor, and we're building the next YouTube killer.

Ambud Sharma: Sure. Let's - Very different

Alex Salkever: models.

Ambud Sharma: Absolutely. But they, they, they go, the decisions go very, very different because - Yes.

Ambud Sharma: What they're doing. Um, in a classic YouTube case, you're doing some recommendations, maybe doing some summarization, um, um, and that would be sort of the territory that you'd probably try to drive. Mostly recommendations, ads, uh, or if not subscription model. Um -

Alex Salkever: Yeah. Well, on the backend, a massive amount of video handling and

Ambud Sharma: transform

Alex Salkever: and -

Ambud Sharma: Absolutely.

Ambud Sharma: Absolutely. And, and you, you might wanna do some [00:07:00] transcriptions on those videos. Definitely. Um, and, and in that case -

Alex Salkever: Touch-ups, improvements.

Ambud Sharma: Absolutely, right? And, and so you're, you're trying to process a lot of that, that, that data. You obviously wanna have the capacity in place, but you wanna have the capacity in place for probably the latest and greatest generation.

Ambud Sharma: And the reason for that is, again, you're doing a lot of processing of, of this raw context. And what's coming out of it is just text, right? And the video is very large. Like what we're doing right now is, is, is probably a very large video file, but when it actually gets condensed into simple text, it's very little amount of data.

Ambud Sharma: Uh, so what's being generated out of the system is, is kinda tiny. What goes into being able to generate that text, auto transcriptions, uh, understanding of context, my tone of what I'm sp - how I'm speaking. If you wanna communicate and capture all of that and, and LLM to capture all of that, uh, it would require, uh, uh, a lot of context processing.

Ambud Sharma: And if it, once it did that, it creates these, these vision language models would go and create that text or [00:08:00] just transcribe the video out. So this is gonna give, give you an example. On the same side, um, the, the, the, the key part here is the model's different, right? We, we picked a vision language model. We didn't pick, uh, a regular text-based model.

Ambud Sharma: So it has a vision encoder layer, which is gonna have slightly different footprint. Uh, it doesn't require as much, uh, of compute, uh, in the very similar construct. Like you, you, you'll find it difficult to sat, and I'll, I can get into details, but, but based on our experience, it gets difficult to saturate the hardware on just the vision encoder layers.

Ambud Sharma: But if you wanna run the, the text part of it, once it gets into the conversion or embedding generation from going from, uh, the video frames to going into the embeddings that it comes out of, and then when it pushes into the language model underneath the hood or the, the, the, the fundamental transformer models underneath the hood, uh, then it'll engage the, the classical, uh, [00:09:00] uh, compute constructs, if you may.

Ambud Sharma: And so- So

Alex Salkever: you need to think about chaining together multiple types of multimodal models to make this

Ambud Sharma: work. Correct, correct, correct. And, and, and each one of those layers, you'd have to think about optimizing, and you'd have to think about optimizing in, in that, in, in that construct if you wanna drive the best price performance.

Ambud Sharma: And if that is. And I would assume that would be existential to this business, right? So, um, if you, if you wanna go after the - At

Alex Salkever: some point, unit costs are gonna be important for AI. I'm not sure when, but soon.

Ambud Sharma: I, I think they're important for us. It, they have always been - For

Alex Salkever: you, they're totally important.

Ambud Sharma: Uh, and, and the reason it's been important is because for, as I said, like, it's growth opportunity for the business. Right. I'll give you an example. If I save resources, that gives me a couple of different things. If I save on latency, let's say I reduce my time to force token, it actually bought the team, uh, more headroom, uh, the product more headroom to do more thinking.

Alex Salkever: Right.

Ambud Sharma: In the same latency budget. At the same time, if, if we, again, saved on resources, it would, again, provide either more [00:10:00] thinking token budget or maybe being able to process more, more requests for more users. So it, it, it pick and choose, but that's how business would basically decide to deploy how these efficiencies sort of land and, and what they materialize into, what the outcomes materialize into.

Ambud Sharma: They're not just dollars. They're, they're essentially resources freed up, now ready to be deployed for something else.

Alex Salkever: So starting, going, that, like, uh, scanning back to these two business models.

Ambud Sharma: Yeah.

Alex Salkever: Let's, let's go ahead and build that cake.

Ambud Sharma: So the, the, the cake on, on, on, let's say that just starting with the, the, the, the YouTube case, right, is this, um, you're gonna have to secure the hardware.

Ambud Sharma: Uh, you're gonna decide the, the hardware and you're gonna factor these decisions in. I'm gonna run vision language models and I'm gonna need to do transcriptions and I'm gonna need to do recommendations. I'm gonna make selections on hardware. You're gonna go out to your favorite cloud provider, you're gonna go secure some capacity, or maybe you will have your own data center and secure capacity.

Ambud Sharma: You have to think about that ahead of time. The hardware's gonna influence what that capacity is gonna look like.

Alex Salkever: So [00:11:00] you might actually have different hardware for the different model serving?

Ambud Sharma: Correct. Uh, and, and, or alternatively, you would slice it differently. So you could also do virtual slicing of the hardware and, and that would allow you to do better because you could do things like trough utilization where you would say, "Hey, at night, I'm not really getting that much traction of, of the users.

Ambud Sharma: My users are based off in certain geolocations." And in that, in that, in that case, you would be able to leverage that capacity for doing more transcriptions or maybe doing transcoding or, or, or, or some other. So

Alex Salkever: an efficiency trade off question of you may, maybe it's not the best hardware for that - Correct.

Alex Salkever: But it's cheaper to slice it and

Ambud Sharma: you've been utilized

Alex Salkever: 100%.

Ambud Sharma: Correct. Because at the end of the day, it's, it's just, uh, uh, to me, I, I, I look at it as like a, a home rental business, right? You have, you have a bunch of properties and you've been going to make sure that they are in use and making use, uh, of the resources twenty four seven.

Ambud Sharma: And if the closer you can get to that twenty four seven marker, the higher efficiency's gonna be. Again, two parts. One is [00:12:00] occupancy and utilization. Second is, are you doing productive use of that resource? Yeah. So two parts. But that baking that in, essentially, these are the, the decisions that you're making, then comes the inference engine.

Ambud Sharma: And you might have to make some tweaks to the inference engine to be able to, to either optimize the kernels that are being used or just what is being made available in the market is might be good enough for the use case. Again, depends on the scale. Um, if you're running at massive, massive scale, um, you probably want to do some specific tweaks in, in the area of, uh, how the encoder's working, how the prefill stage is working, um, including how you're doing quantization at these stages, which goes to the model layer and deciding which model are you picking, what quantization are you running?

Ambud Sharma: Are you running your own proprietary model? Um, are you gonna go and pick an off the shelf model and fine-tune it?

Alex Salkever: So run down all of the things you think, maybe, maybe it's too many, the things you think through at the inference, because I mean, there's frameworks, there's quantization, there's, [00:13:00] I mean, there, there's the hardware that you're gonna be running on, there's the speed, there's distribution.

Alex Salkever: Are you running on Kubernetes or not? Uh, you know, there's all these pieces.

Ambud Sharma: Yeah, so I mean, for when, when it comes to inference stack, and, and there's, when, when we talk about inference stack alone, um, there's the compute layer that is baked into it, right? So we talk about Kubernetes, you talk about, um, the, uh, the orchestrators on top of it.

Ambud Sharma: Uh, we briefly touched base on KV caching and, like, route ca- cache air routing, whether or not that is useful for your use case. In some cases, it may not be. If you're doing coding, it would make sense for somebody to go and resume the code that is running. Sure, to ca - to cache a lot

Alex Salkever: of the things

Ambud Sharma: that - Yeah, but when the session's over, there's no point in preserving that, um, other than for archival reasons.

Ambud Sharma: You don't need hot cache. You can, you need the context history, sure, but you can reprocess it. And sometimes it's cheaper to just reprocess than rather than to just brute force cache it. Um, so this is an example. Uh, but again, these are, these are just knobs. As, as we're talking through this, uh, these are knobs.

Ambud Sharma: Again, there's no right [00:14:00] size or one size fit, fits all knob. It has to be tweaked through experimentation. That's what we do a lot internally for us, ourselves, for even for, for our efficiency use cases. We try things, try them small, see if they work, if they work, provide good results. We have the data to make a, a case to say, "Let's do a larger rollout."

Ambud Sharma: If it doesn't work, we roll back, we try something else.

Alex Salkever: Why would, uh, someone at the inference level only want to. I mean, would they want to use more than one? I mean, like, maybe Triton and LLM or, I mean, are, are there, there's scenarios where you. Like, when does it start to make sense to, uh. I mean, it increases your operational overhead, but on the other hand, there's upside if you, if things are optimized for -

Ambud Sharma: 100%.

Ambud Sharma: I think it's a great question. So I'll give you an example. Uh, if you try to fire up LLMA CPP, it'll likely fire up and depending on the hardware, right less than 60 seconds, and do a decent performance for a given request level. But when you're running something like a VLLM or even, uh, TensorRT LLM, [00:15:00] right, and you.

Ambud Sharma: It is gonna need to compile all the CUDA kernels and CUDA graphs, and it will take some time, uh, and depending on the model size can take up a lot of time. And so you, depending on what flexibility you're looking for, if you're, if you're just, uh, if you're extremely resource constrained on the GPUs and you wanna, "Hey, I wanna try this, uh, model, and then 30 minutes later, an hour later, I need to run another model, so I wanna fire up another container and model comes online."

Ambud Sharma: If it takes 30 minutes for your container to come online, practically serving requests because it was doing graph compilations and optimizing, it's a lot of time wasted, and that time you're paying - It's

Alex Salkever: expensive at scale.

Ambud Sharma: Ab- absolutely, it is expensive at scale. If you have thousands of GPUs, uh, you're gonna pay a lot of money just trying to warm things up.

Ambud Sharma: And that goes back to the occupancy utilization problem, which is, "Hey, uh, do you have enough, uh, in, in the, uh. Do you have enough utilization to call it success or is it, um, is it just, uh, wasting cycles just trying to orchestrate?" Because at the end of the day, it's [00:16:00] just physical infrastructure that has to be running something productive.

Alex Salkever: S - and I assume also part of that is, are you designing your workload so that you won't have to wipe memory and go up and down frequently or -

Ambud Sharma: Yes, and then it, it depends, right? Again, the, the, the, you wanna have certain static workloads and you wanna figure out how can you minimize this overhead, which we, like, internally call runtime overhead, right?

Ambud Sharma: So that runtime overhead can cost you a lot. And it's not just GPUs, it also as applicable as, as much as possible in the CPU landscape, because I can't tell you how many people are wasting resources just doing Git compilations in the industry. Uh, and all it's happening, all that's happening is just compilations before you can run full warmup traffic, and that's just wasted cycles.

Alex Salkever: So to, to dial it back slightly, for the YouTube model, coding model, talk a little bit about inference layer, like how the two would be, like, some things that you think would be different from

GPUs.

Ambud Sharma: So you, in both cases, you would be running a fleet of resources. Right, right. Uh, your fleet of resources would be the [00:17:00] inference engine with the how, what.

Ambud Sharma: The, the, the, the big part that would change is how is the differentiation between the two. Correct. Uh, right. So what models are gon - going to be running at given point of time, as well at scale. So let's say you wanna do transporting at night in your trough traffic, that could be one example, right? So you could, you would have to switch the model over, but then you would do one switch per day, and then you would pay that one-time tax of 30 minutes for, for a, a GPU warmup run, and then you would basically go and load up your model and you're good to go.

Ambud Sharma: But if you're actually context switching all the time where you say, "Hey, I have to run this, and then I have to change the model and run something else, and then I have to run something else," uh, in different use cases, depending on, let's say, if you had different models or different languages, just, just, just give you an example.

Ambud Sharma: If you had different models or different languages, you would have to switch. Um, and in that case, you would have to figure out how to minimize it or size your fleet appropriately so you have enough backlog so you don't have to go and dynamically keep scaling the fleet. Um, just an example, right? So that's where partitioning of the fleet, [00:18:00] partitioning and sharing of the GPU resources and the underlying hardware matters so much.

Alex Salkever: So how do you also factor in, you're in a cloud, you only control above the Hypervisor layer, and there are sometimes variances which are not easy to predict below that layer.

Ambud Sharma: It's to study a lot of documentation, uh, do experimentation, and have very crisp, uh, benchmarkable workloads that you can run.

Ambud Sharma: Things that you understand. The only way you can tell when there's an anomaly in the system is when you know everything else. If everything else was constant, then you know what ch. And, and the results are different, then you know something else changed, and then you can investigate what changed. But if it, if, if you're changing everything, you're changing the model, you're changing the inference engine, you got OS got an update, your Kubernetes control plane got an update, well, I, I.

Ambud Sharma: Could be anything that could be going wrong. And that's an important part of efficiency is to actually have a predictable, testable, repeatable [00:19:00] workload that allows you to say, "Hey, did I. Did something change here? I did something change here for good." It's just baselining.

Alex Salkever: Yeah. So it's all, not all that difference than, like, uh, trailing latency basically.

Ambud Sharma: No, it's not. You measure like P99s or things like that. It's exactly the same. Jan and I were talking about this before, but it, it's exactly the same. It, it just needs to be correlated back to the ba- base primitive. So the things that we talked about in terms of latency, for example, the latency budget part, how's that different than a web server?

Ambud Sharma: If I, if you got more latency budget, would you do more work with it or you maybe reallocate and add more request load? Kind of identical.

Alex Salkever: So swimming up the stack, next layer.

Ambud Sharma: The next layer is deciding the model, uh, which is, again, you could start there. And by the way, you could start in any of these layers.

Ambud Sharma: You don't have to start at the bottom of the layer. They kind of crosstalk and depending on what workload you're talking about, you would start your business somewhere and then. Or you're a problem space somewhere and you travel up and down the stack.

Alex Salkever: But this one seems to be challenging for - Yes.

Alex Salkever: Particularly for two reasons. One, the improvement rate is probably [00:20:00] fastest at this layer. Um, and when, just because of the non-deterministic nature of the way we work with AI, when you change models, lots of things break.

Ambud Sharma: That's true. And when you change quantization, lots of things break. Uh, but when you change quantization, a lot of performance gets delivered as well.

Ambud Sharma: Sure.

Alex Salkever: I mean, FP8 is much better.

Ambud Sharma: Correct. And, and MXFP4 is even way, way better, uh, in performance. And so it depends on. And performance meaning speed, by the way, not model performance. So -

Alex Salkever: Which are different.

Ambud Sharma: Which are different. Uh, and, and it's important to kind of, again, going back to the baselining, right?

Ambud Sharma: Baselining at that layer matters because what's your, what's your canned workload where you can check whether or not the model's performing to your criteria, whether it is providing predictable responses even, uh, whether it is providing similar enough responses that you care about where you would say, "Hey, this is a, a, a, a reasonable but acceptable outcome.

Ambud Sharma: Uh, I'm willing to, willing to live with this as a business or a product." Um, that's important. Um, and if you can [00:21:00] bake that into your baselining again, um, for your use case, for your company, um, you would be able to repeat that process, try the new model out, see if it works, if it doesn't work. What do you need to do?

Ambud Sharma: Is there any opportunity or potential there maybe to do fine-tuning, maybe to do prompt, prompt engineering, where you would allow it to steer? Is it steerable? Uh, or if it's a d - a d- dead end scenario saying, "This is the best we can do with this model and we gotta go and look up for either a bigger model or, uh, uh, uh, a just more expensive model."

Ambud Sharma: And what that means is, when I say bigger model, just, it could be an MOE model which is, uh, just larger in size, or it could have, uh, a lot more, uh, or it could be a dense model. So it could just have a lot more compute requirements to be able to get that answer and have more intelligence.

Alex Salkever: Or a lot more active parameters for burn.

Alex Salkever: Correct. So around that, it sounds like then a, a recommendation you would give would be have a very active POC process around models, and I'm assuming also have a very [00:22:00] well-designed blue-green, uh, so that you can test things in once you feel like they're getting hooked

Ambud Sharma: up. You do. And every layer of the stack you do, but there, there's certain layers of the stack which will be immutable.

Ambud Sharma: Uh, certain layers will be mutable, like software you can perhaps change, model you can tweak, uh, hardware would be immutable. It is what it is, what you, what you ordered. And so you kind of make, kind of get to do the best with what you have. Um, and there are good examples of this, uh, out in the industry.

Ambud Sharma: Many people are working on this problem. This again, uh, a l - a, a very front and center problem for the industry. So many people are working on that. And there are. There's, again, no silver bullet. Tha's the important part I wanna kinda call out with the five Layer keg is that there's no silver bullet answer to this, right?

Ambud Sharma: There's a lot. These are all the knobs we're talking about. That's what we're kinda going segueing into, like, some of the deep aspects, but they're just knobs at the end of the day. H-

Alex Salkever: how do you think about models that tend to be more paradigm shifts? So example, like if a JEPA model lands, [00:23:00] or, uh, the latest GEMA, they didn't have a vision encoder even they had good vision.

Alex Salkever: Correct. So you start to be able to think differently about what's going on.

Ambud Sharma: They do. And th- those paradigm shifts take a while for understanding to be baked in as well as en - us as engineers as to what their performance characteristics are from a, how they impact the physical hardware. Um, I think we're, we're still early phases in that aspect because, uh, when we see a repetition, at least for me personally, this is, this is, this is not, not to with Pinterest, but just for me personally, I kinda try to see how.

Ambud Sharma: Is it repeatable? Is that trend very sticky to the wall? Um, and if it is and it's gonna repeat again, it will start showing traits because one sample, one anomaly is not really enough to kind of establish a trend. But if there's a repeated trend, then we can actually look how that would impact, uh, these decisions, essentially.

Ambud Sharma: A- and,

Alex Salkever: uh, and last question, the model layer. Um, what I noticed a lot myself is that people, even [00:24:00] architects, tend to overuse AI or, you know, very often there's things that work perfectly well with traditional ML or even non-AI processes, but they still point an LLM at it even though it, it doesn't make a lot of sense because there's robust ways to do it.

Alex Salkever: How, how do you think about segmenting out or identifying jobs that should not be in a model because that would also seem to be an efficiency lever to pull?

Ambud Sharma: 100%. So like, again, I, I, it, it's about what level. I, I, I, I wish we had a good way of quantifying intelligence. We, we still are early phases of quantifying and measuring intelligence, but, uh, depends on what that measured level of intelligence is necessary to complete a task.

Ambud Sharma: And the only way can, you can probably approach that is to even, uh, run an experiment. And again, this is where experimentation's so important, because if you try an experiment, and let's say it is 100 times better, then perhaps we're doing it wrong and you gotta go and take a look at it. Uh, and it also depends on [00:25:00] what's the accuracy level needed for some use cases, right?

Ambud Sharma: Sure. So if you're doing some mission critical work, uh, the requirements are very different, uh, than, than something that is more subjectively interpreted, right? So if I, if I do a variable naming convention, you know, modeled at variable naming convention differently, just to give an ex - give an example, um, it, it, it, it's subject to interpretation.

Ambud Sharma: But if, if, uh, if it's some accuracy and correctness code, and let's talk about, uh, let's say object classification, for example, right? Um, you could, you could have a vision model go do object classification or, or just annotations for you if you wanted to. Uh, is that the right, right tool, to your point? Um, it depends on the accuracy requirements.

Ambud Sharma: And, and in some cases, and this is where, if you remember, we talked about the prefill decode part, um, you can have an LLM crosscheck it. If you're having a ultra, ultra fast prefill where you're prefilling at, like, thousands of tokens a second, you can have a traditional [00:26:00] ML and you can have ultra fast prefill, uh, on, uh, an LLM, uh, or a, a vision model, and then basically say, "Hey, crosscheck my results."

Ambud Sharma: And you can, you can absolutely do that. And there's, again, to your point, um, it's not one size or this is right and that's wrong approach. It's most, mostly based on what makes most amount of sense.

Alex Salkever: So it's all within confidence intervals, basically, which is fair enough. It's the last layer up, up, and then we'll, we'll round it up.

Ambud Sharma: Yeah. So, I mean, last layer is more about understanding what is. It's, it's the. The, the, the reason I call it layer five is because it is, uh, at the highest layer of abstraction from the underlying set. Um, and that's based on routing. It's basically saying what, to your point, I think that your, your, your previous question actually is applicable to the route to - It

Alex Salkever: is a route, it was a routing que - Correct.

Alex Salkever: Yeah, I personally, like, I route a lot of my stuff to Homebrew because Homebrew does a whole lot of things very easily with, with Python libraries that I could pay a whole lot of tokens that have clawed to, [00:27:00] but make no sense.

Ambud Sharma: Yes, and, and simple models, a small model, uh, uh, a small open weight model could do a lot of simple tasks really well.

Ambud Sharma: You don't really need an expensive -

Alex Salkever: For OCR, like, call dockling, it's great model.

Ambud Sharma: Correct, correct, correct. And so, so that's the, the. Again, these are more specific models. Um, our, like, when we do recommendations, they're very, like, they're homegrown recommendation models, but, uh, and they capture the Pinterest Acecraft, for example.

Ambud Sharma: And they do a very good job of doing that as, as, as, as people love using our app. But they're not LLMs in a classical sense. They are machine learning models. They use similar technologies, but they're not LLMs. Uh, and that's the key part, so understanding what is the right model for the job, to your point.

Ambud Sharma: And what would. Let's keep it into the, the purely LLM landscape or, or, or, or, or the token landscape. Uh, what task should you send, um, uh, uh, what should you send to an LLM, and how do you wanna route that? You can have [00:28:00] governance in the se - in, in, in a, at a company level, uh, where you can write these policies, where they can be static policies based on workflows or even business units saying, "That business unit has no business re - business reason to go use this expensive model."

Ambud Sharma: And then there are cases like a, a trillion parameter model, like, if you're doing simple tasks, why, why do you need that model? What kind of reasoning do you have to, to establish? Um, and then there are other cases where you want to use that expensive model. Uh, for example, if you're doing coding and you're doing planning, uh, you definitely wanna use a larger model that has higher context understanding of your code basis, especially the large code bases.

Ambud Sharma: So that's just an example of, like, where routing plays in a role, and then we layer governance on top of it where we say, "Hey, um, we can have budgets, we can have limitations of what somebody's allowed to spend without stifling innovation, where people are allowed to play around, they're allowed to innovate and experiment with these models without shutting things off saying, Hey, you can't do that, that's too expensive.

Alex Salkever: It feels like we [00:29:00] swung a little bit too far in the experimentation side and now it's swinging back, uh, because you read these horror stories in the press of, like, uh, I forget, I mean, some company apparently was a 500 million dollar spend or something in token. It was crazy. I, I thought that was probably not accurate.

Alex Salkever: Yeah. But, but I do hear horror stories. So, so, like, how do we, though, strike that balance and, and how do you. I mean, because I think, to your point, if you're smart about it, you can give devs more than enough tokens to play with as long as they Don't use tokens for things they don't need to.

Ambud Sharma: Yeah, and I think this, this goes back to, we talked about, touched base on experimentation through the various layers ofour stacking, including benchmarking and things like that.

Ambud Sharma: Uh, and I think that's, that mindset plays a role here as well. And we use that extensively through, from, from an experimentation standpoint. Like our entire platform is, uh, very experimentation driven. We try something, it doesn't work. So you have to have that harness and you, in the same case here, you have to define what your experimentation is.

Ambud Sharma: And I, I [00:30:00] see that very often in the industry where people are not having a clear definition of what success looks like. And if you're not saying, "Hey, this is my, this is my hypothesis. This is what I was hoping to achieve and I tried it. Did it work or not?" And that could be a boolian answer or it could be a more gray area complicated answer.

Ambud Sharma: But let's say if whatever that answer is, is it meeting the hypothesis that you experimented out or, or set out to do in the experiment? And at that point, you either pause or terminate the experiment. If the answer is great, like you, you actually, the experiment succeeded, you would, you proceed to the next stage.

Ambud Sharma: And if the answer is it did not, then there's no point in putting more tokens or burning more tokens on that same experiment unless the hypothesis has changed. And that, I think like if, if you follow those. And these are similar patterns of what used, we used to do in classical infrastructure -

Alex Salkever: Sure.

Ambud Sharma: Um, at scale at, in, in specifically in, in the ad space, we've, we've been, these practices have been common.

Ambud Sharma: Um, so if we can apply that, um, as long as we [00:31:00] can apply those principles, the same thing would follow in the, the token space in my understanding.

Alex Salkever: Yeah. No, it makes perfect sense. I, I guess it's different just because you're talking to this magic box and you think it can do anything and, you know, it's, it's a, it's not like, oh, well, we, well, we need X numbers of new triple XL instances in, you know, in Amazon East or whatever.

Alex Salkever: It's not the, it's not quite the same thing.

Ambud Sharma: Yeah. I mean, it's, it's more complicated. Yeah. Let's just put it that way. Uh, uh, the, the in - the underlying infrastructure's more complicated. Uh, the, uh, with agents, the, the graph becomes more, way more complicated. You can have agent, an agent can call another agent, and you can have, like, it has controls to be able to correspond some compute if necessary, either directly or making requests in API calls to some other service which is underneath the hood backed by some compute.

Ambud Sharma: So if, if that graph becomes very complicated over time, um, it, it, it, it does complicate governance.

Alex Salkever: Mm.

Ambud Sharma: Uh, but at the end of the day, we gotta simplify. That's why, that's [00:32:00] why, uh, I try to put it in the five Layercake approaches because it, it has to be simplif - simple enough to understand, because we can, we can be only effective if we actually understand the problem really well.

Ambud Sharma: Um, and in order to do that, in order to achieve that, we have to be able to compartmentalize what this problem is. And if we, and, and into specific pieces, and then we can bring in the subject matter experts of those specific areas to say, "Hey, how do you solve this specific problem?" And then you can iterate and, and gradually iterate on it without losing context that it's a bigger problem to go and solve.

Ambud Sharma: So if, if I just went in and procured a whole bunch of GPUs and I wanted to build an, like, Agentic coding company and I had no CPUs to serve that, what good, good, good really is there? Sure. Right. Like, it doesn't solve the problem. Uh, and, uh, and, and at the same time, if you had a whole bunch of CPUs and I was missing out on GPUs, had no capacity to, and, and had no allocations to it whatsoever, went back to square one exactly the same problem.

Ambud Sharma: So again, it's, it's very classical in, in some ways, mapping it out to [00:33:00] what we know and do well in, in the FinOps and efficiency space. And in some areas, it is more complicated, which is things are, A, moving very fast, B, uh, the call graphs and the interaction of systems is more complex. Um, again, we're building on top of what was already there.

Ambud Sharma: And then the third thing is, uh, thing, things, uh, uh, the evolution and the future prediction of the evolution is unknown, which is how things will evolve, what rate will they evolve at? We don't know. Um, again, we've, this, this field has gone through ups and downs, uh, in the last couple of years where we've had plateau states and then we've gone, had, like, last couple months has been a, a surge of new models, uh, and then we're still digesting those.

Ambud Sharma: Uh, and then, uh, maybe there'll be a little bit of plateau and then we'll have more surges. So it really depends on, on, on that. And, uh, but it's a, it, uh, long story short, a- a- at, at, at the end of the day, it's an exciting time to be, uh, be [00:34:00] around because, uh, for the first time we have intelligence that was predominantly just, uh, a trait that we assumed to our species and is now available at our disposal where we can multiply intelligence.

Ambud Sharma: So that, that part I'm actually very excited about.

Alex Salkever: So let's end on one, one final note. Sure. Sorry. I just. Yeah, we got final. Agents.

Ambud Sharma: Yes.

Alex Salkever: Uh, so all the things you talk about with five Layer cake, they, uh, it, it makes sense. It also maps very nicely back to existing systems or existing methodologies. We're just sprinkling the eye dust on it.

Ambud Sharma: Yeah.

Alex Salkever: Once you start to move towards agents, and more specifically agent to agent or, uh, you know, extremely fast, extremely small transactions, things that are way beyond hooping capability to comprehend easily, nor would we even wanna try, h- how does things change? So - And we, and we're getting a hint already.

Alex Salkever: I mean, you saw like CloudFlare agent traffic past, you [00:35:00] know, bot traffic last month and it's just gonna go, right?

Ambud Sharma: O - 100%. But what, what gets interesting is the, the, the five layers we talked about, they're also very similar if you apply them in, in the Agentic and CPU era. There's certain things that, that change around that stack.

Ambud Sharma: Look, you gotta still procure capacity. You still need to decide what hardware you're gonna run your stuff on. The

Alex Salkever: agent may be deciding for you, but yes.

Ambud Sharma: Yeah. Uh, something is gonna decide. Agent would probably have some guardrails around it. Obviously it can't have the keys to the castle and, and a credit card that says go and spend all the money you want.

Alex Salkever: No, but it can optimize if like, if I have to split a hopper or fire up.

Ambud Sharma: 100%. It, it, it absolutely can. It can decide that and sometimes it can do better than us. Uh, and, and, but when we talk about inference stack, that inference stack changes to maybe a runtime and that runtime makes optimal decisions as to whether it is doing over subscriptions in some cases, uh, whether it is doing context switching.

Ambud Sharma: Again, same thing, same problem that we talked about, right? Are you context switching the LLM, uh, on the inference engine? Are you con- context switching the container? Uh, uh, [00:36:00] is, is it context switching requests? And the same thing applies to what are multiple types of agents running. Is this a homogenous agent with a wild true loop that's just, uh, context switching between the different requests?

Ambud Sharma: It really depends. And, and, but again, the problem space remains identical. Uh, when we come into the model, the layer four, layer five, layer five was always there. We had governance, we had budgets and infrastructure. That remains identical. Where did you route? How much capacity you gave somebody for running some workload to an agent in this case, that remains identical.

Ambud Sharma: Layer four, when you talk about model and quantization, I think that changes. Uh, and that changes slightly because, um, what budget was given from, uh, the model to, or the agent to be able to make LLM calls is probably what I would map it out to a little bit more. Uh, where I'd say, uh, what can, what, what LLMs are available for the agent to call for the various tasks, the model routing that we talked about becomes sort of an Agentic problem where it says, "Okay, I, [00:37:00] uh, I, I think this is a small task or this is a more complex task."

Ambud Sharma: Again, it, agent itself doesn't have the intelligence, it's just an execution harness to some extent, but it is making the intelligence calls to some LLM or - Sure. And there

Alex Salkever: might be some like agentic overlord that makes the intelligence

Ambud Sharma: decisions

Alex Salkever: like

Ambud Sharma: this job. You're setting guardrails. Yeah. And as we become more and more like, uh, like, uh, task or, um, goal oriented, uh, we will, we will all evolve to setting goals and constraints, uh, and, uh, desired outcomes, if you may, right?

Ambud Sharma: The pathways of, like, go, no-go areas, uh, what do we actually wanna get done from the system? If you kind of foresee how things would probably evolve in somewhat that direction, um, these are knobs that hopefully agents can solve themselves, uh, predominantly because we would set the context and we would give, make these tasks or skills that are available to the agent that it would go and drive for it [00:38:00] or, or it would be mindful of before it actually makes a decision.

Ambud Sharma: Uh, but again, all of that would tie into what, what's the intelligence levels of the models and that's, that's becoming better and better. And of course, the underlying infrastructure that we talked about.

Alex Salkever: No, makes sense. And, uh, uh, in the end, it basically all comes down to how well can we communicate with the LLMs and tell what we need them to do or the agents and everything.

Ambud Sharma: Yeah. And then, and again, we, uh, the, the, the core takeaway, one message I get core takeaway is the experimentation comes first, basically, right? Like that's the, it, across the entire thing, every layer has to be experimented with. And again, it's not about saving money at the end of the day. The, the, that, it's about powering the business, uh, helping your customers, uh, providing them better quality of service, uh, and just being able to cater to more customers.

Alex Salkever: Ama, thank you very much for joining us.

Ambud Sharma: Thank you. Appreciate

Alex Salkever: it.