Sign in or Join the community to continue
The Caveman Prompting Challenge
Posted Oct 01, 2026 | Views 10
# Agentic AI
# AI Governance
# Tokenomics
# FinOps
Share
Speakers
James Barney
Head of Forward Labs (Forward Deployed Engineering) @ MetLife
Passionate technologist and experienced technology leader with a strong background in cloud engineering, platform engineering, and data engineering.
As a visionary and hands-on leader, I thrive in building and inspiring high-performing teams to drive innovation, achieve strategic goals, and deliver exceptional results. Throughout my career, I have spearheaded the successful development and implementation of scalable solutions in AWS environments, promoting best practices and enabling seamless cloud migrations for diverse business units.
A natural problem solver, I enjoy leveraging open-source technologies to address real-world challenges and optimize processes. Committed to fostering a culture of continuous learning and cross-functional collaboration, I am driven by the belief that empowering my teams leads to meaningful contributions to the ever-evolving technology landscape.
+ Read More
Demetrios Brinkmann
Chief Happiness Engineer @ MLOps Community
At the moment Demetrios is immersing himself in Machine Learning by interviewing experts from around the world in the weekly MLOps.community meetups. Demetrios is constantly learning and engaging in new activities to get uncomfortable and learn from his mistakes. He tries to bring creativity into every aspect of his life, whether that be analyzing the best paths forward, overcoming obstacles, or building lego houses with his daughter.
+ Read More
SUMMARY
Caveman prompting has one rule: why use many words when few do the trick? It saves tokens on the way in and on the way out. Push it too far, though, and the output falls apart. So how far is too far? Nobody has benchmarked it yet, and that question opens our conversation with James Barney, Head of Forward Labs at MetLife.
James spends his days connecting new AI capabilities to old business problems across dozens of regulatory regimes, and he still finds time to push code. He explains how the FinOps Foundation's AI working group took on the most basic question: which model for which workload, and why the answer always comes down to cost, speed, and accuracy. We get into Anthropic's launch pricing for Fable, why a million tokens is easy to price and hard to explain, and why every stakeholder eventually tells you what they really care about once you name the wrong North Star.
+ Read More
CONTENT & TRANSCRIPT
Demetrios: [00:00:00] Back in the day, we had browsers and we would go and navigate to a place and get on there and look around. Put it into another- Spreadsheet ... and yeah, a spreadsheet, then manipulate it in some way and then send it off-
James Barney: Yeah ...
Demetrios: so that it could go onto a PowerPoint and all of that goes out the window.
James Barney: Can be...
James Barney: It goes completely out the window. And so yeah, caveman, why use many word when few do trick, right?
Demetrios: Mm-hmm.
James Barney: Like, that's the bread and butter of the whole thing.
Demetrios: Man, even I used GitHub the other day and I was like, "Why would anybody go on this UI and expect to do anything?" It's so hard.
James Barney: I think the, the trouble that I'm predicting that we'll f- we'll run into is what I call like the, the latent shopper problem.
James Barney: You know when you just like walk into a store and you don't know specifically what you're looking for- What you want ... and the person comes up and they ask, "Hey, can I help you find anything?"[00:01:00]
Demetrios: All right, so now I'm here with James, the Global Head of AI Enablement at MetLife Give me the breakdown. What do you work on?
James Barney: We focus on just giving our business partners, like, the tools they need to, to get their jobs done, right? And so in, in the world of AI today, that means getting them AI, right? And so, uh, what does that mean to, to a large global organization?
James Barney: It, it really is understanding all the various laws, regulations that go in, go into effect for different regions around the world- Mm-hmm ... uh, different countries, um, understanding, you know, customer rights and, and important aspects of, of, of governing how new technology can be applied to old business problems at a scale, uh, and, and faster than, than, than we've [00:02:00] done before.
James Barney: So it's, you know, day to day, it's a lot of meetings. Uh, I still push code. Um, you know, I've... I open up pull requests. I review my, my engineers' pull requests, things like that. Um, like just today, I set up a GitHub action for looking at, you know, some agent skills- Nice ... that we're developing. Whatever the problem is, that's what, that's what we solve.
Demetrios: When you say figuring out the different pieces in different countries, it reminds me of these games that my daughter played in this book where, you know, it would be, like, a drawing and then the actual name or word on the other side, and you have to, like, draw a line to it.
James Barney: Yeah.
Demetrios: I imagine you just have a bunch of laws and rules and regulations on one side, and then, all right, this maps to the EU, this maps to the US.
Demetrios: Yeah. Except in probably a more automated fashion.
James Barney: Yeah, I mean, we, we do have a lot of these [00:03:00] processes automated. The, the nice thing is that we have experts in the region-
Demetrios: Hmm ...
James Barney: who are obviously the experts-
Demetrios: Yeah ...
James Barney: um, who work closely with the government regulators, you know, people like that, to understand what the rules of using these tools either are currently or are going to be.
James Barney: Um, so that way we can understand how we can, how we can apply this technology in a, in safe and reasonable way to make, you know, our customers more confident about being our customer.
Demetrios: And you're also working in one of the FinOps working groups, right? The AI one.
James Barney: Yeah. Yeah.
Demetrios: What does that en- entail?
James Barney: So the, the FinOps Foundation obviously, uh, does a lot of stuff with AI because it costs money and it's hot.
James Barney: Yeah. Um, and so the, early on, the, the question was, well, how do we just even start thinking about... Honestly, I think the, the impetus question was, uh, how do I know what model to use for which workload?
Demetrios: Hmm.
James Barney: Right? I think this was back with, like, ChatGPT 3.5, um, and obviously [00:04:00] that was a while ago.
Demetrios: It only got more complex from
James Barney: there.
James Barney: It only got a little bit more complex. With only just a few couple million more models that you can choose from. Um, so, uh, so really that was, that was what it started with, and what we noticed is there's generally the trade-off between the cost of a model, the speed of the model, and the accuracy of the model.
James Barney: Um, and so balancing these things for your specific workload was, was what the, the working group was trying to put out information about is, like, here's what we see across all of our FinOps member organizations.
Demetrios: Mm.
James Barney: And here is, here's what we found. We don't know. We don't know for certain, but this is what we've noticed and, and here's, you know, the trends, things like that.
James Barney: And so we've, we publish white papers. We, uh, you know, throw together code repositories to de- to demonstrate, you know, how to, how to make a, a token counting and, like, chargeback system. Not as a, not as a way to be sort of the, the end-all, be-all authority, but as a way to start that conversation, [00:05:00] because a lot of people just don't know, and they just need something to read.
Demetrios: So this is a bit of a tangent, but I've challenged a few of my friends, and I've always wanted to see this because of my own workloads. But, uh, and maybe I will challenge you to go and do this. We'll see. I wanna know what the maximum amount of the caveman skill I can use- ... is before the actual output just falls off a cliff.
James Barney: Yeah.
Demetrios: So, like, could somebody benchmark how far I can push the caveman skill? The caveman skill. And there's, like, just the different tasks that you have, and so easy task, maybe. You could do it on a few different metrics, like the complexity of the tasks and then also the amount of back and forth-
James Barney: Hmm ...
Demetrios: that you're getting.
Demetrios: But I know caveman skill is huge for this whole saving money. It's one area, one vector you can use to just, like, save on costs.
James Barney: Right.
Demetrios: [00:06:00] And if you push it too far, then it's kinda crap, and it's especially crap when you need to have more information or you're, like, compacting-
James Barney: Hmm ...
Demetrios: the, uh, the whole conversation, and then you need that.
Demetrios: And so I wanna know what, like, how far can I push it?
James Barney: Yeah, I mean, it, it really depends on the model that you're using, which is, like, I guess the lamest answer. Like, it'd be nice to say, like, it's about that far. Yeah. Right? Like, but unfortunately-
Demetrios: Yeah. No, I'm sure, like, you could do it with a bunch of different models- It's-
Demetrios: and then you'll see, oh, this one, you can push it to this far- Right ... or that far, at least for now. But just as you have those benchmarks that'll come out when folks launch their models- Mm-hmm ... and they say, "I, we got this high on SWE-bench."
James Barney: Yeah.
Demetrios: It's like, I wanna see what's Cavebench. What's
James Barney: Cavebench? Yeah.
James Barney: Yeah, it, it's, it's tricky to know, but I think it, it does strike at the, the core sort of the whole point of, [00:07:00] like, the FinOps Foundation and, like, what the, the Tokenomics Foundation is, is striving to do, which is trying to get the most value out of the, the tool- Mm ... of the moment, the LLM, the generative AI capabilities, with, uh, you know, just maximizing that ratio,
Demetrios: right?
James Barney: Mm-hmm. Um, and so yeah, caveman, why use many word when few do trick, right? Mm-hmm. Like, that's the, that's the, like, bread and butter of the whole thing, right? If I can get a really good response out of just, like, a, a few input tokens, or if I can get a really good response and it's just a really small output, you're saving money on both sides.
James Barney: Yeah. Right? And so as these models sort of mature, and ironically, right, Anthropic just released a new model literally today, um, the, the new Fable model, um-
Demetrios: But they said, uh, they said something weird like, "Oh, you're gonna have this price for... Or you're gonna have this [00:08:00] model at this price until the end of June."
Demetrios: Did you see that?
James Barney: I did. Yeah. It's like they're, uh, I think they're probably... I think they got burned a couple times when they released a new model and started charging people for it right away, and it, you know, the usage spiked, and the, the cluster that's running this model couldn't keep up with demand.
Demetrios: Oh, so you think- And like- ... they're doing it higher.
James Barney: I, I think they're, I think they released it with, uh, with a sort of- acceptable price or like free sort of, you know, kick the wheels pricing-
Demetrios: Mm-hmm ...
James Barney: uh, with, with the anticipation of like it may not be perfect when it launches. It may have some issues or some weird regressions- Mm-hmm
James Barney: that will work out for the next, what is it, 14 days or something like that. Um, and as they learn about that, they're... I, I think they're trying to get, you know, developer sentiment kind of back aligned with what it was- Mm-hmm ... which was, you know, much more positively thinking about the, the company- Their pricing
James Barney: and their [00:09:00] models. Yeah. Yeah. And, and like, you know, when your system goes down for half a day, people start thinking like, "Well, what am I doing?" Maybe
Demetrios: we should-
James Barney: Maybe we should have- ... look
Demetrios: at Codex ...
James Barney: maybe we should have other back, other, other things. And it- Yeah ...
Demetrios: I
James Barney: think it's a natural thing. Um, it's sort of...
James Barney: It feels like to me a test drive, right? Mm-hmm. They're s- they're letting you take the model for a test drive, see what it can do, and then, you know, ideally it proves itself and you're like, "Yeah, that's worth it." And, and I think the business value of me using Fable would be, you know, is worth the squeeze.
James Barney: I think that's what they're going for.
Demetrios: And how do you plan for these kind of like wrenches that get thrown into the mix as you're enabling the developers across the whole company, and then you get something like that where it's like, okay, cool, we can't trust this pricing at all. Uh, so we can use it, we can try and build something on it, but at the end of the month, we're gonna have to reassess.
James Barney: Well, it's not so much that we don't trust the pricing. It's just that the, the [00:10:00] consumption is difficult to, to sort of tie down.
Demetrios: Mm-hmm.
James Barney: Right? So like, I mean, at the end of the day, if you burn a million tokens, apparently it'll be $50, right? And that's, that's pretty easy to understand. The, like, the problem is, is understanding what those million tokens did.
James Barney: And so when the wrench gets thrown and, you know, 10 minutes after the model release we have developers asking for it, it's like, "Yeah, guys, like we just learned about it too." Um, you know, we're, we're working on partnering with these model providers to, to give us Heads up or like early access, stuff like that.
James Barney: But at the same time, they're, they're doing the same thing, right? A lot of times the models that they're building are cutting edge and, and they've been using them just for a couple months themselves. So like, I think it's just kind of this new strange world where, um, the release of models and the new capabilities will, will probably just sort of be something you just learn to roll with the punches with.
Demetrios: Mm-hmm.
James Barney: Um, I think they'll continue to leapfrog [00:11:00] each other in capabilities here for the next year or so, and then I think it'll probably taper off.
Demetrios: Mm.
James Barney: And you can, you know, timestamp this right now, and then a year from now when I'm completely- It's gonna- ... wrong, you can, uh, you know, I'll point and laugh.
James Barney: It's fine.
Demetrios: Who is that guy? You see that Substack of the guy who just goes and fact checks people's, like, claims?
James Barney: Yeah.
Demetrios: There's a whole, uh, there's a whole newsletter on it. I can't remember the name, but incredible Substack because he'll just be like, "You were wrong. You were wrong. Look-" I mean- "... you said that this was gonna happen on this day 10 years ago.
Demetrios: No, still not happening."
James Barney: Yeah. I, I mean, I... Yeah, how could you not know the future? It's just-
Demetrios: Yeah. ...
James Barney: very shortsighted of them
Demetrios: to not know
James Barney: the future.
Demetrios: Exactly. Jeez. Um- Uh, so how are you thinking about then the value that the folks in the company are getting with AI? Like, what are the metrics that you're feeling very confident about recognizing and, and [00:12:00] viewing as strong signals that there's value being produced here?
James Barney: We look at a couple things. One, and it's the most easy to, to sort of measure, is are we avoiding costs somehow, right? Like, did we Are, are we able to somehow, through the use of AI, not spend money somewhere else? Mm-hmm. Um, and that can, that can look like a, a lot of different things. Um, so, you know, maybe we are internalizing a tool.
James Barney: Maybe we've rebuilt a, a capability or, or, you know, you name it. Um, that's one way that we looked at it. The, the second way we look at it is can we, can we drive business outcomes faster? And, like, that's, like, the most corporate, uh, response ever. But, like, it's so broad, right? It would be way really easy if we were just a widget factory- Mm-hmm
James Barney: and we could look at the pile of widgets on the other side of the factory and say, like, "Yep, we made more of them."
Demetrios: Mm-hmm.
James Barney: Um, but unfortunately, we don't just make widgets. Um, and so that's the, that's sort of the [00:13:00] driving conversation that we have with our various business stakeholders is, like, what is your widget, right?
James Barney: What is your unit metric that you care about? And ironically, normally, when you do have this conversation and you find one unit metric or thing that really, you know, aligns to revenue or a KPI or something like that, it'll satisfy one stakeholder, but then the next stakeholder in the next conversation will say, "Well, that's not the right one."
Demetrios: Hmm. "
James Barney: And what I actually care about is this." But what's handy is that once you find at least one unit metric and you start sort of advertising that as the, the source of- The North Star ... the North Star, people are all like... They'll get upset, and they'll be like, "No, that's not what I care about." And then you're like, "Ha ha, excellent, I've got you now, so this is what you care about."
James Barney: And you just kinda go through and collect all the unit metrics. And you start... You know, it, it very much aligns to the, the core principle of, of FinOps, which is understanding, like, what is that unit of value that the business really cares about. Um, and then you're able to, uh, eke it out. 'Cause [00:14:00] it is a different way of thinking, right?
James Barney: Like, if you're thinking quarterly, it doesn't necessarily align to an individual thing that happens 1,000 times a day, right?
Demetrios: Mm-hmm.
James Barney: Um, and so I think those are the two, two main ways we're looking at value. Very
Demetrios: broad. So you get this y- this key North Star metric, and all different kinds of folks have their different metrics.
Demetrios: Then how do you tie that back to the AI? Yeah. Is the idea of however much or however they're changing that metric? Like, I didn't quite make the connection.
James Barney: Yeah. All right. So, like, if a process consumes $10 worth of tokens, and this is just for AI, right? If a process just consumes $10 worth of tokens, then you can say that you, you can go to your business person and say, "Hey, to get one of these outcomes, it cost us $10.[00:15:00]
James Barney: Do you like that or not?" Right? And they'll say, "Yeah, that's great. Keep doing that." And it's like, "Okay, perfect. Conversation over." Or they can say, "Mm, that's kind of expensive. Like, can we do it for $9?" Mm-hmm. "'Cause, like, $9, that would l- that works a lot better with our current, like, revenue estimates." Um, and then the conversation becomes a more classic sort of tech optimization conversation, as opposed to, like, this weird ethereal AI conversation.
James Barney: Mm-hmm. Where you can go from, "Okay, we need to save $1 per unit of work, what are our options? We can do a cheaper model. We can do various, like, prompt efficiency techniques. Caveman. Caveman. We can do, uh, you know... But, but with all this comes different trade-offs, right? Where, like, if you have a, a cheaper model, it might be slower.
Demetrios: Mm-hmm.
James Barney: You might have to run [00:16:00] it twice to get, you know, the sort of, uh, certainty that you need about that output, right? Which, in effect, doubles your cost, right? So it becomes this sort of, uh, balancing act optimization problem where, um, there's a lot of different knobs and levers that can then be pulled, but at least you have that metric that you can track- Mm-hmm
James Barney: with a goal, right? I think once you have a goal in mind, you're able to really drive towards achieving it and testing whether or not, you know, a change actually pushed you closer to that goal or moved you further away.
Demetrios: And does this also work when you're in this exploratory nature of trying to create something?
Demetrios: Because I feel like a lot of times when I'm first building something, maybe it's going to be exploration, greenfield type of stuff. But after I've done that, then I'm much more confident in, all right, I've, I've built it. Maybe it wasn't the most [00:17:00] optimized way to build it, but now that I've done it, I can look back and I can say, "Yeah, it was a little bloated, but-"
James Barney: Yeah.
James Barney: I mean, that's the classic FinOps sort of story, and that's what FinOps people do, right? We, we optimize existing workloads to meet the demand for, for what needs to be done. Um, honestly, it's kind of like what humans do all the time- Mm ... anyways, right? We are very good at optimizing systems. Um, and so yeah, I think generally the, like my, my approach is use the smartest model I can for the initial work, and then if it, you know, knocks it out of the park, the question is, A, do I need to do this task again?
Demetrios: Mm.
James Barney: And if I do, go down one level of a model, right? If it's a 300 billion parameter model, maybe I go down to a 70, right? Just using like the standard open source model counts. Yeah. Or not parameter counts. Right? And like maybe instead of, um, instead of leveraging the 300 [00:18:00] billion parameter model, if I can get the same output from that 70 and, uh, just sprinkle in a little bit more of a, like a harness, agentic harness capability, hey, that's not bad, right?
James Barney: And, and it's that optimization that you just sort of slowly chip away at, um, that ironically AI is really good at sort of doing for you, which is kind of weird because traditionally the cloud was not so good at telling you how to improve itself. Mm-hmm. Right? Like, if you have a Lambda function running somewhere, it's not gonna w- without help from the cloud prevention, or cloud provider, they're not gonna say, "Hey, this serverless function that you've got running needs to be tuned a bit," right?
James Barney: That's a, that's a, that's a thing that they provide you, whereas with AI, you can literally just ask the same model, "Hey, how do I make this work a little bit better?" And, you know, it'll say, it'll take a look at it and say, "Oh, you can jump here or move there or whatever," and
Demetrios: usually it works. [00:19:00] That is a great point, 'cause you don't get that with traditional FinOps.
James Barney: Mm-mm. It's like this weird self-bootstrapping capability. Yeah. Like, once you have AI plugged into whatever that capability is, you can just keep asking it about that capability and, like, where, where certain inefficiencies you've noticed or where inefficiencies perhaps it's noticed. Mm-hmm. Um, and you can really, like, accelerate really fast.
James Barney: You can go from zero to 100 like that, um, in terms of the capabilities that you were originally going for. And then once, once you're kind of elevated, you the human, out of the sort of toil of that process, you can actually start thinking about what, what is the problem that we're actually trying to solve, and is the process that I've put in place originally, is that actually solving that problem- Hmm
James Barney: or is there, like, a different angle that I need to come at it from? And what's nice is that with, obviously, AI, you can just have it, like, iterate rapidly and say, "Okay, [00:20:00] let's try this new angle." And it might not work, right? But at least, at least you can spend the time, or you don't have to spend the time, rather, um, to, to even explore it, right?
James Barney: Yeah. You just kinda like, "Hey, go, go try it. Come back. Run it, run, run a model on that for 1,000, 1,000 invocations and see if it actually pans out."
Demetrios: Well, yeah, that's kind of what my questioning and the reasoning was on the last one with that's, like, exploratory nature in a way, and maybe you have five explorations before you land on one.
Demetrios: Mm-hmm. Do those five still count as this, "Hey, we need to do it for $10"? Th- this, that's part of the $10- Mm ... when you're going to the CFO or whoever to say, "This costs fi- $10," even though it's like at the base it was a lot of exploration, and then you chose one and you went vertical with it, and you, you shipped that one version, but [00:21:00] as part of that whole package.
James Barney: Yeah. I, I think most I mean, I don't-- I would assume that most CFOs wouldn't necessarily classify the exploratory work as, like, a production workload, right? Like-
Demetrios: So it's different. It's like R&D- Yeah ... versus actual production. Yeah.
James Barney: Yeah. That, that's the way I look at it. I'm sure if you get a CFO in here, they'd say something-
Demetrios: Yeah
James Barney: like, totally different.
Demetrios: But, well, I guess where it starts to get tricky is how can you classify those tokens as these are R&D tokens. This whole exploration that I did-
James Barney: Hmm ...
Demetrios: that shouldn't be counted towards the total cost, and you're going back and you're counting what was part of the final build versus what was the exploration, right?
Demetrios: Yeah. You know what I mean? Like- Yeah,
James Barney: yeah. I- That- The way that I think about it is that's just personal productivity, right? Like, obviously it's not directly what we classically [00:22:00] think of as- Mm-hmm ... productivity. But, like, if I'm able to personally iterate and build a thing locally or, like, experiment with a model, you know, on my machine to somehow...
James Barney: Not, not-- The inference is remote. But, like- Yeah ... if, if I'm able to build the thing on my machine and explore it that way, that's one thing, right? It's kind of like building software locally on your laptop. You can run the NPM test command and, like, validate that the new feature works, right? But when you actually push it to production, there's a whole pipeline process sort of difference between the thing you know, it runs on my computer is totally different than it runs on any computer.
Demetrios: Mm-hmm.
James Barney: Um, and so I think that's sort of the same framework that I apply it with. It's... Or, or, uh, that I think about it with, is there's this, this sort of monthly budget, right, um, that can be applied to each developer as they're exploring, [00:23:00] and then that is specifically separate from the actual workloads that you know are actually running and doing things, like, specifically, like, task-oriented, right?
James Barney: Huh. Um, so I think the, the R&D model does kind of work really well there. It's just difficult. I think they've started getting really close to each other.
Demetrios: Yeah.
James Barney: And so that's the, that's the trouble.
Demetrios: Y- yeah, especially as you've, you're kicking off, like, 10 different cloud agents-
James Barney: Hmm ...
Demetrios: trying to build different things.
Demetrios: And like you were saying, like, "Hey, can we run 1,000 simulations on this to try and find the best way- Yeah ... to go about it?" But, uh, it could be one way of looking at it is, like you said, once you merge it to main, then you know this is the production workload. We're gonna go and do that However, I know that, like, some teams, it's just constant, like, merging to main, and some of them are exploration [00:24:00] still, but then they'll go and a new PR gets merged so fast over top of it that it's like you can't really separate the two.
Demetrios: Like you're saying, like it's so unclear how the agents are, are working and which ones are working in what capacity.
James Barney: Yeah. The experiment becomes the product.
Demetrios: Yeah.
James Barney: It's like, wait, wait, what? Yeah. Which, I mean, honestly, every product is an experiment. Mm-hmm. Or at least should be f- I would, I would argue, should be drafted as an experiment.
James Barney: Mm. Like, will this product actually succeed in this market? Yes or no? Mm. And then if it succeeds, then you know it did.
Demetrios: Yeah.
James Barney: But obviously, you have to be able to measure, like, what it was about that product that made it succeed, and that's where, again, I think the, the, the unit economic or the unit metric for that particular product really comes into play, right?
James Barney: How is it actually moving your company's bottom line? And, and that's where I think opening up the door [00:25:00] for all these devs to start exploring and building and iterating while still having the, the, the traditional software development life cycle guardrails in place, even for agents, right, um, is super critical because yeah, you can...
James Barney: Not, not every experiment is a really good idea,
Demetrios: right? Mm-hmm. Like,
James Barney: um, and, and so there is still this higher level, I would say, product thinking that is required in order to make sure that it aligns to your overall strategy, um, while also, uh, being a good investment, right? Because there can be big investments that just don't align at all to your strategy.
Demetrios: Well, and there's a lot of things that I do with my agents, especially, like, the personal Hermes agents-
James Barney: Mm ...
Demetrios: that I would never do as a human, but since it's possible with an agent, I'm like, "Oh, sweet. Of course I wanna know every single, like, Reddit, subreddit that I follow and what's popular there, and Hacker News, like, summaries of the [00:26:00] day, and all of that in my daily briefing," you know?
Demetrios: Yeah. Like, so it goes and it scours the internet, and then gives me this briefing, and maybe every once in a while I'll do that. Or who am I kidding? Like, I do that when I'm trying to take a break from, quote unquote, "work." But there's a lot of, I guess, uh, things that I do with the agents that I wouldn't do as a human.
Demetrios: Mm. But since I have the possibility to do them, why not?
James Barney: Yeah. I mean, that's what complicated organizations are all about anyways, right? It's, you know, you can have a, a business that has all these different departments that all do things that one person would not be able to do or not even have the mental faculties to do-
Demetrios: Mm
James Barney: in the first place, but that's the whole point, right? The, the, the point of delegation is to get specificity towards the problem that you are all trying to solve, right? And so the... I, I think basically what you're [00:27:00] describing is the, you know, sort of kind of organizational theory, right, where the, the complexity of the problems you're able to investigate sort of- You know, expands because you have the capability to investigate that.
James Barney: Yeah. And it's like, that's cool. That's kind of, that's naturally human. That's like what we do. Put us in a dark room, we'll start feeling around, seeing what's going on, and that's just what we do, right? Um, and so I, I hear that a lot from people. It's like I've, I've, I've never even thought about doing X, Y, or Z thing, but now, now I have my Mac Mini sitting on my desk- Yeah
James Barney: and so of course I'm gonna have a do it. Yeah. Right? Like, why, why wouldn't you?
Demetrios: Yeah.
James Barney: Um-
Demetrios: I, I guess the question then becomes How do you look at that from a governance perspective? Mm-hmm. Because I can make the claim that this is very valuable for our North Star metric, but is it really like moving the needle, and how much is it spending?
Demetrios: And so like I'm just trying to [00:28:00] have these frameworks in my mind as far as the cost-benefit analysis or that triangle that you mapped out for, from FinOps and the, and the spend, match that up with the metrics. But then I'm like, man, there's all these really potentially costly personal productivity agents- Mm.
Demetrios: that I can let loose and have them doing things just because I, they can. But like how do you bring that back into like, how do you put the reins on it, or how do you, how do you even know if it's worth doing?
James Barney: Yeah. I mean, I, I think that's the... It's really th- there isn't one single way to answer it, right?
James Barney: There's layers to it. Um, just like how when, you know, the personal computer first landed on somebody's desk, the, the hope was they would use that to just crunch numbers all day, right? Mm-hmm. And then what do you know? They invented the browser, and [00:29:00] like everything just went up in flames, right? Um, but it, it, I think it's the, kinda getting back to the, the previous question, which was like I never would've even thought about collecting all this data and understanding what my favorite subreddit is talking about today, um, sort of is a reflection of what we can do with, with a governance perspective as well, right?
James Barney: Like the, the, it, it means different things to be able to, um, to, to demonstrate that an agent is able to, uh, you know, bring extra value to a company. It, it's another thing to ask whether or not it should be the one bringing that value to the company. Um, and so it, it really depends, I think, on each individual sort of risk tolerance, and I mean like individual company or individual person, what they're comfortable doing.
James Barney: Um, you know, in highly regulated spaces, we oft- you know, luckily we have laws that kinda tell us where, where we have to [00:30:00] be. Um, and, and that's sort of where we start with, with the governance aspect. And then from there, there's, uh, again, getting back to delegation, there's a lot of different stakeholders in play that make sure that the, the use of AI is something that, uh, that should be done.
James Barney: The, the, you know, the type of data being sent to the model, it should be sent to the model. Um, the type of You know, the s- type of server that we're using should be the correct server, right? Mm-hmm. Like, the, it just kinda, the more you use it, the more questions you know that there are to ask and answer, and it just sort of snowballs from there.
Demetrios: Yeah, 'cause I think about governance in different ways, and I'm sure there's so many different layers to governance, and it's almost like peeling back an onion- Hmm ... as you're saying. Like, I know there's the data residency governance or just, like, policies that you have to keep in place. And I also was talking [00:31:00] to some folks about how Their platform will enable the certain policies that company has with spinning up a GPU cluster- Mm
Demetrios: or spinning up a Kubernetes cluster, so that it's done the way that that company wants it to do, instead of just hoping that a skill will be able to be as effective. Be nailed the first time. Yeah.
James Barney: Yeah.
Demetrios: Which, you know, it's, it's great. Usually it works, but if it's something as delicate as this, like, need to get it right every time, you can't always trust a skill.
Demetrios: Right. And, and so you wanna have it be a little bit more deterministic. So how you put those policies and that governance in place, but again, I, I imagine there's a bunch of different layers of governance that you're seeing every day, and love for you to, like, talk me through some of those.
James Barney: Well, I, it, it really rhymes with a lot of the existing stuff [00:32:00] that we had in the cloud, right?
James Barney: There's the proactive prevention, right? Like, for AWS, for example, don't make a public bucket.
Demetrios: Mm-hmm. Right?
James Barney: So how do I con- how can I proactively prevent people from making a public bu- bucket? Mm. If a person can't make a public bu- bucket, an agent can't either, right?
Demetrios: Mm-hmm.
James Barney: And so, or at least it shouldn't be.
James Barney: Like, I'm not gonna say that. Anyways, uh It
Demetrios: can't, yeah, and then you never know.
James Barney: That's on the cloud provider. Yeah. That's not me, right? Like, uh, so if, if a person can't make a, a public bucket, then the agent can't. Um, and then there's reactive, right, where if something does not align to your specific policies, how do you then automatically react to it such that it can sort of become in line with your policies, right?
James Barney: Maybe it's like a, to your point about Kubernetes, like a, a pod configuration is out of, out of spec. I don't know. Mm-hmm. Right? There's so many different ways to configure Kubernetes. Um, [00:33:00] and like, I, I think defining those things in code, right, policy as code is really important, so that way, you know, 'cause the agent's not gonna be able to call up...
James Barney: Well, I guess technically the agent could call up your InfoSec partner and ask, "Hey, how am I supposed to configure this Kubernetes cluster?" But the InfoSec partner probably has better things to be doing at that time- Yeah ... and probably shouldn't be talking to your agent anyways. Um, but, like, if it's policy as code, the agent can just simply read the code- Mm-hmm
James Barney: and say, "Oh, okay, this is why either, A, I'm being blocked by the proactive defenses, or B, why I'm being flagged by the reactive defenses- Mm ... in terms of, like, whatever the heck it is I just pro- provisioned." Um, so I, I, I think- If you, if you remove the word agent from all this, AI agent, get rid of it, um, the same rule book sort of applies for cloud provisioning as it did for, you know, letting humans pro- provision stuff.
Demetrios: Mm-hmm.
James Barney: You wouldn't let a human just go into your production workload [00:34:00] and start spinning up Kubernetes clusters and things like that, right? Burning
Demetrios: GPUs.
James Barney: Burning GPUs. Yeah. If you, if you wouldn't let a human just go in and do it, then you certainly shouldn't let an, an AI agent go in and do it. Yeah.
James Barney: There should be, you know, governance and, like, SDLC. It's all software at the end of the day now. Um, I guess unless you're plugging in a GPU somewhere, that's, that's, that's hardware. But beyond that, right, it's all software at the end of the day. Yeah. And so, um, if, if it can be described in a, in a document, like literally a text document somewhere or a configuration file somewhere, run it through a pipeline, right?
James Barney: Mm. Like, don't do it manually, and if you're, uh, an AI provider and you are forcing people to do things manually, stop it.
Demetrios: Yeah.
James Barney: Right? Like, give us the APIs to be able to control that, so that way we can properly govern these big capabilities now without having to, to manually click around.
Demetrios: Mm-hmm. Right?
James Barney: Like, let's [00:35:00] not go down the path of click ops like we did with the cloud, 'cause getting out of that was ugh.
Demetrios: Painful.
James Barney: Painful. Um-
Demetrios: Man, even I used GitHub the other day, and I was like, "Why would anybody go on this UI and expect to do anything?" It's so hard. I had to go and ask the LLM how to do what I'm doing, and I ended up just being like, "Ah, you know what?
Demetrios: Like, let's just ask Cloud Code to do it for me." Right. Because clicking around is not fun.
James Barney: That's the... Yeah, and that's the weirdest thing, I think, that strikes me about this current sort of wave of, of AI is it seems like the, the sort of modern UI that we're, that we're really used to as humans is kind of, it's feeling more and more obsolete to me, right?
Demetrios: Yeah.
James Barney: Like That-- And, and that's what I'm kind of reconciling internally, like with myself, is like, why do I need a, a dashboard that tells me [00:36:00] the top 10 token-consuming applications of the day when I can just write a command line tool, give it to my LLM, and ask it what the top 10 things are? And then I don't need a web server that hosts this UI anymore.
James Barney: I don't have to worry about, you know, cookies in my browser, like logging in- Huh ... and things like that because my identity is on my laptop already.
Demetrios: Yeah.
James Barney: Right? So, like, why bother clicking around on this ancient UI that hasn't had an update in forever? Just give me the, just give me the API, give me a CLI, and I'll give it to my agent, and it'll call it on my behalf.
Demetrios: Yeah. Yeah. Or even potentially just getting stuff brought into your chatbot or whatever, like, LLM provider you're using through MCP apps.
James Barney: Yeah.
Demetrios: And you get to see it right there in the conversation.
James Barney: Right. And it, it's like have the-- It's sort of an explosion of capabilities that, uh, I don't [00:37:00] think a lot of people realize, right?
James Barney: If you have, like, an information security dashboard, and you have a FinOps dashboard, and you have a cloud configuration dashboard for all your different applications and things like that, right? Like, as a human, you have to go to one-
Demetrios: Mm-hmm ...
James Barney: two, three, and then you bring the information back, and you consolidate it all into one sort of mentally digestible thing.
James Barney: But with an MCP app or something like that, you just... You are already at the center of this. You ask the question, and it goes out and finds these different things for you on your behalf.
Demetrios: Yeah.
James Barney: And, like, sort of processes it for you. It cuts... It's so much more efficient, right? Um, and I, I think that's sort of where a lot of these capabilities are gonna go is, like, um, there's gonna be this unified sort of way of interacting with things And that's gonna be where we work in the future.
James Barney: Mm-hmm. It's going to be sort of in this hub, and the questions that we [00:38:00] ask will involve activating different connections between systems, sometimes in ways that we wouldn't have done ourselves, right? Yeah. And, and that's kinda one of the interesting things is, is as we look at a lot of these AI capabilities, um, often the human way of doing it is not the way that the AI would do it, right?
James Barney: Like g- gathering all the data before, you know, in some sort of like mass collection process is a very human way of doing it, whereas an AI might do it on the fly, right? Yeah. It'll go... It'll start the problem with going to system A, it'll chew on that information, and then it'll go to system B and system C when it needs to, as opposed to like a human-driven process would sort of front-load all that activities, right?
James Barney: Yeah. Like I, I think about like filing my taxes, right? A lot of the work that I did was going to all my different bank accounts and downloading, you know, all the forms that they, they, they publish. Mm-hmm. And it's like, "Oh, man." What if instead you just sort of went there on demand for each bank, right, to, to download that file?[00:39:00]
James Barney: And like that's sort of the way that, that I see AI helping humans in the future.
Demetrios: Yeah. I've heard my buddy Simba talk about how the chat interface is the new browser. Hmm. And it very much feels like in line with what you're talking about. We have a chat that we can ask things of, and it will populate the information as needed.
James Barney: Yeah.
Demetrios: And it's the specific information that we look for. It's not a dashboard that has 20 items and maybe we only need three of them.
James Barney: Yeah.
Demetrios: It's like, no, I'm gonna ask directly to that. And back in the day, we had browsers, and we would go and navigate to a place and get on there and look around and try and click around their UI, and- And find out the feature didn't
James Barney: exist
Demetrios: get what we needed. Yeah. Or get that data and then bring it back to, you know, put it in our head, and then maybe go or translate that data or kind of like do something with that data, put it [00:40:00] into another piece of software-
James Barney: Spreadsheet ...
Demetrios: and yeah, a spreadsheet, then manipulate it in some way and then send it off-
James Barney: Yeah
Demetrios: so that it could go onto a PowerPoint, and all of that-
James Barney: Can be- ...
Demetrios: goes out the window
James Barney: It goes completely out the window, right? Like you just go, you say, "Make me a, make me a slide deck that can, that shows the top 10, you know, token users by security vulnerability," right? Mm-hmm. Like pulling that data manually would be absolutely insane-
Demetrios: Yeah
James Barney: in any like big company. And so like being able to sort of- Just do that on the fly is gonna be critical- Yeah ... in, in years going forward. I think the, the trouble that, that I find or that I, I'm predicting that we'll f- we'll run into is, uh, what I call, like, the, the latent shopper problem. You know when you just, like, walk into a store and you don't know specifically what you're looking for- What you want
James Barney: and the person comes up and they ask, "Hey, can I help you find anything?" And you're like, in your head you're like, "Yeah, I'm kinda looking for, like, where the, the dark chocolate [00:41:00] aisle is." Yeah. "But, like, I'm just kind of browsing." Yeah. "I don't know necessarily what I want." Like, I don't know how AI solves that problem because there's a lot of...
James Barney: Like, you and I are describing known problems that we wanna use data that we know about- Mm-hmm ... to be solved. The problem that I'm talking about now is, like, what about the problems that we don't know that we have, that you do get from, like,
Demetrios: that-
James Barney: Looking at that dashboard with 20 different things ... that dashboard and you're like, "What the heck is that spike over there?"
James Barney: Yeah, yeah. Right? Like, if you don't have that dashboard, if you don't see that data- To
Demetrios: inspire new ideas
James Barney: and- Yeah. What do we do? Yeah. And so I think it's gonna be balanced. There's a whole lot of stuff that's, that's gonna be, you know, trading off here and there. Obviously, LLMs can just look at a big old CSV and see the spike in the data- Yeah
James Barney: even though it's just, like, 100,000 rows. But it, it just, it feels different, right? If you don't see the data, you can't necessarily know what questions to ask. Mm-hmm. And I think that will be sort of the tension in the future is, like, how do you know when to put the data that you're interacting with on a [00:42:00] more dedicated- Dashboard.
James Barney: Ugh. Versus just letting the LLM sort of look at it and say, "Hey, here's the top three," you know-
Demetrios: Most important- Most- ... pieces you need to know Yeah,
James Barney: here's your punch list.
Demetrios: Yeah. Yeah, you, you have to be intimate with the data on some level because that's where you gain those insights.
James Barney: Hmm. And only exploring it manually sometimes is, like, the best way to-
Demetrios: Yeah
James Barney: to sort of-
Demetrios: That's
James Barney: it ... do that, right? Like, there's a lot of... I think once we're removed from the toil of that everyday sort of work-
Demetrios: Yeah ...
James Barney: the, the ability for creativity, you know, in solving these problems comes out naturally, and I think that'll be really interesting to see, you know- Yeah ... where we are in a year, what, what it looks like for work actually to be done with these tools- Mm-hmm
James Barney: when we're no longer slogging through spreadsheets in a year. Ha. That's a gift. Yeah, we're slogging- We're slogging through different spreadsheets. Yeah. Um, uh, but, you know, like, how, how, how will that change fundamentally how we [00:43:00] approach problems and how we solve problems in, in, in a business, in society, at the world at large, right?
James Barney: It's gonna be really interesting. Punch list.
Demetrios: Yeah. Yeah. You, you have to be intimate with the data on some level because that's where you gain those insights.
James Barney: Hmm. And only exploring it manually sometimes is, like, the best way to-
Demetrios: Yeah ...
James Barney: to sort of- That's it ... do that, right? Like, there's a lot of... I think once we're removed from the toil of that everyday sort of work-
Demetrios: Yeah
James Barney: the, the ability for creativity, you know, in solving these problems comes out naturally, and I think that'll be really interesting to see, you know- Yeah ... where we are in a year, what, what it looks like for work actually to be done with these tools- Mm-hmm ... when we're no longer slogging through spreadsheets in a year.
James Barney: Ha. That's a gift. Yeah, we're slogging- We're slogging through different spreadsheets. Yeah. Um, uh, but, you know, like, how, how, how will that change fundamentally how we approach problems and how we solve problems in, in, in a business, in society, [00:44:00] at the world at large, right? It's gonna be really interesting.
+ Read More
Watch More
The Challenge with Reproducible ML Builds
Posted Mar 15, 2024 | Views 454
# Reproducibility
# Machine Learning
# FIMIO
Prompt Injection Game - Large Language Model Challenge
Posted Apr 18, 2023 | Views 1.3K
# Large Language Models
# LLM in Production
# Prompt Injection Game