Sign in or Join the community to continue
Please Stop Telling Your AI to “Search Harder”
Posted Oct 30, 2026 | Views 4
Share
Speakers
Erica Hughberg
Community Advocate @ Tetrate
Erica Hughberg is a technical leader and community advocate passionate about helping engineering teams build scalable, secure, and human-centric application platforms. With a background in software engineering and a deep understanding of cloud-native technologies, she specializes in driving the adoption of open-source projects like Envoy Gateway, Istio, and Kubernetes Gateway API, which enable organizations to simplify traffic management, security, and API distribution.
As a maintainer of Envoy AI Gateway, she plays a key role in shaping the future of API infrastructure. She focuses on features to ensure organizations can securely and efficiently integrate AI-powered services while simplifying traffic management, security, and API distribution. In the Envoy community, she drives collaboration, mentorship, and contributions that advance the project and its adoption.
Lastly, as a believer in the power of storytelling, Erica enjoys translating complex technical concepts into engaging, accessible narratives in the form of social media posts, conference talks, podcasts, and educational content.
+ Read More
Demetrios Brinkmann
Chief Happiness Engineer @ MLOps Community
At the moment Demetrios is immersing himself in Machine Learning by interviewing experts from around the world in the weekly MLOps.community meetups. Demetrios is constantly learning and engaging in new activities to get uncomfortable and learn from his mistakes. He tries to bring creativity into every aspect of his life, whether that be analyzing the best paths forward, overcoming obstacles, or building lego houses with his daughter.
+ Read More
SUMMARY
Humans used to give infrastructure a breather. AI agents don’t.
In this episode of Debug Mode, Erika and Demetrios explore what happens when one person’s prompt triggers a burst of searches, API calls, and compute-intensive requests - and why existing services, gateways, and pricing models can struggle to keep up.
+ Read More
CONTENT & TRANSCRIPT
Demetrios: [00:00:00] Like the majority of the visitors on all websites now are agents, AKA bots
Erica Hughberg: one
Demetrios: human can suddenly put a massive load on systems around it. And now the usage is gotten way out of control because I thought Kubernetes was like the be all to end all for scale. So now you're saying that Kubernetes is not enough for the scale that you have if you're a lovable type service
Erica Hughberg: I've spent the past week going back into the world of infrastructure, middleware, all of the things. All of the things that is an inference that is being pushed to their limits because of the continuous rise [00:01:00] in agentic development and deployments, and just humans using agents more and more.
Demetrios: And so these, there's things breaking, and you've seen what's happening.
Demetrios: What exactly has been breaking, and what have you been seeing out there?
Erica Hughberg: So I'll start with the most simple thing that's all breaking. We, you and I actually touched on it a while back, and I think we touched on maybe more in the relationship of databases and other systems were being, like, really hammered by agents.
Erica Hughberg: Mm-hmm. So just forget about MCP for a moment. Forget about, like, handling MCP traffic, but there's a lot of APIs out there in the wild. So we talked even about searching a quite a while back, and how you ask an, just your little favorite agent a question, it can go and fire off, like, tens of, like, search queries to try and assemble an [00:02:00] answer for you.
Demetrios: Yeah. Okay. Fans out. It does that variant, multi-variant thing.
Erica Hughberg: It's, like, faster than a human, right? So they can suddenly, one human can suddenly put a massive load on systems around it. And now what that-- So that is interesting, not just for databases and bigger systems, but also, like, all of these fairly useful APIs that we might have out in the wild, especially free ones, the ones that don't cost money.
Erica Hughberg: And we talked about computer use recently, just like now you don't even have to have a agent actually be calling APIs programmatically. It can literally use your computer and be faster than a human at doing so. Mm-hmm. Okay. So- I happen to have had a lot of my life and my experience in the world of financial services, financial systems, and [00:03:00] a large part of that focused on financial advisors.
Erica Hughberg: And a lot of companies want to give services to financial advisors, and financial advisors like finding out information. Now, imagine you put agents in the hands of financial advisors, right? Which, I mean, they already do. They can already use ChatGPT and Claude, and there are a-- there are actually companies that are, you know, providing agentic solutions for financial advisors for money.
Erica Hughberg: So that exists out in the wild. But these systems that were originally built for handy-- handling portfolio analysis or what, uh, you know, doing predictions based on capital market assumptions, like what if the oil price goes up, it goes down, the property market crashes, it doesn't crash, it goes up, like testing these assumptions and what would happen to your portfolio.
Erica Hughberg: Those types of A fairly compute-intensive, uh, [00:04:00] requests can now get hammered by agents trying to optimize a portfolio.
Demetrios: Oh, yeah. '
Erica Hughberg: Cause it's like, oh, and then I found... Like, you run one assumption, you learn something. A human going through that process is fairly slow. An agent going through that process can be quite fast and just fire off more and more requests.
Erica Hughberg: So th- this, all of these systems, and, uh, this doesn't just apply to financial stuff, right? It applies across-- I will assume that this will apply across many industries, are not pre- prepared for improvising actors that are not predictable, because even though we've had machine users for forever, they were predictable.
Erica Hughberg: Like, they were behaving in a predictable pattern. They would not effectively code the route itself on the fly and fire off API calls. We could learn. [00:05:00] Same with commerce. We could plan for Black Friday. The whole crazy part is that these improvising actors would be everywhere, but why are they different from our old machine users that sent requests?
Erica Hughberg: They weren't throttled by the slowness of just human behavior. They were at least predictable. So, like, imagine Black Friday. That's very human, predictable event. But then imagine end of month, end of quarter, end of year financial reporting. That often would've been carried out by machine users, but they were predictable.
Erica Hughberg: We knew it was gonna happen. We knew they were gonna come and send all of this, and often they would do the same thing every month or every quarter, every year. So predictable, but faster than a human. Now we have these improvising pieces of software that's like, "Ooh, I'm gonna do this thing, and now I'm gonna do this [00:06:00] thing."
Erica Hughberg: And it's never really the same. Sure, we can learn patterns, but they're bursty, and the only thing that is stopping agents from doing things is humans deploying agents to solve problems. So And then as you said, they fan out to all these things. So what happens to systems underneath? And this is the why I wanted to bring up.
Erica Hughberg: This is not about MCP, 'cause I think MCP is actually getting a lot of love when it comes to gateways and protection, et cetera. If you just look at the traditional API gateways and the, just like agents calling existing services, they are now seeing like a massive increase in traffic. So it's not just about the services behind it on the other side.
Erica Hughberg: It is also the middleware and infrastructure in between [00:07:00] that is seeing more high, like spiky traffic. So-
Demetrios: Yeah, and I've heard this too with websites. Like the majority of- Mm ... the visitors on all websites now are agents, AKA bots, that are using websites. And so whether that's just like scraping the website for some reason or another, or that's a, a human asking an agent to go and do that specifically.
Demetrios: I guess it's always gonna be a human asking an
Erica Hughberg: agent to do it
Demetrios: for them. Yeah. Somewhere, like if,
Erica Hughberg: it could be like many, many steps away, right? Yeah. And I think that's the crazy part. It's like I wanna find out, like we talked about trip planning, the amount of time- Yeah ... my mother's doing it, I'm doing it, you doing it.
Erica Hughberg: I mean, I ain't opening all of those like websites. I even don't even do trip planning sometimes. Sometimes I'm just trying to find out how to plan my weekend [00:08:00] locally, like what other local events is going on. So...
Demetrios: Yeah. Yeah, exactly that. I've, I've asked Gemini and ChatGPT when I was, when I'm in places like-
Erica Hughberg: Mm
Demetrios: what music and what kind of like concerts are happening here.
Erica Hughberg: Yeah.
Demetrios: And when they say, "Oh, nothing," I start getting angry at it. Use your, use more agents. Search harder, because I don't believe you that there's no concert is in this big city on this weekend. It's just that you're not looking properly. So.
Erica Hughberg: You're not-
Demetrios: Yeah ... doing a
Erica Hughberg: good enough job.
Demetrios: Yeah. So- Go bombard some APIs, please.
Erica Hughberg: Yeah, which it may not actually just call APIs. It may be using a website that in turn is calling APIs, but we have all of these like- Increase in traffic. So that, why was this fascinating to [00:09:00] me? Because I used to deal a lot with API gateways, and API gateways love doing something, and that was to, when you used a vendor product, it was an API gateway, they love charging you for the number of requests that go through that gateway.
Erica Hughberg: Okay. So when you sign this big-
Demetrios: Yeah.
Erica Hughberg: Yes, so when you sign this big contract with API gateway vendor provider, you have ended up signing... You're trying to estimate how many API calls you're gonna have over, like, the next three years. Okay, so go back three years a time or two years, and people trying to estimate, like, "Oh, based on our previous patterns of, like, API traffic, we think we're gonna sign a contract for X amount of annual API calls," right?
Erica Hughberg: And then agents, and this even would apply
Demetrios: to- Bottoms out somewhere. Yeah.
Erica Hughberg: Yeah.
Demetrios: Rethink
Erica Hughberg: that. And even, and now I'm like even a step, trying to predict it. If you're sitting there trying to [00:10:00] think about planning, you know, a pu- purchase to, for an API gateway service. Now you're like, "Uh, so what am I going to actually estimate for?
Erica Hughberg: How do I know?" Because It's just really changed, not just how they c- It, it changes the business model, I believe, on some of the things that we, especially in the middleware space, we've been buying products
Demetrios: Mm. Because you've been optimizing for usage, and now the usage is gotten way out of control, and it has not been because you now have 10 times more interest in your products.
Demetrios: Yeah. Maybe because you just have the same amount of users that are using agents or not- Yeah ... even the same amount of users. You're just getting picked up more often by these agents that are out there scouring the internet, [00:11:00] and they-
Erica Hughberg: Yeah ...
Demetrios: don't really wanna use your product or weight your product highly.
Demetrios: It's just one of thousands- Many ... of searches that they're doing.
Erica Hughberg: Yeah. Like, your stuff's valuable. It's adding value in some capacity, right? Probably to- Yeah ... to an entire agentic workflow. And now it's, it's just, to me, it's interesting 'cause I've ne- by the way, I've never been a fan of request-based pricing in API gateways.
Erica Hughberg: Oh. So I kind of feel validated in my...
Demetrios: So that's your take, is like- Yeah ... request-based pricing should go out the window when it comes to these gateways, the middleware?
Erica Hughberg: Yeah. Yeah. That's like, I mean, I've been, I've been saying that for a very long time, so 'cause I was very sad about it before as well. But now it's, like, even more so, I feel, because how can...
Erica Hughberg: Your, your cu- [00:12:00] if you are selling these pieces of middleware and your customer, your customer which uses API gateway, if they are in a situation where they can't even predict because they have all of these unpredictable actors in this space now, that is, you could argue, always had unpredictable actors 'cause humans are unpredictable, but this is at a different scale of unpredictability.
Erica Hughberg: So-
Demetrios: Yeah. And-
Erica Hughberg: I-
Demetrios: That's where you're seeing, you're seeing it there and you're- Yeah ... seeing it in other places too, this, like, manifesting? Or is it, have you just been focusing on the gateways because that's your bread and butter?
Erica Hughberg: Um, so that was, like, obviously one part. But then what is quite interesting in a different way, which is the same-ish problem-
Demetrios: Same
Erica Hughberg: but different
Erica Hughberg: but different. Very, I say same but different. So the sameness is that people are spinning up loads of agents that do lots of work. [00:13:00] Mm-hmm. So for a while, so if we go back to, I was about to call it traditional computing, but now Kubernetes isn't that old. So modern cloud computing. When we started, the, there is a similarity and awful dissimilarity between microservices and agents, okay?
Erica Hughberg: So- Yeah ... we, we started out in this-- we moved about 10-ish, maybe more than 10 years ago, we started moving away from big monolithic pieces of software that we had to move around and schedule on hardware to, um Kubernetes 'cause it helped us manage all of that scheduling. So it's, I like to say it's just a workload management platform that helps you manage all of your workloads and scale it up and down based on demand.
Erica Hughberg: Okay. Now, yes, people [00:14:00] have agents now, and they're not necessarily running on everyone's computers. A lot of them are running in the cloud, if you will, on rented compute. So when you start in, now we have to start agent sessions, right? We have to start them, and they need to run. At the end of the day, there needs to be hardware because physics.
Erica Hughberg: But, like, they're not gonna run on their own without anything powering them. So anyway, there's- Mm. So maybe then you'll reach for Kubernetes, and you'd be like, "Okay, I'm just gonna use Kubernetes and use that to schedule my workloads." But Kubernetes- Your
Demetrios: agentic workloads.
Erica Hughberg: Yeah. Agentic workloads. Yeah, like your agents, the actual-
Demetrios: Yeah
Erica Hughberg: code that runs your, like, imagine, like, effectively the, the little loop, the little harness doing things, right? Yeah. So, okay. But [00:15:00] Kubernetes was designed to handle the idea of how many unique services? Okay, thousand. If we count them in the thousands, right, Kubernetes is gonna be okay. But Kubernetes is, has a conf- an API server, just to mess with your head.
Erica Hughberg: The word API is all over in the world of software. But it's got a server that helps configure and manage all of these workloads, right? 'Cause it's a workload management, uh- Mm ... platform, if you will. If you now explode that up to millions, right? So imagine you are offering a solution. So I mean, I'm a big Lovable fan, so let's use Lovable as an example.
Erica Hughberg: So imagine all of these humans going onto Lovable, and they're like, "I'm gonna build an awesome website for my business," or, "I have, like, this crazy app idea," or you're my seven-year-old niece and you're like, "I have a really cool game idea," [00:16:00] right? "And I'm gonna build it." You have all of these people from different walks of life starting these sessions of, and they get their little coding agent that isn't running on their computer.
Erica Hughberg: It's running out and about in the world somewhere else. And this is, uh, th- these amounts of agents is, again, on a different magnitude than microservices in even some of the largest systems that we can imagine. Because what microservices were really, they were the same service but multiple instances of it, right?
Erica Hughberg: Mm. I think that if you went on Lovable and you have your local coding agent and your session with that, and I have mine, they're not the same, right? Similar- But they- ... but not the same.
Demetrios: Yeah. Why, uh Uh, I guess they're not the [00:17:00] same because as it develops, my agent has different context than your agent
Erica Hughberg: You've got different context.
Erica Hughberg: You will have set up your connectors, 'cause say in Lovable, you can set up all of your different connectors, and I set up mine, which are integrated with my authentication and auth- author- authorization to interact with those connectors, right? So we have both, and even I will have multiple different projects to which I might have set up different things.
Erica Hughberg: Mm-hmm. So this-
Demetrios: But
Erica Hughberg: still,
Demetrios: just as a little side note-
Erica Hughberg: Yeah ...
Demetrios: that for anybody that, uh, uses traditional SaaS, you always have integrations with traditional SaaS programs. Yeah. You have the ways that you're going to use it that are different than other people, right?
Erica Hughberg: Yeah.
Demetrios: So it still isn't that far off, but I- Mm
Demetrios: do understand the-- I, I guess I'm [00:18:00] trying to wrap my head around how different is it and where does it differ?
Erica Hughberg: Yeah. Yeah. So we have the difference in that often when you think about a-- I, I spend a lot of time in SaaS development in my life, and tenants is a headache, by the way. It's not that fun. As in, I might, but I brushed it off as sound like it was all very simple and easy.
Erica Hughberg: But no, yeah, SaaS tenancy is tricky, so there are different approaches to SaaS tenancy, so we can do a parallel here. So in SaaS tenancy, we can do complete isolated tenancy, or we can try and run a system that we reuse. So we have single tenancy. It means like you get a full stack of just Demetrios tenants, right?
Erica Hughberg: You are not reusing services. Erika's tenant and Demetrios tenants are completely segregated. We have parallel instances of this platform. [00:19:00] Then to-- That was the first and easy way we did SaaS, right? In the beginnings of SaaS. Then, of course, that got expensive, right? Of course it did. Uh, because if Demetrios, you only go on my SaaS thing twice a week, why should I keep running an entire Demetrios tenant instance all the time- Yeah
Erica Hughberg: and paying AWS or Azure or GCP for just running the software that's just idly there to be ready in an instant when you open your-
Demetrios: Yeah ...
Erica Hughberg: tab in your Chrome, right? So, oh, that got expensive. So then we got more creative with the idea of multi-tenancy, right? So- We can share certain thing, and this is actually again where if we look at microservices where they could become so powerful.
Erica Hughberg: So instead of you having a completely segregated setup, we carry the authorization in the API request, and we know that Demetrios [00:20:00] can access this data. He belongs to the Demetrios organization, and we start architecting and designing our software around the idea of tenancy. So this, so this is again somewhat similar, right?
Erica Hughberg: Because now with context, you can call that context of, uh, w- in which context are we executing this request.
Demetrios: Yeah.
Erica Hughberg: So how can we in the world of agents, right? The, I think actually you make a very good point, by the way. You, this parallel I don't think breaks. It actually is helpful. So this parallel in the world of agents, yes, we have an identity and then we have a session, right?
Erica Hughberg: In the traditional other software, maybe Demetrios has many projects he's working on in the same, like SaaS product. We have multiple product, projects and they carry different stuff. But- [00:21:00] If we, uh, in the world of agents, this other interesting part, it's like a context here can grow quite big. So imagine you've been building this wonderful website, and suddenly you didn't look at it for a month, right?
Erica Hughberg: Now you come back and you sort of wanna load all of this very quickly into a hot agent, right? Mm-hmm. So that it can run, so you don't have to sit and wait so long. And that's an interesting problem. I mean, again, I used to work in fina- financial systems software, like especially for analysis of, we try to preemptively, um, like cache in like data to memory so that you would feel like it was so fast.
Erica Hughberg: Like similarly to if anyone's read about how Google at one point, I think this is probably still true, but [00:22:00] Gmail. So when you put in your username, you were very like, like your email in the login, you were very likely to put in the correct password. Yeah. So in the backend, on the server side, Google will start pulling and caching your emails, not sending it to browser.
Erica Hughberg: But that would mean, because humans type slowly, humans are great at buying time, right? You type in your password, you hit enter. So by the time that you're loading your inbox, server side had already fetched your emails, so that when it renders on your browser, it can be really fast. So those are other traditional like speed architectural approaches to try and preemptively load things in before the person actually clicks it or needs to see it.
Erica Hughberg: But
Demetrios: yeah. Yeah, so this is, uh, interesting to think about now because you've got these parallels. You realize [00:23:00] that with Kubernetes, there's some downfalls in certain places, and those downfalls are-
Erica Hughberg: Ultimately scale ...
Demetrios: it's-
Erica Hughberg: Ultim-
Demetrios: It's the scale of the services, because I thought Kubernetes was like the be all to end all for scale.
Demetrios: That was like what you need, uh, if you are at scale, you need Kubernetes. So-
Erica Hughberg: Yes, you do. Yeah.
Demetrios: So now you're saying that Kubernetes is not enough for the scale that you have if you're a lovable type service and you're just dealing with agents. Every human who's using it is calling a whole ton of agents on the backend, which need to then be spun up and spun down.
Demetrios: So there's like this serverless piece to it.
Erica Hughberg: Yeah.
Demetrios: And so [00:24:00]
Erica Hughberg: But the exciting part- Yeah ... ah, the exciting part is that actually this project that is started by Google called Substrate is actually using Kubernetes underneath, underneath. Oh. But you can imagine it as a- Wait,
Demetrios: what's Sub- what's Substrate?
Erica Hughberg: Substrate is this agent workload orchestration platform that underneath uses Kubernetes as well. So it stands on top of the foundation of Kubernetes, but the actual thing that I understand, by the way, I've just been learning about this in the last week, so I am no expert. But what the people who I've been speaking to is telling me is that because of the management side of managing and configuring all the services in Kubernetes-
Demetrios: Yeah
Erica Hughberg: that's where we have a What do you call it? Like a limit. Like it's not
Demetrios: possible A nightmare [00:25:00] scenario, yeah, if you want to try and configure all the different agentic workloads.
Erica Hughberg: Yeah. So that is why one of the things that needs to be on top is the, um, this extra management plane that is simpler and agent...
Erica Hughberg: When I say simpler, not simpler in terms of its technical, um, implementation underneath, but geared towards managing agentic workloads. So it's not like you're trying to use something that was designed for microservices management, and you're trying to use that to manage agentic workloads. So purpose-built rather than simple is probably a better way to put it.
Erica Hughberg: Uh-huh. So like designed for the purpose rather than you're trying to use, um, use a hammer for a screw. So you, um, you have that. And then the other thing I [00:26:00] was learning about was how it also have this perspective that you can create an agent and also the policies around what it's allowed to do, right? So you package that with the definition of the agent as it gets created.
Erica Hughberg: But I'll see if I can just share the, share the Git- the GitHub. I'm gonna try and click buttons here and not, like, make a Kanye West song at the same time.
Demetrios: It is 808 and heartbeats when you put that bit.
Erica Hughberg: So what is Agent Substrate? Agent Sub- I'm gonna read it out loud just to make sure. So Agent Substrate delivers a performant, high-density runtime environment for large-scale agent deployments.
Erica Hughberg: Ooh, so many buzzwords in one sentence. The Agent Substrate [00:27:00] control plane provides full lifecycle management for agent sandboxes, delivering sub-second agent resume/suspend operations, and allows heavy multiplexing of agents onto the same computer infrastructure. It support multiple sandbox technologies, including microVMs and gVisor, enabling consistent lifecycle operations for all sandbox types.
Erica Hughberg: So I think that all sounds all great and big words, but then the next paragraph is actually where it gets into like, well, what, well, what is it really? So at the core- Yeah ... agent substrate maps to a larger set of actors with little quotation marks around it. So- Mm ... applications such as agents onto a smaller set of ready workers, again, little quotation marks around workers.
Erica Hughberg: So relying on the fact that agent-like application tend to idle most of the time to achieve heavy multiplexing. So as far as I understand this idea of workers, [00:28:00] so you have actors and workers, it allows you to reuse the same compute. So you know, like how we talked about a moment ago, like, well, yes-
Demetrios: Multitenancy.
Demetrios: Yeah ...
Erica Hughberg: we are like different, yeah. But aren't we using the same underlying like mechanics? Yeah. Like actually software. So that's how I understand the difference between act- like the ready workers are like the common ground, and you can have your actors and I can have my actors, and we can... That's how I understand it, but I could have misunderstood it.
Erica Hughberg: Who knows? But, um, so it provides the ability to manage the actor's life cycle, and then you assign the actor to workers in real time, rather than having to create workers for every actor
Demetrios: Ah, so it flips it on its head
Erica Hughberg: So actors are not tied to a specific, [00:29:00] like, runtime, I guess is a good way to put it. So actors and workers become, workers become the resource we are using for the actors to actually do the work.
Demetrios: And this helps with these problems that we were talking about where you have someone that uses an agent and then a month later they come back to that same agent and you want to spin up all that context real quickly. And in-
Erica Hughberg: That's my understanding ...
Demetrios: A- and it does that because of the, m- mm, like not needing to be tied to one runtime?
Erica Hughberg: So I also don't think that they've solved all the problems yet, by the way. Right. That's the other part where I
Demetrios: always- It's, it's very much in beta still, right? This is a-
Erica Hughberg: Yeah. Should, should we call it alpha even?
Demetrios: Alpha,
Erica Hughberg: okay. So- I, I don't know what to call it, but it seems very early. So from the, my friends that I spoke to.[00:30:00]
Erica Hughberg: So like for example, they, it doesn't have a... If you use one version of Substrate and a new like release happens, there is like no upgrade guidance, so no like, you know, paved path for doing-
Demetrios: Oh,
Erica Hughberg: that's what I was- ... like upgrading from one to... So it's, it's early, right? You know something's mature when there's like, "No, there's a new release and here's the upgrade guidance."
Erica Hughberg: Right?
Demetrios: Well, it's not like it's so hard to create that anymore considering you can just ha- point an agent at it and then tell it to create it. But maybe,
Erica Hughberg: you
Demetrios: know-
Erica Hughberg: So I don't know exactly. It's a good question, by the way. How much have they solved at this point around like quickly loading things into memory, for example?
Erica Hughberg: Not entirely sure. But I, what I understand from the problem space that it wants to solve is not, um, like making the utilization better of the actual com- compute [00:31:00] and allowing it to be faster so that you can continue these agent sessions without having cold stops. So I'm gonna get more, I'm, I'm, I'm more interested in this project, gonna try and learn more about it.
Erica Hughberg: But I also see what they say- What's
Demetrios: this saying here?
Erica Hughberg: Oh, it's a demo
Erica Hughberg: Here are instant actor teleport. That sounds very sci-fi.
Demetrios: Yeah.
Erica Hughberg: High performance suspend and resume of actors onto any available worker in the pool with sub-second activation. That sounds like what we were talking about, how can you bring back the context? And, and then the state persistence, persistent working memory, and file system state preserved per- path- perfectly across hibernation cycles wheref- I like the word hibernation cycles.
Erica Hughberg: I don't know how that-
Demetrios: That does make you feel like you're dealing with a little polar bear or something, doesn't
Erica Hughberg: it? Little brown [00:32:00] bear maybe. I don't know. Yeah. Little brown bear. Little, they go hibernate for- Hopefully
Demetrios: not a grizzly. Shit.
Erica Hughberg: I don't like- I
Demetrios: don't wanna meet one of them. The polar bears are apparently more dangerous though, just FYI.
Demetrios: I've heard they're-
Erica Hughberg: I'd go and hang out with them.
Demetrios: Yeah. But everybody thinks that they're so cute because of all the Coke commercials, and then turns out they're the most vicious out of all of them.
Erica Hughberg: They're white little Christmassy looking bears.
Demetrios: Yeah.
Erica Hughberg: But other cute animals that hibernate are the hedgehogs, you know?
Erica Hughberg: They're real cute, but they're spiky. So...
Demetrios: Yeah. They will get you. Watch out for them. So basically, these hibernation cycles are when I don't look at my Lovable creation for a month, and then I come back and I'm like, "Oh, what was I doing again? Let's fire it up." And-
Erica Hughberg: Yeah ...
Demetrios: it can instantly pull up that creation.
Demetrios: So basically pull the code from GitHub and show it to me, because I think that's one of the [00:33:00] things that Lovable did really well, especially in the beginning. They were the first ones to show you what you've made and let
Erica Hughberg: you- Yeah ...
Demetrios: play around with it instantly.
Erica Hughberg: I loved it.
Demetrios: Yeah. I-
Erica Hughberg: Okay. That was
Demetrios: not hard, by the way
Erica Hughberg: will have to
Demetrios: say just very on brand for you to love a company that comes from Sweden. I'm guessing you also are very much into Spotify.
Erica Hughberg: I do. I, I have like stopped myself from doing a Spotify analogy in this conversation so far, because I was so close multiple times to talk about Spotify.
Demetrios: Yo. What?
Erica Hughberg: Yeah.
Demetrios: Don't let it... Don't, don't dim your light here, Erica. What do you got for me on the Spotify?
Erica Hughberg: It was about like how Spotify has such fantastic, like- Um, what do you call, like how they... You know how I used the example of Gmail doing your, like, prefetching? Yeah. There's some really cool stuff about Spotify, how it pre-downloads music you probably will listen to next [00:34:00] before you even have- Oh,
Demetrios: nice
Erica Hughberg: done it. So it's like this, how do you anticipate user actions before they've even taken the action? Because humans- That's true ... are quite predictable, by the way. They are not that unique, unfortunately. Uh, so he-- and they d- they've done some really cool stuff about you get, like, clustered with other people and their music tastes.
Erica Hughberg: They can like, oh.
Demetrios: Yeah. You know, I actually wanted to... So I knew that one, and I thought it would be so cool if I could meet other people that were, like, in the same cluster as me. Like, wouldn't it be cool- Make friends ... if... Yeah. Like, who, who else is listening to this random, you know, I listen to some jazz from-
Erica Hughberg: Yeah
Demetrios: what, like, I- I've got a lot of different jazz likes. But then I also will go and I'll listen to [00:35:00] some random country. I'll listen to some random, uh, Brazilian music, and I would love to have-
Erica Hughberg: Yeah ...
Demetrios: I, I just wanna know where is that other person? Where's my other, my twin- What
Erica Hughberg: if we have a- ...
Demetrios: musical likes?
Erica Hughberg: What if you have a, like, Spotify doppelganger?
Demetrios: Yeah. I probably have a few of them. That's what I was thinking, and so that's why I was like, where, uh, when can I meet these folks? Can we have a little hangout session and- Oh ... just see who are you and what do you like to do? What brought you to this weird path of musical taste that is, uh, really it's like a smörgåsbord-
Erica Hughberg: Okay
Erica Hughberg: of species of
Demetrios: Swedish.
Erica Hughberg: Yeah. Oh, great. I, I love you bringing the adopted Swedish words and using them. Yeah. And okay, so have you h- now I'm gonna go on a massive tangent, but do you know about So Far Sounds? [00:36:00]
Demetrios: No.
Erica Hughberg: Okay. So So Far Sounds is a, like, I don't know if you wanna call it community, but basically you can sign up to go to a gig, but you don't know who's playing.
Erica Hughberg: Okay?
Demetrios: It's like a blind date?
Erica Hughberg: Yeah. From Pizzic? But it's also often in, like, odd locations. So it would be like, I went to one in an old church in London, right?
Demetrios: Okay, so-
Erica Hughberg: And you literally just bring-
Demetrios: Oh, I think I've seen some of their videos on YouTube.
Erica Hughberg: Mm. It, and it can be- Yeah ... someone's backyard, it could be someone's living room.
Erica Hughberg: Yeah. Like, it, that's what kind of is that sort of a community. Like, Ed Sheeran did one of those, by the way, but people didn't know they were gonna see Ed Sheeran. So That you sort of can find out like the type of music. But now when you're talking about the Spotify thing, right? What if you can get matched with so far s- but you don't know who's playing.
Erica Hughberg: You just know that it should match [00:37:00] your music profile.
Demetrios: Yeah.
Erica Hughberg: And then you can meet the people who are like you. Yeah. You just go and hang out in a random place and listen to some music. And then you look around and you see that all of them look like you as well. That would be crazy, like.
Demetrios: Everybody's got a mustache.
Demetrios: Yeah.
Erica Hughberg: Wild hair.
Demetrios: Yeah. They got their wild hair and it's like, "Wow, I thought I was such a special snowflake, and coming to this just made me realize, like, I'm not. I, I'm not unique at all. I'm a freaking cog."
Erica Hughberg: But, by the way, I, I just realized though, uh, like n- now I'm gonna do my weird mind jumping, topic jumping thing.
Erica Hughberg: What I did think about, you know, I go about substrate and Kubernetes and all of that. The big question that I had in my mind, like That I was thinking about. So you know how Kubernetes is actually quite old at this point. Yeah. But actually, a lot of people don't need it, [00:38:00] right? A lot of people adopted Kubernetes without needing it.
Erica Hughberg: A lot of people haven't adopted it because they frankly have not needed it. But the big, like, if you think like GCP or, like, if you use, Cloud Run on GCP, you are not using Kubernetes yourself.
Demetrios: But it
Erica Hughberg: is under the- But GCP is using Kubernetes, right? And so from what I was thinking about is how many people are actually going to have to use Substrate for their agents?
Erica Hughberg: Are they either going to not have agents at that scale, or if they have agents that they wanna do this with, are they simply gonna use an abstraction service on top of it and it's more like a, you know, hyperscaler type technology and they're gonna worry about it?
Demetrios: You know what my take is? I've been thinking about that this week actually, funny enough that you say it.
Demetrios: I'm not sure [00:39:00] a lot of use cases for agents need cloud agents. Yeah. I think that us in the bubble of, like, coding agents-
Erica Hughberg: Yeah ...
Demetrios: we believe that, oh, all agents need to have this, like, scalability and they need to be able to run in the cloud and all that. But when you go back to the case of your mom using AI-
Erica Hughberg: Yeah
Demetrios: it is not like she's there like, "Well, I need to be able to have 10 different agents trigger throughout the day, whether I'm at my computer or not," or, "I need to expose this agent to my 20 best friends so that they can also leverage it, and then they can consume it, because I built this agent and now they get to consume it."
Demetrios: Like, that's not the majority of use cases. It's really just us going to a screen and then putting in a prompt and letting the agent try and go do it, and quote unquote agent here because [00:40:00] sometimes it's just a response on a chat window. Yeah. And-
Erica Hughberg: I like ...
Demetrios: so that's my, that's my take. Like-
Erica Hughberg: I agree ... cloud agents,
Demetrios: nah
Erica Hughberg: But again, with my mom using, say, Gemini, oh, my mom, I talk so much about you.
Erica Hughberg: But you know, she used her like little Gemini app on her phone, right? Mm-hmm. That from a hyperscaler point of view, I would guess, now I'm guessing, I have no facts on this, but imagine there's lots of retired people doing, planning for their now free time, uh, with their Gemini agents, assuming my mother is not alone.
Erica Hughberg: Like, then would Google have to use something like Substrate to like manage all of these things? But like my mom doesn't, right? Yeah. She
Demetrios: doesn't
Erica Hughberg: have to worry about it. But-
Demetrios: I see what you're saying. So us being the people that are [00:41:00] like the end users, we don't need to worry about it. But the people that are exposing that service, which is right now a very few amount of companies- Yeah
Demetrios: that really have, A, the scale, and B, the offering for that, right? So it's like, okay, you've got Google Anthropic and OpenAI as the main three, and then really- Yeah ...
Erica Hughberg: there's
Demetrios: a handful of others.
Erica Hughberg: You've got Lovable doing Lates, like having lots of agents. Then you have, I cannot remember all of the names of them, but like there are Lovable-like solutions at similar companies- Yeah
Erica Hughberg: that do, I feel so bad 'cause I know the logo. It's like an orange logo, another company, and I cannot remember the name of it. But-
Demetrios: I think there's one called like Bolt, and then there's another
Erica Hughberg: one- And you got Replit ...
Demetrios: Base. Yeah, Replit, Base44 I think is one, and then- Yes ... Procode's another one. And there's, there's a lot of them out there.
Demetrios: And it's true, maybe they need [00:42:00] to be using this, but how many other folks are really having that many agents that need to be cloud agents and spun up if they're not- Like at that scale ... if we're not talking about like their engineering teams.
Erica Hughberg: Yeah. Like even their engineers, could these engineering teams effectively just gonna have to leverage wrapped solutions from the cloud providers to power their products?
Erica Hughberg: So like going back to my life of like financial advisory software, there is a fixed number of registered financial advisors in the US, right? You ca- even if you put your financial advisory agent in the hands of every registered financial advisor in all of the United States, there is a limit. So like- Yeah
Erica Hughberg: the scale again is fundamentally different from consumer, like my mom [00:43:00] on Gemini. Like- Mm-hmm There is way more instances of my mother like, like individuals in the world than there are financial advisors in the United States, so
Demetrios: Again, this episode is called How to Not Feel Like a Special Snowflake. Yeah.
Demetrios: You're really just a sheep in the herd. And I think that it, that is, it is a great point though. Who's gonna need it? Who's gonna be using Substrate? Is it gonna be something that people think they need, but really in reality, push comes to shove and they don't need? We'll see how much complexity
Erica Hughberg: it
Demetrios: adds.
Demetrios: Yeah. Because if it is something where you're like, "Oh, cool, like I can just add this and make it so that those two people at my company that really are pushing the boundaries when it comes to agents, they have an easier way of interacting with it," then great if it's like a no-brainer and it's [00:44:00] very easy user experience, or as some people like to call it these days, agent experience of setting it up.
Demetrios: But if it turns out to be like a whole lot of work and-
Erica Hughberg: And it's... And you might end up using it, but you don't know you're using it 'cause it's a implementation detail in a product you're using, right?
Demetrios: Yeah, exactly.
Erica Hughberg: As many very clever, very well-made, you know, fantastic technology solutions are for, even for engineers, they become implementation details in a solution they're using, so.
Demetrios: Yeah. It, it kind of is the whole like, are we gonna need sandboxes argument? And I've talked to so many people and it's like, of course we need sandboxes. Every agent should have sandboxes, or you should have sandboxes that you can run your agents in, blah, blah, blah. It's like, yeah, all right, but is it [00:45:00] going to be the dominant, like is it gonna proliferate beyond us nerds?
Demetrios: Mm. And so we'll see. That, that would kind of give you a better idea of if Substrate is going to have any legs.
Erica Hughberg: And then also, I just like remembered something. Next time we speak, I think we need to assess if AGI, if we are in the AGI era, era, because I believe yesterday Sam Altman has now declared we're in the AGI era.
Demetrios: Well-
Erica Hughberg: So
Demetrios: Oh, God. Ugh. So
Erica Hughberg: that will be-
Demetrios: Well, yes, I wanna talk to you about that next, and let's leave it- Yeah ... let's leave it at that. We'll see you next time.
+ Read More
Watch More
How to Systematically Test and Evaluate Your LLMs Apps
Posted Oct 18, 2024 | Views 15.3K
# LLMs
# Engineering best practices
# Comet ML
Using LLMs to Power Consumer Search at Scale
Posted Jul 21, 2023 | Views 930
# LLM in Production
# Power Consumer Search
# Perplexity AI
Using LLMs to Punch Above your Weight!
Posted Apr 27, 2023 | Views 1.3K
# LLM
# LLM in Production
# Anzen
# Rungalileo.io
# Snorkel.ai
# Wandb.ai
# Tecton.ai
# Petuum.com
# Mckinsey.com/quantumblack
# Wallaroo.ai
# Union.ai
# Redis.com
# Alphasignal.ai
# Bigbraindaily.com
# Turningpost.com