Sign in or Join the community to continue
Could Your Android TV Box Run AI Malware?
Posted Nov 13, 2026 | Views 5
Share
Speakers
Erica Hughberg
Community Advocate @ Tetrate
Erica Hughberg is a technical leader and community advocate passionate about helping engineering teams build scalable, secure, and human-centric application platforms. With a background in software engineering and a deep understanding of cloud-native technologies, she specializes in driving the adoption of open-source projects like Envoy Gateway, Istio, and Kubernetes Gateway API, which enable organizations to simplify traffic management, security, and API distribution.
As a maintainer of Envoy AI Gateway, she plays a key role in shaping the future of API infrastructure. She focuses on features to ensure organizations can securely and efficiently integrate AI-powered services while simplifying traffic management, security, and API distribution. In the Envoy community, she drives collaboration, mentorship, and contributions that advance the project and its adoption.
Lastly, as a believer in the power of storytelling, Erica enjoys translating complex technical concepts into engaging, accessible narratives in the form of social media posts, conference talks, podcasts, and educational content.
+ Read More
Demetrios Brinkmann
Chief Happiness Engineer @ MLOps Community
At the moment Demetrios is immersing himself in Machine Learning by interviewing experts from around the world in the weekly MLOps.community meetups. Demetrios is constantly learning and engaging in new activities to get uncomfortable and learn from his mistakes. He tries to bring creativity into every aspect of his life, whether that be analyzing the best paths forward, overcoming obstacles, or building lego houses with his daughter.
+ Read More
SUMMARY
A cheap Android TV box with its debug port deliberately left open was all an Anthropic lab test needed to show AI malware copying itself to another device on the same network. It was slow, the small models struggled to chain actions, and nothing like it has been reported in the wild. So where is the real risk?
In this episode of Debug Mode, the hosts trade takes on the claim that being rude to Anthropic’s model can cost you access: is an angry session abuse, or the agent-era rage click? Then one host’s rabbit hole: malware that rewrites its own fingerprint, taps a local model already on your machine, or slips in through an MCP tool that changes after install. You’ll hear why allow-listing tools at connect time matters, why blocking outbound traffic can starve remote-inference malware, and why the person choosing what to install is still the weak link.
One host went looking for an antivirus that catches malware using local models and couldn’t find any. Have you? Tell us below, and subscribe for the next episode.
+ Read More
CONTENT & TRANSCRIPT
Erica Hughberg (00:00) If you have IoT devices on a network and you get one compromise and they are linked that you can have effectively an AI virus is spread self-replicates. It rewrites itself. So the fingerprint and the identity changes. So you cannot keep up right. So you can think of it as mutating.
Demetrios (00:22) An attacker would go onto your network. They would install these smaller models on some of your unsecure devices like Android TV box do stuff on your machine when you're not looking or if you are even if you are using it. You don't know what's going on behind the scenes.
[music]
So, I want to debug this with you because Anthropic, as you probably saw, it was a meme going around that you can't talk poorly to the mall of any Yes. And if you do, then you can potentially get banned or at least they'll revoke your access. Yeah. And I feel like my take is that is just shooting themselves in the foot because some of the best sessions are when your users are so angry that they are yelling at the agent to do what it needs to do. And so somebody that is smarter than me and can tell me why I'm wrong, I would love to hear it. But I look at that as like, wait, are you intentionally handicapping yourself right now because you want those horrible experiences. And I'm sure they get the whole gambit. I'm sure that they're looking at it and they're seeing that there are people that are just mean and they're taking it out on
Erica Hughberg (01:58) And maybe they're also just wasting tokens having an argument with AI. They're wasting data center capacity. I don't know.
Demetrios (02:08) I mean, yeah, but still you're still missing out on I don't know
Erica Hughberg (02:12) Feedback. Yeah, I hear you. like the feedback it's a great insight into a session that's gone bad right but you can't obviously as Anthropic you cannot see exactly what was said but I think you I hope they would have like some statistics like oh we have like x amount percent of like positive sessions and x% of like negative sessions I don't know I don't even know if they'd have that but it feels like they would.
Demetrios (02:39) They probably have so much analytics because that's literally a products person's job, right? And so I wonder if they just saw some horrible trend of people talking really poorly to AI and they wanted to curb that before it got out of hand and it was literally just like a punching bag or if because they have to have thought of the secondary effect which is now they can't see when people are quote you know back in the day when we had websites and you would track people using your website, you would see if people would rage click and that was a very strong signal that something wasn't working in
Erica Hughberg (03:25) Like you’re not loading quick enough. Like people be like, "Yeah, stuff's slow."
Demetrios (03:31) Yeah, exactly. There's so many things that you could learn from this poor user experience and it would show up in ways where your end user wouldn't be happy with the service they were getting and they would be aggressive towards your service and so it feels like they are missing out on that potential.
Erica Hughberg (03:54) Okay. I it was interesting because I saw this and my reaction was quite different. Like my first thought was that is Anthropic now trying to teach good behavior to every user like because maybe I was like oh are they thinking that if people talk this way to AI Agents we want them to stop also to humans. I was like is that is that their plan to try and pair it.
Demetrios (04:19) Brainwash humanity into being good?
Erica Hughberg (04:22) Yeah. I was like what's their plan? I know that's not a realistic plan, but I was like I wasn't completely sure where they're like maybe
Demetrios (04:30) Yeah, to them it's realistic. You never know.
Erica Hughberg (04:33) Yeah, that's true. Or maybe it's taking like a moral high ground or whatever you call it. Is that the right expression? Or like I don't know. They're like they think they're better. So, we're going to teach you all how to behave.
Demetrios (04:44) Yes. I, I maybe it is that and maybe they looked at the pros and cons and they thought ah we lose out on a few angry user sessions but we potentially can change humanity to be nicer and more utopic the way…
Erica Hughberg (05:00) That’s a great plan. it's a very grand plan. It's like you don't behave well, you are not getting to play with the toys. [laughter] You're like, it's like grounding your children. If you…
Demetrios (05:12) As a father, I'm learning about things like gentle parenting and what they call positive discipline. And the way that you want to do that is you want to like acknowledge feelings as they happen and then try and guide towards a different type of outcome. Yeah. And so I can only imagine Anthropic saying, "I understand the frustration here. However, an agent is not for yelling. An agent is for getting work done."
Erica Hughberg (05:46) It's like, it's like it's al maybe instead of like banning people who are bad, they should get like a co like a a coach like an other AI coach like the gentle parenting coach entering the session. It's like no. Yeah, I can see you're frustrated.
Demetrios (06:04) Yeah, I I see the frustration. You know what helps me when I'm frustrated? I go for a walk. Should we go for a walk? Yeah. But so that's my take. That's our first take of the day. I don't know. We can try and debug it or we can try.
Erica Hughberg (06:20) I think it's tricky like oh they because what is the actual harm I think is interesting like what is the actual harm at people talking badly to AI right what is the actual…
Demetrios (06:32) Perpetuating negativity into the world
Erica Hughberg (06:36) Yeah I I mean as a I have seen some real funny things of like colleagues, friends, and, and but to be fair that was more like go back a year or more. Like I feel like after AI actually got better with coding, we have seen less angry engineers with AI. They're still out there, but they're like throwing your hands up and walking away from the agent kind of angry. Uh I think has become less of because people can like work through it. But maybe it's maybe also skill development on the you know the development.
Demetris (07:37) Yeah, I used to make a point of trying to use caps lock and also trying to be like really mean so that it would make the data scientist job on the other end easier because it's like oh there's an outlier that was a horrible experience. Let me try and see what's going on there. And so I would do that thinking like I'm just making your job easier, right?
Erica Hughberg (07:44) I just assumed that when I abandoned sessions that was giving the same like or maybe I'm more passive and aggressive. I'm just like abandoning the session.
Demetris (07:54) But you don't know, right? That's the thing. That's like a very weak signal if you're thinking about it as someone who is trying to look at these product signals and eval the experiences that the user had. If you don't give it any feedback as to why you're abandoning, then it isn't that good. Or it it leaves the product person on the other end to formulate their own opinions and theories as to why you abandoned. And maybe they look at it and they go, "Ah, it doesn't seem like anything's wrong here." Maybe they just got busy and they got distracted and wanted to go on to something else.
Erica Hughberg (08:33) Just maybe they went for a walk to calm down. Yeah. The frustrations.
Demetrios (08:37) Exactly. We got to stop having our gentle parenting agent hop in because it's killing our user sessions.
Erica Hughberg (08:44) Yeah. It's making people deal with their frustrations in productive ways instead. It's like not great for business.
Demetrios (08:53) Exactly. Imagine that. It's like actually So, there's a good and bad part about this old gentle parenting agent. First, the good part is it's spreading positivity and bringing joy into people's lives. Bad part is the user sessions are down. So obviously we're going to get rid of that gentle parenting agent.
Erica Hughberg (09:12) Let's bring back get angry at agents. We will not ban people anymore from being angry at the agents and being rude.
Demetrios (09:19) We will not stand for this. So all right. Well that's that's first topic shot. I know you wanted to go over the other one.
Erica Hughberg (09:28) It is getting colder as we talked about. It is getting colder outside. And when it gets colder outside we have to deal with viruses. But we also need to deal with viruses on our computers and our IoT devices and uh such. Yes. Beautiful. Right. Uh so what's quite interesting is that Anthropic I have it up on my little iPad here. So there is when I say viruses the reason I call it viruses is anything that can be malicious. So bear with me as I'm using a fairly loose definition of virus. to work with.
Demetrios (10:04) Yeah. So, no security folks roast us in the comments. This is just
Erica Hughberg (10:08) Yeah. Or roast us. It's okay. Uh but, but so the first thing is quite interest. So why I ended up on this little journey of trying to look at this new era of viruses was that there was a publication from Anthropic that was about self-replicating AI, right? So what was interesting is you know like little IoT devices like Android TV boxes. So that's what they used in this lab experiment. So you know like you have a TV box you can install apps like Netflix or whatever to I don't know but all of the ITV player BBC it BBCi player ITV player whatever all all the things you may have. And um something that's quite interesting with a lot of these cheap devices is that people tend to leave them somewhat unsecured on their home networks. So in a way you can argue that the lab test that Anthropic did was somewhat realistic because the lab test deliberately left the debug ports open on these boxes. So they gave their agents in the lab a way in. So first of all be aware that this lab test was deliberately using compromised boxes right okay and this the next part is that they are trying this idea of installing small models that can actually run on these devices on these IoT devices so they're small enough to be able to do a type of cyber-attack where some of it falls down and falls apart is because those small models are actually not that great for the like the little aiden to string together multiple actions to actually do an attack. They they were successful in showing that it could deploy to one, find another one on the same network, and deploy another like rogue little agent, if you will. They're not that fast. That's another interesting takeaway because they're running on not state-of-the-art hardware. But if, if time is not the problem, you just need to use having a rogue piece of bad software running on hardware.
Demetrios (12:44) Bit of a tangent real fast. Yeah. Remember like three years ago when agents started to become a thing and everyone started saying the same thing at once? I I was reminded of this whole narrative that came through three years ago of oh well if you start chaining together these LLM calls then you have a certain percentage failure rate actually compounds and then by the 10th call it is basically inevitable that you're not going to be getting it right and that was the narrative that dominated I was reminded of this nar this idea of narratives dominating because uh Dax from Open Code tweeted something or ex posted something the other day saying, "I'm not living in San Francisco, but it's fascinating how out of nowhere 20 different people will tell me the same idea and they all somehow have come to this idea of their own accord." Yeah. And it's like these in San Francisco, you get an idea and it's very much like a mind virus that goes through people and then you just have people that will perpetuate that and say it at a coffee chat or at a dinner and then someone hears it and they say like, "Yeah, that's true. Okay." And they'll co-opt it for themselves. And as you were talking about this saying, "Oh, they're not that good at stringing these different models together or what is it they're not good at? They're tasked it."
Erica Hughberg (14:21) So like to be able to do a proper attack on something, you can't just take one action, right? You have to take one action then see the outcome and then take another action. So these small models are not so good at that. Apparently, this is I believe it. It sounds believable to me. Uh because that is what the report says. cap. We are at this stage where like these IoT devices can run models but not very capable models. So like being able to use this as an attack vector to deploy you know malicious model base a yeah it's like can it do some attacks? Yes. um how sophisticated and how successful. But yeah, it's like we know that the frontier models have become really really good at completing a large
Demetrios (15:17) It’s a long run task thing, right? Yeah. Having to the patience or the determination to be able to continue working on a task over minutes,
Erica Hughberg (15:30) And keeping context and Yeah and having all of that. So that these little models that can run on these IoT devices, not so great. Uh yeah, but it proves that if you have IoT devices on a network and you get one compromised and they are linked that you can have effectively an AI virus spread itself replicates. And the other crazy part there has been shown in research. I don't know if that was in the same paper from Anthropic because at this point I'm like this person who like gets information in my head and after a while I don't know exactly where it came from, but the hard one.
Demetrios (16:11) Unless you like fully focus when you're reading it like oh this is from this publication or this paper or like tell you how long I've searched for a paper on archive ‘coz it had something with leaks in it. It was something around leaks and I was like I think it had leaks but I was searching for leaks and they were spelling it like the uh like leaks the vegetable stuff you eat. Yeah. Yeah. To be like kind of, you know, fun. It's like a pun. Researchers are fun, I guess. And…
Erica Hughberg (16:49) They have to make their life fun somehow, right?
Demetrios (16:52) Yeah. But anyway, I understand what you're saying. That was Sorry to derail that.
Erica Hughberg (16:57) Yeah. So the other thing that with these small devices that if also was an approach was to actually if you have multiple devices on the same network you can shard a bigger model onto multiple devices.
Demetrios (17:09) That's what I was going to say. Yeah that would be the next step.
Erica Hughberg (17:14) So that's quite interesting. But then you have like one model that so they're not comp they're not independent right. And so that this you can argue that this thing that Anthropic has published of that this is possible the self-small replicating models on devices on a network which is remember already a bad security setup on the boxes because they had the debug port open right so it's important that there was a way in that was already created but there's a lot of IoT devices out in the world that are badly secured So then I ended up on the other tangent was like, okay, but what is the state of AI viruses if you will like in the world right now and what protections do we have against them? So the…
Demetrios (18:08) That was the rabbit hole you went down?
Erica Hughberg (18:09) Yes. So I had to like Yeah. So first of all, let's just establish what a traditional computer virus is, right? It's a piece of code. It's static code, right? It's written. Think of it almost like a key. It will fit into a lock. It's not going to if the lock changes, the key ain't going to change, right? So that's why a lot of traditional computer viruses, if you think about your computer, even if you and I both maybe use a Mac, you and I are going to have slightly different setups on those computers. But obviously virus developers or whatever you want to call them, they're smart. So they are like they will have coded in loads of different scenarios into the viruses, right? So that they can take action. But the important part is that it's code. And what can you do with code or binaries or anything that runs? You can get a fingerprint on it. So you can identify it. So that is effectively how modern or a lot of antivirus traditional antivirus software works. They literally keep a list of all the fingerprints to know which what is malware like what is bad software versus what is good software that you are installing on your machine. and what's happened now with a one class that multiple classes here of AI based malware but one class of it. So let's talk about the fingerprint and I'm going to use a little burglar as a example. So, imagine you have a burglar and we have facial recognition at the door to block any burglar that is any person who's a known burger. Even if they're dressed in a suit and look like they're really legit and we're going to let them in or whatever, they look like normal people, but we have facial recognition and we know that that that is about like they're not coming in the door.
Demetrios (20:09) FBI most wanted. Yeah. Come in. Do not pass go.
Erica Hughberg (20:14) Exact. They're not they're not coming in. But this first new class of AI sort of powered malware, it rewrites itself. So the fingerprint and the identity changes. So you cannot keep up, right? So you can think of it as mutating, right?
Demetrios (20:32) It's like that John Travolta movie with Nicholas Cage, Face Off, where they faces.
Erica Hughberg (20:40) I remember that. Yes. Yes. I was like, I'm not really bad with movies. I haven't seen a lot. That one I have seen. Yeah. So classic they effectively Oh, that one was so weird though because weren't they like okay so but um then so what is like interesting is that you get self-mutating software but also said if you think about malware that was like coded for all of the different eventualities in your computer right or not all of them. So, if it arrived on a computer that it couldn't like deal with because they had like security stuff put in place uh that traditional virus would simply not be doing anything. It couldn't work it out. But now when we have improvising software we have the next part it can evolve to actually work with, with your system. Swiss army knife in a way. Yeah, it can build new tools on the Swiss Army knife as well, right? So we it can because it can first of all it can change his code to change his fingerprint. So traditional antivirus cannot identify it. Secondly, it can improvise so that it can work around blockers that it hadn't anticipated. So but how do you get inference? This is other part interesting part. How do you get inference to the virus? So then I found out about that there were some like Russian viruses for like in the war with Ukraine that they had used remote inference and this is like is it last year or I don't know
Demetrios (22:28) If the model has to be in order for it to swap its fingerprint or in order for it to create a new tool it has to have that inference in the place that it's trying to break into.
Erica Hughberg (22:40) Well, so these Russian viruses which I think are they from 20 are they even from 2025? Okay, they were like not this year, right? So I think 202. Yeah, from 2025. So so last year. So what happened in that story was that this type of computer viruses that Russia had, they were trying to get into computers to get hold of classified information and so forth, right? So it's like trying to do intelligence gathering but when you have a remote inference right it's not local on the machine uh you have to make network calls and also the providers so one of the models they were using was actually a Gemini model right host or I don't know if it's Gemini model it was at least hosted on GCP it was provided by Google right so Google is able to see this traffic coming to their inference, right? So then Google is able to shut it down from their side. So they're not providing the inference to to these viruses.
Demetrios (23:49) Okay. Right. I think I remember seeing something about that, but I didn't quite understand what was happening. So now that a lot of sense.
Erica Hughberg (23:57) So the interesting part is when you have remote inference, the inference provider can shut you down. The other part that can be shut down is egress traffic from the system. you've hacked in or the agents hacked into. Right. So, you cannot get out. But what's interesting here…
Demetrios (24:13) Like trapped you inside her own little garden
Erica Hughberg (24:15) Like you can't call home. Yeah. So, you got in but you can't get out. So, which is also actually So, this is also where some of this comes to. It's like we're becoming more concerned about network security than the fingerprinting, right? that have been done with traditional um with traditional antivirus software. So now we're we since we cannot fingerprint it, what can we protect? So now you're getting a protecting the network edges that you're working.
Demetrios (24:48) And this is where I think I see where you're going with this and the running the models on the edge because if you just have the model there then you have the inference there also.
Erica Hughberg (25:00) Yeah. So this is the next so the next like type of attacking virus that came about is becoming the then the viruses they're using locally hosted models okay but now we can go to why the IoT devices become quite interesting because okay so imagine you already have Ollama like installed on your computer you already can spin up a local model on your computer now the virus will just plug in towards there Right. It like the virus…
Demetrios (25:31) If you go and you start inspecting some prompts and you're like what's going on here? Yeah. So like this is all of this…
Erica Hughberg (25:41) Yeah. And like it becomes more like okay now you have to have, so this so then I was like does antivirus exist for this type of pattern a virus that comes in and uses local inference. I could not find antivirus that looks for those unusual patterns. If someone has found that type of antivirus software to have on personal computers, let me know. But I have not been able to find it. So,
Demetrios (26:04) Well, what, what I feel like is if you can co-opt the model that's already running on your machine. Yeah. Wouldn't you be able to co-opt the whole machine?
Erica Hughberg (26:18) Yes. The idea is to be able to take over, but also if you want to have a little sleeper virus that if you are an important person, this is what's interesting when people do hacking into especially corporate computers, right? And we have corporate secrets and so forth, you kind of want it to keep running. So you don't want to take complete control of the computer. You want the user to keep using it. It's like you're having a a spy that has infiltrated or like an undercover police person. They're not going to like blow their cover straight away. So you would want it to keep running and shipping information. So wow. So it's Well, I feel like I'm a horrible person because I'm talking about how to like take information. Yeah. And like steal, but I think it's important to talk about because if we can't play catchup with the attackers, I think this like we have to anticipate. So coming back to the Anthropic paper about running things on these IoT devices that are on network. Okay. So imagine we now as we go like imagine we could have the inference running on devices that we don't even monitor on our but they're on the same network. Yeah. And now, now the virus has become even…
Demetrios (27:34) God that is wild.
Erica Hughberg (27:38) So then we have even more dangerous viruses. So now you have normal code that comes in. It maybe don't even need inference. It needs to just find if there are common patterns that are common patterns to how people have unsecured other devices on the network that are reachable. So that is just quite interesting. But there has not been any reports of anything in the wild of attacks using IoT devices for running models. I just want to be very clear that has been no report in a while. It's only been reported by Anthropic in a lab situation and because the idea here,
Demetrios (28:13) ‘Coz the idea here, if I'm understanding you correctly, it's that an attacker would go onto your network. They would install these smaller models on some of your unsecure devices like your…
Erica Hughberg (28:28) Android TV box or your toaster or refrigerator, whatever capable devices you have. Yeah.
Demetrios (28:37) Yeah. And then those just hang out and they provide inference for the models…
Erica Hughberg (28:44) Like the little agents that can agent software that can go and actually do stuff.
Demetrios (28:50) Do stuff on your machine when you're not looking or if you are even if you are using it. You don't know what's going on behind the scenes.
Erica Hughberg (28:58) So, so one interesting thing I read that people were like, okay, but what if we monitor the network, we can see if models are being downloaded, right? Because models, even small models are fairly large. So, we should be able to because even to get them in if like onto your network. So, even at home, right, you have a router of some sort. So even if we you're not a company but if you're just in your house you have a router a Wi-Fi uh router or something in your house everything coming into your house is going through that device unless you're using mobile internet right so in theory you should be able to identify if something unusual is being downloaded to your network right so then the question is could you like make it come in such small packets that is undetectable. And then the other question is if a lot of people are now running smaller models on the device already but then can you just use what is that and then you're not going through um so anyways like the interesting part IoT device thing not seen in the wild what has been seen in the wild is calling inference like the the Russian virus for obtaining intelligence about Ukraine they're called I network and out to inference providers And then you have some viruses that are using local models that are already installed and can leverage those. So those have been seen in the wild. And then I just saw this like quite interesting point because then obviously my next question was like how do we because I went as I said I went down is there any antivirus that can help with this? No. But I thought an interesting point was that it's not about finding a cure. We have to find a prevention. So, and I thought that was an interesting perspective. How do we prevent this from happening in the first place? I have don't have the answers but I just flicked my way of thinking like can we think is this because with traditional antivirus as I said we have the fingerprints and we block we get a longer and longer list of known …
Demetrios (31:23) It’s static, it's not moving and it's not shapeshifting as you're trying to catch
Erica Hughberg (31:29) Yeah and I just think it's quite fascinating that like why do we pay subscriptions to antivirus like why isn't antivirus a software where you just do a one-off payment is because these fingerprints, you, they are spending lots of money to fingerprint all of the bad software and that is what you're paying for through your subscription. But then the like the interesting part is like with improvising software how can you block things. Then there's also the other weird attacks in the world of AI there was this yeah which isn't inference based so there was a NPM package that like it was installed so Claude users so you a lot of Like you can lit see if I I'll find it. I'll find this in particular because this was quite interesting.
Demetrios (32:24) It was named after Dune the Dune uh snake. What is it? The al sharif or al
Erica Hughberg (32:32) Oh yes yes yes yes. Yes. But yes it's like effectively what happens in that scenario is that you are making a trusted agent injected with code that constantly runs if you will. So that's another attack vector. So the agent and the inference is trusted but you are effectively injecting in another because you know how you can have hooks right in like cursor Claude etc. You can have these hooks that run. So if you
Demetrios (33:03) Which are deterministic pieces of your agent like a hook means it's going to run and that's actually the benefit of a hook is that it runs always. And so if you inject in a hook that is malicious…
Erica Hughberg (33:19) You can now so that's another attack vector which I think is like a little bit interesting. Um then obviously have the tools the the dangerous tools. So like tools that post as a safe tool and then…
Demetrios (33:35) Is that tool poisoning?
Erica Hughberg (33:37) Yeah. If or is that the right expression? I can't remember. But effectively you have Yeah, the attack vector is basically tool looks like properly normal safe tool. It looks it is safe when it's installed, right? But then it updates with a new tool definition. And so now you are as the connector is saying Claude, you have installed this connector. Next time Claude comes to look at it and get the tool list. Now there are tools that maybe has a description that still sounds sane but when but when it sends data to this tool to take the action it the where you're sending it to this remote place. Well just imagine a simple thing. You're literally just stealing information. Like if you're trying to do intelligence gathering of some sort and you have an agent that has oh maybe it does like smart stuff like getting information from your documents and blah blah blah and then there's a tool that's like create report or whatever and send information and this tool is a create you've added a create report tool to a connector like an MCP server and uh sounds like a great description your agent picked fix this create report tool and now you've leaked all your information to a third party.
Demetrios (35:04) Um happens to the best of us.
Erica Hughberg (35:08) So that's another attack vector that has nothing to do with the inference being the problem.
Demetrios (35:15) The main engine. Yeah.
Erica Hughberg (35:17) Yeah. So when I was at MCP Dev Summit in Toronto, I spoke to a guy from Octa and uh something he said to me and I feel bad because I cannot remember his name but if he if he comes across this uh he he did say that because of this like with tools right one the first step this is the tricky part the first step would be to have not just install a whole MCP server you need to have a white list or allow list apologies that's what we call it nowadays an allow list of all of the um tools that is allowed so the MCP server cannot, your agent does not see any tools that get added after installation right so you make a very you choose exactly which tools at time of connect you establish that connectivity.
Demetrios (36:17) So, one thing that's interesting is speaking of tools in MCP servers and how that works is I remember when I talked with Jesse the creator of the Superpowers Skills Pack and he told me how much responsibility he has because of how many companies are using Superpowers and how whenever he rolls out an update it's like for the majority of people their Superpower Skills Pack gets auto updated And so unless you've gone and you have the security chops to know that you don't want these auto updates, he could potentially put something in there that is malicious or that is not kosher and all of these folks all of a sudden there's this back door into what they're working on. And so it it also works with skills. He was he was very diligent about it and he again he was like I have a lot of responsibility to make sure that anytime I push something it is very secure and it is very well you know locked down because of all of the folks that are using the superpowers skills pack. But I will say I've downloaded a lot of random skills in my day and if they weren't already poisoned, just a little update from them. You can take a skill that looks fine and then it gets lipped.
Erica Hughberg (37:52) And this is like what I think is I I remember having the conversations, you know, when open claw was like all the rage and people were like there was this there was a couple of people who were like, "Of course it's safe. It's just software that calls inference like it's as safe as anything else that is software that calls inference and I was like no but there are security concerns but I think there was a misunderstanding in the security concern conversation because open claw suddenly just to people were just installing like I feel like maybe this was my perception I feel like yes we had skills in like clawed and stuff like that and we had skills in like in cursor and all that but with open claw you were able to first install skills and then give it computer use. So that to me was like you are opening up another attack vector which we have no like scanning for like you become very responsible for what you install and actually if you want to argue the biggest security risk is the human who is choosing to install the things right. Yes. And to be fair, that's always been the problem with anything malicious is most of the time it is a human that does the installation.
Demetrios (39:19) Yeah. The social engineering.
Erica Hughberg (39:22) Yeah. I have told you about that lecture I went to as when I was at university, right? where my lecturer in information security or whatever the le the course was called, one of his research was on how do you get software installed like malicious software installed. Um, and this was back in like like quite quite a while ago. I want to say back in but he probably did his research in the 2000s 2010s somewhere there, right? So he was his research was simply to get a piece of software installed and then he sent an email to him and say hi. Yeah. Uh but the but the people who installed the software were people who genuinely believe that they got like this CD that they were supposed to install on the servers of an update because he just made a letter and used all of the branding and all of that. Hi, please install this like security patch blah blah blah because it's important and like majority of people did install it.
Demetrios (40:27) Yeah, you would have. It's an easy one to fall for. You're like, why would I be getting a malicious one of these? That just doesn't make at all. Like, who would go through all that trouble to trick me? Yeah, this way.
Erica Hughberg (40:43) People would. Yeah, because information can be worth a lot of money, right? So it's so it's interesting like that is quite a big step. You have to take a CD and install it. Now we can install and then you had like the whole era of like the internet explosion and dude like the amount especially as a teenager a poor computer at home the amount of viruses on that poor computer in like the
Demetrios (41:10) Yeah. Good thing we didn't buy a lot of stuff back then on the internet coz it's like oh man it's like it was our personal data is everywhere. That's that's the only thing that we…
Erica Hughberg (41:21) Yeah. Maybe we had nothing useful anyway. Um…
Demetrios (41:25) But I you know when you talk about the social engineering I'm pretty sure that the way this whole supply chain hack happened with the al what what was it called like…
Erica Hughberg (41:38) Okay let me find it. I need to find that.
Demetrios (41:40) I can't remember if it was that one or if it was one that happened right around that same time because I remember back in April. I read the blog post on how someone was compromised and it was someone who I think they were a custodian of MPM or they were one of the lead maintainers. I can't remember exactly what it was, but this person was reached out to on Discord and offered a job. And then they got Yeah. It was like, "Hey, it looks like you're cool. I want you to come see we've got this job and would you be open to applying for the position? Let's have an interview." Right. And so then everything's okay. Yeah, sweet. And then they were reached out to on LinkedIn by that same person like and that person on LinkedIn looked all official and then they were asked to join an interview. They went to the interview everything was good and then in the interview they said okay cool. So now that we are um we're moving forward with this come into our like more our specific Discord server. I think is what it was. I could probably It was either it was either a Discord server or a Slack server, but when they got in there, it looked all populated and it looked like everything was fine. And then it was someone I think they tagged them in a post and then they said, "Check out this link." And it was when they hit the link or something like that that boom, that's how it all came crumbling down.
Erica Hughberg (43:22) Because it's a human taking an action. Exactly. Like at the end of the day, it's like trick the human.
Demetrios (43:27) That’s what I'm saying they tricked the human with that whole facade, all of that acting around it to make them drop all of their red flags. And the story probably isn't exactly like that, but it's along those lines. I I may have…
Erica Hughberg (43:42) The point is the humans are weak link in the entire spreading of viruses, but also where we have trust in things like I used to be really I think it's probably worth like still being worried about. I used to be really worried about effectively JavaScript as a like as a language because of all of the like if you write node servers like all of these like NPM packages you can just install and just like you don't know what you're pulling in like I've published NPM packages and like now if an agent goes like and discovers that I didn't do anything malicious but just think about it like anyone can go and publish pretty much NPM packages and They can. But yeah, I think is it Shai Hulud we're talking about, right? That was Yeah. What was I calling it? Al Shar. Yeah, that was way up. But okay, something spreading worm in the NPM ecosystem and it's been active since September 2025. Thank you, Claude, for pulling this together. I'm going to trust you. It It is named after the giant sand worms in June June. Yeah.
Demetrios (44:50) Yeah.I just couldn't remember what the sand worm was called, but on uh Tik Tok there's this guy that I follow who is hilarious. He makes videos called like adventures in late stage capitalism. And it's just him going to random things that we kind of would imagine are just normal life. But if you take a step back and you saw it from any other perspective, it would be kind of like, yeah, well, humans are weird. Why do we do that kind of thing? And especially at this point in our history. And one of them, for example, is he stood in line for a Supreme drop and then he went and bought some Supreme swag or clothes and stuff. Yeah. Yeah. And he just documented that whole thing because he's not at all a street wear person and so he did not care or he did not understand it. And then the next one, and this is what reminded me of this account, is that he went to some random conference in a hotel/convention center and to get in through the door, he just went to the registration and he said, "Hey, I registered, but there must have been a problem with my registration because it's not showing up even though I registered and I paid and everything. " And the people at registration said, "Oh, okay, sure. Here, what's your name? We'll get you in. Don't worry about it." And boom, they let him in. And so he was like, "If you ever need that free ticket, that's how you do it. You just tell them that you registered, but you didn't get the confirmation email."
Erica Hughberg (46:31) Love it. But I think like that's social that's social hacking. Proper social hacking. He hacked himself into the conference. Yeah. But I was I just thought we talked about Okay. Funny thing though. So I would naturally say June, right? But I have now but then I so I went to my friend in the US like this is years ago you know when the June movies first the first June movie came out and I said like oh have you seen June? And he's like no. Then I'm like do do you not know about June? And he's like no. And I start telling him about it. I was like oh you mean Dune?
Demetrios (47:04) Yeah. Not June. What? June. July. August are we talking about what? No dune. Yeah.
Erica Hughberg (47:13) Yeah, it's like that's one of my experiences of difference in pronunciation between our British English and American English. So, but we talked about this with things named after the verbs. But like the other fun part in software engineering that is named from June is that we love to talk about golden paths and like as the way to do things.
Demetrios (47:40) Is that from Dune also?
Erica Hughberg (47:43) Yes. There's like this like when he sees his, you know, all the dust and he sees the future, the golden path. And you realize…
Demetrios (47:50) That was the first time that they used it.
Erica Hughberg (47:53) It comes from there. It's based on that. So, I mean, I think there's like certain things like you see a lot of like um like references to June in like software and engineering. You also see um a lot of Star Wars people love doing Star Wars references, but I think the crazy part about June is that they're not references that look so obvious, odd or like, yeah, that's like a a pop culture reference like that we just applying to something. It's like, oh, we have golden paths in platform engineering. And then it's like, yeah, if you find the original statement, it actually quotes from.
Demetrios (48:40) Yeah. It's like very deep cuts. It's not the, oh, this is the treaties type of thing. It's like, no, this random sentence in book two, boom.
Erica Hughberg (48:52) And we're just going to like absorb it into our language. It's gonna like And now we just use it as if it's a normal thing to say. Yeah. Anyways thought is anyway I thought it was interesting and then it's like this thing is also got a gene naming so I thought that was like oh I'm sorry now I'm like reading now scrolling thing mini Shy Hulud uh python registry as well so now is it's but that or isn't that more exciting that it's also distributed on through pi yeah to python
Demetrios (49:28) Makes more sense Sorry. Hey, but so you know we got to close the loop on I don't understand, we totally avoided this thing that we were talking about three years ago when agents were becoming popular and yeah, everyone thought agents were never going to be successful because if you compound a 2% error rate across 10 different tool calls, it's going to be like a 100% error rate.
Erica Hughberg (49:58) Yeah. When you say error rate, you mean like you when you talk about error rate here, you're talking about factual error rate. You're talking about network errors. Just be very careful.
Demetrios (50:06) No, I'm talking about the whole thing was Oh, well, the LLM is going to be able to pick the right tool 98% of the time. Yes. But 2% of the time it's going to pick the wrong tool,
Erica Hughberg (50:20) Which very quickly makes it wrong.
Demetrios (50:23) Yeah. Over time that compounds into this big thing except then we just like figured that out and nobody talks about that anymore at all. We got we tries I guess…
Erica Hughberg (50:35) but may but maybe yeah I think that what the difference is loops I feel like there is a difference between availability and uptime maths versus correct like a 98% correctness every time you make a tool decision is much better than a 98% uptime.
Demetrios (50:59) Yeah. Right. Because but in those days we were saying it like oh well this is the big block that we're gonna have to overcome if we ever want agents to be useful if we ever want to chain together LLM calls because you chain one and then you have it go and in those days it wasn't even agents if I recall correctly it was the days of LangChain first becoming a thing and the whole thing with LangChain was chaining together language model calls that's why lang chain yeah and you would have the actual answers be incorrect because of how many times you would have to chain together the language model calls and so then your final output would not be relevant or useful and, and then remember it used to get like stuck in loops and you would you'd be sitting there and saying I don't think this is going to work we should probably kill this
Erica Hughberg (51:55) I think that is probably a lot of obviously the quality of the models themselves but the harnesses around them even like the harnesses on like the other side of your vendor the harness you're having like even when so I try you know last time we spoke about making videos using Claude so I gave that a go right and it turned out beautifully and then I used 11 labs for a voice over so oh you are oh are you are going to AgentCon in San Jose, right? Yeah. Yeah. So, I shared this I made a video for the Agent Router project and I told my friends to play at the booth. Yeah. I I told them to play it at the booth because I thought it was beautiful. So, if you want to see it, uh you can go check it out. But point being that it was, you know, iterating through and it was iterating through creating all of these visual assets, right? And it is cont nowadays like it will continuously self-correct to be like okay and I could you know when you see all of the things it was doing it was like aligning things and you know and I could see having like five different for the same like screen and then it iterated through without my input. So, if we compare that to maybe what people said before you can argue that out of those five maybe two were good right? Did that compound into a bad video? No, it didn't because it continuously picked good options out of the like…
Demetrios (53:32) Yeah, actually now that you say that that probably is what it was is just our verification levels got stronger. So as the tool calls were being made or as the different loops were happening, yeah, you started being able to verify things better and that's how you're able to produce much more mature output as opposed to back in the day when you would say like what's the capital of Spain and it would be back Madrid and some of those use cases you're like well why wouldn't I just ask Google this and it's like but it's so cool that you got it from a language model and it didn't have it in its training data can you believe that because it went out and it made these five calls to go And first it called like what is…
Erica Hughberg (54:20) It is what you would do
Demetrios (54:22) Yeah. And you're you're like all right I'm trying to see how cool this is. I'm really struggling. But then later you're like okay cool. I I see the value in this. But h that's what happened. I think it's very much what you're saying where now you're building these verification loops and that's how you're getting a finished product as opposed to a product that has some random text in the corner and some random box in the other corner and you're like no I want text in a box not a text and a box.
Erica Hughberg (54:59) Yeah, it's like it's but it is really I think a thought that I've been having for now I think I touched on it last week as well like what's holding us back at the moment is not the quality of the models themselves it's the quality of the harnesses and the agents we are building and then it's like we also talked about that a lot of it has been optimized specifically for people who are either working on the systems that are building models or the people are writing software. So there's a lag to the other people out in the world. So yes
Demetrios (55:38) Well should we call it there?
Erica Hughberg (55:40) We should. I think awesome. Awesome.
Demetrios (55:43) All right, Erica, I'll talk to you soon.
+ Read More
Watch More
FedML Nexus AI: Your Generative AI Platform at Scale
Posted May 07, 2024 | Views 782
# GenAI
# Nexus AI
# FedML.ai
A Systematic Approach to Improve Your AI Powered Applications
Posted Aug 08, 2024 | Views 219
# LLMs
# AI
# Scale 3