Sign in or Join the community to continue
Can Your AI Heat My House?
Posted Oct 27, 2026 | Views 4
Share
Speakers
Zachary Smith
Co-Founder and CEO @ Datum
Zachary Smith is a leader in the digital infrastructure and cloud computing ecosystem. Zach started his career at New York-based Voxel, a pioneering Linux-based cloud computing business. In 2014, he co-founded Packet to empower tech-enabled enterprises with automated bare metal compute. Backed by SoftBank, Dell Technologies, and Battery Ventures, the business became the leader in bare metal automation and was acquired by Equinix in 2020. Zach led Equinix's edge computing and software connectivity business until he left the company in early 2023.
In late 2024, Zach co-founded Datum, a startup on a mission to provide Internet-scale superpowers to every builder. The company is backed by leading venture firms including Amplify, Encoded, and CRV.
In addition to his board role at Wasabi, Zach currently serves on the board of Cloud One Digital, a digital infrastructure platform backed by One Equity Partners and Pursuit.org, a workforce transformation impact organization upskilling workers for an age dominated by software and AI. He is an active member of several arts organizations, including the National Sawdust and The New School’s Schneider Concerts advisory boards.
Zach lives in Manhattan with his wife and two teenage sons. He graduated from the Juilliard School in 2001 with a Bachelor of Arts degree in Performance, performs with the Camerata Notturna, an amateur chamber orchestra, and is section principal of the New Conductor’s Orchestra.
+ Read More
Erica Hughberg
Community Advocate @ Tetrate
Erica Hughberg is a technical leader and community advocate passionate about helping engineering teams build scalable, secure, and human-centric application platforms. With a background in software engineering and a deep understanding of cloud-native technologies, she specializes in driving the adoption of open-source projects like Envoy Gateway, Istio, and Kubernetes Gateway API, which enable organizations to simplify traffic management, security, and API distribution.
As a maintainer of Envoy AI Gateway, she plays a key role in shaping the future of API infrastructure. She focuses on features to ensure organizations can securely and efficiently integrate AI-powered services while simplifying traffic management, security, and API distribution. In the Envoy community, she drives collaboration, mentorship, and contributions that advance the project and its adoption.
Lastly, as a believer in the power of storytelling, Erica enjoys translating complex technical concepts into engaging, accessible narratives in the form of social media posts, conference talks, podcasts, and educational content.
+ Read More
SUMMARY
What happens when the AI revolution collides with the physical limits of our infrastructure?
In this episode of Debug Mode, Erica Hughberg is joined by Zachary Smith to explore how AI is transforming the world of cloud computing, data centers, and hardware — and why the biggest challenges ahead go far beyond GPUs and model performance.
+ Read More
CONTENT & TRANSCRIPT
Zachary Smith: [00:00:00] The largest inference service provider of the world might be Apple in three years.
Erica Hughberg: I just find it very fascinating that AI probably consumes more drinking water than I do.
Zachary Smith: We can't even support the floor loading of those. They're running to 6 to 8,000 pounds of weight per rack. Most buildings can't handle, even in their ground floor on the concrete, that level of density.
Zachary Smith: I would say positive both cultural and economic things that we have with technology. Are we willing to basically consume all of the resources of our planet to power more GPUs in bad ways, or will we do it better?
Erica Hughberg: Very special version of debug mode [00:01:00] because Demetrios is not here, so I get to do whatever I want. And, uh, I've been down a massive rabbit hole, and I start get really interested in all the weird hardware and infrastructure, and also how it's changing in this AI era. And when you find yourself in trouble and you don't know how to get out of a rabbit hole, you can call a friend, and that's what I've done.
Erica Hughberg: So Zach, my friend, who's gonna- Hello ... get us out of the rabbit hole of how is infrastructure and cloud and everything changing and evolving in this AI era, and also what does it mean for, like, leaders? That's something I'm really interested in. What does this mean for, like, engineering leaders, how they approach things?
Erica Hughberg: I don't know if you have the answers, Zach, but you might have an opinion.
Zachary Smith: I have s- many opinions.
Erica Hughberg: But Zach, I have a first question for you, which is... I actually have many, but s- I'll start with one because it's been bothering me all morning. So [00:02:00] you talk about alt clouds, like alternative clouds- Mm-hmm
Erica Hughberg: which does not just mean people who work in clouds wearing Nirvana T-shirts. So what is actually alternative clouds?
Zachary Smith: Not all heroes wear capes, you know?
Erica Hughberg: Yeah. So what, what is an alternative cloud, and why should anyone care about them?
Zachary Smith: Well, uh, I think we got a definition problem really quickly, so- Yeah
Zachary Smith: I've tried to make one. Um- Okay ... well, mainly I have an identical twin brother. He's much better with words than I am, and we were thinking about this a year or two ago because I grew up in, in, in, in a, in an era where we- Mm ... it was very simple. We had data centers, and we had hosting companies. Well, first it was ASPs for a little bit, so I- Mm
Zachary Smith: briefly entered into the world of technology in the, uh, kind of late '90s, early 2000s, and there was these application service providers, but really it was hosting companies and data centers.
Erica Hughberg: Yeah.
Zachary Smith: And then [00:03:00] emerged this whole new wave of IaaS players.
Erica Hughberg: Yeah.
Zachary Smith: Yeah, infrastructure as a service- Yeah ... which really were providing computers, networking, storage, all kinds of things- Yeah
Zachary Smith: generally with some sort of automation around that, you know? So it wasn't just, you know, hardware sitting in a data center. Yeah. It was hardware sitting in a data center that had some sort of rep- reproducible automation around it, right?
Erica Hughberg: But for me, like, I came into the world of the internet, well, I started being on the internet and actually making websites and stuff because- Mm-hmm
Erica Hughberg: I'm not, not as early in the world. But anyway, I stopped, and then we had-
Zachary Smith: Are you saying I'm old and have gray hair? I like it. Mm. I just, uh, like a little gray hair is okay.
Erica Hughberg: Little, little is all right. So I, my first experience with publishing a website was, uh, getting an account on a web hotel I don't know if that's what they were called in English as well, but I grew up in Sweden, and we called them web [00:04:00] hotels.
Zachary Smith: Hey, man, I just need to rent the web for a little bit.
Erica Hughberg: You just need to rent space where, like, literally you just have to pay to get your files up there, and you can serve your website up, and you got DNS pointer. It's like, great. You know, sorted. So that's, like, what I started, started my little journey in software.
Erica Hughberg: But also it meant that I, for my career, I didn't really care about hardware or data centers much.
Zachary Smith: Sure. Yeah. I would say that the vast majority of people, you know, over the last 20, 25 years haven't had to worry about it. Just kind of like, what do they say? Most technology is good enough, it's like magic, right?
Zachary Smith: And people didn't have to worry about hardware. They basically got to g- you know, go online to some sort of a web hosting or cloud computing company, an IaaS business, and-
Erica Hughberg: Yeah ...
Zachary Smith: clickety, click, click, click, and I've got access to part of or a whole computer with the things I needed to. And, and I think, like, that's evolved to be, like...
Zachary Smith: You, you're probably familiar with the IaaS, meaning infrastructure as a service, and then, you know, PaaS- Yeah ... platform as a [00:05:00] service, and then SaaS, which is software as a service. Yeah. And then we got
Erica Hughberg: kind of- We care even less.
Zachary Smith: Yeah. You just, like, abstracted further away. And then there's this thing that came out in the late, you know, noughties, which was, you know, serverless, which is really an IaaS product, but with you didn't have to worry about the servers, meaning you didn't have to, like, update the operating system or-
Erica Hughberg: Yeah
Zachary Smith: things like that. And I think that that's become, like, really insufficient over the past few years. Because number one, we've had new layers of the stack emerge, like neo clouds, which are-
Erica Hughberg: Yeah ...
Zachary Smith: very specific infrastructure-focused hosting companies that just do
Erica Hughberg: kind of- Is a neo cloud an alt cloud?
Zachary Smith: I believe so.
Zachary Smith: Mm. Um, and so I made this little definition. You can go to the Alt Cloud Awesome List on GitHub. Yeah. Just Google for that. Um, or ask chat, I guess, to find it for you.
Erica Hughberg: Don't, don't
Zachary Smith: Google. Ask- Or we can use the web address. Don't Google. Just ask Gemini to find it for you. Yeah. It'll be fine. Um, but basically we said, "Hey, this, this kept getting confusing," 'cause people are like, "Wait, is it a [00:06:00] cloud or is it a neo cloud?"
Erica Hughberg: Yeah.
Zachary Smith: And I felt that to, to users today, developers today, people who build and run production infrastructure, there's this whole range of clouds. You might use a database cloud like Supabase. You might use- Yeah ... you know, a front end like Vercel. You might use a storage cloud like Wasabi or Backblaze.
Zachary Smith: Uh-huh. Like, there are all kinds of clouds, and so where do they fit? Yeah. And so we came up with this concept of alt cloud, which is hyperscale is one, and then everything else is some sort of alternative to this. Yeah. Which is basically an array of infrastructure-associated or infrastructure-dependent services- Yeah
Zachary Smith: um, that are operated as a service, can be globally consumed- Yeah ... um, and if they go down, your kittens cry And
Erica Hughberg: Like, yeah, it's, it's import- it's an important component It, it's important. It's critical ... of the spot. Yeah.
Zachary Smith: Yeah.
Erica Hughberg: And okay my brain jumped somewhere here. So if you're a big company [00:07:00] and you have your own data centers-
Zachary Smith: Mm
Erica Hughberg: Would you have similar challenges and problems that companies that are running alt clouds would have?
Zachary Smith: I think it depends what kind of company you are. Yeah. I do believe, you know, in our definition that we put together, um, we had the concept of some sort of consumptive base and multi-tenancy in there-
Erica Hughberg: Yeah ...
Zachary Smith: which many enterprises operate internal- Mm-hmm ... private clouds for their- Yeah ... hundreds or thousands of constituents.
Zachary Smith: Yeah. And I would consider that to be very similar, is that, you know, the bar of end users who have grown up in, "Hold on, let me roll over to GCP and magic happens"-
Erica Hughberg: Yeah ...
Zachary Smith: um, or, "Hold on, I've rolled over to Vercel and just Git Push"- Mm. "... magic."
Erica Hughberg: Yeah.
Zachary Smith: I think those are the users who work at companies now, and companies- Yeah
Zachary Smith: are like, "Shoot, I need to have this type of capability." Um, so I, I do- Make it
Erica Hughberg: easy.
Zachary Smith: They have to, you know, 'cause that's what the user experience expectation is like- Mm ... um, if you [00:08:00] have the need for an internal infrastructure or you're- Yeah ... doing that because of, you know, other reasons. So yeah, I think the bar has gotten raised and is more complicated.
Zachary Smith: So take a look at All Clouds. It's, it's a, it's a position. I'm not sure it's a full answer, but I think, um, it is a broader definition than saying hyperscale, Neocloud- Mm ... and then all these other things which we're not sure
Erica Hughberg: of.
Zachary Smith: All
Erica Hughberg: the other things. Yeah. So- Okay. So because I'm gonna say that you're a little bit older than me, um,
Zachary Smith: I was, I was born in the '70s.
Erica Hughberg: Yeah.
Zachary Smith: Just barely, though.
Erica Hughberg: And, uh, so what... So what I worried about in 2006- Mm ... and what you worried about in 2006 were probably different, and maybe owes not just because of an age difference, but also because of an interest difference. Uh, what, say, 20 years ago, which- Mm ... I mean, I think EC2 is turning [00:09:00] 20, right?
Zachary Smith: Yep.
Erica Hughberg: This year. And, uh, what were you worried about 20 years ago? What, like... Or what were you interested in? What were you, what was occupying your mind 20 years ago?
Zachary Smith: Yeah. Well, I mean, like, let's just say that the world of using infrastructure in my day-to-day life, you know- Yeah ... I had probably many concerns as a, you know, 20-something-year-old- Yeah
Zachary Smith: it was in 2006. Um, like, do I have enough money to pay my rent, and how are we gonna get payroll done this week? Yeah. You know, I think was probably a lot of what I was thinking about. Practical
Erica Hughberg: stuff, yeah.
Zachary Smith: Um, but you know, beyond that, actually, I thought a lot about, at the time, just the, you know, community of the internet.
Zachary Smith: Um- Mm ... when I got involved in, in the, the infrastructure world, the internet was still very, very community-based. Yeah. Open source was not a foregone conclusion. Um, you could not really boot Linux operating systems on those computers. Um, you know, the world of open source software is [00:10:00] pretty much just Apache Foundation and, you know, a few other burgeoning products, you know, projects.
Zachary Smith: And so there really wasn't culturally, I don't think-- I mean, were we using GitHub? I was not. We were using Subversion-
Erica Hughberg: Yeah ...
Zachary Smith: you know, and SourceForge. So, like, the infrastructure of community-based stuff was changing a lot, and the internet was changing a lot. Really-
Erica Hughberg: Yeah ...
Zachary Smith: the internet was pretty concentrated still with a couple of major telcos and, you know, a few content networks.
Zachary Smith: And we were all just trying to figure out how to make, make it more, like, usable. We knew that this content-driven web was coming. It was happening. Yeah. Things like video and, you know, distributed systems- Yeah ... were, like, barely h- but, like, people were starting to have, like, real content on the internet versus just email and, you know- Yeah
Zachary Smith: some whatever. And so I think, like, this was the- Like static
Erica Hughberg: websites, a forum.
Zachary Smith: Yeah. Yeah. Exactly. And so this was starting, and we were just struggling to figure out how to scale it, how we would help get peering [00:11:00] between networks- Mm ... how we would, you know, bring more of the internet around the world, which was still pretty US only at the time.
Zachary Smith: Yeah. Um, and so yeah, I think that was mainly what we focused on, and the answer to that for us was, you know, there was-- We were not the only one. I think, uh, there's a, there's a... Dan Rayburn has a CDN, dead CDN list. Okay. And I think I, I'm on it with three different CDNs we, I helped create. And so, you know, like CDNs- Yeah
Zachary Smith: was a way to how do you distribute content and deal with this- Mm ... internet scale load and improve routes around the world.
Erica Hughberg: We should probably also define CDN here for those who may not know what it is. As far-- Okay. I, I may mess up this acronym. I think you got it. Content delivery network.
Zachary Smith: Yes. I thought it was a cat GIF delivery network, but you could be right.
Erica Hughberg: Oh. Or it could be... Or is it, or is it content distribution or delivery? Is a D distribution or delivery?
Zachary Smith: Yeah, I think the common definition is content distribution network.
Erica Hughberg: Distribution.
Zachary Smith: Yeah. Um, I'm sure that's what Akamai has a patent on.
Erica Hughberg: Maybe we just get [00:12:00] another one then. Uh, anyway, the-- I remember that being very exciting actually.
Erica Hughberg: So not even in 2006, but like, uh, back then, like 2013- Mm-hmm ... I remember being very excited about that. That was when I learned about what a CDN was. Very exciting. Ooh. Do you know how fast we're gonna be able to get websites to computers now? Huh?
Zachary Smith: And JavaScript files, they're gonna download so quick. Oh,
Erica Hughberg: so exciting.
Erica Hughberg: People are not gonna understand the internet suddenly got faster, right? Like, we used to-- I, uh, in early in my career, like, all we worried about was trying to get a website to load under 500 milliseconds- Sure ... right, on someone's machine was very exciting problems, right? So that was really- Well,
Zachary Smith: now we're just trying to get an inference request to return in under 500 milliseconds, so.
Erica Hughberg: Well, dude, a inference request in under-- I mean, like, I thought we'd be happy if it takes seconds and not minutes. But and, uh, so it was [00:13:00] really-- Well, I, I ended up... So talking about networking, actually, we weren't, you know, networking is a shared interest here among us. But- I started getting really fascinated with what was happening in this AI space, actually because of all of this bluefield stuff.
Erica Hughberg: That's really what piqued my interest, right? So, so that piqued my interest because of that work that I guess people were doing, like the Tetra Datum people. There's- Who are those guys? Yeah,
Zachary Smith: whatever, you know, there's a thing.
Erica Hughberg: Yeah. There's a blog post, but the important part, the exciting part, and this is what gets me down a rabbit hole, right, was, okay, wait a minute.
Erica Hughberg: The-- What does inference requests really mean? So obviously, the internet was exciting at the beginning 'cause we started chatting. The computers were talking and talking and talking, and it just talked more and more and more. The more stuff there was on internet, the more it was to send around the network.
Erica Hughberg: Mm-hmm. Then now inference, quite interesting. I hadn't thought about this problem until it was spelt out for me, was if the [00:14:00] requests are bigger and bigger and bigger, decryption and encryption maths starts getting more and more like, you know, complicated- Resource intensive ... to do. Resource intensive is a good way to put it.
Erica Hughberg: Yes, much better than complicated. Uh, so I was like, okay, and then now how am I in this rabbit hole, Zach? We're gonna try and explore this rabbit hole. The rabbit hole then became, wait a minute, say we have this entire AI era on our hands- Mm-hmm ... that is-- Everyone's been like, "GPUs, how exciting. What a wonderfully, um, you know, um, multi-use platform."
Erica Hughberg: How do I have them?
Zachary Smith: Do you have any GPUs? Right. Can I borrow the GPUs?
Erica Hughberg: So the GPUs were like really hot, hot stuff, right? And then I started to realize that the ripple effect of all of this stuff is beyond the GPUs and the TPUs and all of that stuff, right? So yes, we have to train models, and then we have to do inference to serve [00:15:00] them.
Erica Hughberg: Then I was like, wait a minute. So that's how you got me interested. I feel like part of this is- So it's my fault ... on you. How I got down this rabbit hole, so that's why you have to be here to get me back up from it.
Zachary Smith: Well, this is a good tie-in because in 2006, I was worried about things like how would we possibly do video on the internet?
Zachary Smith: I mean, like watching like something on your phone, a video, we all laughed at this, right? Yeah. Mm. There's simply not enough capacity, and nobody could do that, right? Um-
Erica Hughberg: Streaming video, was that a crazy
Zachary Smith: idea? Exactly.
Erica Hughberg: Yeah.
Zachary Smith: Yeah, it was a crazy idea. Um, and so like that's what we were worried about. 2026, what I'm worried about-
Erica Hughberg: Yeah
Zachary Smith: you know, I would say there's a lot of things I'm worried about. Maybe it's all that gray hair. Maybe it's just the world's more complicated. But, um, I would say two things really, you know, kinda keep me up. Yeah. One is broadly speaking, um We're just, we're on a path right now where the physics of compute hardware, um, Moore's law as it were- Yeah
Zachary Smith: like, uh, you know, [00:16:00] and the, and the physics of smaller nanometers and transistors have kinda run out. And so we're just putting more of them. We're stacking more of them, and that's why you've got computers and networks that are consuming not kilowatts of hardware, but, you know, hundreds of kilowatts of har- you know, power- Yeah
Zachary Smith: for a single computer. In some cases, a rack of computers, sorry. Yeah. And, and like this is causing a massive, huge resource challenge, and we'll, for all of this, you know- I would say positive both cultural and economic things that we have with technology, are we willing to basically consume all of the resources of our planet to power more GPUs in bad ways?
Zachary Smith: Yeah. You know, or will we do it better? And will we use the billions of dollars or trillions of dollars that we're investing as a [00:17:00] society in infrastructure around technology for the benefit of our communities and our planet? Um, I think therein lies the riddle. Um, and I believe it's possible, but it's not a, a, a, a guaranteed outcome by far.
Erica Hughberg: Mm.
Zachary Smith: Um, and so, so that's one thing, you know, I worry about. I do
Erica Hughberg: f- uh, on that topic of like resources, I just find it very fascinating that AI probably consumes more drinking water than I do. Like as in if I had a little agent, I'm not sure I drink enough water. That's probably my problem. But I was just fascinated at the cooling because you don't want to have like bacteria growth.
Erica Hughberg: You need to have like actual drinking water in those data centers. Like or as- Yeah ... good quality as like drinking
Zachary Smith: water. You need to have clean water for- Yeah ... yeah, evaporative cooling.
Erica Hughberg: Um- Yeah, you don't want like stuff growing, like mold.
Zachary Smith: No. I mean, th- there's all sorts of resource side that, that can come down to it.
Zachary Smith: Luckily, I think like as a species, we [00:18:00] remain incredibly innovative. Um, unfortunately, I think that the world of computer hardware has a problem in it- Mm ... um, which is that the incentive systems of the people who make most of the things- Okay ... are not how to make it last longer.
Erica Hughberg: Okay. So tell me more about it.
Erica Hughberg: Right? What does that mean? Why are they not incentivized to make it last longer? Because they wanna sell
Zachary Smith: more? I've had many good relations with my chip maker friends, but they only make money when they sell you more crap.
Erica Hughberg: Yeah. Can you recycle- And- ... these things to build new things easily?
Zachary Smith: In theory you could, but the incentive systems are not made up for that.
Zachary Smith: And so- Okay ... what, what we've done over the past 10, 20 years of p- as things move from consumer in my office, I saw the computer, and then I'd have to get rid of it when I wanted to be done. Mm. Now you don't even see it. Sit in some data center in Ohio. Yeah. You just turn it off and delete it.
Erica Hughberg: Yeah.
Zachary Smith: And so the, the, the, the association to that, [00:19:00] the physical just massive amount of waste and just basic things like steel and racks and cables and things is unbelievable.
Zachary Smith: And there's not really a mechanism on which you give it back and like, "Oh, I should pay to recycle this. Where would I even do that?" Why- Yeah ... hey, Intel made this thing or NVIDIA, let's give it back to them and, you know, they'll, they'll turn it into a new one. Yeah. I think if you look in our consumerized world, we've done a good job at that.
Zachary Smith: Um, I like
Erica Hughberg: to point- Like Apple is very nice at like incentivizing you to give your stuff back to them.
Zachary Smith: Yeah. And you know what they do with it in most cases.
Erica Hughberg: What do they do?
Zachary Smith: They don't take your three-year-old iPhone and sell it to somebody else for less money. They take your three-year-old iPhone, take it apart, recycle like- Yeah
Zachary Smith: 99% of the things- Oh ... and make a new iPhone.
Erica Hughberg: Well, that's pretty good, right? That-
Zachary Smith: Which is pretty smart- Yeah ... 'cause you're gonna design the iPhone so that one day you're gonna get it back, right? 'Cause what they wanna do is sell you a new iPhone, and they wanna get rid of the supply of the old one. But that's [00:20:00] not the case with a NVIDIA, who never wants to see those chips again, ever.
Erica Hughberg: That's really fascinating, though, about the whole, like, if you think about if you know that, okay, people normally have the phone two to three years. If I design it in a way so it's also easy for us to take it apart and reuse the things, so then that becomes a design principle, just a foundational design print- 100%.
Erica Hughberg: Wow. That's so fascinating.
Zachary Smith: The one, the one computer manufacturer who's really, in my opinion, great at this is Cisco- Mm-hmm ... which is really, you know, targeting towards high 90s of recycling their own supply chain. And they learned this from the early days of the internet.com boom, where they wanted to get rid of the gray market in switches- Okay
Zachary Smith: right? Um, or at least that's my-
Erica Hughberg: What is gray market? What does that mean here, gray market?
Zachary Smith: Well, let's say you sold a bunch of switches and then- Mm ... those companies went out of business, and then there was all these switches just sitting around.
Erica Hughberg: Hmm.
Zachary Smith: You know? Well, what do you do? You, they show up on eBay.
Zachary Smith: They show up- Oh ... on somewhere else. Somebody's selling you a used switch, and that switch might've cost $5,000 new, and now it costs $200 'cause somebody's just trying to get rid of it. And so Cisco, you know, has done a great [00:21:00] job. They want that old switch back, right? Hmm. Um, because they will recycle it and take it out of the market.
Erica Hughberg: Yeah.
Zachary Smith: Um, and I think that that kind of, which, you know, in some ways just helps to manage their secondary market- Hmm ... um, and- But also- ... kinda deal with unlicensed reselling- Yeah ... you know, is the same thing that I think Apple's done, which is like, it's better for us to have a circular supply chain-
Erica Hughberg: Yeah
Zachary Smith: because we don't need a billion old iPhones sitting around dragging down the price of our- Mm-hmm ... next iPhone 19. We'd rather actually take those back and make- Yeah ... iPhone 19s out of them, if it, if that's possible.
Erica Hughberg: Yeah. Okay, so since, this is all fascinating. Okay, all right, there's so many thoughts in my head as you- But
Zachary Smith: because it's in a faraway data center that you never see-
Erica Hughberg: Yeah.
Erica Hughberg: You don't feel it. But I do give my, my resources back. I do go use some stuff in GCP, and I give it back to GCP. I'm very kind- Yes ... like that.
Zachary Smith: You're super nice about that. We appreciate you.
Erica Hughberg: But okay, so the promise [00:22:00] of the cloud, right?
Zachary Smith: Mm.
Erica Hughberg: Okay, I'm gonna hope- so we talked about, like, these physical limitations, right?
Erica Hughberg: But the promise of the cloud is there's infinite compute.
Zachary Smith: Ha. That's so funny. It's just somebody else's computer, Erika. You know that.
Erica Hughberg: I know that, but I also don't have to worry about it if I need more, right? I guess, but that's like, that's what it felt like. I remember h- like- Yeah ... talking to my friend, uh, many, many years ago, like, as we were at university and we were doing all of these interviews for internships back then, and she came to me, "Erika, what is the cloud?"
Erica Hughberg: And I was like, "Hmm, how do I explain this?" Right? And But I, I think it's so fascinating though, like you're renting space, right? It's like you're renting space- Yeah ... somewhere. Um, but-
Zachary Smith: And there are actually really positive- Yeah ... utilization effects from shared resources in hyperscale compute farms or centralized compute where you can say, [00:23:00] "Hey, Erica gave that VM back.
Zachary Smith: I'm gonna use it for somebody else very quickly." Yeah. D- And that's awesome, um, because that would probably not happen if it was sitting under your desk or sitting in your back IT closet. Um, so I think those are really-- we've really benefited from those, um, overall in society, and I don't wanna say that, um, cloud companies are, uh, you know, bad at this.
Zachary Smith: They're doing a good job. Yeah. But the structural issues are, are real, which is we're in this kind of like more, better, new upgrade. I like to akin it to the- Hmm ... another big industry. Let's- Okay ... let's use a completely different industry of
Erica Hughberg: hardware.
Zachary Smith: Mm. Love it. Okay? Love it. Which is like, have you ever driven in a car or sat in a car?
Erica Hughberg: Oh, just this morning.
Zachary Smith: Fantastic. Okay. Well, this is an industry that is basically really geared up to make sure that you can get a new car. They s- make new models every year, right?
Erica Hughberg: Yeah.
Zachary Smith: 2026 thing, 2027. Oh,
Erica Hughberg: I li- I love the car industry. It's like a such a fascinating industry. Yeah, okay, continue. Yes.
Zachary Smith: So it's the same car, but there's like a bunch of different ways- Yeah
Zachary Smith: that you, [00:24:00] Erica, could participate in owning or having or using that car. Yeah. You could rent it from Hertz. Yeah. You could go to the dealer and do a l- three-year lease. Yeah. You could buy it with cash. You could- Yeah ... finance it with the little loan they'll give you. Yeah. Heck, if you have a N- Like, let's say you're buying a Honda, but you have a Toyota.
Zachary Smith: They'll even take back your old Toyota.
Erica Hughberg: Yeah.
Zachary Smith: That's pretty nice of them. They might not give you a great price, but there's- Yeah ... this thing, Kelley Blue Book. They'll give you a price, and then, you know, they'll deal with, like, selling it used car, scrapping it- Yeah ... whatever they're gonna do, right? And I think that the, the car industry, if you look at, you know, what-- Let's use, uh, BMW.
Zachary Smith: Um, BMW has a lot of dealers. Uh, those- Yeah ... dealers have a bunch of brand-new BMWs on their lot. Um, and when you walk in and say, "I would like that BMW," um- Yeah ... you're offered some things like BMW financing or a BMW lease 'cause the people who made the thing-
Erica Hughberg: Yeah ...
Zachary Smith: are like, "I have the capital to finance the thing for [00:25:00] you, and when you're done with it, we have ways for you to give it back to us because we're gonna want you to buy a new thing."
Zachary Smith: And in the data center industry, like NVIDIA, they don't provide NVIDIA financial services to- Mm ... your dealership or your end user. You know, they, they kind of say like, "No, maybe that's Dell's problem to deal with," and, you know, they push that down the line. And there also isn't this kind of like, "Hey, when you're done with it, you just bring it on back.
Zachary Smith: We'll make sure- Yeah ... you get a shiny new one." And, um, I think that that's a real-
Erica Hughberg: Lease it. Yeah. Lease hardware.
Zachary Smith: Well, we do. We have, like- Yeah ... rental car companies called, like, cloud computing businesses and- Yeah ... stuff like that, but we don't have this Kind of realization of a mature, regular cycle of upgrades.
Zachary Smith: Um, and I think that's because over the past, you know, couple decades we've gone from like, "I will buy the mainframe and we will keep it for as long, forever. We will build the building around the mainframe," to this kind of like TikTok cycle with Intel, you know, where we're doing- Yeah ... you know, every four to five years, to now like GPU infrastructure.
Zachary Smith: I mean, [00:26:00] Jensen's goal is like an 18-month cycle right now.
Erica Hughberg: Yeah. '
Zachary Smith: Cause get- And the goal of the, the new version is- Yeah ... to make the old version really irrelevant.
Erica Hughberg: Hmm.
Zachary Smith: And so- Okay ... I think that's setting up this ... 'Cause you don't say like, "Here is- Yeah ... this brand-new thing. We're also selling the old thing."
Zachary Smith: They're like, "No, no, no, that one's no good anymore. This one's three times better." Yeah. And I think that's really the problem, is that we're not set up, 'cause NVIDIA doesn't have incentive mechanisms that say- Mm ... "Hey, if Erica kept using the two-year-old model because she's gonna drive that Honda for 200,000 miles 'cause she just like, it's a fine car for her and everything's okay," there's really not an incentive mechanism for NVIDIA to have that happen.
Erica Hughberg: Yeah. Okay, so this is quite fascinating, the whole like non-circular economy, because when I talked about the car stuff though as well, see, I got quite interested in ... Because I think we all know that we were told that the Apple, the n- next Apple iPhone is going to be more expensive. [00:27:00] But why? Because memory.
Erica Hughberg: But not because- Mm ... the data centers are taking the memory that should've gone in my iPhone. They, the factory, so pr- you know, they are producing memory. The place
Zachary Smith: that makes
Erica Hughberg: the memory. They are realizing that they have better profit margins selling memor- different type of memory, but they just rearrange their production line to produce this other memory for data centers because data centers-
Zachary Smith: Yeah, and it turns out it takes a long time to make a new production line.
Erica Hughberg: Yeah. Yeah, apparently. It's
Zachary Smith: very complicated.
Erica Hughberg: Uh, it's like all of these, like, physical constraints, isn't it? Like, we're living in this virtual world, but we're still constrained by reality. So-
Zachary Smith: Yeah ...
Erica Hughberg: this, this, right? And then... Okay, this is part of my weird rabbit hole, Zach. So can you-
Zachary Smith: I'm here for you.
Erica Hughberg: Great.
Erica Hughberg: Thank you. But also, you'd brought me back to the car stuff. So I learned some cool stuff from my brother, who works at Volvo Group. And one [00:28:00] thing, and he was mainly interested in Volvo trucks, to be fair here, 'cause Volvo produce a lot of different machines, but the trucks, for example, right? They had to innovate to make sure that, because the demand for different type of trucks varied, right?
Erica Hughberg: Mm. So how can you quickly turn a production line from doing one type of truck into another type of truck as quickly as possible to meet the demand?
Zachary Smith: Mm-hmm.
Erica Hughberg: So they were able to, like, really quickly flip over a line from producing one product to a different product. Mm. And that was all about, like, how...
Erica Hughberg: Anyway, it was quite impressive. But And I don't wanna quote him 'cause many years ago he told me how quick it was, but it was, like, ridiculously fast. We're talking, like, 24 to 48 hours, right? To switch over. They could
Zachary Smith: change the assembly line to, like-
Erica Hughberg: Yeah ...
Zachary Smith: a different thing.
Erica Hughberg: Yeah. Well, a similar thing. Same-
Zachary Smith: Sure
Erica Hughberg: you know. But it's not like radically- Yeah, model this to model
Zachary Smith: that.
Erica Hughberg: Yeah. Not a radic- I don't think they can go from, like, truck to tractor, right? That would be [00:29:00] quite crazy. But, you know, within- I get where you're going.
Zachary Smith: Yeah. Yeah, pretty impressive, right? That you could do that.
Erica Hughberg: Yeah. Because apparently that was an engineering feat, suppo- you know, the way I understood it.
Erica Hughberg: I was nodding and was, I was very impressed by this because-
Zachary Smith: Mm, mm,
Erica Hughberg: mm ... supply chain stuff is apparently very interesting.
Zachary Smith: Yeah,
Erica Hughberg: totally. Make sure you c- supply is good. Mm-hmm, mm-hmm. But then, okay, back to the memory problem. That was then also another part that I started, apart from DPUs and other exciting things, like memory, all of these other things.
Erica Hughberg: This seems to be affected by this era, right? Of the things is cha- Are things changing? Like, okay, so you are much closer to the world of infrastructure than I am, 'cause I'm literally just watching BBC, like, news stuff about it the other day. Um, probably there was an entertainment value to it as well.
Erica Hughberg: Anyway, are, are people, [00:30:00] like, scared... You know, you talked about this 18-month thing that NVIDIA's like.
Zachary Smith: Mm-hmm. Mm-hmm.
Erica Hughberg: Are people scared of over-committing and buying things maybe? Like, are they-
Zachary Smith: I mean, I think they are to some degree, but, um, you know, right now there's such a kind of high demand for the, like, whether it's FOMO, like you're an enterprise- Yeah
Zachary Smith: and you're like, "We gotta be in on this, otherwise we might get left behind." Um, or it's, you know, service providers who are seeing a huge demand, right? And they, they're basically like- Mm ... "If I don't get this, I might not get the next thing," you know? And so there's- Yeah ... you know, dynamics of the end user offtake world and supply that you wanna make sure, like, "Hey, no, I wanna take this."
Zachary Smith: But I think, like, the m- you know, the, the challenge that we're just running into is these things are incredibly complicated, right? We have a-
Erica Hughberg: Yeah ...
Zachary Smith: pretty amazing global manufacturing ecosystem that can produce, like, incredibly complicated devices through- Yeah ... you know, specialization basically, right? And whether that's TSMC [00:31:00] just only specializing in two nanometer fabrication processes, or, like, Samsung making its, like, DDR6 memory, you know, and fine- Right
Zachary Smith: there's other companies who make, you know, certain power supplies. The, what happens is the market doesn't- allow, because it's gotten so complex and so expensive to get in this high manufacturing, that it doesn't allow for there to be 20 manufacturers who make trucks. There are, like, two or three that can possibly make the memory, and so you are really dependent upon this.
Zachary Smith: Like a, what do they call it? A duopoly kind of market. And that's, I think, where we've gotten to is in so much of this supply chain, it's really down to one supplier or maybe two, and there's not a lot of excess slack in the system
Erica Hughberg: And I would assume that it would take-- it would be incredibly expensive if anyone wanted to enter this production market, like they wanted to start producing this [00:32:00]
Zachary Smith: Or impossible.
Erica Hughberg: Or impossible.
Zachary Smith: I mean, maybe all things are possible, but like it could be impossible because you just can't get enough of the machines you need to buy- Yeah ... to power the factory or because you couldn't get the workforce- Get the raw materials ... because like, yeah, you know, where would you get it? 'Cause like, you know, maybe you- it's already being locked up by the current supplier.
Zachary Smith: So like I think talent is a big part in manufacturing as well- Oh, yeah ... um, which is just are there enough material science engineers who understand and have actually built- Right ... fabrication processes? And the answer is like, no. We have the same thing in the nuclear industry right now, right?
Erica Hughberg: Dude, that's so interesting.
Erica Hughberg: Okay, I love this weird thing. So like you find a, like a role or like a specialization that there may not be like enough of. Okay, so material like scientists or people who understands this material, whatever it is. Sure. Right? So- I
Zachary Smith: think a, a really good example right now is nuclear, right? Okay. Japan's going through this, which had- [00:33:00] Hmm
Zachary Smith: like the US has had for a long time a lack of a nuclear industry from a s- from a human perspective- Hmm ... because we haven't really had a consistent we're building nuclear plants industry. Japan did, and because of that had robust people graduating from like, you know, every year from universities with degrees in nuclear physics and all the things necessary to like, you know, build and run nuclear plants.
Zachary Smith: Yeah. And then after Fukushima, basically people stopped going to school for that because they weren't building them, and now they don't have an industry and it's super hard to restart it. Like they're really struggling and like if we don't restart it soon, we're not gonna have the talent pool 'cause people will stop teaching it.
Zachary Smith: They won't- Yeah ... have the university pipeline. I think that's what we're dealing with here, like they built the Vogtle plant down by you in Georgia. You know, they basically had to train an entire generation of workers to build one plant. That's not really very sustainable. It's very expensive. It goes away.
Zachary Smith: People don't wanna get into the business, et cetera. So I think therein lies some of the challenge with these [00:34:00] things- Mm ... is just labor force benefits.
Erica Hughberg: It's also like, so if you got into it, there's not gonna be like loads of projects you're gonna be on in your whole career because there may be not gonna be-
Zachary Smith: Or maybe there's only one or two companies or very concentrated, and you just don't know about it, right?
Zachary Smith: I don't know how many of your friends are going to, you know, material science, chemistry degrees or
Erica Hughberg: whatever, but you know. I literally asked like one friend who went into that. She did start her own company, and uh, she was in the Forbes 30 Under 30 at one point, which was very cool.
Zachary Smith: So it worked out pretty good for her.
Erica Hughberg: Yeah, it worked out pretty good for her. But like, I mean, I don't know that many people. Like I literally can think of one person I know who went into material science. And And it's-- But do we s- so is that-- Okay, I'm like thinking now. Hard work's hard
Zachary Smith: business.
Erica Hughberg: If you were, if you were to like advise someone, this is literally not a question I thought to ask before, but [00:35:00] now we're here.
Erica Hughberg: If you were talking to like a person who was choosing- My, my
Zachary Smith: 16-year-old son who's- Okay ... deciding what he wants to do for university.
Erica Hughberg: Okay. Okay. Uh, do you decide at 16? Wow. Okay. But yeah, someone around that age who's thinking about what they wanna do, would you, what would you advise them to look at? Apart from do something you're interested in, you know.
Erica Hughberg: Assume you're interested in-
Zachary Smith: That's the problem, right? Yeah. 'Cause like s- I, I presume, 'cause I don't know, I have the attention span of a gnat, so like if I was going to, uh, uh, looking back at myself- Yeah ... you know, I'm not sure I could put myself into an industry like, oh, it'd probably be good in to get in like, I don't know, nuclear energy and all that stuff, and, you know, go to school for 15 years to do that.
Zachary Smith: Or, I don't know, man, like you gotta love it, right? Yeah. Or you gotta really think that it's, yeah, something on the other side. I think, like, that's probably... We don't do a pr- we're gonna have- Yeah ... this labor force problem, I think, you know, where we've seen a lot of this with, with AI where you don't need junior roles, you know?
Zachary Smith: And I, I kinda think of it as electricians, which is a major bottleneck in our [00:36:00] current supply chain. Okay. Which is just the trades, right? Um- Yeah ... concrete, electrician, you know- Yeah ... plumbing. This is all part of data centers or building- Yeah ... you know, energy systems or related, and we don't have enough of these people.
Zachary Smith: And the thing is, like, you don't wanna, like, get a junior... Like, you would do it in s- in software. You'd be like, "Well, we'll just get a junior engineer who can, like, you know, start to ship code. But if we, if we mess it up, we'll, you know, we'll just fix it."
Erica Hughberg: Yeah.
Zachary Smith: But, like- Just keep fixing ... you don't want a... Yeah, you don't want a junior- It's just-
Zachary Smith: like, electrician working on the power grid. You want a master electrician working on the power grid, right? Like, and so how do you become a master electrician? Mm. Well, we have an apprenticeship program- Yeah ... through the union that allows you to, like, shadow a, an electrician- Yeah ... for many years and, and get paid while you're doing it- Yeah
Zachary Smith: so that way you can become a master electrician. Yeah. We don't do that very well in our industry.
Erica Hughberg: No.
Zachary Smith: So I think that would require us to, you know, have apprenticeships or long-term training programs that, that, that allowed people to go [00:37:00] into those harder careers and become experts.
Erica Hughberg: Yeah.
Zachary Smith: Um, so anyways.
Erica Hughberg: Oh.
Zachary Smith: We've gone into labor force dynamics now, Erica. Yeah, which
Erica Hughberg: is very fascinating.
Zachary Smith: I, I thought we were gonna work, talk about hardware.
Erica Hughberg: Okay, yeah, but this is the people behind the hardware, and people- ... aren't they interesting? Uh, okay. But okay, what now? Okay, come back to, like, Erica's weird shit. That, okay, look, I said the bad word.
Erica Hughberg: Look, I told ya. I told you. Come back to the
Zachary Smith: stuff.
Erica Hughberg: I told you I say bad words. Anyway, I, I'm wondering, I have so many questions in my head, of course. But, so I've identified the, the whole memory thing, which- Mm-hmm ... I think a lot of us have identified is, like, a challenge.
Zachary Smith: Yeah. Making your iPhone more expensive and all the other things.
Erica Hughberg: And I, I now think I should've bought an iPhone last year. Anyway, here we are. And, um, like a new one. I have a, I have a 16. So anyway, now... And then we have the, uh, DPU [00:38:00] stuff, which was quite interesting. That, that was useful and exciting. And, uh, the cooling, the cooling, by the way, very fascinating. I've now understood that AI makes things a lot hotter than things used to
Zachary Smith: be.
Zachary Smith: Yeah, and that comes back to my previous point where the easy wins of we will go on physics journeys from You know, 10 nanometer to 5 nanometer- Mm ... which means our transistors got, you know, half as, as thick, which means we could put the same amount of 'em- Mm ... or we could on half the amount of space or- Yeah
Zachary Smith: we could do double the amount in the same amount of space. Well, we're kinda done with that. We're going to 3 nanometer and 2 nanometer now, or 2 nanometer-ish.
Erica Hughberg: Yeah.
Zachary Smith: Which means the only way to get more performance is to add more transistors, and, and, and the way they've done that is start to layer vertically.
Zachary Smith: Yeah. So you have layers of that in both memory and, you know, in CPUs, obviously GPUs are, are that same way. And so every part of the stack is just layering up with more transistors, which means [00:39:00] more power.
Erica Hughberg: Yeah.
Zachary Smith: And I believe there's a little physics rule about for every- ... you know, reaction there is an equal- Yeah
Zachary Smith: opposing reaction. Yeah. Which I believe when you put a bunch of electricity through something, it gonna get hot.
Erica Hughberg: Yeah.
Zachary Smith: Right? And, and so you gotta, you gotta remove that heat. And so I think therein lies, you know, we are using more power because we're kind of- Mm ... running out of good physics, and so we're just- Yeah
Zachary Smith: stuffing more. One of the things that I think about in our industry right now is we have a big density issue. Okay. Because not only are these things very power- Mm ... hungry, they're getting really dense. You know? Mm. And like a typical, um, computer-
Erica Hughberg: Yeah ...
Zachary Smith: you know, five, 10 years ago would be like 4 or 500 watts-
Erica Hughberg: Yeah
Zachary Smith: um, of power draw, and now you've got computers that are, I don't know, 2,000 to 4,000 watts- Yeah ... for one computer or more.
Erica Hughberg: Yeah.
Zachary Smith: Right? And th- but the real issue is that, oh, well you're like, "Okay, well I'll just put like... If I can't cool that density or I can't power that [00:40:00] density, I'll just put like two computers in one rack instead of 20 computers in a rack."
Erica Hughberg: Okay.
Zachary Smith: You know? Get that going on. And it's kind of like, yeah, I'll put like one here and I'll put a bunch of space, and I'll put another one up here. For
Erica Hughberg: airflow.
Zachary Smith: So I could... Yeah, I got more airflow. Excellent. I lo- I love where you're going with this. And more airflow, which is a very inefficient but cheap and flexible way of doing cooling.
Zachary Smith: Um, you know, so we could have like one up here and one down here. The problem is we're running out of networking, so these things are moving data between them so fast- Yeah ... that we have to put the cables right next to each other- Yeah ... using some sort of a copper back plane or something else. So we can't just- Yeah
Zachary Smith: string the computers all across the room. Yeah. We gotta have them in the same place. So now we're seeing this extremely dense rack- Yeah ... design where instead of maybe a f- like when I was at Equinix for many years- Yeah ... the average rack density of Equinix was about four and a half to five kilowatts per rack.
Zachary Smith: Yeah. Now you're seeing designs of 100 or 120 kilowatts per rack just a few years later. I mean, that is [00:41:00] incredibly hard to power and incredibly hard to cool, and you think of unintended consequences.
Erica Hughberg: Yeah.
Zachary Smith: We can't even support the floor loading of those. They're running to 6 to 8,000 pounds of weight per rack.
Zachary Smith: Most buildings can't handle, even in their ground floor on the concrete, that level of density of, you know, pounds
Erica Hughberg: per square foot Well, I had never thought about this problem. This becomes- Well- ... like a building engineering problem as well. Wow.
Zachary Smith: 100%, right? So we don't even have the buildings capable of holding that kind of floor loading.
Zachary Smith: So this is the, this is the array of events that-- And I wanna give, like these are all the issues right now where people are like, "Why can't I get GPUs," right? And where I'd like to like say that there's been this kind of false paradigm, what do you call 'em? Mm-hmm. I don't know. There's must be a good phrase for this, but like-
Erica Hughberg: Okay
Zachary Smith: either like- It's
Erica Hughberg: okay.
Zachary Smith: Go with that ... we'll have to just build more buildings, you know? Mm-hmm. Or like we'll have to just have less AI. Mm-hmm. And I think that that's a false choice. I think that there's a third [00:42:00] choice, which is like well, we might have to figure out how to do it better.
Erica Hughberg: Yeah. Well- Right? ... I think that's what you were saying earlier though, like humans are pretty good at innovating, especially when things are constrained, which is really fun and exciting.
Erica Hughberg: Like coming up with like-
Zachary Smith: New ways ...
Erica Hughberg: creative, yeah, creative ways of doing things, and it's quite fun and exciting to see where it may be going. And oh, d- like I'm s- like I'm so fascinating about the whole like weight part. But something
Zachary Smith: that- You wouldn't think about that. One of the biggest problems I always had were people were like, Facebook for a while was designing these computers- Yeah
Zachary Smith: called OCP, and everywhere in the world we use a 19-inch, I don't know why, but there's probably a centimeters, millimeters association for that, 19-inch rack.
Erica Hughberg: Mm.
Zachary Smith: Which all computers will slide into a 19-inch rack. Okay, yeah. Everybody's using it around the rest of the world. Super good. Facebook decides they're gonna use Open Compute, they're gonna do 21-inch racks.
Zachary Smith: That's interesting. Okay. I'm sure they had a good reason. I don't know. Hmm. Wider thing, they got to fit more [00:43:00] stuff in it. I don't know. But you know what? There's a lot of data centers I've been into where you can't turn that around the corner 'cause you get it out the elevator- Oh ... and you're, you're, you're just trying to move the rack and it's not possible to put it in the darn building , you know?
Zachary Smith: So things like this you don't think about, um, or you decide now we'll just have to build our building around a 21-inch rack now.
Erica Hughberg: So. That's so interesting. Like d- that sounds so fascinating, just like even-
Zachary Smith: Real world problems. That's why people wanted to go to the cloud, so we don't have to see it. It's too complicated.
Erica Hughberg: But it's fun. I also ended up down the little other rabbit hole about like power-
Zachary Smith: Hmm ...
Erica Hughberg: and power in the sense of, okay, this may be f- m- you pr- people probably know about this and can explain it better than I do. But when we train AI models Apparently, all this TPU stuff pulls energy like as if it's-- It goes like peak, valley, peak, valley- Yeah
Erica Hughberg: peak, valley. [00:44:00] Right?
Zachary Smith: Yep.
Erica Hughberg: So as far as I- The
Zachary Smith: thundering herd problem.
Erica Hughberg: Yeah. It's like you're stamping. So for anyone who doesn't know, the Millennium Bridge in London... So when I start university, I start, start university in London, computer science. The lecturer, the welcome, like, you know, "Welcome to computer science-" Welcome to university.
Erica Hughberg: Yeah. He showed a picture of the Millennium Bridge, and he said, "When, when, um, um, civil engineers make a mistake, this is what happens." The Millennium Bridge had a problem, you see, because when people walked on it, and people apparently-
Zachary Smith: Booj, booj, booj ...
Erica Hughberg: yeah, it started to move Yes. Quite a lot. Uh- Yeah. And, uh, so, and then he showed me the next- The weight
Zachary Smith: displacement made it move more, and it was zoor.
Erica Hughberg: And then he showed a picture of a blue screen, and he said, "When computer scientists make a mistake, this is what you get." Right. So he's like: Don't [00:45:00] worry about it. You're gonna make mistakes. It's gonna be okay.
Zachary Smith: It's gonna be all right. Move fast and break things. We'll just fix them later.
Erica Hughberg: So he's, he basically told us the stakes are not that high for you.
Erica Hughberg: But what I then found very fascinating, that computer people can cause quite interesting physical problems.
Zachary Smith: Oh, yes.
Erica Hughberg: And I thought that was really... And then I learned about, and now there's batteries to try, like, well, no, batteries may be the wrong term for it, but pow- like, on the rack- Yeah ... like, holding power so that it has a more, like, smoothing out the whole, like-
Zachary Smith: Yeah.
Zachary Smith: Yeah. No, you have to smooth out the currents, and you have to put, you know, areas to absorb that energy in certain places or to store it. I mean, this is, Erica, where I think there's... This makes me-- I'm an optimist at heart, otherwise I wouldn't be an entrepreneur, and so my fear is that we will, you know, put all this money into things and not realize really necessary societal benefits that we have to do, and there's not, like, [00:46:00] unlimited money or desire from society to invest unlimited money in everything.
Zachary Smith: And so, like, I think we have to go through some pretty serious infrastructure upgrades related to climate change and other things in our, in our society that we really need, and this is an amazing opportunity to do so. Yeah. Right? Which is a demand of hundreds of billions or trillions of dollars going into mainly physical things.
Zachary Smith: You know, we say AI. What we really mean is, like, electrical substations, data centers, power cables- Yeah ... battery systems, you know, new generation, uh, of energy. Wouldn't it be amazing if we did that in ways that was really helpful for our society, you know, of the coming decades? I think so. And so that's where I'm optimistic, is that we as engineers and we as, you know, product builders and people throughout this can work towards better solutions so that those batteries in the rack can maybe do something for society.
Zachary Smith: Or even a better one, I lo- I love this one-
Erica Hughberg: Okay ...
Zachary Smith: is like, hey, these, these racks, we've [00:47:00] had a huge problem over the past couple decades where the computers were egressing heat, but we were blowing air around it. Yeah. And when you blow air around it to do a heat exchange, the air is not that hot, okay? Mm-hmm.
Zachary Smith: We call it low-grade heat in the data center world. Okay. Okay? So you're egressing heat, but the, the heat differential between the air and the-
Erica Hughberg: Yeah ...
Zachary Smith: kind of exhausted, you know- Mm-hmm ... is not that big.
Erica Hughberg: Okay.
Zachary Smith: But now we're doing liquid cooling. We're taking very, very high-end heat. We're maybe turning it into steam- Yeah
Zachary Smith: actually,
Erica Hughberg: you know, because- I think that's how some water des-
Zachary Smith: We evaporate, but if you use it in an enclosed loop system-
Erica Hughberg: Yeah ...
Zachary Smith: what do you got? You got energy. You got heat. Yeah. Really, really high-grade heat. Now you can do all sorts of things with that heat, right? And so-
Erica Hughberg: Heating houses ...
Zachary Smith: heating houses, municipal systems.
Zachary Smith: I was just in Denmark with a firm- But then people- ... that does this.
Erica Hughberg: Oh, yes. I mean, I'm from Scandinavia. Um, the whole, like, wonderful- But they have
Zachary Smith: municipal heating,
Erica Hughberg: like- Yes ...
Zachary Smith: you know, networks.
Erica Hughberg: All the... Yeah. It's very fascinating, right? I, we'll talk [00:48:00] about it 'cause I love this. T- tell people what this is, the communal heating.
Zachary Smith: Well, New York has one. It's just kind of-
Erica Hughberg: Ooh ...
Zachary Smith: not great. It's steam- Okay. Yeah ... and that would be not great for a variety of reasons. But we do have one, but we're, you know, unique for North American cities. Mm. Um, but like in Scandinavia, they'll have municipal cooling, like heating loops, right? Yeah. And you can basically have medium-grade heat water or related moving throughout different buildings.
Zachary Smith: Yeah. Um, the, and the, and the city would provide that as a heat source, right? And then- Yeah ... in each house you can have a heat exchanger or related, right? Well, what I- just imagine if you had a long-term client who was just generating a ton of heat- Yeah ... for you called a data center next door. Yeah. And you were, you figured out your incentive mechanisms- Yeah
Zachary Smith: so that way the s- the city could rely upon the data center for high-grade heat-
Erica Hughberg: Yeah ...
Zachary Smith: for a long period of time. You might not have to build a new boiler or natural gas plant or related because you can do that. And I think that means the data center saying, "Hey, we'll [00:49:00] guarantee you this amount of heat- Yeah
Zachary Smith: for 20 years," not just like, "If I have it, I'll give it to you." Yeah. Like, and I think therein lies some of the things that we need to do to work together, because that could really make, that people would see a data center as an asset to their community-
Erica Hughberg: Rather
Zachary Smith: than a problem ... which is usually an extraction to their community, right?
Zachary Smith: Yeah. Which is you're using my energy, and then the AI bits travel to somewhere else, and I don't get to benefit from that. Yeah. But maybe if I could see an output like, oh, you helped us by solving our, you know, municipal heating upgrade program. Thank you. Yeah. Our electric bills went down. We love you.
Zachary Smith: Thank you so much. Yeah. Right? I mean, at least that's how I think if, i- digital is a big part of our society-
Erica Hughberg: Yeah ...
Zachary Smith: we need to figure out how the inf- infrastructure investments around digital are also benefiting and aligned with society.
Erica Hughberg: This is like-
Zachary Smith: That's the optimist view of hardware.
Erica Hughberg: Yeah. Okay. So I...
Erica Hughberg: Coming to, like, looking forward then, looking forward into the [00:50:00] wonderful world and what's possible, what are things that you are excited, apart from how can we live in symbiosis with AI, um, right? What are, what are, like, things that you may have learned about recently or heard of that you think makes you even more optimistic about the future?
Zachary Smith: Well, I'm not gonna go into the things that make me scared and, and whatnot. I'll go into the things that I think are good, right? I am-
Erica Hughberg: What, the optimistic ones. That's what we want. What makes you optimistic?
Zachary Smith: Well, I love, um, that... Remember earlier we debated what the meaning of alt cloud is?
Erica Hughberg: Yeah.
Zachary Smith: I see so many new clouds and service providers emerging right now, and I think that that is a really interesting and needed part, is that we need to- Yeah
Zachary Smith: benefit from a more diverse ecosystem. Yeah. I think diversity in [00:51:00] thought, business model approach usually creates better outcomes. Yeah. And for the past 10 or 15 years, we've been on more of a consolidation- Yeah ... on infrastructure, and that's scary to me. I don't think it creates- Because it might become-
Zachary Smith: the best outcomes ... '
Erica Hughberg: cause it could become the same problem that we have with all of the manufacturers of memory- I- ... and stuff, for example maybe.
Zachary Smith: I think it could be that. It could also just be used for things that we don't want. It could pre- present a lack of choices. And I loved the early phase of the internet where it was messy and it was a bunch of small people getting ISPs and- Mm
Zachary Smith: trying to cr- I don't think we'd be where we were today if there wasn't tons of random innovation happening. Yeah. And I would like to see that again. And so I, I believe that's the most exciting thing right now, is just a lot of entrepreneurship, a lot of people thinking, "Well, why not this way?" And- Yeah ... um, that to me is, is ambitious, but it's exciting, and I hope that more of those kind of clouds and operators and approaches to technology will [00:52:00] survive and thrive.
Zachary Smith: Yeah. Um, and that
Erica Hughberg: we have- What is- ...
Zachary Smith: more choices in the future.
Erica Hughberg: What is an unexpected alt cloud that you... Is there anything that's been unexpected to you, or, like, something you think other people find surprising?
Zachary Smith: Yeah. Well, I mean, I would say that, like- A lot of people would think, "Hey, I've got
Zachary Smith: Huge amounts of capabilities and investment and advantages and money sitting in one or two or three hyperscale cloud businesses. Okay?
Erica Hughberg: Yeah.
Zachary Smith: I don't think it's true. Like I always like to say, they own undersea cable systems through satellites and space. Mm-hmm. It's pretty, pretty wide swath. Um, but, you know, not considering things like antitrust, breaking up companies, which- Yeah
Zachary Smith: you know, we've had a history-
Erica Hughberg: Mm ...
Zachary Smith: in society when things get too big and too monopolistic that we, you know, capitalism comes in and the, and the, you know, we are like, "But that's not okay," and [00:53:00] the big hand- Uh-huh ... you know, weighs down. Yeah. That's currently not where we're at, but I think- No ... that it could happen.
Zachary Smith: Okay? Mm. Um, but you know, I've just been so impressed with, um, some of these new cloud operators. Like- Yeah ... I would even say Anthropic is a really interesting one that could make the legacy cloud, like AWS, feel really awful.
Erica Hughberg: Mm.
Zachary Smith: Right? Like, why do I wanna deal with all this stuff? Yeah. I don't wanna deal with this stuff.
Zachary Smith: I wanna have this cool abstraction layer called like- Yeah ... I just tell it what I want and it does, does it for me. Like I think that is really fascinating, because we have an entirely different paradigm for what it meant to interact with technology, which was much more, you know, uh, fair and open to people, because right now, bless your heart and all the work that people have done in the cloud native community- Yeah
Zachary Smith: for making it easier to run software deploy, it's still freaking complicated.
Erica Hughberg: It is.
Zachary Smith: Oh my God,
Erica Hughberg: it's complicated. I was literally talking about how complicated, like, is, is to configure things in Kubernetes, for example, just earlier today. Right.
Zachary Smith: And so it's still super [00:54:00] complicated, which means very few people, like on planet Earth, have the capability to actually do it.
Erica Hughberg: Like- Like
Zachary Smith: point
Erica Hughberg: something percents of-
Zachary Smith: Yeah ...
Erica Hughberg: of
Zachary Smith: percentage points of humanity can do that, which is unfortunate. Like, I don't know. And like I think that with better abstraction layers and newer, you know, newer clouds and newer services like, like- Yeah ... even what Claude's doing, I'm doing more deployment and capability with infrastructure than I've ever done.
Zachary Smith: Maybe we can get 1% of society that could have access to this type of capability. So this to me is super exciting, which is like new interfaces that really change the kind of surface area of who gets to innovate on what.
Erica Hughberg: Yeah. Okay. I have like my little question I sort of want to round up.
Zachary Smith: But you wanna hear my dark horse?
Erica Hughberg: What? Oh, your
Zachary Smith: dark horse. My dark horse is that-
Erica Hughberg: Oh.
Zachary Smith: My dark horse is that, um- Like maybe we need to meet, meet back here in three years or five. I should give myself more time.
Erica Hughberg: Okay.
Zachary Smith: Um, I mean, three. Three might be okay. [00:55:00] I think like the largest inference service provider of the world might be Apple in three years.
Erica Hughberg: Oh, interesting.
Zachary Smith: Like on-device local access- On device, yes ... is gonna just completely change our paradigm and we're gonna solve like what we run where and our data.
Erica Hughberg: Yeah.
Zachary Smith: But like th- might just be amazing.
Erica Hughberg: Mm. Okay. Like, uh, me and Demetrios on a couple of other of these, uh, episodes, we have sort of, um, talked about this idea of can it in the future be that some inference happens on the cloud and we have-- And the local, the local like on my iPhone or on my Mac or on-
Zachary Smith: Mm
Erica Hughberg: some more powerful machine in my house, whatever. Mm-hmm. But it's something that what can you run reasonably on consumer device and still produce quality and only offload what needs some crazy power to the place- Yeah ... that has the crazy [00:56:00] power.
Zachary Smith: Or some state or some something or other.
Erica Hughberg: Yeah. Yeah. So like I think-
Zachary Smith: I totally agree with this, and so that's my dark horse is that like Apple, who probably has none, none, none percentages of the inference market today and is considered completely laggard, might actually show up with some pretty amazing stuff that changes the way people do stuff, and I think that would be pretty interesting 'cause maybe I could just buy my computer and be able to do all the great things instead of wondering what my token budget is.
Erica Hughberg: Then you have to worry about what your electricity bill is instead, but hopefully that is way lower than paying for token. I'm already-- Yeah,
Zachary Smith: I'm already doing it to like, you know, watch my Netflix and stuff. Yeah. Come on. It's fine.
Erica Hughberg: Exactly. I'll be fine. But like I have-- I think we actually need to speak again.
Erica Hughberg: I'll tell you what we need to speak again about because we didn't talk about it, and I think this is something we missed out on, and we are running out of time. But I am super curious about how companies today, right, big enterprises-
Zachary Smith: Mm.
Erica Hughberg: If you're a [00:57:00] big enterprise, you already maybe have data center. Many big companies do.
Erica Hughberg: Yep. And if you have your own data center and you have- Like now when we may be using compute and stuff more intensely, so you don't have a-- 'Cause that was the whole point of the cloud, right? I don't use it all the time, so I give it back when I'm done, and then I go get it when I need it. But assuming you use it in a more constant fashion, then owning the hardware starts becoming more reasonable, right?
Erica Hughberg: We were all trying to move out of our data centers into the cloud, and now it's like, wait a minute, if I'm using it all the time, why don't I just have it?
Zachary Smith: Long all data centers die, long live the data center.
Erica Hughberg: It's like it's... Anyway, so I'm, like, quite fascinated by this notion of how, and this is where I think alt clouds may be interesting as well.
Erica Hughberg: So what if I have my general purpose, my own data center, [00:58:00] and then I use or specialized alt clouds for specialized stuff that is, like, almost the idea of, like, outsourcing in a normal company, right? I have my- Yep ... core business.
Zachary Smith: Yep.
Erica Hughberg: I have that, and I outsource stuff that is not my core business, and I use those service providers.
Erica Hughberg: Anyway, that's like a weird Erika thought, and we didn't get into-
Zachary Smith: No, I think we, we, we should, we should call it the rise of the alt clouds. Um, but, uh, I think the dam is broken, Erika. We used to say multi-cloud-
Erica Hughberg: Yeah ...
Zachary Smith: five years ago, and that really meant I'm gonna figure out how to use AWS and GCP, so I could make sure to beat them up with each other or have a little bit of diversity.
Zachary Smith: And now alt cloud means, you know, multi-cloud means I'm using Databricks, I'm using- Mm. ... Mongo Atlas, I'm using my private thing, I'm using CoreWeave, I'm using, you know, Supabase, I'm using, you know, Resend, I'm using, I'm using Twilio, I'm using all these clouds. Yeah. Oh my gosh. And so therein lies the challenge is we're gonna ha- And then that [00:59:00] we didn't even get into sovereigns- Oh
Zachary Smith: where I'm gonna be forced to use this cloud. And so-
Erica Hughberg: That's another such interesting thing, like, yeah.
Zachary Smith: Because IT and technology and AI is politics now, and we are in an age of disaggregation. I call it the splinter net. It's like the scarier, less good version of the internet. Um, and I think-
Erica Hughberg: Is splinter net bad?
Zachary Smith: Well, I mean, splinter net's more complicated and not everybody gets to use it. And so, you know, this is Countries saying, "I can't rely upon that thing," or, "I can't get- Yeah ... chips from them," or, "Their clouds are located in those places." And, you know, we're unfortunately dealing with a lot of non kind of globalization, let's all figure out how to work together stuff.
Zachary Smith: We're working on how we protect each other, protect against each other and- Yeah ... or something like that. I don't know. It's not as friendly as, as we wanted it to be. No. And so I'm hopeful that all clouds can play a role in that too, which is like maybe we could [01:00:00] get together and work on open standards for things like interchange on accounts or how you could use the right cloud in the right location- Yeah
Zachary Smith: and it wouldn't be completely awful and different. Like- Yeah ... that is a cool open source, like let's do it together if our governments aren't gonna figure out how to work together.
Erica Hughberg: Yeah.
Zachary Smith: Let's figure out how our systems and- Oh ... you know, software can work together.
Erica Hughberg: I like that. That's really b- li- little bit poetic.
Erica Hughberg: So if you are, if you are, like this is my, this is my end question. So if you are working as an engineering leader in some fashion, or you have some, you know, you work in a company and you're an engineering leader, this whole like navigating the world that is set compute and inference and connecting systems, et cetera What is obviously tricky, but what, 'cause you've been talking a lot about giving back, how would you advise someone [01:01:00] to help better understand what's happening in the world of infrastructure, data centers, hardware, and how to actually have a positive impact as a consumer of that?
Erica Hughberg: Can you, as a consumer, as a customer of these things, can you have a positive impact?
Zachary Smith: I mean, I think if you're an engineering leader, one of the first things I'd look at is the Green Software Foundation. It's part of Linux Foundation. It's focused on how do you bake better software to use your resources appropriately and to understand them.
Zachary Smith: And so I think, you know, being intimate with the impact, the energy side, the optimization, which we used to do a lot of, and then the cloud just said, "I'll just scale it out." Like, where we can figure out how to use the right resources for what we're trying to do, and does that make sense beyond the bill, right?
Zachary Smith: Yeah. I mean, it should align economics, uh, over time, but I think, like, that's a really good stuff. Um, you know, the Green Software Foundation has been doing a lot of great work around normalizing kind of energy impact [01:02:00] and resource impact of software, which I think, you know, if that's your job is to help make software or turn business outcomes into-
Erica Hughberg: Yeah
Zachary Smith: you know, uh, to code, like that'd be a cool place to start. Have a look and, you know, see if there's something you could incorporate into your SL- SLDC, SDLs, SDLC-
Erica Hughberg: SDLC, yeah.
Zachary Smith: Software development ... in your software, in your cloud development life cycle. Yeah, I think that could be a good idea.
Erica Hughberg: I like that, the cloud development life cycle.
Erica Hughberg: I love that. Okay. I, I could talk to you for so long, and, uh, I really enjoy having this opportunity to chat with you, and I really enjoy the opportunity of having other people have the ability to listen to a conversation with you so they can pretend they have a chat with you as well. But thank
Zachary Smith: you. Well, longtime listener, first-time caller, as Frasier Crane would say, so thanks for having me on this, Erica.
Zachary Smith: It's been super fun to chat with you, and if you're a hardware nerd and you love that, um, reach out. Would love to talk about it, [01:03:00] and, uh, let's figure out how to do it in the way that makes it all better for us and the planet.
Erica Hughberg: Just clearly, Zach is quite friendly. That's a good place to be.
Zachary Smith: I'm a, I'm a nice enough guy.
Erica Hughberg: Yeah. I think so. So far you've been nice to me, so I will say Zach's friendly. Don't be afraid. Reach out. If you're in New York, see if he's around.
Zachary Smith: Lower Manhattan, you'll find me there
+ Read More
Watch More
How LlamaIndex Can Bring the Power of LLM's to Your Data
Posted Apr 27, 2023 | Views 2.9K
# LLM
# LLM in Production
# LlamaIndex
# Rungalileo.io
# Snorkel.ai
# Wandb.ai
# Tecton.ai
# Petuum.com
# Mckinsey.com/quantumblack
# Wallaroo.ai
# Union.ai
# Redis.com
# Alphasignal.ai
# Bigbraindaily.com
# Turningpost.com
Beyond AGI, Can AI Help Save the Planet?
Posted Apr 20, 2024 | Views 1.2K
# Environmental AI
# AI2
# Allenai.org
We Can All Be AI Engineers and We Can Do It with Open Source Models
Posted Nov 20, 2024 | Views 761
# AI Specs
# Accessible AI
# HelixML