Sign in or Join the community to continue
Token Maxxing Is Over and Here Is What Comes Next
Posted Oct 19, 2026 | Views 3
# AI Coding
# Agentic AI
# Developer Productivity
Share
Speakers
Pepijn Senders
Principal Engineer @ Epassi
Pepijn is a Principal Engineer at Epassi, bringing over 10 years of experience building and scaling front-end infrastructure, as well as managing engineering teams. Before joining Epassi, he was a Principal Engineer for e/acc GenAI at HelloFresh, where he helped spearhead the company's GenAI transformation until February 2026. Previously, he served as Fractional CTO at Liv Longevity Labs, co-founded ELEVE in Berlin and Knekelhuis in Amsterdam, and led technology at GoReadyMade. Based in Prague, Pepijn is also a music label co-founder, event organizer, and vinyl enthusiast.
+ Read More
Demetrios Brinkmann
Chief Happiness Engineer @ MLOps Community
At the moment Demetrios is immersing himself in Machine Learning by interviewing experts from around the world in the weekly MLOps.community meetups. Demetrios is constantly learning and engaging in new activities to get uncomfortable and learn from his mistakes. He tries to bring creativity into every aspect of his life, whether that be analyzing the best paths forward, overcoming obstacles, or building lego houses with his daughter.
+ Read More
SUMMARY
What happens when you give 200 engineers who have barely touched AI a coding assistant and a six-month deadline to rebuild their platform? Pepijn Senders, a principal engineer who previously led AI enablement at HelloFresh, has done exactly that, and he chose Sonnet over Opus on purpose.
+ Read More
CONTENT & TRANSCRIPT
Pepijn Senders: [00:00:00] I feel like sometimes, like for example, with Opus 5 or something, like Opu- Opus would ask me something and I don't even know what's going on anymore.
Demetrios: Yeah, dude.
Pepijn Senders: And like why... I really wonder why nobody's like, "Why the fuck are we like listening to this model who doesn't even speak English," right? Why, why am I not taking any second guesses here and be like, "Okay, what is actually...
Pepijn Senders: What's happening here?"
Demetrios: Yeah. Right? That's the thing with Fable, man. So many times you're just kinda like, "I mean, I guess." Yeah, yeah, yeah. "That looks good. Are you speaking English?" Yeah, exactly. "Or are you... Or are we just like talking in some made-up language now?"
Pepijn Senders: Take, take your repository, whatever you're working on, open the code base, and stop when you don't understand anymore what's going on, when you cannot place a function.
Demetrios: Mm-hmm.
Pepijn Senders: Like, what is this function doing? Why is it there? What's the business value of this piece of code? I will guarantee you, and even now, like even when I'm like really trying to like make sure that we are in control, there will be p- parts of code where people have no clue what's going on.
Demetrios: Mm.
Pepijn Senders: But I will guarantee you it will not take you a long time before you will find that function.
Demetrios: Yeah, first line.
Pepijn Senders: The first line, [00:01:00]
Demetrios: yeah. Indeed. Well, I guess I lost.
Demetrios: Pep, give me the lowdown on what are you working on, just so we can ground the conversation and let people know who you are working with. Sure,
Pepijn Senders: sure, sure. Yeah, so I'm a principal engineer at epass right now. It's like a customer benefits company. Uh, before that I was a principal engineer at HelloFresh.
Pepijn Senders: This one you might know- Mm ... a little better. Um, I've always been like, uh, trying to be as generalist as possible in my practice. Mm-hmm. And then, uh, my baby was born February 2025.
Demetrios: Congrats.
Pepijn Senders: I took, you know, as a European, parental leave. Uh For two
Demetrios: years. You're still on parental
Pepijn Senders: leave? No, no, no. Not anymore.
Pepijn Senders: Not anymore. Um, but in this time, of course, that was around the time that AI, AI was like really becoming very f- Oh, true ... quite effective at coding. And- February
Demetrios: 2025,
Pepijn Senders: yeah. Of course ... February [00:02:00] 2025, long time ago. Like in AI years, very long ago. Yeah.
Demetrios: That
Pepijn Senders: was- Um- Yeah ... coming back from parental leave three months later, talked to the CTO.
Pepijn Senders: He's like, "We gotta change stuff-"
Demetrios: Uh. "...
Pepijn Senders: today."
Demetrios: Uh.
Pepijn Senders: Now you're back- That was interesting ... to work. Yeah, yeah, yeah. And now I'm back and just working. And so I'm also on the side of my helping out, uh, Dex, uh, at HumanLayer. He was on stage earlier. Um, he's very, very interested in like seeing how AI is impacting the engineer's life, how you can actually get products from working with AI- Mm
Pepijn Senders: that make sense, that don't go off the rails. And again- ... Sonnet maxing is one of the things that I say.
Demetrios: Let's go deeper there. Yeah. What is Sonnet? W- why the enamoration with Sonnet, I guess?
Pepijn Senders: So imagine, right, everybody at this conference is like already super AI pilled.
Demetrios: Yeah.
Pepijn Senders: If you would give everybody like Opus 4.8, 4.6, whatever was out at the time- I don't know where I would've been-
Demetrios: Oh
Pepijn Senders: you know, six months later. I see what
Demetrios: you're saying.
Pepijn Senders: So for me, [00:03:00] like a-
Demetrios: Oh ...
Pepijn Senders: a model that doesn't go through as many thinking steps, doesn't do as much reasoning, stops earlier, keeps the developers more in the, in the loop. There is way more to this, but- And it will
Demetrios: get to the productivity gains that you can show, and then you can say like, "Hey, do we want more of this?"
Pepijn Senders: Yeah,
Demetrios: yeah. It's just like, go- This is
Pepijn Senders: enough.
Demetrios: Yeah.
Pepijn Senders: Yeah, yeah. So we basically rebuilt an entire platform in six months, uh, wrote like 800,000 lines of code with that team. Uh, part of the team, like 50 people with the team. So that was, like, quite a big thing, and they launched it actually two days ago- Mm-hmm
Pepijn Senders: at a big conference in Germany. So everything's fine, you know. There are some gaps here and there, but people have been b- have been staying in the loop, and 60% of that project was written by Sonnet.
Demetrios: Yeah.
Pepijn Senders: So of course there's still some Opus, maybe a little bit of Haiku here and there.
Demetrios: Mm-hmm.
Pepijn Senders: But yeah, we were, we were using Sonnet for the most part because I really believe that if you're in the loop enough, then it's fine.
Demetrios: Mm.
Pepijn Senders: And I wanted people to be in the loop because it's [00:04:00] just like hard matter that they're working with. Banking software, uh, hard business rules.
Demetrios: Yeah.
Pepijn Senders: And if you just tell, like Son- like tell Opus, like, "Hey, go bu- go build this, uh, transaction engine," it's like, "Okay, sure. I'll bu- build you a transaction engine."
Pepijn Senders: And Sonnet will get stuck at the first step, and you're like, "Okay, I actually have to steer this a little bit." Mm. "I actually have to think more." So, like, I try to always tell these people in this company, like, "Hey, don't take off your thinking hats." It's like, stay on the table. And we carried, like, systems around that, like, you know, like the specter of development, RPI kind of stuff, where Sonnet was, like, consulted for the right things, and it was, like, bringing up human feedback at the right time as well, so.
Demetrios: Was the adoption from the engineers easier too with Sonnet? Uh- Or were they already interested 'cause they were playing around with it on the weekends?
Pepijn Senders: Everybody was super excited to get, get, uh, get on, get on board. They... There was, like, nobody doing, like, the, the, the, the adoption preaching or whatever- Mm
Pepijn Senders: at the company. So nobody was really like, "Hey, let's go." So we were using GitHub Copilot before this.
Demetrios: Okay.
Pepijn Senders: [00:05:00] Um, and their token spend was almost zero.
Demetrios: Yeah.
Pepijn Senders: So
Demetrios: They weren't really using it. They had the subscription- No, they weren't
Pepijn Senders: really using it, yeah ...
Demetrios: through GitHub Copilot, but it wasn't like anyone had told them, shown them the magic-
Pepijn Senders: Yeah, yeah, exactly
Demetrios: of building a software factory with AI. Exactly. And so you found that it was a much smaller step to use Sonnet to get them just like, "Hey, ask a few questions. Try and get it to bu- write you some code," and then it would write a little bit of code, it would come back to them, they would be much more in the loop and therefore be much more trusting.
Pepijn Senders: Exactly
Demetrios: And your
Pepijn Senders: spend And that work, that, that work, yeah, and the spend was... So one thing that I try to do, so, like, maybe that's something that we can get into as well. So, like, I'm very passionate about showing the ROI of using L- uh, AI for coding-
Demetrios: Mm-hmm ...
Pepijn Senders: just because I think it's very interesting that for some time, hope- I'm pretty sure we're over that time right now, but everybody's, like, token maxing for a little bit.
Demetrios: Mm-hmm. [00:06:00]
Pepijn Senders: Uh, that doesn't work if you have enterprise billing.
Demetrios: Yeah, or a CFO.
Pepijn Senders: Or a CFO- Yeah ... or both, right? So- ... um, like, I basically built a dashboard where we could calculate the amount of, like, full-time engineers we're gaining-
Demetrios: Mm-hmm ... by
Pepijn Senders: using AI. Also, because we had a very clear cutoff point, okay, here we are not using AI, and here we are using AI- Oh
Pepijn Senders: again, this year in February. So we can really see, like, the difference. Like, we can actually see the difference in productivity- Awesome ... before and after, before and after.
Demetrios: Yeah, tell me more about these metrics that you're looking at.
Pepijn Senders: So there's a couple ones that I think are interesting. So we are, of course, like, lines of code and those kind of things are very difficult to actually- They, they don't really tell a story- Yeah
Pepijn Senders: as well as they, yeah, they, they cannot really tell a story. So basically we were trying to, like... Like, one thing that I really like to do is, like, to see, okay, this team is gonna build, like, a new home screen for the app. Mm-hmm. And then I can actually see for that specific feature [00:07:00] how long, how much time they spend on it and how many tokens they spend on it.
Demetrios: Mm.
Pepijn Senders: But if that new home, home screen is, like, maybe let's say 1% more valuable, it- was that worth spending $20,000 in tokens?
Demetrios: Mm.
Pepijn Senders: Right?
Demetrios: That's like a-
Pepijn Senders: That's like- ...
Demetrios: asymmetrical bet.
Pepijn Senders: Exactly, and I never think that these metrics should be like, okay, that person's now fired because you used $20,000 of tokens on a home screen.
Pepijn Senders: But it should always be a conversation, right? Yeah. And one thing that I've, that I've-
Demetrios: Oh, you're saying it the other way. I understood it as it's like, wow, you have the possibility of hitting a really big home run-
Pepijn Senders: Mm ...
Demetrios: when you're using these. Uh, but you're saying, like, eh, was it really worth it to do it-
Pepijn Senders: Yeah
Demetrios: and use this much money for this, like, not that valuable of a thing.
Pepijn Senders: Exactly.
Demetrios: Mm.
Pepijn Senders: Exactly. And when I was at HelloFresh, like I'll, uh, there was, like, there was a way bigger engineering org, and I was also doing, like, the AI enablement there, again, from, like, May [00:08:00] 2025, which is, like, very long time ago in AI years.
Demetrios: Mm-hmm.
Pepijn Senders: Um, and HelloFresh is such a company where... And now I can speak freely because I'm not you know?
Demetrios: You didn't sign any NDAs?
Pepijn Senders: I didn't sign any NDAs. I'm not- All right ... employed there anymore. Let's
Demetrios: hear it then.
Pepijn Senders: Uh, but it's a very product-driven organization.
Demetrios: Mm-hmm.
Pepijn Senders: There is, like, a gazillion different funnels for every single customer profile, a lot of different variations.
Pepijn Senders: Everything gets A/B tested. Everything gets, like, weighed 1,000 times before it actually got, gets even built, right? Oh. So the product roadmaps are, like, the leading indicator of everything that's gonna happen at the company, and everything is measured, okay, how much customer value can we get out of this work item?
Pepijn Senders: Mm-hmm. So everything is measured to that. And if Pep comes back from parental leave and says, "Guys, we should start using AI for coding," they're like, "Well, what's, like, the gain?"
Demetrios: Yeah.
Pepijn Senders: Is there more customer value at the other, other side of this? We don't know. So I was fighting with them for, like, six, seven months, like, trying to, like, actually prove that ROI, and I managed to do it at the end.
Demetrios: What... How... And how did you [00:09:00] prove it? I'm sure it was a million conversations, and it's really hard to just summarize in- Yeah ... you know, five minutes. But what were some of these big talking points that you had?
Pepijn Senders: So for me, it was all about- Speed, of course, productivity increase. And then again, also like, you know, variations that I could show and like the things that they could ship.
Pepijn Senders: But honestly, in the end, I think I kind of failed-
Demetrios: Oh ...
Pepijn Senders: at this company.
Demetrios: Oh.
Pepijn Senders: Because I wasn't actually able to show the full story, which I'm got pretty good at right now.
Demetrios: Yeah.
Pepijn Senders: Because I think I should have really focused more on, like, what the, the company was focused on. They were really like, "Okay, which feature is bringing the most money?"
Pepijn Senders: And I should, should have told a story like, "Okay, if we put AI on this feature, then we ship it like two, two times faster." Right? Wow. That would- that, that could have been a story.
Demetrios: Okay. But it wasn't. Not a lot of guests on here are so open and vulnerable, so I appreciate you- ... talking about that and saying...
Demetrios: You know, 'cause it's not all roses and tie-dye.
Pepijn Senders: [00:10:00] No, no. We're all learning, right? That's crazy.
Demetrios: Yeah.
Pepijn Senders: Yeah.
Demetrios: And so now you're recognizing the biggest feature that is going to really be generating the... Basically what I was saying before is there are some bets that you can take that will give you outsized returns.
Demetrios: So focus your energy there, focus your token spend on those as opposed to making a home screen 1% better.
Pepijn Senders: But it's always been like this. That's the thing. I feel like- Yeah ... with these token max- maxing budgets, people just forgot about that.
Demetrios: Mm.
Pepijn Senders: Uh, which again, we saw as well. Like I just like I was, like, looking at the dashboards and I was like somebody was, like, spending, like, billions of tokens a week, and I was like-
Demetrios: What are they-
Pepijn Senders: what are you doing? He's like, "Yeah, so I'm just, like, I'm building my own tooling." I was like-
Demetrios: Why? ...
Pepijn Senders: on the company bill? Okay, nice. I'm not your manager, but maybe...
Demetrios: And so first thing you did was you put together some dashboards on just token spend to-
Pepijn Senders: Yeah, yeah, yeah, yeah ... like- Yeah, my favorite dashboard for this, uh, you can steal my idea.
Pepijn Senders: Uh, like, I've... A lot of people [00:11:00] when they look at it are like, "Wow, that's actually..." Like, again, it's not, like, a conclusion, but it's a beginning of a conversation. Mm-hmm. And I don't wanna throw any engineers under the bus right now. They're, like, using tokens for their personal projects on the company bill.
Pepijn Senders: I hope you can still do it for the foreseeable future. But what... What... The, the thing- You're
Demetrios: speaking to me so much. All right. Yeah. Uh-
Pepijn Senders: So we, we bu- we basically just, we built, like, a scatter chart of person versus, like, production.
Demetrios: Mm-hmm.
Pepijn Senders: Or token spend versus production. So you can see, like, the token spend on the X-axis and how much they are producing on the Y-axis.
Pepijn Senders: Like- Oh ... number of features shipped, number of lines shipped, lumber- number of PRs merged. And, you know, I'm spending, like, 20K, but I'm also shipping, like, a million lines of code a month. Oh. Right? And there's somebody also then spending 15K, but he only merged one PR in four weeks. Like-
Demetrios: Oh.
Pepijn Senders: So again, it could be that he's just researching, right?
Pepijn Senders: That's like-
Demetrios: Yeah ...
Pepijn Senders: but at least you, there's, like, a conversation starter, and I think we should have those [00:12:00] conversations, and I think we should be more open. And honestly, I feel like we should share these metrics with each other because I know I don't wanna be at the mercy of these AI companies, just, like, send them, like, a, you know, $1 million, $1 million check every month-
Demetrios: Mm
Pepijn Senders: which is the situation for a lot of people.
Demetrios: Are there any other data visualizations or just data that you've been able to catch the outliers, both good and bad, that help you see, like, "Oh, wow, we should have a conversation"?
Pepijn Senders: Yeah, yeah. It's very important it's the start of a conversation.
Demetrios: Yeah. It's not accusing you of anything, but what are you building?
Pepijn Senders: Yeah, yeah, exactly. It's like... And again, there is a lot of-- I would say in my experience, it's al- it's, it's become better now that we're, like, talking about these metrics better and people are more realizing what, uh, yeah, you know, they're actually spending a lot of dollars for the company and the company has to know what the ROI is.
Pepijn Senders: But before that was big, like, I would say, like, 70%, 30%. 30% were people just writing their resumes. Uh, Tinder pickup [00:13:00] lines.
Demetrios: Uh- On the, on the company dime?
Pepijn Senders: On the company dime. Asking Claude code, like, "I have a date tonight."
Demetrios: What should I do? Oh, man. And then you start to see that, and you recognize, like, uh-oh.
Pepijn Senders: Yeah, yeah, yeah.
Demetrios: We need to stop.
Pepijn Senders: And then I start thinking about my own prompts and, okay, fuck, actually maybe I'm also not-
Demetrios: Yeah, you need to- You know? ... use your personal computer.
Pepijn Senders: Yeah, yeah. Is it your personal account? But-
Demetrios: Yeah ...
Pepijn Senders: um, to your question, like, what, which other visualizations? So again, I think, like, this, like, really thinking about, okay, what is, like, using AI, giving our, our team, giving back to our team- And I don't like to thinking like productivity increase, like, okay, we're 2X, 3X better.
Demetrios: Mm-hmm.
Pepijn Senders: But I rather think, okay, like how many-- like if we would have to do this with... Now we're doing this with 20 engineers. Like before AI, we might have needed like 38, 40 engineers.
Demetrios: Mm-hmm.
Pepijn Senders: So now you're suddenly saving the company 20 head- like 10, 20 headcount, right? And that's like your, that's their ROI.
Demetrios: And the, uh, almost like false [00:14:00] dichotomy of that, maybe that's not the right term then, is that when you have 40 engineers, you're organizing 40 people, and so it's not a guarantee that with those 40 people, you're going to get the same output and the same outcome. Yeah. And so, so it's a little bit like, yeah, I could see that.
Demetrios: You can kind of like say, all right, there's metrics of how many people would we need if we wanted to do this? Are there other things? Like, like, 'cause I'm thinking about when you're going through this, you potentially wanna go and get the folks that are actually shipping the most, and really in the quadrant that you drew- Mm
Demetrios: they're in the top right, right? That's the best quadrant to be in.
Pepijn Senders: Yep. Yep, yep.
Demetrios: Most spend, most shipping.
Pepijn Senders: And that's the other side of the story, right? They kinda wanna get people to that, to that position.
Demetrios: Yeah.
Pepijn Senders: Right?
Demetrios: Uh, but are you going then and talking to those people and saying like, [00:15:00] "What are you doing?
Demetrios: How are you-"
Pepijn Senders: Yeah, learning from them. Yeah. Definitely. Yeah, yeah, yeah. That's also the... And also like, you know, the, the, the thing with like token spend limits that a lot of companies have right now, uh, the, you know, post-token maxing days and- Mm ... CFOs freaking out and, you know, wanting to know what h- what's happening there.
Pepijn Senders: I feel like, you know, I think there should be... These, these topics are still like so early, but it's like- Uh, and like, you know, I was talking to somebody, uh, at the dinner two days ago, and they were like, "Yeah, we have 60K per month tokens spent."
Demetrios: Just their team?
Pepijn Senders: No, no, the-- individually.
Demetrios: Oh, wow.
Pepijn Senders: Yeah.
Demetrios: So do whatever you want with it.
Pepijn Senders: Do whatever you want with it. We don't, we don't have that kind of luxury. Yeah. So...
Demetrios: Wow, that makes me uncomfortable, but all
Pepijn Senders: right. Yeah, that's a lot of tokens. Um, but I don't know, like if somebody is like super successful at like producing value, right- Mm-hmm ... with AI, and they can show this, why not give them more money?
Pepijn Senders: Yeah. Because they are basically maximizing [00:16:00] themselves, right? Where other people might actually be more on the negative side, where they're using a lot, but maybe they're... I don't know. There's like a lot of, lot of stories here, and again, that's why it's always the start of a conversation- Yeah ... not like a conclusion.
Demetrios: Yeah, 'cause you may be saying you're not being productive by these metrics we have- Yeah ... but if you grabbed other metrics, they are being productive in ways that- Or- ... you just can't see ...
Pepijn Senders: or they are like, you know, super new to the domain, and they're just using AI to learn about it. Mm-hmm. You know? Also happens.
Demetrios: Yeah.
Pepijn Senders: Right? And then they're not producing as much for may- maybe three months. But then in three months from now, they're gonna be, you know-
Demetrios: Kicking it up. Yeah ...
Pepijn Senders: kicking it up, right?
Demetrios: Yeah. I c- I can see that. So all right, what else you got? Because these are some, these are some kind of spicy takes, and I like it.
Pepijn Senders: Yeah, yeah, the spicy takes are there. Um, so Like what else? In th- in this space or like some, just generally?
Demetrios: I actually, I would love to hear, yeah, what, um, what [00:17:00] basically under the guise of spicy takes in the space, maybe not necessarily ROI because we've beat a dead horse on that one.
Pepijn Senders: Yeah, yeah.
Demetrios: And that's what everybody wants to know, right?
Demetrios: The, again, closing the loop on it, it's very hard to know what token spend is quality token spend and what is not. I really like your quadrant idea of who sh- shipping, who's faster and more-
Pepijn Senders: Yeah ...
Demetrios: and who's spending like less and more. But are there other s- things that you think about, uh, that aren't related to like the company ROI things?
Pepijn Senders: Um, so
Pepijn Senders: Yeah, I think a lot about, um, just like Making sure that humans stay in control and stay the driver-
Demetrios: Mm ...
Pepijn Senders: in production of codes. Um, I've seen many problems, uh, over time [00:18:00] where LLMs just take over and you don't know what's going on anymore.
Demetrios: Yeah, that's basically it.
Pepijn Senders: Um, I've, like, everybody at home can try this.
Pepijn Senders: Yeah. Take, take your repository, whatever you're working on, open a code base, and stop when you don't understand anymore what's going on.
Demetrios: Mm-hmm.
Pepijn Senders: When you cannot place a function.
Demetrios: Mm-hmm.
Pepijn Senders: Like, what is this function doing? Why is it there? What's the business value of this piece of code? I will guarantee you, if you've...
Pepijn Senders: And even now, like, even when I'm, like, really trying to, like, make sure that we are in control, there will be p- parts of code where people have no clue what's going on.
Demetrios: Mm.
Pepijn Senders: But I will guarantee you, it will not take you a long time before you will find that function.
Demetrios: Yeah, first line.
Pepijn Senders: The first line.
Demetrios: Immediately. Well, I guess I lost.
Pepijn Senders: It's, it's, it's really freaky and I think, you know, I think Dexal said it nicely on stage, like, yeah, you can, you can, like, improve those, um, [00:19:00] uh... You can, like, point the agent at, like, say, like, "Hey, you have to improve this," but if you don't even know that it's there-
Demetrios: Yeah ...
Pepijn Senders: like, it's like a ticking time bomb, really.
Pepijn Senders: It's the
Demetrios: unknown unknowns.
Pepijn Senders: It's the unknown unknowns, and I just have way too many examples of this, of blowing up in production and, you know. You know, I've worked on, like, quite critical systems with a lot of, like, customer data, and you don't wanna be there.
Demetrios: Well, and so what are you doing then? I know Sonnet is one method of trying to make sure that things...
Demetrios: It, it's constantly asking you questions. Yeah. But it's so easy to get out of your depth, especially when it suggests you to do something that you're like, "Well, I was gonna do it this way, but I guess it probably knows best, so whatever." And especially if it's not, like, a language that I'm super proficient at or it's something that I'm not really that into and I just wanna get it done quick, and boom.
Pepijn Senders: Yeah, and then you got a problem.
Demetrios: Yeah, exactly.
Pepijn Senders: So- It's, it's really, like, I really wanna flag those things. Like, we have, like, a company harness, [00:20:00] uh, now, and we're trying to measure, like, uh... So we, like, we have, like, a lot of steps where, uh, people have to go through. It's kind of like the Grill Me, but then our own version that we had before.
Demetrios: Mm-hmm.
Pepijn Senders: And we're trying to see, like, when is a human, like, steering against AI.
Demetrios: Uh-huh.
Pepijn Senders: And we also, like, use different models for this, for, like, this, this stage. For example, we might use Opus actually, or we might use Fable, depending on, like, the, the complexity- Yeah ... of the problem that they're working on.
Pepijn Senders: Um, but we wanna know when we're steering away from the dataset that the LLM has. 'Cause I'm really convinced that these, like, you know, they're trained on data.
Demetrios: Yeah.
Pepijn Senders: And, you know, we wanna do something, you know... Most of software engineering is super generic, right? It's just like, okay, build this fucking model or-
Demetrios: Yeah
Pepijn Senders: you know, build this database adapter. Doesn't really matter.
Demetrios: Mm-hmm.
Pepijn Senders: So it's all, like, very reproducible by AI. They can just churn it out for you, whatever. But there's a lot of moments actually that you have to steer back-
Demetrios: Mm ...
Pepijn Senders: and say, like, "Actually, no, we need something different." And I'm getting a little bit in the weeds now, but we really try to measure, like, okay, when [00:21:00] is, like, the, the, the humans are steering back?
Pepijn Senders: We have, like, some dashboards on that as well, seeing, like, which teams are doing it more and less. Again, conversation starters, not like-
Demetrios: But that's a positive signal, or no?
Pepijn Senders: So it's a positive sig- uh It depends.
Demetrios: Mm-hmm.
Pepijn Senders: Again, like it's a, it's a conversation starter.
Demetrios: Yeah.
Pepijn Senders: Yeah. So we can-
Demetrios: Being able to steer is really like understanding that we don't want the LLM-generated stuff, and so it's almost like I would look at it, I guess y- you're gonna find the pros and cons on each.
Pepijn Senders: So it's interesting because we have like graphs over time that can show this.
Demetrios: Yeah.
Pepijn Senders: And also with models and like, for example, like I feel like sometimes like for example with Opus 5 or something, like Opu- Opus would ask me something, I don't even know what's going on anymore.
Demetrios: Yeah, dude.
Pepijn Senders: It's like such a...
Pepijn Senders: And like why, I really wonder why nobody's like, "Wait, why the fuck are we like listening to this model who doesn't even speak English," right? Yeah. Why, why am I not taking any second guesses here and be like, "Okay, [00:22:00] what, what is actually... What's happening here?"
Demetrios: Yeah. Right? That's the thing with Fable, man. So many times you're just kinda like, "I mean, I guess."
Demetrios: Yeah, yeah, yeah. "That looks good. Are you speaking English?" Yeah, exactly. "Or are we just like talking in some made-up language now?"
Pepijn Senders: Yeah, yeah, yeah. So again, I've seen just so many different things and we also look back at like sessions a lot to kind of like look at like, okay how are features built and also like now we im- we can try to look at session history for postmortems for example.
Pepijn Senders: There was like a authentication problem, and I was like looking at the code and the ADRs and all the documents that were put out and they're like, I was like, "What the hell? How-"
Demetrios: Mm-hmm. "
Pepijn Senders: How did this, how, how did this get decided that this was a good idea?" You know
Demetrios: how, and this is the funny piece, is that when you're given the selection of choices, it kind of s- is saying without saying it that here's what you can do.
Pepijn Senders: Yeah.
Demetrios: And it doesn't, it makes you think that like these are the choices, and es- if you're not gonna put in the energy to figure out what other [00:23:00] choices there are, you kind of go down the path that it wants.
Pepijn Senders: Yeah.
Demetrios: And so by having those questions that are being fed to you, like, "Do you wanna... We're-- I suggest you do it like this," and it usually will recommend something-
Pepijn Senders: Yeah, it always does
Demetrios: that will throw you on a path that maybe you didn't wanna be on.
Pepijn Senders: Yeah. Yeah, yeah, yeah. It's, it's hard. Like the, it's, it's almost the, the one of the ways I've found to have, give some agency back, uh, to the engineers to kind of like help them steer through that. And the way that we actually do it right now is like, 'cause I feel like when you're one-on-one prompting with an agent, you're more likely to say, "Okay, recommend this, recommend this, recommend this, recommend this."
Pepijn Senders: Looks good to
Demetrios: me. I'm in.
Pepijn Senders: So-
Demetrios: Yeah ...
Pepijn Senders: what we do right now is actually we make it like a multifaceted story where it's not just like engineering questions, it's like product, UX, whatever. So basically you ingest like the feature, the PRD or whatever you wanna t- whatever call it, w- the, the product brief or whatever.
Pepijn Senders: And the idea is with the teams that we're working with right now, they come together on Monday, and we call it [00:24:00] the refinement session, like we used to call it as well in the, in the past, where you had this annoying engineer that put their hand up and be like, "Hey, we cannot build this," you know? Or, "It's too difficult."
Demetrios: Yeah.
Pepijn Senders: That's the same vibe that I wanna get back.
Demetrios: Ah.
Pepijn Senders: So basically we're shifting left, we're putting everybody in the same room. We're discussing what needs to be built, and then AI as a partner is trying to like grab like the, the hard parts from the, the requirements.
Demetrios: Mm-hmm.
Pepijn Senders: Like the hard parts to implement, the hard parts to...
Pepijn Senders: That were like the, the, the code base would get, you know, where it would be difficult to implement in the current code base that we have.
Demetrios: Ooh, okay. So talk to me more about this, 'cause this is very much an organizational way of working- Yeah, yeah ... with-
Pepijn Senders: Definitely ...
Demetrios: AI.
Pepijn Senders: Yeah.
Demetrios: So y- I'm one-on-one with my agent talking about what I'm gonna build, and I have this feature idea, or I have this way- You're context building.
Demetrios: Yeah, I'm context building.
Pepijn Senders: Yeah.
Demetrios: And then- I go into a meeting with the rest of the team and I show them the-
Pepijn Senders: No, no, you do live.
Demetrios: Okay.
Pepijn Senders: So basically you have the-- Or either it's asynchronous or, uh, uh, during a call, you just [00:25:00] like open up the agent screen, make it pretty in some way.
Demetrios: Uh-huh.
Pepijn Senders: And you just talk about the feature, just like how you would do like a refinement session before-
Demetrios: Okay
Pepijn Senders: AI.
Demetrios: Interesting.
Pepijn Senders: Yeah. And then the agent's able to get more questions. It's able to drive the conversation more because there are a lot of conversation starters that the agent will give you.
Demetrios: Uh-huh.
Pepijn Senders: And then in, in a group setting, I feel like, you know, peer pressure, people are less likely to be like, "Oh yeah, recommend it."
Demetrios: Yeah.
Pepijn Senders: Or, "I suggest it."
Demetrios: That's the social cues. Yeah- Yeah ... because you can't just go through every step.
Pepijn Senders: Exactly.
Demetrios: So your-
Pepijn Senders: It's still a value there because the product manager can be there. It's like, "Hey..." They, they can also steer the conversation, right? Because I feel like if you're in your silo, it's not, it's never been good to work in a silo.
Demetrios: Yeah.
Pepijn Senders: Like agile is great for that reason because there are so many ceremonies, there are so many points where people can actually talk and like get out of that silo, right?
Demetrios: Mm.
Pepijn Senders: Yeah.
Demetrios: Dude, okay, so this is brilliant. So you literally are going through these questions with the rest of the team.
Pepijn Senders: Exactly.
Demetrios: And then it, it's [00:26:00] almost like fun because, "Oh, cool, what's it saying now?
Demetrios: Uh, it's s- suggesting that we do it this way." And everybody, that group knowledge is able to not allow the, "Just looks good to me, I'm gonna accept everything."
Pepijn Senders: Exactly.
Demetrios: Okay.
Pepijn Senders: Yeah. And it works really well.
Demetrios: That's a little trip.
Pepijn Senders: And like the, the, the artifacts that get generated over time for this actually are very valuable for somebody like me who's like sitting on top of this gigantic project of like building like a, rebuilding like a huge part of like their platform in six months.
Demetrios: 'Cause you're also recording those calls, and then you have the transcripts
Pepijn Senders: too? No, no, like we just have the, we just have the transcripts. So like, so like, yeah, record it, but then-
Demetrios: Yeah ...
Pepijn Senders: um, I'm not gonna go for like the actual call, but I will, you know, also like using LLM to kind of say like, "Hey, can you like give me like a, a summary of like the last, of the, of the refinement sessions that happened this morning?
Pepijn Senders: You know, what is like the general direction? What should I watch out for?" Yada, yada.
Demetrios: Nice. Yeah. And how, if you wanna dig into something specific, how is this decision made? Why did we decide to use this or that? [00:27:00]
Pepijn Senders: Yeah. So honestly, I've been too late- Uh-huh ... with some of these decisions sometimes, but it, it, like I can see- Yeah, I mean-
Pepijn Senders: I can see myself use that in a better way-
Demetrios: Uh-huh ...
Pepijn Senders: where I can actually be ahead of, ahead of things getting implemented in a, in an unscalable way or, you know, non-desirable, whatever.
Demetrios: Okay. So shifting that left, what are some other ways you're thinking about shifting left or just like organizational changes?
Pepijn Senders: Yeah, I have another hot take.
Demetrios: Oh.
Pepijn Senders: This is, uh... So- We have to demystify LLMs for us to be productive with LLMs. Otherwise, we're always gonna treat them as like some magic box is okay. I'll put something in, it's gonna do it for me, done. But if you actually really demystify it and like know that the input that you give it is very-- is actually very, uh Like the, the output will be very different-
Demetrios: Yeah
Pepijn Senders: depending on how you're using the tool. Um, I did a little experiment because a lot [00:28:00] of UX designers were, were like knocking on my door. It's like they wanna write code. I was like, "Sure." "But let's try it out first."
Demetrios: Mm-hmm.
Pepijn Senders: So I basically gave the same task to a bunch of front-end engineers and a bunch of designers to, like, generate a page on the website, and I compared the results before and after-- or re-results between the two groups-
Demetrios: Yeah
Pepijn Senders: and the prompts that they used. I told them one prompt, only, only, on- on, yeah, only you can do only one shot to generate- Ooh ... this page. And the results were stunning because, sure, they look the same on the outside, but the all, all the designers but one were, like, single HTML page with JavaScript and CSS inline.
Pepijn Senders: All the engineers had, like, React apps, uh, whatever view, doesn't matter. And if you... There's, like, a very small thing if you look at the prompt that actually is, uh, uh, the, the changing the way that these outcomes are different from each other. Um, and it's like the engineer would write [00:29:00] just React or scalable-
Demetrios: Uh-huh
Pepijn Senders: or maintainable.
Demetrios: Yeah.
Pepijn Senders: Like, any of those tokens-
Demetrios: Keywords ...
Pepijn Senders: steer the model in a completely different direction.
Demetrios: Yeah.
Pepijn Senders: So if you take that very simple example, and take that and then apply that to the way that you wanna work with LLMs at scale, then you really realize that you have to make sure that the right tokens get put in for the right task.
Demetrios: Hmm.
Pepijn Senders: Because they really, they really steer the outcome a lot. And then going back to the Sonnet maxing, with a wrong token in Sonnet is not as bad as a wrong token in Opus.
Demetrios: Oh.
Pepijn Senders: Because it will take... Opus will take that token way further-
Demetrios: Yeah ...
Pepijn Senders: than Sonnet would.
Demetrios: Yeah. Which in a way is like a double-edged sword- It is a double-edged sword
Demetrios: 'cause you want Opus to go further-
Pepijn Senders: Yeah ...
Demetrios: and not ask you. Yeah. But if it's going the wrong direction, you're screwed.
Pepijn Senders: Exactly.
Demetrios: This is something that I was literally just talking about with [00:30:00] Bauke on the last segment, and we were calling it, like, magic words. And so it's being able to place the model into the right latent space-
Pepijn Senders: Yeah
Demetrios: to understand that, oh, when I make this, it needs to be maintainable or it needs to be scalable, so I'm going to use these constructs instead of those.
Pepijn Senders: Yeah. Yeah, yeah.
Demetrios: And to know that, that's the key, is that you have to understand the value of these things, and so that's where either you learn that for that specific thing that you're trying to do.
Demetrios: So if it's a UX designer that's trying to write code, they have to now learn that and that... 'Cause it's not, like, so hard to learn Just like add a few extra words to your prompt, right?
Pepijn Senders: Yeah, yeah.
Demetrios: But you gotta know those words.
Pepijn Senders: No, like with the thing that we did though in the end was actually the designers are writing code now.
Demetrios: Yeah.
Pepijn Senders: But they have like a repo where they're doing that in where we set the context, and basically I did an evil thing where I basically said that every designer writes in [00:31:00] Svelte.
Demetrios: Uh-huh.
Pepijn Senders: Because Svelte is like, it's like not as safe as Rust, but it's a lot safer than React, for example.
Demetrios: Yeah.
Pepijn Senders: It's like way harder to make mistakes.
Pepijn Senders: So like we just tell Claude in that repo like, "Hey, whatever the designer asks you to write it in Svelte with these like, uh, principles, yada, yada, yada." And then we have another like output style thing that is supporting Claude Code, and then it basically talks back to them in designer speak.
Demetrios: Brilliant.
Pepijn Senders: It's like, "Yo, I built your page. It's done," right? "Find it here."
Demetrios: Smart. Smart man. Well, before we sign off, you did mention you have a company harness. What does that entail?
Pepijn Senders: So it's like a meta harness they call it right now where we're bu-
Demetrios: Harness above the harnesses ...
Pepijn Senders: Exactly. So- Yeah ... we're orchestrating Claude Code.
Demetrios: Mm-hmm.
Pepijn Senders: Uh, I did a talk in Berlin. I was very harsh on people there. Uh, it was like the ri- the, around the time of the rise of the skills.
Demetrios: Mm-hmm.
Pepijn Senders: Uh, the doomsday. Um, like [00:32:00] I wanna control what goes into the context window. Like, I'm really, really, yeah, I'm really anal about it.
Demetrios: Yeah. Making sure that you don't just get random skills thrown in there?
Pepijn Senders: Yeah, so like I will give you an example. Like, I was... We had this, like, a workflow where we used, like, the classic commands that Clockcode used to have. Still are supported, but... And it was... Some guy was like, "Yeah, dude, I just, like, cannot get res- good results."
Demetrios: Uh.
Pepijn Senders: I was, "Okay, let me, like, open up your, your Clockcode.
Pepijn Senders: Let me see what you have." And he had, like... He basically went into some skills repository and downloaded every single skill.
Demetrios: Oh.
Pepijn Senders: It's like he had superpowers, all that shit. Oh, man. Everything was turned on. It was- Every...
Demetrios: So it's, like, the same skills trying to do things, and they're fighting between each
Pepijn Senders: other?
Pepijn Senders: Exactly. Exactly. Yeah. And, like, his context window was already at 40K tokens before he even started- Ah ... so I was like, "Yeah, well, that's why." So going back to the company harness, so I... Everybody here, probably this conference, is, like, more on the innovator side of things. Mm-hmm. They are, like, wanna do [00:33:00] things themselves.
Pepijn Senders: They wanna, like, explore, whatever. If you go to, like, a just a normal company, that's not the case. A lot of people are just, like, very happy to just do whatever, you know, we tell them to do. They just take the ticket, build it out, ship it. And I didn't think it was as ba- or it's not bad, but I didn't think it would be as that many people that actually would just follow the process-
Demetrios: Mm
Pepijn Senders: when I would give them the process, or we gave them, like, the process, and then we actually build it out into, like, a, into a, into a company harness. But we... When... While building, we found that 85% of the code was written in ex- exactly the same way.
Demetrios: Oh.
Pepijn Senders: So refinement, uh, specification, implementation, verification.
Demetrios: Okay.
Pepijn Senders: So we could see that after the fact because the artifacts were all there. We can measure how much code they were generated from that, and it's like... Then we're like, "Okay, let's just, like, make this a harness-
Demetrios: Mm ...
Pepijn Senders: and make sure that whenever somebody is in that stage, it's the most optimal context window, the most optimal [00:34:00] skills for whatever they need for that specific stage.
Pepijn Senders: We measure exactly what happens there, uh, so we can see, okay, this team spent this much tokens for this specific stage for this task.
Demetrios: But you started from scratch, right? You're not pulling up Pi and trying to build on
Pepijn Senders: top of
Demetrios: it? No.
Pepijn Senders: No. It would be too much, right? No, it's very, it's very simple. Yeah. Like, it's actually, uh, this is a, this is a cool trick you can do, uh, with Clockcode, and you can make, uh, your own meta harness, um, where basically what we do is we actually start Clockcode.
Pepijn Senders: We inject a small thing into the system from Sentick. If you're done refining, go touch this file.
Demetrios: Mm-hmm.
Pepijn Senders: Then we have a watcher that watches that file, and then whenever that touched file is touched, we close Clockcode and open a new session- Oh ... for the next stage. And then we capture all the metrics from that and we get structured data, all that stuff
Demetrios: Brilliant, dude.
Demetrios: Well, this has been awesome.
Pepijn Senders: Yeah. Thank
Demetrios: you. But we gotta go to the next talk.
Pepijn Senders: Next talk.
Demetrios: We're gonna send it off. We will see you all soon, [00:35:00] and I'll be back with more folks in the podcast room. But until then, enjoy this talk from the main stage.
Pepijn Senders: Thank you. Dude. Wow. That was freaking awesome, bro
+ Read More
Watch More
The Creative Singularity is Here // Pietro Gagliano // Agents in Production
Posted Nov 26, 2024 | Views 1.1K
# Singularity
# Transitional
# Forms