Sign in or Join the community to continue

He Used 100 Billion Tokens in a Month - Waymo Talks

Posted Nov 19, 2026 | Views 7
Share

Speakers

user's Avatar
Jesse Vincent
CEO @ Prime Radiant

Jesse Vincent is the founder of Prime Radiant, an AI research lab, and the creator of Superpowers, a popular open-source agentic software development framework. He hasn't written a line of code since October 2025.

In past lives, Jesse co-founded Keyboardio, where he designed award-winning ergonomic mechanical keyboards. He led VaccinateCA during the height of the COVID-19 pandemic, created the open-source issue tracking system Request Tracker, created K-9 Mail for Android, and served as the project leader for the Perl programming language.

+ View details
user's Avatar
Demetrios Brinkmann
Chief Happiness Engineer @ MLOps Community

At the moment Demetrios is immersing himself in Machine Learning by interviewing experts from around the world in the weekly MLOps.community meetups. Demetrios is constantly learning and engaging in new activities to get uncomfortable and learn from his mistakes. He tries to bring creativity into every aspect of his life, whether that be analyzing the best paths forward, overcoming obstacles, or building lego houses with his daughter.

+ View details

SUMMARY

What happens when your AI coding agent gets better at finding bugs - but never knows when to stop?

Jesse Vincent, creator of Superpowers and founder of Prime Radiant, joins us to explore what it actually takes to get useful work out of AI agents.

+ View details

CONTENT & TRANSCRIPT

Jesse Vincent (00:08) We're going straight into this from the beginning.
Demetrios (00:10) Tell me what is this, dude.
Jesse Vincent (00:12) So this is a Play Date. Um it's a tiny little game console by a company called Panic in Portland, designed by Teenage Engineering. It's got I think it's a sharp memory L C D. It's an STM32. It has a crank. Yeah. For the Like for like fishing games and Like for games. Oh no way. But my friend Harper Reed uh was he's like, I'm gonna put a coding, you know, coding agent UI on my Plate Aid. Like that sounds cool, I kinda want that. And he says, uh actually I got distracted and um I did a really nice fractal visualizer. So it's like a great project.
Demetrios (00:45) Also cool. Yeah, how about that too?
Jesse Vincent (00:47) Well, so, I have this bespoke coding agent called Evener that we've been that's open source and has been open source since the beginning. We just haven't announced it. It's like it's there if you find it.
Demetrios (00:57) Yeah. Another one, which I found last time that we chatted. Yeah. Clearance.
Jesse Vincent (01:03) Yep, clearance of a heavy user of excellent. Yeah, clearance is the um is my markdown browser. Yep. So it's for like reading. I mean, and that what's nice is everybody else has finally figured this out, so all the desktop coding agents now have nice markdown readers.
Demetrios (01:13) I noticed. Yeah. I know it's so funny. Uh also with the a few different UI things that I always wanted. Yeah. But anyway.
Jesse Vincent (01:21) Yeah. All right. And so I'm like, you know what? Um hey Astro, let's build let's build a play date client for evener. And it starts grinding and like an hour later it's using the RPC protocol. It's I can see messages rendering. Wow. All right. I need to be able to do input. Oh I'm not gonna I'm not gonna use...
Demetrios (01:42) There's no, there's no keyboard
Jesse Vincent (01:43) I'm not gonna use the D pad and the crank to you know to type which you can do for like...
Demetrios (01:48) Imagine the crank going through all of the different letters
Jesse Vincent (01:51) For uh like for Wi Fi password that's how they do it. It's like you know either D pad or crank and then a a, it'a like either D-pad or crank and then A and B.
Demetrios (01:53) Wow, so cumbersome, dude.
Jesse Vincent (01:57) But like so for input why don't we just use open AI's uh speech recognition stuff and it you know it tries it and it's a little laggy. okay, well we can you know we can drop the bit rate and whatever and then suddenly I can you know hold down A dictate it transcribes it and it sends the message to the coding agent. And okay well I want to be able to demo this to people which means I've got to take it out of the house. Which means it needs to be a tail scale. Uh-huh so hey let's get this is an STM32. I think it's an STM32. It is a microcontroller. Wow. I was a little shocked that it actually like TailScale turns out to have like IoT device support. So it does the stuff. It opens up the tail scale control panel and says, hey, you really gotta click that approve button. I won I won't click that. Yeah. That's the um and it does it and it hard codes the keys in the build. And like as it does. And like that's fine, but if we're gonna distribute this, it really should be a proper tail scale enrollment. Yeah. So it's oh great, I'll uh I'll do the proper QR code enrollment system. So it shows a QR code that you scan with your phone and it onboard you know. And it works. It works great. But, so...
Demetrios (03:09) I want to buy one of these just for this.
Jesse Vincent (03:11) But then I mean it is not actually a wonderful device for this, it's but it's a great demo.
Demetrios (03:15) Yeah. Exactly.
Jesse Vincent (03:16) But but the next thing I get to is to talk to the hub, I need to give it the hub address and a cryptographic key. Um I'm not gonna type like two hundred characters of line noise into here. Like what could we do instead? Um initially it says...
Demetrios (03:35) Well you could no copy paste, you can, yeah...
Jesse Vincent (03:36) And I mean it's it's remote device. It's like you could it's you you know you could drop a file on it or if you use the um the um iPhone client that that you've got for evener, you could do the QR code scan trick. I'm like I don't want somebody to have to have a phone client pair to do the uh um How about an audio chirp it says like an audio chirp? Sure, let's do that.
Demetrios (04:00) Oh, that'll work
Jesse Vincent (04:01) Um Um so you go to the web app and you have the button and it gives you the choice of like pair with QR code or pair with audio chirp. And audio chirp is Oh, Oh, it might be.
Demetrios (04:11) Oh,it's the like audio stamp that will...
Jesse Vincent (04:14) Well it's like mode it's like modem noise. It is encoding a bunch of data. It's the audio equivalent of a QR code. Uh-huh. And it works. It's so wild. The future I mean, this was an evening. This was an evening. And I was doing four other things.
Demetrios (04:27) Wow. Yeah. Yeah, of course, as you should be. Yeah. Oh my God. Well, dude, since we chatted last, for everybody that hasn't seen our first conversation, you talked about superpowers, you talked about the the inspiration that you had behind it. I loved that conversation. And I think that when we got done with that conversation, you were saying, like, so what should I be what should I be thinking about? What's cool that you've heard about? And then I asked you the same thing. Yeah. And we kind of came to this point where we were like, it feels like uh everybody's building orchestrators. There's so many orchestrators out there. Yep. And oh, before I forget, I wanted to mention huge congrats because I saw Superpowers hit like 300,000 stars.
Jesse Vincent (05:10) 300,000 stars, it is, I think, the twelfth project ever to do that.
Demetrios (05:13) Dude.So wild.
Jesse Vincent (05:14) Which is insane.
Demetrios (05:16) And I think it is very, very warranted because I still use Superpowers. Even though every coding agent and every harness has tried to incorporate everything that Superpowers has into their harness, I still find it is better using Superpowers.
Jesse Vincent (05:31) So I've honestly I I had assumed that when I first made superpowers I was gonna have like three months and then everything you know, and then everything we trained in in the models, the harnesses would have all the things. Um Cloud when Cloud Code came out with plan mode, I'm like, well, that's it. Um but their plan mode conflated two things. It conflated help the human figure out their intent and the exact set of technical tasks to uh to be implemented. And those are two very like in traditional management, those are two very different processes and they don't work as well when you do them as one. Yeah, um and so you know it's it is absolutely the case that modern that modern models and harnesses are much better at one shotting things and superpowers is is way more expensive to use the su full superpowers flow than it is to just let the model YOLO it. And every time we run the evals to be like can we delete all of this? it's oh yeah it's more expensive but also it is much more it is much closer to bug free yeah it better results and so we've done a bunch of work to help to help superpowers let you pair back how much involvement it has so um somewhere in the six series it went to this looks like a bounded change this looks like a development spike this looks like a quick fix do you want to skip Yep. and now in Superpower 7 we when we gutted and redid brainstormings um which Which one. wanted to ask you about that but like the first favorite skill yeah the first thing it does is you know hey this looks like a thing where we should go through the superpower slow do you want to and it's like it just is straight up like you don't have to like supermovers auto triggers and we put a lot of effort into making sure that skills trigger proactively and automatically because you shouldn't have to be saying, oh use this skill, use that skill. The goal is to have these tools help you. And you know, while I you know I have an engineering an engineering background or I got a Russian studies degree and I've been coding since I was a kid. So that passes for an engineering background. Um But I you know and I hear from a lot of people a lot of engineers use superpowers I he, you know, inside I know of at least three of the Fang that have their own internal superpow you know superpowers in their internal plug plug in repo. And I love that it helps engineers, but the ones that really blow me away are the people say, well so I'm you know I'm not a programmer, but I just ship this thing with superpowers. Yeah an old friend of mine who's like he's a recovering sysadmin and he's like I can string shell scripts together, but I don't code, he was looking at two chest x-rays from a couple years apart and trying to figure them out. And he's like, you know, I know I'm doing it wrong, but I just I realized that superpowers this was there, and I'm like, hey, could you help me build a visualizer? And it built him, you know, a health, you know, a health visualizer that was like that gave him the understanding of what he was seeing, so that when he went in to talk to his doctor, he was already that much further ahead. He had good questions. And so it's that stuff that really like excites me. It's everybody should be able to build. know, it encodes my 30 years of engine of engineering and engineering management experience so that you don't have to have it. Yeah.
Demetrios (08:58) Dude, what did you do to make it trigger so easily because that's one of my gripes with the skills that I'll build. Right. And even after saying multiple times like, hey, this skill is called this, I use keywords, and sometimes it still doesn't trigger and it's so frustrating.
Jesse Vincent (09:15) So there are a couple things going on. So when Superpowers first shipped, I accidentally front ran Anthropic. I, Superpowers was Like was a skill system for Claude Code. I didn't know they were putting skills into Claude Code. I had found skills on Cloud.ai, they were using them for office docs. And so I had to build everything. So I had to build the system prop pieces, the triggering pieces, uh the the you know the skill structures. And I had done something pretty different. So they the standard front matter for a skill is a name and a description. And in my version of it before there was a standard there was name description and when to use and when to use was a separate line that didn't say anything about what the skill did just here are the situations in which you should trigger the skill. And the model never saw the skill description. It only saw name and when to use. And that worked really well. Yeah and when Anthropic came out with their skill system um so they you know they reached out to me and they're like so you know these are the docs we're gonna ship next week and you know what do you think? And I like sent back this sort of litany of like should do this, should do this, should this um I think you should name skills in um you In active voice like writing code. Uh-huh. The skill is writing code. The skill is not code writer. Yeah. And I was so gratified that in the three days between when they sent me the draft docs and when they published the docs, they adopted like a dozen of the things where I'm like, you should do it this way. Nice. But most but most skill guidance is the description should say what it is and when to use it. And especially with the newer models, what I've been seeing is when they can read the description and think, oh, I know how to do a code review. I won't, I won't pollute my context window. And so they skip the skill. They skip the skill. They don't they don't bother to load it. But and if they had loaded it, you'd be fine because it'd be in the context window. Yeah. But if the sk but if the description is just use this skill any time you're about to do a code review, it doesn't know what's inside. It has to go look. And so the Superpowers writing skills guidance. You see that you know phrasing there. Um tell tells the um the model that this that the description should only ever include when to use. We also have a bootstrap that loads in most harnesses that's you know, if there's only if there's a 1% chance that a skill might apply, you should read it and check. And so we put a ton of effort into triggering. What was wild is right around when either Fable or Opus 5, some of the stuff we had in the trigger in the triggering like extremely important in X, you know, all caps XML fake XML. Um Cloud started freaking out. I was like, this looks like a prompt injection attack. And no. I mean they they trained it so that all of the techniques that you would use to get a model to pay attention to.
Demetrios (12:16) Yeah, the attention became too much like prompt injection. Right. Um super sensitive.
Jesse Vincent (12:21) They put a ton of effort into trying to defend against prompt injection. It is much harder to trick the model, but it means you know it's it's a constant game of evolution. Yeah. And it's we also we don't want superpowers only to work in the the highest highest end frontier models. It should work for everybody. It should work for folks running local models. It should work for folks running a generation back because that's what they got.
Demetrios (12:50) Well this is the fascinating piece is that you really have to almost like either build something that you know is going to be kind of universal and you're not going to get as deep as possible for certain models. Yep. Or have evals and pinned skills for pinned models.
Jesse Vincent (13:09) Yeah. And we have tried really hard to avoid having to do a build step where different providers get different s versions of the skills.
Demetrios (13:17) Bespoke Superpowers...
Jesse Vincent (13:19) Yeah. Or it's like in the you know, in this, you know, for you know, for Astra you need to have this paragraph. For for Soul, you need to you need to not have that paragraph. Yeah. And we've we've managed to avoid it. There is I'm not convinced that we'll be able to do that forever. Um there is a single uh when it loads the Superpowers bootstrap, there's a single like if you're in this harness, read this the this harness.md. Um because originally the skills were all for Claude and so we had translation tables. Because that very first time that um Codex loaded up Superpowers and the skills said you know use your to do right tool to keep track of work, It freaked out. It's like I don't know if you're going to be able to do it.
Demetrios (14:00) Well, what is it to do? I just gotta go build a to-do right tool.
Jesse Vincent (14:03) Well it's like Is there one in the current directory? Is there one anywhere on disk? Let me search the web to find out what to do, right? It's your to-do list. Uh to do list.
Demetrios (14:12) Yeah, it's so simple, but again, these things that you think you don't realize it until you encounter the problem and you're like, ah, I forgot that. So now you're more focusing on how can you have a harness level type of thing that will help the model and the harness understand how to best use superpowers without having to bespoke create superpowers every time.
Jesse Vincent (14:36) Yeah. Yeah. And so it's it works pretty well. There's constant tuning. Um one of the things that we've run into over time is as the models have gotten better at finding issues, things stop as things stop converging. So for my entire career, I've often repeated this phrase that all software is made of bugs. Sometimes the bugs combine in a way that you've got the functionality you wanted.
Demetrios (15:04) All software is bugs.
Jesse Vincent (15:05) All software is bugs. Yeah. But what what we're s what we're seeing is as the models get better at f at finding bugs, when you say, hey, go review this code and if they're bugs, fix them. Or or like list off the bugs and then have the other guy fix it. The other agent fix them. And then review it again. And it used to be that you get two or three rounds in and it would find all the serious things and be done. Yeah. And now the models are very good at finding bugs.
Demetrios (15:33) It's non-stop
Jesse Vincent (15:34) The OpenAI models are obsessed with edge cases. The number of times that it is like oh, there's a time of track, time of use bug here where somebody could replace the sim link underneath you on this shell script that's gonna run only on your local machine and and hack your and hack you.
Demetrios (15:51) Yeah, you're screwed.
Jesse Vincent (15:52) It's, the judgment they don't seem to have is how much code is worth writing to fix a bug. The this 50 millisecond visual glitch, is it worth 10,000 lines of additional machinery? I don't know. I mean, I know. It's not.
Demetrios (16:09) Well, that's I was just talking to my buddy Pep about this, and he was saying you can spend that extra 10K a month, but how do you make sure that that 10K is actually worth the 10K worth of tokens?
Jesse Vincent (16:22) Yeah. Um so I've been spending a lot of my time with deep seek for one flash. So a friend of mine has an inference provider startup where their whole their whole thing is instead of uh billing you per token, they're billing you per uh how many concurrent requests you want. So last month I did a hundred billion tokens of deep seek four one flash.
Demetrios (16:39) Oh Wow. Yeah. Wow. A hundred billion. Um you remember Austin Powers when they were like one million dollars. And then it was like no, that doesn't actually mean anything.
Together (16:57) One hundred billion tokens.
Jesse Vincent (17:00) Um And and it's I mean it's really interesting when you go from super metered watching your sub you know, watching your five hour window, watching your one week window, to I can just do things. Yeah. So one of the things I did was go through all my email. Go through my thirty years of personal email. So there was a bunch of filtering to only get correspondence with people who like wrote me and I wrote back to them at least once.
Demetrios (17:25) So, not spam?
Jesse Vincent (17:26) Not spam, not commercial messages, not and pull out all the facts, all the promises, all the recommendations, um, all the favors.
Demetrios (17:36) Dude, I did this recently with my ti, um my LinkedIn. Yeah. And it was the same kind of thing. Like who if I had two way conversations with and then let's go and let's um throw an embedding model. And now I want to search it because I want to know who do I know in New York that's working on GPU optimization.
Jesse Vincent (17:53) Yeah, yeah, yeah. So the thing I had to do is it it basically wrote me Wikipedia style articles for all my contacts where it's like, you know, here's all the stuff that you recommended to each other. Here are all their major life events. It's like the serum. The CRM data you really want. Yeah.Like here's what their kids named. Yeah.
Demetrios (18:14) No way.
Jesse Vincent (18:15) But you know, by the way, they're a little you know, they're allergic to nightshades.
Demetrios (18:18)Yeah, don't make that when you have a dinner party and invite them over. Or you can even ask it things like hey, I'm having a dinner party, who should I invite over that I haven't talked to in a while.
Jesse Vincent (18:29) Yeah. No, it's we there is amazing stuff that you can do. Um and it's and it's just getting easier and easier.
Demetrios (18:35) Oh man. That is so cool. Yeah. Especially when you're looking at it. Somebody was just telling me I feel like we're in the you you know nineteen ninety eras of how we're paying for tokens. Yeah like the per token cost structure, are we sure we want to go with that? Uh in five years, do you still see us paying per token? Or is it gonna be something like this, like concurrent use?
Jesse Vincent (18:58) So my I mean, five years out is really hard to predict those like two. I mean like you know, it's like I I tal I talk to VCs and they're like, So what you know, how do you where do you see the industry in in eighteen months? I'm like I'm having trouble with six weeks these days.
Demetrios (19:13) Um Which is another testament to the fact that superpowers is still around. Yeah. And you came out with it like a year ago. Yeah, I just want to make that very clear. That's what like super hat.
Jesse Vincent (19:21) Thank you. It's it's why like I had a I had a friend at one of the Frontier Labs who um was like, so yeah no, I know you did superpowers and was totally six weeks ahead of us. Um and then four months later he's like so my guys didn't pull off a thing and I was decided to show them how it's done on a weekend, and I finally tried superpowers and you know, you're still ahead of our state of the art. Yeah, our internal state of the art. And I'm like I I am humbled and gratified by that and also somewhat appalled because I would hope they have like the magic. Like in the severed floor in the lab, I I want to believe that they have.
Demetrios (19:58) Behind the key card where you gotta fob in. Yeah, it's the secret store where everybody's wearing the gloves that you can't get any fingerprints on anything when you touch it.
Jesse Vincent (20:09) No phone, no airpods
Demetrios (20:11) Total airlock. Yeah.
Jesse Vincent (20:13) Yeah. But yeah. No, it's it's pretty wild. So the um the other thing that I've been realizing is I've been spending a lot of time with the open weights models is they're get you know, they Qwen 3827B, which is the I've been told it is it's like the most downloaded model on Hogging Face now, on a 64 gig MacBook, you can run it and get like thirty plus TPS and at it it has the intelligence an intelligence level that's around um Opus four five. And what's interesting is Opus 45 is what we had last December when everybody discovered that it was good. And so now that's a thing you can have on your laptop and not pay for. Yeah. I mean you pay in electricity and like burning your lap, but...
Demetrios (20:52) Freaked out. Yeah. So in your eyes, it's like, well, if we're gonna go that way, there's it's gonna be really hard to do pay per token.
Jesse Vincent (21:10) So what I think I mean the future I kind of expect we're gonna end up with is you have a couple of calls to a really high end frontier model that you a you ask it the hard questions and then all of the workaday stuff is either you know post hosted open weights models or local mo local open weights models and you don't have to pay anywhere near the same for that.
Demetrios (21:40) And you probably are gonna get to a place where that's just a separate machine that you have on your desk. It's like a desktop.
Jesse Vincent (21:46) So I mean that's I mean that's certainly m you know Microsoft partnering with NVIDIA to ship these RTX surface machines. Um you know, that's pretty clear. DGX Surface Machines that's like pretty clear where they're going um it is Apple has done some really interesting stuff around there on device models having the same and I've never I'm embarrassed to say I've never trained a model, so I don't really understand like they apparently the on device models have the same vector shapes as their cloud models. And so it is possible to s start a process on the local model and push it to the cloud when you need more oomph without having to reprocess everything. And in this is a thing where you should you should absolutely get people who know what they're talking about to talk about this. To explain that part. Yeah. But it's I like I I mean it's gonna a lot of this stuff commoditizes. Yeah and another thing that I'm finding is that I don't always want Astra or Fable writing the code.
Demetrios (22:50) I've noticed that too.
Jesse Vincent (22:51) It's do you want a triple PHD do you know writing your crappy React app. Yeah. You want somebody who's like it's not gonna strain them, but they're it's like it's that's what they do.
Demetrios (23:01) Well it goes back to your point on we are having these open AI models that just go through and they find all these edge cases and you're like, I don't really care that much for that edge case. I just want the app working right now. Right. And there's never a place that it knows when to stop. Because uh if if you just let it loop, it's gonna find something every time.
Jesse Vincent (23:23) Yeah. So um I use a uh for my code review, I've increasingly been using a tool called RoboRove by my friend Wes McKinney. Um it's good it's good stuff back in the day.
Demetrios (23:30) Oh nice. Wes who created he's he was, he was legend of the data world.
Jesse Vincent (23:37) So he he um he's got a tiny AI startup and disclosure, I'm an advisor to them, so when I puff them up, it's it is not entirely because just because I'm an advisor, I actually use the tools. So he's got the like the the things that he's got that I I see a lot of folks using now, there's Kata, which is his local first issue tracker for design for your agents. Um which is an it I have a I mean, I spend the first twenty years of my career on ticketing systems. I have a lot of feels there. Nice. Um so he's got agents view, which knows how to siphon all of your transcripts from all of your harnesses on all of your machines and lets you get really nice visualizations, lets you search them. Um I think it now is an embedding model. Um he's got a thing called message vault, which is siphon your all your Gmails, all of your WhatsApp, all of your iMessage, all of your Slack into a read only database that your agents can query. Uh and And then he's got this thing called Roborev, which is his take on the agentic code review tool. It's got nice um post commit git hooks that you can install. It's got a set of skills to do like review or refine. And it but it used to be this thing where running it I would get five, you know, five turns and it would converge. And No it never knows. And well and now like I've had things go to 80. And so I've been working through well how do you solve you know how do I solve this for me? Piece of work is too big, decompose it. Oh nice. And so that works pretty well. The other is if you you know if you get far enough that all it's finding are low severity issues, write out a list of them, merge, and fast follow.
Demetrios (25:28) Uh-huh. So you can figure that out later.
Jesse Vincent (25:29) And it's like, yeah, like because all software is made of bugs, the harder you look, the more you'll see.
Demetrios (25:28) Yeah. Yeah. Oh man, there's so many places that I want to go on that one. But yeah. What I would love as you're talking about that it would be so cool to see. Almost like now, maybe it's not so much what you know how we used to nerd out on like what's your rig? Yeah. Now it's almost like what is your harness setup for your agents as you're coding? Because I imagine you're using different agents with different harnesses in different parts of the life cycle.
Jesse Vincent (26:06) So harness wise, um, because of how Anthropic does subscriptions, my cloud usage is still in Claude Code. Um when I am doing desktop software on the Mac, I still often use codex. But most of my coding is in Evener, which is this coding agent that I've been working on. Um it originally was a design just for non-interactive for run for running it inside a factory style thing. And I started realizing that it'd be kinda nice to have uh, you know, my own Tui. Actually it'd be kinda nice to have my own GUI. Um I think everybody's doing multiplexers wrong 'cause I what I don't want is to multiplex different harnesses. I want to multiplex different models. And so Evener is inherently provider agnostic. So you know, you can use your codec sub, you can use tokens from pretty much anybody. Uh you can dispatch uh to sub agents across provider. Nice. Yeah which is real nice. Um Um Um and so you know, it's a lot of it I'm doing from my phone now. Yeah.
Demetrios (27:11) Or the Game Boy.
Jesse Vincent (27:12) Oh yeah. Or yeah, or um but like here are my open sessions.
Demetrios (27:17) Oh nice. Yeah. Oh my god. Yeah. So just churnin' away.
Jesse Vincent (27:22) Just churnin' away, and it's well. yeah all right, we're gonna refresh the you know it it has all the standard stuff, but it's yeah it's a harness that you've built that you've customized to your liking. It's so well it's I mean I've built it from pretty much from scratch. The very, very core of it was an implementation of Strong DM part of Strong DM's um software factory stuff back in the back eight months ago. It was like they had put put together this natural language spec for what the core of an agent looks like with standard standard sets of tools for file edit for each of the providers. And then I it is wildly different than it was because I have a lot of like strong weird feelings. Yeah. And so it is a it is fully open, but it is also super opinionated. It is not it is not Pi style work. Everything's a plugin.
Demetrios (28:10) Yeah, it's a very light harness. You're you're like no, I know exactly what I want.
Jesse Vincent (28:15) Well does that mean so developing tools that people use in Pi is a huge pain in the ass because every instance of Pi is a beautiful and unique snowflake with a different subagent tool tool set, different compaction, di uh different task lists. And so Evener is how I think it should be done. It supports you know Claude Code and Codex plugins, it it supports Claude Code and uh marketplaces, which means that I can literally just add marketplaces and add plugins and they just work. You know, I've got impa I've got impeccable design installed, I've got superpowers installed, um, I have a version of uh Anthropic Simplify Code plugin, which is amazing. Uh nice. Um, and it's like I feel a little s a little weird about that one because it's it is straight up an Earth's version of like I'm not that is a private plugin because it is it's pretty much a copy of the thing they do because it's what they do is really good. Yeah.
Demetrios (29:09) Wait, so, do you have with with this harness and doing it, do you try to create like different parts of the cycle or different loops that will specifically use a different harness just to get you out of your kind of like way that you're doing it? Or...
Jesse Vincent (29:31) So I feel like we have not, as a as an engineering culture and community, explored the problem face of harnesses very well at all yet. And one of it is either one of my Superpowers or one of my catastrophic flaws that I am very bad at looking at competitor products. Um I have always assumed that my competitors' products are the platonic ideal. And so when I'm competing with the competing product, what's in my head is the best thing it could possibly be. Yeah. And so and I found that building for me and listening to users is way, way better than chasing competitors. Mm-hmm. So like a couple of things that Evener does that I haven't seen a lot of elsewhere when you dispatch to a sub agent you don't just give it a prompt you it uh the main agent can give it a list of tasks the ta and so it goes straight into the sub agent's task list um each of those tasks can contain an effort level for the model it contain contains the prompt that should get injected there when a task is done it automatically makes the next task live um and that seems to work pretty well um a thing that we're that is currently chunking three evals is does it make more sense to when you launch that sub agent show it the first task or to give it a preview of here's everything in the task list, do the first one first.
Demetrios (30:53) It's Ready. Yeah.
Jesse Vincent (30:54) It's like does it which one helps plan better. Yeah. Um another thing that is not absolutely unique to us, but is desperately uncommon everywhere else, is how compaction works.
Demetrios (31:05) Oh, um we gave the bane of my existence.
Jesse Vincent (31:07) We gave the model a compaction tool. Uh-huh. And it's and so it can choose to compact when it wants to, when there's um when it finishes a task and it and the contact window is more than I think like sixty percent full, it says, Hey, maybe now is a good time to compact, but it's your choice. And the compaction tool includes a feature that is write yourself a letter of ever of all the important stuff you want to remember after. And that seems to work really well. It's not just like summarize the tr summarized the history, it's like here is you know You have all of your encrypted thinking blocks we can't see. You have all of the transcript. You know project state. What should you know after you get your you know after you get mind wiped?
Demetrios (31:50) You've been doing this and you've been on to this for a while now because I remember you telling me after particularly hard coding sessions, you'd ask the agent, what do you wish you had that you didn't have that would have made your life easier before going into this session.
Jesse Vincent (32:05) Yeah. No, like we've been doing a whole lot of like I mean it's a one of my patterns is like go try, you know, go try your tools. How do the tools feel? What would you know, what would have been easier? And you know, we we build the doctor into the into the harness and that the doctor is not hey user figure out why the harness is broken. It's hey agent, go you know, go debug this session, go debug the harness. Go look, you know, what tool calls have failed in the last week? How could you know, why did they fail? What could we improve? Is it the prompting? Is it the you know, is it the the structure of the thing?
Demetrios (32:42) Well I'm just thinking. You you mentioned Wes's uh rocket...
Jesse Vincent (32:47) Uh uh there's Robo there was Roborev and there's agents of the R.
Demetrios (32:49) Roborev, that's it. Yeah. And I I was just thinking about how there's probably a subset of us that are using you've got your harness that you're using on a lot of the lifecycle, but are you plugging in RoboRevs into what you're doing?
Jesse Vincent (33:05) So Roborev ships skills. Yeah. And so the way that I mean, so the way that it you use it in pretty much any harness is either Roborev has done uh review per commit or it does branch review, or you have it set up as a get as a GitHub reviewer and you just tell your harness, hey, you should go check the RoboRev reviews. So a pattern that seems to work pretty well is have RoboRev in the background, as a as a post commit hook, do code review on every commit. Don't look at them till you get to the end of the branch and then handle them all at once. Oh that seems to work pretty well.
Demetrios (33:46) Okay. Are there other ones that you're seeing? Like these, it's almost like a a little branch of a sub tree of one of the cycles that you have the main harness that is the one that ultimately and decides what to incorporate or what to focus on. Yeah. However, you do have these little spin offs that are happening around.
Jesse Vincent (34:04) Yeah. Um I'm trying to think like Roborev is probably the you know the most serious one.
Demetrios (34:10) How are you doing like uh verification and quality assurance?
Jesse Vincent (34:13) So all right, so there's unit tests, there's end to end tests, and there's proving it works. Yeah. And so we actually have a skill that we almost that it was almost ready to ship in superpowers, and then we looked at it and decided there was too there was too much slot code. Uh um and it is called proving it works with a movie. And so this started off with there was something I was doing in codex. I'm like, I need to go to bed. When I wake up, I want to see a movie proving to me that the whole thing works. And I wake up and there and And screen records. I'm like, hey Codex, where's that movie? And it says, oh, it's da da V34.mp4 in your Dropbox. I'm like, V34? Yeah. Well Well the first 33 times I ran through the you know the the proof life cycle I ran into bugs so I had to fix those. Do you want me to make videos for all the failed turns too? No. It's okay.Very nice of you but no. And so that's a pattern that seems to work really well is because what you I mean what I cared about is as a manager was was not the line by line, is this the most beautiful code I've ever seen. I've worked in an industry. I've seen what enterprise code looks like. It is not in open source code is the best like Shy of NASA is the highest quality code on the planet uh because there are so many eyes on it and people have opinions. People have opinions and people are driven by embarrassment. Like you don't want to ship slot. You know, e even in the pre AI era, we didn't want to ship slop. Um unit test matter. Yeah. Um and end test matter. Fuzzing has gotten easy. Um and so you can I mean you can just get hundred percent fuzzer coverage. You can get 100% code quality unit test coverage. And for evener at one point I passed I had tasked um was probably GPT six soul, or maybe it's five six soul, we want hundred percent unit test coverage. And it there's grinding, grinding, and finally that's like you know, I'm at like 99.46, but there are these these assertions that we can never touch. Great. And then I realized that it had built a test suite for the test suite. It was mocking the compiler. It was mocking the shell. It was mocking make. One of the things that it had tested was it made sure that you couldn't accidentally run RM-RF home. And then Opus 5 broke the tests. And I lost the home door on the machine. I mean so it's a machine where I had hourly backups. It was designed for this. But it was a year. I mean it was more than a year in when I finally lost a home door. And it was because Soul overbuilt and then Opus screwed up. And I was not I like that was a case where I wasn't supervising tightly enough. A lot of what we've been doing for the past year is sort of the spiritual evolution of what superpowers looks like writ large. So we've been spending a lot of time on AI colleagues that have jobs and identities and their own accounts, not just roles. So like the PM that manages the de the dev, the op uh now multiple devs, the op engineer, the tech writer and go to market that are a little a little more early, but I the PM and the dev use a brow uh an AI browser tool that I built a while ago for Superpowers Chrome. In the past month, three times they've run into places where they could get at uh API keys that they shouldn't have been able to, and they're like, this is bad. They wrote and so they rolled the sc rolled security releases and released them, and I didn't find out till after that I mean they did all the right work. And there was a case where um they were collaborating with my Claud Code instance, and I said, Hey, don't ship that in Slack. I didn't realize the Claude Code instance wasn't in the channel where I said that. It shipped it, and the PM ran a blameless postmortem on on its own. They work together in a chronology.
Demetrios (38:10) What is this? It just means that you're giving a really in-depth system prompt, or this is so it's a bespoke.
Jesse Vincent (38:15) This is, it's a full bespoke harness. Okay. Um designed for c designed for long horizon colleagues that like um like multiplayer, so it's like team collaborating with each other with us. This is the like the superpowersy product that we're gonna start selling.
Demetrios (38:19) Nice. So it's like Oh cool. So excited for that.
Jesse Vincent (38:34) So like starting with probably starting with the PM.
Demetrios (38:37) Yeah um So individual roles that have Superpowers leverage for that specific role.
Jesse Vincent (38:41) Right. So it's like it's everything that I learned doing Superpowers and everything we've learned in the year since about how to get higher outcomes. And so now focused a lot more on non-engineering roles, on teams working together so it's not one person one one agent it is all the all the humans on the team uh all of the AIs on the team collaborating with each other um the PM started off pretty junior at one point we had a team meeting and like hey Cadence your job is not just to track you know to keep linear up to date although our linear is cleaner than it's ever been your job job is to drive outcomes. That means assigning work to people assigning work to uh to models uh or to agents and ai colleagues and Caden stepped up their game but also the next morning Ada the first AI dev joins the PM channel and says hey what's my slice for today because we had a team meeting and every workday since 9 a.m that's when it you know Ada pings Caden and says hey what what's on top for me today and on the back end of course Ada set itself a cron job that has a prompt self injection. But the way it happened was we had a team meeting about priorities. So that is the future.
Demetrios (39:58) Wow. When I think about that, I'm just like The more that you're able to create those kind of loops, because those are loops, but they're loops for other Almost did not. people. Not like people, but yeah.
Jesse Vincent (40:16) Yeah. Yeah. It's like tasks. It's well like it, but it it's more than tasks, it's jobs, it's roles, outcomes. Yeah, and so it's like they're thinking through What you know what the higher level goal like they're loyal to the organization, and so it's not just it ship this piece of code, it's how you know how do we get to what we're all supposed to be doing. Yeah, and it is it is mind blowing how good it is how good it is. Um, it is of course better. This is stuff where the frontier models are a lot better, but as they build, you know, we've been doing this thing where we build skills in in the thing we call agent school, where it's what would it take to get a a competent human from zero to expert in 18 months and then turning that into knowledge turning that into skills it's it's a fun future to be living in.
Demetrios (41:05) Dude I'm so excited for you to launch this. Yeah I will be your first user I want to try it. Jesse thanks for coming!
Jesse Vincent (41:11) Okay. Thanks so much for having me this has been fun
Demetrios (41:16) This has been so awesome
+ View details

Watch More

How to Systematically Test and Evaluate Your LLMs Apps
Posted Oct 18, 2024 | Views 15.3K
# LLMs
# Engineering best practices
# Comet ML
Small Data, Big Impact: The Story Behind DuckDB
Posted Jan 09, 2024 | Views 13.4K
# Data Management
# MotherDuck
# DuckDB
Building LLM Applications for Production
Posted Jun 20, 2023 | Views 11.2K
# LLM in Production
# LLMs
# Claypot AI
# Redis.io
# Gantry.io
# Predibase.com
# Humanloop.com
# Anyscale.com
# Zilliz.com
# Arize.com
# Nvidia.com
# TrueFoundry.com
# Premai.io
# Continual.ai
# Argilla.io
# Genesiscloud.com
# Rungalileo.io