How To Delegate To An Agent Like You Would An Employee?

Demetrios: [00:00:00] This episode is incredible. I am here with a former comedian turned head of ML engineering at Monaco, Mikhail Eric, and former influencer s- turned head of Codex, uh, developer experience at OpenAI, and myself, former podcaster, uh, but nobody told me that I shouldn't do it anymore, so I'm still podcasting.

Demetrios: Let's get into it.

Mihail Eric: All right, guys. Unbelievable.

Demetrios: I like to ask folks what would make this a home run for you? I think it would be a home

Jason Liu: run if people just understood how powerful computer use was. If you're actually able to do the things you don't like about your job with computer use, and then do the things you do like with computer use. Uh, like, you know, with, with a [00:01:00] computer, right?

Jason Liu: It's like, I think many people talk about this idea that, oh man, I, I can't believe like, I wanted robots to like do my dishes so I can do the things I love. And instead, like the, the robots are like making movies and I'm doing the dishes, right?

Mihail Eric: Yeah.

Jason Liu: So like computer use is like the first step in like doing the dishes, at least on your computer, then everything else comes second.

Jason Liu: And so for me, success is just like- Set up computer use and, like, get all that stuff, like, out of the way, and then you can do the fun stuff of, like, talking to people and understanding what to build and how to build it, and not the mundane things of just, like, scrolling Twitter all day to, like, figure out what's wrong

Demetrios: And why is computer use versus just, like, using APIs?

Demetrios: Why computer use specifically?

Jason Liu: I think the question would be, like, what percentage of the, of the software has APIs that you can use, right? And with APIs, you have to, like, figure out how to install the APIs. You have, like, API keys, you know. Like, I have, like, four computers. I never know which one has, like, my X API credentials.[00:02:00]

Jason Liu: Uh, but I don't need that anymore, right? Uh, any application I can control with Codex in the background with multiple tabs open, and, uh, I just don't have to worry about these things, right? Uh, recently I had, like, a bunch of... I mean, this is, this is a tangent, but, like, I had a bun- I had a shipment coming in from Europe.

Jason Liu: There was, like, a bunch of, like, forms I had to fill out and I'm just like, "Codex, like, fill out these forms and, like, sign the documents." And it would download the PDF, read the website, fill out my form, search my address, then it would, like, upload it to Dropbox, press sign for me, download the image back, and then re-upload it to the form, right?

Demetrios: That's so wild, dude.

Jason Liu: Like, I had a Codex, like, uh, fax medical records, 'cause it figured out, like, the fax website that I can pay $10 to... So I'm just, like, getting my coffee and, and, and Codex says, like, "Hey, I need the, the confirmation number." And I'm like, "What?" And then, like, link pops up. I send the confirmation number.

Jason Liu: It's like, "Great, I've just paid for your, like, fax." [00:03:00] Dude,

Demetrios: wait,

Jason Liu: and you're giving- And I'm like having coffee with my friends. Like, that's what... Like, this is the, this is the weird thing. Like, that's weird- Yeah ... right? Like, you, you have to admit, like

Demetrios: E- explain this a little more because you're just giving it the higher level objective, and it's figuring out how to do that.

Demetrios: Like, you're telling it, "Hey, I need these medical records faxed." How much detail are you going into?

Jason Liu: Well, so for this example, it was the fact that there was a long email thread that I had been managing, and at some point they're like, "Hey, we just need you to, like, fax these X-rays to your doctor and, like, sign this form," right?

Jason Liu: And it was, like, 9:00 AM in the morning. I was like, "Great, okay, well, clearly I..." Like, I'm good enough at using Codex now or just AI in general where I read this email and I think, "Okay, everything that the AI needs is in this email. What is missing in this email?" And I press a button called App Shots on my computer that basically takes a screenshot of the app I'm building.

Jason Liu: Oh, uh, in, sends it to Codex, and then I press record and I just go, like, "I need you to, like, download these forms and send it to fax. Just [00:04:00] find a service you have my permission to, like, check out," right? The address should be in the form. Like, i- it's not just like, "Do this for me." And I hit enter, and I just, like, leave the house , right?

Jason Liu: Like, that's the world I wanna live in. Like, I don't wanna live in a world where, like, I'm the person up at 4:00 AM, like, trying to, like, get- Accepting ... more out of my systems. Yeah. Um- But now, like, l- like, everything I'm doing is something I wanna do

Mihail Eric: I mean, this is obviously a, a huge shift in how people, uh...

Mihail Eric: I mean, how people wanna work, for sure, but a shift in how maybe they, they do work and how they will need to work. W- what does it take for us to get this to mass adoption, right? Like, what... It feels like there's some guardrails- Yeah ... concerns here of some kind, uh, or some security things, or, "Hey, I'm gonna pay for me.

Mihail Eric: Yeah, then you're using my credit card." I mean, what, what needs to be in place for people to stomach this more? I mean, 'cause it seems like the technology is there, but maybe it's, it's the comfort, maybe, like, a social comfort piece that still isn't there.

Jason Liu: I don't, I don't even think it's that. [00:05:00] I just don't think people know.

Jason Liu: I think we know, right? Like, I think, first of all, I think paying for something is very scary. Like, it's... I, like, I only n- know that I felt comfortable doing that because I knew, like, a fax would be, like, $4. Right? Um-

Demetrios: It's gonna just do it 400 times and

Jason Liu: Exactly. It's like, "Oh, there's an error. Let me try again."

Demetrios: Yeah.

Jason Liu: But I think the big part really is the fact that people just don't know this is possible, you know? Like, when I see someone in, like, Photoshop and they don't know the short- like, the short keys, I'm like, "Oh, like- Yeah ... why are you clicking and dragging? Why are you r- why are you right clicking and then pressing copy?"

Jason Liu: And so I was like, "Yeah, there's shortcuts. Like, learn the shortcuts." Now, there's, like, shortcuts for, like, work.

Demetrios: Life.

Jason Liu: Right? The fact that, like, someone asks a question and I go, like, "Codex, answer the question for this person," versus, like, them just asking Codex itself. Like, so much of it is just people not knowing that it's even possible.

Demetrios: Let me Google that for you has now just become you tagging Codex in a thread and saying like- Yeah ... "Hey Codex, can you answer it?" Yeah,

Jason Liu: like, my answer to your [00:06:00] question on Slack is, has, like, sent by ChatGPT, you know?

Demetrios: Yeah. That's the pat- new passive aggressive, is like, actually, you probably should have asked ChatGPT first.

Demetrios: And I act- I was tongue in cheek. I posted this probably when I really started diving deep into figuring out what I could and couldn't do with coding agents. Mm-hmm. And I posted something along the lines of, "I got in a fender bender today and my first instinct was to ask my agent if it could fix it." And y- what you're saying is a little bit like, yeah, but that's not that far off, right?

Demetrios: Like, at least maybe not the car itself- Yeah ... the bumper's not gonna get fixed, but all the insurance bullshit, and all the, like, communicating with the person that you hit-

Jason Liu: Right ... that's

Demetrios: done.

Jason Liu: But just like, you know, find me like, three different mechanics and, like, look at their prices, and actually I want you to, like, email them and try to negotiate the prices to figure out if there's anything better.

Jason Liu: Like, that's not really an [00:07:00] API, but for you to find the API would be even more obscure.

Demetrios: Yeah.

Jason Liu: But just, just use my computer, right? It's already logged in. Yeah. And now I can do the actual fun stuff of, like, meeting my friends or, like, actually getting to, like, do some real work rather than just like... True Yeah.

Jason Liu: I, I- Or getting you another- I don't want to fill out a form ever again.

Demetrios: Yeah. Yeah, the forms piece. Dude, and so you gotta tell me more about that because I have a document that is just, like, me, all about me. All of my personal- Yeah ... information that would be- Yeah ... incredible if it was all over the internet.

Demetrios: Uh, but I seem to have enough trust to give the agent that and say, like, "Hey, whenever you encounter a form, here's all the information that you're gonna need," right? Is that how you go about it? Or do you have something, do you use password managers? What are you doing?

Jason Liu: Yeah, so this is really tricky because I think something like password management, like, should be in the Codex app.

Jason Liu: Like, I, I can't imagine a world [00:08:00] where, you know, by the end of the year you shouldn't be able to just have one password, like, enabled, right? I think many other browser agents already do this where the-- it's not the agent that's actually copy-pasting the password in, it's just triggering this API. You can already do this with, uh, computer use.

Jason Liu: Like, sometimes I'll just get a notification to, like, put my finger somewhere and I'm like, "Okay, let me... Like, where is this computer asking for? Okay, it's trying, it's trying to log into Gmail. I fine-- I'm fine with that, log into Gmail." These quality of life things, like, go really, really far, right? Like, even for folks who are tracking, like, experimental results, right?

Jason Liu: Some, some data is, like, in a Jupyter Notebook, some of it is in some, like, custom, like, Streamlit app, and then you wanna, like, share it in an Excel spreadsheet. That's also computer use. That is my job, but that is, like, the least fun thing about, like, anything that I'm doing, right? And I'd rather just, like, tell Codex to do it and then try to actually have a conversation with a human being at the office.

Demetrios: Yeah. This, um, [00:09:00] th-this does feel like you can leverage different pieces, so tell me more about the loops you've set up or the workflows. I don't know if they're loops. We'll just use that term because it's the term du jour.

Jason Liu: Yeah, I mean, I can start with the simplest one, which is I think anyone can benefit from just, like, a morning brief, right?

Jason Liu: So I have a pr- I have a pretty expensive morning brief because I don't really have to pay for tokens anymore. But basically, at 8:00 AM every morning, my computer wakes up, it reads a bunch of Twitter, it reads every email, it reads every Slack message, and basically builds out this, like, list of things I should be focused on, right?

Jason Liu: And it knows this because it's, it has all those memories about me. And when I read this list, I'll basically use voice input, I'll press the microphone button, and I'll just talk over every single thing that happened. It's like, "Oh, okay, it does seem like four or five people have issues with, like, their rate limits being expired too quickly.

Jason Liu: Can you just check-- Like, check for any kind of, like, SEVs on Slack? Okay, this person asked for, uh, a podcast. You know? Okay, figure out if comms has been attached to it and, if [00:10:00] not, attach comms. Someone gave me some feedback about, like, some chain of thought leakage. Is, are people aware of that? If there's a feedback ID, ask for the feedback ID, then once they reply, send it to the research team."

Jason Liu: And then I hit enter and I go back to doing something else, and that's a really simple, like, loop. The other thing that's really cool about the Codex app is Codex can create its own threads, rename them, and pin them. And so maybe the Codex comes back and says, "Oh, there is a SEV." Great. Start a new thread called, like, SEV monitoring, pin it in my sidebar, and make sure that thread wakes up every 30 minutes and checks on the status of the SEV.

Jason Liu: Right? And that, that thread should check Twitter to see if people are still complaining about it, and check Slack, and try to marry the two so whenever I come back to my desk, I just know the status of these things, right? And I... There's been cases where I'll look at my thread and it'll say, "The team has said that they addressed this," [00:11:00] but there was a tweet, like, two hours later that still had the same issue.

Jason Liu: "Do you want me to, like, share this reference in the Slack channel?" I might say, "No, I need you to DM them and ask them for a feedback ID, and then send it to this team." And all of a sudden, they're like, you can just manage so much more and my- the only discrimination I need to give is, like, permission to, like, post on Slack or DM someone on Twitter.

Demetrios: Damn.

Jason Liu: This is the same with features, right? In between a seven, like, oh, the copy is wrong or the alignment. It's like these are such small things. I can just say, like, "Okay, just make a pull request, fix that," and then any time anyone leaves feedback, fix that, and then just make sure that whenever I'm coming back to this, all the tests are green.

Jason Liu: Who should I... By the way, search- figure out who I need to DM to get this reviewed and merged. And once it's merged, do one final check where you, like, quote tweet the tweet and say, like, "It's merged." You know, like, that stuff sucks to do . Like, I don't wanna just be, like, on Twitter being like, "Issue has been resolved."

Jason Liu: But people care, right? [00:12:00] 'Cause it makes you... Pe- people will feel like we're more approachable, that we listen to folks. Yes. And it is just, like, a, a one reply saying, like, "We're looking into this," or, "There's been a pull request that has been made."

Mihail Eric: Yes. So I think there's, like, one category of things that you described which is the, the, the chief of staff type interactions where you walk in and, and there's just...

Mihail Eric: You're, you're glorified chief of staff, which we've been... We've wanted this for decades. Maybe it's finally here for real and people just don't know about it. And the second thing which you just started touching on is, is the, the feature building, right? And, and in fact, it sounds like the interface is still the same.

Mihail Eric: Well, what, what changes? Like, what, if anything, is different in that second category of tasks, uh, i- in terms of your interaction with it outside the chief of staff? You know, 'cause are you still reviewing the code? Are you not reviewing the code? You're just sort of talking about features, uh, signing off on features.

Mihail Eric: Well, what, what does that look like as maybe a second broad bucket of, of, of work?

Jason Liu: Yeah. So transparently, I don't work on the, like, biggest features of the product where, like, I'm not figuring out how to, like, unify, like, work mode versus, like, Codex mode. A lot of my job [00:13:00] on developer experience is just making things feel a little bit more seamless, a little bit smoother.

Jason Liu: And so a part of it is actually just having Codex help me understand the code itself, right? So for example, we have... We're making, like, thousands of pull requests, like, every day. And maybe someone is asking about whether or not the- the browser's gonna have, like, multiple tab support in the future. My coders will see this question, read the source code, and answer it, right?

Jason Liu: And it's like, oh, it is being worked on, but, like, things are not being reviewed. Okay, now it's my job to help figure out, like, how can I be, like, the human in this loop and, like, incentivize the team to, like, ship this feature or prioritize this feature? Um, and then for smaller things, obviously I can just fix them myself, but usually if it's, like, 40 lines of code, it's actually much easier to get these things merged in.

Jason Liu: And if the AI tries and it's bad, then we just, like, close the pull request, and maybe it's not a priority or maybe I have to make a case for it. But as more people are working on the software, I see my job as [00:14:00] Providing the data and the motivations for someone to really care about the thing that they're building, right?

Jason Liu: Like, anything is possible, you just have to care now. It's like, it's just my job to, like, care about these things,

Mihail Eric: right? So, so actually, like, it is fun about this, like, it's a unified interface then, right? I mean, even among... I, I, I, I separated a line sort of artificially between chief of staff work and, like, the development work.

Mihail Eric: But- Yeah ... but functionally, your, your interface is the same. Maybe, a- and sort of like where you insert yourself as the human in the loop, I mean, is that your most active effort here? Is, is like, what, yeah, I mean, may- maybe that's, that's a question that wonder- is left here, is like imagine a world where everyone has this.

Mihail Eric: Yeah. Where are we still, where do we sit in all this, in this whole process?

Jason Liu: I think we just have to care. Like, I think that's, that's literally it. Like, anything that happens at OpenAI is because some individual really cares about this result getting done, right? It's like, like the code, I- again, I, I don't really think about it.

Jason Liu: It's just about like, do I believe this is gonna be the right decision for the company, right? Like, one thing I think about when I was, used to [00:15:00] give people, like, advice on, like, building up their, like, businesses was just, like, like hard work is not really the thing that gets you all the results. It's about, like, making really good decisions.

Jason Liu: And AI- Right ... has made, like, everyone able to work hard, but it doesn't change your, like, decision-making ability, right? And like, we could build all these features, but with AI, I can just, like, really prove to people that, that might not be convinced because they might not have the same pain points as anyone else.

Jason Liu: I can say, "Hey, like, it's, I think it's really important that, like, I don't know, tabs per- are persisted across threads, because this is how people are working. And here's all this data to, to motivate this, like, this, this, uh, issue," right? "And this is the reason we should build this feature." Building the feature is pretty easy, right?

Jason Liu: And then how do you, like, broadcast this back to the world and say, "Hey, we hear you. We care about the problems you're running into, and we're gonna get this across the finish line"?

Demetrios: Do you ever, uh, the first thing that I think about when you have the agent [00:16:00] have the insight into the code base is like, I'm now on Twitter, I can @ Jason and be like, "What is, you know, the latest on the code base?"

Demetrios: type thing. Yeah. And potentially I'll get this, some really good, juicy details.

Jason Liu: Yeah, I mean, I think that's definitely a possibility. Generally, I have a lot of, like, protections on, like, y- like, replies. Basically, like, you cannot reply to things. This is mostly done with a auto review. So we have a second model review every, like, outbound request.

Jason Liu: So for example- Yes ... we, we have classifiers that determine, like, is this destructive information? Do we believe there is data exfiltration? That's a separate sidecar that runs on the main model. Um- And for the most part, it's always been the case that I think our models tend to not take the action. Hmm. So I'm always trying to, like, push it to go a little bit further.

Jason Liu: Um, so those are the things that I don't really [00:17:00] find that these are big issues, um, in my personal work. Well,

Demetrios: and talk to me maybe a little bit more about the different, like, checkers that you have and ways that you've set it up so there are these fallbacks and you don't just have one model doing whatever it's asked.

Demetrios: Yeah. It feels like you've probably got overseer agents in there?

Jason Liu: Um, I think the only overseer agent is the one in auto review. So if you go in the Codex Composer, there's, like, allow all, ask every time, and then auto review. And I think our alignment team does a really, really good job in just, like, making that auto review experience very seamless.

Jason Liu: Um, so it's like, so that's one thing. And the second thing is you just make sure that, uh... And this is something I'm less familiar with, but for example, uh, Codex is not able to send emails to anyone that does not have an openai.com email.

Demetrios: Oh, nice.

Jason Liu: So there's, like, at the MCP layer, different, uh, permissions.

Jason Liu: So for [00:18:00] example, like, you cannot commit, you cannot push to, uh, external repos without manually doing it, right? Like, it's actually, like, in the, it's in the environment to do that. And so even if I clone my own repo from, like, my Jason, you know, /blog- Codef is not allowed to push. Like, I have to run the push command myself.

Jason Liu: And these small things happen at different layers, right? The sandbox layer, the auto review layer, and also in the system messages.

Mihail Eric: One thing that you said that, that, uh, I think was... It was, it was a statistic you shared and, and sort of like, well, you know, we, we went by it really quickly, but I wa- I wanna zero in on it a little bit more and, and, and go deeper here.

Mihail Eric: You mentioned that in a code base where thousands of PRs are r- are merged every single day, which, which I think for most companies that would hear this would be like, "What?" Like, I mean, that's- Yeah ... that's unreal, you know what I mean? Like, I think many people's code bases are probably like a few thousand, uh- Yeah

Mihail Eric: commits and PRs in total. Um, and your actual business is probably at, have, have that sort of scale. Uh, what is it-- what did [00:19:00] it take to get there, right? I mean, how is this not just a Wild West of, of... Which clearly it isn't, 'cause we see the, you know, we see the, the, the platform working. Yeah. So it's clearly not a Wild West, but, but what, what did it take to get to that point where that was not the case?

Mihail Eric: That, that... Which was very much a real concern that one could have.

Jason Liu: Yeah, I mean, I don't know the details of sort of how we are doing our, like, source code management now. It is basically like one big monorepo. I think even, even now, like, a lot of the efforts we've... A lot of the investments we've made have basically been to make sure that, like, source control management is working well, make sure that, like, CI is running well and things are reliable.

Jason Liu: Um, but I'm just less familiar with that side of the world. A lot of it is just... Like, I can, I can speak to the Codex side, which is, like, one of my favorite skills in the Codex app is the main developers or the, or the lead developers of the Codex app have these, like, really great, like, finalized code skills And all it does is these, like, very detailed and opinionated [00:20:00] ways of reviewing the code.

Jason Liu: And so Codex knows to trigger them any time a pull request is being made, and just finds all these issues and starts self-healing. So I think one of the big things too is, like, the Codex team has made a lot of investments in self-healing before the pull request is even sent to GitHub and, like, CI is being run.

Jason Liu: Then it's the- it's a matter of making the right investments so, like, CI can run effectively and, like, you know, it's not just, like, terabytes of CI running every day. Um, but I think that's a problem, like, every, every big company is, is trying to figure out now, really. And I, I think for, for Better for Worse, I find that we are actually a company that, like, reviews way more code and, like, you still have to, like, work with humans to figure out, like, what are the right decisions to be made, at least in the desktop app or in the app server.

Jason Liu: And then everything else I think is, um, you know, a different level of- a different bar for quality, right? Like, even in our, in our monitor, like, all of my side projects are there. All of the, like, random demo apps are there. [00:21:00] And those things we can acknowledge are, are just a different bar in terms of, like, what gets merged in.

Demetrios: Dude, and the merge conflicts, there's not, like, a hell hole around that?

Jason Liu: Codex.

Demetrios: It just will figure that shit out. That is incredible.

Jason Liu: Yeah. You know- Yeah ... you invoke, like, babysit PR, right? Like, your job is to check the pull request every 30 minutes to make sure it's on, it's rebased off of main, make sure that every comment is addressed, make sure that CI is always green.

Jason Liu: And so it's very rare that I come back to a pull request at any given time and, like, nothing's, like, at the latest stage. Those are again, like, that's where I want computers to be working, right? Like, I need the computer to verify that the desktop app has, like, loaded and to click around and make su- and send a message and know it works.

Jason Liu: And I can again focus mostly just on, like, what I care about and what I think the users want.

Demetrios: Tell me more about the skills, 'cause you kind of hinted at different skills that you have and you- your favorite ones. I know that you had a QR code with a bunch- Yeah ... of cool ones [00:22:00] that we can, we can just drop the link in the show notes.

Demetrios: Talk to me through the... Or talk me through these.

Jason Liu: Yeah, I mean, I think the, the simplest ones that I care a lot about are just, like, code organization. So I have, like, my own, like, Git commit skill that, like, looks at the diff, organizes the commits by, like, groups, and then makes sure that, like, commit messages are just good and, like, you know, clustered together.

Jason Liu: Also different... I have different skills on making sure that the model writes like me, and that skill is updated every week. And, and all it does is it says, "Every week..." The automation says, "Every week, read my Slack messages and my emails and figure out the different kinds of, like, writing tones that I adopt."

Demetrios: Oh,

Jason Liu: nice And it's like, okay, like when Jason is like asking for deadlines, talk this way. When it's talking to a friend, talk this way. If you're talking to an executive, talk this way. If you're talking to a coworker, talk this way. And that's made it like much less painful for my coworkers to be, be sent AI-written messages, right?

Jason Liu: Uh, it still feels [00:23:00] like it's coming from me and, and like being respectful of their time. And then we have all of these different like triage and review skills. So in our documentation, for example, I have a skill that spins up a sub-agent that reviews the docs the way one of our teammates reviews their docs.

Jason Liu: And that skill is basically built by saying, "I need you to look at the past 100 pull requests that have been reviewed by, you know, Dominic and Charlie. Figure out what they care about, and then every time a pull request is being made, spin up these skills and review the pull requests." And Dom really cares about the fact that we've cross-referenced the documentation with the source code in the monorepo.

Jason Liu: Charlie really cares about the fact that m- like things are up to date and that these, uh, the wording is like correct, and, uh, those have been really, really valuable. And the last thing is just triage. I have one skill that just says, "Given a screenshot or given an app shot, [00:24:00] uh, here's like how you think about what I'm asking for."

Jason Liu: And in the same way that like you, you know those CEOs that would just like forward you an email with one question mark, and it's like your job to figure out like what the CEO wanted? Like, I'm basically doing that all the time. So I might send the codec a slap shot and it says, "Okay, clearly, clearly this is a, this is a tweet, and the user's complaining about something.

Jason Liu: It's complaining about the browser. Okay. Well, let me look into the docs to figure out like what is the Slack channel for the browser. Okay, it's the like browser feedback channel. How do they need to format the feedback? They want it formatted this way. Great. I'll post in the Slack channel with a, a citation for this tweet, and then I'll check in 30 minutes if anyone has responded, and if they have, I'll reply back to the tweet that says someone's looking into it."

Jason Liu: Right? So anytime someone complains on Twitter, I just go like app shot, question mark, Enter, and I just like go do something else. And these things take so much of your time, like you would not believe.

Mihail Eric: Right?

Jason Liu: Um-

Mihail Eric: This is, this is the Elon Musk skill or [00:25:00] the, or the, or the angry s- skill.

Jason Liu: Yeah, yeah, yeah. But like, this skill now is like so long, you know?

Jason Liu: Like if you, if you email me about like asking for credits, app shot, question mark, send, and it's like, "Oh, he's asking for credits. Let me read the email thread to see if Jason's promised anything. Okay, he hasn't promised anything. Check if there's an org ID. If there's no org ID, send an email asking for the org ID.

Jason Liu: If there's a number, if it's less than X, just grant it. If it's more than Y, you post in this Slack channel with context." But all I'm doing all day is just, like, sending question mark and a screenshot to Codex. Like, it's, it's, it's, it's awesome, right? And again, it lets me do the other things I wanna do.

Demetrios: And did you-- Do you have to-- Because I'm in the beginning stages of this, right?

Demetrios: And there are certain workflows that I want to figure out. But are you consciously architecting them and saying, like, "This is how I want this to be done," or how much are you just, like, letting the model cook versus you deciding [00:26:00] XYZ?

Jason Liu: Well, I think when I started my consulting business, I hired an EA, and the first month was just a ton of misunderstandings, right?

Jason Liu: It's like, "Hey, why did you say no to that meeting? Who d- who, who asked for that meeting? This guy? Well, that guy's famous. Like, why did you tell them, like, I wouldn't go to their office? You thought I was busy? But, like, this guy's..." "Oh, no."

Mihail Eric: Yeah,

Jason Liu: it's more important than this. "Actually, when you start reaching out to people," right?

Jason Liu: Like, "When you, when you start reaching out to these people, I need you to check their LinkedIn and figure out if they're important and really build an understanding of my values," right? I just-- It's impossible to believe that this assistant just knew who this, like, you know, famous investor was. Like, that's not your job.

Jason Liu: So I've really learned to just build these things as I go, right? You ask for a couple times, you build the skill. You know the skill is bad, and you just, like, give feedback constantly. And I, I think the reason I, I find so much success in doing this kind of work is because I, I have dealt with humans, and you become much more empathetic, [00:27:00] um, versus just like, "AI sucks, I can't do it."

Jason Liu: Like, for the triage skill, it is just a bunch of checklists, right? Like, and I just built out these checklists as if it was an SOP, as if I'm onboarding a new employee. And then the benefit is I can now share this with the team. And now the entire team can benefit from, like, this, like, feedback triage s- uh, plugin that I've built out, right?

Jason Liu: But it's just a markdown file that says, "This is every Slack channel that takes in feedback. These are the DRIs for this channel. Under what conditions do you tag the on-call? How do you, like, figure out there's a Sev? What's the comms that you should figure out here?" Right? You build that over time, just as you would an employee.

Demetrios: Are you sharing all these skills with the team? And how often are the teams or teammates saying, "Cool, I'm gonna take this skill but then make it mine," versus, "I'm just gonna run with Jason's," versus, "These two skills are very similar. Maybe we [00:28:00] can merge them"? That kind of, like, skill management type of thing.

Demetrios: Yeah.

Jason Liu: I, I think those are still pretty unknown, right? Like, if there's no real, like, simple ways of doing, like, batch, like, push updates of skills, right? Um, I think that we also find that everyone sort of tends to do things differently. But that's the same thing as, like, actually having employees, right?

Jason Liu: There's, like, a way of doing things, and everyone finds their optim- like, their optimal paths. And maybe they can, like, reconvene and communicate. But, uh, you'll be surprised. For some, for some things, I've had my Codex and a coworker's Codex communicating in DMs to build out an application, because, like, I had credentials that I couldn't share, and it was, like, a whole thing of, like, making sure our two applications could talk to each other based off, like, credential sharing.

Jason Liu: And they just, like, spoke to each other until we figured it out. It's very bizarre. Like, people will just go... Like, you'll look on this thread on a public channel, and it's just my... Every message I send is sent by ChatGPT, and every message, like, Gabriel sends is also sent by ChatGPT. And they're like, "Thanks.

Jason Liu: Can you implement, like, [00:29:00] uh, you know, JWTs so I can make a request using my credentials?" "Great. I've sent you like, a key in, in, like, you know, the secrets, uh, like, 1Password stuff. Can you please fetch that and then make sure it's set in an environment variable?" "Hey, it's done." It's, like, really bizarre. And this is just Codex running in a loop.

Jason Liu: It's just check, check that thread every, like, 10 minutes and make sure there's no updates.

Mihail Eric: Well, this is like the, um, there, there's, like, the, the OpenClaw, like, Arena or whatever it was called, where, like, where they put the different agents and have them effectively run their own, like, Reddit for a little while.

Mihail Eric: And I guess that, that's- Oh, yeah. Mulbook. Mul- Mulbook. Exactly. Yeah, yeah. Something to that effect. I have a, I have a bit of a, a, a, a different question or, or kind of line of questioning here, which is, you know, I, I literally had a, a, a meeting with a, with a, like a, a student or, or soon to be like junior eng in, in the work- workforce right now.

Mihail Eric: Uh, and I'm curious, you know, given what you've seen, trained in, in the old way, now a, a, an ambassador of the new way when it comes to engineering [00:30:00] and, and, and working with agents, like what is, what is still the defensible skill set for a human here? And, and specifically like, you know, I, I'll make it very concrete.

Mihail Eric: Like a student was asking me like, "You know, what do I do?" Right? Like, like I'm, I'm just talking to this agent all day. Like what... I was t- told CS was, was engineering and I was ta- had all these algorithms. I'm like not really doing that. Like what... It's not to say that it was useless, but it's definitely not what I signed up for when I got into the major.

Jason Liu: Yeah.

Mihail Eric: Um, and now I'm in this new world where I've- seems like all I'm doing is talking to agents. Uh, go, is this what my career is gonna be? You know what I mean? So- Yeah ... just curious what, what, what you've seen and, and, you know, living on the frontier, working on the frontier, what, what recommendations you have for folks like that.

Jason Liu: Yeah, I mean, I think the, the right way of framing is like my job is to like care about things that I think are important and be incredibly dissatisfied with like how things are, right? Because again, like at some point you're gonna be in a place where everyone works hard. So like working harder is like, [00:31:00] like spending more tokens is not gonna be the thing that matters, right?

Jason Liu: It's about making good decisions and like understanding like what are the things that I should be dissatisfied about and how can I care enough to fix these things? Like, you know, performance for example. It- the details of like some algorithm is not the thing that matters. The fact that I want things to like load quickly because this is like the right decision for people, and people should care about like apps feeling snappy.

Jason Liu: And like if you are unhappy with like how q- quickly things are building or unhappy with how things are loading, you will now be incentivized to figure out how do I actually make these things faster? And maybe, maybe the next model can't figure out these performance optimizations, and now your job is to be creative and make a good decision in the company.

Jason Liu: I think this is comes from the fact that, like one essay I wrote a while ago was like, if you want taste, you have to eat, right? And just like knowing what good taste, like knowing what good looks like is just about using different products, being dissatisfied with them, coming w- up with the vocabulary of how to describe why I like something or why I dislike something, [00:32:00] and like that's the real skill.

Jason Liu: And then once you figure that out, once you can put it into words, yeah, you just tell Codeus to fix it, right? But the front end won't be as good or the performance won't be as good because you care about the experience, you care about the design, you care about the performance. And then, yeah, you just, you just tell y- the agent to go fix that.

Jason Liu: Um, but I think- Like talking to the agent is not the job. But like making good decisions and caring about products is, is the job. And now you can just do that even more. But because you can do even more, you, you really have to figure out like where you put your attention and what, what matters the most.

Jason Liu: And those, that goes back to this idea of like your job is to make great decisions and like have taste and have discernment and just like be uns- be dissatisfied with how things are right now. And this is a durable skill for the rest of your life, you know? Whether it's like I think my food is like under-salted to like, huh, the design's not good, we should fix it, so like the performance is not good.

Demetrios: The beautiful thing I found with that, the act of [00:33:00] learning how to describe the delta between like what you're looking for versus where it's at and what is not quite right, is that you can just ask agents. It's like, "I am not enjoying this for some reason. I don't quite know what the words are to describe it, but let's go through it and figure out why."

Mihail Eric: And I wonder, um, i- I mean, I think that it's, it's such a, such an astute point, right? I, I, and you said something to the effect of like we all can... Agents have made it very easy to work hard, now we just need to work on the right things, and I think that's, that's a very succinct but, but on-the-nose way to describe exactly this phenomenon.

Mihail Eric: I think that, uh, I'd love to like go a little bit deeper here then. Like, like tactically, so, so someone... Yeah, we need to have judgment, we need to eat to have taste.

Jason Liu: Yeah.

Mihail Eric: Metaphorically, totally agree with all this.

Jason Liu: Yeah.

Mihail Eric: I am today trying to develop better taste. Like, am I going and s- you know, standing on the street talking with users saying, "What do you hate about this?"

Mihail Eric: Am I going and consulting on, you know, being a four-point engineer? Am I like... What, [00:34:00] what tactically am I doing to, to develop and, and exercise that muscle?

Jason Liu: Yeah. I mean, I think for me, like one thing I need to do more of is just like try out every coding agent, right? And some, like, and try them out and like try to find something I don't understand how to describe.

Jason Liu: They co- I don't, I don't... Like you'll know when you've hit your limit of understanding when you don't know why something feels way better than something else, and now it's your job to like understand, right? You can't, like you can't just like learn things by like looking at the menu. You really have to like try things out and try, try things for a long period of time and figure like, "Wait, why does this feel so much better?"

Jason Liu: If I can't describe it, it means I don't know, which means I'm missing some knowledge. If I don't know, then I can't build it, 'cause I'm missing some knowledge, right? And like developing that vocabulary I think is the, the right thing forward, right? Like for example, like how do we figure out like when, when does it make sense for the AI to block on a task and ask you a question?[00:35:00]

Jason Liu: I don't have a good theory as to, like, what that means. Like, like under what conditions do I wanna get blocked, right? I would hate to, like, tell AI to do some work, come back and realize 45 minutes ago it asked me a question.

Demetrios: That's

Jason Liu: happened to me- And it

Demetrios: hasn't done anything, right? ... so many times, dude. Oh my God, it's so frustrating.

Jason Liu: Right. But like, okay, so that means, like, there's this notion of, like, deference, right? This is a word I've been thinking about a lot, which is, like, AI deference. Like, when should the AI, like, reach out for help? What does it mean for someone to, like, need help? Wh- what, when do I decide that? And now I'm, like, thinking about this, like, theory of mind of, like, how, how do I simulate my boss's, like, behavior to figure out when's the right time to ask for help versus when's the right time to, like, try harder and struggle, and when is the right time for me to fail versus for me to, like, realize that I should not fail and should ask my boss?

Jason Liu: Like, that part of designing an AI system I think is completely unclear, and, like, AI is not gonna solve that for me. And like, my job is to really care about that, right? How do we [00:36:00] have, like, AI deference, and how do we figure out when AI needs to escalate? And these are the problems I think are, are gonna kind of be the problems of the next, like, couple of months, right?

Jason Liu: Um, but the implementation details will just be, like, some machine learning problem, right? Like- It will just be some like, oh, you're just modeling their behavior. Like, you can use the, you can use the accelerometer on their phone to figure out if they, if they like just sat down or something, or if you can look at the ambient sound in the, in the room to figure out they're talking to someone else.

Jason Liu: Like, the solutions are gonna be technologies, but like what you're trying to do can now be much more abstract. I don't know if that feels like a good answer to you, just like try things and really think about like these bigger problems, but just the- These dream outcomes are still just implementation details that we will have to solve with software.

Demetrios: Yeah, I've felt that while I'm trying to become a little bit more astute in interior design because I'm currently building a house. I have no idea what the hell I'm [00:37:00] doing at all, but I'm like, "I'm gonna live in this place for the next hopefully 20 years." And so I'm trying to like upskill as fast as possible on the basics, and you realize when you come into it, you're just like, "Dude, I don't know what the hell is going on here.

Demetrios: Is it better to have this stairway like that? Or should I... Oh, this person, you know, I watched that YouTube video and it said this. That kind of makes sense to me." And you just are floundering in the beginning. But it reminds me of that, I, I can't remember who had that incredible monologue about the gap between when you're like starting a endeavor and then when you become fully professional, like when a really good at it, and how ugly it is all the way up until you get good at it.

Demetrios: And that whole process is like the gap. Yeah. And you notice that there's a gap. You know there's something wrong because the [00:38:00] professionals look like that and you look not anything like that. And so that journey is almost like what the, I think it was like Ezra Klein potentially, uh, put

Jason Liu: out on the- Yeah, I mean, and that speaks to this idea like, you know, my taste for something usually exceeds my ability, and AI has like greatly raised the s- floor of my ability.

Jason Liu: But if I can't like tell AI what, like I don't have a lot of words for how I describe JavaScript. I don't have a lot of words to describe like interaction patterns other than like, "Can you make it move cooler? Can you make it be nicer?" I know easing is a word, c- you know. "Can you add some more motion?" If I don't have these, this language like that, it's still gonna be bad, right?

Jason Liu: And like obviously a better model will make it better. But, um, again, a part of it is just like it's my j- it's my responsibility to like educate myself on the language I need to use to describe what I want, and that has like always been the case.

Demetrios: Yeah. Uh, one thing you also were mentioning is when [00:39:00] to Bring, have, ask the agent to escalate versus to, like, just keep going, keep trucking.

Demetrios: And it feels like there potentially is a horizontal move where you don't necessarily need to escalate, but you could get help from some other surface area that, whether it's somebody else's agent or it's just some other way of looking at it. Yeah.

Jason Liu: Yeah, but that, that, that goes back to deference, right?

Jason Liu: Because I don't want my agents to, like, be DM-ing the on-call person at 4:00 AM in the morning to get permissions for some database. Like, I need you to, like, just go to sleep and wake up at 9:00 AM like a regular person, right? Yeah . 'Cause you're gonna make me look bad.

Demetrios: Well, well, I was saying,

Jason Liu: yeah . But, like, these skills of managing, like, I just don't, I, I, I can't describe what I need to put in my agent's MD for this to be useful for me.

Jason Liu: But I think once I do, it is my responsibility to, like, help the company steer the model development in a direction that shows deference or is respectful, right? Just because a model is more determined doesn't mean it's a better [00:40:00] coworker Does it matter that it's a coworker? Should the problems be that, like, it should be completely autonomous and not need help?

Jason Liu: These are now the problems that, like, AI can't help me answer, right? But these are the-- these are, like, really exciting problems to be had.

Demetrios: And

Mihail Eric: that goes back

Demetrios: to the taste.

Mihail Eric: Uh, I was gonna say o- one other kind of thing that... Since you probably spend a ton of time, I mean, as a, a, a very developer-facing- Yeah

Mihail Eric: uh, engineer, you work with a lot of developers. Yeah. I'm curious, um, what's the spectrum that you've seen around people that are... Like, what does truly AI-pilled mean today, and what is, you know, what are-- wh- where, where are we, uh, what does that full spectrum of, of adoption and proficiency look like? And how does one maybe go from the, the, let's say, the noobs all the way on the left side to the, to the, to the most AI-pilled, you know, like, like that, that, that meme where it shows, like, the brains getting bigger.

Mihail Eric: Yeah. You know, the brain is exploding or, like, sending up beams of [00:41:00] light. Um, what does it take to get to that level, uh, if you start from the normies, uh, or, like, the newbies sort of, uh-

Jason Liu: I mean, yeah. I think the biggest thing is just, like, you should be giving AI tasks that the AI can't do. Like, if you cannot do that, like, you are, you're, like, not believing in your teammate enough, right?

Jason Liu: Like, a part of being a good manager is, like, allowing your employees to fail in, like, safe ways, you know? Like, I'm not gonna just throw my employee into, like, a board meeting and, like, be like, "I'm not gonna be there. I'm, like, at a birthday. Go. Good luck," right? But maybe it is like, "Ah, I need you to prepare, like, a first draft that I can review.

Jason Liu: Then I'll, I'll go, like, build out the meeting material," right? Or it's like, actually, it's about, uh, you know, like, "I need you to do all this research for me, so when I'm in this meeting, uh, if there's really detailed questions, I'm gonna rely on you to, like, help me answer a question. And if, if you don't know, then I'll say I don't know, and maybe that's safe."

Jason Liu: Um, so I think a lot of it is just, like, [00:42:00] coming up with tasks that AI will actually fail at in hopes that, like, the future versions of these things will be better. Or even better, just by, like, changing the configuration, adding some more plug-ins or, like, updating the agents MD that these things can succeed.

Jason Liu: Because once you realize that you can delegate and once you realize that you can make your life easier, like, that's when things get really exciting. Uh, when I started my business, I was really scared about hiring an assistant Oh man, they're really expensive. What would I- what if I don't have enough work for them?

Demetrios: Yeah.

Jason Liu: And then you realize, wow, I sleep two hours earlier every day. I think I'm gonna live longer if I learn to delegate.

Demetrios: Dude. It's so funny.

Jason Liu: I had- Like, I, I really thought that way. I was like, "I think I'm gonna live longer 'cause I'm going from five hours of sleep to eight hours of sleep." Like, that's the thing I'm pricing against.

Jason Liu: It's not, like, how many dollars per hour I'm paying this person. It's like, like like, oh, oh my God, like, if I, if I just learn to delegate [00:43:00] and believe in people, and really, like, lean on the people who are, who are here to help me, like, that's the dream outcome. And then it's like, yeah, you know what? Fail.

Jason Liu: Don't worry. Like, just try it. It takes you longer, that's okay. It took you two hours to do this, like, you know. Next time, what can you do better, right?

Demetrios: I had that same kind of realization when I was writing out what I wanted an EA to do.

Jason Liu: Yeah.

Demetrios: And I realized, like, whoa, if I give them this information, that means I won't ever have to do that crap again, and I can- Yeah

Demetrios: just tell them to do it. Oh my God. Was like, now I understand why Stripe loves their fucking writing culture so much.

Jason Liu: Yeah. Yeah, but people just don't... I think developers are very much like, "I wanna build everything myself. I wanna do everything myself," 'cause, like, only-- Like, everything I do is well-engineered and everything you do is over-engineered.

Jason Liu: I just wanna do everything myself. But once you learn to delegate, I think it's, you just realize that, like, it's so exciting, right? [00:44:00] At, at some point it's like, "Oh man," like, "I wonder what else they can do?" What un- what- It's

Demetrios: a power up. Yeah ...

Jason Liu: what skills do I want her to start learning? Can I, like, put her into a course and, like, help her understand how to use, like, how can I make them better at AI, right?

Demetrios: Hmm. Yeah.

Jason Liu: Anything I've do- everything I've experienced with an EA, I experience with AI. What tools are they lacking? What courses should they take? You know. And, like, what information does, do they need from me to succeed? Like, if I don't even give them that credit card, there're just, there are just things they can't do, right?

Jason Liu: But if I say, "You know what? You're allowed to spend anything under $50," so many things open up, right? You're like, "Oh, every time I fly, I want you to, like, send my girlfriend flowers."

Jason Liu: You know? But you- Right ... you have to write things down, and you have to put in the work, and you need to make sure they have the information. So just, like, g- making sure these AI agents fail and just providing more information and, like, just having this memory bank, it's really gonna just make your life, [00:45:00] like, just not even, like, better, but just, like, more enjoyable, you know?

Jason Liu: Like- It, it's just like automate all the crap. Like, it's like you want the AI to, like, do the dishes so you can go, like, make art and, like, read poetry, and not the other way around.

Demetrios: Yeah. I found that quote, by the way, speaking of making art. It was from Ira Glass. I knew it was some NPR dude. And it is, uh, it's one of those, I recommend everyone go check it out.

Demetrios: It's the... I'll leave a link to it in the description also. Nice. Dude, the only last thing that I wanted to hit on, unless Mikhail, you wanna go down a different rabbit hole, is the threads and how you kick off new threads- Yeah ... and how you let the... Y- you, like, pin the threads and you have them continuously go back and check and make sure that it is on a schedule or, like, that heartbeat.

Demetrios: Can you walk me through that real fast?

Jason Liu: Yeah. So one of the things that I [00:46:00] find has been, like, an interesting UX decision is we have less custom commands than other, like, comp- company products, right? For example, I think, uh, many, many agent tools have, like, a /loop that creates these things for you, right? Or, like, a /newthread.

Jason Liu: Uh, we don't have those. And primarily it's, it's, it's two design things. One, I'm basically never typing, so, like, I would never have to, like, say /loop. I just say things like, "Make a new thread. Keep an eye on this." Right? So because I dictate so much, I never need to use these slash commands. So the two magic spells and incantations you have to remember is just, "Keep an eye on this every X minutes," and it will just do that.

Jason Liu: It will send a scheduled message every X minutes. So if I say, "Keep an eye on this every 30 minutes and make sure all the tests are passing," it will basically register a message that is sent to its own thread every 30 minutes, and it's like, "Hey, I was asked to keep the test passing." Like, "Let me check CI.

Jason Liu: Okay, this test failed. Let me fix this." [00:47:00] And that, that's one loop. And the second thing is just literally just saying, "Make a new thread." Right? And it'll c- it'll call the create new thread function with the, like, starting message. And it also turns out, again, if you type things, you're gonna be really lazy, right?

Jason Liu: Like, I'll type... If I type, I'm, I'll, I'll say like, "This does not work. Please check." But if I use a voice command, it's like, "This is not working. I tried these buttons. Maybe it's about something here. I saw some pull requests made last week attaching the surface. Can you double-check all the pull requests?" I just wouldn't type that, so talking is really useful.

Jason Liu: And then when you give all this context, when an agent makes a new thread, that first prompt is usually so much better, right? When you ask the agent to write its own automation, it is just so much better. And when you ask the agent to write its own goal, it is just so much better. And, like, that's how I think about these things.

Jason Liu: Like, I almost don't wanna ever be responsible for writing the prompts anymore, right? Like, I don't use /goal. I'll say, "Make a goal." [00:48:00] And that goal is usually much more detailed. It'll be like, "These are the tests that you need to pass. You should, you should, you should focus on having these tests pass incrementally by running, like, the pytest, like, specific file commands, and log them here."

Jason Liu: You know, that's way better than just like, "/goal, make the test pass."

Demetrios: Yeah, I

Jason Liu: love that too. Same with new threads,

Demetrios: same with automations.

Mihail Eric: I love that that's the right interface, yeah. I think that's, that's... I, I very much appreciate that. I think that there's a, uh, minimalism, I think is the name of the game here, and, and meeting people where they are in terms of how we want to be interacting.

Mihail Eric: If, if the, if the mental model here that we're trying to replicate is that of i- is that of the, of the, you know, assistant chief of staff-

Jason Liu: Yeah ...

Mihail Eric: that you're not going /goal, like go, go fix my, you know, go, go do my laundry or whatever, and like, don't stop until the laundry is done, right? Like, instead-

Demetrios: Yeah

Mihail Eric: more casual than that,

Demetrios: I think. What kind of chief of staffs do you have, Mikhail? What the hell? Well, I mean- I need to find one of them one that does the laundry shit. [00:49:00]

Jason Liu: Well, see, the chief of staff is not the one who does the laundry. The chief of staff is the one who finds and hires- Oh ... the laundry person.

Jason Liu: That's true, yeah. Right? It's like my assistant is not the cleaner, but if I say, "Hey, I've been really busy. I need you to go find me someone on like TaskRabbit to do this task. Look at my schedule to figure out when it makes sense for somebody to come to my house for three hours, but make sure there's like..."

Jason Liu: You know. Yeah. And in the beginning is you, you do have to say all of these words.

Demetrios: Mm-hmm.

Jason Liu: Right? And it would be impossible to like lo- well, you hire someone day one and just like, "Yeah, my house is messy." And then you're like, "Why didn't they book a cleaner?" AI sucks.

Demetrios: Yeah. But I have noticed that with the voice mode, even on AI products, right, if you're building a product that is user-facing, you have this potential to get so much more context if you have voice mode.

Demetrios: Because as humans, we just are willing to blab. Uh, just by [00:50:00] default, we talk more than we would ever type. When we type, we're very much lazy. We wanna get it done as fast as possible. We wanna stop typing as fast as possible and go do the next thing. But when we're talking, you're kind of like thinking, and when you're thinking, you're filling it with ideas, or at least for me, I give way more context when I talk.

Jason Liu: Yeah. My joke has been, uh, we have been pointing at things and grunting for like probably a million years, and we've been typing for like 30. Yeah. You know, we're just like evolved to just point and blabber, right? Like screenshot and like voice message is the closest we'll get to like pointing and grunting at something, uh, to get some work done.

Jason Liu: I mean, let me also add one more comment, right? I think in the beginning, I said, w-what does success look like? Success is just people really need to understand that computer use is amazing, it works, and we should be using more of it to get our, like, laundry done. I think the second call out here too is- I tell a line which is just Tony Stark has never been seen, like, typing into Jarvis, right?[00:51:00]

Jason Liu: Yeah. I, I promise you, by the end of this year, you will feel like Iron Man, right? You'll be able to use your voice and orchestrate your work, and the work will span code or your computer. It will spin other threads. These threads will start monitoring each other, right? And, like, the sooner you believe that computer user works and, like, start talking to a computer, the, the mag- more magical it's gonna feel.

Jason Liu: Uh, and like- Oh, yeah ...

Demetrios: that's the, that's a crazy future. I'm literally thinking through all of these different ideas and like, shit, all this stuff that I've been doing by hand, I do not need to do that. Like, how am I going to just try and push harder on what I can get my agents to do? But

Jason Liu: check out one, one of my favorite automations is, uh, I, I travel a lot for work, but when I book flights from work, it only goes to my open at email.

Jason Liu: So I used to have an automation that would just forward the booking thing to my personal number. Oh, nice. Now the automation is, if you see it, just check in and download the boarding pass, and then [00:52:00] text me the boarding pass. And so sometimes- That's cool ... when I'm just on my computer, I'll just see a tab open, and it just says like, "JetBlue check-in."

Jason Liu: And I look at the page and I... It's already, like, choosing the window seat, 'cause I know it knows I like the window seat. Sorry, the aisle seat. And then some other days, like when I'm in New York, right, I'm like, I'm usually out, and I'll just get a text message of the boarding pass for the flight tomorrow.

Demetrios: That's awesome. To

Jason Liu: then- These are just, like, things I like... These should be invisible and, like-

Demetrios: Yeah ...

Jason Liu: I shouldn't have to do, do these things, right? And it'll, it'll text me if it's like, "Hey, there's a good upgrade if you wanna, like, pay out of pocket for the upgrade." And it's like, "Oh, cool." Like, "Yeah, do that."

Demetrios: Um- And it'll call Comcast too, right? We can get it to call Comcast and...

Jason Liu: I don't know about Comcast. You might have to use a human for that, but...

Demetrios: It'll hire a human to call in for you.

Jason Liu: Exactly, exactly. Master

Demetrios: of trades. Yeah.

Jason Liu: But, like-

Demetrios: Oh, man, s- there's just, there's so many things that right now I'm doing and I am almost, like, frustrated [00:53:00] with myself for doing them, because after I hear you, I'm like, "Fuck, I could probably not do that.

Demetrios: Actually, I can figure out..." It just is front-loading the time and effort to make it a, like-

Jason Liu: A- and I think the difference is for humans who have never had an assistant, they're, like, annoyed they have to front-load this time. But when I'm paying someone $50 an hour, I'm like, "Man, I've, we've already... Like, you've signed.

Jason Liu: Like, I'm sending you a 1099 whether I like it or not. Like, I need to, like, make sure you know how things work, and if you mess up, I, I need... I can't have you mess up again."

Demetrios: Yeah.

Jason Liu: Right? Yeah. I think having experienced that has been- It was very eye-opening for me from the perspective of like being someone who is now contributing to what it must feel like to have like a super assistant.

Jason Liu: It

Demetrios: is like, it is just a lot of hard work. Dude, you know where you, um, and, and I totally forgot about this, how much [00:54:00] in the early days of LLMs you were looking at LLMs and, and I remember the joke was like, "Great, you just built yourself a recommender system-

Jason Liu: Yeah ...

Demetrios: with an LLM, with like this RAG. Oh, wait, RAG, you just, this is a glorified recommender.

Demetrios: Now it feels like we're so far past that. Does it not-

Jason Liu: It's incredible.

Demetrios: Dude, like thinking about how much that was in a conversation versus now what the conversation is, it's night and day.

Jason Liu: I mean, I can't even, like, I can't even imagine using like 5.5. Like, like things, yeah, I mean, internally, externally, it feels like I've been at OpenAI for like a year now, but it's only been like 120 days.

Jason Liu: But, um, you know- That's rough ... this is why I say that machine learning works. It like- Yes. And it rocks.

Mihail Eric: Yeah, exactly.

Jason Liu: More and more things are like in distribution, and it is [00:55:00] just incredible that like we can make this kind of progress. And the moment you find tasks that like the AI can't do, that just gets addressed in like a month