12-factor Agents - Patterns of reliable LLM applications // Dexter Horthy // Agents in Production 2025

Dex Horthy [00:00:00]: What's up everybody? I'm Dex. I could talk a little bit about myself, but I'm not going to. This is 12 factor agents and we're here because of all of you. So this talk wouldn't have been possible. I'll give a couple shout outs at the end, but this is basically my journey, Talking to about 100 founders, AI, engineers, CTOs, people building agents, specifically people building agents that are deployed in production, especially to enterprise, and making hundreds of thousands, if not millions of dollars in revenue and my journey in understanding how different it is that the top 1% of builders, the way they work compared to the bottom 99% and everything I learned. So if you're here and you're building agents, you are a part of this. And I'm really excited to share my learnings and especially excited for the questions, discussions.

Dex Horthy [00:01:05]: I might even try to wrap a little early so that we can have a longer discussion. What's it like to build agents in 2025? If you're like me, you went on some kind of journey that looked like this, where you decided you want to build an agent, you did some product design, some ux, you figured out what you wanted to build. And then we're all engineers here, we know how to use libraries. We don't build everything from scratch. So you pick up a framework and you get to work and you can ship something really good really fast with a lot of off the shelf tools. And it's 80% is good enough to get the CEO really excited about the project and double your budget and tell you to go hire a bunch of people. But it's probably not good enough to put something in customer hands. And we'll get into what I mean by 80% in a minute.

Dex Horthy [00:01:44]: And you realize that in order to get past that 80%, you end up nine layers deep in a Python call stack trying to figure out how does this prompt get built or where do these tool calls get injected? Where's this all coming from? And if you're like me and a lot of people I talk to, you kind of just start over from scratch and use the base API calls and you build your own thing. Or one project I went on was I realized that this wasn't even like that good a problem for agents. One of the first agents I built was over a year ago. I had a make file with a bunch of tasks to build my projects and I created a small agent and said, you have a tool to read the make file and you have a tool to run the make tasks. Go build my project. Couldn't figure it out. Again, this is GPT4. This was almost like pre 4.0 even, but it did things in the wrong order.

Dex Horthy [00:02:32]: It skipped steps. It just didn't understand how it worked. After two hours of just kind of messing around, tinkering on a Saturday afternoon, I just kept updating the prompt and I had the prompt very specific of like, use this one and then use this one and then take the output from this one and feed it into this one. And by the end of it I got it to call the tools in the right order and it could actually access the file system and it could build the project. And then I reflected and I said, wow, like, if I had not tried to build an agent, I could have written the bash script to do all. I already knew what the workflow was. I, I could have written a BASH script to do all that in probably about 90 seconds. So not every problem needs an agent.

Dex Horthy [00:03:10]: But I want to talk about performance and what 80% really means, because we talked about this because AI changes the rules. Can you imagine if your REST API failed on 20% of requests? I'd call that an outage. I'd tell all my customers the API is down. In pre AI, you had things like latency, uptime, cost and security. With AI, we also care about accuracy. And there's this other dimension of performance where it's like it's normal for your thing to be factually incorrect, to hallucinate. People have a tolerance for that. But your goal is to maximize accuracy.

Dex Horthy [00:03:43]: And so 12 factor agents. And we're going to talk about context, engineering. It's really all about performance in that sense of not speed, not cost, not latency. But how do you actually get the AI to reliably solve problems in a way that people can trust it, in a way that it can be trusted to do high stakes work. So I went on this journey. I think a lot of you have also been on similar journeys or about to go on such journeys. And what I came out of it with was that there are some core things that make agents really, really good. Doing a greenfield rewrite.

Dex Horthy [00:04:14]: Throw out, throw out the old software and we're doing the new software now we're doing agents. And we got our seven agents and here's what we're going to build. It's like, please, no take small modular concepts from your existing code base from agent building and apply them to your existing code base. What's cool about this is I don't think you need an AI background to do it. This is really about. It's a lot less about making models better or training or fine tuning or all the stuff we've been talking about. The RL talk was super interesting. This is about how do you take the best model you can get your hands on today and get the most out of it.

Dex Horthy [00:04:48]: Just like Heroku published This thing called 12 factor apps, which was a system of kind of conventions for how to build software that would run well in the cloud. We don't even have the term cloud native yet, but if you're deploying software in the cloud, you are doing almost all of these things and you don't think about it. You have version control and you have tests and you have a build pipeline. This doesn't need to be said anymore. And I think basically my thought is that we need something similar for agents. Just like what are the core building blocks? No matter how you're building it, no matter where you're shipping it, no matter what problem you're solving, what things hold true for all agent implementations. And again, this is just. I can't take credit for coming up with a lot of this.

Dex Horthy [00:05:25]: This is me talking to 100 people who figured this out in a way that let them build really cool products and really cool businesses and then writing it all down. So we put up this GitHub repo in April. Turns out a lot of people felt this. We were on the top page of Hacker News for all day. A bunch of impressions on social. I will just present this slide without comment and is on track to hit 10k stars with all of your help. I'll put up a QR code again later. That one was only up for a sec compared to something that feels super hypey like mcp.

Dex Horthy [00:05:59]: I think a lot of people identify with what we found and what we were saying here. So there's already 14 contributors to 12 factor agents. And what really shook out of this was this idea that blew up in June called Context Engineering. So this is. This is a picture. I made the text a little bit bigger, but this is basically a picture that was in the original 12 factor agents from April. SWIX even went so far as to put the main text on the AI Engineer talk from June as Context Engineering. And by the way, as we're going through, I am going to blast through the center meat of this presentation because if you really want all those details, you can go watch the other talk.

Dex Horthy [00:06:35]: And I want to get to kind of like looking back after three or four months and what we've learned since then. If you want to go watch the talk, you can go watch. It'll be easy to find. So that was June 3rd. On June 12th, Walden from Cognition wrote this really cool essay on how they learned and what they learned building multiple agent systems and what they realized doesn't work well. Six days later, Toby from Shopify came out and talked about essentially how he thinks about context engineering versus prompt engineering. Andrej Karpathy jumped on a week later. They had this whole thing of what is context engineering? All this stuff and like, the definition of context engineering in my mind, again is all about performance, but it's expanded a little bit.

Dex Horthy [00:07:20]: When I talk about context engineering, it's really about how do you build a really good agent? How do you build a bunch of software that's powered by LLMs that feels like a cohesive digital person that is helping you do your job or helping you accomplish your task or entertain you? Context engineering now as it is used today, can also be mean. Like used to mean how do you use an agent? So you're jumping into deep research or you're jumping into Claude code and you're giving it all the right pieces of information in a single prompt so that it can do its job properly. That is now also, I think, a valid use of the term context engineering. But I'm going to talk less about that. I'm going to talk more about building software that is powered by LLMs and you're engineering the context that goes into your individual API calls. So just to clarify that, you're going to listen to this talk and it's going to sound like I'm here to talk trash on all the frameworks. The things that are out there are incredible. The frameworks are really, really good.

Dex Horthy [00:08:15]: They moved us forward in a really important and powerful way. This is, if anything, a list of feature requests, a list of design principles. This is the things that frameworks in AI, the next generation of AI frameworks need to enable us to do in a way that saves us time on the stuff that's tedious but. But allows us to move fast and have control over what's going on. So if I'm not here to bash frame, what am I here to do? I want you all to question everything you know about agents. Unless you already watched the other talk, don't question any of that, but open the black box and understand agents from scratch and build great agents with sound engineering practices. Sounds good. Again, I'm going to blast through this middle section of the talk.

Dex Horthy [00:08:57]: If you want it in more detail, you can go Read the repo and you can also go watch the AI engineer talk, which is another 17 minutes of your time. There will also be QR codes at the end if you want to go read. I want to stop on this one though. This is the most important factor. The thing that makes LLMs good, the things that makes agents feel magical, is the ability to take a string like this and turn it into JSON. That looks like this is taking raw unstructured input and, and turning it into something that a program will use to go execute something meaningful in the world. Whether it's fetching data or changing data or pushing data out. Doesn't matter what we do with the JSON, those are the other factors.

Dex Horthy [00:09:38]: But you can start moving towards agendix software just by doing this pattern. That's factor one. Factor four, tools are structured outputs. I'm going to start speeding through. But basically tools are not that fancy. The idea of tool use, like I don't say tool use considered harmful in the sense of it's bad to give AI access to the world. I mean that the framing of the idea that tool use is some alien entity interacting with the world, I think is counterproductive to building good agents because all that's happening during tool use. I'm sure many of you have seen this under the hood is an LLM is outputting JSON.

Dex Horthy [00:10:15]: Deterministic code is going to do something with it and maybe you'll feed those results back to the LLM. So if you can make models like this, you can feed them into a switch statement like this and you can run some code, but there's nothing special about it. The orchestration and control flow stuff, again, I'm going to go through this pretty quickly, but the way we got here is basically we're building software which is already a graph. We've got these DAG orchestrators like Airflow Prefect, all these sorts of things. Really good technology that gives you a lot of affordances. But the promise of agents is you don't have to write the DAG yourself. You give the LLM a event and a goal and it will traverse through the graph in its own way and try to figure out exactly what needs to happen next. We model this as a loop and if you were going to animate what happens, you have your event that goes into the LLM.

Dex Horthy [00:11:04]: It's going to pick tools over and over and over again. At each iteration, you're taking the updated context window and feeding it into the prompt as a whole. This whole thing is your agent, but this is A simple software construct. If we can separate out the core pieces of this architecture that let you do this dynamic graph building in real time as the data is being processed, then you can get a lot more flexible and powerful applications again. It turns out the bigger that loop gets, it doesn't really work. We'll get back into that long context. Windows are really hard. Even as models get smarter, you'll always get better results with a really small focus prompt and context.

Dex Horthy [00:11:44]: But we can build on this. If you say what's an agent really? You've got your prompt, which is going to turn something, usually a context window, but could be initial event, into JSON. I don't know about you all, but I'm a software engineer. I've written a switch statement before. I've written a for loop and I know how to turn data into strings and strings back into data. A lot of these pieces are not that complicated and you can build them yourself. And so that's what we're going to talk about today. And you can do all kinds of fun stuff like breaking and switching and summarizing and judging and all these things you've read about really cool papers on what agents do.

Dex Horthy [00:12:17]: I would say like implement that stuff yourself because it's not that complex. 5 and 6 are similar. I'm not going to go a lot here, but this is about launching and pausing and resuming agents. Basically the idea that if you have your agent loop built like this, and let's say it gets triggered by a REST or an MCP call, we build some context and we push that into the model. Then we call long running tool. We can take that context with the long running tool selection, put it in the database, interrupt it and then go launch that long running job. When the job is finished, it will post a web hook back to us with the state ID and the results. We can use the state ID to fetch the context out of the database and then we can append the result.

Dex Horthy [00:12:59]: Then we push it right back in. The core of this thing is all stateless. So agents are just software. I think it's the big takeaway there and you should maintain flexibility. So that's 5 and 6 owning your prompts again. There are frameworks that can template you out a pretty good prompt as you get to that 80 and push through the boundary. You probably want to own every token of the prompt. Because LLMs are pure functions.

Dex Horthy [00:13:22]: The only thing that affects the quality of your agent is how good are the tokens coming out. Is it selecting the right next step? Is it Selecting the right parameters. The only thing that influences, other than getting a better model or changing the model parameters, the only thing that influences the performance of your agent is which tokens are you putting in. So I think Nimrod had a good talk about like you could pick from three different models and five different prompts and a couple other things and you have this explosion. So really focused on like if you're not changing the model and the temperature and the settings, how can you get the best results? By messing with the actual tokens you put into the prompt. I don't know what's better, I don't know what the right approach is. The prompts off the shelf from the templating frameworks may be the best, but I know the more things you can try, the more likely you are to stumble onto something that is really, really powerful and differentiated. Similar in your context building.

Dex Horthy [00:14:12]: Not going to talk too much about this, but the way you pass information in, you can have a string of tool calls, or you can just pass everything that happened in a single user message. As long as you're conveying to the model what happened and you're asking it, hey, choose the next step. You can do a lot of things that are not the standard recommended context window management. You can do this in Python really easily by stringifying stuff. We do a lot of traces in this XML stuff and just put the whole thing in the user message. I'll show you a real example later. If you wonder why all the pros use XML instead of JSON when they stuff data into an LLM, there's a lot more token efficiency. This is more meaning dense.

Dex Horthy [00:14:50]: The LLM has to read less stuff to get the same meaning out. And if you don't own your context window, you can't do these kinds of inspections. So that's owning your context window again. Tokens in, tokens out, and everything. Everything in building great agents is context engineering. The prompt, the memory, the rag, the agentic history, structured output. It's all part of this idea of how do I get the right tokens out of the model? And I don't know what's better, but I know you want to try everything, so that's on your context. Building the error stuff is pretty fun, but I'm going to skip over it, but basically by telling a model it did something wrong.

Dex Horthy [00:15:26]: Okay, it doesn't want to advance past this video. Yeah, there we go. So if you have an error, you can catch the exception, put on your thread and try again. Obviously this can spin out. You have Issues with the model making the same error over and over again. I'm sure lots of you have experienced that the answer is on your context window. Don't put the whole error in clean out errors once you get a valid tool call. All kinds of things you can do here to make this better.

Dex Horthy [00:15:48]: Contacting humans with tools this is another one I'm going to gloss over, but you should go watch the other talk is basically like a lot of people I've talked to are using tool calling for everything and having a specific call or multiple different types of functions that allow the model to declare its intent to respond to the user in various ways. Whether it's request clarification or final answer, we're done or et cetera, et cetera. Lots of different options. This is what some of our traces look like that enable the model to request human input as a tool call lets you do these outer loop agent things. All kinds of fun stuff. Again, I don't know what's better. If you get good results by using that final stop token that is like hey, we're outputting plain text now instead of JSON, you should do that. But I know you should try everything that's Contact humans with tools.

Dex Horthy [00:16:34]: If you are going to contact humans with tools, you might as well give the model the ability to talk to the human where they already are instead of of forcing everyone who's using agents to have like six different browser tabs for every agent. Put it in their slack, put it in their email. Your adoption will go up. And this is hard to do because human communication over email and slack is genuinely asynchronous. So without the control flow stuff you lose a lot of the flexibility you need to make these stateful asynchronous workflows work well for your problem space. The last one is this small focus agents thing. We talked about this loop and why it doesn't really work. What people are doing that works is using agents in specific parts of the pipeline.

Dex Horthy [00:17:13]: This is really like use LLMs for the things that LLMs are really good at. Don't try to have an LLM. Do your whole workflow. Find the parts where there is natural language or there's human in the loop, or there are things where it is valuable to take plain text and compact it into an existing problem to output a better result. Whether that's a content review or lots of other use cases. We do this at Human layer. We have an agent that does our deployments. Most of the workflow is deterministic, so let's say A PR gets merged, we will pull that in and run it, test it and actually, we're in a little short on time to skip this part, but basically it's about using agents.

Dex Horthy [00:17:53]: Go watch the other talk if you want to. Small focus agents, less than 100 tools, less than 20 steps. You'll get really, really good results that way. What if LLMs keep getting smarter? We talked about this a little bit, but basically what's going to happen is as they get smarter, there will always be a thing they're not good at. Yesterday there was a thing that was really hard, now it's easy. And there's a new thing that's really, really hard. And these engineering principles in 12 factor agents are going to help you push the boundaries and find that thing that the LLM can't always get right all the time and, and tune it up so that you're the only one who can get that right all the time. This was why NotebookLM was so magical.

Dex Horthy [00:18:33]: No one had ever seen something that would do things of this high quality, this reliably before. So that's kind of one of the big takeaways there. Stateless reducer, stateless transducer, call it what you want. Manage the state outside the LLM and you'll have a better time. And really quickly. A lot of people ask about evals and why evals isn't in here. I think the idea is like if you embrace this elements or pure functions and tokens in, tokens out then you get this and that everything is context engineering. Then you get this really clean just input output, pair of a trace of everything that happened.

Dex Horthy [00:19:09]: Here's what the next tool is and you can email on that really easily. I'm going to show you some code from open source agent that we built that manages issues in linear for us. Here's our prompt. You can see at the top the list of the list of different structured outputs we give it. Again, this is about 10 tools or so, including three different ways to contact a human, then our testing and our evals. This is using something called Boundary mlbaml, which I highly recommend. Very cool product, very cool team, but you can pass in. Okay, here's the email and you can make assertions about the intent of the output issue.

Dex Horthy [00:19:43]: So you can do these deterministic guesses on what's going to happen and you can make sure if I change my prompt I don't change the description. So this is a longer thread. Work out an error about team ID must be a uuid and we can assert that the next intent is to list the teams because it picked a hallucinated a team id. Here's a full the point is not to read this. You can go read the code, it's open source. But this is a full trace of Once all of this is done, we want to assert that it decides that it's done and we get our final answer. In summary, we're still finding the right abstractions. There's a couple of links like why libraries versus frameworks? Is it like a way to think about this? Whether you should prefer duplication over the wrong abstraction? This is from Ruby comps in like 2014 if you want to make a 12 factor agent this is coming soon.

Dex Horthy [00:20:28]: I know I said this a month ago, it is still coming soon. Thank you for everyone who reached out. We are looking for some collaborators on this as well, so I'm excited to share more there. Basically this is shadcn for agents. I won't go into what that means, but if you've been in this long enough, you will know what I'm saying there. In summary, agents are software. You can write a for loop, you can write a switch statement, you can serialize a string. So write the Dang software.

Dex Horthy [00:20:51]: LLMs are stateless functions. Everything is context engineering. Own your state and control flow. Find the bleeding edge. Be like the notebook LLM team, find the thing that the LM can't do and write a bunch of code to make it do that thing. Agents are better with people. There are hard things in building agents, but you should probably do them anyway. You should most of them.

Dex Horthy [00:21:09]: And one of the things we work on a human layer is this especially factor 11 making it easy to connect your agents to humans asynchronously. Because wrangling APIs and web hooks isn't the fun part of this problem. So most of what we do is open source because we think this is important to solve together. We do the boring stuff when we sell that. This is the agent to human protocol, which is in draft right now, but it's just a really simple four endpoint spec for how to connect agents to humans asynchronously. That's all the pitching I'm going to do today. I mostly just love automating everything and I had a great time learning from the community and writing it all down and sharing it with everybody. Thank you all for being here.

Dex Horthy [00:21:48]: Let's go build something. One shout out is to I do do webinars with Vaibhav, who was one of the kind of like original architects of the 12 factor agents methodology. We came up with this model together at a hackathon back in November. So if you want to learn more and do hands on workshops, there's a link for that on the last slide. Thank you so much. There's the GitHub, there's the link to the workshops. If you want to chat more, find me on X. I'm Dex Horthy.

Dex Horthy [00:22:17]: You can find me on LinkedIn. I'm much more active on X though, so please hit me up. We'd love to chat more. We'd love to see what you all are working on. I think that's it. That's 12 Factor agents.

Skylar Payne [00:22:28]: Awesome. Awesome. Again, another information dense talk. Loved it. Some folks were asking in the chat about the other talk that you were mentioning. So those of you who are asking, I put the link in the chat. So go ahead and pull that. And so just like maybe rip ripping through some questions here while we have some time.

Skylar Payne [00:22:51]: First thing you, you mentioned that for the Looks like you're building some kind of tool to like build 12 factor agents and you're looking for collaborators. Where do people go if they're interested?

Dex Horthy [00:23:04]: That is a great question. I would say ping me on Twitter or open an issue on the GitHub repo. The discussion will probably happen there.

Skylar Payne [00:23:15]: Cool. Perfect. Awesome folks. Again, if you have questions, feel free to drop them in the chat and I will transmit them back over to Dex. But maybe while we're waiting, a few other questions. Maybe they'll lead into some hot takes. So one of the things that caught my eyes talking about tool calling in this, and I'm just curious, given your experience of talking to a lot of people developing this 12 factor framework, what is your take on JSON tool calling versus code execution? Generating code instead of actual JSON tool calls.

Dex Horthy [00:23:54]: Okay. Instead of wiring up, let's say the resend.com MCP server, just have the model write Java a script that calls the API and then execute that code in a sandbox.

Skylar Payne [00:24:05]: Yeah. Or something that I often see is people defining util functions which are the tools you can call and then just generating code. And there's a lot of people that point out that JSON tool calling models are pretty bad at outputting JSON. And also it's hard to do composition with that. So just curious how you see this.

Dex Horthy [00:24:28]: Juxtapose with 12 factor agents. Yeah, I mean I think it all comes down to like thinking at the, like thinking at the level of abstraction inside the agent loop. Because you don't just have to blindly Call tools in a loop. And so if you can get the agent to output code, you can then pass that to another agent that reviews the code and asks a human for the right secrets. And you can kind of like delegate all these responsibilities out in a way that is deterministic where you're not depending on the agent to call code. As far as like agents calling tools with JSON, there's a bunch of different ways to do that. I know that the quality goes down when you do constrained generation. One of the examples we talk about on the AI that works show is this idea of when you do code generation and you're writing Python and it's output in a JSON object and you force constrained generation.

Dex Horthy [00:25:15]: When you get to the end of that new line, the model's really going to want the next logit is going to be 99% to pick the new line character. But because it's in JSON and that's not valid JSON, it will be for you're going to zero out that thing and then you're going to be stuck with like 1% for the backslash and then 1% for the n. And like basically like it becomes way harder to get the model to and backslash and N and almost every tokenizer are two separate tokens. By the way, there is no single token for the backslash N. I think that would be like if you had a code optimized model, maybe that would be the case. That's another thing to think about. If you can have the model write code in the way that it wants to write code, which is the way code is written in the training set, that's better. I think most models are much better at writing code that way.

Dex Horthy [00:26:05]: That's why the JSON tool calling is off. A lot of people have written custom prompts that tell it to output XML and custom XML parsers that handle the output. It's. It's tricky. Again, the Boundary ML folks have a really crazy rust based parser that can take busted JSON and turn it back into real JSON which is a really interesting approach to the problem. There's one other thing I wanted to say but I forgot so much here.

Skylar Payne [00:26:32]: Awesome. I think we got some other questions in the chat and we might have a couple of minutes. Maybe we'll take one more question somebody asked. Thoughts on safety Prompt injection, jailbreak and MCP with root permissions is an unsolved problem, but increasingly there's edge cases solved by the frontier models. So what's your mental model for working in this space safely, but still quickly.

Dex Horthy [00:26:55]: I mean, again, it comes down to get your hands in the logic of the loop. Mcp, I think, is really good for adding AI functionality or adding new functionality to AI for people who are not technical. So, like, someone or like, just technical enough to be dangerous. Right. If I'm not super technical, I can paste a pile of MCP JSON into Claude Desktop and like, suddenly connect it to all my stuff. I think if you're building agents and you're controlling the loop, and it's not just a blind call. Tools, call tools, call tools. Until some token comes out that says we're done, then you have a lot more control.

Dex Horthy [00:27:33]: And you can say, cool, we're going to call this tool, but before the output comes out, we're going to use a really small dumb model that's not going to follow instructions and use that to, like, as a guardrail. So, like, the answer is like, if you build in this way and you control every token of the prompt and everything that goes in and everything that comes out, then you get a lot more flexibility and control. And the answer is, like, it's going to be different for every use case. There's no, like, magic. Just like with memory, there's no magic abstraction that solves the memory problem. And the people I know who are building agents that have really good memory, these, like, AI tutors that help kids learn math and stuff, and they have really good memory over weeks and months. All of that is built from scratch. Awesome.

Skylar Payne [00:28:16]: I think that's all the time we have left. We have our next speaker getting ready, but thanks so much for sharing, so much. So remember, hit up that GitHub repo, find Dex on Twitter, and, you know, let's make better agents.

Dex Horthy [00:28:32]: Super dope. Thanks, Skyler. Thanks, everybody. Have a great day. Enjoy the rest of the show.