A Discussion on Loop Engineering

[00:00:00]

Good morning, everybody. While we're waiting, we have a very interesting topic today on loop engineering. Um, I need the guiding principles slide, please, Sparsh. Um, and it's gonna be a very hands-on session in terms of we're all gonna be chatting about real-world experiences, all the speakers are. We all have a little bit of time to talk.

I will let Sparsh take you through who's gonna speak when, and then we're going to open it up for you to do the same. There has been talk, and Sparsh, I don't know if we agreed on this, to have a follow-on session where people could take the ideas from here, implement them, and then we come back and actually let people show what they've done, just the way we're about to.

Um, have we decided to do that as a follow-on? I am not aware of that author Okay, then leave that alone. But if people are interested in doing that, put your notes in the comments section and we will see what the vote is. So the guiding principles today for a very hands-on [00:01:00] session is you are-- we all are here to learn.

We all are here to contribute. And basically, the more you put in, the more, uh, you will get out of the session. So we have some rules. Number one, this is a no judgment zone. That is, no one should be afraid to ask a dumb question because there are no dumb questions. And let me tell you something, you'll find out in my presentation, I'm probably the master of dumb questions, okay?

So it's as much giving me freedom to ask a dumb question as all of you. But the fact is there are no dumb questions. This is a safe zone, no judgment zone. And so everybody please realize that and bring that attitude to the table. There is a document, it is in the chat and I will put it-- It's in the Slack, and I will put that in the chat as well where we take Q&A.

It's all based on first come, first served. So what, it's like a stack, right? The first person's question gets answered and we move on from there. And so you can find that [00:02:00] document, um, as soon as I chat, put it in the chat or you can find it in the Slack for the reading group. And that's basically it. So feel free to ask questions and Sparsh let them know if they can interrupt us.

I don't know, I'm happy to be interrupted but not sure everyone is. So go ahead and take it away Uh, sure. Thank you very much, uh, for that

I think it's more of, uh, the way how I believe it's more of a interactive session, not exactly when the speaker starts, but in between when they give some inputs. Maybe let's have more of a interactive session today

So starting off with some, uh, today we are going to talk about something that's really interesting, and that is the loop engineering. Every time I go on the Twitter, I go on the X, I hear the same thing again and again. I don't write prompts anymore. [00:03:00] I write loops And before we dive deeper into this thing, the way how we are going to set up the agenda for this thing is, the first is to understand what exactly are we talking about here when we say loop engineering, how we exactly go here And the reason why we do that, because we want everyone to start from the same page.

That would be the second part or the second segment of the session for today, where we would have some really great practitioners who will come and share their experience on how they started to build some of these systems, what are the good things that they have done, and where you can improve to make your systems better.

And then finally, it would be more of an open discussion where everyone can pitch in, share their own thoughts, and share their experiences Now coming back exactly to what, where we started. I don't write prompt [00:04:00] anymore. I don't prompt my cloud. My job is to write loops. Is this a new buzzword or is this some sort of a thing that has already been there and-- or is there some sort of a fundamental shift?

Sorry, I don't see the video anymore. Did you stop sharing? Or is it- Yeah. Sorry. Baldemar, is that young Baldemar?

Valdimar Stórö.

It will lose sparse? Yeah. I suspect he will be, be right back I suspect the same

Okay. Well- Can, can you hear me? There we go. All right. Yeah, you're back. Okay So we have been talking about, we have been hearing about these loops. Is there-- Is this a new buzzword in the town, or is it some sort of a fundamental shift that we have? To understand where to start with the journey that brought us here, we wanted to solve our software problems and to industrialize the process of how we do the software development.

We started with prompt engineering. It solved one problem that expo-- then exposed new ones. We moved to context engineering. It solved some of the problems, then again, new problems emerged. Then harshness engineering, and then finally the loop engineering. The whole higher vision that we are talking about is we are industrializing this process of software development.

We started with isolated prompts, and now we are moving towards building these software factories. But how did we exactly came to this point? Let's understand a brief of the journey. [00:05:00] First came the prompt engineering, where we started to define the prompts and give it an ability to think. We give, we give an input, it thinks about it, and it gives us an output.

And you were verifying what exactly is happening Which brought to one of the problems that when we were looking for complicated workflows, what if prompt or the model is not the problem, but model having the right information is the problem? That brought us to the second part of the engineering that we sometimes call the context engineering.

Now, to solve this problem, we gave the tools, the instructions, the, uh, the history, everything to the model, and we thought our problem of building this software factories or automated software development is done, is fully solved. But still, it brought us to new problems that whatever [00:06:00] software we built, we built in these environments, production environments that we have.

And even in the context engineering, we were still verifying what is happening in every turn. Sure, we were able to see a few things which were working, but still, for working and automating these long horizon tasks, we are not yet there. And then finally came something that we all talked about a lot, harness engineering.

Now, we not only gave the model the internals, but we also gave the model the external environment, the whole scaffolding for the model or the system to being able to do everything by itself. And this was like being blown away what CloudCode is able to do, or some of these harnesses are able to do. That you give them a task and sometimes they do that, uh, cognitive planning, they do some sort of a, uh, hierarchical folding, and in the end, [00:07:00] give you the code, give you the full-blown feature.

Still, the issue with this is we were-- we are still verifying what is happening after a turn. After every turn, we see if it's right, if it's wrong. If it's not right, then we fix it, repeat the whole process. But this is where things started to get interesting. We started to see few things we have started to do on repeat.

We made skills for that. We made sub-agents for that. We started to have initial versions of workflows. Then finally came the point where we realized what if we make the loop or the feedback loop smart enough that it can verify and optimize itself. That is where we start with loop engineering Now, the model is not the only thing that's smart.

It's a feedback loop that is smart enough that it can verify and optimize itself such that finally, at the end of the day, we [00:08:00] are getting a full-blown feature or a full-blown product. And that's where we start to industrialize the whole software engineering. It's an era of building software factories. And here we are no longer verifying what exactly is happening.

We have shifted the parcel or the main thing to the verifier or, or the optimizer of the feedback that we have. And the system is now so good that we have built this feedback loop into the system. Now, in today's session, we have some really interesting people who have not only read about it, but who have started to implement these things in the production environments that they are working on, and they'll be sharing some of the great insights of how they did it and how we all can get started.

So let me call upon the first practitioner that we have right now within us, uh, Arthur, to the stage. Oh, wow. I get to be first, huh? All right. Let me [00:09:00] share my screen, if you don't mind

Hello? Whoops, what happened? Uh, can you share the screen? I'm trying. My thing is locked. Hold on

Um, I'm gonna have to do

I-- no. Valentin? Here, I thought you were back. I'm back. Yeah. Okay. Um, let me bring this over here 'cause I have two screens and it's over there, but I have to work here so I can look at you guys. I'm timing this 'cause I was promised I would do it in 10 minutes, and if I find I'm running out of time, um, then I'm going to cut it short.

So I'm gonna start now. All right. So if I can bring up my... There it is I'm gonna blow through this pretty quickly, and if you have any questions... And also, I am just now hitting the button to put two documents into the Slack channel, the reading group channel, that are for you to keep that may be helpful to you.

They're my working documents, but I think you'll find them interesting, and they can work as examples for you. Now, what I'm gonna do is I'm gonna talk about where I started, because unless I talk about where I've been, it's hard for you to understand where I'm going. Secondly, I'll talk about my plan, and then from that, I'm gonna talk about what I've already learned.

I haven't finished this yet. I'm in the midst. I started last Wednesday when we were doing this session and said I would go as far as I could go in the time that we had. So let me start with some numbers. 109,895. That is the number of lines of code that Claude, not me, has written since January 1 when I started my project.

Now, I am not a [00:10:00] production-grade coder. I've been a chief product and engineering officer. I'm a data scientist. I write in Python. But I have never written production-grade enterprise code myself. And as a result... But I'm building a production-grade platform to run for hundreds of thousands of people. It has an SDK, it has APIs, it's got all sorts of fancy architecture, and so it needs to be stable, it needs to be scalable, and it needs to be secure.

And I certainly have never done that, and I don't know that I can trust Claude or anyone at LLM to do that. And so I've had to depend on my harness to make sure that that's the case. All right? Now I'm gonna take you through the evolution of that harness very quickly because of time, and I'm gonna talk about where we're going from here.

But there's some other numbers. 1,233. That is the number of hours that I've put into this since January 1. That means I've written 89 lines of code per hour. Now, I have no idea if that's good or [00:11:00] bad, but I bet the enterprise folks who've worked in big enterprise engineering teams, if they only wrote 89 lines of code per hour, they probably wouldn't have a job in a year.

So I feel slow, and I also feel burdened, and I wanna go faster. All right? So let me tell you how I evolved the harness, and I'm gonna go through this very quickly and stop on only a couple of points because I wanna share a couple of things that I think may help you. I started with a very simple site just to test things.

It was a file upload site. I did vibe coding, and by the way, you may think of me as a vibe coder. If I am, I'm a pretty damn sophisticated vibe coder coder But I just did this the way you do, threw it out there, did it without a spec, got it built, and merge and push. Great. Except what I learned was that Claude, while writing code for the next thing I wanted fixed, would very easily break what is already done.

So lesson number one is you needed regression testing upfront. And so we did that, and I evolved with putting in architectural [00:12:00] plans. Then I had to do validation testing, and the reason is I was riding along and, for example, I noticed that Claude was hard coding variables, okay? And it's like, hey, I'm not the greatest coder in the world, but I know that a variable like that should be in a config file that I can easily change and is used in all places.

And I thought Claude knew how to do this. So now I'm starting to have to write validation tests. I'm writing them as Python tests and going forward. And then I get in a more formal ticket management, typical board management, just finding epics. And what I found was that my epics, and I'm still finding this, that we get in the middle of them and they grow.

I... Things I didn't anticipate, things I find that Claude did wrong, and suddenly this epic gets pretty big and breaks lean principles. And this is one of the things that I really want to fix. So There we go. So after going back for a moment, so as I built the validation, at one point Claude is still doing dumb stuff and I'm seeing stuff that I don't like.

So I said, "Timeout. There must be software that's [00:13:00] out there open source for a QA platform. There must be best practices. Let's go research that." And so what I'm gonna do here is bring up a, what is somewhat of an eye chart, and I hope you can see this because I'm having to work over here. And what I did is I built a Q...

And this is the doc- one of the documents in the, in the, um, uh, chat. Um, I looked at two things. What did I wanna QA for? What are the things that I was worried about? And you can see on the list sort of what they are. Um, and I can hardly see them, but this is the list of things like maintainability is one.

And then these are the steps in the process where I wanted to implement those cha- implement those quality controls. And so what I ended up with, and this is a bit of an eye chart, don't worry about it, is this is the steps in the process. These are the things I wanted to optimize for. These are the code packages that I found that would solve the problem at that particular step.

I spent two weeks taking that and [00:14:00] putting all of those code packages in and putting the processes in. The yellow ones haven't been done yet, but the green ones are actually in place. And the problem that I found is I put it in place, I put it at push, and push would just like die. It would take forever because the number of thi-

Uh, so that wasn't just me. Sparsh, you also, uh- Uh, yes ... froze, right? Yes, yes. Arthur is maybe running another- Okay. Seems to be reconnecting. Okay. All right. So by the time Arthur is joining back, people who are attendees with us, if you have implemented some sort of loops, would love to discuss it at the end of the session.

But, uh, you can put in the chat some really great observations you had by building these loops. Yeah, we also have the questions document. There's a link in the chat. You wanna ask something there we can talk about later

Um, I'm posting, uh, this Q&A site again. If anyone has any questions they want to take at the, take at the next part of the segment, then please put your questions here

Yeah

I can start my presentation if Arthur is not back in a bit. So

And show off a seven-month-old baby in the meantime

We have some comments in the chat already from Tanmay Tiwari

Maybe it's taking Arthur some time. Uh, Vlad, would you like to go? Yeah. Um, yeah. All right. I will share my screen and start talking about my-- what I've been doing

Do we see my screen? But yes, you're good to. Okay. So hi, my name is Valdi, and I've been involved in this reading group for a while, and, um, I haven't done much like professionally in terms of this loop engineering. I've worked a lot with agents, but never gone around and closed the loop and trying to automate everything.

Um, so recently I started playing around with, um, loop engineering since we had this, this reading club session and I built- We had a power outage. Power outage. Okay. I started my talk. I just started, so if you wanna go back to yours, I haven't really started. If you wanna take over again, Arthur. Are you ready or?

About that. Can you guys hear me? Yeah. Okay.

So Where I was, was this issue of the QA harness that I took two weeks to build and it blew up my process, my push process, and I'm just starting timing again. So I had to pull it out. I do run it manually now with a little bit of changes. So that was, um, and I need to bring this up over here. Sorry. That was, uh, one of the learnings.

And then I was still getting code that I'd find things that were wrong, and this is one of the most important things that I've done, is to create a dialectic where Gemini reviews Claude's code in an ant- an adversarial way. And what they do is they, you know, Gemini's very arrogant. It'll say, "Oh, [00:15:00] Claude doesn't know what the hell he's doing."

Not that quite that way, but pretty close. And they'll argue until they reach consensus, and that made it get better code, definitely more stable and more scalable. And what I ended up doing is automating that, and now I use Gemini and Fable. And I mention this because Gemini and Fable work at two different levels.

Gemini is very good at the lower level of architectural review and implementation review, and Fable's much better at the big picture. And what I'm finding now is between the two of them, I get even better results by having all three of them doing it. Yeah, it's a little expensive and I don't let... I mentioned before in the chat that I do not let these guys go off and run totally automated, and I won't for a while because just a couple of days ago, Fable ran, started 75 freaking agents in parallel and burned through all my tokens in about 10 minutes.

And that's a lesson learned. That's why I don't trust them. So then at the end, I'd get code that came out and I put it into then a [00:16:00] post-code review before push and let Gemini look at it, and now Fable, and they would come back with 10 things. And I'd end up in this loop where I had to go back, redo it, go back, redo it, and then we go to push, and even then there are some things that I have to redo.

So this is my process today. Don't worry too much about it. I'll leave it up here for a moment. All right? But it's a 14-step process. It's well documented, it's well defined, it's, there's, it is architected, and it is explained in detail in some of my architecture documents, and I am a manic person about documentation in case you can't tell.

So the problem is it is still hugely painful and error-prone. I mean, uh, driving me utterly nuts, and if you knew the number of F-bombs I have thrown at Claude, if I ever did that to a real employee, I'd be fired in 30 seconds. What I need is a self-evolving platform to help me optimize, speed my process, and reduce my stress.

'Cause the stre- uh, ask my wife when I come out [00:17:00] of these things. I, I literally, the other morning at 6:00 a.m., she was asleep. Her be- bedroom's like right there, and I was about to shout to the world. I had to do it silently. Trust me, that is very stressful. So What do I wanna optimize? Now, I'm going to actually show you a different page because I think it's easier to see it there

Here. I hope-- let me blow this up a little bit. And so, um, the number one thing I care about is this code issue, this rework after I get ready to push, where this code loop keeps coming back with the code review. I want the code right first time every time. And then there are 10 other things that are what I want to optimize for, and they're here on the list, and some of them you may recognize, like instrumentation should always ship with the code.

And Claude is very bad about following that rule, even though it's in his [00:18:00] instructions, okay? So Sorry. Before I could start, you know, here's the optimization loop. So my design debates that I taught you about at the beginning, the architectural implementation plans, those are all documented in a file called Integra- Implementation in a folder.

Those go in as signals. My code reviews, now I'm keeping in the same folder. They go in. There's also telemetry. I have built some in feedback. That goes in. And now we go to recognizing issues, diagnosing them. This is the loop. Proposing, approving them, I have to still manually approve, integrate them, measure, and come back.

And at some point, this is going to get very, um, I'm not writing code, I'm not writing harness code. It'll write itself. But I'm not there yet, and I never will trust, at least not anytime soon, trust the AIs to work totally without me. And this is the point I'm gonna come to in a minute. This is the real pain point is how you write the code and the integration code while you have to build a business.

I'm only one person. So how does that-- how did I do [00:19:00] g- do both at the same time? So the other thing you have to do first, I said I want quality code. What is quality? You know, right? Quality can mean different things to different people, and you have to define it for you. Okay? So for me, it's architecturally my code has to meet best practices in the industry, A.

B, this thing we have called our hexagonal architecture that our thing is built on. It's the high-level architecture. It has to have separation of database layer from logic layer from presentation layer. You think that's-- I mean, I'm again, not the great programmer, and I think about that. My first business, Claude did not do that.

It was like, duh, why didn't you do that? So I have to have that as a guideline. There's this thing called meso-level architecture, the middle level, et cetera. I won't go into the details. Then follows code at a level-- deep code level best practices. And this comes back to is my existing platform complete?

You saw that it isn't, but are any of those yellow items, are they really so important I need to stop now [00:20:00] and put them in rather than keeping going? Um, it has to follow the design and the relevant architecture docs, 'cause I have a whole set of architecture docs. Then it has to follow the implementation plans.

And then... Whoops It can't miss. This is the problem. There are related issues that start to pop up in the middle of my epics. I told you that. And it's like, no, that shouldn't happen. You should identify all of those up front so that once we get into the epic, it just flows. And it's driving me utterly nuts.

Then the code has to behave as expected, obviously. It has to be clean, well-documented, maintainable, and easily explainable to you if you were to come in and look at the code for the first time. I'm looking for a VP of Eng- head of engineering to partner with me, by the way. Anybody's interested, come talk to me.

Um, and then it also has to be self-check. It has to ship both with its tests and with the instrumentation into Sentry and other things. And the UI, we have UX guidelines, believe it or not, or I do, and it has to follow [00:21:00] those. So here are the lessons that I have learned so far for what I've gotten through in the last week.

First of all, like I said, identify what you want to optimize very clearly in as much detail as possible. Identify what quality means to you and your code. Third, and this is what starts to happen, you have to identify your failure modes that need to be handled. And k- what happens as you start this process, you start finding them, you start being aware of them.

You need to keep them somewhere so you can make sure that they get fed back into the loop. And now I'm gonna show you one other, uh, document, which again is somewhat an eye chart. But this is the document I put into the Slack as well. And if you come down here, this is my design document that I'm using as a guide.

And in this, and let me blow up this These are the examples that I found. So [00:22:00] this is where I put my, my problem children. And you can see it's listed by what area it covers. It captures what is the problem, what it thinks the problem was the cause, and then how we're going to handle it. And those are three things you should capture.

Those three things you should capture for everything that you find and categorize them because you're gonna come back and try to fix those because now what you're gonna do is-- By the way, before we go to the next step, is you probably already have built a great deal of this. I went to do this in the document that you saw, and Claude said, "Oh, you already built a lot of this."

I was like, "Oh, wow, I'm smarter than I thought." And so there you go, um, and it's about building a loop now. So most of you have already built a lot of this. So then you write the plan. That's what I put into the Slack and what I just showed you, and that's where these failures come in because you can now identify very clearly each failure mode to create a detailed set of implementation approaches and options that you will then come back and work [00:23:00] through.

You need to have very clear metrics of success. What is success for that loop? And identify the enfor-- This is really important. Identify the enforcement mechanisms for each loop. Claude is great at building things like tests, and then it fails to build the rest of it, which is the enforcement that actually makes sure it runs.

I've been through this so many times, I'm so pulling my hair out. And then issues for implementation, you-- there-- it's not one loop, it's multiple loops. Do one at a time, go for your optimization, and more importantly, this is the one that's really scared me, and going back to my point about integration. In the last two days or three days, I have written more QA tickets around the things I found that are wrong as I've been building the loops than I've actually written tickets for the code.

Just like when you write code and you put it into QA, you get the expansion of tickets, and then it shrinks before release. It's exactly the same thing here. Now, I don't know how I'm gonna handle all those tickets. It's one of those things I have to work through, but it will [00:24:00] happen to you, and you need to come prepared for that.

I am done. I'm sorry that I had the interruption. I hope I didn't run too long, and I'll turn it over back to, uh, Sparsh.

Hey, I think we're, uh, we're gonna transition to Val. I think you also have a demo you wanna show? Yep. Cool. Let me share my screen. Welcome, David. Hey, yeah. It's our- Glad you made it So while, um, Arthur had, like, his real-life experience of, uh, building these loops, I'm just gonna share some toy experiments I've been doing, uh, in order to kind of understand this topic a bit better.

And, uh, um, I'll just walk through it, and hopefully you will get some insight into, into the topic if you, if you haven't [00:25:00] been working on it every day since January, like some of us. Um, so what is it about? Uh, loop engineering can be se- viewed as automating the human in the loop. And the development of coding agents in the last couple of years has, like, been a lot due to this.

So we originally, or like couple years ago, would maybe ask, uh, ChatGPT for coding assistance, and then we'd copy-paste the code and integrate it ourselves. And we got systems like Cursor, but... And we do individual, uh, individual coding tasks, but you're always really managing it. And then more recently, people started using Claude Code and, and other similar systems where a lot of this has been automated.

Um, so I decided to try to, like, do this myself [00:26:00] using OpenCodes, which is an open source Claude... S-something similar to Claude Code, um, where I added some loops around a code- coding agent and an outer loop around the whole thing, which I'll sh-show you here. The goal is to take away the tasks that the human in the loop usually handles and giving it to the system.

What I did was I tested on, uh, Deep Software Engineering Benchmark. It's what they use to evaluate coding agents, like do-- com- usually comparing models. But here I keep the same model, but just tried a couple of different kind of orchestration s-systems around it or like, uh, I thought of it as loops where you have like review and, uh...

So it's basically things that I think Claude Code does behind the scenes, [00:27:00] but I was just tinkering with it. And DeepSWE-Task is, uh, a benchmark of real GitHub tasks where you have a repo with, uh, an issue that needs to be fixed, and we have that as, like, labeled data. We-- 'cause we-- uh, the problem has been fixed by developers, and it's made into a benchmark where, where you can evaluate the agent and how good it is at patching the issue Um, as part of all this, I start to think of coding agents in some abstract way.

I like to think of it in terms of just a controller. There's like a system. When you have a controller that wants to steer the system towards a specific goal, and the controller, which is the coding agent, can make observations of the system, it gets feedback from testing, and makes actions on the system to try to steer it in the right direction.

And, uh, [00:28:00] um, yeah. Somehow simplified things for me viewing it this way instead of just thinking of coding agents and all their jargon

And what about the loops? So when you want to work on an issue in your code base, you first need to discover it. You know, it's discovered somehow. And that's something you can automate by having a re-review agent, uh, review the repo and find the problem. And once you have the issue, you need to, like, observe the, the system, figure out what's wrong with it, make a plan, and then there's the implementation loop, where you implement, verify, review, and repair again and again, and finally confirm it.

And what I would do in the past is, you know, make the plan probably with, uh, a-assistance with the coding agent's as- uh, agent's assistance, [00:29:00] and then leave the implementation step to the coding agent and then verify myself. Um, but in this experiment, I'm trying to automate that. So, like, to enforce a planning step in the beginning, which I call a prepra-preparation, uh, row here, where there's-- we use open code to make the plan and make it testable in a programmatic way.

And then we can also use open cod-code to, uh, run this loop of repairing, verifying, and reviewing using an independent reviewer. Um- Yeah, OpenCode was cool. I hadn't used it before. You can set permissions for the agents. We have a special research agent, implementation agent, and a re-review agent. They have, uh, these Git permissions where you can make sure that the, uh, [00:30:00] agents do only what they are supposed to do.

Maybe they're only reading and writing documentation and not editing. And, uh, put all of this into the control flow around the coding agents, where first there's a preper-preparation exploration phase, where a permission locked agent reads the repository and w-writes a machine-readable plan before any code is changed.

Where, you know, the, the controller needs a good map of the system before it tries to steer it the right way. And, uh, this improved the results a bit on the deep software engineering, uh, task that I tried. This is not like a proper scientific ex-experiments with reliable s-results, more just tinkering and understanding a bit how you can add control flows and loops around coding agents.

So don't have like proper scientific results here, [00:31:00] but that helped in my tests. And then there's the verification loop, where instead of me asking the agent to verify the results being in the loop, like he codes something, I had programmatic tests generated by the agent. And, uh, then I had an independent review run by a separate agent to tell if the step which is currently being implemented, you know, the feature involves maybe five different steps to finish, but it would check every time, uh, if, if the step is ready and, uh, repair what needs to be repaired.

Um, so it was kind of cool to have this just control flow or like coding on top of agents. Instead of function calling, you have these agent tasks like first you make a plan, uh, based on the task specifications. [00:32:00] Then for every step of the plan, you implement, verify, observe, and retry. You can have set limitations on how often you, you retry to not like spin it on forever.

Um, resulting in a Updated repo that's probably fixed, hopefully fixed, depending on the complexity of the task. Um, but to make it fully, like what we usually call loop engineering now, I added this outer loop on top of it, because now I had this system that could do some coding, uh, using some like loops and control flow and verifications that I didn't need to touch.

Uh, but the, the DeepSWE benchmark, which I made it from, is like, um, made to fix issues in GitHub. But issues, they pop up all the time. And so [00:33:00] I played around with putting like a, a first like a web hook that listens to, listens to a GitHub's issues. If a issue comes in, comes in, it would trigger the coding agent loop to try to fix it.

Uh, and then I made like an audit agent that discovers issues, which you could run like, uh, every day or something, to discover issues with repo, uh, address the issues, and note down the learnings. And that's an important part of building these loops, is to persist some, uh, learnings every time. So it is somewhat continuous and it, it improves.

Um, and I guess this is what people are doing. This was in the pa- uh, reading material we looked at for the, for the, uh, loop engineering from Anthropic, where, uh, you have this audit agent, agent that discovers issues, can be run on a [00:34:00] scheduler, takes the issue and possible learnings from the past, puts it through the loopy workflow to try to fix it, and, uh, would finally push it.

And that's about it. I was trying to have results comparing th- this to like this vanilla way of just asking DeepSeek to fix the problem. But, uh, unfortunately don't have, uh, scientific results. Um, anyway, this is me. Hopefully shed some light on the problem or the concept where you have loops around, loopy workflows around coding agents.

With verification, we need to have the independent verification and review and, um-

Have a way to have a machine-readable plan with machine-readable tests and goals that you can verify. [00:35:00] Um, that's all for now. Valdemar, are you gonna share your slides? 'Cause these slides are really important. I think they're 10X better than mine. Can you put them in the Slack or, you know, whatever you wanna do?

Yeah. 'Cause I think people would like to have these. Yes. And that's another thing I've gotten used to doing is having Claude generate HTML presentations. Makes them, like Pretty neat. Uh, I think I can share this link Yeah, you don't, you don't have to do it now. I'm just asking somewhere in here while we're having the next speaker, if we could do it, that would be great.

Yeah, I'll do it. And, um Do we now go into just a dialogue? We have 50 minutes left, or do we have more stuff planned? I'm just saying. Who... David's back. Who's next? Hey, guys. I think that's about it, though. Um, Sparsh, I don't know if there's anything that you wanted to talk about specifically about some of your [00:36:00] experiences in the work that you guys are doing within your work or we can kind of just jump into questions.

Uh, yes, David. Maybe I will try to give a short briefing slide, three to five minutes. Yeah. Yeah. Go for it. Okay. All right. So for us, we have started to build some sort of, uh, early loops in the work that we are doing. And what we realized, the hardness that we have and the technical infrastructure that we have built was not enough for building these loop-based infrastructure.

And for getting started with this loop-based infrastructure, we have to build a few layers of our own. Once everything was done, we built this observability layer, which gives us bird's-eye view, very zoomed out to do very zoomed in view of what exactly is happening in our system. Once we have that observability, our systems or the agents that we are running, they were able to understand what is happening at what place.

The second layer that we built on top of those was the [00:37:00] eval layer. Now, eval layer is very important, and I'll come to the fact why exactly is that. And then finally, once we have observability, we have evaluation, we build this looping infrastructure on top of it. And the way we started with it is building these inner loops, and at some point we'll go towards the outer loops.

For us, inner loops are somewhat, uh, LLM based and outer loops are somewhat deterministic or design based. And now coming onto the different aspects of these layers. The one thing that we realized, even if we have built these loops, if e- even if we span more and more agents, the output, it doesn't get better with time with the more doesn't mean better.

And the reason is everything boils down to the eval layer that we have. If we have everything in place, the systems need to understand what to measure, how to evaluate it, and how to determine our system is improving over time. And once we understand those things, then we have to communicate, [00:38:00] not in a human way, but in an LLM friendly way or agent friendly way so that the system understands what- Sparsh, do you have slides that you're showing?

Uh, no, I think, Arthur, I think in the- Okay ... uh, in the sense of time, maybe I won't be sharing the slides. But to the concept that I was telling about is observe-- building these layers, observability, evaluation, and looping on top of those gives us a better control over the loop, loops that we built, and it helps us making a system more automated with time

And this is how we have started to build the inner loops. And a lot of people, when they talk about loops, some things people already do that having these skills, having these sub-agents which help them automate some inner, inner part of the workflows that they, they have

Uh, and while building this, we have a lot of, uh, pitfalls that [00:39:00] we ran into l- again and again. I think some of those were mentioned by Arthur. And if anyone wants to discuss, I'll end it, stay back at the end of the session and share those. Maybe David, we can go on to the next part of the session. Awesome.

Yeah. So I think that closes it out. The goal was to like, kind of like, you know, set like a shared mental model for everybody to understand, like, as you're starting to hear about loop engineering and whatnot, like what does that mean? Where is that coming from? Even though there's a lot of things that you probably have been doing in your day-to-day workflows that kind of map back to these, but now we're in more of a let's formalize it stage.

And then, um, thank you Val, thank you Arthur, thank you Sparsh for kind of like just showing some demonstrations of, you know, what are the things that you're working with, whether it's projects or in, um, your professional life, uh, that kind of demonstrate some of these things and the challenges that you're coming up with.

And the last part of this, what we wanna do is kind of pivot this into the open discussion part. So if there's any questions from the group of, or just things that like have been your experiences to give everybody like an opportunity to [00:40:00] chat, which is the whole goal of these conversations. So- David, how do you wanna do this?

We have questions, we have questions in the Word doc. Cool. And do you wanna just open, do an open discussion first, or do you wanna go to the- I, we could do, we could do part of the questions and make those as part of the open discussion, right? So I guess we can kick those off. Um, and by the way, like this is meant to be like, not like, hey, hosts have answers and then, uh, guests have, uh, questions.

It's more of, you know, if you hear something that is a question and you think there's an interesting answer that you have, just raise your hand and you can kind of contribute too. Uh, this is meant to be more of a round table. Um, so- I, I think the first question is really important. Yeah, go for it. Go ahead.

Jim, are you on the line? The guy, Jim, who asked it, are you on and could you please ask the question of the T in the group? Let's have a conversation on this one. This is one that I'm sure is on everybody's mind

Jim, you there? I don't see anyone named Jim here anymore. Okay. All right, then I'll ask the [00:41:00] question. His question was really critical, which is, doesn't loop engineering have huge token burning costs relative to the way we do it today? I'm asking the question. I have my own answer. I want other people on this call, I wanna learn from you.

Tell me how you're handling this issue. You can raise your hands or you just say you wanna talk P-personally, for my exper-experiment in open code, I was using a free version of DeepSeek Flush version V4. So it's both-- it's a cheaper model than using, um, uh, you know, Claude Opus or Fable. And, and, and in my particular case, it was being given away for free.

It would have cost probably a bunch of money to run coding agents, running tasks. It was like millions and millions of tokens. But like of-- one thing is to ch- use the cheaper, uh, open source models, but obviously that's not the whole answer. Just my two [00:42:00] cents. Yeah. I think, uh, this is an interesting convers-- like, uh, point that came up in a lot of our conversations internally too, is, um, you know, as you're, you're getting to like building out those harnesses on your loop and figuring out like what are your breaking conditions and starting to kind of configure that as part of like each agent run, right?

Like number of loops, et cetera, or tick sets in a go through. Um, for example, like on tool calls, like, you know, are people using like length use or anything that they're emitting, um, you know, log runs to, to also query against to get a better sense of like, you know, how much tokens are you burning or like what's the cost of this particular workflow and run, and thinking about those as like SLAs or SLIs.

Um, so kinda curious if anybody on the call has been kind of, uh, experimenting with any of this and or avoiding jumping into it because, um, you're not trying to like, you know, run through your GCP budget at the end of the day. Yeah. So my, uh, thing is what like Arthur said, that's what I started. So basically my thing is the [00:43:00] task has to be like a binary tree task, sir.

That's my thing. It shouldn't interfere with, uh, whatever is happening. So let's say we got a front end, you got a back end, and you got a database layer. These three are different layers. These are not like same layer. So database architecture is the most critical part and what business it is solving So database should also tell what businesses is solving and the interface of it, like what the properties are, what they do on it.

Then most important is I don't think in coding, even if you have a very powerful system, max two agents in front end, max two agents at back end can run, not more than that because it's a generative technology, the probability increases a lot. So it is always evaluating what changes it has done. So let's say this file changed five lines, then it will get confused what happened.

And when it gets confused, if the loop is running, it will create chaos. Right. Doesn't matter it's Fable, GLM [00:44:00] 5.2 or anything. Model difference doesn't matter that much then. So you have to be very, very careful with it. Second, one more thing I have found is in scaffolding, create a scaffolding MD and tell it what each component it is there.

So it will not create, let's say extra new component for, uh, let's say anything component or anything it is because it always-- it doesn't follow the MD very well. It doesn't matter it's Fable Max or... Fable in my view is not that good of a model actually because it, it hallucinates a lot. It thinks everything you are asking is illegal.

Yeah, that's true. It does that a lot. That is true. It does that a lot. So I, I was like trying to create a face, uh, verification app for it-- on it. So it gave me a lot. So gap is doesn't matter if it's a six-month model also, it will work very well if you have given it on. So like other thing is what like I've created like the thing was on task.

So what I found is [00:45:00] there is a limit. Let's say it's-- let's say I've created a real estate agent on it. So prompt engineering matters a lot more there. So because it has to know what task it is doing. So, uh, like, uh, so it can tone. So then things can be, let's say practical acronyms. So let's say I'm searching for New York City.

Anybody can type NYC. So you will have to give it on some properties. What does that mean? NYC means New York City Some things like it, you have to give it, and on those properties, you have to fix it. And the prompt that there is, it should have, okay, you should ask this and that, this and that. And it should not be more than eight to 10 things.

So let's say when somebody is asking for something, there is a limit to, okay, it will ask the price, it will do. On that basis, it will do. You can run parallelly many things like this. So this task will be fully automated. Let's say thousand people ask it, a million people ask it. When their [00:46:00] complexity starts happening, let's say thinking about a law or anything, then human intervention is required.

Loop, there is a limit because once it gets a little off, and it will get off a little very easily- Mm-hmm ... it will create chaos with you. And if you do not have log traces, you will find the report very convincing, that it is very good of a report. There is nothing wrong with it. Everything is fine. And when you go to the market, you will have a problem.

So that, from what I have found, uh, it really happens with it, in my experience. Yeah. I-- Those are all really good thoughts, Tanmay. They reflect some of my experience. But I'm gonna let other people talk because I do have one thing to add to this, a very practical suggestion. Anybody else have thoughts beyond Tan-Tanmay?

Uh, I want to add based on what Tanmay and Arthur you mentioned. So for us, we started to run in a lot of these things, and what we did is we created some sort of a repute agent. Now, even before our agent or any of the loop [00:47:00] starts working, it tries to verify all the assumptions that are there. So ideally, before even starting, it might burn two or three X of the total tokens that we are going to use.

But as soon as the system starts writing code, starts doing things, it, at least in the starting fifty percent of the cases, it doesn't go wrong. The stronger reputal we have, the more tokens it takes at the start, the, but easier it makes our whole pipeline. That is the planning mode. That is the most important part of it.

Uh, I wouldn't even say the planning mode, but we have built an ante to the planning that even in the planning thing, how exactly you have your own independent verifier. Because as humans, you look at things from a different perspective. Let's say I'm an architect, I want to keep my technical debt as low as possible.

Right. And those are the properties or dimensions that I've defined, we have defined in those agents. Exactly. And that [00:48:00] keeps the cost of, like, running into loops again and again lower. Yeah, yeah. Uh, that means, and one more thing is templates are also. We can have template of a back end, like back end some kind of template or a template for a front end.

That also works very well. Uh, some kind of components you have already built, that can also help in future for it to do Uh, what does Fable do is basically it tries to view the design, what is running in co-work mode, and then basically it does the design on it. So if you have already in place in kind of some of MD format, it will follow that a lot better.

So you can have certain templates where let's say your company has some kind of a, like a framework for some kind of a back-end and front-end. That actually reduces hallucination, let's say if you're starting a new project or something else also. That's from my experience I have found actually. It helps, uh, uh, doesn't have to make a lot of assumptions then.[00:49:00]

All right. Let me add one thing, then we'll go to the next question. So the, my dialectic that I mentioned, this adversarial review, um, it's sort of a loop, right? Because it's automated and it runs on its own. I put in a me- a piece of code that actually shows me after it runs how many tokens have been used.

So if I were at an enterprise doing this, I would have the code automatically generate that to a log file or to some report that I could look at for any group using the loop, and then I would optimize it. So what I'm gonna do for my loop engineering is I'm gonna put that at every stage of that measurement, at every stage, and look at it.

And as, as I go through it and start to see what's going on, I will try and optimize it. My better optimization is go to open source. But for now, I want the higher end paid versions. So if you do that, you can start to see where you're burning tokens, what kind of questions are burning the most tokens, and then figure out [00:50:00] how to change your prompts to reduce it.

Now you can only go so far, Tanmay, and frankly, if you ask me if it's worth the number of tokens, even if you burn a few extra, my answer is, given what I've accomplished, um, I would argue that it's well worth the cost for the few extra tokens. But I understand where you're coming from

Okay, next question. Sorry, I forgot. Um, let's go to the third question. How is any of this different from goal/goal command? I have no clue because I haven't used the goal command. So does anyone, first of all- I think- ... know what that is? I think you probably do. I think it was dur-during-- Sorry, go on. I think it was during my talk.

Uh, there's a goal command in Claude code, and, um, I don't know how it works, but it's probably similar to what I was doing, where you have... But there's different types of goals. You can have like a goal evaluated by a LLM, which can be some kind of abstract goal. But [00:51:00] then you can have like programmatic formal verification of the thing, and that's the more, maybe more important one to be able to generate a formal way to verify that the goal is there.

I don't know if they do- Yeah ... that in Claude code. I didn't, like, manage to do that really well in my tinkering with my open code stuff. But that seems like- Yeah, my understanding of the goal- ... most important, important thing maybe. Yeah. Yeah, my understanding is the same. If, uh, if you're using Claude and you've used the goal command, um, it, it kind of goes through those same steps, um, that are critical.

You'll have a sub-agent that's the verifier that they use a smaller model on, and that'll essentially kind of compare or like look at the output to ultimately optimize towards the goal that you're defining for it. Uh, they also have like a loop command, right? And, yeah, I think CodeX has like its own version of these things.

Now you have like automations and whatnot that you can define where you can put these things on schedules and whatnot. But, um, yeah, those are those attempts to kind of like, uh, featurize them within, uh, the existing, um, whatever your [00:52:00] coding agent or preference tool is at the end of the day. Um. Will it work to do this, David, or anyone else?

Please, people, if you've used this, we wanna hear the experience. I'm, I'm curious if it does the level of sophistication that I need it to do. Yeah. Um, I, I-- That's where it's like I feel like if you're your average user, you're just kind of going through this and like using, say, Claude Code, for example, of, you know, you're working on a work task.

It makes kind of sense just to throw it in there and help it kind of like you could step away, hopefully come back and review it. You're still that kind of like piece in the middle, so I don't think it really gets to like the loop engineering piece where ideally you're kind of like having a route for loop that runs and you're keeping state from the previous thing to kind of hopefully continue to just like run this on some, some cycle.

But I think for a subset of tasks that you're probably working with every day, um, it's a perfectly good tool to, to try to incorporate. If you're building a system and you wanna incorporate these into a system, I think that gets a little bit towards for the, the architecture and everything that we're talking about.

Does anyone else on the call, not the speakers, have any experience [00:53:00] with slash goal and its use for loop engineering? Because, uh, really this is a great question. If you can build, use a built-in tool, why not?

Leo, you got anything? I see you're on the line

By the way, I've, I've asked Leo to come in and review my harness and see how I can get faster on top of doing all this other work. So I have very strong interest in hearing what Leo has to say.

Um, not too much for it. Uh, Goal has been used for only in like Mod Go. Um, but I don't think Cursor has anything for it. So if you'd bring your own tool, then that's actually pretty useful for it on how to keep through. And I think just like the planning and I don't know, I think everything is a loop. Like just like how you, I don't know, like Crew AI had everything.

Uh, LangGraph has been out for a while and that, you know, had agentics in the loop and [00:54:00] it's just more of as a lot of models become better at tool calling that the variety of models that you pass things off to has grown. So, um, nothing too specific, but Goal has been useful on and off. They have increased basically the thinking time itself.

So they've got two, three modes for it. There is a another mode that was released, I'm forgetting the name of it. So what it does, it runs longer actually, so it does better. So they have found that basically if you increase the thinking time, but evals has to be in place, otherwise it, uh, will not run very well.

It can be bad also for you if the eval is not there good. So, and, uh, what happens with mo-most thing is anthropic models have a thinking command in it. Yes. So when you give it a prompt, you have thinking this kind of stuff. Claude is very well built for that, so that's why it runs very well. Uh, but, uh, if, if you can do it for other models also, [00:55:00] it'll run.

OpenAI likes Agent70, others also like Agent70, open source models also. They like have followed that also. Funny thing is every model te- uh, tells, uh, gives you, uh, reference to OpenAI and, uh, Claude always, even open source when they're giving you reference for anything also. So that's what they have done with goals actually and, uh, Claude or Co-Codex also has done.

Uh, that's it. Not much more difference on it. Okay. We are at time. We're five minutes over. Um, I'm happy to let it go on, but I think we should probably be good about time and let people do their thing. Um, I don't know about you guys, even as one of the presenters, I learned a tremendous amount this morning from this session.

So I want to thank our speakers. I wanna thank all of you who participated. Tanmay, Leo, great to have you chat. I wish more people would jump in in the future, and we'll let you go from here. So thank you everybody, and talk to you in [00:56:00] the next reading session. Thank you, sir. Thank you.