Link to Presentation

Demetrios [00:00:05]: Nathan, you're up next. And Sam, are you going to be able to stick around and chat with us after this?

Demetrios [00:00:15]: So we can open it up and have more conversations. Nathan, did you have something you wanted to share your screen?

Natan Vidra [00:00:25]: I'll share my screen. It's nice to meet you guys today. My name is Natan. I actually like the founder of this AI company called Anote. I won't really talk about this today and attempt to just kind of share what I'm like currently working on which I'm trying to open source for everyone to use. And I've recently gotten really interested in agents. So I'll kind of talk about some things that I'm interested in and what we're trying to build this winter as like an open source, fun AI research project and and also stick around for questions at the end. But I guess the general idea is there's this kind of rise in AI agents and multi agent AI system.

Natan Vidra [00:01:08]: I know everyone uses the words agents. It will kind of define what I mean by agent. But essentially I wanted to kind of share a few things. Today we'll talk about the architecture of how you can build these multi agent systems. We'll talk about how you can kind of coordinate these teams of AI agents that will work together. We'll talk about how you can kind of monitor and optimize these agents to be intelligent. We'll talk about a few use cases of where agents just can be applied and super helpful. And we'll talk about the end this kind of idea of a generalist agent framework and why it's a really exciting concept.

Natan Vidra [00:01:48]: So I think today just to start we should kind of get on the same page of like terminology. So just kind of taken from some sites like CREWAI and Phi Data, we'll call an agent like an autonomous unit that can basically perform tasks, make decisions similar like how a person is there. There's like an agent and these agents can kind of do tasks. So you can kind of give these agents tools and a description of what they should do in a system prompt. Then it can kind of do these tasks. And what's cool about agents is rather than just having like a single agent, you're going to have these teams of agents or crews. So similar to like how in the real world you'll have like product managers and software engineers and writers and researchers and salespeople and they're all kind of combined together, a crew will basically be these kind of teams of agents that do these tasks. And the tasks that you can kind of give them could either be things that are done in parallel or in series, where you have one thing going to the next thing to the next thing.

Natan Vidra [00:02:48]: Or it could be hierarchical where you have if then then do this. Right? And this kind of entire end to end process of the task is sometimes called like a workflow. And the building blocks of these kind of agents is essentially like an LLM like GPT or Claude or Llama or Mistral tools, which are things like searching the web or being able to use a computer or you know, actually being able to implement code. So you can kind of have these teams of agents that use LLMs and tools to execute really complex tasks that individual agents can't. I kind of first got into the space when I was building my company Anode AI. We built a chatbot that looks something similar to this, where you'd kind of upload these files, you'd ask questions and you'd get answers. And we realized that if you uploaded tons of files and you asked questions that were super complex or domain specific, it was really difficult to get high quality answers to questions. So we were looking into a lot of different ideas like fine tuning or enhanced rag or labeling or different evals.

Natan Vidra [00:04:01]: And then we kind of stumbled along the idea of agentic rag and that kind of went to like a whole flow of agents. I'd say what I'm trying to build as like an open source product this winter just for anyone to kind of use and try this way to like build autonomous agents, have these teams of agents like work together and like have the ability to kind of monitor and optimize these agents. You can kind of have them perform really, really well. I think it's worth kind of talking about like how this could work. So like one way is maybe you'd have a specific agent that would be good at these specific use cases like email outreach or financial analysis, or an automatic coding agent. And then another frame of thought is like, you can have agents that would be like this generalist framework. So my opinion personally is that the latter approach, the generalist framework will kind of be similar to like how you have generalist models like GPT and Claude and Llama and then there'll be like domain specific agents similar to how you have fine tuned models for tasks. But I'll kind of first talk about a few of these use cases and then I'll talk about why I'm really interested in the generalist framework and how I think it could work.

Natan Vidra [00:05:18]: And I'll share a link to what we're building and where it's going to be and or why I think it's cool. So I'd say like one use case is like an obvious use case, like email outreach. So a while ago I was actually building this random product called Upreach and essentially what this was, it just like a hacked together product where it was like you'd kind of want to search for people, then email them, like automatic spammer, right? So you basically find people, you'd generate emails and you'd send it and you'd ideally do it at scale. And the idea was that this could help save time to reach out to people if you wanted to like host events or host drive awareness or kind of causes. Right. So in the ideal world, rather than kind of have this product where you'd actually have to go on a UI and search and find, add people, list and generate emails, it'd be cool if you could just basically tell like a chatbot, like ChatGPT, hey, let's reach out to a list of 10,000 people that are heads of AI in New York that work at these mid sized finance companies and invite them to our AI event. And you'd have this kind of agents that would basically be almost like a series of APIs that might find the right people, look and enrich the data from the web, generate emails and send them out. Right? And maybe you'd kind of repeat doing that, or maybe there's kind of like a use case where it's like, hey, you want to apply to all these grants to get funding, but let's say you wanted to apply to these RFPs, but you, there's maybe like a hundred RFPs and it just takes a ton of time to find them, to write these applications and submit them.

Natan Vidra [00:07:02]: But if you could actually use AI to do that, that would be amazing. That way you can get a lot of really awesome opportunities. So the idea is like when you would apply to these grants, maybe you'd want to do research, you might want to write these grants, you might want to ensure that it fits the format, you might want to revise the grant, and then you might want to upload a testim portal and submit it. And for each grant you want it to be like super tailored and unique. So you can imagine in a general software engineering framework you might have a series of APIs for each of these things. But with agents, ideally there could be like a way that you can kind of use tools like like writing, researching, using the web to kind of automate this process. And we built something related to this too. Obviously there's kind of use cases like finance, right, where this is an example using the Phi Data framework, where you might want to kind of have a web agent that searches the web, a finance agent that can kind of grab information about stocks.

Natan Vidra [00:07:58]: Then maybe you'd want to kind of call these in the team to summarize information on a specific stock you're interested in. And then there's obviously like the event planning where it's like, hey, you might have these different agents that might. One might help you find a venue, one might help you do logistics, one might help you market and communicate event. And you can kind of chain these together and give each agent these different tasks to basically, potentially in the future, not today, but maybe in the future, host these awesome AI events. I'd say these are kind of specific use cases that you might think of in a framework. And each of these kind of use cases could use these open source building blocks like DSPY or things like Pedantic or LangChain or LOM Index, right? Where you can kind of use code and have each agent be some sort of module and you can kind of give each agent tools and techniques and try to optimize them. And obviously each framework has its pros and cons. And I think what's cool about them is you can have these kind of compound systems and modules with inputs and outputs, and each module can have these, these tools and tasks.

Natan Vidra [00:09:05]: There's been a shift recently within the research community to this general idea of generalist agents. So what this basically means in theory is that rather than basically being like, hey, I'm just going to do task X and Y and Z for outreach, what I'm going to do instead is I'm going to have these kind of templates of agents that have access to tools. And then when I'm going to basically put in a query like, hey, I want to reach out to like all the people in AI and send them an email about this cool event I'm hosting. You'll have this thing that's called this Orchestrator. And this orchestrator essentially is ideally going to be really smart and it's going to be able to spawn and create these different agents in the order that they should be in, either in series or parallel and whatever kind of, and give them the tools they need. And then this orchestrator will assign these agents the order, give them the tools and tasks they need, these agents who are going to go out and actually try to do these tasks. And then ideally, if they can all return the right values. You'll be able to just have this framework that will be like super generalizable in theory.

Natan Vidra [00:10:18]: It's great. You know, if it all worked, you just kind of ask a chatbot to do something similar to like ChatGPT and just basically be able to use all the tools and templates and figure out how to do everything. In practice it's going to be like really hard. I'd say there's like a lot of these open ended questions of like how do you optimize Orchestrator and make it really smart? How do you actually evaluate these agent systems to ensure reliability? Can you fine tune like each module and what data do you need? What if you want to run these agents privately? How do you monitor these agents via logs. So we're actually basically trying to do this project over the winter where we're going to basically open source our code and just let anyone who wants to learn about agents and how they work work together on this centralized place. We've already have just a typical standalone thing where you can have this chatbot, you can ask questions, get answers and we're providing that to just everyone for free, no strings attached. So you can just kind of clone the repo. Try it.

Natan Vidra [00:11:19]: We have like instructions to actually like set up the code if you want. I included also like this deck which kind of just has the information I talked about today in case you're interested. And I'd say like some of the kind of themes right is you'll have this general purpose agent framework that we want to build this like registry where you're going to have these domain specific agents, the ability to kind of do smart orchestration and use these tools some way where you can like log and monitor and understand how each of these agents work. And then ideally as a developer you might want to interact with it via code. So something that would be able. This will be like a really interesting project and hopefully we'll learn a lot and ideally in like middle of February we'll have some cool presentations on things. So yeah, there's a lot of use cases where this could be applied. I kind of share this all here from videos to resources on things we've built at my company Anote that we're just kind of going to provide to the community as well for free.

Natan Vidra [00:12:28]: And I think really our goal is to try to learn as much as we can about how these agents work, how you can build these teams, learning about all these awesome frameworks that are out there. And we're excited for like the opportunity to hopefully like learn together. And happy to answer any questions too.

Demetrios [00:12:46]: Awesome, dude. The last slide. So while if anybody has any questions, throw them in the chat. The last slide or second to last slide that you had in there about what is going on in research. I'm not sure I fully understood that with like the. It's a, it's the common design pattern. What was it?

Natan Vidra [00:13:15]: Yeah. So essentially what's going on is rather than having to build an agent that would be like your logistics manager or venue coordinator or you're like marketing and outreach person, you're going to have these things that are templates and when you will kind of input a query to a chatbot, similar to chatgpt or the chatbot I showed this orchestrator is going to basically be able to take in this query, figure out which agents or templates that you kind of call, what order they should actually like do the operations, and then from the order you're going to have and the agent you're going to call in this order, each agent will kind of do the task. You'll ideally be able to kind of spawn via YAML or markdown or be able to edit, you know, these kind of agents to ensure it works and monitor and log them and ideally optimize the system. But it's going to basically be able to figure out which tools to use and actually like do the thing like autonomously. And that's I think like the cool idea that I think it's very open ended on how to do it. But like there's a lot of things like evaluation and fine tuning and which frameworks to use and we're trying to like learn about it.

Demetrios [00:14:31]: Yeah, awesome.