Samuel Partee [00:00:21]: Every time your intro gets better.

Demetrios Brinkmann [00:00:22]: Oh, just wait till the intro that I got for you, man.

Samuel Partee [00:00:27]: Wait, wait, wait.

Demetrios Brinkmann [00:00:28]: Let's do a little throwback, because last time that you came on, I introduced you with an amazing video. I don't know if you remember it. Do you? Let's just read jot.

Samuel Partee [00:00:44]: All right.

Demetrios Brinkmann [00:00:45]: Let's rejot everyone's memory a few times. Let's play it. Let's play it for everyone. Excuse me.

Samuel Partee [00:00:52]: I knew it. No. How did they not know?

Demetrios Brinkmann [00:01:02]: How did they not know? So a lot's happened since a year and a half ago when you spoke at this conference last, and I will say that you've gone off and you started a company and you got kind of famous. So the intro's been updated. Let me give you the new Sam party intro.

Samuel Partee [00:01:23]: Love it.

Demetrios Brinkmann [00:01:26]: Excuse me. Do any of you know who Sam Pardy is?

Samuel Partee [00:01:33]: You do?

Demetrios Brinkmann [00:01:34]: All right, good. They know you. You're world famous.

Samuel Partee [00:01:43]: I heard him. I heard him.

Demetrios Brinkmann [00:01:45]: All right, so don't let us down. Bring us home. Talk to us about what you've learned in the last year and a half building arcade.

Samuel Partee [00:01:53]: AI wouldn't dare. I think I've never had a harder time picking which slides I want to put into a presentation. So we're going to try and get through it. But as usual, I won't get through it. All right, we ready?

Demetrios Brinkmann [00:02:11]: Oh, yeah.

Samuel Partee [00:02:12]: All right, let's do this. All right. Share and then present.

Demetrios Brinkmann [00:02:22]: Sam, have you done this before?

Samuel Partee [00:02:27]: No, actually, this is my first time.

Demetrios Brinkmann [00:02:28]: This is your first virtual call?

Samuel Partee [00:02:30]: Yeah. You didn't know that?

Demetrios Brinkmann [00:02:31]: No.

Samuel Partee [00:02:32]: I can't believe you invited me.

Demetrios Brinkmann [00:02:33]: Oh, no. What did I get myself into? All right, well, it's live, so nothing could go wrong.

Samuel Partee [00:02:41]: Great. You can see my awesome Apple background.

Demetrios Brinkmann [00:02:43]: Yeah, I see it all.

Samuel Partee [00:02:44]: Did we get it?

Demetrios Brinkmann [00:02:45]: Yeah.

Samuel Partee [00:02:46]: Go for it. Ah, well, look, I know I only have 25 minutes. Thank you, Demetrios, for yet again, an intro that never ceases to amaze me. As I said, it has been really hard to pick what I want to put into this presentation because never before have I worked so hard on something and cared this much. Because now you know, this company and what I'm putting this work into is it's all. It's, you know, it's mine in some ways, and it is a fascinating area. There's so much to learn. There's also so much that we're going to find out in the next year or so.

Samuel Partee [00:03:30]: So without further ado, this is just going to be a quick intro to tool calling my opinion of what an agent is, some insights that I've learned, and then we're going to get into what Arcade is because I can't wait to talk about it. It's been forever and man, it sucks not talking about something you're working that hard on. All right, so motivating questions. Can you scale the tools called by agents? Think about that separately from the agent. Can you scale the tools call by agent? I'm not going to give you any answers. These are motivating questions I want you to think about. Why is API key based authentication authorization so pervasive in the LLM space? Why is it pretty much every service? Why isn't there a CICD equivalent for agents? I'm sure there's some body in the audience says that's my product, but in general it's not like the ones that we see every day. Vibe checking is still, unless somebody else wants to disagree with me, pretty much the best way to check if your agent actually even works at all.

Samuel Partee [00:04:41]: And lastly, I think the most important one is why can't chat GPT send an email for me? This is the question that bugged us so much. We built a company around it. So agents and tool calling Potentially spicy take, I believe an agent is not an agent unless it can call tools. Otherwise it's just fancy search and it's creating precision for generality and user experience. Having a really nice retrieval system doesn't make it an agent. It's just fancy search. Tool calling is the future of agents. It is what makes something agentic, in my opinion.

Samuel Partee [00:05:23]: That is also why I'm so fascinated with it. So what is an LLM tool? An LLM tool is essentially a function f of x equals what? I say this because a lot of people overcomplicate this, and I think it's due a lot to the term tool calling. It's annoying because calling has this implication that execution then follows. But right now the execution is completely the onus of the developer. Execution entirely happens either within an orchestration framework or manually once you get predictive parameters back for using the Bear client. So note in the rest of this presentation, when I say tool calling, what I mean is prediction of parameters, not execution. Minimally, tools consist of four things. Now there are other things, but minimally necessary, not sufficient.

Samuel Partee [00:06:24]: I think that's right. A set of m inputs, each with v values, a set of y potential outputs, all of the which are typed by the way, a description of that function and when it should be used usually, and a method of execution. I say A method of execution. Because a lot of tools are different, there are some that don't necessarily need to be executed, but in general this is the minimal constitution of an LLM tool. A tool definition, as in what you send to an LLM, is made up of the first three what are the inputs it needs to guess, what is the type of the output that it needs to return in terms of usually some type of structured output, and then what is the description of that function, as in when should it be used? This tool definition is important because it is what informs the LLM when and where to use that tool. So quick intro to tool calling Client application invokes the LLM with some type of prompt, usually a conversational context, and those tool definitions. The LLM I said LLM provider here, but returns a response with that selected tool. If your parameter, if your tool choice is something like auto, or if you're using OpenAI then required and the predicted parameters.

Samuel Partee [00:07:47]: Note again, this doesn't mean that it is executing. Even when people say parallel tool calling, it doesn't even mean parallel execution. It's just implying that there are many tools in terms of the parameters, the sets of parameters being returned, and that they can be executed concurrently because they have no association or shared state. All of which bugged me, to be honest with you, as someone from HBC calling that parallel tool calling seems annoying, but you can actually run them concurrently. And that is a very important point. Now after this, when you get that response, what usually happens is essentially RAG retrieval, augmented generation, but in this case it's a tool. I wish I could start the term tag as a thing, but I'm not going to try. You run that tool with those predictive parameters usually returned in JSON and then you invoke the LLM again with those results, and then you have a informed response with some type of tool result.

Samuel Partee [00:08:45]: This is usually how these things go in terms of what happens within an agent. When an agent is calling tools now, what is an agent then? So we know what a tool is now and we know how it's used, but what's an agent? This is where some of you are going to start to disagree with me, and that's fine. Text medium. There has to be some type of text medium, whether it's a prompt, whether it's a built in, whether it's a chat medium, whether it's some type of context. There has to be some type of text medium to start. Usually the next thing is orchestration system and a language model. With these three things, you almost have an agent in my opinion. But as I said earlier in one of those title slides, an agent is not an agent until it has tools.

Samuel Partee [00:09:35]: And if you have tools, you need to be able to run those tools. Now usually this is today caught up in the orchestration system. I'll talk about this a little bit later, but that is not where tools should be executed. There are a lot of reasons for this. Orchestration should be orchestrating execution of tools needs to happen where the execution of tools needs to happen based on the diverse requirements of that tool. I'm going to talk about that later. I'm not going to get ahead of myself. Honorable mention.

Samuel Partee [00:10:06]: Here are retrieval, system state persistence, chat storage. All the things that people are probably like. But wait, what about. Yes, I understand, but minimally this is what I define as an agent. Now, given this and tool definitions, I know I'm skipping over this, but only at 20 minutes. What did I learn? Well one agents aren't that good at fixing bugs. It's a bit of an inside joke. But first I will just say I spent about a year trying all different types of agents.

Samuel Partee [00:10:41]: Everything from text to SQL to having agent pulled datadog time series to put on a plot. It was a lot and not everything worked. But throughout all of it I got a few perspectives that I think are worth sharing. So without further ado. First, as I mentioned earlier, kind of tools need a dedicated runtime separate from orchestration. Now when I say orchestration I usually mean LLM orchestration, lang chain, llama index, yada yada, crewai, what have you. The reason for this is because typically those are going to be built in Python, which is GIL locked. Unless you're on 313 and you're using the one that dropped the GIL.

Samuel Partee [00:11:29]: But what ends up happening is that forces tools to be essentially just concurrent HTTP requests and so there's no room for anything that might be computationally intensive. Think about BM25 like a text search that is somewhat computationally. Even scraping a website is somewhat computationally intensive. When you're processing the text you get back that'll lock the GIL and you can go look at LangChain issues and see a couple people like why did it take forever this time it's because the GIL locked and so because of that and also because agents need to be able to scale their tools. Think about why can't agents scrape 10,000 websites all at once? Well, how do you scale the tool calling up without scaling the agent? If the orchestration framework and the tool execution are co located, you'd have to scale both. The client application in this case subsumes the tool execution, in which case the compute resources are intermixed. So there has to be a way to separate the two. I show this here, but these are essentially the problems.

Samuel Partee [00:12:38]: What if you wanted your tool to run on modal with the GPU who was generating images, but you didn't want your agent to run there? And there are ways for this in some of the current orchestration frameworks. But essentially what I'm saying here is that it is beneficial to be able to separate the two in an effort to be able to have more diverse tools that can do more interesting and capable things. It's also easier to maintain. Agents will never reach their full potential until we have better systems and processes for managing the tools they use. Tools are code. They need to be packaged like code, versions like code, tested like code, and right now they are sub packages of a sub package of a repo. For most things, agents need the same thing. They need versions, pinned dependencies, things we figured out in software a long time ago, accurate and relevant monitoring.

Samuel Partee [00:13:40]: We're getting closer there, you know, like Agent Ops and you know, Log Fired and Lang Smith and a few others. But we also need scalable runtimes for those tools. Like I mentioned earlier, as we get to things like multi agent, this problem will only get worse. We need to start treating it less like jupyter,_bookbook,_126 and more like a real software project. I say that and somewhat include myself. If we can treat creating agents more like creating software, agents will get more stable and they will perform better. That is something I learned though, right? Lastly and most importantly, agents should be able to act, but not just for you, not just a bot token, not as some other entity that you have to give the world of the keys to the castle to. They should be able to act as you.

Samuel Partee [00:14:47]: And this is really hard. It's a lot of brute force effort. But this type of authorization is what is limiting right now, the tools ecosystem from being able to do things for users that are actually impactful. Have you noticed how most tools are HTTPs requests to something protected by an API key that's going to return text, something like Google Search, serp, Fire Crawl. Doesn't that seem kind of limiting to anybody else? So this is going to be a little bit of an overview. It's going to be a little bit dense, but I wanted to go through what I think are the minimal hierarchy of needs for an agent right now and add on a Few things that I think are necessary and the reasons why we've been working on Arcade. This kind of represents the top of it. At least a lot of what we learned is actually stopping this ecosystem from thriving more that is.

Samuel Partee [00:15:56]: Everybody knows. Oops, sorry. Everybody knows that. By the way, prompt orchestration there. I'm solely referring to the inputs that actually go into these processes. But everybody knows that you can have a large language model and you can have a prompt. If you put the prompt in the large language model, you get an output easy. Then came vector databases and retrieval and search.

Samuel Partee [00:16:18]: And Joe from Vespa would probably kill me if I said that vector databases weren't invented 10 plus years before this was used for a large language model. So I'll mention that. But essentially retrieval and search just add to the input. If you think about it, RAG is just finding a way to put more information into the input. Agent orchestration comes next. So how can I have specific prompts with specific LLMs in specific scenarios and have them orchestrated such that they can work together on things? You see things like Langgraph and Crewai and what have you. Llama index doing this right now. Tool calling could have come before this, but I'm putting it after for a reason because it's not as used.

Samuel Partee [00:17:09]: If you take everything below tool calling right here, it's much more common to go into production. Even at redis, I saw significantly more retrieval search down in this pyramid internal doc search use cases than I ever saw. Tool calling maybe one or two. Right. There's an interesting step here that is new. Now I call this tool orchestration because I think it'll be familiar to people in terms of what it implies. But what I'm talking about here is the same thing that I keep mentioning, that tools need to be separated and treated as more important than they are right now. They're probably the most neglected part of this ecosystem and the part that needs the most work.

Samuel Partee [00:18:00]: If we can facilitate the usage of an agent with hundreds if not thousands of tools, the capability of that agent grows significantly. The problem with that is if you have 100 tools in the context window of a single request, you're probably going to not even be able to use llama32. We need better systems for determining which tools to include, when to include them, based on what context. That is exactly what the same thing does for agent orchestration. It's just solely for tools. This also includes executing those tools in different environments and making sure that things like failures and retries are handled properly. That's why I Call it orchestration. And lastly, and most importantly, AUTH agents will never be able to do anything that is helpful in our day to day lives.

Samuel Partee [00:19:04]: If they can't do it for us. If they're always doing things that aren't protected by an authorized API, then they will not do enough useful things to ever fulfill their potential. And I realize that's a strong statement, but think about the last time you did something really useful or you know, even day to day at work where you weren't signed in. If agents can take actions on behalf of you, the world of the things that they can do for us opens up drastically. All right, what I've been waiting for, I can't wait to talk about it. What is Arcade AI? We call it a tool calling platform because it's meant to address all of these things that I've been talking about in the talk so far. This is a little terse right here, but there are essentially three things that you need to think about. When I explain the APIs and how to use them.

Samuel Partee [00:20:11]: Obviously for the people who just want to go read the code, I put the QR there. That's usually what I do during talks. So I did it for you. There's a bet. Three things that you need to focus on with Arcade. First is the tool SDK. The Tool SDK in Arcade is meant to be opinionated on purpose, if you notice. Essentially, Arcade new is the CLI command that essentially creates a cookie cutter of a Python package.

Samuel Partee [00:20:41]: It comes with everything you need to be able to essentially write one function, one Python function, and then immediately serve that to a large language model. We've tried to make it as easy as possible, but it is opinionated. It's still standard library Python, but we require that you annotate every parameter. We're not parsing docstrings and the reason for that is because I've seen too many times that kind of thing fall through the cracks. That's another lesson to learn is that all of the various Python docstrings don't really agree with each other in terms of parsers. Even if you're using something like unstructured, the engine actually serves these APIs. It's essentially middleware. It can be used in three different ways that build on each other in terms of their abstraction.

Samuel Partee [00:21:28]: There's an authorization authentication API that was first. Then there is a Tools API that utilizes that. Auth API Tools is essentially Information, Add, Remove, Enable, disable, Execute. Then there's an LLM API that's an open API compatible Wow. Open AI API. Too many as and I's compatible API to call the chat endpoint with tools automatically injected and handled for you. And it adds two new tool choice parameters of execute and generate, which I'll talk about later. That LLM API utilizes the Tools API which utilizes the AUTH API.

Samuel Partee [00:22:14]: So what you get is essentially a incredibly powerful new type of LLM request that seemingly by itself can access the data it needs when it needs to, as you. Lastly, the actor is essentially a worker API. You could in theory write this with anything. We open source the schema, but we provide a Python CLI command. Just like Yarn Dev, there's Arcade Dev. It spins up the actor with whatever Python packages you have installed. Given the Python the Python based tool SDK, all you have to do is write those functions, Poetry install and then Arcade Dev and you're writing custom tools. It is meant to be as easy as possible to get the tools you write tested, evaluated and running with any large language model.

Samuel Partee [00:23:08]: Lastly, to mention the tool SDK does come with a tool calling Specific Evaluation Framework. There are critics, rubrics. I don't have enough time to explain it, but it is built specifically for the purpose of saying I have a CI CD pipeline. I want to know every time a new model is released how it affects my tool calling capability, how it affects my agent capability, how it's going to affect what's in Prod right now. It's not quite yet, but it will eventually integrate directly with pytest. So it'll be like Code Cove essentially. And in that way you'll be able to evaluate every time someone even changes a prompt. So first the AUTH API.

Samuel Partee [00:23:58]: In theory, this is a generic AUTH API. You could use this without ever writing any type of AI program. However, it is specifically geared towards LLMs and LLM based workflows. Like agents. It provides this really simple interface wherein you supply a provider, an AUTH provider and a set number of scopes. Given that you provide a unique ID for the user for your user, like a UUID or something, something you're ideally storing in postgres or knowing this crap, probably Supabase. But it is really easy because we don't care who that user is, we just need a unique ID to associate that token with. So you can handle that however you'd like.

Samuel Partee [00:24:52]: And that token storage you see there can be ephemeral and this entire thing can be stored inside your vpc. So in that way all of the long lived tokens can be held within your environment and you don't have to worry about any of those SOC2 type things. When would you use this? So let's say you're using LangChain's Gmail toolkit, right? Or you already are and you went through the pain of rolling your own null system well instead, now you can have one API call for any user that logs in, obtain that token, we store it whenever you need it again, we'll be able to get it with the same call. That just makes the process of using any tool that might need that authorization a lot easier. Also, pre author, even if you're using Arcade tools, or if you're using any tool that might require those set of scopes with that provider, you can pre authorize any type of action you see a lot of the authorization providers you see above. So as I said, all of the APIs build on each other. So the tools API that provides that distributed runtime, that solves the aforementioned problems that I was telling you about builds on the AUTH API. This provides authenticated tool calling.

Samuel Partee [00:26:08]: It also provides management of tools. So what tools are going to what agent? How can I group these tools together into toolkits, synthetic toolkits, so that you can actually have features that are based on one or more combinations of those tools? Because we all know models, especially smaller models, start to fall off at about 20 or so tools, managing those is a non trivial task. Tool selection is something we're going to be investing in. But for right now you can be able to group and have these tools contained to specific actors like you see below. You can imagine the far right ones. The social media actor, Google Actor, Slack Salesforce, Atlassian Actor. These actors serve specific tools and if you're getting a lot of requests for a specific tool or that tool needs to be scaled, it's as easy as either letting Modal do it for you in two lines of Python code, or scaling it up on something like ecs. And that was the point of being able to separate out something that you can scale very easily in terms of a runtime.

Samuel Partee [00:27:23]: So all of the tools that you're going to be able to be running from an LLM, you can scale really easily. We actually have a Slack bot hosted on Modal right now and it is incredibly easy to host. It's about four lines of code shout out to Modal for a really great API. When would you use this? When you want to add specific capability to your agent? Let's say you want to be able to draft emails for your users upon some event. That tool already exists in Arcade. It's a pre built tool. You just have to add the name and point at Arcade. That's it.

Samuel Partee [00:28:00]: That way all of your users will be able to draft an email with Google or Outlook or Microsoft's. I think it's Outlooked immediately with your agent if they ask. There's tons of those pre built toolkits. If you go to docs.Arcade-AI.com and then go to the integrations pane, you'll see all of them that we have there. All right. Are you kicking me off?

Demetrios Brinkmann [00:28:28]: Keep going.

Samuel Partee [00:28:30]: I'm close. I have the time. I'm doing all right. Actually, we're at 25 minutes so I lying. But lastly, the LLM API. It uses the tools API and the auth API. So this means essentially if you're using the OpenAI client, you point the base URL at Arcade and you immediately get every tool that we have. I'll skip over this a little bit.

Samuel Partee [00:28:59]: You can read the documentation, but what does this actually do? Before Arcade, you have the left. You're responsible for tool execution. You're responsible for handling the inputs and outputs, you're responsible for handling getting the tool results back into the prompt. And now afterwards you have one HD HTBS request if your tool choice is generated.

Demetrios Brinkmann [00:29:19]: Can you hear me?

Samuel Partee [00:29:22]: I have a demo, but I don't think we have time. Do we have done?

Demetrios Brinkmann [00:29:27]: Yeah, go for it.

Samuel Partee [00:29:30]: I'll skip through it a little bit. This is made by one of our users, Sergio. I hope he's okay with me showing but he already shared so I figured it was okay. He has done some great work in integrating to Twilio and Notion already. Given that our OAuth API is generic, you can integrate to anything that has an OAuth API. You see there, he just asked the agent to send Twilio a message and it showed up on his phone right here. So even though ChatGPT can't send an email, we can create a Notion doc, we can send a Twilio message, we can read your Gmail, all in a secure way that any SEC team in the world couldn't fuss about because it's the exact same as they would ask you to do. And that's Arcade.

Demetrios Brinkmann [00:30:18]: Boom. All right, I got a comment on the first question here. And somebody saying that sometimes the LLM is self confident and it's almost as self confident as you yourself wearing that sports coat. How did you think you were going to be able to pull that off? Give me a break.

Samuel Partee [00:30:42]: Not even a sports coat. It's like cloth.

Demetrios Brinkmann [00:30:45]: Hey, man, this is great. In all honesty, there's a lot of questions coming through and I want to get through.

Samuel Partee [00:30:50]: Yeah, let's do it. Let's do it. Come on.

Demetrios Brinkmann [00:30:53]: If off is the limiting factor, what does security look like with agents?

Samuel Partee [00:30:58]: Oof. How much time do you have? Oh my God. Great question. I don't have the time to answer that question. But there's two different connotations of security. I wish I could follow up with a question with them. With them I should say security in this case, if you're pertaining it to the execution of the function is determined by who is running Arcade. If it's our cloud, then it's our responsibility.

Samuel Partee [00:31:26]: If it's the user and they're deploying because Arcade is very easy to deploy in your environment. In that case it would be their responsibility because it's their execution. They're responsible for that runtime environment. I'm not exactly sure that's what you meant by security, but that there's a lot of great bents that I could take that. So honestly, follow up with me. There we go.

Demetrios Brinkmann [00:31:47]: That's a good answer. A good non answer. I would say for my team. Skill is the definition of a code function that can be called by an LLM. Is a tool too much different? I'm not sure if I understand that word skill is.

Samuel Partee [00:32:05]: One more time.

Demetrios Brinkmann [00:32:07]: Skill is the definition of a code function that can be called by an LLM. That might be. Hold on, I might be in the wrong part of the chat. Hold on, bear with me. Let's just pretend like that never happened.

Samuel Partee [00:32:23]: Honestly, I think skills would be a good word. I wish I thought of that earlier.

Demetrios Brinkmann [00:32:27]: Skills.

Samuel Partee [00:32:27]: I think tools are pretty overloaded.

Demetrios Brinkmann [00:32:29]: Yeah. Yeah. Tools have a lot of connotation. All right. How do you continuously manage Arcade's tool integrations for maximum performance when the tools you're integrated with may change in behavior underneath?

Samuel Partee [00:32:44]: Incredible question. Cicd, fake accounts, they go all the council on every single one. CICD runs the tools every night. We have to do that for all of the packages. Every single one. We have a difference between community packages and Arcade packages. Arcade packages will be tested that way community packages will not be great question. I have a lot more but let's keep going.

Demetrios Brinkmann [00:33:06]: Is Arcade open source?

Samuel Partee [00:33:09]: Yes. Everything is downloadable except the engine is closed source binary. But you can download the doc container. You can run it just like at some point you gotta pass. But single, single developer free. If it's just you hanging out developing, you just want to use it at home. Do whatever you want to Control your Hue Lights with a chatbot. Have at it.

Samuel Partee [00:33:30]: You don't have to pay us.

Demetrios Brinkmann [00:33:32]: Okay, Okay. I like that World Chatbot. Hue Lights. All right. Wouldn't creating a new tool calling platform essentially mean you'd, one, be creating your own protocol that would work only if others use it, or two, how do you scale and become that gold standard?

Samuel Partee [00:33:51]: So what we did was a definition format. So a tool definition format can be translated into any model format, which makes us kind of Switzerland. We can play nice with anybody. But you're not wrong about the protocol, except for the fact that if they just need to add an execution like the video that Harrison just put up, in that case, we're just a similar kind of HTTPs API protected by an API key. Except for you're going to be hitting endpoints that actually do things.

Demetrios Brinkmann [00:34:21]: Incredible, man. There's a lot more questions that are coming through in the chat. We sadly don't have time to go through.

Samuel Partee [00:34:30]: Yeah, it's my name.

Demetrios Brinkmann [00:34:33]: We'll try and I'll get you into that chat right now. So you can keep answering those questions because I think you're going to like some of them. This has been awesome. I appreciate the talk. I was wondering myself the first time I heard you say tools should be treated as code, and that resonated with me.

Samuel Partee [00:34:54]: It's good.

Demetrios Brinkmann [00:34:55]: Let's get there. Let's get there. All right, I'm kicking you out. Now, sadly, I don't have a goodbye video of the cows saying moo, but it's time for you to move on out.

Samuel Partee [00:35:06]: Give me one of those sweaters. That's what you have to do.

Demetrios Brinkmann [00:35:08]: That's it. Let's trade you give me that sports coat. I'll give you this little poncho. All right, you can have it. Bye. Bye. I'll see you later.