Under the Hood of an Algorithmic Hedgefund

Beginning with an overview of spatialedge.ai, the company spearheading this innovative approach, the talk navigated through the vast landscape of the platform's extensive data backend. Spanning over two decades of historical data alongside real-time data streams, this backend serves as the bedrock for informed decision-making.

Central to the discussion was the fully automated trading strategies which are developed and trained to capitalise on market dynamics, as well as the rigorous performance benchmarks these strategies must meet before being included into the fund's portfolio.

Transitioning to the operational architecture, the talk provided a glimpse into the intricacies of strategy/model deployments, hosting, monitoring, and the CICDCT pipeline. These facets collectively ensure the seamless execution and optimisation of trading strategies, underscoring the sophistication of the algorithmic hedgefund's operational framework.

Join us at our first in-person conference on June 25 all about AI Quality: https://www.aiqualityconference.com/(url)

Pierre Cilliers [00:00:00]: Hello everyone. Yeah, unfortunately I'm not going to tell you guys about LLM making millions. I'll just be speaking more about the operations of the actual over ethnic age fund. Cool. So I'm just going to give a very brief background, use the opportunity to just introduce myself and where I work. So basically, as you guys have not realized, I'm from South Africa, recently moved to the UK about twelve months ago. My background is in operations research, so heavy mathematical statistics. All my postgrads were more in the machine learning aura and then I went into doing some research for an algorithmic edge fund, hence me being in this space.

Pierre Cilliers [00:00:43]: But my current focus, I'm a machine learning engineer at spatial edge or spatialedge AI, and the tech lead at a fund called the Sonic fund. So I might say sonic, one might say spatial edge. Same thing. It's just a project or the company, but yeah, so yeah, we do some machine learning operations, quantitative development, which is more the mathematical calculations of price, data in, actions out, and then some fund management kind of, because I let the algorithm do it. Okay, cool. So just to introduce spatial edge, just very briefly, so what market we are in. So there's crazy statistics of McKenzie explaining the whole machine learning AI realm that we are in. They say that there's a 13 trillion opportunity for businesses to unlock value.

Pierre Cilliers [00:01:35]: But a shocking state is that 18% of mister models that go into production don't even add value to companies within the first three years. More shocking is that 87% of products don't actually reach production. They're more of proof of concepts, more of exploratory work. Cool. So just a bit of our track record. So we help companies improve profitability by automating and optimizing their operations using AI. So we have about 170 solutions delivered. 90 plus of our solutions, or 90% or more of our solutions, are currently still operational.

Pierre Cilliers [00:02:11]: And we've created some value, about $300 million worth of value in our clients. So before you start judging my presentation, these are the brains behind what I'm going to present. So we have about 115 employees. Just under 20% of our employees have PhD. So a very specialized group of people, we have about 600 years of experience, and we consist of data engineers, data scientists, machine learning engineers, software engineers, and some business people obviously doing some unlocking value touched quite a bit of technologies. So now that you might not question the technologies I might touch on, now we'll get to the tech stuff. So everyone knows the saying garbage in, garbage out. So I'll just touch on the data that we typically use so we did quite a bit of work.

Pierre Cilliers [00:03:02]: I'm not going to touch on how we got the data and how we ingested and all of that, but basically, for a hedge fund to run, you can trade different financial assets, you can trade forex, you can try trade the stocks, cryptocurrencies, commodities, whatever you'd like. So we have all this data in our back end, more than 20 years worth. And if, you know, financial data in some sort, they typically burn it. So it might be like daily data. So you have the open price of the day and the closing price of the day, and it goes all the way to tick data, which is any event that happens in the market, we get that as feedback. So we have all of this data that we train our models on and run our strategies on, hence models and strategies. So I'll touch on the strategies quickly. I'm going to use these two concepts quite a lot, and they are different.

Pierre Cilliers [00:03:53]: So a strategy gets data in, it runs logic, and it sends trade executions out. So that's very simple. Okay, so it's basically a set of conditional rules. It can be some if statements, it can be a machine learning model, it can be basic technical analysis. But yes, this is basically a strategy. Data in logic transactions out. A model, on the other hand, which I think is very valid for this talk, this can be anything from traditional machine learning models, forecasters, predictors, transformers, some sentiment analysis that you want to do. I'm not going to touch on the specific models.

Pierre Cilliers [00:04:32]: We use all of them, but if you have questions, please ask. So, yes, so very basic, I think. This is Python. We have a model, we feed it data, it makes a prediction. But now this strategy uses this model prediction with other rules. So it will say if the model prediction is one, and other rules, such as price greater than XYZ, we want to open a trade. So that is what a strategy is. It's more of a brain model, is one feature of the strategy, let's call it that.

Pierre Cilliers [00:05:01]: And it's very important to know that a good model does not equal good strategy performance. Right. So you might enter the market brilliantly, but you still have to decide, when am I taking my profits, when am I cutting my losses, all of that. And that is what we, what we do. Perfect. So I'm going to focus on a single strategy for this example. So this is typically the timeline of a strategy, is you have some ideation and development. A quantitative developer, so very, very smart person, would get an ideation and start developing and trying some manual techniques, get a trading strategy, how it wants to manage trades when it goes against you, how to manage it when it goes with you.

Pierre Cilliers [00:05:44]: So this takes about two to three weeks and then we do training and backtesting. So this is when we run it over the historical data. So we feed the historical data in as if it's real data. So we withhold everything in the future and stepwise allow it to see historical data. So this strategy would think it's running in real time, but perform on historical data and give us a performance of how the strategy would perform in the real world, theoretically. So we do have some pain points here, and I'll touch, we can maybe bring this up a bit later, but having complex machine learning models in our strategy is time consuming. Imagine running 20 years of minute data, which is millions of entries, and having your machine learning model score in real time. So having it in real time score at every single minute, that's no problem.

Pierre Cilliers [00:06:35]: But having it score over 20 years of data within an hour, it gets quite complicated. Not solved. Pain point, but we're getting there. Then as soon as we get some favorable backtesting results, we deploy it to demo trading. So this is when it trades on fake money, right. And this is just to ensure that we interact with the broker correctly, that it executes our trades and that it does what the developer wants it to do. And as soon as we're happy with some trade executions, which typically take two or two or three days, we deploy to live on live fund hedge funds and hope for the best, I guess. So some features that I'll, I'm going to call it our product.

Pierre Cilliers [00:07:17]: So some features that our product offers is, for instance, you might have a strategy, you might train the strategy on one asset pair on, let's say the apple stocks or whatever. And it does extremely well. Before you allow to continue, before you submit it, we run it on every single financial asset we have in our back end, right. And based on a performance filter, which might be what you'll draw down if you know what drawdown is or what's your annual gain or your monthly gain, it would output what financial assets the strategy typically would perform well on. So developer can actually investigate a bit further on. More to this is that we do provide some optimization. So this is for models, this is for strategies for both. You give it a few configs, it would run over the historical data and give you outputs of your different configs, right? So you can choose your drawdown features.

Pierre Cilliers [00:08:13]: So it might have better gains, but it might have a, a bigger drawdown where drawdown is typically how far do I fall or how far does my account fall before recovering. Right. So all of these statistics have to be taken into account. So I'm just explaining all of these very briefly because they all tie into our whole ML ops process. Perfect. So just for those in the finance industry, typically for a backtest, we would get a graph compared to a benchmark. I chose the s and P 500, for instance, and you'll get many statistics. You'll get, for instance, how long is your strategy in the market? How long does it have open trades? What's your accumulative return? Sharpe ratios are financial metrics to rate your risk versus reward.

Pierre Cilliers [00:09:05]: And then we also have things like expected daily gain, expected monthly gain. So our strategies typically have to meet this criteria. So it has to have five to 8% monthly gain, which is 80% annual gain, an average drawdown of -10% so this, at any given time, it's allowed to have 10% drawdown. Our max drawdown is 30%. That's our hard cutoff. Strategies can't lose more than that. And that is typically a sharp ratio between one, one and a half or two. So now that you have individual strategies, which have brains on their own, which take different financial assets in, have different trade executions, we want to start structuring our fund or our portfolio.

Pierre Cilliers [00:09:48]: So consider we have a portfolio in the middle. We have an arsenal of 2030 strategies, but we don't have enough funds to distribute it to all the strategies, right? So each strategy would have some risk associated with it. Some would be high frequency trading, some would be medium frequency trading. They are different strategies. Right. So we compose the risk appetite and the gain appetite that we want, and we assign the strategies and funds to it, weighted on how our risk and our desired gain is, which we're currently working on, is if strategies do not perform as they do in backtesting, or whether we want to up our risk or up our gain, it would automatically select strategies from our unused strategies and substitute them into the portfolio. So this, we wanting to do more of a reinforcement learning approach, but that's still work in progress. So this is still, the portfolio part is still done.

Pierre Cilliers [00:10:49]: A bit more traditional. Now that you guys know hedge funds, let's get to the platform. Cool. So you guys would be happy to know that everything is monitored and CI CD is taking place with every single step of the way. So we have a developer, it's, they can design the strategy, they can test different machine learning algorithms, but as soon as they are happy, so they bt backtest on their local machines. As soon as they're happy with a model they submit it to our Kubernetes cluster and or in git to a develop branch or to a backtesting branch where final backtest is kicked off. So we don't check the codes. As soon as the backtest actually runs successfully and it passes the performance faulters we are happy with the strategy to go forward.

Pierre Cilliers [00:11:38]: Then with a merge request we go to demo trades. It automatically spins up. Next slide actually spins up our deployments. So it spins up a broker, spins up the strategy and spins up a message broker in the middle. So three containers in one deployment and then it trades demo. So it creates a demo account and then deploys the strategy on this demo account for you. And then as soon as we are happy and we want to include it into the live portfolio, create a merge request and go to live. And this is an iterative process.

Pierre Cilliers [00:12:12]: Models do get outdated, we do get better ideas. We do realize that we make bugs. So that is why we keep iterating perfect. So this is only one section of our framework. Unfortunately I can't show you everything that's under the hood of the algorithmic hedge fund. I'll only show you part of it. But essentially this is our deployment. So this is one strategy deployment.

Pierre Cilliers [00:12:38]: So this is not the fun. This is just a single strategy, right? So we have a strategy container, we have a message broker and we have a broker container. Okay, so strategy container is the most complicated and probably that you guys that I explained right now so you guys would understand. So we have a strategy file. This might contain models, this might contain just logic. And for instance, I'm inputting the same model. So model one with different financial assets that I want to trade on. So I want to trade the pound dollar, pound New Zealand dollar and the GBP yen.

Pierre Cilliers [00:13:12]: These are the three currencies I would like to trade on. So we have sonic Libraries. I mentioned that our project is called Sonic. So the sonic Library that actually structures the trades into something that the broker can understand. So we try and completely make this process broker agnostic. So if a broker goes down, we can substitute it with another broker, etcetera. So we have a sonic Library that connects to a sonic server. This basically just translates it into something that the broker understands.

Pierre Cilliers [00:13:44]: So we spin up a broker with whatever login credentials we create as a startup script and it spins up this broker in a container with our custom client that accepts trades and executes them into the broker. It also sends real time data to the strategy. So it is a circular process. Sends data, consumes data, sends trades, executes trades. So this is what a deployment would look like. Okay, so that's one deployment. So maybe we can get stuck into this. So just to talk about model performances and to focus on that, there's obviously some Model risks I'd like to bring it up for in case anybody has some advice or has experienced this in the past.

Pierre Cilliers [00:14:30]: So a big thing that we encounter is data drift. But this you can cater through. I hate saying this word, cicdct. So, continuous integration, continuous development, continuous training. So basically, we don't have to redeploy the model. We can continuously train it within the deployment. And this is for new price ranges. So, for instance, the yen reached all time low in the past week.

Pierre Cilliers [00:14:58]: Our model has never seen these price ranges ever before. Right. So how is it supposed to predict within these price ranges? We need to retrain the model on new price ranges or anything like that. So trending markets, right, or bitcoin hitting all time high. There we go. And then we also do new price behaviors. Right. So we don't have to redeploy our models if a new Covid hits or anything like that.

Pierre Cilliers [00:15:25]: Yes, it might not know how to cater for this instance, but it should train for it in the future. Obviously there are some validation checks and performance metrics that have to be met before continuing. Right? But yeah, new market is influences such as the 2008 housing crisis and Covid. That is fine to do with continuous training. What is very complicated is financial concept drift. So dynamic markets are subject to changing conditions. So this is if an asset gets decommissioned. So if a stock gets delisted or loses volume on the stock exchange.

Pierre Cilliers [00:16:07]: So people aren't interested in trading it anymore. Our model is used to trading with very high volumes of this specific stock or the specific forex pair. Right? So, for instance, if our model wants to enter, it is used to getting in or out of a trade within five minutes. Now that there's less volume, it might take a few days. Right. So we don't want to use the same model for if financial concepts change. Okay, perfect. Then regarding model retraining procedure, we prioritize strategy uptime instead of improving our model a little bit.

Pierre Cilliers [00:16:43]: It's not ideal, but uptime of a strategy is a lot more crucial than getting that little percentage of a model performance. Especially that a model performance isn't the sole characteristic of making the predictions and making the trades. Our rule of thumb is every two to three months, it's not great, but financial markets don't move that fast or don't drift that quickly. And on the other hand, decommissioning a strategy is not trivial. As you saw on my previous backtesting slide, one strategy was in the market for 94% of the time. So when do you decommission the strategy? So you might decommission a strategy which means you shut it down and you want to introduce a new strategy. But they are how many open trades open now? Who's going to handle those trades? You don't want to get involved, you don't want to close it yourself because you don't know necessarily how the strategy wanted to manage it. Okay, you do know.

Pierre Cilliers [00:17:38]: The developer would know, but the fund manager would not not know. So it's not a trivial solution, but it's something we are working on. Perfect. So I showed you individual strategy. I try to mix it up but it is pretty small. We have a second strategy with three different models. So first one would be the same machine learning model transformer, whatever you want to do for any forecasting where the second one, it can be three different models based on what symbol or what financial asset you want to trade on. And here we have different strategies.

Pierre Cilliers [00:18:17]: As I mentioned, our portfolio comprises of different strategies and that's how we kind of distribute our risk, right? So every single flow of data within our cluster is tracked in our metastore which is basically a postgres sequence where we capture every single data flow that happens. This includes every single price action in the broker. The reason for that is we need to do our own auditing. If a broker closes our trades, we need to know what prices they were offering, if they just closed it because they needed to or if it was a valid close. So we need to do audit on our trade executions. We also track all trade executions. This is what we want. So what our strategy sends.

Pierre Cilliers [00:19:04]: So let's say it wants to enter the market at $10 but in that few seconds or split seconds that it goes to the broker, there's some movements and it gets executed at dollar eleven. There's a little bit of spread there, right? So we track that. So we track what we sent out and we track what comes back. Obviously there's some bans that we find acceptable and if it's not acceptable, we don't execute the trade and let the strategy know. We also track all historic and pending orders, which obviously makes sense. And then we also track our account data which is our balance and our equity and our drawdowns and all of this. I was said to include that our cluster does auto scaling, right? So this is the more strategies we add, the more complex strategies we add. Cluster is developed to manage this.

Pierre Cilliers [00:19:58]: Scale up, scale down, deploy more nodes, node pools, etcetera, etcetera. So it's pretty dynamic. Yes. And we also have dynamic resource allocations as some strategies. If you have a reinforcement learning model in a strategy, it's going to be a lot more resource hungry than just having a few if statements and technical analysis rules. Perfect. So you don't necessarily have to read the detail here, but from the metastore, I'm going to touch on two last topics before I hand over to the floor is alerting and monitoring. So within the deployments we also have alerting and monitoring.

Pierre Cilliers [00:20:40]: I know some of you love that this is on deployment health. Are all of our strategies up? Is our cluster within our capacity? Has anything dropped all of that? We also have broker connection. This is if anything is closed on the broker side, you can't trust brokers. Then we have strategy execution, as I just mentioned, and strategy performance. This would remind us to say accounts X, Y and Z are in profits. Make sure that in month end you scrape your profits and start from your starting balance again. So that's typically the approach that we take. Then we also have an Ops and performance dashboard.

Pierre Cilliers [00:21:25]: So this basically is just a SQL sidecar and Grafana just to say what deployments have to be up, what deployments are up days since failure, and also strategy performance and portfolio performance. And this is more for shareholders and stakeholders. Perfect. So just regarding the alerting, obviously we have extensive logging and alerting in Google logging. So we host everything in GCP. We found, well that's all our data, all of our deployments, all of our alerting, all of our ops. It's just that it is beautiful. Yeah.

Pierre Cilliers [00:22:02]: So the Google login would be comprehensive description of the alerts and strategy logic. And then we also, we use slack at work. So this would actually notify us if there's any irregular broker behavior. So if they are offering very big spreads, which means that they're falsifying what prices you may enter and exit, then agreed upon cluster resource health, broker connectivity and strategy uptime, and I can show you some value. So we have some monitoring dashboards. So this is just one of our strategies. So you'll see we can select an account. So triple 3202.

Pierre Cilliers [00:22:40]: And the strategy running is sonic DS, which is data science, market flow, and this one's been running live for a while, I think since December. And these are the statistics that it got a developer can then view in real time the drawdown, the balance, the number of trades. If I scroll down, you would see the win rate for longs, the win rate for shorts and some cross referencing can be done with backtesting to ensure that the strategy is performing as we trained it. So thanks for that. I'm happy to go through to any slides to answer any questions. Yeah, you'd find when you're trained models against historical place that you need to remove flatness one events or extrude the titles to prevent it in front of the model. So we accept that typically. So we accept black swan events if it's within our performance filter.

Pierre Cilliers [00:23:39]: So if it's still but less than 30% drawdown, we would still accept the model. But we typically include black swan events. Yeah. Can you talk a little bit more about the strategy aspect, like how much of it is the model? How much of it is domain knowledge of somebody actually writing those logic scripts? Yeah. What does it look like? Some are 0% model, so some don't have machine learning models in them or anything. Some of them are just a set of quantitative rules. For instance, I can take you through an example. If you google trading strategy or an algorithmic trading strategy, you'll find an entering the market with a take profit and stop loss, which means the rule might be if the price goes above a moving average.

Pierre Cilliers [00:24:29]: So let's say 100 moving average, I anticipate an uptrend, right? Or you can say if the price is above a moving average and my volume is high, it's going to continue in an uptrend. So I want to enter in a long or buy. Alternatively, you'll say if a price is below moving average, you'll enter short. So that's typically how the rules work. Right. But then on the other hand, what we refer to it is as trade management. So now you have seven trades open, right? And it's how you manage them. So you manage your exposure.

Pierre Cilliers [00:25:01]: So you say, I have too much JPY exposure, I need to reduce my JPY exposure. So you increase other financial symbols instead of a JPY exposure. So it does get very complicated as soon as you get to the trade management part of it. But this entry signal is very trivial. It is how all manual traders trade. You can even draw, you know, trend lines. Have you seen people draw trend lines before? You can mathematically program that, right? You can program support and resistance lines and you can enter and exit the market like that. So anything that manual traders could do, you can do programmatically.

Pierre Cilliers [00:25:41]: Yeah. Well, you said speed is sometimes more important than the performance of an individual model. Yeah. Because these patterns, you can. Yeah, yeah. So the speed I was talking about is more of the backtesting. It's just pumping the data through a, let's call it a pickle. So we pick our models.

Pierre Cilliers [00:26:02]: You pump it through a model and it spits out if it is a binary classification, a one or a zero. So to do that for 20 years takes extremely long, where if you do it every 15 minutes in real time, that's fine. So we typically don't have speed issues in real time. It's more in time consuming with backtesting. Yeah, very good question. I pray for that day that I'm willing to influence the market. No, we don't have the volume to do that. The reason we trade forex is because it has the most liquidity from any stocks.

Pierre Cilliers [00:26:44]: So a tweet from a president doesn't move the forex market as what it would do on stocks or crypto or anything like that, because big central banks, they move money every single day. So, yeah, fortunately we don't do that. Yeah. So there to just be to class. So the examples we have is just to classify Reuters, Reuters financial models, just to see if it has a positive, neutral or negative outcome on a currency pair. We haven't found a legitimate way to do Twitter feeds or Reddit feeds as there's a lot of noise on there. But regarding transformers, we've used transformers for price prediction. It's currently not much better than your traditional, your traditional ways of predicting momentum and stuff.

Pierre Cilliers [00:27:42]: But yes, we are using them. I would love to say we are 100% systematic, but yes, if things go so, we do have measures. So automatically it would start pulling out if we reach 25% drawdown because 30% is our limit. But obviously rapid movements in the markets, it might not work. And if a broker connectivity drops while the market falls, we're going to have to get in v. And so you could just say, boom. What we need is last step. You risk strategy, which seem to work better in these conditions.

Pierre Cilliers [00:28:22]: Yes. So the beautiful thing of our CI CD process, which is on git, right, is git has version control. So there you can roll back any strategy that you have. Yes. And so as soon as we do a backtest in the cloud, it actually stores a job loop of the actual state of the strategy, the params of the strategy and what data the strategy ran over. So we have the complete state and performance metrics of that strategy. Even if we update it, even if we develop models from it. Yeah.

Pierre Cilliers [00:28:54]: So we keep version controlling of it. How far do we backtest? So 20 years, 2007, let's say from the platform. So now it's hands off. We're not developing more on the platform. So it's a viable product? Yeah, it was done by the software team. Unfortunately, I won't be able to give too much details to the integration testing, but, yeah, please reach out and I'll put you in contact with someone that I would love to speak about. But, yeah. Do you buy all the strategies to come back in a quick enough time before you can action a trade? Or could you have like two hour free come back and say, yes, go ahead, that'd be enough for the system to act, or do you have rules around that? So a single strategy would execute or not based on the delay.

Pierre Cilliers [00:29:54]: Right. So a fund. Are you talking about the fund with multiple strategies? No, it would be per symbol. So per financial asset you want to trade on. So the financial asset data goes through to that specific strategy, makes a decision if it's within the bound or in the second to execute the trade, it would execute the trade. If not, it would send a response back, realize, do I still want to enter the trade or not? Or alert us or any action that it needs to take. Yeah. So luckily not all three.

Pierre Cilliers [00:30:30]: Yeah, yeah, yeah. So our strategies aren't that linked from cross asset? Not there yet. So all I'll do act. Single asset. Yeah. There's 22 main forex symbols, and typically we design strategies that perform relatively well on all 22. Cool.