Distilling 200+ Hours of NeurIPS: What’s Next for AI

Nikolaos Vasiloglou [00:00:00]: There's been only one area that for many years, people are trying to use deep learning to beat traditional shallow models, and they haven't managed to do that. And that's your typical XGBoost type of models, like tabular data predicting on data frames. My name is Nicolas Vasiloglu. People usually call me Nick the Greek. I'm the VP of ResearchML at RelationalAI and I got an award for finishing my PhD without drinking a single cup of coffee. But as I get older, I need to get some caffeine. I'm also a marathon runner, but I'm allergic to coffee, you know, so this is how I take my coffee. I take coffee pills.

Nikolaos Vasiloglou [00:00:54]: They make my day. I wake up in the morning, I open the bottle, I take a pill, drink some water, and then everything looks great. Take them in the morning, not at night to wake me up.

Demetrios Brinkmann [00:01:05]: You went to Neureps. You distilled the plethora of information in talks and papers that came out at neurips into almost like a cheat sheet. Can you explain what you did and what made you. What drives you to do such a thing?

Nikolaos Vasiloglou [00:01:23]: Yeah, I don't think it's an exaggeration to say that I tried to become a human language model, which is a little bit of strange because language models were invented by humans. This is how we speak. And I think now we're trying to imitate them, in a sense. So I do have a passion for research. Although I work for the industry since I finished my PhD, I've been very close to research. I try to publish. I have interns. But I think I remember what somebody told me many years ago.

Nikolaos Vasiloglou [00:01:55]: He said it was a researcher in biology, and he said, well, we should stop doing research and just read our papers over the last 10 years to see what we've done. And there's this particular thing that people always want to publish, they always want to talk, but. And we produce a lot of material, especially in research these days. It's amazing. But I don't think we ever take the time to read what we produced. And I've noticed through the years that there's always papers that they just don't get on the spotlight for some reason, whatever reason. And then it takes us 10, 15, 20 years to say, oh, yes, actually somebody had mentioned that, but we never paid attention to it. So I think there's a gap in the scientific world, in the industry world, which is, okay, we say what we say.

Nikolaos Vasiloglou [00:02:52]: We produce all this research. Can we somehow try to ingest that material and communicate it to the leaders, to the decision makers. These decision makers can be research managers, they can be CEOs, they can be VCs so that we actually know what we're doing. So yeah, that's a passion that I've been doing for many years. This was the most difficult year because we're talking about a conference with close to eight and a half thousand publications. Hundreds of hours of video, just like reading, going through, skimming through the material. This is like 10 hours, just like going in sync everything one by one. I think probably the last year that I'm able to do that for neurips.

Nikolaos Vasiloglou [00:03:45]: I think next year I will have to use the help of language models. Maybe we'll have to train the language model with.

Demetrios Brinkmann [00:03:51]: So you didn't use the help of AI this year?

Nikolaos Vasiloglou [00:03:54]: No, I didn't actually I did in the end, let me tell you this, this was a little bit, you know, should I say disheartening. So after I finished my presentation, I created, you know, 600 slides which I broke it into 12 different lectures if you want. I recorded the videos. I had this idea, people are using NotebookLM to generate podcasts. And to my surprise, you know, NotebookLM did such a great job explaining the material. I thought I did a good job creating representation by just taking the transcripts and giving them to NotebookLM was experimenting with something else called PDF to audio. We can talk about that if people are interested in that. I said, wow, you can communicate ideas better than me.

Nikolaos Vasiloglou [00:04:51]: Now I just want to put an * Because AI is running so fast. And I had the honor to meet the product manager of NotebookLM at Google. So when we started, NotebookLM was a little bit hard to tame. It was sometimes was going off road, it was oversimplifying. So we had to kind of like learn how to tame it. But in the past six months it has improved a lot and I don't really think it needs that taming anymore.

Demetrios Brinkmann [00:05:20]: Fascinating. So what you did, if I understand this correctly, is you took all of the recordings and all of the papers that were submitted to NeurIPS this year. This last year. Right.

Nikolaos Vasiloglou [00:05:32]: Sort of. I didn't go through here. Going through eight and a half thousand papers would require four years of work for me. Wow. I just went through the presentations, the workshops, selectively, some papers and I'll explain why I did that. So it's kind of interesting knowledge. It's already self organized through the. So we can talk about that hard work.

Nikolaos Vasiloglou [00:05:56]: But essentially I try to, like if you see my process, it really looks like the process of doing gen these days. So I think there was another benefit for my professional job about how, how do you basically use gen AI in the enterprise? Like if you think that I got, you know, this huge material and I try to identify places where this is well summarized or, you know, well ranked if you want, and working iteratively by myself and getting to this 600 slides. I think this is something that now you can start doing with language model where you have to prepare material. Yeah.

Demetrios Brinkmann [00:06:43]: Why did you choose certain papers over others?

Nikolaos Vasiloglou [00:06:50]: So here's the thing. Let me explain to you how it works. So there's three major conferences every year in AI and they happen with this order and this is the order of importance, iclr, which actually happened a few weeks ago in May. Then in the summer we have icml, which is bigger, and then we have new rips in December, which is much bigger. And I think it's the most complete and probably the oldest. What happens is when you go to Neurips, there's a lot of keynotes, you have the orals, the oral presentation, not all these eight and a half thousand papers. It's about four and a half. I don't remember the numbers I have in presentation that you have them, the main Track and another 4,000 in workshops.

Nikolaos Vasiloglou [00:07:32]: Not all of them make it to oral presentations. So you first go through the keynotes, you go through the orals, and then you go through the workshops, the keynotes and the workshops and presentations that they look interesting. I think that's part of me. I've been many years in the area, so I think I'm a good ranker. So I basically use what the new RIPS reviewers have done by elevating a paper to an oral to a keynote, the workshop organizers, by selecting which one's going to be, it's more important. And now all these presentations, all these oral presentations, they usually cite and reference papers that they've already published, let's say within the last or two years. And sometimes when they go to the conference, because things are running so fast, they're also presenting about things that they have submitted for publication in the next year. So by attending one conference, you kind of like get an idea about what is happening in the past six months and what's going to happen in the next six months, at least as a publication.

Nikolaos Vasiloglou [00:08:40]: And now then I start, you know, as you watch the presentation, you start digging through the references and then you find, oh, you know, this is a reference that maybe wasn't very well perceived or, you know, or it Got published like a year ago and we needed a year for people to kind of like start talking about it. And that's how it goes. You take snapshots, you take. I took a about 2,000 screenshots from papers here and there. I have a way of organizing the material anyway. And then you keep refining and then you keep cutting, you keep cutting, consolidating, cutting until you bring it. This year, typically I can make it fit it in an hour and a half the previous years. This year I had to basically have 12 presentation close to, I think six hours, six and a half hours of me presenting.

Demetrios Brinkmann [00:09:30]: Well, it's 200 slides, you said.

Nikolaos Vasiloglou [00:09:33]: No, no, it's 650. They're public. I actually have everything available. The initial screenshots, which I think is close to 2000. And then after distilled that to I think 600 slides.

Demetrios Brinkmann [00:09:49]: And then you had the six hour presentation that you yourself are talking about, your understanding of all of this and why it's important.

Nikolaos Vasiloglou [00:09:58]: And then after that, the podcast, which makes it even, you know, even smaller.

Demetrios Brinkmann [00:10:03]: The Notebook LM podcast. What are some things that stood out to you?

Nikolaos Vasiloglou [00:10:08]: Yes, you know, surprisingly, I don't know, but I managed to break it into 12 topics. That's why I named this. It is like the 12 days of new Rifts. It kind of coincided with Christmas, but I didn't do it on purpose. It just happened to be 12 things and the presentation is broken. There's two slides. That is a nutshell. Then I have a slide with 12 topics and then there is three slides for every topic and then there's a whole section for every topic.

Nikolaos Vasiloglou [00:10:37]: So you can go hierarchically as deep as you want. I think the very first thing that kind of stood out is that we see that with every technology it pops, there is buzz. It's kind of like the hype cycle either makes it or doesn't make it. So now it's the year that we can say that language models are everywhere, but not for everyone yet. And it seems to me that I don't remember neuripskiing around the Deep Seq kind of boom or buzz. But it is true that the open source language models are winning. There were a lot of, there were a lot of, how can I say this? Papers about language models that are very, you know, they are very open, they give you everything. They give you all the checkpoints.

Nikolaos Vasiloglou [00:11:53]: You can reproduce them, you can pre train them from those checkpoints. If you want, you can take their data set so they open sourced a lot of data. Sets that you can modify if you want and you can include them in your own deployment. Okay? And there was a presentation, I think if there was one presentation towards was from the inventor of the lstms. So for the audience that doesn't know what an LSTM is, it was the precursor of the transformer that made the language model what it is. And he basically had a slide where it says, the way that I translate it is everybody loves to hate language models. Okay. And he lists like a bunch of complaints from people.

Nikolaos Vasiloglou [00:12:54]: It's not going to give us AGI. He can't do this, he can't do that. And when you hear people complaining about something, it means that they're using it a lot. And these complaints are actually constructive because these are the complaints that are going to lead us to the next. To the next big leap. I think it's kind of fascinating that this was the 10th year anniversary from the first sequence to sequence model. So the sequence to sequence model is basically what a language model is. And you give it a sequence and it generates another sequence.

Nikolaos Vasiloglou [00:13:34]: You give it a prompt, it generates. And that won the Test of Time award. But the author was Ilya Suskevertz, the famous OpenAI guy with the whole drama left. And so he was there and he gave a very nice presentation about that. It was also the 10th year anniversary of another monumental paper, instrumental paper. It was the gans. It was the first time we synthetically generated images. And it's been 10 years, those years.

Nikolaos Vasiloglou [00:14:06]: I remember when these papers came out, everybody was very critical because they said, oh, your references were only one year old. And that's not a nice thing. When we write a paper, you want to always refer papers that go back in time. But that was the beginning of deep learning. There was nothing before that. Now we can go 10 years back and, and reflect and talking a little bit about deep seq. I just wanted to mention that this is kind of like this is the second time that it's happening. I remember, I think it was 2023.

Nikolaos Vasiloglou [00:14:43]: It was in the middle of NeurIPS where OpenAI announced ChatGPT. And all of a sudden everybody was talking about that. I think they do it by choice. If you ask my opinion. These announcements are coming around the conference to kind of like leverage the congregation. We're talking about 16,000 people. Okay, so this is the place where you want to go and throw the bomb. And obviously there was a lot of discussion and something that I noticed is that both years, like back in 2023 and now in 20.

Nikolaos Vasiloglou [00:15:17]: Actually, no, it was 2022. 2022 and this was new year 2024. It was two years ago. There was no one from OpenAI or very few people from OpenAI then and now there wasn't anyone from Deep Seq. I think I found only one paper that A small paper that was someone from Deep Seq. And that's a sign, in my opinion, that these people are waking day and night. That's why you don't see them there, I think. Or maybe they don't want to be there because there is so much that they want to withhold.

Nikolaos Vasiloglou [00:15:51]: They don't want to be.

Demetrios Brinkmann [00:15:52]: They'll get trampled.

Nikolaos Vasiloglou [00:15:54]: Yes. But you know, this year there were a lot of Great talks by OpenAI. It was my favorite one. So a specific one about how trying to improve ourselves in gaming basically brought us to the reasoning models and very nice presentation, which means that now OpenAI starts opening more and more and giving out to the world and explaining what they're doing and how they're doing. So, you know, that was. Obviously there was a lot of stuff around Agents, if you ask, they basically dominated the conference. And you know, I could go through the whole 12 of those, but I'll just, you know, you let me know maybe on.

Demetrios Brinkmann [00:16:50]: On the Agents thread. Was there anything that jumped out at you with the agents that still to this day, because it's been six months since Neurips and it feels like you were saying that things move very fast and sometimes it's a flash in the pan, sometimes it is a signpost that needs further research. Did you see anything there with respect to agents that now you're seeing more and more a common pattern or it is something that is more established?

Nikolaos Vasiloglou [00:17:27]: There were two things. First of all, there was a lot of buzz about, you know, the smaller models and if you remember, like Nvidia stock and OpenAI, there's not a stock. But anyway, the kind of like they dipped in there was this thing. Are the smaller ones challenging them really? And I remember there was a slide about. There's a lot of evaluation frameworks about. It was a benchmark on agents between different language models. That benchmark had the frontier model, as we call it, like Claude Token, AI Cohere, Gemini. These are the frontier models as being better than others.

Nikolaos Vasiloglou [00:18:05]: And of course they hadn't included in their benchmarks the newer models, you know, the Quinn and Deep Seq. And there was this impression that, you know, are these outdated or what's going on? I think six months after Neurips and all These buzz of smaller models, I think they're useful but the buzz they made was a little bit bloated. I think the frontier models are still ahead of the game and they're worth their money. Of course, people are now experimenting with this reinforcement learning technique for fine tuning and we're learning a lot. So that track kept going on and I think it's producing results. I personally use them in, in a project at Relational AI. It has to do with. I guess we can talk about that since we're going to talk about it in a week at the Snowflake Summit about text to SQL competitions and how we've modified and used, I want to say simpler but more effective method of using that.

Nikolaos Vasiloglou [00:19:19]: So I can see that from a personal experience, but I follow what the others are doing. So this idea of reinforcement learning, chain of thought is producing results. The other thing is something else which I personally find it fascinating. There were papers that they were talking about building a lab from agents, a biology lab or a materials lab or an antibiotics lab where there's agents trying to do research and discovery, a new antibody, a new material. And that was really impressive. These are the cases where the hallucination, the imagination of language model is a feature, it's not a bug because you want them to think out of the box and you don't care if 9 out of 10 are wrong or they are hallucinations. One of these hallucination could be useful. Of course there was a small setback because there was a paper from MIT that supposedly discovered, that came a bit later, a new material out of nothing.

Nikolaos Vasiloglou [00:20:34]: And it created even bigger hopes and it got retracted. I read recently that it was also cited by Nobelist in economics. Now they took it back. But if we see iclr, that happened five, six months after Europe and we see the workshops, we see a lot of activity around that, like building an army of agents that they're actually going to produce new stuff, invent things. I think there's definitely a lot of interest by components, but there's also a lot of pitfalls. We can talk about that if you want. Want more?

Demetrios Brinkmann [00:21:15]: Yeah, let's go down that path.

Nikolaos Vasiloglou [00:21:18]: Yes. So I think while it is useful when you're working on a research lab or where you're trying to, you know, it doesn't have to be a research lab. They say you are the finance department and you want to run like a simulation. You can ask if we do this or that, what's going to happen? And you can have, you know, agents talking and creating scenarios and Ideas that you, you never thought about it. This is great. But remember, this is what I called agents as decision support. Okay? Not decision making and decision execution. Okay? So.

Nikolaos Vasiloglou [00:21:52]: And I think that's a great path and I think that's the way that you should be using agents. But I think slowly companies are trying to build agentic models that they're doing decision making, decision execution. Actually, you and you might have seen, like some companies that they're trying to organize your calendar. There's a nice research preview from OpenAI. Again, it's called the Operator, where, you know, say, book a restaurant for two people in a nice restaurant with. Book a meal for. Yes, with seafood, whatever you want. And it starts opening a browser and comes back to there is a Vasa for you, which is probably fine.

Nikolaos Vasiloglou [00:22:35]: If it messes up, it's not a big deal. But I don't know if you're going to say, go buy me a car or I'm giving you the bank account making a decision for me. I don't think we can trust them. And I think in many cases, remember, these are also probabilistic systems. People are trying to trust deterministic tasks to probabilistic systems that they're extremely difficult to control and extremely difficult to debug. So I feel like we're reliving the time where multithreading came out and people jumped to adopt that and then they very quickly realized that you can put yourself in a very difficult situation where the system is failing and you don't know why and you can't fix it. So I think there's a bit of a, you know, rush over there. You know, we're asking things, there's some.

Demetrios Brinkmann [00:23:31]: Threads we need to pull on and we need to get things that are a bit more reliable before we can really make the significant advances we're looking for.

Nikolaos Vasiloglou [00:23:42]: There's also a few other things I think the things that are more important in, in my opinion, if you want, like ages, you know, obviously people were talking about them, we see them now. What I think is interesting from this presentation is research that we haven't really seen in production yet, but it's coming very aggressively. It's being baked. You can see a lot of momentum, but you probably don't see, or the audience, you don't see it in your everyday life because there's no companies producing the products with them. So there is. For example, Genai, or deep learning, has conquered every traditional area of machine learning. They did it with images, they did it with text. Anything you give them this recipe just give us lots of data to pre train and try to put predict the next token, the next thing, the next something unsupervised in any way and we can finish that.

Nikolaos Vasiloglou [00:24:49]: There's been only one area that for many years people are trying to use deep learning to beat traditional shallow models and they haven't managed to do that. And that's your typical XGBOOST type of models like tabular data predicting on data frames and tables. And I think this is the first year that we had the series of models that they managed to beat the, you know, the undisputable Queen of models, XGBoost that everybody's using for predictive analytics, of course at a higher cost. So you know, there's a workshop that I've been attending for three years and it shows signs right now that I think next year we're going to see language models for tables for database. They're going to take database sets of tables as input, get pre trained and you will be able to do, you know, predictive models and ask questions directly on those data.

Demetrios Brinkmann [00:25:52]: So this is one of the things on that point. I know that in the community a lot of folks have asked time and time again about time series and forecasting models and, and you have these forecasting foundational models, but the jury's still out on how valuable they are. And it feels like that's kind of in that same vein of yeah, maybe the research is really pushing in that direction, but is it. Has it overtaken what the common industry uses are? Not yet, but it's coming fast.

Nikolaos Vasiloglou [00:26:32]: Yeah, it's coming fast. Definitely for the first time we saw them, there's still this argument which is like, even if you are marginally or a little bit better, is it worth spending all these GPUs and these resources? But I think people are waking like they're making lighter models. And there's another aspect that you have to see which is in the enterprise world people think that you hate is having 20 different platforms, you know, one for your predictive one, your prescriptive one for your, you know, graphs, one for your whatever bi. People want one thing, one platform. This is by the way, why the data cloud. AI data cloud companies are marching right now like Snowflake, you know, databricks, bigquery. It's because they're kind of like consolidating this, you know, this set of tasks in one place. Okay.

Nikolaos Vasiloglou [00:27:27]: So you know, so if you have like one right now, let's say foundational model that can do everything for you, it can do your sentiment analysis, can do your knowledge extraction, it can do the query answering, it can do the predictive analytics, it can do the optimization for you Prescriptive analytics. This is what people call the generalized rag, something I'm working on. But Amazon also presented the auto glue and this is part of their vision as well. So you find more in my presentation where they're trying to have one box where you ask a question and if that question is a predictive model, calls it predictive model. If it needs typical RAG retrieval, it uses that. So if it needs to solve an optimization problem, it solves an optimization problem.

Demetrios Brinkmann [00:28:15]: I guess that's a little different than this model that's trained and it's like a foundational model for tabular data. Right. But the idea that you're saying is that will hopefully that type of a model will be able to generalize across many different of these questions. So you don't need to think, call a rag, call a, a prescriptive model, call xyz, you're just going to call that model.

Nikolaos Vasiloglou [00:28:43]: So here's the thing, let me make a small digression. The reason why we don't have language models that can deal well with tabular data is because we don't have a lot of public tabular data. Enterprises don't open source their databases. We don't have an Internet of tabular data. Now there's been a lot of effort to gather that the equivalent of a C4. So C4 is the, the big data set that people crawl the web and train all the language models. I think there's been an effort and that's part of the workshop over there that described all these efforts to create that in some cases the language model can natively do that task. That can natively.

Nikolaos Vasiloglou [00:29:28]: If you pre training with databases, there is language models where you say okay, here are some examples of training data. Here's my test data based on that predict how much I'm going to sell tomorrow. And that's great. There are some other tasks like for example, if you want to do integer optimization, we don't have language models that they know how to do that. And right now we have some pretty good solvers that Simplex, Kurobi and others that people have worked for many years and they are quite accurate. So in some cases you want the optimization to be. You have to be careful because if it's a predictive model, a language model, making a mistake is not a problem because any other solver will do the same. But when you're solving an optimization problem.

Nikolaos Vasiloglou [00:30:15]: In some cases, it could be you can have soft rules or soft constraints where it doesn't really matter if you're violating them. But in some others, you might have hard rules, like you're trying to do your supply chain and. Or you're trying to schedule your airplane fleet and you want to make sure that you're not going to have more than 100 people. Like, it gives you 150. That aeroplane does not fit 150. You got other problems and you have.

Demetrios Brinkmann [00:30:42]: Some pissed off customers.

Nikolaos Vasiloglou [00:30:44]: Yes. Which of course takes us to another thing that was very popular during the conference, which is verification systems. Okay, so this is becoming a central theme. You know, how are we going to make AI models more reliable? And long story short, if the task you're trying to solve, you can attach a verifier that's a perfect competition. That's the perfect combination. For example, you're trying to prove theorems. You got a language model which is very creative. Also a huge part of the conference, like math for AI or AI for math.

Nikolaos Vasiloglou [00:31:32]: You being creative to prove a theorem. But in the end you can check if you're right or if you're wrong. But there's other tasks where you don't have a way to verify them, such as what's the capital of France? You don't have to, you know, if it does you wrong, there's no way you can, you can verify that. So you're relying, you know, 100% on the language model.

Demetrios Brinkmann [00:31:57]: So the verifiers are going beyond like the guardrails and output checks type thing.

Nikolaos Vasiloglou [00:32:05]: Yeah. So the verifiers could be, you know, verifying, for example, a mathematical theorem. It's not. There's a system called LIN4 that does it for you. It's a complicated system. You can do it, but in some other cases. So here's the interesting thing. This is something we all have to get used to, thinking one direction and immediately try to think the other direction.

Nikolaos Vasiloglou [00:32:29]: So whenever you have the language model doing something for something, then you have to reverse. What can this something do for the language model? Okay, so, for example, when we generate software, okay, let's go to the biggest elephant in the room. Everybody's working on using language models to generate software. Okay, you need to verify, okay, you got your compiler that it compiles. That's like, okay, that's easy. But how do you know that the software does what you want to do? Okay, that's like a. So there are AI systems that they're trying to take your task. Like I'm saying, I want the software to do this reverse a string.

Nikolaos Vasiloglou [00:33:15]: Okay. And they are trying to create the verification tasks, the tests, if you want the verification tests that they produce, code has to satisfy. This is, you know, this is one of the most difficult tasks, actually. The reason why verification has been there and even Turing Awards in model psychic and all that stuff for many years. The problem is writing the tests or writing the verification processes that you need to pass through. And if you pass through, you say, well, it should be correct.

Demetrios Brinkmann [00:33:56]: Okay, yeah, it's basically unit tests for the output.

Nikolaos Vasiloglou [00:34:01]: Yeah, it's a little bit more than unit tests because it could be abstract. You have to show me that this variable never proved, that this variable never becomes negative in the code. You're saying that I'm having some variables and you have to prove that they will never exceed those bounds. Like which part it should be, all that stuff. There was a workshop over there and lots of people talking about that. Also a great topic.

Demetrios Brinkmann [00:34:35]: But the verifiers themselves are not generated by foundational models. They're generated by humans.

Nikolaos Vasiloglou [00:34:43]: Yeah, the verifiers are symbolic systems. You know, this is how we like calling these symbolic systems. Like if I tell you, for example, if. Let me give you an example. A verifier that everybody was going to understand. Let's say that I'm asking a language model to generate a path from Atlanta to San Diego and it gives me a path. Now I can go to Google Maps that they have a graph and I can say, okay, you told me to go from this road to this road and take this intersection. Well, does this intersection exist? I have to go from this node to the other node.

Nikolaos Vasiloglou [00:35:20]: Well, does this exist? And then you told me that the whole thing is going to be, I don't know, 3,000 miles. Like I can go and measure. Well, actually it's not. It's 2,000 miles or it's three and a half. So that's what I'm saying. You can think about. Most of the verifiers that we would use in enterprise applications will involve some type of a graph or some types of rules. We like calling them the whole package.

Nikolaos Vasiloglou [00:35:45]: That's knowledge graphs in the enterprise. This is again, relationally, I where I work. So the knowledge graph, and that's why they're becoming very popular with Genai, is, you know, they have a very strong interactions with language models. You produce something, and I'm using the knowledge graph to verify that this is working or something. Sometimes, you know, Maybe I'm creating a symbolic path that I go and I execute it on the knowledge graph. Lots of things and sometimes even people use the knowledge graph to train the language models. Anyway, we divert a little bit, although there was some discussion about that and mirrors. Wow.

Demetrios Brinkmann [00:36:26]: Well, so there I'm trying to think what we were talking about before we were talking about the verifiers and we went on that little tangent because there was also some good stuff there. But man, these verifiers are wild in trying to figure out the actual verification of them is quite difficult. And so all the talk and all the fun stuff around that I'm. I can only imagine is going to be getting more and more complex as time goes on.

Nikolaos Vasiloglou [00:37:00]: No, no, you're absolutely right. And just keep in mind that our. You remember when I talked before and I said gaming, trying to play poker, trying to, you know, play and all these games, they basically evolved and brought, you know, the kind of like chain of thought and test time inference. Like what we see right now, the old class of models or the deep seq class. I think people use the O class because OpenAI was the first 1010304. You know, that was by working with games. That's a beautiful talk. It's historical, it's nice.

Nikolaos Vasiloglou [00:37:33]: It has a lot. It could be a movie because there's students beating, playing poker, winning. We can make it Hollywood. Kind of like the same thing has happened with the verifiers and math. I mean, there's two reasons why the community is so focused on math. The first one is because this is kind of in human nature. We want to prove the Riemann hypothesis. You know, we want to go to the moon.

Nikolaos Vasiloglou [00:37:57]: Like people have this aspiration that this is, you know, we, we have the money, we have the, the brain power. Let's go and solve a difficult problem. Okay. The second thing though is it's, it's hilarious. You know, you know how you want to study history at Yale and people tell you, well, you know, in order to increase your chance to get at Yale, you should take, you know, AP calculus, an AP Algebra, like advanced algebra. You say, well, I want to study history. Say, well, yes, but when you're going to submit your application, they want to make sure that you are smart and you're hardworking and if you are good at math, then it means that you are smart and hardworking. Of course this is a little bit unfair because there's other ways of testing people if they are smart or hardworking.

Nikolaos Vasiloglou [00:38:49]: But this is what people know over the years, you're good at math, you're smart. Okay, it's a little bit. I don't like that. But that's how the. Now we do the same thing with language models. It's actually been proven that if you teach a language model to do math, it becomes a better agent, it becomes a better thinker. So you would see that people are pushing math theorems and code into the training data not because they want the model to do math, but it was to be a better reason. It's like Wall street, you know, they're saying, oh, this guy is a quantum physicist, you know, studies abstract math, let's hire them because you know they're going to be able to figure out the stocks.

Nikolaos Vasiloglou [00:39:27]: So these are the two reasons why. And that's where the whole discussion about the verifiers, of course is coming up.

Demetrios Brinkmann [00:39:39]: No, you've been working with graphs a lot. Talk to me about what you're working on.

Nikolaos Vasiloglou [00:39:44]: Let me first give you my definition. It's not actually mine. It came from someone who worked for us in sales. What is a knowledge graph? Okay. Knowledge graph is a language that both humans and machines understand. Okay, If I write code, code is something a machine understands really well. I don't think the average human or anybody understands. It's very difficult even for a software engineer to read code.

Nikolaos Vasiloglou [00:40:13]: A knowledge graph is kind of like an intermediate representation. It has like facts, has qualifiers, relations, entities, rules. So this is the truth for every company, for every organization. Even if my favorite example of knowledge graphs is Euclid. Euclid basically created the first knowledge graph of mathematics with his theorems, his axioms and all the axioms, he built theorems. And that was an imperfect. It was actually a wrong knowledge graph. It took about 2,000 years, two and a half thousand years until David Hilbert completed a, and put all the pieces together.

Nikolaos Vasiloglou [00:41:02]: And because of that knowledge graph he built, it helped him post the famous 20 or 10, I think, I don't remember questions that have to be answered. So how beautiful this is. You know, the knowledge graph helped us systematize everything in the back, in between, even before Hiltbert. It let us go to different geometries, like you know, non Euclidean geometries, like hyperbolic, spherical and all of them. And then Hilbert said, okay, now that we know what we have, these are the 20. I think they're 20. Either 20 or 10. I think they're 20.

Nikolaos Vasiloglou [00:41:36]: The ones that he said in the famous conference in 1900 and that because he said these questions, then Getel came and the Others, and there's still some of them unproven. The same thing is happening on a corporation, an enterprise. You kind of have to create this ultimate source of truth, which is your knowledge graph. And once you have that, it helps you build apps on top of that. Answer questions, do your research as we described before in the future, like show me what product I need to develop. The problem is, as I said, building the perfect knowledge graph, it's not a task, it's a journey and it can take time. And usually what happens is that the information is spread all over around here. You have them in databases, you have them in documents, you have them in code, you know, on Slack, yeah, you have them in Slack, you have them in zoom transcripts, emails and emails everywhere.

Nikolaos Vasiloglou [00:42:33]: And somehow this needs to be, you need to build like the skeleton if you want like this ultimate source of truth. Now I remember when I started my, my job at relational AI back in 2018, I would go to customers and say, hey, you know, if you have a knowledge graph, like we have this system and we can help you do xyz, it says great, but I don't have a knowledge graph, what would it cost me? And you know, you put down the cost, it would cost you, I don't know, a million dollars or something like that because it was too custom. You know, you really needed a lot of quantitators and there's example you can go, if you go to my Neurips presentations a few years back, not that that long, you can find the famous talk by Luna Dong from Amazon where she was saying that the product graph they built was 100 million project, basically more than 100 million. I think it was a hundred year project and it was several millions of dollars to build up just for the products of Amazon. LLMs didn't exist, an army of data scientists, annotators, blah blah, blah. And now the language model comes and all of a sudden that process, the, the bar is the cost is going down, everybody can do that. I wouldn't say it's trivial. You just don't get things.

Nikolaos Vasiloglou [00:43:50]: You put them on the language model and it happened on its own. But you need to build tools. So part of my day to day job is building tools natively in Snowflake because there's another aspect that we haven't talked about which is called security. So moving things around makes them insecure. So you basically want to do everything. I talked before about people wanting to do things in the same stack, in the same, you know, cloud provider, you know, in the same space. In the same language, if possible, and not. And not move around.

Nikolaos Vasiloglou [00:44:28]: So my job is how I can take all the sources of data and create a knowledge graph in an environment that is very friendly to databases because the majority of the data is stored there these days. And that has, of course, its challenges because databases, they are optimized for a particular task, which you have to respect and you're trying to do something else. So I'm there bridging that gap.

Demetrios Brinkmann [00:45:00]: Well, there's a lot of questions that come up instantly because I can imagine from the data engineering perspective, you have hot data and cold data. And so how does that play with the knowledge graph? You also have data retention policies. So how does that play with the knowledge graph? All of these things seem to make it a very sticky situation.

Nikolaos Vasiloglou [00:45:25]: It does. And you know, we tend to conflate two different. And I didn't make it clear because I didn't want to confuse. But maybe it's the time to talk about that. There's something called the ontology and there's something called the knowledge class. The ontology is like the type system. You know, you got the location, you got this and that. General, you know, types that.

Nikolaos Vasiloglou [00:45:46]: These are kind of like invariant over time. And then on that you attach like a fact, the facts, the value types, as we call them, which change and they have an expiration date. Now, I think here's a clever thing that we're doing. We belong in the category of the relational knowledge graphs, which means that our view of the world is very, very closely tied to the database. So you can think about a knowledge graph as a, as a view, as a layer, semantic layer that writes and reads everything from a relational database. So relational databases have kind of like solved this problem. Retention, access, roles, scaling, all these things. So of course, there are still some challenges because when you fine tune a language model, doing that, like incrementally adding facts you can do, removing them is not that easy.

Nikolaos Vasiloglou [00:46:53]: So you got to be careful what you fine tune and what you do dynamically. So it is a scientific. But it's mainly an engineering project. Let's say it's an engineering problem where you need to be smart about AI and how you use it. So it's R and D, you know, if you.

Demetrios Brinkmann [00:47:14]: But how are you making sure that your worldview is not limited to only that database? Or is that the way that you like to do it, where you say one database, one worldview, one type of knowledge graph?

Nikolaos Vasiloglou [00:47:26]: It could be multiple databases, but when we bring, for example, text data, you know, we ingest them and they see it as, you know, let's say triples. Nvidia is located in Mountain View, Menlo park, wherever they are, that's like a fact. It has a timestamp, but it leaves. So the document is there. You ingest it after you ingest everything, it sits on the knowledge graph and the knowledge graph sits on the database. Some of the data, they already sit on databases. But I think what people don't realize that when they build databases, they build them to serve an application. And they put more emphasis on how it's going to be fast, how it's going to serve the application, but not how it reflects to the concepts that you as a business owner or stakeholder, think about them.

Nikolaos Vasiloglou [00:48:16]: So that's why you see some databases that are super wide and you look at all these columns and you don't know what's going on with these cryptic columns. And somebody says, yeah, if you actually meet the customer, you have to say, select this when this and this, like it's like a, I don't know, a 10 pages query that basically defines what a customer is. Something like that.

Demetrios Brinkmann [00:48:35]: It's a secret handshake.

Nikolaos Vasiloglou [00:48:37]: Yes. So that's because that database was built to serve production, to be optimal, to minimize whatever you want, like whatever constraints you had. But that's not the way that you think about the data, the world, your company. When you're thinking about changing a gear in your car, it's a sequence of things happening. You don't care about all these things you're saying it's a gear, it's a wheel. You don't need to know the material that the wheel is. You have to abstract that when you're writing things. Now, of course there's a lot of collaterals.

Nikolaos Vasiloglou [00:49:12]: One is answer questions that. The other thing is, it's not very well known to the masses, is that if you had designed your database according to how you think of your business and that representation could also be efficient, then it means that you can build your apps much faster. Because every time somebody wrote an app and defined the active customer somewhere with a big SQL query and it's like in a Java file and then somebody else wants that and doesn't know how to find it and goes and writes that query. And that query is a little bit different from the other. Another is like inconsistencies. And so if you have things in a knowledge graph and you're trying to build an app, you can do it much faster, much cleaner, it's more maintainable. And we do have customers where we took hundreds of thousands of lines of code like UI where we could able to compress them with something smaller because we had a better representation of data. So that's something that's probably not very popular or familiar to the AI audience.

Nikolaos Vasiloglou [00:50:19]: But when it comes to application development and enterprise application development, this is another huge use of Node as gas. The whole enterprise is running on applications.

Demetrios Brinkmann [00:50:31]: There's a lot of meat on them Bones, that is a hundred percent things that I've heard people talk about, especially when it comes to self serve data and analytics. And each person is, like you said, creating their own data pipelines in their own way or they have their own dashboard and then they leave the company and they, the data engineer doesn't know. Do I need to keep babysitting this dashboard? Is anybody looking at it? Can I just start turning things off? I know I talked to a guy who said, yeah, we had this big push about five years ago to go to self serve analytics and it was great. And now we have over 10,000 data pipelines that the data team has to support. And so they're literally just chaos engineering and turning things off and then waiting for somebody to come and say, hey, why is this not working?

Nikolaos Vasiloglou [00:51:28]: And again I'm just going to go back to the mathematicians who are very disciplined. Every time you prove a theorem, you don't say and say according to that theorem, it refers to specific. So there is a very well known knowledge graph of theorems where you go, it was Newton's that or you know, like you go to the specific person, it has a name. Sometimes they have a name, but sometimes you refer to them as publication theorem X. So there's a reference system. So it's very rare that the same theorem exists, proven twice in two different places where you don't know about it.

Demetrios Brinkmann [00:52:07]: In two different kind of unique ways. That's the other thing here because you don't have to be limited to just doing it one way. Maybe you do it one way and.

Nikolaos Vasiloglou [00:52:18]: Then so think about, you know, you brought a very nice example here, the Pythagorean theorem. You know, there's about, I don't know, fifty, a hundred ways to prove it. But when you are proving something says according to the Pythagorean theorem and you don't care if it's like, you know, 50 different ways to prove it. Imagine now if we didn't have this, you know, knowledge graph, you would reference, you know, the Nick's proof and then somebody else will have the John's proof that they refer to the same thing, but they don't have the same name. And I will say, well, are they doing the same thing? Is it the same? Yeah, yeah.

Demetrios Brinkmann [00:52:53]: You have to be a expert on all 50 different ways.

Nikolaos Vasiloglou [00:52:57]: Exactly.

Demetrios Brinkmann [00:52:57]: Or you at least have to go and reference and do a little bit of Sherlock Holmes and figure out is there actually that way to prove it?

Nikolaos Vasiloglou [00:53:06]: Yeah, that's typically the person you can fire from the company who lives last. Like the person who's been there for 20 years and remembers that in 2021 there was an audads and we weren't getting data. It was a bug. So whenever you're writing a query, you have to exclude those dates. You see this query that says, but not in these dates to get the average or something, because that app was polluting the data. So, yeah, we have lots of stories like that.

Demetrios Brinkmann [00:53:32]: Man, this is great. Anything you want to end with?

Nikolaos Vasiloglou [00:53:35]: Well, you know, there's 12 topics. I think I'll close with one. Maybe I close with two because I have two more favors we have been covered. The first one is something, and I'll close with that because I think I have a little bit of a contribution over there. Something has to do with data monetization. Okay. There wasn't any workshop or any paper about data monetization, but there was something about code called data attribution. Okay.

Nikolaos Vasiloglou [00:54:04]: We all know that there's a ton of copyright lawsuits, all sorts of problems. What's going to happen? The famous Facebook meta case where they download it illegally pirate books to train the model. What are we doing with people Flag? If Genai is doing all that stuff, how are we going to live? How are we? Is it fair? So it was the first time that I saw papers that they can say, I'm generating this token, I'm generating this sentence. And I did that because of that page in my training data. And if that training page was created by Demetrios, then we'll say, let's go and give one penny to Demetrios for that or whatever, like 10% of what we're making from that. So that is huge for me. Because if we continue in that direction, we should be able to, basically, because right now you're gonna say, well, you know, I do that, but there's no way I can pay, you know, I found it on the web or something like that. So if we can have an attribution model, and especially which is like per token, like per sentence, let's say per word that I generate, then we can start paying off people, dividends.

Nikolaos Vasiloglou [00:55:20]: And these models are smart. It's not that they find the same thing. If you go in my presentation, they're just saying to the language model, I need to shut it down. And the language model complains, says, no, I still want to be alive. And they trace it back to a novel, a sci fi novel that was about a robot that was supposed to be sat down, that was actually written in a different language. But the language model was able to do that. So that's one thing. And the other thing that again, it's one of these things that we haven't seen that taking off, but I think it's going to take off.

Nikolaos Vasiloglou [00:55:52]: It's basically building smaller language model and putting them like Legos, composing them to build a bigger one. Because right now, one of the problems that language models are very monolithic. You train them, put everything. All this effort is millions of dollars to build one. You know, it's like as if you were writing code, you know, the whole Windows system in one file. So. But that's not, you know, like this composability. There's a competition, really great.

Nikolaos Vasiloglou [00:56:21]: And I'll leave you with that.

Demetrios Brinkmann [00:56:22]: Yet this isn't in your eyes, the directions that loras are trying to take.

Nikolaos Vasiloglou [00:56:29]: Well, loras are, you know, they still, you know, you still have the llama, which is, you know, 100 millions of dollars to chain, and you're modifying it. That's one way. But I think what people are trying to do is you can take, let me give you an example. You can take a language model that is specialized, very good at translating from, I don't know, Greek to English and another one from English to Italian and you can literally add them. And now you can translate from Greek to Italian so you can compose the tasks. So it's much faster than going and training a Lora, you know, so there are some pros and cons, but it's an interesting direction. I think it can also help when you are building the language models. Like we can say you train on this, I train on those.

Nikolaos Vasiloglou [00:57:14]: Let's put them together and build. You know, it's more about, you know, all of us working in a collaborative way. I.