AIAW Podcast
AIAW Podcast
E170 - Building ML Systems with Feature Store - Jim Dowling
Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.
In Episode 170 of the AIAW Podcast, we’re joined by Jim Dowling, CEO of Hopsworks, co-creator of featurestore.org, and author of the upcoming O’Reilly book Building Machine Learning Systems with a Feature Store. Known as "Mr. Feature Store," Jim walks us through the evolution of AI infrastructure. From traditional batch learning to real-time, agentic workflows powered by vector databases, RAG, and LLMs. We discuss how feature stores serve as the memory layer of AI agents, enabling contextual awareness and low-latency decision-making, and why they’re central to the future of scalable, ethical, and sovereign AI systems. Tune in for a deep dive into the systems powering tomorrow’s most advanced AI applications.
Follow us on youtube: https://www.youtube.com/@aiawpodcast
Irish Profile To AI Sovereignty
Jim DowlingThey were called the wild geese. So they started by saying, let's do this. And then the business editor said, hang on, this is relevant. This is about digital sovereignty, about AI. Let's make it a business profile piece. I mean, I've been covered in the Irish Times a bunch of times before because we get funding. And it's kind of one of these things. But do you have a family member there or contacts with some companies? No, I mean, well, we have some customers there. And like one, but we've one employee there. But otherwise, you know, we actually have initial funding round as well, partly from Ireland. Um, but um All right, we're ready, so we could.
Anders ArptegYeah, yeah, yeah.
Jim DowlingNo, but otherwise, like we're we're fully Swedish company, you know, and um Swedish-owned, Finnish-owned. Uh and then the Irish Times this journalist basically had listened to this interview and said, Hey, you know, I'd like to do this. And it got upgraded from like a profile piece for for kind of immigrants or emigrants to uh business piece. Um and then it went through another round of kind of uh discussions. But I think that you know there is an interest now in the kind of the story about digital sovereignty and and digital infrastructure and uh Europe really having some challenges and and and we we play a small part in the in that story company. What was the title of the article then? Um if you Google Jim Dowling Irish Times, you'll find it. I don't know what the title is. Um but it was a profile piece, I think. So I'm not sure what I think it was something about building the data for AI systems, so at scale, something like that. Big large volumes of data for AI is needed.
Anders ArptegBuilding a platform to allow companies to handle use much of data that goes through we goes with AI. Could that be it? It probably, yeah.
Jim DowlingIt's not not a very snappy title. No, is it? No, it wasn't.
Speaker 4There it is.
Jim DowlingIt's very very building a platform to allow companies to handle huge. I think it probably couldn't get anything snappier as an explanatory title. It should not be a good idea. Not something you would kind of like just say, oh, I want to read that.
Anders ArptegNo. Okay, so so what was the gist of the article you would say? What was the core messages you wanted to bring forward?
Jim DowlingYeah, I i it was kind of a partly my story. So, how did I go from research to um to building a company? And and it drew an analogy. So, my my supervisor in uh in Dublin, I did my PhD in Trinity College Dublin, he started a company called Iona Technologies, and they're really Ireland's first kind of really massive software company valued at billions in the at the 99 boom. I think they were right at five or six billion. So he was the first CTO, and and my old research group left on mass. They all left to start this company. Company was massively successful, the boom happened, the boom crashed. They were they were actually the people who who who invented Corba. If you remember, there's a standard called Corba for distributed computing.
Anders ArptegOh, yeah, yeah, before the Chrome kind of components thing.
Jim DowlingSo yeah, so Corba, they were Corba, they were the Korba company globally. And um then they got hit by web services and they got sold, I think, for a few hundred million. But um, but you know, a lot of companies spawned out of that.
Speaker 2Yeah.
Jim DowlingUm, so uh, you know, I I told the story of of Hopsworks, our company, which is very similar. We came from a research lab. Um, we kind of had a more of a background in MySQL because the product is built on a uh fork of an open source database from MySQL. Um, but we did, we were a research lab, and our our our journey was uh very similar to Iona technologies and similar to, for example, Databricks who Ali Godzi uh left, where you you you build very deep tech um uh software in a research environment because business won't necessarily take the high risk in in building something that has a maybe a lower chance of of paying off. And and then I think you know, so the story is partly about that, but then it's it goes in to talk a lot about digital sovereignty and some of the challenges that we have today and and what we're doing as a company, so we're a European AI uh a company for building and operating AI systems at scale, and and there's not many of us in Europe. So it's kind of unique as well. And I think, you know, given that there's a lot of um anguish at the moment about the fact that we don't have our own digital infrastructure and AI, we don't have our AI infrastructure either. That, you know, are we gonna build it or are we gonna continue to rent? I think that's the question.
Anders ArptegYeah, I mean it's it's one of my favorite topics as well. So let's get back to that a bit later, perhaps, in the podcast. And I'd love to hear your thoughts how we really can big uh build digital sovereignty in in Europe, because I think it's harder than people think. Um but why is that? And that would be super fun to just talk more about. And I and I guess we also should be uh rather thankful for Trump in this case because it's uh really boomed, I guess, the interest then for the actually building sovereignty in in Europe. And for your company, it must be in some way positive,
Research Roots And IONA Legacy
Anders Arptegright?
Jim DowlingI don't I don't like the word thankful, but like you know, it's um uh I mean it it it it's a situation we're in, and uh yeah. I think most companies are are here to kind of fill needs, fulfill needs, you know. So you might have a need for software to help people more productive or to save costs. And in some cases, those needs become weird needs, and that's kind of where we are today.
Anders ArptegI guess we all certainly want digital sovereignty for one reason or the other, and anything we can do, and I think also what you're doing and have been doing for your company is certainly something that truly tries to make that happen. So I'm very thankful for that, and I'm very grateful for you coming here again, Jim Dowling. You're also one of my heroes. I think you know, for your knowledge that you have, but also done a similar journey as I have from academia into more industry uh use cases, uh is something I really respect. And I think you have so much detailed knowledge about so many questions, and especially these kind of cloud and sovereignty questions that um I love to sit down and discuss more about that with you, but also because you actually are releasing a new book, right? Released. It's uh it's done, it's out there.
Jim DowlingCongrats on that. Yeah, thanks. It was a big I always wanted to write a book, it just never got round to it. It's one of these things, and then eventually did get round to it.
Anders ArptegWell, let's uh let's go more into the book shortly, but perhaps you can give just a quick background to really who is John Showdown. Yeah, it's right. That's the book. Perfect. Okay, yes, now it show it now.
Jim DowlingThat's the book. It's an O'Reilly. Uh it's published by O'Reilly. It's called Building Machine Learning Systems with a Feature Store, and it goes from batch to real-time to LLM genetic systems.
Anders ArptegNot the most sexy story, but I think it's very sexy content. So let's get back to that shortly. But please, who is Jim Delling?
Jim DowlingI mean, I'm uh I'm a computer scientist if we talk at the at the technical level. Um I did uh I I come from Ireland originally, did my undergrad and PhD in Dublin. I've lived in Germany for a couple of years as well. And then I came to Sweden 20 years ago, worked for MySQL for a couple of years. Yeah, and that's kind of the genesis of Hopsworks, because we we forked an open source database called NDB Cluster that I'd worked on. And um that was the core of our kind of scalable data platform. And when we started the company, um we left it, so I I you know I went to research to I got actually a Marie Curie Inter-European scholarship, which was nice. It meant I could get back into research. Um and I did that at at what was called six at the time, um, which I stayed at Rice for for many years. Um and I also then got a position at KTH. I was an associate professor at KTH for I don't know, 11, 12 years, maybe.
Speaker 2Oh, that's true.
Jim DowlingSo finished officially last year, but I'm guest lecturing at the moment, of course. So it's I'm still involved.
Anders ArptegAnd and without trying to get in too many rabbit holes here, but but I have my thoughts about you know how academia and research works today. And our research, of course, can happen in industry as well. But but if we just focus on academia and KTH, KTH is a good example, I think. But still, if you look at academia in general, what what do you think, perhaps specifically in AI, how the academia works today?
Jim DowlingWell, that's a hard question. I mean, AI, I mean, like if you look at the at the if you take a step back, like I published an ICML 20 years ago, right? And you look at ICML, you look at at some of these conferences now. UmRips, I think is the current name. Yeah. And um, you know, and and an ICLR. And they're they're just insane. These are like uh, I don't know, it's like Woodstock or something, isn't it? I think you know, the numbers of people coming. So obviously, academia as a whole has taken a very large interest in in fun in foundational AI. And you and I remember I did my PhD in reinforcement learning. Yeah, me as well. Which is insane, and and there was no interest, you know. So um at the time. So now there's huge interest in AI. I think a lot of academia has switched AI. That's the big thing that I see.
Anders ArptegBut still, you know, of course, these kind of a few selected conferences is super big, of course. But I think, you know, isn't most research would call it that not published at all in conferences these days and just put on archive, right? That's a good point.
Jim DowlingI mean, it's things move so insanely fast. Um I think when I talk to my students before, I always talked about impact rather than just number of citations and so on. And impact can come in many forms. Yeah. You know, it could be a library that you build that gets um adoption. If you want to get a PhD, you do have to have a research publication or two. But if you want to build a career, um, impact is is important. That career could be in research or it could be in industry. So, what did you choose to go into actually pursuing a PhD at that time? I think I was intellectually curious, really. I wasn't, I didn't do it for the money. I mean, it was horrendous money. I mean, I lived on you know, this was a long time ago. I I had lived on the breadline, like, you know. Um I was intellectually curious. I thought, you know, I wanted to dig deeper. I was interested in in computers. And you know, I did my my my bachelor's uh my final project was in AI, but like uh I like distributed systems. I went into distributed systems and then I did AI in my distributed systems PhD. But do you do a PhD today if you were at that age? Oh, that's uh you know what the diff biggest difference is um there's a pyramid in academia, which is you know, you've got PhDs and then you've got postdocs, and then you've got uh full-time academics at the top. And I think when I finished it was a it was a much narrower pyramid. Now the base of the pyramid is so wide that and and this was actually partly true in my old research group. Um we had, can you believe this? In our research group in Dublin, we had like 40 PhDs by the time I I was I was a lecturer at that point. So I was a lecturer in my hometown, I was nice and sorted. Um, but the um there were so many PhDs they couldn't have any ambition to continue in in academia to get a lecturing job or something. So they all started doing startups during their PhDs. Oh during the PhDs. Oh, yeah, they're all talking, oh yeah, and a lot of lot of a lot of high value startups came out of that group a lot.
Anders ArptegSo, okay, so if I phrased a question like this, uh imagine your kids at some point uh are thinking the question, would you recommend them to do so? I mean, it'd be hypocritical if I said don't do it, wouldn't it? I mean No, it's not. I mean, times are changing.
Jim DowlingIt could be so I mean, like, you know, would you recommend your I have my my my my eldest son is 16 and he's just gone to gymnasium in Sweden. And um, you know, we're he's asking for advice. What line would you choose? Like so yeah, in Sweden you you specialize somewhat at that age, and he's doing technique uh vetenskap, which is um sort of theoretical computers, is kind of what what his old man did like. Um, so he you know, will computer industry be the same in a few years? I I will there be as many jobs? We can already see a sort of fall-off in jobs, and then you're wondering how much will this continue? I think there's certain areas we can see already, you know, who's changed this more than anyone we already mentioned, lovable. So web designers, um, we can look at um, you know, jobs like graphic designers, they've disappeared a lot thanks to Gen AI. Um, I can imagine in the very near future, data analysts is a job that will be um it'll be revolutionized. So there's going to be a lot of these kind of jobs that there have been a lot of job
Founding Hopsworks And Vision
Jim Dowlingum positions where will disappear. Will there be new jobs created? I think it's a great question. Probably.
Anders ArptegBut I think this is a question that a lot of people are thinking about, and and you know, people are trying to think how they can help their kids with good advice. And I can at least see two extremes here or two alternatives that I don't know which one is is the proper one. But one could be that because of AI is happening and it's providing all this kind of very you know wide breadth of knowledge that you can have, um that you can think that humans do something else and perhaps they give the context because they do the reasoning, perhaps they have the innovation and and power that that AI do not have. So what's happening really here is you let AI, especially when it comes more to agentic work, do the more detailed work. And what humans need to do then potentially is to widen up to become more of a generate, perhaps understanding the the STEM field, so to speak. So we're steering. Yes. So we are more in control of them, but then we need to understand mere the basics, of course, but perhaps be a bit more general. But I heard the other way, yeah, and the other extreme could be that uh companies do not hire like grad or people right out of universities, and it's because that's something that AI can do well. So what they're really looking for is more specialized people, people that are domain experts in some field. So what you really should do then, then, uh if you believe in that future, would be that you might focus on becoming experts in you know medical diagnosis or whatever. And doing a PhD.
Jim DowlingI think I think it's an interesting point. My my take is that I still think we're quite early on the journey to AGI. I don't think we're quite there yet. And the one thing I would like to do if I was new is learn as much about this as possible. So the one thing I I've learned from my years in computers is um my sister's an architect, right? And if you're an architect, um, you have to understand the property of all the materials you're using to build the houses that you're building. Otherwise, if the house collapses, you're in trouble. Right. And we have had a term called computer architect, but in in my experience, because compute computer science is based a lot on abstraction, layering on top of layering, and you know, this assumption that, well, you don't need to peel but peel behind that abstraction, you can just work at your layer, it never really holds true, right? So to be really an expert in your field, you need to understand as many of the layers as you can, as far down the stack. And you know, in my company, I I said I was at MySQL. We have Michael Ronström, who was the top guy in MySQL. He he's found bugs in CPUs. Right. So if you go down the level, you know, the core of our database, there's assembly in there. Yeah. And then you're actually then stepping through debugging on assembler instruction.
Anders ArptegAnd I guess there will be some very few jobs and very specialized people that can.
Jim DowlingYeah, but I you know, I think just for intellect, I you know, AI is still so early that I think understanding like if we talk, you you mentioned that you provide the context and so on, but like this is something I'm working on quite a lot. The the amount of the how you provide it, where you provide it, where it comes from, how you process it, how you acquire it, retrieve it, um, query it, how the format, all of this is important. So, you know, if you go in with an education about this, you'll have some skills that will be very valuable for being able to build the next generation AI. So I think I wouldn't be negative on computer science as a field at the moment. I think it's still going to grow because we're becoming more digitalized. Um, but I think a lot of jobs will disappear. But I'm not sure which ones. That's the question. That's the hard hard question. But you're still also partly working in academia, right?
Anders ArptegNo, I okay.
Jim DowlingNo, I'm I I said it was guest lecturing KTH. That's what I'm doing at the moment. I have a course I'm guest lecturing on at the moment. The book is the course material. Yeah, yeah. But uh, I'm not I'm not employed now.
Anders ArptegOkay, so you've gone completely to industry, so to speak, in that aspect. Right.
Jim DowlingI mean, you can only do it for so many years, you know, that you can try and try and keep both going, but it's not Jan Le Kun tried, right?
Anders ArptegBut now we'll see what happens. But uh in Benjamin, that's tried it. Anyway, okay. Uh and then you've been working now uh with Hop's work. I mean it started off in KTH years, I guess. How how long has Hopsworks been around?
Jim DowlingSo we we actually left and and and took in external venture capital money seven years ago now. Yeah, seven years ago. It's quite a while. Um I would say time goes fast, but maybe I've aged more than that, seven years, I'm not sure. Yeah. Um, but yeah, I mean it it it flies along.
Anders ArptegAnd just to give a super quick uh pitch of what hops works do, how would you phrase the mission of the company?
Jim DowlingI mean, if I'm talking to somebody who knows computers and so on, I'll say, hey, it's Databricks on-prem, on-premise.
Anders ArptegI haven't heard that before.
Jim DowlingYeah, well, I mean, it because it's easier for people to visualize. Like so, I'm I, you know, we we run, we build a whole platform on Kubernetes, so we can deploy in the cloud or on-premise. But just to explain, people don't know what Databricks is. Um we're a platform for if you want to develop AI applications, if you want to operate AI applications, and then how do you get your data to your AI applications and at scale? So if you're a very large company, let's say you're Ericsson or Saab or if you're Salando or Patty Power, um, so these are all kind of global brands, um, then we we provide the software that enables them to build AI systems uh and be more productive in doing so. So get more models built and deployed faster and get the data from where the data is siloed in these organizations into a central kind of place where then you can actually build AI using all of the enterprise data, not just small parts uh from different uh verticals or or business units.
Anders ArptegI think you phrased it nicely as well, and then you called both the development aspect and operations aspect uh of AI applications to call it that.
Jim DowlingUm I use the term AI systems more, all right? Because the system incorporates the the application plus the plumbing and infrastructure underneath it. Because it kind of is a system and an application we think of as at a high level as you know, it has a maybe a user interface and and and maybe it stores some data in a database. But but AI systems actually need a lot more plumbing, right? So they need like uh data to be pulled in from different sources, they need models and and so on.
Anders ArptegAnd I'm sure since we are going to speak about the book shortly, that we'll hear much more about what that really entails. Uh okay, so that's the majority, I guess, of your work today. Uh and and you're the CEO of uh CEO co-founder, yeah. Cool. Okay, but with that, uh, let's go into the book. And um can you perhaps start just a bit about you know how how did you come up to with the idea to get started writing a book, which is super tedious. I've done that partly, but not the first one.
Jim DowlingThis this comes from some the the company we we were kind of um category creators in a space called feature store.
Speaker 2Yeah.
Jim DowlingAnd the feature store is uh it's a data layer for for managing data for training and and um inference in AI systems. So I you know I originally talked to Manning about doing a book, and then that didn't really work out.
Anders ArptegManning and that's it.
Jim DowlingManning is a publisher, they're a publisher. They're they're quite well known in computer science. Um, and then someone on our side said, let's approach O'Reilly. So we talked to O'Reilly and they said this is really good. And you have the conference. So we have kind of a natural base. So the conference is about 4,000 people register every year. There's kind of this natural community, and that's
Academia, Careers, And AI Jobs
Jim Dowlingattractive to publishers because you have a natural place for people to come in and and and have interest in the book. So they like the the feature store element of it. And this was about two years ago or something. Yeah, it started, project started two years ago. Um and uh took about 20 months to write, I guess, which is uh probably longer than I thought at the beginning.
Speaker 2Yeah.
Jim DowlingBut I like I didn't like it, let it take over my life. I mean, one of the things about writing a book is um it particularly in AI, you you it becomes it it an AI book is not going to be fresh forever. So um it needs to be built on some sort of fundamental principles if you want it to last.
Anders ArptegAnd just to continue on that topic, you know, a book, of course, is a big thing, takes a lot of time, and it it also gets perhaps a bit outdated in a rather short period of time. But what is the reason that you wanted to really get started writing the book?
Jim DowlingUm I I've been um advocating a certain way of building out systems. It started at the university. Um that that we that that I said, okay, this and I think the course, yeah, ID223 at KTH, I think it was the first course globally where students built AI systems. Everyone was just training models, doing inference, but we've been they've been building AI systems for four years maybe now. And um an AI- That's such a great thing, by the way.
Anders ArptegI wish more universities were doing this.
Jim DowlingYeah, and and like and my my my bug bear at the time was Kaggle, right? I said, look, Kaggle will not help you build an AI system because any AI system that, if it's using Kaggle, you train on the model and then you have some static test data, you make a prediction once and you're finished. An AI system that makes one prediction is not creating a huge amount of value. Maybe it does create some, but but most AI systems will continue to make predictions because new data will continue to come in. So the idea at the beginning was that okay, everyone's gonna design a unique AI system. You're gonna pick a unique data set and define the prediction problem for that data set, and you're gonna build an AI system. And that's actually harder than you think, right? I mean, it's kind of but what what I did in the course, I mean, with the company, I would say originally was um came up with a very simple architecture for decomposing any AI system. So if you have a batch AI system which makes predictions on a schedule, so for example, Spotify Weekly will give you a recommendation of songs once a week, that's a batch AI system. Or if you have a real-time AI system, so imagine you're a Zolando and you're browsing the website, it's going to recommend clothes for you based on what you've just looked at and what you've just done. Uh, that's real-time AI. And then you've got, of course, nowadays we have agentic systems which use LLMs and they may use context data to help make better, better responses, provide better responses. So for all of these systems, there's a very natural decomposition, which is that you have raw data which comes in, and that data needs to be transformed into input data to the models, both for training and inference if you're doing training. We call those feature pipelines. So we know in machine learning a feature is the input to a model. It's typically some measurable quantity. Um, if you're doing large language models, we could call those context pipelines if you prefer that terminology, but it's still input data into the model. Okay. But there could be data pipelines. It can be structured, unstructured. The point is that there's data that is required that is external to your system, and that data needs to be transformed into some format which can be input to the model. And often we do things like I'll give you an example, feature pipelines, context pipelines. What do we do? We we do compression. We have maybe some raw data out there, and we want to compress it into a signal. We want to do that for features and context. We don't want redundant data coming in. We want to have relevant features or relevant context. Um, we may have features that you're not allowed to use, so maybe gender or something like that. And you have context you're not allowed to use. So you're prohibited from using something or other. So there are a lot of analogies between just taking data and preparing it in such a way that it can be used by models for both training and for inference. So that's the first part of NEA system, is these feature pipelines. And the second part is if you're training a model, you take the features that were created in the first pipeline, you train a model, and then you save the model. So pipelines, just to for people that don't know what a pipeline is, a pipeline is a it's a program which uh has an interface, it has some data that it brings in and has an interface as output, a schema that it outputs data as. So typically these pipelines in AI systems are have data frames as outputs, so they'll write out a pandas, a polars, a spark data frame, or even Flink. You can have like data sets which are like data frames. Um, training pipelines take these data frames in, which will be selections of features and output models. So they have very well-defined interfaces. And the last pipeline is an inference pipeline. So inference and machine learning means making predictions. So to make a prediction, your input will be the features, the model itself, or one or more models, and then the the output of predictions. And this is true whether it's a batch, AI system, or a real-time or or an agent. Um they're all going to take an input in and they'll put uh some output cut out.
Anders ArptegAnd of course, you have the operations side as well, we should speak a lot about in the book, but um, but yeah, this is at least the AI part, right?
Jim DowlingSo this was my motivation that we had this very clear way of decomposing any A system. Um and then the students, for example, they can easily talk to each other. You do the feature pipeline, you do the UI and the interface, because the inference will typically have a UI. You want to have some way to consume the predictions, otherwise, your A system is not worth much if it's not um producing predictions that are useful. So you can decompose work quite easily, and then uh one of the problems when you decompose things in computer science, when we modularize, is how do we compose them together into a system? So this is where the feature store makes it easy because there's these natural interfaces. Data frames go in, data frames go out, models go in, models go out.
Anders ArptegSo I get that, but but just to I mean, I think in some way you can say some things are lacking here. I know you don't think so and and have the right view on this, but if people were listening to this and and thinking that you're not considering all the necessary framework necessary to do the proper operation of an application as a whole, make sure that it's up and running, make sure that you have the proper latency, make sure that you know have security issues or whatnot. I mean, of course, you have a big operation side that requires a lot of work, sure, which is not perhaps AI specific, it's more like general application specific as well, right?
Jim DowlingYeah, the book actually covers that in chapters call we call ML ops. So I I so people use the term ML ops and LLM ops. And I actually didn't use those terms for the chapters because the way I see it is you can kind of more easily understand the operational requirements as being offline testing requirements. So when you're developing, you'd like to run tests. So they might be unit tests or integration tests, um, you know, model bias tests, feature pipeline tests, inference pipeline tests. And then when you have an operational system, you have monitoring, observability are key properties that you want to have in any AI system. So that's kind of a uh so I don't use the term ML ops, I split it up into the runtime or operational um uh aspects and then the offline aspects of testing before you deploy. Um and yeah, of course, that's key to everything. But I think I think like in terms of how we we build AI systems, a lot of people have a challenge in just getting to a working small AI system. And that's the goal of this book in particular is how do I get to a working AI system as fast as possible with as simple a mental model as possible, which is where the feature pipelines, training pipelines, inference pipelines come in. I give you a mental model, you can now decompose the problem of building your A system into these three three main components, and then from there you'll get a minimal viable, I call this term minimal viable prediction service. So you probably know the term, um, and minimal viable products. But AI systems are prediction services. So I call it a minimal viable prediction service. And from there, you then of course add testing and uh operational concerns and layer those on over time.
Anders ArptegCool. So if I understand correctly, I mean that was the reason that you took on the big investment in time and effort that you have to spend to write the book to give your kind of mental model in how to decompose these kind of systems and you know bring a piece of knowledge, right?
Jim DowlingYeah, if you read the intro of the book, it says this is the book I wanted for my course, ID223, which is true as well. Which you use it for the course. Yes, use it for the course now. Yeah, yeah. Yeah.
Anders ArptegYeah, I mean cool. And and uh then if other people are interested in uh writing a book as well, perhaps you can give some kind of advice. You know, how do you approach this? I mean, you're a famous person, you have a name, and that means that you can speak to publishers like O'Reilly and they they will get interested, so to speak. But how how do you go about you know finding a publisher or get started to at least have you know some way to uh get started with a work or writing a book?
Jim DowlingYeah, I I can tell about it. Like for me, I I I wrote quite a lot in the space, and particularly for feature stores. I did a lot of content, um, wrote a lot of blog posts, gave a lot of talks, and I think, you know, and then I organized this conference, the feature store summit conference, which is relatively high profile in that small space. Um, I think you know, one of the issues with a lot of public there are different publishers. So there are some publishers where it's a bit easier to get in, and there are some that are a little bit harder. And I think uh many publishers don't invest massively in marketing. It's easier if you can have somebody who can market your book because they're they have a social media profile. Um so it saves them, you know, going out and marketing your book as such. Um so I think you know, if you're if you want to just write a book and get out as fast as possible, there are publishers who uh, if you have good content and it's in a relevant space, will will listen to you straight off the bat. Like packed. I know a guy, um uh Paul Azustin's from Romania, and he you know, he's quite big on LinkedIn now, but he he wasn't when he wrote this book, and he got it in because it was about writing LMs and agents. Um they they they said, fine, great, let's let's publish it. And um very successful book because he was first out. So, you know, you you don't you don't have to have like that kind of deep background in in research or as an influencer if you have good content and you're you're very timely with it. Um but I mean in general, I think I've been writing for most of my adult life. So for me, writing is kind of just a natural skill that I developed through academia and and so on. Um you know, if you're if you're not as as comfortable with writing, then you should take it as a big project.
The Book: Feature Stores Explained
Jim DowlingAnd kind of dedicate. I didn't dedicate, you know, I kind of fitted my life around writing the book rather than fitted my life around writing the book, you know, the other way around. So keep uh the work-life balance, so to speak. Uh I live in Sweden. I try to. I wouldn't say, you know, my wife wouldn't maybe agree with me on this, but you know.
Anders ArptegIf I phrase a question like this, is there anything in terms of how you wrote the book that you would do differently if you were to write uh another book or a follow-up for this?
Jim DowlingSo I I know a guy, um professor in the Netherlands, Martin van Steenen, he wrote a book in distributed systems, which is used by most courses around the world. Uh it's not AI, but it's he he basically told me um, I have a I I wrote my book and it took me uh 1,327 hours. And I'm like, what are you talking about? And he said, Well, I had you know booked in my calendar these three hours every day, and I count the number of days it took, and it took 22 months, and and I know exactly how much time it took. I didn't have the luxury of being able to do that because you know, we get customer meetings, I get uh things come in all the time. You know, the CO job is kind of the job of last resort when there's issues, they come to you and you just have to react. So um, I've you know I've kind of fitted the writing around um my availability for doing it. But if i if I could do it in a structured way, that would be the way I would do it. You know, just assign time and say, right, two hours every day and have that discipline of sitting down and and cranking it out. Um that would be the best way of writing a book.
Anders ArptegI will certainly have a look at this, and I think a lot of people need to learn so much more about build their proper prediction services, we call it that, and not just you know building a model or a prototype and believing that's the the end goal in itself, which so many people seem to do. Right. I I would like to just go into one of the terms or the terminology about you know feature stores, because uh I think that is something that has partly confused me a bit in the past, and I think could confuse other people. And and there is a lot of similar terms, of course. Uh you could have data pipelines, you call it feature pipelines and feature stores. Could you just try to give us your view of what do these terms really mean?
Jim DowlingYeah, I think um so the the the feature store is a concept, it wasn't me personally. So Uber were the first people to write about their data platform to support their AI systems, and they gave the term feature store to it. So that's where the term came from. And this is 10 years ago? Or this is 2017. Okay. End of 2017. So they their platform is called Michelangelo. And um, as a data platform, it's it's unusual because you're gonna have a large volume of data, historical data, that you need to collect because you're gonna train models with it. But they also have real-time systems. So when you're on Uber and you want to use the app, and when drivers get you know um pointed at certain um passengers to collect, they models need access to real-time data. So they're gonna look, they need data for things like uh activity levels in a given area, reliability to the driver, requirements on the person who's ordering, and you need to match all this up. And and people don't write if-then rules for it anymore. We we train models to try and match those drivers and um passengers up. So the platform um has low latency real-time data you need access to, but also needs large volumes of historical data. So, you know, when I worked at MySQL 25 years ago, we had relational databases and that was it. But since then, we've had the emergence of data warehouses and what we call columnar data stores. So that's Databricks, Snowflake, um, what we call the lake house now. So in Uber's case, they stored large volumes of columnar data for the for the training data for the to train models with, and then the real-time data in uh what we call um a row-oriented database, so a low latency database. Because you know, if you go to Snowflake and say, hey, I want to get this uh information about um how many drivers this are how many passengers this driver's picked up in the last week, it'll take about a second to get it back. And you know, at Salando we have some two millisecond uh requirements. It's kind of you know very low latency. So so so the data platform you need to support operational AI is is complex. It needs this columnar store for large volumes of historical data and a row-oriented store for for real-time data. And we actually had this in our platform when we were um uh when we started and left KDH. We we our our goal was to build a scalable data science platform. And then we knew about Uber's um Michelangelo platform. We said, well, that's the kind of platform we need to provide because Michelangelo was not a public platform. So I think you know, when you're a developer, I can give you two different databases. Uh let me give you an example from this week, a very topical one. Okay. Um, this week Cloudflare went uh was down for 18 hours. Most of the internet was down. They the reason it was down is because they built their own feature store.
Speaker 2I'll I'll try and explain what I'm saying, right?
Jim DowlingSo they have a machine learning model that's deployed to all of these edge nodes, and it takes as input features. We said already features are the input to the models. So some of these features are pre-computed. So they're there may be global things around activity globally. I'm not sure what the features are, they didn't tell us. But it's uh pieces of information that they use as input to the models. And then when the models are deployed at the edge, they'll get local data from the edge as well. And they'll combine both of those to make predictions about things like DOS attacks and so on. So what they have is they have a big database called Click House, which is a column store. And um, so they're storing these features in Click Out. And what you have then again, as you said, is you'll have that they might use the term data pipelines. So we're computing features in our data pipelines, we're storing them in ClickOuse, and then we need to materialize those features, meaning copy them from ClickOuse somewhere. So often materializing features you do when you're creating training data, you'll say, I've got all my features here, and I I want to get last week's training data. So you go to the feature store, it'll give you the training data for last week, maybe as files or maybe as data frames, and then you go and train your model. So, what a feature store does is it there's certain things that are hard to do when you're materializing data from many different tables. So, firstly, Clickhouse is a tabular data store, it's not just a columnar data store, it stores data in tables. Um, now it what what they did was they had a they wrote they wrote their own feature store. So they had a query which would um read the data from this table and store it as features for the model to pick up.
Speaker 2Right.
Jim DowlingAnd that query was buggy. Um, I think it might have been dependent on some permission or access control that got changed. I'm not sure if it was because of a configuration change they made or whether it was an upgrade in ClickHouse. But basically it wasn't tested, right? It was just something they kind of hacked together and it resulted in duplicate data in the feature file. So we actually had a debate internally this week. We we we have different lakehouse file formats. We wouldn't call Apache hoodie. It prevents duplicate rows. And we also support uh Delta Lake, which is another Lakehouse file format, and it doesn't prevent duplicate rows. It has a primary key, but it doesn't enforce uniqueness. Hoodie does because an index for it. So um somebody we talked about this internally and saying, well, should we not do this? We shouldn't, because you know, we've seen this problem happen before. We saw it happen before customers. Um and you know, it this change needs to happen in the Lake Has file format, and we haven't done it. And it's not it's not a proposal yet. So we say, you know, if you if you don't want to prevent duplicate rows, just use hoodie. So, you know, a feature store will will ensure these guarantees that things like when you materialize feature data, that it you don't have duplicate rows, and so on. And uh um, you know, click or the clickhouse database didn't do anything. I mean, they wrote some code themselves. And then um when this code got distributed, uh it caused all their systems to crash. So the feature source provide a bunch of guarantees that a normal data warehouse wouldn't provide, and they also provide real-time data. So I'll give you a quick example of one. In the book, we have weather data and we have air quality data in two different tables. I want to build a machine learning model to predict air quality. So I have air quality observations, and there's sensors here in this part of town. Um, and you can find the timestamp of the observation, and then I can get the weather from a different source, uh, and I can put it in different tables. Now the timestamps will never line up exactly. It's time series data. So you have a problem of oh, if I get this air quality observation, what was the weather at exactly that time? So in database terminology, it's something called an as-of join. It's a it's a temporal join, and many databases don't support it. Actually, most don't support it. So if you use Databricks, there's no support. Um, and we built in support for that type of query. So you get uh what we call point-in-time correct training data. It you know, it doesn't have um stale feature data, it doesn't have future data leakage, and it's important when you're training correct models. So if it's more likely you have some flexibility in the exact timing so you can still match up things, or well, I mean the the rule is the following if I have an observation of air quality at this point in time, I take the closest weather observation, but before it, not future value. Right.
Speaker 2Okay.
Jim DowlingYou know, as it maybe it's not so important in um air quality prediction, it could be a little bit before or after, but if you're doing credit card fraud prediction, you don't really want to know be based on a future usage of your card, because then you're going to predict on something that happens in the future, which is is not what we want to do in machine learning.
Anders ArptegOkay, so it's uh doing I mean it it's simply that's harder than people think to build high-quality feature stores, if we call it that.
Jim DowlingSo if you do that, that's that's that's one one aspect. We'll get into another one, right? So when I want to ingest data, let's say take the credit card for example, it's in the book as well. So we have credit card transactions arriving, and typically they'll arrive on an event platform like Kafka or Kinesis or somewhere like that. Um, what we'd like to do is we'd like to be able to understand activity levels on your credit card because machine learning models will use those as features to predict things like a chain attack. Somebody, somebody manages to find your card or steal it, and they go run around town and they start using it at lots of small payments. All right, not good. Not doing it, but being victim, yes. Sure. And of course, we you know with geographic tax where your card is used at some infeasibly far away uh location within a short period of time. Um these are kind of what we call features that help predict fraud. So you have all this data coming in, and you'd like to do it in real time. So what you what you can do is you can say, okay, um, I'm gonna compute these features, and we can call them, let's call them counts over windows of time, right? Or sums of money spent over windows of time. And those features we want to have as input to the model, and we want them to be what we say as fresh as possible. We don't want them to be half an hour old, we want to be two hours old, three hours old. We want to compute them as uh as you use your credit card to basically know exactly what's happened. So feature stores provide the infrastructure to do that computation and to serve those features at low latency.
Anders ArptegSo you keep being old all the time the freshest, so to speak.
Jim DowlingAnd this kind of logic that you date. I think the the most famous one is TikTok. Right? So have you heard the term TikTok um digital crack?
Speaker 2No.
Jim DowlingNo? So you know it's so addictive. Yeah, I think Andre Kaparthy said that. He called it digital crack. Because it's so the the recommendation algorithm is is so good and so reactive for many people. It's like it feeds dopamine. Like basically, when you click in TikTok or swipe or share or like, every event you do, every every action you take is an event that gets sent to a very large Kafka cluster, and then those events are processed by an even larger Flink cluster with tens of thousands of servers. All right, they're using are using Flink in TikTok. It's Flink and uh Alibaba, didn't they buy they did? They bought the company um Ververica behind Flink from Berlin. That was a classic. We won't we need to get into that, but that was Europe losing. So it's just geostrategically, a little side side discussion briefly, but um uh at the time Alibaba couldn't build that technology. So um, but
Why Feature Stores Matter
Jim DowlingChina have now built on Flink. All of Alibaba and Huawei and WeChat and ByteDance are running on Flink, so the Americans can't shut them down. So but it's European technology. Yeah, anyway. We should be proud of that. And it's from KTH originally. The original streaming engine was done at Rice and actually Rice and KTH. So Flink is cool. Um Paris Carboni and ULEN Safer ED. Yeah. Ah, awesome. Okay, so so yeah, so so back to the point, TikTok. Um, within a second or two of you taking an action, it will make very good predictions about what if you're getting bored or if you want to get more interested. And that's all powered by this feature store technology.
Anders ArptegCool. Good. And I think that could be a good reason to dig more into the book to truly understand what the feature store is, because I don't think it is a really well-known concept, right? And I know you also are speaking a bit about more agentic workflows, and of course, it's a super popular term these days. Everyone is speaking about that, and I think a lot of people are abusing the term agents and agentic workflows in extreme these days. But still, perhaps you can get a bit more into how feature stores are related a bit more to agentic workflows.
Jim DowlingYeah, sure. Um so agents, as we know, um basically they're they're they're think of it as being you have an LM as an interface. It takes in a request and returns a response. Um, but whenever you want to uh to an agent to perform some task, it could be a chatbot, that's what we're everyone's familiar with. That's the typical task we have. But you know, we see Clarina here in Stockholm have agents to help them respond to customer support requests. So if we take that Clarin example rather than the chatbot example, we say, okay, um, somebody goes to Clarna and says, you know, uh, there's a problem, you you you're charging me too much for this thing I paid for on order, and I didn't, I missed my payment. But you know, so for the agent to answer that question, the LLM doesn't know anything about your private payments. So there's a lot of private data that uh Klarna have that the agent doesn't know about. Some of that data will be policy documents and PDFs, and many people know about things like RAG. So you would say, okay, let's chunk the policy documents about you know payments and late payments and cancellation policies and terms of payment. Um, some of the data will not be in vector databases and probably will never be there. Um, like your orders, the transactions you take, and your history with that data is stored in operational databases. A lot of it is in data warehouses, it's in tabular data systems, and that data will not, I don't think, end up in uh probabilistic search engines, which is what vector databases are. So um, and then so so the LM has a has uh has some private data can't access. Some of that private data will be kind of static, but some of it will be very, very recent. You know, so maybe I just performed an action and I went, oh no, that was terrible. And I called Clarona within you know 30 seconds or a minute. This is where we get into feature stores now. We're into the TikTok style case. So when you take actions, that data then gets uh fed into your system. Maybe it arrives at a Kafka cluster and spreads out to the data warehouse and the feature store. Um when you ask, uh, and and this is the what this is what's covered in the book, right? Um many people think uh think of agents as kind of like LLMs, where you have the interface to an agent is you send it a string and it gives you a response. That's kind of the assumption, right? You've got string in, string out, because that's what an LLM has. LM is tokens to be more specific, but but but agents can actually have an interface, a schema, an API. So if I have an application which is a Klarna help desk, you're probably logged in, right? Um so you probably have a user identity, you may have a um a problem and an uh an ID for the transaction, you may have an order number that that you made that you can quote, or you don't even need to provide it because the application knows it. You clicked on it, you say, I'm gonna complain about this. So it picks up the order ID, it knows your ID, it knows the transaction ID. So the agent can be sent these IDs along with your query.
Speaker 2Yeah.
Jim DowlingAnd then at the back end, the feature store pulls out the data related to those IDs and injects it into the context, the context window as as context. So that's the role of feature stores at the moment for agents. They're prov they can provide real-time context to um agenc workflows.
Anders ArptegUm good. Um okay, so we can mix the private data, something coming coming from a Kafka stream and whatnot, and then also fetch data from feature stores and combine that into um a recent and relevant context to the agent.
Jim DowlingYeah. I mean, like it just for the listeners, again, if you take a little step back, I mean, I have a the second lab in my course in the course this week is about um fine-tuning LLMs, and people kind of wonder how do fine-tuning relate to uh context and so on. I mean, LLMs have a cutoff date and they have certain data. And to incorporate data outside of that training data set, uh private data, let's call it, you can you can fine-tune. You can collect your train data, fine-tune the LM, and you know, add uh something like uh LoRa adapter to enable to answer questions on that private data. But but again, like there's gonna be a cutoff time for the fine-tuning. So everything that arrives after that it will have to be context. And then the context is gonna be the LM can use it to perform something called in-context learning. Yeah. I hate that term, by the way, but okay. But it is that this is um it's covered in the book. Like, so in-context learning is the next branch of machine learning.
Anders ArptegOkay, let since you brought it up, um let's discuss it a bit more. But because the reason I hate it is I agree with you know what you are what people are trying to say, man. It's a well-used term, of course, but um it it's um for me at least learning and adaptation is two rather well-defined terms. And learning means in some way that you actually change the behavior in a way that can be reused at some later point. You have to build experience in some way. Um if you just have in-context learning, you're not really changing the parameters or changing anything that can be reused afterwards, right?
Jim DowlingI mean, that's one definition of learning. Another definition of learning would be that I can give um, I can take supervised learning. I can say, um, here's an example of, and this is what I think I covered. You have an example of two different types of stone. Uh, with one that doesn't exist called lumerite, another one called xeronite. These stones don't exist, and they have features or properties. They have a shape, a size, and so on. So I can provide in the in the prompt examples of these two different stones, and I can do a binary classification machine learning challenge, which is here are examples, here's here's a test set, an example of a stone you've never seen. Can you classify it as xerite or lumerite? And they work. Now, you know, that so it's not like classical machine learning where we have a training phase and then we have this train model that keeps the learnt's um uh patterns and then can use them to make predictions. Isn't that just understanding the data that you provided in a clear shot sense? Yeah, you can call it that as well. But I mean it depends on what you define as learning, right? So if we agree that that classification tasks involve learning, because what we're saying is there there may be a nonlinear relationship between the features, that it's not just a case of um you know statistical kind of property that you're trying to infer.
Anders ArptegUm but learning is something that you learn. I mean, it is literally the term that you use that you add some piece of knowledge to your system, right? Or your brain. And it doesn't really happen.
Jim DowlingYeah, so then so this is where we get back to context, right? So this is what's interesting. So the learning process for agents is not training models, it's it's actually collecting relevant context. And it's no, it's collecting lots of information to make it available as context, and then being able to retrieve relevant data.
Anders ArptegSo not if I were to defend the term here, the way I would defend it and say it can have a proper meaning, is that if you differentiate the context from a single prompt to like a multiple or a set or a session that you have, you can certainly have reuse of a certain prompt in a longer session by having a set of intermediate kind of predictions happening that you are later reusing. So, in that mean meaning, in the end of the session, you can do stuff that you couldn't do in the beginning. Yes. So it it that at least in some way I would say argue positively for the term.
Jim DowlingI mean, I think look, that's what that's the trend in in large language models. That has been the trend in the last couple of years. So, you know, the the the notion that's where we have reasoning models, right? So reasoning large language models, that we we take an input string and we say, okay, I'm not going to try and answer the whole question. I'll I'll try and maybe do some entity extraction from some entity recognition from that. So um and then I'll ask questions about the entities and I'll I'll try and build up some context data if we use the term.
Speaker 2Yeah.
Jim DowlingSo that later on in the it'll actually put in the full question and have much more context information available with which to answer the question. And I think that's the general trend that we're seeing in agentic architecture is that you try and um decompose uh whatever the queries are into components from which you can basically bring in more context. And if you want to call it learning, or if you want to call it um, context engineering, actually. You can call it context engineering. Yeah.
Anders ArptegIt's kind of a nice term, I think.
Jim DowlingI think, you know, because for me, context engineering, I you know, someone, one of the students this week said, Oh, is that rag? And I said, Oh, you're so 2023. Um and none of them actually knew the term context engineering. I was surprised. They weren't um keeping up to date on uh what's happening.
Anders ArptegNow they're you know, I'm getting into terminology here, but it's just because you are such a knowledgeable person, and I think it's interesting to try to just understand this properly. So we we can say that you know a string in and a string out from an LLM is one thing. But what would you say the difference between a like traditional LLM and an agent is is I mean, a lot of people have so many different uh questions or definitions of this, but it I think it would be interesting to just hear your thoughts about this.
Jim DowlingI think an agent is there to solve a task, like uh so a well-defined task. So if you take the most popular agents, let's say we'll bring up lovable again, it's a coding agent.
Speaker 2Yeah.
Jim DowlingAnd I do have actually a specific philosophy in this because we're building our own coding agents. We call it Brewer in Hopsworks. Um, and it's that um so coding agents are are there's two types of users. There's there's the users who want to write code and help them be more productive. And I've heard people use the term um instead of using it vibe coding, they call it vibe engineering. I don't know whether whether it sticks or not, but but the point is the LM will help generate code and you understand the code and you can fix the code. So this is you know, proper software engineers, and they're getting a lot of use out of it. And in our company, a lot of people are using it, and it's very productive. Then you have the lovable people, and a friend of mine showed me his website, they did it in lovable in Dublin on Monday, Jason, call out. So, and he was really proud of it, and he's really happy. The guy can't write a line of JavaScript, has no idea what it is, no intention to learn it either. But um, the reason why it works for him is because uh lovable has a very natural intermediate representation of the output. It's the the web page. You type in the commands and updates the webpage, you can see um as you progress, you're you can see this intermediate representation, the website that you can iteratively improve. You don't care what the what that gets what's actually compiled to the JavaScript. Um so we're doing this for for pipelines, for feature pipelines, training pipelines, and agents themselves. That for people who don't want to necessarily use cursor or um you know uh clawed code, but would like to be able to write these AI systems because they know I know what my features are, I know roughly what model I'd like to use, and I know what how do I like to make predictions, that you can still, as a domain expert, generate code. So we have an intermediate representation that the domain experts can iterate on. So they'll need to have some understanding of what they're doing, right? They'll have to know this is my data, I you know, this is
Agents, Context, And RAG Limits
Jim Dowlingwhat bad data looks like, these are transformations I'd like to make.
Anders ArptegBut getting back to the LLM versus agent, because it at least in my view, because you know, even in my PhD, long, long while back, I spoke about agent-oriented development. So for me, you know, the term agent is is rather important.
Jim DowlingUh still I think it's been taken from us. Like so for me, when I heard it, I'm like, come on, I'm reinforcement learning. An agent uh exactly an agent has a goal, and um, it's gonna achieve that goal through um, you know.
Anders ArptegBut but even the classical reinforcement learning definition uh is usually using the terms agents, of course, but then also saying, of course, understand the environment, but also to take some action that influenced the environment. And you have a time horizon as well. Yeah. And and if you take a classical LLM, let's say the only thing it can really do, it's just an API, you put some string in and you get the string out. I mean, then you know, if you have a really shallow definition of an environment and say that is the only environment, then yeah, perhaps you can say it's an agent, but I think it's too loose.
Jim DowlingNo, but that's not what they do, right? So, like at least from from my understanding, what an agent does is your your query comes in, right? And and like I said, sometimes there might be an intermediate step. I talked to some people at bookbooking.com, what they do in their agent, and they do entity um entity extraction, right, from from the query uh to get extra context. Named entity recognition or sorry? Was it named entity recognition? Yeah, named entity recognition would be kind of the so any or um so um but what the agent does, uh maybe there's these processing steps like that, pre-processing steps, but then it goes into what's called the agentic loop. And the agentic loop is But that's a different thing. Okay, so we're gonna be able to do that. Well, that's that's what an agent did. I mean, that's what agents are. But that's not what old people use it for. No, but if you're if you're working in agentic programming now, the agentic loop is what it is. Yes, there's nothing else.
Anders ArptegBut that was a completely different thing. Because then I think if you have a loop, at least, you know, one I think definition that I like at least is that the agent should have at least a choice of what action to take. If it is a loop, it certainly has a choice to get out of the loop or continue the loop. Yes. Um at least make it if you just have a single input to an LM, and if it has it is evolving, right?
Jim DowlingSo you probably have heard of these protocols. Like there's there's um MCP model context protocol, which is to the agent will access tools. They'll get a description of the tools and be able to invoke them. There's another one called A2A protocol that Google have to call other agents. But there's another one called AG UI, which is growing massively. I don't know if you've heard of this.
Anders ArptegNo, not that's all.
Jim DowlingOkay, so it's AG-UI. We're adding it to our agent as well. It's taking off quite a lot, but it's basically a way of getting um feedback from users. So it's assuming the agent, this agent has a user interface.
Speaker 2Yeah.
Jim DowlingAnd uh one of the things about agents is they they are kind of um asynchronous because they're, you know, they don't just give you one answer at the end, they they stream the answer out. Um, but they also may need to get input from you and say, hey, can you clarify this? I w I need some future direction or some direction because um I can't accomplish this task without your feedback as to what I should do next. So I think that's becoming part of kind of naturally, because like you said, the agent will need to maybe clarify and go back to you and talk to you. And I I think that that's becoming part of it as well.
Anders ArptegYeah. But even if it's I mean, if you take the reasoning loop and say that it will do some kind of chain of thought thinking and then say it's an agent. And I think even that in that case is too, you know, too broad of a definition of an agent.
Jim DowlingFor me, for me, the a the chain of thought thought reasoning in the book I covered, it it's not it's not part of the authentic loop in my worldview. Um, chain of thought reasoning is just the way to say, I'm going to take the query and get more context from it by asking questions about parts of the query or asking meta questions about the query and the inputs, and then summarizing that and maybe following some paths down, and then getting all this context together, and then say, here's the query, here's the context, give me a better answer. So that the agent for it is in fact doing planning. That's the thing. That the agent, you you're you're basically saying, here's my query, here are all the available tools, and you ask the agent, what should I do? The agent is doing planning in the classical planning sense.
Anders ArptegOne way, a metaphor that I usually use here, and it'll be fun to hear if you think it's useful or not, is actually self-driving cars. Yeah. And uh it's kind of easy to understand what an action is there. It's no question. I mean, it's so easy to understand, you know, they have perception, basically me mean needing needing to have knowledge management to understand all the cameras and whatnot. Then they need to do some planning, you know what's how to get to the end destination. But even with a plan, they they can't do anything unless they take an action. And an action in that case is very simple. You accelerate, break, return left and right.
Speaker 2Yeah.
Anders ArptegIt's very, very simplistic. But then it certainly lives in an environment, a physical environment in this case, not a digital environment, but the physical environment, and it's clear that it influenced the environment by taking a very concrete action. So I mean it it's super clear. It's no question that the car, self driving car, is an agent, right? Yeah.
Jim DowlingAnd and and I'll I'll decompose that a little bit further, right? Because um how do you evaluate an action? Because it's easy to evaluate the self-driving car, right? We know we we don't want it to crash. We we we we we have a way of measuring whether this agent works. So in the book I covered, there's a new um area called evals. If you heard the term evals, evals are considered a data set that you with um, let's say it's it's uh an a query that goes in and expect a response that goes out. And you you pass in the query, you get the response, and you need to somehow evaluate if that was what you expected your agent to do. But you can decompose agents' uh evals into two sets. One are objective evals, where you can actually measure the output. So the agent took a task, maybe in the digital world. Um, and then you can validate in the digital world did it write to my feature group? Did that write succeed? And is the data there that expected to be there? Um, but then we have unfortunately subjective evals. Yeah, sure. And that's where agents like Clarna, your customer rep, was it a nice answer? Did you feel good? Did it did it satisfy you? Um and that's where people look at you know LLMs as judges and um and and humans, humans to kind of to to to really um you know bootstrap them. And just to end off the talk. So that in that case, uh you know, agents aren't, I don't think they're as as well defined as as when we we worked on them before.
Anders ArptegNo, no. But we we know it's it's supposed to at least do something more than just knowledge management tasks, I hope.
Jim DowlingOr here's here, I'm gonna throw something interesting at you, I think about because I know you I knew you get deep into all this stuff, right? And I'm just trying to find a good definition of the Okay, but but let me throw you something crazy at you, right? Because Jan Lacone has gone to do his world model thing, right? And I was thinking about it and I was going, hang on. I was shocked that LMs were as successful as they were. And I'm and and you know, um it's because being able to predict the next token, you need to have a model of the world to do that. That's you know, that's what um uh you know, some people have said. But um so I was thinking about it, and I was saying, you know, if an agent actually can take use tools and and it has context information, which you can get from anywhere in the world, that information, as long as it comes from any source, be it the physical or digital world, and it comes in as context, the agent has a the LM has a world model. So there is that interaction between the physical world or the world of actions, and and it's it's on the terms of the LLM's model, which is the model of language. So if you can translate things from the physical world into the world of language, then the agents can be um active in the physical world.
Anders ArptegThat's kind of my take on it. Who did anthropics speak about this? And I think they define it. I'm paraphrasing them now because I don't recall exactly what they said, but uh, for one, they it it was that it should be a choice in some way. And I think they had like three categories of different agents. So and and yeah, they workflows, um agentic loop, and uh yeah, and also Andrew Ng, you know, he said it well, you know, it's basically a spectrum of how autonomous it is. I mean, you have super low agents, it's basically just an LLM, and then you have super uh autonomous agents that are taking a lot of actions, you know, without even interacting with the human. And then there's a big spectrum. But I think there should be a lower bound here.
Jim DowlingI mean, I I yeah, I I I I I I don't don't to be uh unkind to Anton, but I I met him about 14, 15 months ago before Lovable took off, and and we're having an argument about um a discussion, an argument discussion about uh um how you generate code, and he was like, Prompts are the way, you know, great prompts and static prompts. And then I saw an article from them in August, I think, where they said, well, they're gonna have to agents.
Speaker 2Yeah.
Jim DowlingSo it's it's that unwrank discussion.
Anders ArptegYou're going from But they usually agent. I think we had him here on the podcast as well. So he explained the loop that you do. What they're doing is uh an advanced set of agents, I would even call it the lovable. So it's no question that that is agents to me. Um what I'm thinking is more like if you write, I actually did this recently myself, just for the podcast to get some planning help. So I have a planner agent. Yeah. But the only thing he's doing is basically going out, doing some research on LinkedIn, pulling that together, uh, summarizing the person, uh, looking at what it has he has written before, and then you know, trying to come up with a good set of questions and a plan for that. But I've hard-coded what it should do. I've told it to go and look in these places and then do this until something is finished, and then continue with that. Do that. So I've hard-coded two and set of tasks to take to end up at the goal. I wouldn't really be comfortable calling that an agent because I've told it exactly what steps to take. Do you see what I mean?
Jim DowlingI I I do, of course. Um, you know, uh this is like the notion of priors versus what you can, you know, how much how much knowledge are you building into the thing. Yeah. And at the moment, we have to build in a huge amount of knowledge. And that that's kind of why I said I'm not so worried about all the jobs being lost, because we do have to encode a huge amount of knowledge in them. There's only small, very focused tasks that they can perform at the moment. And giving them a wide open task like find out about this person on the internet, I don't see that.
Anders ArptegThat's actually a great topic. And perhaps um we can get back to that uh Goran later as well, and uh speak more about you know where are we really today when it comes to more agentic capabilities, so to speak. Um, that would be fun when you get to a more philosophical questions.
SpeakerIt's time for AI News, brought to you by AI HW Podcast.
unknownCool.
Anders ArptegYes, yeah. So we usually take a small break um in the middle of the podcast to just speak about our favorite news topics that we heard about, and there's been a couple of them. But let's start with uh with you, Jim. Do you have any favorite topics uh that you'd like to share?
Jim DowlingI do. There was a uh a large um conference or summit, I think you'll call it in Berlin two days ago, and it was called the European Digital Sovereignty Conference. And there's a nice picture of uh Macron, the French president, with uh the CEO of Mistral, and then the CEO of SAP with the German uh Bundeschancellor.
Speaker 2Right.
Jim DowlingUm and they're kind of lifting up so you know we're we're a European company, we're based in Europe, and I think a lot of people are worried about the direction in which you know our friends are going in the States, and we're worried about a democratic future. And um, the the one thing that that we can see is that you know, we we we're renting our digital infrastructure, and that's okay. Um I mean the were the Americans rented the were the Americans invited in the British to build the railroads in the 1800s? I don't think so. But we we you know the the Americans have built our digital railroads and um the news basically is about um how can we build this digital infrastructure in Europe? Because we can see that if we don't have digital sovereignty, you can't really have sovereignty. So, you know, Danish people are very worried about Greenland. Um they you know the you I don't know if you saw Aarhus and Copenhagen have left Microsoft to go to open source solutions. And um, you know, there's a real notion we know that, and and you can see it just at every every every large summit Trump has, all of the tech people are there. Tech has the leverage today that
Agentic Loops, Evals, And Tools
Jim Dowlingit never had before.
Speaker 2Right.
Jim DowlingAnd if you have that leverage, you can extract extractive rents, you can save 15% on everything, and um all our digital infra runs on that, so there's nothing we can do. Chinese don't, they don't pay. Um, so so the summit was very interesting because um I think there's now kind of an understanding that that we need to actually take control. And you know, the Americans have complained for a long time we didn't take control for defense, and we're trying to do that. And I think the same is going to happen for digital infra. Um, in my space, uh enterprise computing, I think every large company I know has left Europe to go to the States. And it's kind of tragic. And I'm hoping that that will change. I think this is a starting point for some change, and that first you have to identify the problem and agree that it's a problem. And Europe always has that as a challenge before we can actually decide what to do about it.
Anders ArptegOkay, so this was the summit of European digital sovereignty, right?
Jim DowlingYeah, I think we had there was some Swedish representative representatives from the government, and there was a bunch of other you know, prime ministers. It was a big one, right?
Anders ArptegDid they come to some conclusion or what was the highlight? Do they all agree that we we have this big problem with European sovereignty? Or what do you think?
Jim DowlingI mean, you you probably you might have read in the news there were there were some agreements that like regulation needs to be watered back, paired back a bit, so the EU AI Act will, the adoption of it will slow down. That's a big news in itself, by the way, right? It is, it is. And and and the I think one of the the unfortunate things about if you're a business, right, let's say like like our company, and you go all in on the European AI Act and you spend a lot of money being the first to be compliant, and then it's not enforced, because that's what happened with GDPR. GDPR was introduced and it wasn't enforced for a few years. It's enforced now, but it took a few years. Um, then it's very hard for you to plan, you know. So I think it's better to be honest and say, okay, we know that people won't meet the regulations for a period of time, let's let's delay them. But at least when whenever they come up with a clear timeline, please keep to it and then enforce the regulations because otherwise it's kind of crazy, you know.
Anders ArptegNo, but let me not go into the rabbit hole of GDPR and the AI Act. But these kind of recent changes, if I'm not mistaken, I think one of the GDPR changes was to allow personal data for training purposes, for AI training purposes, in in not all of it, but it were it was a relaxation, so to speak, of the requirements at least to use the yeah.
Jim DowlingYeah, I mean, there there is there is uh there there what what the U AI Act uh defines levels of data. So you're risk levels. Yeah, the the highest risk level you're not allowed to use to train models with. I I think it's it's reasonable to be honest, right? Who wants a model that's going to be biased by gender or race or or things like that? And I I think there's there's a lot of reasonable ideas behind it. Yes. Um, to be honest, like you know, in our space, move fast and break things has worked, right? Ignore all regulations if you're Uber and come in and just run it. And that model, you know, which is the Peter Thiel model of tech technology, I think we'd be better off maybe without it, you know. And if I I know it's fast moving and so on, but um my take is that like I don't want models. Like we have a huge problem with the voice of the chatbots today. You know, you ask one chatbot, I think in the book I gave an example. If I ask an American train chatbot about um Gaza, I'll get a very different answer to a Chinese trained chatbot about Gaza. It's the same thing, but they have different worldviews about it.
Anders ArptegAnd perhaps you can rephrase it that it's beneficial to be agile, so to speak, for tech development. But I wish we could be as agile for regular regulatory purposes as well, right?
Jim DowlingYeah, I mean it it's it's not easy. Like honestly, it's I I I if I knew the answer, I'd be at there.
Anders ArptegBut I think one was it was some relaxation about you know how you can use personal data for training. It was also some um some reduction for the cookie rules, horrible cookie pop-ups that we get all the time. So I think that's really, really good that we're seeing that. But then it was a set of you know um relaxations for the AI act that I don't really recall exactly. But it was something for smaller. Yeah, I think that was one of the outcomes.
Jim DowlingAnother outcome was, you know, I think they're they're trying to promote European technology, right? So there were rules about uh there is an interesting new proposal uh on the table, which is how do we define uh sovereignty levels? So um the commission has a proposal in in the works which is saying, okay, if you can take uh European people's data and transfer it out of the EU, um, then you're not fully, you're not the highest sovereignty class. And that will cover any companies covered by the US Cloud Act, which would be Google, Microsoft, and Azure. Um and you know, like uh I I hear all the arguments about you know, regulation will slow us down and so on, but in Europe we have no cloud company. It's kind of tragic, right? And the Chinese have great cloud companies, and the Chinese didn't open themselves up, they regulated themselves that way. So, look, I mean, I think we we're in a kind of crisis situation where um, you know, we want to be able to enforce our own laws. We want to decide these are European laws, these are our values, and we're currently not doing it. There are European laws on the table that are being shelved and changed because we don't control our own digital infrastructure. And for me, that's not okay.
Anders ArptegYeah. Okay, so great uh with that conference. Great that they were speaking about the problem. Great that they're actually taking actions as well uh to try to improve on the situation and perhaps get better preconditions for actually having some sovereignty.
Jim DowlingI will add the last point too. I think the the who's got a right way Scott free or large European companies. They're all hiding behind and blaming the commission and so on. But like let's be honest, if Ericsson or SAP uh or one of these large European companies had started European cloud, we'd have wrote in behind them. We all would have. I'm pretty sure they would they would be large and successful. But they're all too conservative. And I think that's so you know, what is one of the one of the most promising clouds is the little little cloud. Yeah, like that's crazy, you know.
Anders ArptegWell, I just so want to got get more into this topic. I'd actually add that as a topic later, you know, how how we could be potentially building European cloud, but um perhaps later, if we get time for it. Cool. Any other news that you'd like to share?
Jim DowlingWell, we also Jan Le Kun left his job. I mean, you know, I think we were talking about that earlier. Um very interesting, but you know, not clear what it'll do.
Anders ArptegNo. Um, and we spoke about it in the last week's podcast as well, a bit, but now it was at least some clarity on what he should do, right?
Jim DowlingUm I mean, if he he's building a company, so he's raising capital for the company. He's talked about the Jappa world model, but I think it's gonna be a long way off, whatever he's doing. It's probably on a longer time horizon.
Anders ArptegYeah, I I wish I had the article here now, but I think it was some company that he actually has started with for some time back, but not really perhaps worked that much with that he's going to continue on.
Jim DowlingUm the other news this morning was that NVIDIA reported good results. So AI bubble burst for a little bit longer.
Anders ArptegYeah, the stock market has really moved in the last couple of days.
Jim DowlingThat's uh yeah, I think there's a lot of nerves and people down. Uh well.
SpeakerObviously, not too much. It's uh if you look at the if you look at the numbers, it's the meta actually that is uh dropping the most, the rest of it is pretty slow, stable. Google is going up after the Gemini. Yes. Uh Nvidia is pretty solid, so they went up after the announcement.
Anders ArptegBut it was a general like tech uh reduction in the last couple of days.
SpeakerYeah, yeah, but Apple is now orienting themselves with Google. So if you look at their stocks, it's completely steady. So the only Amazon is completely steady. So the only one that is very volatile are just two, right? So you have Tesla and you have uh what is called Meta.
Jim DowlingI never understood Tesla being up there. If you look at I can't understand their valuation, that's yeah.
Anders ArptegSo you you bring up some interesting topics. I'm biting my time all the time to not go into it. But yeah, anyway, the the general stock market, if you just look at the index, actually moved quite a lot uh in the last couple of days compared to at least a couple of last months. Anyway, uh Jan Likun, yes. Uh and just to recap, perhaps, then he's been the hero of I think you phrased it very well actually before the podcast. And um, do you recall you know his new boss kind of thing?
Jim DowlingOh, I mean he yeah, he I think he got moved, they did a reorg, and his boss was Alexander Wang, who who uh who founded a company called Scale.ai, and they were acquired by Meta uh for several billion. But I mean they did data labeling. So, you know, if your new boss for your AI is uh a next data labeler, it's kind of I mean that's an that's not an AI kind of but he as you said, he's a touring winner, you know. He's one of the three he's been great at predicting, at seeing. I mean, he's been hammered a lot because of LMs, and he kind of talked a lot about you know the fact that they they're regressive, auto-regressive, and uh um actually that was one of the tasks for the students in their course. They said they're doing uh air quality prediction, which is auto-regressive because he's using predicting tomorrow.
Anders ArptegSo but it's not like it's against LLMs or against transform. I mean, he's the first biggest foresproker for for um you know self
AI News: Sovereignty And Policy
Anders Arpteguh self uh self-supervised learning, you know.
Jim DowlingOh, he I mean that like that that was actually the key thing. Like so you say, well, he he didn't do LLMs, but he was the number, you know, in the cake. You remember the cake where self-supervised learning has been the key technology for training uh large language models, you know.
Anders ArptegAnd he doesn't want to get rid of that and transformers it allows almost to continue with and has to answer. Anyway, such an interesting topic. I wish the best for Jan Lee Kuhn, even though it's been a bit of a not the best of times, so to speak, for him, I think, in recent months. No. Cool. Uh quick update otherwise. Uh I think you know, Gemini 3 was released this week as well. Oh, yeah, it's quite good, yeah. The scores look good, yeah.
Jim DowlingYeah, it's and trained on TPUs, not on GPUs, not on video GPUs. So that was interesting.
Anders ArptegYeah, it's it's leading the leaderboards now, taking over LM Marina, etc. And it's been a large number of releases now in just a couple of weeks, right?
Jim DowlingYeah, on the hardware, I was at a conference, a Ray conference in SF two weeks ago, and there were actually talks on Hue Way's uh accelerator. Do you know who way have a competitor? Oh yeah, I was kind of shocked, you know. That's the first time I actually seen it in the real world and live. Yeah. So it might be a thing in a few years' time. Who knows?
Anders ArptegYeah, cool. No, but uh just to give some kind of uh summary of the recent releases here, we had um, of course, GPT 5.1 in in last week or something, I think it was, right? It was uh yeah, kind of an incremental sport. I don't no, nothing, I don't think. The thing, the way that some old men phrased that was that we don't care about leaderboards anymore, we just do it because we want to have a more appreciated style of language or something. I think if he were to be leading the leaderboards, he would be bragging about that all and every day, right? So I I think that was just bullshit and and uh big excuse. And they have not been leading in the leaderboards for a long time. They were in the top of Elemarina for the normal 5.0 for some time, but it's not been that. And normally over all of 2025, it's been other frontrail labs, I would say. Gemini primarily, but also Grok has come up, and of course, Claude in coding and a lot of these kind of tasks. So I think this is just, and now we just saw Gemini 3 released, and of course it took over uh from all the others. Uh Claude and the Opus one has been leading a lot, even in text-based, kind of normal text-based tasks. But now Gemini took that back as well.
Jim DowlingBut do you not think like that that they're you know, Ilya Soskir says, right, we've hoovered up all the data, they're they're kind of tending to kind of converge.
Anders ArptegNo, but I don't think so, actually. No, you don't think so. I think it depends on what how you measure them. Uh, if you just measure them, uh measure them on knowledge management tasks, being able to take a large piece of data that you train on and just recall features from it, then then they're really good, and that will be hard to surpass. But I think actually they're surprisingly bad in reasoning capabilities right now, like arc AGI kind of tasks, which are still really bad. And even to humans, they are still bad in that. And I would even argue, and perhaps you don't agree with this, but I would even argue also in action capabilities that are really bad today. And uh, we can get back into that later. But I think that is something that we will see a lot of improvements in in coming years.
Jim DowlingOne thing I do notice is they're using a lot of context, like they they they tell me things like uh because you work in feature stores, and I'm like, how do you not work in feature stores? So they are they are storing a lot of information about us and feeding it as context. So that we'll notice that over time.
Anders ArptegDo you have a I mean, given all the models we're seeing now, the Gemini, the Clauds, uh, the Groc, Groc 4.1 came recently, I think uh Illness saying Yeah. Uh it wasn't a super big improvement. Uh Elnus said that Groc 5 will come this year, we'll we'll see, and it will be AGI. Um, yeah, let's not even comment that perhaps. No. Um, and then we have the Chinese models as well. What do you see now in in 2026? Uh, who do you think will will start to rise to the top there if you were to guess?
Jim DowlingI mean, I I you know uh there's for me, there's two two types of models. There's the frontier ones and then there's the open source ones. Yeah. And I guess my interest is more on on the gap between them, which has kind of been that that that gap has been closing over time. And I I think in the long run, I do I do think Susker's right, and that that it will tend the current with the current technology we have, I don't see it's growing to the sky. I don't see us reaching AGI on the current path in the near term. So yeah, I think the the interesting story will be convergence of uh open source models with or open weight models, I should say, rather, and and frontier. I mean it looks like Google are really taking this seriously, and Google are extremely competent engineering um culture, and I think they'll probably keep keep improving relatively. That's the only thing I can see. So Groke Anthropic and uh we use anthropic and open AI actually for for coding agents. But um you know, maybe we'll have a look at Gemini. I don't know.
Anders ArptegYeah, yeah. Gemini's getting really good in coding as well. Okay, interesting times as usual, and a lot of stuff is happening in the world of AI. But getting back a bit to um to perhaps Hop's work a bit more, and uh also going a bit more philosophical and thinking you know, we had the old time of you brought up like Spotify's Discovery Weekly kind of batch systems, and then we have Salando and other systems doing or King, for example, is doing a lot of uh real-time workflows. Yeah. Um what would what do you say the the challenges are in in moving perhaps a bit more towards these kind of real-time systems and perhaps even agentic workflows? And how is like Hopsworks trying to address that?
Jim DowlingYeah, I mean, we we took a bet early that that people would move more from batch to real-time. So we we built out the infrastructure for helping companies build real-time. So you know, you it's a little bit of a change. So go so batch is this notion that that you take data on a schedule maybe once every hour and you you copy the data over. If you take Spotify weekly, it's gonna pull up all the data from the last week. It's a big spark job. If it's a little bit late, or you know, it it it it the spark job has failed, or some workers fail, and you have to restart them, and or it does that automatically, but it it's not the end of the world. So batch is kind of operationally not as challenging. That's why most companies will start with batch. Um, but if you want to add real value, like TikTok is the most valuable AI system in the world today, still, it creates the most differentiation and the most value, and it is the real-time aspect of the prediction of the recommendations that it makes that differentiates it. And building that is hard. So um, you know, I think a couple of years, a few years ago, companies were more inclined to go this way, but when LMs arrived, budgets just swung right over to LMs and everyone went all in LMs. So I think the real-time AI space has not grown as fast as as expected. We we had some competitors in the space. Um when we did the when we started in the feat the feature store space, I think there were four companies in America raised 20 million, and another raised 165. They're all gone. None of them are left. Um, because the space hasn't grown as fast as anticipated, and um, there's only space for a few winners, like it's one of those things. So for us, um, it's been uh you know, it's not been the the the the the the hockey stick kind of from day one. It's been it's been building a database company. And and and you know I don't know if you heard the cliche, it takes 10 years to build a mature database. We're at year seven, so you know we're getting there in terms of the data platform, yeah.
Anders ArptegOkay, so um for one, perhaps you know, you you said real time is most valuable. I guess there are a lot of examples for that, but I I guess there could still be use cases where batch could be useful.
Jim DowlingUm, but most mostly most use cases in enterprise are batch, yeah the vast majority. Things like oh, you know, email uh classification, marketing campaigns, you know, churn prediction, a lot of these kind of you know, grunt work in in that data science teams produce at their batch. I think what what's changed actually are agents because most agents aren't batch. I think you know, there are batch. I I would argue coding agents are kind of batched. They don't have a real time, it means that you have some expected response within a bounded amount of time. Like so if I have an interactive agent and it builds on a competitor platform and it takes two minutes or five minutes to respond to adapt to what you just did, you you won't perceive that as being intelligent, right? So you know, I mean, at I at a very basic level, um the an agent that is interactive and will respond to actions you're taking, um it needs to do that in a second or two. Otherwise, humans won't perceive it as being intelligent. And that's kind of the tech we're building for that. Um and it's not, you know, the uh But it depends on the task, right?
Anders ArptegI mean, if you're playing chess or if you're even a coding agent if it takes a minute, I can't.
Jim DowlingBut a coding agent's not an interactive agent, right? You know, so there I mean there's gonna be there's gonna be like your your agent that goes out to LinkedIn and you know profiles that that's a batch agent, right? So and so there's gonna be both, of course. Um but I think I think a larger fraction of agents will be interactive. That's that's my expectation. I might be wrong, but I I think a larger fraction of agents than than the AI systems we have before, machine learning systems. So interactive in like a second or subsecond kind of time frame, or I mean, you know, I mean, well, that's that's what we that's what we we provide. We provide info to get you to the sub-second level. And now some people might say, Well, I don't need subsecond, I need 10, 20 seconds. And and and that's fine. So there's going to be a spectrum of use cases. I mean, companies who buy our platform, they do because they have many use cases. They don't just build one AI one AI system, they build many use cases on them.
Anders ArptegWhy is streaming or these kind of real-time use cases harder than batch? Oh, that's all right.
Jim DowlingYeah, that's I mean the operational aspect I touched on, right? So if your batch pipeline fails, yeah, you can go grab a coffee, come back, fix it, and then it's ready for the next time it runs. Operational, the warning flags go flying and uh things aren't working, you know, let's quick fix it, right? So you have to have high availability built in typically, like so components fail, they have to keep on running. Um, you have uh if you have spikes in traffic, you have to handle them. You know, in batch case, you know, data can pile up fast or slow, won't matter. You're gonna you're only gonna process a bounded amount. But in streaming, you know, often the data is coming from external systems and and you don't have control over those, so you can get spikes in load and so on, and you need to handle all of those. So there's a lot of operational work in it. Yeah. Yeah.
Anders ArptegI remember, you know, the back old like Hadoop days and writing the first batch jobs and comparing like uh a single processor kind of algorithm and needing to paralyze that into to a batch mode. It was a big thing and really hard to do, and you had to rethink, you know, how the even wrote the article or sorry, the algorithm to make it work. And I guess in some way, doing a streaming version of that is also a big step where you and that was like maybe 10, 12 years ago, right?
Jim DowlingYeah, it felt like it was huge. It wasn't that long ago, right? In the great greater scheme of things, it's not that long ago. Yeah, I mean, streaming is um is part of it, right? Because real time, if if I a lot of people write applications, what happens is all the actions, the applications, you just kind of stream them out to an event bus like Kafka or something like that, and that's where the data comes in. But but your application can also you can enter text and data, so it can come in through requests. So so you know, you can have the query
Hardware, Models, And Leaderboards
Jim Dowlingstring, of course, as an input parameter. So so real time is a combination of both, you know, data that comes in when you have a request for a task to execute, and also then this kind of background uh activity level uh data. So data can come in in many different ways, and you kind of need to have this central view of the data so that you can you can take that data, whether it's context or features for machine learning, and then um process it in such a way that it's gonna be, you know, help you with your query, give you a better answer. Um, and then you've got to think of the operational aspects like is am I generating too many tokens or um is it feasible to compute this? Is it taking too long to compute this or to retrieve this? So there's there's a ton of different things that that come into play that that are that you can just kind of ignore in the bash case.
Anders ArptegUm I remember playing you know a lot with the flink uh founders as well uh in the back in the days and how they they came from you know the from normal kind of traditional database world really in the beginning and and then they rewrote or reused some of the ideas. But still came up with a real noble way to do streaming. Yeah. Which you know, Spark didn't do because they tried to have batch and then add streaming on the code.
Jim DowlingAnd it's still a problem. Like so if you know Zolando's biggest, uh the big Databricks' biggest customer in Europe is Zolando. And Databricks say everything has to be Spark or SQL. Yeah. And they can't do real time. So you can't do the click and then the data gets through. Firstly, you get to the Spark microbatching, and then you get to their lakehouse first kind of world. So the data gets right to the lake house and then sync to the databases. So it's minutes. And uh we have a different kind of philosophy where the data is real time first and then goes to the lakehouse after or in in parallel.
Anders ArptegSo what is the tech stack that TopSperx is based uh mainly building on, so to speak? If you can share something about that, yeah, sure.
Jim DowlingI mean uh a huge amount of the platform is open source. So so the core kind of technology is a database we build called Ron DB, named after the inventor of the database, Michael Ronstrom. Um the database came originally, it's a it's a Stockholm story. It was originally built at Ericsson and then MySQL bought it over where I started working. Um, and it's been open source, and um we've forked it, and now we we're the largest contributor to that um NDB cluster uh platform. So, what's specific about ROMDB if you were to just quickly Yeah, I mean it's a it's an in-memory distributed database. You can have data on disk, but primarily it gives you very fast access, low latency access to data. So if you're building a real-time AI system, you might have a budget of 50 milliseconds or 60 milliseconds or 100 milliseconds for for doing something, taking an action. And that budget has to be split between getting the data, making the predictions, processing the data. So you want to keep that budget for the database as low as possible. You know, and if you're going to DynamoDB, it's 20 milliseconds or 10, 15 milliseconds, and we can get down to two milliseconds. So you're going, great, I can spend more time doing AI instead of just going to the database. Um that's what makes it unique, and it's kind of high availability. So you can have lots of servers and they can crash, some can crash and it keeps running.
Anders ArptegAny other major components you'd like to highlight?
Jim DowlingWell, with the feature store is kind of the the the I guess what we're best known for, which is it's effectively you can think of it as being um a way of managing the data that comes in and the transformations of the data coming in and storing it for training and inference. So we do have a lake house as well as Ron D B. So Ron D B is the fast serving of the featured data, but we build an open standard. So open lake. Delta Lake or Delta Lake is one of the OTF open table format standards we support. Uh we support Apache Hoodie, iceberg will come soon. Um and we we have built in uh Kafka as well to ingest the data. Um and then we provide compute, so it all runs on Kubernetes, so you can run jobs in Hopsworks, like you can run Python, you can run Spark, you can run Ray. Um what we also do is we'll manage GPUs because if you have GPUs for compute, um Ericsson, for example, have a a massive cluster with you know tons of GPUs. Can't say the number, but it's lots. Yeah, and they use Hapsworks to um for training and inference um and and running things like Ray and so on um at scale.
Anders ArptegSo and uh Kubernetes, do you use the vanilla open source or do you use somewhere?
Jim DowlingWell, we we we don't use Kubernetes in a sense, we just run on any Kubernetes distro that's there. You know, so it's kind of we when we started HopSerks, we we ran on virtual machines because we didn't think Kubernetes is it wasn't mature enough to run databases on. And um, it took us a while to get there. Like we should have gone earlier, I should say, because the the great advantage of moving to Kubernetes is um that we started out doing a managed platform in the cloud where you would deploy it into the customer's account and and we'd have a control plane and we could manage that. And we still do that, like for example, with with Solando. But but now that runs on Kubernetes, and we can take the same whole software platform and anywhere Kubernetes runs, we can run. So if you're in an air-gapped environment off the grid, Opsorg still works there. You know, so this is great for for for you for running in European clouds or running on private data centers. We can, you know, we have some some customers, some government and law enforcement customers that we we run on and they're they're air gapped.
Anders ArptegAnd um you use Ray as well. Can you just elaborate a bit more?
Jim DowlingDo you have a lot of people? Yeah, so we we we provide compute in the platform, and but it this is where we're a bit different from let's say Databricks or Snowflake. We don't force you to use our compute. So if you want to run compute outside Hopsworks in Flink or Spark or anything, that's fine. But we do provide compute. And I would say in the cloud, a lot of customers would bring their own compute, they're already running EMR or Databricks or something like that, and so we just keep using it. Um but if you're in a non-prem environment um or in a European cloud, maybe you don't have all those um compute frameworks available to you, then then they use hopsurce compute. So we can run Spark or Python or Ray Jobs, the the primary so Python means anything, right? So it could be polars or pandas or you know, it could be training with Torch or um, you know, these kind of things.
Anders ArptegSo cool. Um so you support a number of open source. So I guess that's a big important point here. You called yourself like the uh on-prem version of Databricks in some way, right?
Jim DowlingOr yeah, I mean it's to make it easier for people to understand, because it's like um it really, you know, we we provide uh analytics and we so it's not just the um you know we have the the the lake house for storing large lumps of data. We actually have a query engine as well coming in now called Trino and Superset for doing analytics, right? Yeah, so then then like because uh the the trend in our space has been very Databricks have done incredibly well in the last two years because they unify analytics and AI. And you buy one platform and you get your analytics and AI in that stack, and um, this is very attractive to uh a lot of um enterprises because a lot of the you'll know probably from your experience, integration is a huge amount of work. You have different platforms and you have to talk to multiple vendors. So so we're we're we're following the same strategy where we started with real-time AI, and we have then you know batch and and LLMs, and we've just added uh now this this analytics capability as well. So we're you know effectively saying, hey, you know, we're providing similar capabilities to Databricks, um, but with a different, I guess philosophically, what we're most different is we're open compute. That you use your bring your own compute, that's fine. And if you have 130 billion valuation, you don't want open compute.
Anders ArptegHow much of if you go with Bitcoin, perhaps the normal kind of tic stack and the jobs in a compute and a storing you provide and a feature source, etc. But think more about you know the full life cycle of MLOPs or DevOps or whatever you prefer to call it. Uh I mean things like uh perhaps even data governance or having data catalogs in some way or even becoming compliant in some way, or you know, all these kind of things and monitoring, of course, and security. Anything of these kind of aspects that you're also helping out with in some way?
Jim DowlingYeah, of course. I mean, if you're if you're kind of providing a platform, you have to cover all these bases, you know. Enterprises, uh the hard thing about selling to enterprises is you build something great, and then they say, Well, I would buy it, but you're missing this, this, this. And uh, of course, security is a is a table stakes. And yeah, and to be honest, governance is table stakes as well. Like you, if you're not doing audit logs, if you're not doing access control and all that kind of stuff, there's you're just not gonna be there. Um you know, monitoring is is is built into the platform as well. Again, a lot of it is open source components and so on. Um, but you know, I think in the the days of you buying from six vendors, six different products and pulling them together
Batch To Real Time: Hard Parts
Jim Dowlingare pretty much coming to an end.
Anders ArptegBut I guess also that's what the big top like cloud providers help with a lot, and which do provide a lot of value. But if you don't have that, that's surprisingly hard to do by yourself.
Jim DowlingIt's extremely hard. I mean, I think you know, the the the amount of teams we've met who spent years building infra instead of building AI systems is insane. Because I think that's so I mean at least we we don't want our customers to build infra. We we we want them to build pipelines. We want to build AI systems. An application. An application, no, of course, applications so the the pipelines are there to feed the apps, but without the app, you have no value, you're not creating any value. So so companies should focus on you know getting their data, preparing the data, training their models, making their predictions and integrating with applications. And then, I mean, we just provide the infra to put to connect that together. And whatever you're building, you should build with open source tools, not with proprietary techs and compute, because you know what you're gonna be locked in. Yeah.
Anders ArptegAnd it's stuff like the CloudFare incident will probably happen to camp this time. No, but I think this is so important, and I think so many people are missing this, and and I think also it's a bit perhaps not the most sexy topic to speak about, even for you. I think in Hobbs work sometimes. And like here so many people saying, you know, ah, but we have a great working demo here. You know, you can see it's training and it's doing some inference, and it seems to have some KPI that works. And they're missing like it's 100 or 10x more work required to actually put this in production properly. And and to just do that, I mean, it is easier if, of course, if you run on the top cloud providers, it is. And that kind of value that they do provide, I think people do not really understand. Yeah. Uh, and if you want to not be super dependent on them, we need something else, right?
Jim DowlingAnd yeah, I mean, I think I think there's a general like the the if you look at the we come from a database background, and and there's not two, you know the way we have this thing where we converge and there's two standards, there's there's Android or Apple, right? Yeah. There's not two databases, there's like FIFA of them. There's hundreds, you go to DB rankings, you'll see actually three, four hundred. So, you know, I I think at least for building and operating systems, it would be a tragedy if we have two or three. You know, I think I think we really need diversity. And I one of the most interesting trends I see at the moment is um these data processing engines. Um, there's there's been this Cambrian explosion in data processing engines. So if you if you go back 25 years ago when I uh when I worked at MySQL 20 years ago, everything was a relational database. And along came big data, and there was key value stores. If you remember Dino and DB, people were excited. It's uh and then we had Hadoop, and then we had Neo4j, Graph Database, and we had to still do Neo4j. Yeah, but I mean the point here is that we had we had all these specialized databases like a JSON store, Mongo, and a document store and Elastic, they all became viable databases when data volumes increased. And everyone didn't want to do relational databases, so we see the same trend in data processing engines. Everyone has done Spark, or not everyone, but a lot of people have done Spark. A lot of people know SQL. SQL is always gonna be there, but SQL isn't always the best tool for the task. Um, so we see frameworks like uh every uh pandas has been forever, but Polars is growing a lot. Really great, interesting data science framework. Um, DuckDB has grown massively. I was at a DuckDB conference two weeks ago in SF. Um, and um we work, uh an ex KTH person actually has a startup called Feldera. And Feldera does something called incremental uh stream processing, it's really interesting tech. So so uh let me give you an example of a problem. So imagine you're doing um credit card, we'll take credit card fraud again. So we have a lot of activity in this region here in Stockholm, and we and we want to maintain statistics about um number of bad transactions or or anomalous transactions. So every time an event arrives, why flink is really good is because it'll it'll recompute um aggregations and so on as that event arrives and output it. It's real time. But the compute is is is uh proportional to the size of the number of events. So the computational complexity of recomputing that aggregation depends on how big that bucket is, how many events are in it. When you get to a million events and there's you know tens of thousands of events right per second, it doesn't work. That's a kind of dirty secret of flink. Okay. Um so these guys have rewritten this as an incremental compute engine. It's called incremental view maintenance. And the the the computational complexity of recomputing the aggregates is is proportional to the amount of data. It's order one, the number of data computers. Yeah, okay. So it scales and it's really beautiful. So that's a new compute engine that's out there. It's open source as well. Also name of what this is? F-E-L-D-E-R-A. It's SQL. Feldera. I mean, it takes SQL and compiles it into Rust, so it's ultimately a Rust engine. But um, I mean this and and there's another one called materialize, uh, Frank McSharry. They're they're trying to compete in that space. So we see tons of these engines. DAFT, you might have heard as well. I don't know. I can keep going on all day. So I I suspect that there is a role for a lot of these engines in in different specialized workloads. Ray, we mentioned. Um, and that's kind of the future of data processing we'll be in.
Anders ArptegSo it's good that's happening, but it's also kind of hard for companies that want to or you know orient themselves in the space, right? So then we need to yeah.
Jim DowlingI mean, you know, the the the the answer is use Spark and SQL for everything. You know, um but but like you know, when you're doing specialized workloads, if your name is Orlando, you're not using Spark, right? Yeah. You're using Flink.
Anders ArptegUh I think you also you mentioned, I haven't really read through the all the book, but but I saw uh in the digital version at least that uh you also spoke a bit about function calling for LMs.
Jim DowlingUm yeah, that was kind of going going at the beginning. So pre-A, like without even getting into agents, you know. So because a lot of people jump in straight into agents and MCP and say there's tools and so on, but but but MCP is built on function calling, right? So so I I I think it in chapter three, we kind of just went in and said, okay, how do I get external data? In this case, it was the air quality prediction example and description of it. And how do I get that and insert and inject into my prompt? And function calling is the is the kind of the so your LM will return and say, here's the parameters for the function, uh, and then the client can call it and get the result and insert it back into the prompt.
Anders ArptegGet it better, better with JSON strictness and special output, right?
Jim DowlingSo yeah, I mean, you know, they nowadays people take it for granted that all LMs will return JSON in the structured context, but it wasn't the case maybe a year ago or so. So it's moving quickly.
Anders ArptegSo yeah. Okay, cool. Um I would like to move to a more difficult to answer question, perhaps, but still, we touched upon you know European sovereignty a number of times. But if I give a bit of a longer intro here, I would love to hear your thoughts about you know how can we really build European, perhaps not cloud providers that are better than the American or the Chinese, but at least are sufficiently good so we can rely on them. Um besides Hopsworks. Um and in my view, one way to phrase this in a negative sense, if I start there, is to say that if we take all the big ones, both American and Chinese, I would say, they have been successful with their super big cloud service because they worked 10 years with internal infrastructure and built up that kind of infrastructure for their own purposes and then turned it into a public cloud to a cloud where they can simply externalize uh the functionality in a very, you know, not very, but in an easy way. But it is a ton of work. It's like tens of years and thousands of people that worked with it. You know, Google have like, you know, probably five uh exact like copies of just how to do key value stores that they built internally, and then you know they uh chose some of them to just you know put it in a public cloud. And who in Europe could ever do that? I mean, if you take a company like Spotify, I mean then they went to Google early on, you know, and used them. Uh Salando can't do it, SAP, no, I don't know way, I don't think. Um I mean, do we have any such company, or is is it really required to have a big company that built up their internal infra if we're going to have a competitive cloud service in Europe, you think?
Jim DowlingSo I don't agree with the assumption that you have to first have I I'll give you the counterexample. So the proof by negation, if you will. Oracle. So okay. I mean, our Oracle have now I mean I laughed at Oracle. I think it's only five years old, right? It was horrendous at the beginning. I mean, it was horrendous. Yeah. Um now I know um uh Eric at Mo at Modal, they be they start on Oracle. I had a lot of their GPU utilizations.
Anders ArptegOracle is a huge company, it's been around for a long time.
Jim DowlingThey have SAP are bigger than them by market cap. I think at the time SAP were bigger than them by market cap. I mean, there there are I mean I can name a bunch of companies in Europe. Let's the name the shame the SAP are definitely one of them. German engineers are really good quality. I think Ericsson, I mean, Ericsson's had an Ericsson cloud. Yeah. You probably remember like 15 years ago or something. But you know, we had uh I'll be brutally honest, we had bean counters from handles running the company. They were never gonna have a vision to uh it's three times the market cap. Yeah, it's three times now, but I mean if you go back two years ago, it wasn't uh the case or a few years ago. Um yeah, but but there are some bean caps. In 2020, they're around the same market cap, you know, like roughly. You know, but then that's the you know, there there are companies, but I think look, there is the capital on the one side, but technology I I had a meeting this morning at Everc and if you know Evrock. Of course, of course. I know that's so Everc have a really I I really like their vision. Um and their vision is not to repackage OpenStack. I won't say like I won't get into the details, but but but a lot of cloud providers in Europe are like, okay, there's there's an open um open source. You mean the standard open stack, like so open stack, as you know, is kind of an open source kind of cloud infra, if you will. Or a standard for standard. And but but a lot of the cloud providers in Europe just kind of repackage OpenStack and kind of put it out there. It's a bad one, if I may say so. But still, okay. Yeah, I mean it it's it it's like LibreOffice, right? Like in a sense like you look at it. It's clunky and kind of terrible. Yeah, and it but the thing is it gets better over the years, but it's still it's still not Amazon quality, right? It's not and and I think they they kind of understood this every said, okay, we're we're gonna build a lot of this cursor for ourselves. And I think that's the right way to do it.
Speaker 2Yeah.
Jim DowlingUm certainly and they've raised some capital, but I think they do need help. So what I'd love to see, and it'd be crazy, right? But um, and this came out a bit. Um, Europe is gonna, I mean, Germany's gonna spend 200 billion on defense. How did the internet start?
Speaker 2Yeah, yeah. I'm weird on this.
Jim DowlingWhy why why don't why don't they they say we need a German cloud for for for it's gonna cost us 50 billion and um we're gonna build a German cloud for our defense and uh go and build it, you know. I think I would say that we talk we're we're partnered
Hopsworks Tech Stack Deep Dive
Jim Dowlingwith quite a lot of um clouds around Europe, not just uh Everk or friends, uh OVH in in France. We actually moved, we have a premium version of Hopsworks that we ran in AWS and we moved it to OVH because we went from spending around $15,000 a month to around $5,000.
Speaker 2Oh interesting.
Jim DowlingAnd you know what? It's better performance. And do you know why it's better performance? And this is the the this is the other thing about clouds that people don't talk about. The public clouds are not very good for some things. So so the the disks that you get on the virtual machines are disgracefully poor performance compared to what you can buy here in in ale giganton or somewhere. So I can buy an MVM disk here for 1500 crowns, it'll give me you know five, six gigabytes per second. And the best I can get in Amazon is one gigabyte. It's hard. And when you're when you're a prop platform like Hopsworks, that makes a huge difference. But OVH give us good disks.
Anders ArptegBut it's also you can argue the the opposite. If you go to Google Cloud, you can get the top of the line like hardware in the same thing. No, you can't get the MVM disks there either. Uh take AI hardware, it's for example.
Jim DowlingAI hardware is very good. But but the disks is a is an anomaly, right? And it's because my take is that it's because S3 is the cash cow that keeps everything locked in. And if you provide low-cost, high-performance disks, data services that are not S3 can kind of get it. So I think you know, there's some things can be done a lot better than there. Because I'm not as negative to you as you. I think tech technically, uh, what is a cloud computing platform? It's two things. It's software, and then it's it's it's infra. Infra being the buildings and the capital. If you get the software and you get software that's good enough, then it's just a case of scaling up the infra and scaling up the investment.
Anders ArptegAnd I I did want to start on a negative point. So don't say that I am that negative personally. Uh-huh. I'm not saying you're just I'm just But it is. I'm trying to find a negative reason. And if I were to argue against myself then, um, I think you know a reason that we do have a chance to build a European cloud is that we don't need the extreme number of services that, for example, Amazon provides. Most companies need a fraction of that. And they have spent so much time building things that very few selected companies may need. So if we just could have the core essence of the services, actually, like a thousandths of the amount of work they have spent on that would be necessary to have something that is sufficiently good.
Jim DowlingI I can tell you, like the my my investors listen on this, but we we burnt a lot of money doing this, right? So we we started out by building a cloud-managed version of Hobster's where the control plane was in AWS. And uh we had all the cloud native services for spinning up the virtual machines, and you know, we're using um, you know, uh the DNS service, we're using the internet gateways, we're using all of these things to make you know, make it secure. And in the end, we're like, let's go to Kubernetes. So a huge overlap in terms of the cloud native services that you use to build applications are available in Kubernetes. So the advantage of building Kubernetes is you can go to any cloud. You can just move it and migrate it. Um, and that's I think that that's helpful for building a competitive. If you have a simple managed Kubernetes platform, a lot of workloads will be able to migrate it to a European cloud. Um, and I think a lot of people are developing applications now and and systems, particularly if you take data platforms, so databases and um data processing platforms, they all run in Kubernetes natively. So if you support managed Kubernetes, you can have all of those um in there without needing to rebuild them for such.
Anders ArptegSo that's that's a trend that helps them.
Jim DowlingI think that that trend helps any any any new cloud coming in.
Anders ArptegAwesome. Well, let's um let's hope that we actually do get sufficient investments and push. And and perhaps also, I think a problem was in Europe is where it's so fragmented. Um both in terms of investments that's being done. And we take the big Stargate project in US, you know, $500 billion for building out the AI infrastructure. Then in Europe we got $200 billion, not bad, but it's it's a completely different kind of structure for who gets what. Oh yeah. So it's not like three major companies that's driving everything that have experience in doing these kind of things. It's it's spread out to the huge number of small institutions, and the success rate of that I think is very low. It's the wrong way.
Jim DowlingIn this case, I'm I'm sorry to say, but you have to pick the winners.
Speaker 2Yeah.
Jim DowlingThe Chinese pick the winners and um it it worked for them, right? Yeah. Um, so yeah.
Anders ArptegBut I like the the uh the defense part. We we will have need to build for for some purposes so much infrastructure, and it needs to be sovereign and it needs to have quality. And perhaps, you know, as the internet got started, so could perhaps the cloud business get started. Cool. Um I see the time is flying away here a bit. And um I would like to just think a bit more about you know, we have Foxworks today. Um, you have the book that speaks a bit about how we can prepare for some of the agentic workflows uh in the future. But what do you think? If you were to not think 20 years, but perhaps three to five years, what do you think will happen in terms of how agentic works or other AI works will will change? Well, that's that's a that's a big question.
Jim DowlingI think you can make a lot of money if you get the right answer on this, couldn't you? We're getting more philosophical. So it's yeah, but I mean it's great to speculate. I think um the world of work will change, there's no doubt about it. Um, I don't like to quote, I mean, a lot of people quote Andre Kaparth said it's the decade of agents, not the year of agents. Right. I I do I do think he's right in that, you know. Um the the the there's this cliche in in machine learning which says that 90% of the work is data. It's it's data preparation, it's it's getting the data to where it should be. And uh agents will not be about um yeah, LLM leaderboards, it'll be about the grunt work of context uh engineering that we talked about earlier. It'll be, you know, uh agents are will will be performing tasks, tasks that we want to automate to make easier so that maybe some a lot of people lose jobs because those tasks will be automated, but those tasks are knowledge tasks, and they'll they'll need knowledge that is not just encoded in LLMs, but knowledge from the world around them. So private data, real-time data, and that context data will be need to be transform trans translated or let's call it transformed into a format that can be fed into the to the agent. So that's a lot of work. We talked about you know stream processing and how long it's taken it to sort of mature. It's still not like mass market yet, but it takes 10 years. So I think it's gonna take that time. Um, a lot of things will need to be digitized if they want to be used by agents. They're not ever not everything is digitized at the moment. So I think it's gonna be quite a long journey. It's hard to say what will be next. I think coding is has been one of the first things to be um, let's call it upended by by agents. Um I think uh my my near-term prediction is data analysts. Yeah, I don't think it's you know, you might say it's controversial, but uh I know from experience of you know, writing these uh SQL queries to generate these dashboards. At the moment, the way it works is that an executive in a large company might say, Well, we need more information on sales in this particular area, and we need to know about disruptions, and they'll create a request which will become a ticket, and the data analyst will get the ticket and they'll get a sprint, they'll do out the dashboard, and it'll come back three weeks later. The the the executive will have their dashboard to help them make a decision. Um, that will go from three weeks down to you know three minutes.
Anders ArptegYou know, you quoted what I just said myself, and some other people in Global Connect just um very recently. So I love that. And and just to phrase it the way that I usually speak about it, is I think AI for coding, as you say, is coming a long way. I think it's more or less almost like a window to the future in some sense. Yeah we can see what AI can do with agentic abilities for coding today, and it cannot really do it for a lot of other workflows we have in the organizations today, but it will happen. So we can see a bit how how that is affecting us and impacting us, and that will be a big change in the coming three to five years, I think. And I think agentic analytics, if we call it that, that's what we usually call it, is certainly such a case. We're not having the same capabilities for AI in analytics as for coding, but we're starting to see it. And what Snowflakes and others are doing, and also database, of course, uh with that kind of works and hoppers. Okay, cool. It's really doing exactly what you say. You bring um a workflow that normally took like uh three weeks due to three down to three minutes. Yes. And and the same for coding, then that's yeah, come even further.
Jim DowlingI think what the what the what the first task that would be automated have in common is the context information needed to them is kind of like almost like closed. So if you think think about coding, um you have your source code tree, right? And then you have some libraries and some API docs. So the context is readily available and it's there. Um if we think about data analysts and SQL again. So one of the problems, for example, to get technical is um SQL warehouses and SQL databases, they have columns and they have tables, but they don't have metadata to describe columns and tables. And and we've had that in the feature store. So that made it a bit easier for us. Um so what we could do is we could take a table that's in Snowflake, mount it into Hopsworks, and then we'd we'd the LM would read some of the data and and propose descriptions of the columns and and the table. And LMs, once they have that metadata, they can make that. So if the context data is completely available for a task, I think agents will pretty quickly come into. I mean, you can see in legal, for example, if you take law, a lot of the documentation uh is available, legal documents are available. Um, do you need context information beyond that? Probably not. So that's another area will be up to handed quickly.
Anders ArptegBut I still think, you know, which I think you also said, you know, we still need to make sure the data is properly available then and with the sufficient, not perfect quality, but sufficient quality to to make the agent work. And um we've actually done that a number of things like that in in at work and If you do get it right, then it it seems to be magic on top of it, what the agent can do. But if you don't do that, it's still a lot of work to prepare the data properly, right? Yeah, yeah. But uh it there is a big promise there, I think. And just to elaborate on another thing, and thinking more like geopolitically, perhaps, or at least more from a company point of view, we can see that some companies, especially the big tech provider, tech giants, of course, are adopting to these technologies very, very rapidly. Yeah. And uh and they have done so for many, many years, but are going to continue to be very very fast, probably, to adapt also to these new agentic abilities. Yeah. But perhaps not so for a lot of other companies. And uh, one term we use a lot of here in the podcast is the uh AI divide, um, playing on the digital divide. But now speaking about that, some companies are, we can already see the most valuable companies in the world. If you look at the top 10, they're all these kind of AI-enabled companies, I would say. And uh and then we have that long tail of other companies, and especially in Europe, uh, right? And then the question could be in coming three to three to five years, um, either we are pessimistic in saying the the big tech giants and the one that are you know working really, really fast are just going to accelerate even faster. And the other one is is going to continue to be slow and the gap will increase. That would be a negative scenario, yeah. But we can also see companies that are perhaps not adopting at all, and they will probably die out, yeah. Right. So there will be, of course, a lot of tech progress uh in coming three to five years. We're continuing to see that. But adoption, you know, what really will happen with adoption? And will the AI divide continue? I think it will, yeah.
Jim DowlingI mean I mean, let you go back if we go back, you know, 25 years ago when digital photography started coming. Yeah. Fuji were a massive company, and I had both the early digital camera, they had the physical
Open Compute And Analytics
Jim Dowlingone. You could see it was coming. I mean, you started using a digital camera, it was just so easy. And you knew that this would die out, but it still took people years to kind of figure it out. And the same with going to smartphones, we saw that trend. You you saw it, and we can see the same with AI. It's pretty clear that um if you're not adopting these technologies to build intelligent systems, and your competitors are, you could be in big trouble. So it will be a big change for a lot of industries. Yeah, and I think you know, um it's gonna be it's I mean, uh, here's a big one journalism. Yeah, I mean, journalism, there's no journal, there's so few jobs there now. That was a big industry 20, 30 years ago, yeah. But the internet upended it. So I think it's this is gonna be a lot of industries that are gonna be upended now.
Anders ArptegI want to find a positive twist on this. So let me try to also do that and see if you agree with that. When it comes to American versus European possibilities here in three to five years, and then the negative spin, of course, is the AI divide will continue. Yeah, but if we take the average company, average company in America, yeah, compared to average company in Europe, I think if we actually get European companies to adopt, not compete with the tech giant, but adopt to this type of technology fast, which is something that we in Sweden and Europe have actually been rather good at when it comes to technical adoption or digital adoption. If we just could do that, then I think Europe could actually be faster in adoption for average companies than American ones.
Jim DowlingI yeah, I mean I look, I don't I'm I I think I think there's a huge opportunity here for Europe as well. Right. Now, my working thesis at the moment is that LLMs and open source LLMs will effectively commoditize intelligence. That's that's maybe you consider a strong point. But I think LLMs, we understand the technology, the transformer technology. And um if that if that is the case, then you know the largest LLM labs around the world that are training these models mightn't have the competitive advantage that we think they will have. And you see this on a lot of the leaderboards, right? So a lot of the leaderboards that are improved are hey, it's now an agent, it's using context. So just this notion that I get context information and I'm and I have a query that I want to ask the LLM, and I'm able to augment that query with the relevant context information. And now suddenly it performs much better. That is the technology we need to develop. And we can do that in Europe, right? But to do it, we need to go data first. You need to have your data shop in order. So you need to have, you need to be digitalized. So if you're not digitalized as a company, that's the first thing you need to do. You need to manage your data, and then you need to uh look at okay, what are the tasks that we perform? What, what if which of these tasks could be automated with agents? What context information will be needed? Will it be? I mean, we can see in the future we'll see things like, you know, oh, this this task is performed incredibly often. Um it requires a lot of data and context. Okay, I might need a fine-tuned model, for example. So there's going to be lots of these small different areas. A small, you doesn't, you know, a big moon for small models. I I was at um this Ray conference and and I met someone from uh Salesforce and they're running a lot of small language models, actually, because they have they've broken down the workflows into small tasks. So I think if you can manage context data and you're able to deliver that context, and if the models are are, you know, there's not a huge difference between them. They're somewhat commoditized, then you're going to be competitive. And, you know, we're building out some of those companies here in Stockholm, right? Yeah. I mean, you know, we have Lagora, we have um Tandem Health, we have uh Lovable and a few of these others. They're doing the right thing. And that is uh, I think key competitive advantage. Um and big industries need to look at this as well and need to move faster. Right, enterprises as well. They have to. Yeah.
Anders ArptegBecause they're they're gonna be under threat. And just to continue one point there, I just uh the OSS or the open source models and how they will spread out in some ways and forms. Um I'm trying to find a positive spin on this, which I think is clearly possible here. Of course, the big you know, AI frontier labs will continue to build these super big models that will be in like GPT 4.5 with 20 trillion parameters. That is, you know, they didn't really improve that much, but and was too expensive to run. They can't really use it.
Jim DowlingWell, in 20 trillion trillion.
Anders ArptegThey haven't really published, but the rumor is about 20 trillion. And um, there will probably be a few of these kind of super big models, but they will not really be used in practice, I would say. Only for very, very specialized use cases, and perhaps for knowledge distillation purposes, meaning that they can be used to build other more specialized open models or on-prem models. Yeah. Uh and um and that actually is then if it that's the case, then if we now let China and US spend an insane amount of money on these kind of big models and the infrastructure to train them and also to run them, perhaps. We can focus a bit more on running them, perhaps, or just doing the distillation to practical models that is useful useful for for certain countries or companies, then we don't need to perhaps waste some of the money that they are doing now.
Jim DowlingI I agree. I mean, if you're right, so Databricks is is run by a Swedish guy, Audi Gazzi, who's an American company. They're an American company, we know, you know. But so Databricks bought a company, I don't know if you remember, for two billion dollars who were training large language models. Yeah, mosaic, yeah. So they spent two billion. And um I think maybe a month after they they they bought them, they were buying the best open source models in the market. So they were going to dominate LM open source models. Unfortunately, Lama released was released by Meta, which just blew it out of the water. And they realized, hang on, can we train large language models? We may be a hundred billion dollar company, but can we be in that business? And they're not. No, they don't even care about it. So they're they're in the business of managing the data infrastructure around that. And I think that's that is a key uh a key infrastructure component we need to build these agentic systems.
Anders ArptegBut I think also I must say to there is a possibility for training, not the pre-training perhaps, right? Not the pre-training of the super big models, but the fine-tuning for small models that are potentially distilled from big pre-trained models and then adapted to the specific workflows that they need to be run for.
Jim DowlingI I I think it's a journey, right? If I'm if I go to an enterprise and I say, okay, you have private data or you have new data, yeah, I think you start with in-context learning and context because it's easy. You know, a new you can replace a model and once the infrastructure is in place to manage it, but then when you when you're when you're in industrialization phase, right, when you need to actually really make money, and the cost of this LM is a large part of the total cost of delivering the agent, then you go and fine-tune, then you optimize the hell out of it and get your cost down. I think but when you're building, you know, it that that would that would be for me, that would be more like kind of like, hey, I'm gonna do ML ops, I'm gonna start with experiment tracking, right? It's like I'm gonna just look at my loss curves and until I'm happy. But like, you know, that's not you need to build an a complete end-to-end ML system first, then optimize it. And I think fine-tuning is in for me, is in the same category. Build your agent, build everything, fine-tune then to optimize and and to to make money. Yeah, so but it is it is definitely an important part of it. And I don't think you need the money to do it, right? You don't need the same same resources, you just need a few uh CGX machines, or Spark. You got a Spark one here.
Anders ArptegWe have a DDX Spark, so yeah, we could just got uh my hands on a DDX Spark, which is cool. I think it's the only one in Europe or something, but then go to the case.
Jim DowlingI was shocked how small it was.
Anders ArptegIt is, right? It's insane. But yeah, that will be fun times. But I think it is a great opportunity. And if we just understood how to use it properly, you know, adapt it for our needs, forget the super big like pre-training and uh and then focus on adaptation and the applications and have you know, don't spend or waste time building the infrastructure use, you know, uh ready-made systems like Hopsworks. We can actually focus on the value here in Europe. 100%. That's what everyone should be doing. This should be, right? You know, um and then you know, uh the average European company could actually be much better than the American, and actually the European GDP could actually improve more.
Jim DowlingI mean, I'm I'm not you know, I think the the the fundamentals are are things like you know, education and uh which we're good at, right? We were very good. I mean, the just we don't um I think our universities are in any way inferior to American universities. And and obviously there's been a change now in terms of we're gonna get better quality more better quality students maybe from India and China than than before. So um capital has been a challenge in Europe, if I'm to be honest. Yes. So if you're you know a rich kind of person from the Middle East or from Africa or South America, and you're gonna invest a certain amount of it in in a high-risk investment, you you probably put in Silicon Valley and not in a European uh venture capital fund. That's that's something we we we we have a challenge with, I think. But but you know, that can we have enough money in Europe in the aggregate, and it's just not going there.
Anders ArptegI mean it's lovable. I should turn that uh negative, as you said, for for a potentially not the same capital as in San Francisco to a positive thing. Saying for them, they probably wouldn't be at all as successful if they would have moved to to San Francisco, to the valley. Uh and the argument was more or less that uh of course, well,
European Cloud: Can We Build It?
Anders Arptegcompetition is not as stiff here in Europe, so you get more attention, and you actually also uh being a European or Swedish provider has a big value in itself. So given these kinds of, you know, yeah, the capital is not in the same size, but there are other advantages that actually could be beneficial for European companies. So I hope.
Jim DowlingYeah, we're a deep tech company. We have, I think, 11 PhDs right now. And the average you look in LinkedIn and you see the average time people spend at companies. So we're seven years old. I think average time is about four or five years. So if you're building a really deep tech company, imagine you're building rockets like Space, uh Star, uh Star SpaceX. SpaceX. You know, you can't have a turnover staff leaving every year. Yeah, it's really hard. Yeah, um, and the same is true of database companies and so on. So so that there we have a big advantage here. Um, that you know, people tend to stay in their job and work on the deep tech things for longer. Um, so but that's just my experience. I mean, I guess there's other factors at play, but that that's helped us a lot, definitely.
Anders ArptegJim, um, so ending up with the standard question that we have here, I think we've asked it so many times now. We should write like a research article about this, uh, trying to draw some conclusion for what people think about the future when it comes to AI. And of course, we can imagine the two extremes here, either the dystopian kind of Terminator matrix matrix future where the machines try to kill us all. Um, or we could end up in a more utopian extreme where we actually have a world of abundance, and uh we've used AI to solve cancer and fix fusion energy, and we start to move towards some kind of world of abundance where the cost of products and services goes towards zero, and we don't have to work 40 hours a week for every person, etc. And we'll have a better life in some way. What which of these two extremes do you believe?
Jim DowlingI don't think it's set in stone, right? I think firstly, I think there's agency out there. I think the trend is not good. So from where I'm sitting, the trend is currently um an entrenchment of power. And um the biggest trend I see, which is not good, is that a lot of the techniques that that we've learned in in particular in computers where we we align on one or two standards, um, and then we, you know, like we have I mentioned it earlier, Uber, but a lot of companies go break things, move fast, and get things done. Those principles of how to build computer companies specifically, because people don't do that in other industries, those principles are applied in politics now in the states. That's crazy when you think about it. Is it crazy or is it good? Uh no, it's not good because well, it it's not good because um, you know, I think governments have to be accountable above all else. And um I think you know the the rules of business uh as applied in IT, IT companies, which we we which tend to kind of just you know, Peter Thiel said, I'll only invest in a company that's a monopoly. I'm not interested in anything else. Because with monopoly becomes uh you get powers that you would otherwise wouldn't have. So I think AI at the moment is um unfortunately uh helping entrench um some powerful interests in the states that that I think are not compatible with European democracy. And that's kind of scary. Yeah.
Anders ArptegUm so I do think Are we speaking of like the concentration of power that we're seeing?
Jim DowlingThe concentration of power and and the ability to use that power. So the worst thing, of course, is the language that the model speaks. So we we people normal people associate large language models with open AI, with uh grok, with so on. And ordinary people already understand that Grok speaks a different language to open AI. And you pick the model that talks the language you like. And for me, that's not a good trend in terms of um aligning uh people on common shared interests. And if you're if you want to entrench your power, you don't want people to align on common shared interests. You want to you want to to do the old British way where you break them up into smaller groups who then do a bit of infighting and then you you keep your power. So I don't think the trends have been great, but I do think, like I said, I think AI hopefully will commoditize intelligent commoditize to some extent, which means we will have an opportunity to use that to make a better society. And that's my hope. And honestly, that I you know, when I ran the company at the beginning, I was kind of like, yeah, we're building it, it's going to be cool tech and so on. And in in the last year or two, I've felt more of a calling that hang on, we need to build this infrastructure to help us have infrastructures that we can have our own values. If we don't have our own infra, we may not be able to keep our own values and define our own laws and the basic fundamentals of what makes a free society. And you know, if if if AI is being used to interfere in intellect in elections, which has happened this year in Europe, you know, German elections, Romanian and even Britain to some extent, um then that that's not a good harbinger for what's going to happen come in in the future. Um, because more and more people are using LMs to get their information. They're you know, you see that just look at the search trends in Google and they're dropping off. People are going to do agentic search or agentic chat chatbot-based features. And if that is all concentrating the power of a few people who meet for dinner in the White House every other week, that's that's not great, right? You know, so I think I think I think the future is not set in stone. I hope we can we can um uh go to, like you said, an age of abundance. That would be uh, you know, and an age of uh of of um you know a better society for everyone.
Anders ArptegSo okay, that was a bit negative, Anne, but um do you think it still will happen? Do you think Europe can continue to do it?
Jim DowlingI'm working towards it. That's my job. You are I'm trying to do my bit.
Anders ArptegUm and we're so thankful for that, uh, Jim Dowley. And uh I hope you continue to do the great work and uh provide the infra for the rest of Europe so we can actually do that properly, right? Thanks, Andrew. It's been great. Thank you so much for coming here and best of luck with the book and uh see you soon. Thanks a million.
unknownThanks, Yoran.
