AIAW E134 - Reflection 70B, Strawberry and other AI rumours

What if our AI models aren't as smart as we think? Ever wondered how AI intersects with entrepreneurship and the arts? Are bold claims in AI model development leading us astray? 

Join us as we uncover the surprising gaps between expectations and real-world performance of language models, featuring expert insights from Jesper Fredriksson and Robert Luciani. We dive deep into the challenges of model quantization and fine-tuning, and you'll hear first-hand about a fascinating case where a model trained for reasoning and reflection came up short

We tackle a recent controversy over a model that promised to surpass existing frontier models in reasoning, only to fall flat in reproducibility. The initial excitement quickly spiraled into skepticism and accusations of fraud. Through Jesper and Robert's expert lenses, we explore the steps taken to mitigate these issues and ponder whether the creator was merely misguided or intentionally deceptive. 

Our conversation also touches on the complex interplay between human cognitive biases and AI, the intriguing parallels between human memory and artificial intelligence, and the critical importance of navigating AI's transitional period with care. 

Tune in for a thought-provoking episode that challenges the boundaries of AI development and its future possibilities.

Follow us on youtube: https://www.youtube.com/@aiawpodcast

Chapters

• 0:00 AI Engineers Discuss Model Deployment Issues
• 5:31 AI Engineer Discuss Latest Model Developments
• 10:22 Understanding Compelling Model Development Claims
• 22:29 Model Reflection and Self-Teaching Approaches
• 29:38 Reasoning in Model Development and Evaluation
• 32:45 Future AI Models and Reasoning Capabilities
• 44:04 AI Engineers Discuss Human Brain Biases
• 53:01 AI Models, Memory, and Intelligence Evolution
• 1:02:13 Exploring AI Model Reasoning and Creativity
• 1:16:23 Predicting AGI and AI Hype Evaluation
• 1:23:14 Fear and Hope in AI Misuse

Transcript

Jesper Fredriksson: 0:00

No, not from Hugging Face Olamah. Olamah, yeah, I don't know if they pick it out of Hugging Face or what they do. But it wasn't so good, so I was like maybe it's the quantization that is too strict, too restrictive, and then I sort of left it at that.

Anders Arpteg: 0:18

But you tried a couple of prompts.

Jesper Fredriksson: 0:20

Yeah, and it's just like… and it did output the thinking and reflection thing. Yes. And then it's just like and it did output the thinking and reflection thing, yes. And then it's like it had the sort of the style of reasoning that you would expect from the story behind this, but it just wasn't good. In the end it came to the wrong conclusion.

Anders Arpteg: 0:35

So it sort of did the reasoning in how you would expect it to, step by step, but in the end it had the wrong conclusion step, but in the end it had the wrong conclusion so but it had to be at least fine-tuned then, I guess, because you didn't really prompt it to say thinking reflection kind of thing, it really did it by itself, right yes so it had to be some kind of specialized model at least definitely yes and it couldn't have been like, because in that case it was an api right, it was really downloading in a model and it can't have been some prompting in between or some kind of system instruction or anything.

Anders Arpteg: 1:09

It really needed to be the model itself. Exactly it was trained to do it.

Jesper Fredriksson: 1:13

Definitely so. That was the first part of the story, and then people were complaining that it's not possible to run it off of the deployment. They had to run it off of the deployment they had, and somebody was saying that the way it seemed to not make sense. Somebody was saying that it's actually based on LLAMA 3, not 3.1. Right, and at that point it seemed like odd. It was also in the beginning, when the initial benchmarks were released by matt schumer, somebody was complaining on twitter saying uh, you're quoting gsm8k, I think it's called the benchmark above 99, which is known to be impossible. How does that? How does that? Why is that? And he had no good answer for it. So it's. It started to smell a little bit quite early actually he was.

Robert Luciani: 2:08

He was very active, uh, early, I don't know. He seemed to have posted in a couple places, at least um near each other in time. I don't know if he went first to x and then to reddit. I saw it on reddit first and uh, he was replying to people in the threads very quickly. And I saw also in the threads people immediately downloaded it from Hugging Face and Olam and said I'm not getting it to work the way you're describing it. And he would reply and say, hmm, that's strange, there's probably something wrong. And immediately I was like oh no.

Robert Luciani: 2:37

I mean, if he needs to go into crisis management mode that quickly, something's not entirely right. And I'm always skeptical of when people say the bench, like if somebody has a really good benchmark, cool, that's a sign of something having potential. But you know, it's easy to train on benchmarks and stuff, so like it doesn't mean anything. And so I was thinking, okay, so these guys have great benchmarks, let's see what it's about. And they were really exceptionally good. So I was like, well, maybe it's a little bit better than usual, but you never know.

Anders Arpteg: 3:07

But did you smell something funny about it? Because I didn't. I was really excited when I first heard about it and I thought, you know, it made sense how they described it. I mean, I have head canon.

Robert Luciani: 3:17

So when he says that the benchmark is 99%, I'm like that's in his perfect world and it's probably closer to 90%. It doesn't bother me that he says 99%, because I can temper my own expectations, but the approach that they had sounded super compelling because I was like, yeah, this is how I would approach it, this is how chain of thought sort of approaches it and this is how everybody's sort of thinking is. The next step in attacking this kind of….

Anders Arpteg: 3:42

It's adding some kind of reasoning, you know.

Robert Luciani: 3:43

Yeah, a strawberry cinnamon kind of way. Yeah, yeah, for sure.

Jesper Fredriksson: 3:48

I liked when somebody I think on the first or second day somebody posted just a ranking of all the different models, so it's like the big ones, and then the post said so it's Google, anthropic, meta and Matt from the IT department. That was a cool thing and at that point most people actually believed that it was true. It was funny.

Anders Arpteg: 4:14

Super interesting times for sure. Yeah, we have a theme today to speak more about reasoning in LMS and reflection tuning and other approaches for that. But before we go into that, let's just quickly introduce yourself. So, jesper, if you were, can you just quickly introduce who is Jesper?

Jesper Fredriksson: 4:33

Fredriksson yes, I'm Jesper Fredriksson. I'm currently working as an AI engineer at Volvo Cars and I have a background recently in data science and, before that, medical imaging and brain research.

Anders Arpteg: 4:53

Cool, and I think you also are a person that is working in production with.

Jesper Fredriksson: 4:58

LLM.

Anders Arpteg: 4:59

So, I think you're a very knowledgeable person in this space. I have some real experience, which I think is very rare and really valuable, so very happy to have you here.

Jesper Fredriksson: 5:09

Thank you.

Anders Arpteg: 5:10

Robert, please introduce yourself.

Robert Luciani: 5:12

Yeah, I'm Robert Luciani. I'm a computer scientist, I have a background in high-performance computing and everything from GPU programming to cluster systems and that kind of thing, and nowadays I do consulting and helping companies speed up their generative AI ambitions. And although I think the technical stuff is the most fun, sometimes it is a little bit entrepreneurial and that's also something that right now is a little bit difficult sort of conceptualizing these things that we talk about today.

Anders Arpteg: 5:49

You're also this kind of extremely multi-talented kind of person that's really envious about, you know. You're a very you know talented musician as well. You do art in various forms and you're a serial, serial entrepreneur as well.

Robert Luciani: 5:59

Right with companies, right there is an irony in that everybody thought that the first thing to disappear with ai was going to be the boring sort of automated business stuff and as it turns out, the art stuff is the first thing to sort of be really accelerated by ai. So for me it's fun, but uh, I guess it's funny how that sort of turned out I think you have the best data science.

Anders Arpteg: 6:20

You know song and music video of all. So if anyone were to Google for the bigger, the best. This is something I still love to watch and listen to. So it's a true pleasure to have you here.

Jesper Fredriksson: 6:36

I'm a big fan of yours as well. You have to say the name of the song so people can search for it on YouTube, the biggest the best, Robert Luciani yes, true, and partner the best, Robert Luciani yes.

Anders Arpteg: 6:45

True, and partner of the AI framework, right? Yes, and so many more companies. You have something called what was it Nerve Dynamics or something.

Robert Luciani: 6:53

Yes, and that's I mean, that is a software platform company where, you know, now there's a lot of tools for big data that you can buy that's sort of out of the box. But there's a lot of tools for big data that you can buy that's sort of out of the box, but there's nothing for generative AI. We're sort of building the ship while we're sailing it, and so we're trying to give this package where people can get started with a software stack. And it's funny because there's no best practice around a lot of these things like how to do RAG, how to do safety and all that kind of stuff. You know, we think we know a good way to do it, but we're all trying to figure this out while we're doing it. And I guess one thing that's interesting is you can look at OpenAI and some of these guys and see how they're doing it and try to copy them, but it's not entirely evident that they're doing it the best way. I think everybody's trying their best right now.

Anders Arpteg: 7:41

And tooling around these kind of technologies is really immature right, yes, yes. We need to have expertise in being able to do it properly, but you also have these other companies. What? Meditech or something?

Robert Luciani: 7:53

Yes, that's getting a generative AI in healthcare. And again, it's not necessarily the technology that's immature, because I think if the tooling is right and you have feedback loops, with guardrails and lots of things, you can get good applications. But it's very difficult for people to narrow down and formalize exactly what it is they're doing in their line of work and then just apply it to that. But we'll see.

Anders Arpteg: 8:21

Awesome, well, true pleasure to have you here as well, and you're super knowledgeable in lms and the latest ai tech and also a big julia fan as well, which is kind of interesting I have a feeling.

Robert Luciani: 8:34

No, I don't know, it's one of those. It's like bsd, right, you just hold on to it for a long time, bsd unix, but uh, I wanted to get back into it, but when I went back they still have a command line installer and it's still old and crusty but, I don't know. Netflix uses them. They're still okay.

Anders Arpteg: 8:51

I'd love BSD as well. Anyway, it's going to be a bit of a more technical kind of podcast than usual today. So please bear with us, the people listening. We're trying to potentially explain some topics, but I'm thinking, you know, let's go a bit deeper than usual. I think I'm excited.

Anders Arpteg: 9:11

Awesome and let's start by perhaps giving a bit of a background to the whole Reflection 70B model that was released I guess more or less exactly a week ago now right, Pretty much and it was a person, Matt Schumer, that actually released the model and they said basically it's based on LAMA 3.1, the 70 billion model and it was beating basically any kind of frontier model, including the closed source big chat, GPT, GPT-4.0, 4.0, gemini, claude and everything into a number of benchmarks and it's going to be open source and it sounded like awesome yeah, it was very lofty claims from out of nowhere and that it beats those giant models in almost everything right yeah, and then.

Jesper Fredriksson: 10:03

Then he also said that we're going to train 405B Lama 405B over the weekend.

Robert Luciani: 10:09

To be godlike.

Jesper Fredriksson: 10:10

Yeah.

Anders Arpteg: 10:12

And he also said it's going to release a report next week and they're working on the 405 billion model as well, based on Lama 3.1. And it was. You know, I was certainly.

Robert Luciani: 10:24

I still not sure if it's fake or not, why did you have reason to believe that it was compelling? So you saw that, I mean, those are huge claims and normally you would say yeah, right, but there was something in it that made you say aha.

Anders Arpteg: 10:37

No, but I think we all know that. You know today's LLMs. They are really good in storing and recalling knowledge, but they have very shallow kind of reasoning techniques and, as like Demis, hassabis and others and even Oknei are saying, they are now trying to add reasoning to this kind of model. So any way or any kind of technique that is starting to add some kind of multi-step reasoning to a model is something that's very compelling, and I think what they actually said is based on a paper that's also called self-reflection. Before it was very similar to what Strawberry and Q-Star the top AI are claiming to be releasing soon. So for me it had a lot of connections to how reasoning can be added to LLMs going forward, so I bought it.

Anders Arpteg: 11:27

And I'm still not sure if yeah, but anyway, you know, if we just continue what happened after that? Then they published the model, you know, on Hugging Face and on other sites as well and on OpenRouter and so forth, but no one was able to reproduce the astounding claims they were doing. So being able to beat these kind of frontier models was not possible really to reproduce. And then the whole kind of Twitter storm or whatnot started to appear and people were saying he is a scammer and is frauding. Whatnot started to appear and people were saying he is a scammer and it's frauding. But I, I still don't. I I very like I don't want to blame people. I have the benefit of a doubt, I I don't.

Robert Luciani: 12:13

It was, it was so stupid of him to to publish and say something like this, unless he truly believed it himself there were a couple of sort of um mitigation steps that he was taking that added, a couple of sort of mitigation steps that he was taking that added to the unfortunate sort of bad look that they started collecting, which was when people said I downloaded it from Hugging Face and I'm not seeing what you're claiming. He was like, well, Hugging Face messed up the weights or something, and you're like come on, and so he did a couple of those doubling down.

Robert Luciani: 12:46

And so he did a couple of those doubling down which made it seem even worse, so that when he finally said, okay, probably there might have been something on our side that we….

Jesper Fredriksson: 12:54

I think the most incriminating thing about it is that when he couldn't produce the new weights or it didn't add up for some reason he couldn't produce the new weights or it didn't add up for some reason he decided to to give out a private endpoint to some people who were doing the the evaluations, and then people were looking into what's really behind that. So people were really clever in how to inject prompts into it and they could see that it's most likely a cloud, most likely cloud 3.5 Sonnet that it's using.

Robert Luciani: 13:25

It's just calling another API on his side and if that's true, then it's like and it's worth noting, though I'm glad that you said it's most likely cloud, because it is really hard to discover what model it is you're running because sometimes one model will answer that it's another model because it's been trained on that model's data or whatever.

Jesper Fredriksson: 13:43

So can't know for sure, but it did seem very suspect. Yeah, yeah, I like the technique they tried to.

Anders Arpteg: 13:49

They tried to get the model to say cloud yeah, and then it just came out as an empty string, so they were thinking like he's just doing a regex on on cloud I also saw a pod where matt actually was interviewed by another person and it was very open and he said very compelling things that oh no, we made some mistakes, but they are really going to reproduce results, they're going to retrain the model and it's strange. The first claim was basically they uploaded it wrong or hugging face, messed it up somehow and they're going to re-upload it but it didn't help. And then they said they're going to retrain it and we still haven't seen the results of that. But in any case, I cannot see it was like malintent. I cannot see that he on purpose was trying to fool anyone, because he must have realized it's going to be detected directly, which it has, and it's going to backlash on him in a very, very negative way.

Robert Luciani: 14:49

so I don't think a technical person would have suggested that retraining would be one of the mitigation strategies here it has to have been something that he wasn't really understanding himself yes, right, yeah but the question is here why are we even here discussing this right people? People make up crap all the time. It's specifically that this one was so believable. Everybody wanted to believe it and was very excited about it. It would be awesome.

Robert Luciani: 15:12

If it were to be true, yes, it would be an amazing achievement and everybody's waiting for this kind of breakthrough, and this seemed to be the most plausible way of achieving it, like you said, similar in spirit to how we imagine strawberry might be working, and all the approaches seem to have something in common in terms of how they think the next step in reasoning is.

Anders Arpteg: 15:40

Yeah, and just thinking about, you know what could be the causes of this. You know one thing that he could have done wrong, you know if he had no malintent but simply didn't have the knowledge how to do science properly. I mean he could have been training on the test sets, which would explain a lot. You spoke about the GSM 8K right. It says grad school math, something like that, and they basically had 99 plus percent accuracy. And some people say that we know that more than 1% of the data set is wrong.

Anders Arpteg: 16:19

So to say that you have 99% accuracy means it has to have memorization more or less.

Robert Luciani: 16:24

Oh, I think people were reasoning about that in the thread in reddit. They're saying, wow, this is so impressive because what it has to have done is realized that the training data was wrong and taken that into account to get a better score.

Anders Arpteg: 16:38

This is like god-like capability yeah, yeah, if it's true so, so okay, so in some way, learn that. Okay, I know the proper answer is this, but I know the answer is wrong, so therefore I learned to respond.

Robert Luciani: 16:51

Maybe a little bit. You know, like people say, that if you train a chess machine on a thousand ELO players, it'll play at 1600, or something like that that. This is something similar. All right, right. It learns to make errors similar to the dataset in some way.

Anders Arpteg: 17:06

But in simple terms it could simply have memorized the test dataset and that could be, the easy answer to it, or he actually, with intention, actually added some of the test dataset to the training data.

Jesper Fredriksson: 17:16

There are some subtle ways. So Jim Phan from NVIDIA posted that it's very easy to inadvertently or with malintent sort of make it believable that it's actually true, even though you train on the benchmark. So, for example, just generating synthetic data that looks similar to the benchmarks and training on that, that's one way of doing it, and it could be something like that somewhere in between the sort of altruistic version of it or the really malintentious version of it. It could be somewhere in between that where he thought that maybe this is the secret sauce that will crack it.

Anders Arpteg: 18:02

And one other source of you know what could really be happening underneath is he was using some other model, you know, using Claude or something as a backend and, just you know, rephrasing the output in some way and perhaps instructing Claude to put up these kind of. We haven't really explained what it does. Perhaps we should go back to that saying you know, what is really special about this model?

Jesper Fredriksson: 18:28

And I can start and please correct me to fill me in.

Anders Arpteg: 18:31

But what they really did with the reflection tuning which he claims then, is the novel kind of approach is that when you get the question or some kind of prompt, you potentially add some kind of thinking tag in the beginning saying you do this kind of chain of thought, kind of reasoning, ask it to really, you know, explain your thinking in how to come to the answer.

Anders Arpteg: 18:51

And that in itself will kind of reasoning, ask it to really explain your thinking in how to come to the answer, and that in itself will of course improve the quality of the answer.

Anders Arpteg: 18:57

But then potentially it can also, at least they seem, to do some classification, saying if it's a really hard problem, they also add a reflection tag after the thinking tag saying you know, I think the answer is wrong and you need to reflect on this and see how can we improve on this. So then they can add a reflection which can basically have some kind of retraction capability of saying the thing I said before is wrong. I think you should do it this way, and then he can change his mind and come to another conclusion and potentially have multiple kind of reflection tags and then, when it's satisfied in some way, it adds an output tag saying this is my answer. So it basically had this kind of number of steps in being able to come up with the output answer. First, some kind of thinking kind of traditional chain of thought kind of thing, some reflection tags where you can retract or reconsider what you said and then come to the answer.

Jesper Fredriksson: 19:58

Is that fair? I'm thinking of it as something of a bootstrapping method, where you start with some model and then you iteratively refine the training data and while retraining you get actually better. You could do that with some kind of what do you say like a better model, like maybe a cloud, or GPT-4.0 was the model behind the scenes that was doing the decision. Is this part of reasoning true or not? And if it's not true, then they just change it.

Anders Arpteg: 20:32

Right, because if you take GSM 8K, for example, you know the answer, and in the beginning you can just do the question without having the answer.

Anders Arpteg: 20:41

Then you can check it and see is the answer correct and you can see is this correct or not. If it's correct, then fine, you can add it to a data set. So what they basically did, they also did fine tuning. So it's not just a prompting approach, it's really trying to create a new data set, and they had this partnership with another company called Glaive, which apparently he's invested in as well. So some people are thinking that that's the reason behind it all. But then they add these kinds of data sets. When it gets the answer correctly, it just adds saying okay, this is the proper answer. When it gets it wrong because you know if it's correct or not then you add the reflection thing and once it got it right, you add that piece of data to the data as well, and then you start to fine tune on that, start to fine tune on that.

Robert Luciani: 21:32

I think it's worth saying for the audience here that I think this stuff is particularly confusing and why we're sitting here musing on what's actually going on. Because you know you train these models in the pre-training step on just predicting the next token, but what you really have is like multiple layers of meta optimization and those things are just implicit and we're trying to guess. Like what is it really sort of tending towards in this overall optimization thing?

Robert Luciani: 21:53

so we're guessing here and, yeah, we're guessing if I'm thinking about, like what a transformer does? You have first the decoding of the tokens into some kind of latent space representation and then some kind of computation happens there and then it has to be encoded again into some kind of token, serialized token stuff, where it needs to sort of take into account also that this needs to be presented in English, you know, in a sequence and stuff like that. And so, and you have 40 steps to do that or whatever the depth is for your particular transformer. Yeah, and so imagine you have a question that takes more than you know. The first couple of layers and the last couple of layers are just dedicated to the serialization and deserialization and encoding steps, and you have those little soft layers in the beginning, in the middle, to do the actual thinking, and it's thought for 20 steps, and then it's like, oh shit, now I have to output the first token. I haven't finished thinking.

Robert Luciani: 22:47

Obviously it doesn't do that because that's not what it's been trained to do, but that's what we're trying to sort of mitigate here, that it's allowed to look at its tokens and say, ah yeah, now that I look at it it looks pretty dumb, and I think we often forget because we're so used to these chat versions that we imagine the output like a fixed sort of block of text, but they're actually trained to just go on forever, right, and we fine-tune them to do something else and I guess this reflection, my impression is that you can instrument this bot to sort of cut it off when you think it's done enough, look at its own input and do all that kind of stuff, and really what the reflection is is just a fine-tune to give it a bias to be aware of that process and sort of that input format and sort of force it in that direction, perhaps also not to be as biased as it was originally, to just, you know, continue saying what you said in the beginning, right right To question itself.

Anders Arpteg: 23:42

Question itself, because I listened to the podcast where he Matt was actually part of it and he got the question about what, if you simply prompted to do this, what's the benefit of really fine-tuning it? Why do you need to fine-tune it? Why not simply, you know, prompt it to do the thinking, reflection and output? And he said you know, if you don't fine-tune it, it will keep saying that what the original you know words that you have predicted said is true. So it's really hard for it to say that something I said in the past was wrong that sounds like kind of answer, that is, that comes from experience.

Robert Luciani: 24:20

You know what I mean. So matt didn't do the experiment himself. He spoke to some engineer and the engineers is like we have, uh, found evidence that this is the way to make it happen. And matt's like right and so like he has no, but what he's saying is totally true and it's derived from actual experiments. It's just that his claims have nothing to do with that right.

Robert Luciani: 24:37

We found something promising and he's just reporting on it and there's reason to believe that that's a good approach, because we know, for example, that models that are fine tuned with quantization awareness also perform better when they actually are quantized. So you know, fine tuning with awareness of certain things, we know that's a good approach.

Jesper Fredriksson: 24:56

SL. What do you think about the approach of? It's rumored that the strawberry approach is something that happens at test time, so we're doing some kind of search in sort of real time when the user is asking the question. Is this approach in any way similar to that or is it just a better prompting built?

Robert Luciani: 25:17

into the training. We should say what we think. Strawberry is right.

Jesper Fredriksson: 25:21

Yeah, yeah, good point. Let me start over. We're thinking we don't know this, but there are many… it's already. Now. It's possible to take what comes out of a large language model and somehow vary the sample from the output. And if you take it one step further than just sampling from the output and taking something like the mode or the median or whatever, if you take it one step further you could organize what you get out into a tree. That's a thought way. If you, if you're smart in how you, in how you sort of go through the tree, you can do something like a multicolor tree search. So already by doing that you can sort of get something like a test time compute. It's like a poor man's version of this, and that's what I'm thinking is what's in the strawberry model, something like multicolored research.

Anders Arpteg: 26:23

So it's either that you know, then it goes more into the Q-star, which Q-star is also another name for strawberries.

Anders Arpteg: 26:26

So, these are similar things, but Q-star has a multi-faceted kind of explanation. One is the reinforcement learning which you are basically saying here, building up like a search tree and you have some kind of reward that trickles back and you can calculate some kind of credit to the actions you're taking. But QSTAR can also be interpreted another way, which is at the star. But Q-star can also be interpreted in another way, which is that the star should be pronounced at S-T-R-A-R, which is basically meaning self-taught reasoner.

Anders Arpteg: 27:08

So there is a paper, you can read it and find it's called self-taught reason. When you use the model and going to train a model, you you do more than autoregressive kind of nature. So you you try to, for one, find the, the motivation, the reasoning for why the answer is is there and you add that to a data set. So you add synthetic data, so to speak. It's self-taught to the data set itself and that will improve the model continuously without human input. So it's self-taught and is going to improve by simply using its own input.

Jesper Fredriksson: 27:47

And it can also realize if it makes an error, because then it doesn't arrive at the right conclusion at the end, and then it can fix the reasoning.

Anders Arpteg: 27:55

So when it actually comes to a conclusion saying this is actually the wrong answer, it can question that and change the reasoning. Then you don't add the wrong motivation, you just add the right motivation and that way you can self-taught or teach yourself in some way to become increasingly good at answering questions. So this is the self-taught kind of reasoner approach, which is very compelling because then you don't need to have human annotation and it can basically use synthetic data to continuously improve yourself.

Robert Luciani: 28:28

I have two points that I feel like I have a really hard time holding on to, so I need to let them out before I let go of them.

Robert Luciani: 28:35

One is with regards to whether we're trying to find a correct approach or just an approach that converges more quickly, whether, you know, using existing approaches and adding more data would also work, or whether it's just sort of a dead end. The second one is with regards to sort of reasoning as a philosophical concept. You know Wittgenstein was saying that he was only wrong on a couple of things, but otherwise quite good. He said that you know, the rules of math are enough as is, but not are not a guarantor of the students not making mistakes. And adding more rules is not going to prevent the student from making mistakes, because the rules of math are sufficient as is. The only way to prevent mistakes is for the student to keep practicing, and that kind of stuff there's no like number of layers to prevent mistakes.

Robert Luciani: 29:30

So what are we going to do? Kind of stuff. There's no like number of layers to prevent mistakes. So what are we going to do? And he mused on that for a long time, Like how can we guarantee that we're reasoning correctly, and that kind of thing.

Robert Luciani: 29:38

I think this is sort of similar in spirit and, on the one hand, what I would grant is that being methodical is probably a good approach in general. On the other hand, it's not clear to me that there is a correct way to reason. The only exception to that is when you and I are in agreement. Here's how we reason, let's do it. And the only place where that happens is in formal systems and games, aka mathematics. So, for example, with formal proofs and this is what I think Ilya Suskovor was saying we can have a math proof and then we can ask a model to look at it and say does that look reasonable? Yeah, and then we say it's not, because this proof is not correct and you can correct it and stuff. But if you say, you know, I asked you the question what is the best color? And you're like blue, how can we sort of do that same?

Anders Arpteg: 30:21

game. That's a spectrum right From the formal mathematics to some kind of more philosophical or some more subjective kind of areas where there is no like right answer. But there's certainly something in between. Yes, and I think for a lot of questions that you have in these kind of benchmarks, or you know everyday life that humans use, we can at least come to some conclusion saying this is probably wrong.

Robert Luciani: 30:47

Yeah, I can't say it for sure, but there's something in between, and that ties into this convergence thing that I was thinking of, where what we have is the human body of knowledge. That is our best effort to be structured and we're running out of that. And how can we create synthetic data, given that what we have is fixed? Either we wait for the models to converge on what we have, we try to accelerate it by giving it really really good data, which is what PHY is sort of trying to do, like cut away the crap and give it really good data, and that seems to be a good approach.

Anders Arpteg: 31:18

Yeah, but I guess also one way to figure out if it's the right answer is simply to look for consistency in some way.

Robert Luciani: 31:25

Right.

Anders Arpteg: 31:26

Right, that's true. So it doesn't. You don't necessarily need to know if it's right or not. It's simply that similar kind of questions leads to the same kind of conclusion and you have a world model. I'm moving a bit to Jan-Li Kuhn kind of world model thinking here, but some kind of understanding how the model works. If it's consistent with that, then you can start to reason that this is probably true or not.

Robert Luciani: 31:49

That's an excellent point that scientific knowledge should reinforce and be consistent with other scientific knowledge.

Anders Arpteg: 31:57

Okay. So, robert, do you believe the kind of? Do you believe Strawberry, q-star? Of course we're all guessing here. It's pure speculation, but what do you think really Q-Star and Strawberry is about? Is it more like a multicolored research kind of thing? Is it more kind of self-taught reasoning kind of thing? What do you think really Q-Star Strawberry is about? Is it more like a multicolored research kind of thing? Is it more kind of self-taught reasoning kind of thing? What do you think?

Robert Luciani: 32:17

I think it's just a way of allowing it to think for longer and, um, really what we need is more and a better way of generating data, and a more power, efficient architecture that is designed around the kinds of goals that we think are interesting. I think you and I have talked about how, you know, even dense networks can, in principle, do anything we want, but right now, what we want to do is maybe to boil it down to really specialized things. I actually think that the cursor is a good example of it. You know cursor? I think people just explain what cursor is. Cursor is a good example of it.

Anders Arpteg: 32:50

You know Cursor. I think people just Please explain what Cursor is Cursor?

Robert Luciani: 32:52

is a branch of Visual Studio that does sort of co-pilot stuff, where the AI helps you code, but in just a much better way. It's more integrated and stuff.

Anders Arpteg: 33:00

But the cool thing here. Only on Mac, by the way, so not for Linux. So I'm not using it. But okay, damn.

Robert Luciani: 33:15

I mean cool about cursor is it looks like it's all seamless and there's ai in the background, but in fact there's like five little or even more ais one for the embedding, one for the thinking and stuff like that and that's probably the way it should be right. It shouldn't be one big ass ai. That's super expensive. Why would you want that? Our brains have little dedicated functions they do, and so I imagine that the ai of the future is also going to have little components that are smaller and dumber, more specialized, but helping each other out. So, yeah, this Monte Carlo tree searches or whatever it is, is probably what I also think Strawberry is, and I think that the real future is in building something. I don't know if they call it neuromorphic nowadays something more in that direction.

Anders Arpteg: 33:48

That's another interesting. What do you think, jesper, what do you think Strawberry and Q-star is really all about? I already said it, the multicolored research or some other form of exploration and test time Combined that is then used to train as well, or is it just the test time or the inference time compute?

Jesper Fredriksson: 34:11

it could be both. I mean I'm thinking they probably update the training all along. It's interesting to see that they're now I'm talking about open ai.

Jesper Fredriksson: 34:22

They're outputting a new model every time somebody beat them on the benchmarks, so it seems like they have some way of iteratively releasing more or less the same model, and I'm guessing that they they do some way of iteratively releasing more or less the same model and I'm guessing that they do some kind of fine tuning all the time where they employ something like could be star, it could be verified step by step, which is similar I would like to refine my answer, which is that we are sitting here imagining Skynet but OpenAI is just a business.

Robert Luciani: 34:51

And there's a class of problems, which is monetizable, that people value, for which this approach could be fully sufficient for doing very smart things, for example, generating verifiably runnable, bug-free code and stuff like that. And so maybe this Strawberry or Qstar is just very good at certain classes of problems that people would find very valuable. And so we're sitting here thinking like, oh, this is the next generation AGI. Maybe it's not even about that, it's just about a product that OpenAI can monetize, which makes sense.

Jesper Fredriksson: 35:23

I think it's something that you can use if you think that your question will be benefited by it. I don't think that if you ask what color is your best color, I don't think you your question will be benefited by it. I don't think that if you ask what color is your best color, I don't think you want to ask that to a story.

Robert Luciani: 35:36

I think it's for a very specific class of questions.

Jesper Fredriksson: 35:38

Yes, I've actually seen some reported leaks today on X where they're saying this is evidently part of the documentation that somebody has seen from OpenAI. I'm not sure if it's true, but they have actual use cases where they point out that this is where you would want to use this model called something with release date. Today it has something like GPT-01 preview and then today's date one preview and then today's date, and then there's part of a documentation where they're outlining something like a scenario where you want to create an agent, a customer care agent, where they take some kind of knowledge base and out of that they create some kind of categorization or something I don't know, that that would take in your, your knowledge base and make it into something that's usable by data. So they have something like a use case where you would want to use it. So I think that they're trying to teach us. This is something else.

Anders Arpteg: 36:42

I think it's interesting. You know, sam Altman has himself said that they want to have a more of a continuous kind of very iterative releasing of new models going forward and they seem to be releasing like once every month or something. I just saw myself, you know the. I got an email from an opening I saying they're going in second of October release a new GPT-4 O model with that release date and then you can see all the you know the news articles saying in two weeks they're going to release Strawberry, and I've heard that for a number of weeks now but who knows when it's going to happen.

Anders Arpteg: 37:17

I think actually they are experimenting already with more reasoning in GPT-4.0. It's just that it's happening for a limited set of perhaps users or a limited set of type of questions where they are trying to see that now it actually has to reason a bit more For very anecdotal kind of reasoning. I can see if I ask a simple question you get an answer much faster than if you ask a much tougher question where it has to take some action and go and fetch something on the web or whatnot.

Robert Luciani: 37:48

I feel like there's two kinds of capabilities that they are stealthily introducing. The first one I don't get those news updates via mail, so I just discover these things. For example, I asked it to generate a PowerPoint for me and I think a while ago it would have given me the Visual Basic script that I need to paste out myself into PowerPoint, but now I just generated the PowerPoint using python and gave me the file right. I was like wow, you just gave me the file right here in the chat, um, the second thing I think and this would probably be if they have big like enterprise ambitions is to handle rag for you, right, um so, and maybe strawberries tied somehow to that. Like, given your knowledge, give me an authoritative answer on this question or something like that, and let it sort of think longer on that.

Anders Arpteg: 38:34

yeah, I mean cool I remember still, if you just go and think about you know what are the future incoming months going to be in lms. I remember listening to demis hasabis. You know from deep mind and now google, uh brain and all of google ai. He said you know very clearly that we and I think this is a very nice way to put it we know the LLMs of today are really good at knowledge but poor at reasoning. They can do some shallow reasoning through the layers and the tokens that they have, but it's not advanced reasoning.

Anders Arpteg: 39:06

If you take the AlphaGoes and AlphaZeros, that could beat any kind of human in chess or go or StarCraft or whatnot. They are really good at reasoning and can do this kind of multi-step reasoning in a way that's much more advanced than any kind of human ever could, but they're really poor in knowledge. So we have some models that are really good in reasoning, others that are really good in knowledge, and what would the obvious next step be? Well, it would be to combine the two. This is literally what Demetri Sabis said, that this is the focus for Google right now to combine the power of AlphaZero with the power of Gemini, the LLMs, and it seems obviously correct to me.

Jesper Fredriksson: 39:47

Yeah, yeah, everybody's sleeping on, uh, on, uh. Google's new models uh they're. They're also going to come with something, so they are. They're lagging a bit behind, right they're a bit behind, but, uh, I'm guessing they're. They're, I think that they're catching up. It's like the, the top of the top of the line models. They're more or less the same by now. It's like google, meta, open, ai and tropic. They're more or less the same now, and I I think the one that releases the first will be the one that that gets poor mistral.

Robert Luciani: 40:18

Are they not in that list? And the other little gpu poor companies?

Jesper Fredriksson: 40:22

no, they're close to it, but they're not there. You know what I'm going to add a question about.

Anders Arpteg: 40:25

You know, is the gap between the big tech giants and the small one, like mistral, going to increase or decrease?

Robert Luciani: 40:32

You can think about that. Yeah, definitely I think and also I guess that ties into just out of as an aside is sort of Europe in the whole scheme of things.

Anders Arpteg: 40:43

Yeah, that's another interesting one. Ok, so, but if we believe that some kind of multi-step reasoning is going to be added to LLM, I think we can all agree that some form of that is going to be added to LLM. So I think we can all agree that some form of that is going to happen and it's already happening. But how would you, if we just go back to that, what is the proper way to think what reasoning really is? Because some people are claiming even you know the lambs are not doing reasoning at all. Perhaps we could go in more philosophical, saying what is really reasoning? Robert, do you have any thoughts about that? Sorry for the very philosophical question. No, I love this question. You spoke about Wittgenstein and whatnot, so you should.

Robert Luciani: 41:20

I think that what we're not doing here is discovering what reasoning really is. What we're not doing here is discovering what reasoning really is. We're simply making sure that the three of us are using the same word to describe a phenomenon that we're sort of identifying. So, you know, you and I look at an animal and we say, look at that thing, it's doing that behavior. Are we looking at the same? But yeah, yeah, this is the behavior. We're all in agreement on this specific behavior. What do we want to call it? Do we call it reasoning? And you're like no, no, let's call it something else, because reasoning is.

Jesper Fredriksson: 41:54

I want to use that word for the thing we do whatever.

Robert Luciani: 41:55

I just feel like that's a game of semantics, so we're not actually gaining any insight by trying to figure out what reasoning is uh, and so it doesn't really bother me whether or not lms are reasoning or anything like. I don't think that that's a helpful question.

Jesper Fredriksson: 42:11

But what is the root of it? What is it that we want from the better model?

Robert Luciani: 42:16

Okay, so this is like the foundation of machine learning in general. You have something that is so complex that you don't really understand and you want a system that self-teaches itself, that stuff, whether it's how to win the game, what the rules of physics are, or whatever the case may be.

Jesper Fredriksson: 42:35

But that's the method of getting to something.

Robert Luciani: 42:37

Yes, that is like the machine learning approach, and the question here is can we get LLMs to learn the universe Not LLMs, but like can we get these advanced models to learn the universe automatically, Because we haven't been models to learn the universe automatically, Because we haven't been able to figure it out, and then you know if you want to, if it does it in the same way a human does, or whatnot. I don't think it's super important. There's probably a lot of learnings from nature in terms of energy efficiency.

Anders Arpteg: 43:02

that we can steal and then the rest is unimportant. I think that's an interesting point. You know, if we take the human brain, we know it has a lot of neurons in it. It's like 86-ish billion neurons, but the number of synapses is at least like times 100 or 1000. So that means it's significantly more than the biggest kind of number of parameters that a model can have today. Still, we potentially have a more powerful way of training LLMs than human brains has, because you can feed in all the data of the internet and it can actually use it and come to some kind of knowledge or fact that you can and process a lot in parallel.

Anders Arpteg: 43:40

Yes, what you cannot do with LLMs today, at least not in the training phase, is to very, very quickly learn how to do something Because a human. If I tell you, jesper, this is the way you should chug a beer and I give you some tips. You can, by a few examples, learn how to do it. Can I challenge that? I wonder.

Robert Luciani: 44:04

I think the human brain has a number of biases that serve it very well within the context within which we operate. So, for example, we have some parts of our brain that are specialized at navigation, and in particular they're specialized in two-dimensional navigation, because the surface of the earth is two-dimensional, and so if I were to give a couple of examples of rotations of spinners, it might not feel super intuitive to people, whereas Euclidean 3D stuff would feel super intuitive, and so I do think humans are super intuitive on human stuff. On throwing rocks in the savannah, I'm not sure it extends to general cases.

Anders Arpteg: 44:44

Yeah, Okay, so perhaps there is a bias in our brain for the Euclidean kind of physics that we have in the world. And if you want to learn how to ride a bike or a car, you know, child, you know are learning the physics like the only person I say you know? In a very self-supervised kind of way yeah and you can use that kind of you know, understanding of the physics of the world to to learn new tasks much more efficiently.

Robert Luciani: 45:21

Yeah, I think, um, there's a lot of cross-domain knowledge here that is really beautiful, from biology to physics to computer science and everything, and everybody always makes fun of each other for not knowing these things, but it's not easy to know.

Robert Luciani: 45:33

For example, I was chatting with chat gpt trying to understand how a baby deer can learn to walk as soon as it's born, and apparently there are these nerves that oscillate and there are harmonics in the oscillations that you know. They just happen to move you in a way that becomes like a gate, and, and a gate is a very knowable thing. For example, we know the best way for a spider with, or an insect, even a spider arachnid with eight legs. We know the best way for it to walk, and you can encode that with a very minimal number of neurons that oscillate in a fixed way. And so you think that it's this oh, baby deer doesn't need a lot of training data to walk. Well, no, but it doesn't need more than a couple of neurons either, and that's something that evolution could have created.

Anders Arpteg: 46:24

And it is okay to be a little bit humble to the power of gradient descent. Can we move to that?

Jesper Fredriksson: 46:27

Are you saying that evolution moves by gradient descent?

Robert Luciani: 46:32

Yes with death as the fitness function.

Anders Arpteg: 46:35

Okay, this is an interesting one. But if we take Jeffrey Hinton, he's one of the three godfathers of deep learning and had the Turing Award and everything. He's worked at Google for a long time and he just retired. And he retired for a reason, saying he's getting increasingly scared about the dangers of AI and he has been studying neuroscience for a really long time.

Anders Arpteg: 46:58

That was really what he started with and he said, basically, if I want to understand the human brain, I need to build it, and the way for him to build the human brain is to do AI. So his goal really was to try to understand the human brain and by doing so he invented the back propagation and the way to train these kind of models by gradient descent. And then he suddenly said and please correct me if you don't agree, but what I heard him saying is basically, I think the way AI trains today is much more efficient than the human brain, and that scared him suddenly. And then you know we recently said you know this kind of back propagation techniques is obviously we know that's not how the human brain works, so it's like an approximation.

Anders Arpteg: 47:45

You know, something we do to you know, this is what we can do today, but then suddenly you can flip and say actually, this is much more efficient than the human brain. That means that we can suddenly build something that is learning stuff much more efficiently than the human brain can. That can be scary. Do you agree with that kind of thinking?

Jesper Fredriksson: 48:07

I mean, I think it's fun to think about ways to use human brains as inspiration for new methods, but what we have is sort of unreasonably effective. The human brain. No, I'm talking about transformers. The transformer architecture seems to be, in a way, even though it's very costly to run and not energy efficient, et cetera, it's still. The scaling laws are so attractable, it's so attractive. If you're thinking about it from a business perspective, it's amazing to have something like that. You can just follow the trend until it bends and then you're fucked, but up until that it seems to be working well for us.

Robert Luciani: 48:53

Nature has to balance a couple of things. So I was once looking for like a very good example of what can you do when you have an IQ of something like this. And so for one standard deviation you can do first order theory of mind stuff and for two standard deviations of IQ you can do second order, which means I can guess what you're thinking, that I'm thinking that you're thinking.

Robert Luciani: 49:14

That's three orders of theory of mind and that would require three standard deviations of IQ to do, and so on. The human brain has very little use in general for fourth order theory of mind, sort of guessing in social contexts. So why would it optimize for that kind of intelligence when power, efficiency and a bunch of other stuff might be more useful in social context and that kind of stuff? And so it's only reasonable to believe that AI should be able to. You know, we can just make absurdly efficient at learning AI systems. But I think I do agree with you that AI can learn faster than us, and I think what people miss sort of or underestimate is how similar we are. So, for example, there's lots of really cool examples of. I think the bad thing to show is how similar AI is to us, and it's better to show how similar we are to AI.

Robert Luciani: 50:11

And my favorite example of it is, you know, remember in the beginning when image classifiers made mistakes from adversarial examples. You know you would show it a picture of a penguin and it would say banana, and you're like, oh AI, stupid AI. As it turns out, there's you might've seen it, there's a paper of a team that shows a human a picture of a water bottle and says is this water bottle a train or a dolphin? And you'll say it's 50-50, either are. And you can adversarially change the pixels to sort of force a human to 15% more, classified in one direction or another. Imperceptible pixel changes, which means and presumably if I could fine tune it to your specific brain I could push that even further and I imagine a future where, like imagine movies, you know the death scene. It forces you to cry and you don't even understand why because it's like fine tuned to your brain brain.

Anders Arpteg: 51:09

There are so many optical illusions that the human brain gets so wrong and it's. It's amazing how many bugs you have in the human brain. I think sometimes people don't speak enough about you know the limitations of the human brain? Yeah, I still think you know. Of course human brain is significantly superior to the as that we have today, but still still it's worth, I think, just reflecting, so to speak, about the limitations of the human brain. For one, our memory is horrible.

Anders Arpteg: 51:35

It's easy to see that the LLMs have superior knowledge in orders of magnitude than any single person in the world. We can also see that our limitation in reasoning, in the number of steps backwards or forwards, is also limiting. We can do it to a certain extent, but it's easy to see that, like alpha zero can learn without any human data to be better at chess than any human very quickly, very, very quickly. We can also see that the human brain is really bad at focusing on multiple things at once. You can think especially at least for me, I can think of one thing, potentially two things at once, but not more, and it's easy to see a computer doing a thousand things at once. So I think there is so easy to consider a machine intelligence could be superior in so many ways to a human brain, even though it's not today, but it's easy to see it's happening.

Robert Luciani: 52:37

I don't think people think of these things as weaknesses. Like you know, a magician will know that we have a super strong bias to movement, and that's how they do sleight of hand right. They'll make you look at one hand and then they'll do the thing in the other hand, and then you're like, wow, did you?

Anders Arpteg: 52:49

do that.

Robert Luciani: 52:51

That is a bias and a bug of sorts that they are taking advantage of, and it's so easy to imagine that we could tell an AI don't have that bias.

Jesper Fredriksson: 53:01

I was thinking when you were talking about memory. Why do you think we have such bad memory? If you think about the transformer architecture that can remember basically anything it's seen why are we? We are also a neural network of some sort. Why are we so bad at remembering?

Anders Arpteg: 53:19

things. I guess, from an evolutionary kind of point of view, we don't really need to have a perfect recall, right?

Jesper Fredriksson: 53:26

Yeah, I guess there are probably things that you don't want to remember. So we probably have circuits that that erase memory, just because we don't want to remember everything.

Anders Arpteg: 53:35

Human brain is actually very good at forgetting stuff. Yes, and that could actually be a feature, right?

Jesper Fredriksson: 53:40

yeah, that's probably a feature, because you probably need that to learn new knowledge.

Robert Luciani: 53:44

Yes, I had a project that never really came to fruition, which was very centered on either using a vector database or a latent space RAM, and what I really wanted the LLM to do was develop a sense of whether to save something there or not and what to erase actively. So it has a limited amount of space and you know something you talked about recently put it there and if it doesn't seem important anymore, take it away from there. And I think the human brain does that, that, and not only does it do that, I think it compresses it. So you know, the conception you have of tom the cat and tom and jerry is probably comprised of components. You don't have like a pixel recollection of it. You have a cat concept, a gray cartoon, and then you sort of put them together and that's very space efficient and that's why you can make mistakes, because it's not perfect but it's

Jesper Fredriksson: 54:34

also super efficient it's super funny, as you was. As you were saying the, the characteristics of tom, it's something.

Anders Arpteg: 54:41

Yeah, yeah, I was building it while you were saying it and at the third item, it just popped up yeah it's super fascinating to think about how the brain works so I'm trying to keep this kind of podcast not too long, but but I would love to see you know, if we think a bit further ahead, what would be the perfect next step or the best next step for ai models. We know they're really good in knowledge today, no question about that. I hope you agree. Then we want to combine the reasoning, I guess, in some way, but it could be other things, like the modular kind of approach that you spoke about. Robert, I'm a big fan of Jan LeCun. I've said that many times. He has this JEPA architecture, which I think is really awesome. We could go into that. But what do you think the best direction would be to improve intelligence in coming AI models?

Jesper Fredriksson: 55:32

Jesper, if you were to start, I think it's obvious to say that we should be doing what we think we will get ahead with the further I mean I think we should look at where we're standing. So it's obvious that scaling things works. Now that's not a super good thing because it will take so much power and etc.

Anders Arpteg: 55:53

It is getting limiting right, even though we have a fewer number of parameters and models, that we have synapses in the human brain, and still the human brain requires like 10, 12 watts to operate and the AI clusters to run the models of the day is like megawatts kind of power, so the power efficiency is really horrible yes, yeah and uh.

Jesper Fredriksson: 56:16

The next thing we have you already said is we also have something else that we know works, and that's the sort of reinforcement style of of learning that we know from alpha go, alpha zero and etc. So, combining the two approaches, that seems to be the most fruitful way of going forward today, based on what we already know.

Anders Arpteg: 56:36

And for me, you know, for 20 years ago, when I did my PhD, it was actually in reinforcement learning. I'm still not saying, I'm not convinced that reinforcement learning as a technique itself is perhaps the best way.

Jesper Fredriksson: 56:50

Probably it's optimal. It seems to be very slow in converging, but it depends a little bit on if you do a process optimization or if you do a goal optimization, because then it takes ages to get the signal.

Anders Arpteg: 57:04

I think the concept of search that you have in reinforcement learning is something that's missing today. So if we add at least that concept, the question is is the type of cumulative reward assignment really a good thing? I'm not sure, but at least having search is something that we're missing today.

Jesper Fredriksson: 57:21

Having search and some kind of incremental updating of the model through the search. I think that's what's needed right now.

Robert Luciani: 57:28

Really, we're talking about how can we speed up convergence, because what AlphaGo had was first they had a heuristic for what the next best step was, and then you added Monte Carlo on top of it and that improved performance. But it was still a very strong performance even without the tree search. It just got even better. When you combine them and you have two approaches. You have a sort of a, you have sort of a uh, a brute force approach, or a broad um, what do you call it? Uh, uh, x, not explore versus uh exploitation versus exploration, yeah, and and sort of.

Robert Luciani: 58:07

So what we're really talking about is what is the sweet spot for converging as quickly as possible in these questions? I just thought there was one more limiting factor which we might've not realized, but which Reddit feels very strongly about, which is you know, we've been having these human leaderboards for the various models, which ones people prefer, and they're really getting close to each other. And here's the question are the models actually getting equally good or are people just getting less capable of judging whether one model is better than another.

Robert Luciani: 58:39

That is interesting, because then reinforcement learning with human feedback as a heuristic is a bad heuristic. Now we have to come up with a better. And so what you were really saying is how do we set the goals in a meaningful way? I think this is what Strawberry is about. They probably think they have a good goal set somehow.

Anders Arpteg: 58:55

I think it's also connected to what people are saying are like AI hitting a wall in some sense, but in that benchmark that we're using, it's getting to the human level and then when it actually reaches a human level level, how can we judge?

Jesper Fredriksson: 59:09

if it's better than humans, because so will the models become sycophantic or whatever it's called like. It will only try to please the humans, because that's the way to to climb the ladder.

Jesper Fredriksson: 59:19

It's better for ai to be stupid, because that's what people think is the correct thing to do I love the the chat we had just before when you, robert, were talking about what you were doing with Udio. Maybe you want to talk about that, like how you were looking at different sounds and when something seems to be a bug, that's when you react to it. I like that.

Robert Luciani: 59:40

I think well. So we started off here by saying that it's ironic that art became like one of the main things that really accelerated with AI development and we still haven't been able to apply it to the stuff that you would think it would be very good at. But music generation now is really really good both singing and guitar and everything. And you know, even within music there are these more pretentious forms of music where you think like, oh, you have to be a genius to come up with this stuff, specifically like jazz or fusion and that kind of stuff. And I had. You know, I know this, I know that they're better than us humans. I know it's coming, and when I heard it I was still shocked. You can write Alan Holdsworth and you know all these advanced jazz guys, and it'll just write stuff that is incredible. All these advanced jazz guys, and it'll just write stuff that is incredible.

Robert Luciani: 1:00:33

And so what we do in my band is we ask it to generate hundreds and thousands of solos, and sometimes we just take little bits and pieces here, as is, which is kind of fun. But actually the most fun things are when it messes up, because sometimes we prompt it to do like a blend of two genres and it doesn't really know how to cross over. And the thing that it does right in the middle, where it messes up, is super interesting, which is sort of the essence of Prague, which is the kind of music that I like, where you try to push the boundaries. And AI is really good at doing creative stuff. It was actually a report last week that said that it's good at doing novel research. Did you guys see that?

Jesper Fredriksson: 1:01:07

Yeah.

Robert Luciani: 1:01:08

Or the AI scientists. Yeah yeah, novel research Did you guys see that yeah, I saw it. Or the AI scientists, and it was very difficult for the humans to evaluate whether or not it was feasible or not. The suggestions that the AI came up with.

Jesper Fredriksson: 1:01:17

I think you were maybe thinking about Sakana AI, the AI scientist.

Robert Luciani: 1:01:23

No, no, this is a report where the AI was supposed to suggest potential avenues for research and the humans were supposed to suggest potential avenues for research and the humans were supposed to determine whether, first of all, whether it was a novel idea and two, whether it was feasible or not. And by sort of blind testing, the humans said that the AIs were just generally better at coming up with new, novel sort of directions for research, but that the feasibility was like.

Jesper Fredriksson: 1:01:45

I can't really tell if this is even going to work or not, but it's cool for sure feasibility was like I can't really tell if this is even going to work or not, but it's cool for sure. I was trying to think uh, just to finish off, finish off the the thinking around, uh, finding the bugs. Maybe that's what we want to do. Let's say that we find a way to, to simulate uh, or to uh, to iterate on prompts or iterate on on generation, and then we find something really really strange. Maybe that's what we want to find, like the AlphaGo move. What was the name of it? We can muse on what creativity is just for one moment, because this is also

Anders Arpteg: 1:02:18

philosophy.

Robert Luciani: 1:02:19

A lot of people this is my opinion a lot of people think that creativity is this spontaneous, random, non-deterministic magic thing. You know, you sit there at the piano and nice, I just came up with that. That doesn't know how it works. Jazz people have thousands of hours of training, and programmers that come up with a novel algorithm have hundreds of thousands of hours of training. And so I think, you know, creativity for some reason seems to only come to people that have a tremendous amount of knowledge and then apply it in in a neat way.

Anders Arpteg: 1:02:50

so intuition is based on knowledge in some way, I think it's.

Robert Luciani: 1:02:53

It's strongly coupled with that, so it feels reasonable that llms should be able to be creative in the same way, if I may.

Anders Arpteg: 1:03:01

You know, one favorite trend, at least that I'm seeing, is that, yes, reasoning should be added to LLMs. But if we take the latest kind of image generators that we have, like Sora or stable diffusion version three or whatnot, what they're doing is, I would say, reasoning, but in latent space, and Janneke could also say some similar stuff. But we're actually not seeing that for text models or for text generation, and the reason potentially for that is that for text, text is actually a rather high level of abstraction already, but pixels are not, or audio time samples are not, and they are very high dimensional, so it doesn't work to do it in sensor space, if you call it that I would argue. So you can't do reasoning or generation in sensor space for images and audio, but you can for text potentially.

Anders Arpteg: 1:04:02

But if you take, like you know, Sora, and yeah, I guess Stable Diffusion etc. They basically have an autoencoder, first a variational autoencoder, and then they use a diffusion transformer inside it and you do the like 50 steps or 100 steps of diffusion in the latent space. And this is a very attractive idea to me and I think the same should apply to text. I think we should stop doing all the next token generation in text space, move it to latent space as well and do the reasoning in latent space. Do you think that's the direction we are heading?

Jesper Fredriksson: 1:04:39

yeah, what? What does the reasoning look like then? So I think we need to go into that as well. With with the jepp architecture, where we where you have some kind of energy function and you would try to the energy function would need to know what is a more clever answer.

Anders Arpteg: 1:04:55

The JEPA if we go to that. You know it stands for joint embedding prediction architecture. It basically means that instead of predicting from x to y, you do an embedding, a joint embedding of both x and y first, so you have an embedding of the sensory data that you do see, which can be text, audio or images or whatnot and then in that embedding space you do some prediction. That's prediction architecture. And for doing that they have a number of things. You have the perception part, which is basically the embedding part. Then they have a world model trying to understand what is the way the model works.

Anders Arpteg: 1:05:32

Given an action in current state, what would the next step potentially be? So that's kind of a world model. In some way you have some kind of actor to say what action should I take in this certain state that you have? And then you can basically do some kind of search in that space and saying, okay, I have this kind of perception of x and y. I want to make them as similar as possible, minimize the energy between them. How can I make that happen? Well, I take an action from x, I see what kind of you know latent representation I end up with, and you use the world model to understand that and you continue to search through in the latent space to find a minimal energy between no-transcript. I think that would be so much more efficient than having to do the encoding decoding all the time for every token that you have.

Robert Luciani: 1:06:30

I don't think that necessarily it's the latent space thing that is the make or break. So one reason to believe that this is a good approach otherwise, is that you know alpha mu, which has its sort of dynamics.

Anders Arpteg: 1:06:45

Mu zero.

Robert Luciani: 1:06:46

Mu zero, sorry. That has the dynamics simulator and does everything in pure latent space is cool. We know, for example, that it converges much more slowly, so even learning tic-tac-toe is very difficult until it doesn't, and so forth, but it's a good approach. However, you know, I think that the middle layers of the language model are still doing the same kind of search. The only problem is and I don't think it's in sort of fine grained token space, I think it's an abstract token space that doesn't mean anything, and the main problem is it's not allowed to go back and think for as long as it feels like it should think, for you can't retract either.

Anders Arpteg: 1:07:25

Yeah.

Robert Luciani: 1:07:25

It's. It's you know first, token is done, token is done, you're committed. And it's the same thing, sort of with the diffusion model, where you know you've thought for 50 steps and then you're committed, you have to keep going. So it feels like there needs to be a separate component, like the value network or something that's allowed to Because the JEPA has the cost function as well.

Anders Arpteg: 1:07:42

Yeah, right, and that is basically what you're saying. So you can retract, so you can take some actions. You evaluate the cost and that's a separate part of the brain, so to speak and then, if it actually goes down, you can actually retract and do some kind of proper Monte Carlo kind of research.

Robert Luciani: 1:07:59

But it's not intuitive to most people that the difference between guessing what the next best move is and the difference between predicting what the likelihood of winning is is so significant in the performance characteristics of the resulting model, and that that value network that is trained on likelihood of winning rather than next best move is so significant in improving its performance. So the question here is how do we set this value network, or whatever we want to call it? How do we set the goal to the right goal so that it converges well, performs well and all that kind of stuff?

Jesper Fredriksson: 1:08:34

that is the million dollar.

Anders Arpteg: 1:08:35

Question isn't it yeah yes, they have the world model there in some way.

Jesper Fredriksson: 1:08:40

This is the value you know because it would be attractive to just have, like a delete token when you're when you're generating tokens and then you just go back. No, one done that, but then what is a good indication that you should actually delete?

Anders Arpteg: 1:08:54

But I guess the reflection tuning that we started with is the way to do that, saying you should retract what you just said in the thinking phase and now try to come up with another thing.

Robert Luciani: 1:09:03

So it is kind of a good retraction possibility. Doesn't that remind you of like a Turing machine, a tape machine? It goes back and sort of goes over its own code, but it still.

Anders Arpteg: 1:09:12

It can't really erase it, it just says skip that part and continue to generate on the tape without moving to the left, so to speak.

Robert Luciani: 1:09:20

Many people in the audience might not know that the N and NP is not non-polynomial, it's non-deterministic polynomial, which means that this tape machine if you can imagine a multi-dimensional, multi-world tape machine going in every direction is actually like a tree or a graph, which is what the Monte Carlo search is a little bit like.

Anders Arpteg: 1:09:40

Anyway, super fun stuff and hard to know what will happen here. Here's a question Is it the haves and the have-nots, are we? And hard to know what will happen here.

Robert Luciani: 1:09:47

Here's a question. Okay, is it the haves and the have-nots? Are we just here, relegated to watching what happens and musing at it, or are there things that we can do? I'm just very impressed by all the thirsty guys on Reddit that are making RPG fine tunes and super advanced quantization things. I do feel like they've contributed maybe not 50%, but 30%, to the breakthroughs of the past six months. Are people still in the game? Is Mistral still in the game? Is there anything for people like us to do? I can start that's a great question by the way.

Anders Arpteg: 1:10:27

I think it will be increasingly hard to move to the frontier models unless you have the insane compute infrastructure that is now being built by Xcom and Elon and Meta and Microsoft and Google. But there will be a lot of innovations happening on smaller open source models. But I think, actually my guess, is that the gap will increase and that's a bit sad, or it's very sad. That's sad.

Jesper Fredriksson: 1:10:57

So you're in the other camp, that there's not so much we can do.

Anders Arpteg: 1:11:01

I think we can do a lot in innovating by using the frontier models.

Jesper Fredriksson: 1:11:06

That's what I was going to say.

Anders Arpteg: 1:11:08

From them. You can distill it into some much more specific kind of models that you use for building recipes or whatnot, and there you can innovate a lot.

Jesper Fredriksson: 1:11:17

Exactly so. That's a good bridge over to me if you want to.

Jesper Fredriksson: 1:11:22

So I was thinking I already started talking about this multicolored research, I already started talking about this multi-colon research and for me, the way to sort of see that I'm trying to see where we're going is by experimenting with doing this on the prompting level. So just generating prompts and trying to see can I get to something smarter by doing that, and that I think everybody should be doing, Trying to. If you're already active, it's relatively easy to pick up these ideas and start playing with it, but not at the training level, but at the prompting level. I think that's the easiest starting point. Then you can see what would happen if I did this in the fine-tuning step.

Anders Arpteg: 1:12:07

It's very easy to say people that do not take advantage of AI and these kind of frontier models or the open-source models that we do have, they will be left behind.

Jesper Fredriksson: 1:12:15

No question, right it's like I was about to say, it's impossible to keep up with the frontier research. It's impossible for me and it's impossible for many others. But at least if you're already using the tech, then you're in a decent space.

Robert Luciani: 1:12:35

I would like to give a recommend to anybody that's a tech enthusiast to do the following three things Install Cursor and sort of try out programming with the help of an AI.

Robert Luciani: 1:12:46

Yeah, any of them, it's very cool. Second one is download Ollama and try out a with the help of AI. Yeah, any of them, it's very cool. Second one is download Ollama and try out a couple of models yourself. Any kind of computer you can have, you can run the little and compare what Phi versus Aya versus the Nemo fine tunes, and they're all different. I found, for example, when Gemma was going to summarize a chat log for me, that I couldn't get it to stop being helpful. Sure, I can do that for you. Here's the learnings and takeaways. I'm like I don't want that, and you know, you got to get a sense of their personalities, I think. And then the last one is there just came out a new version of Flux that can run on smaller GPUs for image so Fl.

Robert Luciani: 1:13:28

And.

Anders Arpteg: 1:13:29

X is using it in the GROK models.

Robert Luciani: 1:13:30

Exactly, and there came out a new one called NB4, I think it's called that can run on a laptop with a GPU. It'll generate an image in like 15 seconds and the text adherence is really good. So these three things coding with AI, running language models and you can run this on your laptop and it's pretty damn easy. You clone the repo and it just works. You should try it.

Anders Arpteg: 1:13:56

And I think anyone, even if you are not a programmer, should start to experiment with these things. Now, we've been really techie this podcast and it's been really fun, but I think anyone can really start using it today because there are so many tools and APIs it's fantastic.

Jesper Fredriksson: 1:14:10

You can just start with chat gbt. If you haven't done coding and using the API, you can just start playing around with chat gbt and you can do like a lot of stuff in there as well there's some pretty sci-fi.

Robert Luciani: 1:14:20

There's one sci-fi I just have to mention this because I think it's so cool. So I record music, as some of you know, and I have my son in the background as, like, he's 12 years old and so he has a beautiful like boy choir voice that I have in the background. That sounds spooky and cool and I want to save his voice because he's turning into a man and he's not going to have a beautiful little kid voice anymore. So I want to use some models to sort of preserve his voice for all eternity. And you know, I think many people are many people are thinking how can we use ai to transfer ourselves? And this is just a little precursor to that saving his voice.

Jesper Fredriksson: 1:14:53

I think that's, don't? You think it's possible to go back in voice if you, if you make a model of it?

Anders Arpteg: 1:14:58

probably, probably, yeah you know, that brings me to a rather sad topic. You know we had, I had a personal friend called frederick levden, who was that we lost a few weeks ago to a very unfortunate accident, and there are times when I just wish I could speak to him again or I could have his voice or his thoughts or his ideas or his truly very friendly kind of attitude and personality back somehow. And actually I, yeah, I I wish we would have said actually we do have, you know, the podcast we did with him.

Robert Luciani: 1:15:37

It would be super fun to just take that and try to build some kind of avatar or some kind of model of him that that brings him back in some way I think there's something special that I don't know should or could, but, like there's some, there's something there that, um, it would be nice to be able to, to save, save people somehow it's going to be.

Anders Arpteg: 1:15:58

I can tell you this already. It's going to be memorial for him next week here in stockholm as well. It's already been one in link shipping and we can publish more about that if it's okay for the family and stuff and stuff. But I'm wondering if I should yeah, I would be. It would be super fun to do something avatar-ish for him. Okay, let's not go there, I'm getting sad just thinking about it.

Anders Arpteg: 1:16:23

Anyway, trying to wrap up here a bit what do you? You think about reflection tuning? How is this story going to unbow or evolve?

Robert Luciani: 1:16:33

It's going to die. It's crap.

Jesper Fredriksson: 1:16:34

It's going to die you think so On to the next one, yeah.

Anders Arpteg: 1:16:37

You don't think there is a chance that it actually is on to something and you will come up with a model.

Robert Luciani: 1:16:42

Now, this is the stuff that everybody's working on right, like you are already working on getting your stuff to be more reliable, and that's just another approach, and I don't know.

Jesper Fredriksson: 1:16:57

there is something to it, obviously.

Anders Arpteg: 1:16:58

Just not the GPT-4 beating magic silver bullet that we thought it was. Yeah, I agree, unfortunately, but it would be fun to have this kind of. You know just in three weeks, basically he took this idea, he generated the of. You know just in three weeks. Basically he took this idea, he generated the data set for it, he built this kind of you know the process for training it and publish something, and he had basically very little resources. It's something so attractive if that were to be true. I still wish it could be true.

Jesper Fredriksson: 1:17:21

The TLDR we want to believe.

Robert Luciani: 1:17:25

Oh, that's. That is a great thing to end on. I? Um, here's a question everybody's talking about ai hype. Is ai overhyped or underhyped?

Anders Arpteg: 1:17:34

I'm of the conviction that is underhyped I actually agree, but I think, you know, a lot of people are overhyping it and a lot of people that are not really understanding what ai can be, and there there are so many people that claim there are generative AI experts out there today and they don't really understand what it can and cannot do. So in some way there are a lot of people hyping it to an extent, which is wrong, but there are and I agree with you so many more people that do not really understand the full ramifications of what AI will do to our society going forward.

Robert Luciani: 1:18:07

So, yes, it feels like we've been in this for five, six years, but we've only been in it barely two years, right.

Jesper Fredriksson: 1:18:13

I think it's interesting. It takes some time to really get to know AI. You need to work with it a lot to really understand what it's about. And until you do that, you can't say if it's underhyped or overhyped. I agree, I think it's underhyped. Say if it's underhyped or overhyped.

Anders Arpteg: 1:18:28

I agree, I think it's underhyped. And let me end with the standard kind of question, then, and perhaps in retrospect to the previous time, you have been here in the podcast thinking you know what will happen when we have an AGI, when we have an AI that is more intelligent than any and or all humans combined, will we end up more towards? And Nick Bostrom just came out with a book called Deep Utopia. He previously was very dystopian speaking about the dangers of AI. Now he has a book on the opposite side saying what happens if it actually does the most positive things AI can do the utopia of the world where we live in a world of abundance and we basically have all the goods and products and services for free. We don't really need to work unless we really want to.

Anders Arpteg: 1:19:16

But it could be that kind of more utopian kind of future. But it could also be this dystopian kind of machines going to kill us all the paperclip kind of argument, the terminator and matrix kind of scenariobert. Do you want to start? What do you think? What's your likely scenario?

Robert Luciani: 1:19:31

it's easier to predict uh 10, 20 years in the future than six months into the future so that's the first one.

Anders Arpteg: 1:19:37

That's a good good comment yes and uh.

Robert Luciani: 1:19:40

I think it's most fun to try six months because they end up spectacularly wrong. You know, I've made some specific technology guesses. I wrote on on LinkedIn almost a year ago so a year's not up but I wrote that we would get strong signs of consciousness in the next year, and it's one of those things that I'm like. Now I realize it's going to be debatable forever, no matter what kind of results we get. So that was a daring prediction from me that I will have to eat up, I suppose. I think okay.

Robert Luciani: 1:20:14

So the short-term prediction is we're going to be using AI to generate data in companies. So AI is going to map, you know, knowledge, map processes, map people, all sorts of stuff, and they're going to be generating data for us. That's the short-term prediction. And then the long-term prediction is we're going to see emergent effects. So what I mean is we're not going to make a killer AI that's going to destroy everyone. There's going to be a paperclip robot, but it's not going to be one robot. It's going to be seven systems that are sort of emergently reinforcing each other to do things that we hadn't predicted, sort of like the economy. You know, you have like one screw yeah, and you think you're doing something, but you're actually affecting eight other things at the same time and you really have no clue what you're doing. And these systems are sort of feedbacking on each other, so that's where I think it's going to be headed. Super cool.

Anders Arpteg: 1:21:05

Yeah, I haven't thought about that. That's an interesting point, jesper.

Jesper Fredriksson: 1:21:10

I think, first of all, that AGI, for all we know, it could already be here. We just haven't noticed it. I think that's what's going to happen. We've said this many times before that I would say, if I would see GPT-4-0, if I'd seen it 10 years ago, I'd say this is AGI. So I think it will happen, and it will happen gradually, so gradually that we will not even notice it.

Anders Arpteg: 1:21:37

It's the boiling of the frog, kind of thing.

Jesper Fredriksson: 1:21:38

Yeah, it's the boiling of the frog. It's a little bit dependent on how everybody goes about it, because there are strong forces going that will try to get to it first, and that's that creates a competition for being the first which will, which could make it faster than we think. But I think it will be something like three or four years maybe, and that's my optimistic so ray kirchwell 2029 kind of thing, something that and by then we're already so used to it, so we're going to be.

Jesper Fredriksson: 1:22:12

Well, there was a new influencer on the TV, or whatever.

Anders Arpteg: 1:22:17

Something else. I think humans are more adaptable than we think.

Anders Arpteg: 1:22:21

And we will adapt to this as well, and there are a lot of things and a lot of people that are more intelligent than me certainly out there, I can still survive and a lot of people that are more intelligent than me certainly out there, I can still survive. And if you listen to Nick Bostrom when he spoke about Deep Utopia, he got the question about what will happen when AI is better than any kind of job you can think of. Will it not be lack of meaning or life or something? And he basically said well, there are a lot of people today that do not work. There are children that doesn't have to work. They are happy, are able to have a fulfilling life. There are retired people that do the same. We have people that are super wealthy. Perhaps they don't have to work for a living, but they still are very happy. And people are surprisingly adaptive.

Anders Arpteg: 1:23:06

I think it's one of the strongest benefits and strengths that we have as a human, and I have no doubt that we'll be able to adapt to that. The only thing I'm really scared about is not about the long-term prediction, because I think that will be very positive if you just manage to get there. The real danger is because the world is looking as it is and there is increasing turmoil, so to speak. Because the world is looking as it is and there is increasing turmoil, so to speak, in the world, if humans are starting to abuse AI in coming years, that's the big danger for me and I'm really scared about that. But I hope that we can surpass it because I know a much more super intelligent AI will help us to manage that in a good way, but I'm scared about the way there.

Jesper Fredriksson: 1:23:51

Coming from you. It has a lot of weight.

Anders Arpteg: 1:23:53

Okay, awesome, this has been super fun and I hope you can stay on for an increasing number of philosophical and very much in-depth, super technical kind of discussions. But with that, thank you very much, jesper Fredriksson to come to this podcast and Robert Luciani, it's a pleasure, as always, to have you here. Thank you, it was super fun. Thank you.