A world model predicts consequences, which LLMs cannot do

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 9:06

   training data, you know, things like that. Okay, then there is world models. Okay, so what is a world model? A world model at a very general level is something that allows an agentic system to anticipate the consequences of its own actions, okay, predict the consequences of its own actions. From my point of view, I cannot imagine how you can even think of building an agentic system without that system having the ability to predict the consequences of its actions.

2. 9:35

   I mean, that's pretty essential, right? When we act in the world, we have this ability. And when we take an action without thinking about the consequences, we are taking a big risk. And very often, you know, other people think we're an idiot. But we have plenty of examples on the international political scene at the moment of people who have, you know, no ability to predict the consequences of their actions.

3. 10:03

   So that's the one model. That's all it is, right? Ability to predict the consequences of your own actions. If you have this ability, then you can plan a sequence of actions to accomplish a task, to, you know, satisfy a goal. And you do this by planning, reasoning, by a process of search and optimization. You don't do this by predicting one action after the other autoregressively, like we do. You do this by searching for a sequence of actions that will accomplish the task you set for yourself.

4. 10:37

   So the blueprint for this is completely different from what LLMs can do at the moment. LLMs do not have the ability to predict the consequences of their actions. and they do not have any planning abilities because inference is by predicting the next token, right? It's not by search, okay? So right there, you have the two characteristics that I think are essential for intelligent behavior, ability to predict consequences of your actions, and second, ability to plan by optimization, by search, find a good sequence of actions that will produce the correct outcome.

5. 11:12

   And then there is a third characteristic, which is how you predict the consequences of your actions. Okay, so, you know, if I have a water bottle in front of me, I realize some people would just listen to this and not have the picture. So I have an open, uncapped water bottle in front of me. If I push at the bottom, it's going to slide on the table. If I push near the top, it's probably going to flip.

6. 11:39

   we can't predict exactly how the the bottle will will fall in which direction uh we can't exactly predict how it's gonna slide you know how the water will spill you know whether the table is tilted in one way and the water will uh you know kind of flow in one direction or another there's no way we can predict at a pixel level. And so our mental model of the world predicts, but at an abstract level of representation. So as you were working on this architecture, was a lot of it inspired by the human brain? I mean, obviously, like, you know, the way you're articulating things exactly how we do things. Right. Or at least by, you know, cognitive