Why predicting raw video beats VLAs, and where that bet could fail

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 56:36

   But I think at the end of the day, what we call the models is less important than what we believe the model needs to be able to do at a first principle level. So I want to explain that a bit. I think for robotics, the most important thing is that you need to be able to predict the future. Like robotics, any physical system is fundamentally about based on the information you have at the present moment and before what should you do next so it's really about predicting what comes next so predicting robot actions is like predicting what the robot should do next but if you think about it robot is just a part of the world right robot influences the world but the world also influences how the robot should behave in that environment so if you think about it this way then the maximal the most first

2. 57:26

   principle thing you can do just predict as much of the world that you can possibly predict right so that's why we believe predicting the future is extremely important and today the instantiation is world action model and vla only predicts robot actions where some more abstract representation of the world but whenever you're predicting abstract representation of the world you're encoding your own bias into what's important to predict and the bitter lesson tells you let's remove human biases.

3. 57:58

   Let's just take the most raw representation, the camera inputs, and just predict all of that. So that's how I think about modeling for robotics today. So just to press you a little bit on this, you're basically saying your approach is better than the VLA, right? Because you're emphasizing prediction. I fully buy that. You've sold me on it. If you were wrong, what is the most likely number two that someone else is doing?

4. 58:24

   Interesting question. Yeah. So yeah, there's a caveat in what I've said before. And I think that's where I could be wrong. Right. I said that you need to predict the world as much as possible. Right. But what that doesn't say is that does your sensor have everything about the world that you need to predict? I think I touched on this a little bit. Right. Like if you're trying a world model that's purely based on videos, i think you'll be a very powerful world model but it may not be a sufficiently rich enough world model because you're missing critical aspects of the world like sound like touch right so i think what where we could be wrong is like okay i have the best world model on vision vision-based world model but tomorrow someone else comes out with a vision plus tactile world model that's just like

5. 59:16

   100x more data efficient just because, you know, vision is a very high information or like a highly redundant, you know, input because, you know, every other pixel, they don't change that much. You can see, you can understand the world without perfect vision, but tactile, you know, other modality might be much denser information, right? So maybe having that, predicting that would just allow you to use much less vision so you can get to the same performance with much less compute with much less data.

6. 59:48

   I think you will be able to adapt very easily. No, exactly. There's also a level of how much data... Right. So the reason why I bring this up is because I think if a vision tactile world model or a vision tactile audio, whatever world model, is the final thing that solves robotics, and I think robotics companies will be able to do that better than video generation companies is trying to pivot to robotics.

7. 1:00:13

   The reason is to understand sensing it's actually very complicated like we've been building that since the company started so we know the bottleneck we know how to combine them but for if you're just trying to adapt the video model to robotics action then like you know if we were wrong if you know video model is not enough then like you don't really have the know-how to get tactile to work very well but we have that this is a side question but i think it's a quick one uh heat seems very important is infrared just another color to you or is that a yeah you could be another color could be another color another