Chain-of-thought monitoring will fail as agents grow more capable

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 47:43

   that ensure that this goes really well. Fascinating. And it seems like one of the primary things that current safety researchers rely on is being able to monitor the agent's chains of thought in English. And so when they start speaking in a release, that scares people because we can no longer understand and monitor it as well. In your case, you have your agents kind of going straight to rediscovering language.

2. 48:05

   What do you think that means for monitoring? Or is the point that if you get the reward right and if you get the environments right, then you shouldn't need to monitor? Yeah, I think monitoring is something which is still really helpful. I think that, like it or not, I believe that the approach of monitoring the human language that these systems use is going to be increasingly difficult as agents get more powerful.

3. 48:39

   And we're already seeing that, that the monitoring agents are, maybe they're producing like a tome of a million words of reasoning, or maybe they're actually actively using subterfuge to kind of think in one way and then act in another. It's increasingly difficult to rely 100% on that. So I do think we need other elements in our armory anyway. So I think it's actually really important that some people are looking at how to get this right.

4. 49:07

   I think there are ways to monitor agents that don't use language. I think we can do things like start to, for example, you can use shadow actions where you start to have actions taken where you can look at what they would actually do if they were allowed to act in the world, but you don't actually let them take that action yet. You can start to have other superintelligences which are acting as the monitor.

5. 49:35

   So for example, you might have the idea of like an incredible scientist agent that sort of treads lightly upon the earth. And then you have another superintelligent that's kind of monitoring the kind of magnitude of its actions and regularizing it to make sure it can't do things which are kind of affecting the world too much. I think there are lots of ideas. I would say that there's a lot of research to be done here. I'm not claiming to have all of the answers, but I think it's something that deserves a lot of investigation. And if people aren't looking at this because they're only looking at the language based approach, I think we're in trouble because I think that language based approach is not enough in itself to actually get to where we want.

6. 50:14

   I should say in terms of having said in trouble, I am optimistic about where the world is going. So by trouble, I mean that we're not going to have agents that do the things that we want them to do. I do think we're going to end up in a world with lots of complex, interacting, super intelligent beings. And I'm optimistic that that will lead to a kind of a balance of actors, some of which are supporting the flourishing of humanity and some of which may not be.

7. 50:43

   But the ones which do support the flourishing of humanity will end up kind of supporting us and protecting us against the ones which aren't acting on our best interests. So that's the world I think where we can work towards. Well, given your super forecasting abilities, I'm happy to hear that.