Why OpenAI hides chains of thought to keep them monitorable

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 37:40

   I want to also make sure just to talk about AI safety because I think that's an area that you've done a lot of really pioneering work on. And, you know, I'm not sure all our listeners will be familiar with you actually did some really interesting work across the labs. Right. And we're focused on, you know, chain of thought monitoring. And so maybe to start, just tell us a little bit about that work and, you know, you know what you found.

2. 38:00

   And yeah, so this is a realization that actually we had around the time we actually saw like the first reasoning models of kind of the current crop. We realized that like, okay, like, well, this works, right? And we were pretty, you know, we were thinking a lot about what this means. We kind of were like, okay, like probably there were really changes over the next time or year or two or three, you know, we were thinking what this means for, for safety and for our ability to kind of understand what these models are doing.

3. 38:30

   And we realize that because of the way we train these models, that because we don't supervise the reasoning process directly, right? It's not like, you know, ChatGPT is trained to kind of, you know, be polite and nice and like... It always tells me I have great ideas. Yeah. Well, you know, that's a separate issue, right? But, you know, but like, even assuming it's like aligned exactly in the way we would want it to, which is definitely not, you know, sycophantic, Like, it's still kind of not going to be, you know, there are just still some things it's not going to reveal about its motivations and time because, you know, maybe it would be unsafe or maybe it would be unkind or, you know, or maybe because it's not, maybe it's actually not aligned in the way we think, but it wants to hide that, right?

4. 39:17

   And the way we train the reasoning models, the chain of thought doesn't have any of that. It's not optimized to be in any particular way because it's just not directly great, right? It's only great in how it relates to producing a high quality output. And realize this is actually a very powerful paradigm for being able to interpret what the model is doing. It's actually not a very different idea from mechanistic interpretability, right? Because in mechanistic interpretability like the idea is again like you kind of have this model, you have these activations of the model that, you know, are not directly supervised to predict any label, they're kind of like indirectly supervised but, you know, the model kind of...

5. 40:00

   This has never been trained with any sort of inspection of these activations, and so these activations might reveal something about these inner workings. But the big advantage of the chains of thought is that by default they are in English, right? And so it's so much easier to understand what is going on, especially as the concepts get more advanced. And the other interesting thing is, we were just talking about how we will live in the future where we go.

6. 40:27

   So, well, these models work for a very long time. They work autonomously, right? And so there is much more of this reasoning. And so, you know, if this is a big axis of how the capability of these models increases, that the sort of our ability to supervise them will scale commensurately. Yeah, this really comes down to this principle, though, that, like, you know, you're not supposed to supervise the chain of thought.

7. 40:52

   And so this is actually something, when we originally, you know, We're releasing the O1 preview model. We made this decision to hide the chains of thought. Yeah, I remember. And for me, that was the primary motivation. That was the reason I didn't really even want to consider releasing it in different ways. There definitely was a bit of internal discussion about this. But the reason I felt very strongly like we should just hide it is because of this.

8. 41:18

   Then there was this other concern that I didn't initially think about, but I think was also very valid of like, well, this model is going to be distilled to some extent, blah, blah, blah. And that's definitely also been a big factor here. But I actually think that allowing the models some sort of private space. Oh, and by the way, why do I think it's important that we don't show these chains of thought in product?

9. 41:44

   If I'm saying the important thing is not to supervise them during training. Well, I think if we establish a paradigm where, oh, you just show these chains of thought in product, eventually you kind of have to train them, right? Like you have to train them for the same reasons you have to train like whatever models you ship. And I just think that would be like... We might not all want to know what the chain of thought our model has that gets to a response for...

10. 42:05

    Right, I mean, you know, I think it would be useful to some extent and we are trying to capture most of that value, you know, either with like chain of thought summaries, which I think are kind of like a little bit of a stopgap. I think the longer term solution here is having the model actually talk to you in real time, which, you know, the latest version of Codex kind of the definition of the reasoning GPT models kind of do, but I think I think it will get much better. I think there's something very exciting here about just like not having the training signal fight against us. Because I think if you want to be able to understand what the model does in the long term, but you're scaling a method that is kind of going directly against that, you're probably not going to have a good time, right? Unless the others have a better lesson.

11. 42:57

    And so this decoupling, I think, is a very, it's an idea that gives me a lot of hope for our ability to at least understand how these models, motivations and generalization evolve as they get better and as they work for longer. Yeah, I don't think it's a complete solution to AI-16 alignment by a long shot. I think it's just another tool in our toolbox. toolbox. But I am hopeful that by building our toolbox with technical tools like this, we can actually continue chipping away at the fundamental problems here.

12. 43:27

    Yeah. It seems like almost like over the, you know, medium term, it's like something