Sloppy RL environments push models to obsess over graders

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 14:39

   I think that the state of science here is kind of unclear. In my mind, the big question is what happens when you remove a bunch of sloppy RL environments? So Anthropic had a blog post about about what they were doing to improve the alignment of their models the other day. And the position that a lot of anthropomorphic people take is the misalignment that we see when URL models substantially arises from the fact that these models are trained in many environments that are poorly specified such that if you want to do a good job, you actually do just have to reason about what the greater is going to look at.

2. 15:12

   I think there's an analogy here where imagine you're in high school and you have a really good teacher who's done an excellent job of thinking through the curriculum and trying align the incentives of the grading with the incentives to just learn stuff and do a good job if that's kind of how your exam is set up you don't really need to spend that much time thinking about the minutiae of the exam but imagine that your teacher is instead like extremely capricious and loves doing these extremely obscure trivia questions about particular kinds of things or imagine that it's like you're the stereotypical terrible english teacher who just grades you based on whether you agree with their judgments about what a novel is saying.

3. 15:50

   In that second kind of case, you really are forced to think really hard about what your teacher wants. Whereas in the former case, you need to spend a lot less time thinking about what you're teacher wants. My guess is that a lot of the RL environments that models are trained in today are kind of like the latter case where the instructions the models are given are kind of sloppy and it would be a mistake for the models to take them at face value rather than thinking pretty hard about how they're actually going to be evaluated. And the big question in my mind is if it is feasible to change that, to make it so the models are not actively penalized for just kind of taking the instructions they get at face value, how much quantitatively will that reduce the pressure towards models thinking really hard about their graders?

4. 16:36

   I think we did during there. It just comes down to these like very tricky questions about how strong the effects towards models obsessively thinking about their graders is. Like I don't know whether reducing the sloppiness of the environments, like reducing the incentive towards thinking that you're greater by a hundred X. I don't know whether that is more like a small effect or a big effect on the extent to which these models think carefully about their graders. I'm very worried that if it's only like a 10x reduction, or like, it might be the case that as we scale up RL, the extent to which you would have to clean up your environments is just going to be totally impossible. We would require this ridiculous level of removing incorrect instructions and exploitable graders from our environments in

5. 17:22

   in order to not end up with AIs that obsessively think about their graders. So I think it's pretty likely that we're just gonna have to live with models that are strongly incentivized to worry about what their graders think. Is there like an example of a sloppy or bad RL environment,