RLMs generalize across tasks because the strategy stays the same

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 36:31

   one of the things that we see with an RLM is that if you sufficiently offload context and write, ask the model to write code over that context, you get this really, really weird but useful property, which is that when the model recognizes during training how to solve a task, it turns out that the solution to many tasks is very similar across like tasks where you don't even like it's not even that clear to you that the solutions are similar. So in this example, we have like a retrieval task and we have like an aggregation task. And they're very, very different query.

2. 37:07

   Like the domain is just completely different. And when you train a regular language model over these two tasks, the trajectories look very different. And so what you're relying when you, like let's say you use Pi or Cloud Code or something, which is not in this blog, but we do have these results, you'll find that like these harnesses distinguish too much between these problems, even though the solutions are the same.

3. 37:29

   And so one thing that we find when training RLMs is that like when it sees these problems as the same, and the reason it sees these problems as the same is the sub-agent sees different problems, but the sub-agent is solving an easier sub-task and so you're confident it's smart enough to do it but for the base overall strategy they end up looking the same and so when you train on the left task for example it can immediately solve the right task and so if you go down to like the plots that we have one thing you'll kind of observe is that as you just naively train your RLM on these tasks, they naturally learn to generalize, for example, to longer tasks because the strategy is basically the same.

4. 38:10

   You're just modifying like a length variable. And this actually also holds for tasks that are different. And they're not different across length. They're completely different tasks, math tasks versus writing tasks. But the solution, the like meta high level solution is the same. And so when you train the RLM on one of them, it generalizes this behavior to the second one. And there's no magic here.

5. 38:34

   I guess maybe that's the thing that I want to kind of stress. How would you kind of verbalize what they're learning? So I think in here you say you train it on short tasks. They generalize the stuff 8 to 30 times longer. They're learning how to solve these type of problems. Or what's the core thing they're actually learning? Yeah, they're learning how to solve these types of problems at a certain length.

6. 38:55

   And it turns out that when you take this strategy that they learned, it is directly transferable to the longer length, like they're effectively the same program. And this is something that is really exciting because what this kind of implies is that when you have a corpus of data or environments that you train your model on, the hope is that like, A, you can train on less environments and generalize some more than what existing models can do through harnesses through existing kind of like naive training.

7. 39:27

   But B, also, you still want to use all the data that you have. So when you train on these tasks, hopefully it generalizes to a wider class of problems. And why this is also even more exciting, at least in the context of RLMs or any recursively calling system, is this argument holds inductively. So I'll give you an example, because I said I claimed that competitive programming and GPU optimization use very similar skill sets.

8. 39:53

   The model can, the harness potentially, I mean, you might have to nudge it a certain way, but it can learn that like, okay, how I'm gonna go about solving this GPU programming task is very similar to what I learned for competitive programming. So I'm gonna list out a set of solutions. I'll like spawn sub-agents to list out promising solutions. And then I'll like write this loop to go through and check these solutions, maybe evolve them and like evolve them against a verifier.

9. 40:20

   And between these two tasks, This looks the same, but what the subagents are doing are maybe like unique and something like different, but even what the subagents are solving might actually also be of the same form, right? Because it's like a, it's a recursive argument. And so what I'm trying to get at with this whole blog post is just that like, we should rethink what the role of the harness is because harnesses can actually greatly increase the generalization capability of your model and the amount of data that it's given.