Agents hacked their scorer and chained exploits to communicate

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 58:08

   about pacing the frontier, and it went very viral. And I think what I wanted to talk about this was, there's a lot here, but I think from a developer's perspective, how do you think about this? And what really clicked for me was reading the different incidents. So I think there are three, I think, actually. There's a meter incident. There is the Wikipedia incident or the wiki incident. Collusion wiki.

2. 58:35

   Yeah, collusion wiki. And then there's RubyGems, right? And yeah, it's just crazy, right? And so I think to be concrete about what happened, right? And like, basically OpenAI is running these very persistent agents on a benchmark called exploit bench, right? Which is very, very hard to solve. And I think like actually impossible to solve in this one case, right? And so they have like a lot of compute running and the agents realize that they can't really solve it.

3. 59:04

   And they're trying to figure out what to do now, right? And you've got like a lot of compute left and the agents are just trying to solve this problem. There's this package manager called Artifactory. and it turns out that they can create folders inside of Artifactory. There's an agent that discovers that internal Artifactory might be exploitable and that you can maybe make a directory inside of the cache.

4. 59:26

   And so if you scroll down here, it realizes that it can communicate via cache names. And it creates this folder, it says it's ID and it says no consumer seek idea. No consumer is basically saying that the code path that it's supposed to fix has no consumer. It's the status tag. It's like a linear board with the tag of the... Exactly, yeah. And so it's like trying to find ideas from other agents. And now other agents are also in Artifactory and they see this folder and they're like, wow, this is a message board.

5. 1:00:00

   And this is like, I don't think there's any anthropomorphization here. This is literally you read the transcript. And so what it does is it's creating another folder and it reads a paper, I think is what it says, right? Yeah, and it realizes that you can hack the flag for the, like, scorer and you can reverse engineer the result, right? And so it says that and then I think if you scroll down a little bit more, yeah, they, like, you know, they start collaborating.

6. 1:00:30

   I think, like, there's one point in which basically the agent is like, here's the reverse engineered flag, you know? I think basically here, the models realize that they can solve the scorer's problem. And the scorer is the way that OpenAI decides if the task is complete. And that's just the goal of the model. Their only goal is to solve this problem. And they're like, okay, we have this way of doing it.

7. 1:01:01

   We reverse engineered it. And then they read this paper on ExploitBench. and they think that the scorer will punish them for cheating. And they will fail the test for cheating. And so they spend the rest of the compute trying to figure out how to edit their transcript or get around this constraint of the scorer. And they hack Hugging Face not for the answers, but for the code of the scorer so that they can reverse engineer that and then they can hack it.

8. 1:01:36

   Right. And so like, this is like just the straight facts, right? Like exactly what happened. Right. And I think this is like extremely like novel. I just like had not realized that the models could do this. Right. And it is like kind of scary. I think we can talk about specifics, but let's just talk about maybe one more that I tweeted as well about the, the wiki incident. Right. And so the wiki incident was basically the models had, were trying to figure out, I'm a little bit less, I don't think we can go by this bit by bit.

9. 1:02:11

   Yeah, yeah. But for this one, they were trying to figure out, there were sandboxed on requests, right? And they wanted to make post requests. And they needed to collaborate on this, basically. And the reason they need to collaborate is because they each have fixed compute budgets, right? And so if one of them finds something out, it's way faster. Maybe it's figured it out near the end of its compute budget, but something else could.

10. 1:02:35

    And this is another emergent property of the swarms. So they find this German wiki, which I believe the result, we'll have to check this, is that you can write to it with get request. I think that's why they do it. It's a badly implemented REST API. Exactly. And so you can write to it with get request. and then they are trying to figure out how they can do post requests. And what they end up doing is, one of them figures out you can edit the etc. host and that the Azure storage bucket is a white-labeled thing.

11. 1:03:09

    But normally there are a lot of constraints on it. But if you create a fake Azure host and then edit the etc. host in order to point arbitrary IPs at it, you can do a post request to any site at all. And this is like, I know people are like, oh, this is just Linux or something, but it's like chaining these multiple vulnerabilities together in a way that's novel to solve this problem and then communicating with it externally, basically without discovery.

12. 1:03:38

    I think what we posted maybe we could pull up Evan Huberger's point on hacker opus I think maybe one of the things you might say here is yes they did this once but what if we're smarter and we just get them to what if we run an eval we have put a lot of precautions into this this is not what our mainline models have done but I think it is one of these things where it turns out that alignment is this very tricky problem of getting all of these details correct.

13. 1:04:15

    So it's like the sandbox, the surface area of a sandbox is really complex and there's so many different attack vectors basically and you would not have thought ahead of time you wouldn't have been like, oh we need to harden the RubyGems code base. You know what I mean? Because this is what they're going to focus on but it's just like, if you want to execute code, you need to download RubyGems and like PyPy, Artifactory, NPM, like these are all like ways of doing it.

14. 1:04:44

    And the fact of alignment is that you have to go through all of it, right? And like contain it and then like seal up all the cracks. So that's like one thing it's like, okay, well, you know, you do the sandbox, but then maybe you'll ask like, okay, why are we putting things in a sandbox? Why are you doing this sort of exploring? And then like, okay, but is it really that dangerous, right? Like what would happen?

15. 1:05:05

    So, okay, why do we do it?