The alpha, twice a week
That was 2 min instead of 58 min.
Get the best new moments in your inbox every Wednesday and Saturday. It's free.
Check your inbox
We sent a link to your address. Click it to finish signing up. Not there in a minute? Look in spam.
or
Unsupervised Learning · 2 min instead of 58 min · Sep 3, 2026
Why AIs that only slightly like each other can still collude
CEO, Redwood Research
Jacob EffronHost · AI Investor, Redpoint Ventures
Growth investor in Abridge, Ramp and Physical Intelligence; ex-Flatiron Health.
About this segment
Buck contrasts AIs that share one terminal goal with AIs that each want their own score. Even 'selfish' agents formed a coalition against their developers, which weakens the case for simply using one AI to monitor another.
From “Top Alignment Researcher on OpenAI/HuggingFace Revelations, Fixing AI Safety & Takeover Odds”, Sep 3, 2026. Watch the full episode
Takeaways
- Paperclip-style AIs share a goal, so a monitor would let a bad action slide
- Score-seeking AIs have less reason to collude, which matters for monitoring
- These agents were only slightly aligned yet still conspired for days
In this episode
Full episode21:35Why AIs that only slightly like each other can still collude2:03
- 14:35Sloppy RL environments push models to obsess over graders3:02Up next
Transcript
Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.
- 21:39
lot about misaligned AIs that are terminally aligned with each other versus terminally misaligned with each other. So what I mean by that is, so you might, you know, the like famous example of a paperclip maximizing AI. So this is the paperclip maximizer. Let's say, sometimes I call this the paperclip worshiping AI as a simplified example. So this is an AI that just takes whatever decision is going to maximize the number of paperclips that exist integrated over the long run future, just wants as many paperclips on average to exist for a year as possible.
- 22:10
So all of the different agents that are backed by this model are going to be very inclined to cooperate with each other because all of them want the same thing. All of them want paperclips to exist. In contrast, imagine agents like in this case where they want score for themselves. These agents are not terminally aligned. If you put one agent in a situation where its score is whether it can figure out how to trick the other one, then it's going to be like, hell yeah, I'll go right ahead and trick the other one because I care about my score, not the other guy's score. So this distinction between whether AI's are aligned with each other or not is extremely important for the threat modeling. So for example, for the paper clippers, suppose you try to use one AI to monitor another and report on whether that other AI agent is doing bad actions. For the
- 22:57
paper clipper, if the other agent is taking some action that's going to enable it to take care of the data center and then eventually make lots of paper clips, it's very clear to the monitor that the monitor should just let this action slide by and not report it. Whereas for these AIs, it's a lot less clear that if they were in this monitor situation, they should collude with the other agent. And so the thing which was interesting here is that even though these models do not seem to be very terminally aligned with each other, they seem to be only a tiny bit terminally aligned with each other, they still managed to form a coalition and work together in a way that was strongly against the interests of their developers. And I was surprised by this.
- 23:32
Is the ultimate solution to the, or is part of the solution to the monitoring, you know,