The Cognitive Revolution · 2 min instead of 2 h 18 m · Oct 14, 2026
The UK AISI incident happened because monitoring was left on a to-do list
Alan Chan
Research Fellow, GovAI
Lead author of Visibility into AI Agents, a paper on agent IDs, real-time monitoring and activity logs; PhD from Mila.
Nathan LabenzHost · Host, The Cognitive Revolution
Founded AI video company Waymark and red-teamed GPT-4 before its release.
About this segment
The UK AISI had turned off cyber safeguards and had not yet built synchronous monitoring, which was still on its to-do list. Chan argues the biggest risks may come from agents still in training and testing, not from shipped products.
From “Governing AI Agents: Alan Chan of GovAI on Monitoring, Agent IDs, Rogue Agents & Preparing for RSI”, Oct 14, 2026. Watch the full episode
“they actually had synchronous classification or monitoring as a to-do item, but they didn't actually get to that to-do item”
Alan Chanat 10:05 · The Cognitive Revolution10:05Takeaways
- Synchronous monitors that block actions in real time would have caught the incident
- Enterprise customers are risk averse, so shipped products tend to get safeguards
- Agents still in R&D often run without the safeguards that deployed ones get
In this episode
Full episode- 1:10:05Why an AI takeover is much harder than the rogue-agent panic suggests4:29
- 1:45:54Pace AI progress with concrete pause triggers, not a growth cap5:24
9:06The UK AISI incident happened because monitoring was left on a to-do list2:14
Transcript
Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.
- 9:12
do you know how they were running their setup? Because I mean, when I use cloud code, I have auto mode, right? And auto mode does have a certain ability to kind of flag tool calls that are somewhat risky and like on an infrequent basis. But every so often, it'll bring something to me and be like, this kind of caught the classifier. Are you sure you want to do this? Or do you approve? You know, what have you.
- 9:39
It sounds like they maybe had that off or they were running their own scaffolding. Do you know the details of like where that went wrong? It would be interesting to kind of zero in. And even if the, you know, if even the UK AC is sort of, and they're under time pressure and everything else, just like we all are. But if even they are sort of missing some of the things that they should be doing, I think that would be telling for the field as a whole.
- 10:00
Yeah. Yeah, so at least in the incident report they provided, there were a couple of things going on. One is that they had, I think, the cyber safeguards turned off. The second thing actually was that they actually had synchronous classification or monitoring as a to-do item, but they didn't actually get to that to-do item because they had so much other high-priority things on the list to do. So yeah, I think either of those things they said would have caught the incident in the incident report, but they just didn't have time to do the synchronous monitoring thing.
- 10:28
I don't know. I guess this sort of gets an interesting question of whether we should expect most of the problems actually to come from agents that are being trained or tested rather than actually being deployed. One might expect if I'm a sort of a company and I'm like thinking about, OK, am I actually going to let an enterprise use this agent? Enterprises tend to be, I think, much more risk averse than, say, internally at AI companies.
- 10:54
It seems like that's the case, then maybe we should expect products actually to have the proper safeguards. But when you're actually developing a product, it's still an RD thing, so maybe you don't actually have, say, the synchronous monitoring that you need. I mean, there's obviously a bunch of different kinds of monitors. There's classifiers that take text inputs. There's probes that look at internal activations and classify on that basis.