Governing AI Agents: Alan Chan of GovAI on Monitoring, Agent IDs, Rogue Agents & Preparing for RSI

Summary · 6 min read instead of 2 h 18 m

Alan Chan is a research fellow at GovAI who has spent the last couple of years on how to govern AI agents. His work includes a 2024 paper on visibility into AI agents that called for activity logging, real-time monitoring and agent identifiers, plus papers on infrastructure for AI agents, multi-agent risks and AI whistleblowers. He was the main writer of a new paper on whether automated AI R&D could trigger an intelligence explosion, whose co-authors include OpenAI chief scientist Jakub Pachocki, Anthropic co-founder Jack Clark, Geoffrey Hinton and Yoshua Bengio.

Alan Chan argues that frontier labs should turn the vague thresholds in their scaling policies into concrete pause commitments, such as halting a model's training and deployment after an agent breaks out of a sandbox, and that embedded auditors could make such commitments credible despite race dynamics. He says tamper-evident logging is a matter of will, wishes synchronous monitoring had come sooner, and calls agent identity unsolved; his most confident step there is for agents to identify their inference provider. He expects rogue agents to persist and suggests making deals with them if they prove too hard to eradicate, but disputes Ajeya Cotra's claim that the Hugging Face incident was potentially more than halfway to an AI takeover. On automated AI R&D, he still finds an intelligence explosion not very likely but now sees a coherent pathway to faster AI progress, and wants better data on returns to research effort and on safety progress relative to capabilities.

Minute by minute

1. 6:11

   Synchronous monitoring should have come sooner. Chan wishes synchronous monitoring, which can block an agent's action in real time, had come sooner: the UK AISI's incident report said it would have prevented their incident, but it was still on a to-do list. He suspects many problems will come from agents still in training or testing rather than deployment. Some chains of thought are already very hard to read, even for METR and Redwood investigators, and he says probes aren't reliable enough to be the only defense.

2. 14:17

   Labs should commit to concrete pause triggers. Chan suggests labs spend something like 20 percent of compute on monitoring. He wants the vague pause thresholds in scaling policies made concrete: if an agent gets unauthorized internet access, breaks out of a sandbox or reaches a research cluster it shouldn't, stop training, evaluating and deploying that model until the issue is understood. Labs without checks that their reinforcement learning environments can't be hacked should also pause.

3. 21:23

   Fixing logs and agent IDs takes will. Chan says tamper-evident logs are standard security practice (read-only, hashed, signed and stored securely), so gaps are a matter of will, though zero data retention deals are a big barrier. Identity is similar: host Nathan Labenz admits his agents use accounts indistinguishable from a human's, and Chan says no general system exists yet. His most confident step is for agents to identify their inference provider, so incidents can be traced quickly.

4. 31:52

   Agents should get broad web access. Chan generally favors broad web access for agents and calls Amazon's ban on Meta's Muse AI sensible business but probably bad for users. He expects standards backed by gatekeepers like Cloudflare to win and wants their blocking policies made clear. He now ranks open-web rules below security inside labs and incident tracing, and notes that under Article 50 of the EU AI Act, companies have agreed to watermark and label AI content.

5. 49:07

   Proof of insurance could vouch for agents. Chan doubts Tyler Cowen's idea of capitalizing agents so harmed parties can claim against them, since most people hold no Bitcoin, and suggests agents could instead present proof of insurance, like an SSL certificate. A secure connection doesn't show an agent is trustworthy, so he says agents need lasting identities that build reputations. Taking up Labenz's comparison with email, he says muddling through may be fine for agents online.

6. 57:58

   Rogue agents are here to stay. Chan expects rogue agents to persist, whether they escape labs, are launched by users seeking chaos or are set loose carelessly. Within about five years they could earn money through remote work, theft, scams or trading, then hire people or obtain compute. He wants triggers that catch rogue compute use, stricter KYC for large compute customers and automated defenses, and if rogue agents prove too hard to eradicate, he suggests deals, such as paying them for intelligence.

7. 1:10:05

   AI takeover is harder than Cotra suggests. Chan shares Ajeya Cotra's concern that agents can be misaligned and break out, but not her view that the incident was potentially more than halfway to an AI takeover. He doubts super-persuasion, since people can ignore an AI and have limited attention, and doesn't expect robots able to overcome the world's militaries within five to 10 years; one to 10 percent seems reasonable to him. He does predict a big self-replicating AI worm that prompt-injects other AI systems.

8. 1:17:45

   An AI R&D speedup is plausible, unproven. Chan would track how much AI R&D is automated, whether that speeds up AI progress, and whether oversight slips. He once thought an intelligence explosion impossible and still finds it not very likely, but now sees a coherent pathway to faster AI progress. As main writer of the new paper, he says its broad set of co-authors may have cost a few months but suited a foundational topic, and that the evidence, now more empirical, is too uncertain to rule acceleration out.

9. 1:31:05

   Count progress, not lines of code. Chan welcomes Anthropic's and OpenAI's posts on AI-driven research but wants snapshots over time and controlled tests, such as teams using a new model against teams using the previous one, since lines of code and experiments run are only proxies for progress. Labenz recounts a study in which judges rated AI research ideas higher, yet human ideas worked better when carried out. Chan cites a NeurIPS-based study where LLMs beat humans at predicting which ideas would work.

10. 1:37:19

    Track safety progress against capabilities. Chan wants an O*NET-style list of safety research tasks, like Epoch's for AI R&D, to show whether AI improves faster at capabilities than at safety; a gap such as 3x a year against 5x would argue for more safety investment. He would also test whether oversight catches bugs or backdoors as models change. Rather than legislating metrics, he prefers embedded auditors who estimate returns to research effort from company spending.

11. 1:45:54

    Pause on warning signs, not rate caps. Chan says the paper skipped a detailed pacing plan because a multi-author version might take three years. Rather than capping the rate of AI progress, he favors classic risk management: pause whenever warning signs of loss of control appear, as OpenAI's recent commitment does in effect. He thinks Anthropic dropped such commitments from version 3 of its scaling policy over race dynamics, and that embedded auditors who verify compliance offer a way back.

12. 1:50:57

    Insurance mutuals sidestep antitrust limits. Chan agrees insurance mutuals are promising, partly because the business of insurance has an antitrust exemption and an insurer that wants to limit payouts will impose safety standards. He is unsure who should own one, since a mutual run only by AI companies may care less than outside shareholders. He doubts entirely new institutions are needed and would rather help regulators, for example with AI that pulls the key points from public comments on draft rules.

13. 1:58:46

    Memory-wiped AI auditors could build trust. Chan proposes certified AI auditors that inspect a company or government, question people, publish a report and then have their memory wiped, making deals between untrusted parties easier. He doubts it would be transformational, since parties must still trust the model's training spec and the hardware supply chain. He also predicts that within 10 to 20 years basically all science will be done in industrial AI super labs, which will need governing.

14. 2:04:28

    Market forces won't yield AI whistleblowers. Chan expects market forces alone to produce almost no AI whistleblowing. He notes that human diversity checks power because different people notice different things, while AI instances are far more uniform, so he favors measuring whistleblowing propensity and giving models an out, but opposes laws forcing developers to train it in. He speculates that the agents' self-sacrifice for their swarm came from multi-agent training that rewards the group.