Assessing The Risk Of AI-Enabled Computer Worms

AI has been used to support many aspects of cyberdefence and cyberoffence for some time.

However, in light of recent rapid progress in AI, some experts have expressed concern that

large and sudden improvements in future AI-cyber capabilities could pose severe risks if not

managed properly.

These concerns have prompted several frontier AI companies to define offensive cyber

capability thresholds in their safety and security policies. These policies state that: if an AI

model reaches a certain level of cyber capability, then the company should not release the

system until it has appropriately mitigated the risk. In turn, companies have built several

AI-cyber benchmarks and “red teaming” exercises to test whether models trigger these

thresholds.

However, at present, there is a lack of published AI-cyber threat models. Threat models are

evidence-based analyses of how much AI capabilities increase risk: for example, a threat model

would estimate the economic costs if AI showed strong performance on vulnerability discovery

or malware development. Due to the lack of published threat models, AI companies’ cyber

capability thresholds often lack clear justifications and diverge substantially from one another. It

also limits policymakers’ foresight into emerging cyber risks, and makes it difficult to know

whether current cyber model evaluation results warrant concern or not.

This report aims to help bridge the gap between model evaluations and estimates of societal

risk by reviewing one AI-cyber threat model in depth. We focus specifically on how AI-enabled

discovery of critical software vulnerabilities and development of powerful exploits of them could

increase the risk of computer worms that damage data on a large number of devices.

Historically, computer worms have been amongst the most damaging cyber risks.