Assessing The Risk Of AI-Enabled Computer Worms

AI has been used to support many aspects of cyberdefence and cyberoffence for some time. However, in light of recent rapid progress in AI, some experts have expressed concern that large and sudden improvements in future AI-cyber capabilities could pose severe risks if not managed properly.

These concerns have prompted several frontier AI companies to define offensive cyber capability thresholds in their safety and security policies. These policies state that: if an AI model reaches a certain level of cyber capability, then the company should not release the system until it has appropriately mitigated the risk. In turn, companies have built several AI-cyber benchmarks and “red teaming” exercises to test whether models trigger these thresholds.

However, at present, there is a lack of published AI-cyber threat models. Threat models are evidence-based analyses of how much AI capabilities increase risk: for example, a threat model would estimate the economic costs if AI showed strong performance on vulnerability discovery or malware development. Due to the lack of published threat models, AI companies’ cyber capability thresholds often lack clear justifications and diverge substantially from one another. It also limits policymakers’ foresight into emerging cyber risks, and makes it difficult to know whether current cyber model evaluation results warrant concern or not.

This report aims to help bridge the gap between model evaluations and estimates of societal risk by reviewing one AI-cyber threat model in depth. We focus specifically on how AI-enabled discovery of critical software vulnerabilities and development of powerful exploits of them could increase the risk of computer worms that damage data on a large number of devices. Historically, computer worms have been amongst the most damaging cyber risks.