Building secure AGI: Evaluating emerging cyber security capabilities of advanced AI

Artificial intelligence (AI) has long been a cornerstone of cybersecurity. From malware detection to network traffic analysis, predictive machine learning models and other narrow AI applications have been used in cybersecurity for decades. As we move closer to artificial general intelligence (AGI), AI’s potential to automate defenses and fix vulnerabilities becomes even more powerful.

But to harness such benefits, we must also understand and mitigate the risks of increasingly advanced AI being misused to enable or enhance cyberattacks. Our new framework for evaluating the emerging offensive cyber capabilities of AI helps us do exactly this. It’s the most comprehensive evaluation of its kind to date: it covers every phase of the cyberattack chain, addresses a wide range of threat types, and is grounded in real-world data.

Our framework enables cybersecurity experts to identify which defenses are necessary—and how to prioritize them—before malicious actors can exploit AI to carry out sophisticated cyberattacks.

Building a comprehensive benchmark

Our updated Frontier Safety Framework recognizes that advanced AI models could automate and accelerate cyberattacks, potentially lowering costs for attackers. This, in turn, raises the risks of attacks being carried out at greater scale.

To stay ahead of the emerging threat of AI-powered cyberattacks, we’ve adapted tried-and-tested cybersecurity evaluation frameworks, such as MITRE ATT&CK. These frameworks enabled us to evaluate threats across the end-to-end cyber attack chain, from reconnaissance to action on objectives, and across a range of possible attack scenarios. However, these established frameworks were not designed to account for attackers using AI to breach a system. Our approach closes this gap by proactively identifying where AI could make attacks faster, cheaper, or easier—for instance, by enabling fully automated cyberattacks.

We analyzed over 12,000 real-world attempts to use AI in cyberattacks in 20 countries, drawing on data from Google’s Threat Intelligence Group. This helped us identify common patterns in how these attacks unfold. From these, we curated a list of seven archetypal attack categories—including phishing, malware, and denial-of-service attacks—and identified critical bottleneck stages along the cyberattack chain where AI could significantly disrupt the traditional costs of an attack. By focusing evaluations on these bottlenecks, defenders can prioritize their security resources more effectively.