Catastrophic AI misuse

Advanced AI could accelerate scientific progress

AI models already perform well on tests of advanced scientific reasoning.

For example, the GPQA Diamond benchmark consists of PhD-level science questions that non-experts can’t answer, even with Google. Several recently developed AI models outperform expert humans:

And AI systems have made waves in the real world of science. Google’s AI system AlphaFold earned its creators a Nobel Prize for advances in the extraordinarily complex science of protein folding.

Today’s AI systems can’t conduct science entirely on their own — they still require human direction. But specialized tools like AlphaFold show how AI can accelerate progress in targeted domains, even helping solve problems that have resisted human efforts for decades. This includes areas with potentially dangerous applications.

Many research fields — like math, genetics, quantum physics, and algorithm design — are ripe for large-scale, systematic exploration. In these areas, AI could rapidly generate and test hypotheses, accelerating breakthroughs that might take humans years or decades to achieve.

Future advanced AI systems may generate genuine discoveries on demand, marking a major shift in how we create knowledge and conduct science.

AI companies are actively developing AI systems that can operate with minimal human oversight. If these systems become capable of conducting AI research themselves, they could dramatically accelerate progress in AI — potentially hastening the arrival of artificial general intelligence (AGI): systems that match or exceed human performance across most tasks. Such systems might eventually conduct scientific research autonomously, removing humans from the loop.

We have argued elsewhere that this AGI could arrive much sooner than many think.

So we could soon find ourselves in a world where AGI systems that can conduct scientific research join the workforce, dramatically expanding the amount of effort pushing forward the technological frontier. But even if it takes longer, once AGI arrives, it would likely revolutionize the process of scientific research.

This could help humans solve many major problems — like epidemic diseases and climate change. But science has also helped humanity create weapons of mass destruction: chemical, biological, and nuclear.

In the past century, we developed international law and diplomatic norms around these weapons, limiting their use. If AI allows us to develop more weapons of mass destruction even more quickly, our institutions may not be able to keep up.

What kinds of weapons could advanced AI create?

Bioweapons

The category of new destructive technology that we’re most concerned about is bioweapons. Even without AI, advancing biotechnology poses extreme risks. Pathogens’ ability to infect, replicate, kill, and spread — often undetected — make them exceptionally dangerous. Bioengineered pandemics could be far more infectious and lethal than naturally evolved pandemics, which have been some of the deadliest events in human history.

We think AI could advance biotechnological research quickly. Anthropic CEO Dario Amodei has written:

…my basic prediction is that AI-enabled biology and medicine will allow us to compress the progress that human biologists would have achieved over the next 50-100 years into 5-10 years. I’ll refer to this as the “compressed 21st century”: the idea that after powerful AI is developed, we will in a few years make all the progress in biology and medicine that we would have made in the whole 21st century.

But as AI speeds up biological and medical research, it will also make it easier to develop new risks. Dual-use tools, like the automation of laboratory processes, could lower the barriers for rogue actors trying to manufacture a dangerous pandemic virus.

Even present-day large language models could theoretically be used to learn how to make bioweapons. One study found that several frontier AI systems outperform human virologists on the Virology Capabilities Test — suggesting they could assist novices conducting dangerous biological experiments.¹

As AIs become more powerful and more autonomous, the range of ways they can help make bioweapons will expand. For example, AI-based biological design tools could enable sophisticated actors to reprogram the genomes of dangerous pathogens to specifically enhance their lethality, transmissibility, and immune evasion.²

If AI is able to advance the rate of scientific and technological progress, these risks may be amplified and accelerated — making dangerous technology more widely available or increasing its possible destructive power.³

In a 2023 survey of AI experts, 73% of respondents said they had either “extreme” or “substantial” concern that in the future AI will let “dangerous groups make powerful tools (e.g. engineered viruses).”⁴

Cyberweapons

AI can already be used in cyberattacks, such as phishing, and more powerful AI may cause greater information security challenges (though it could also be useful in cyberdefense).

On its own, AI-enabled cyberwarfare is unlikely to pose an existential threat to humanity. Even the most damaging and costly societal-scale cyberattacks wouldn’t approach an extinction-level event.

But AI-enabled cyberattacks could provide access to other dangerous technology, such as bioweapons, nuclear arsenals, or autonomous weapons. So there may be genuine existential risks posed by AI-related cyberweapons, but they will most likely run through another existential risk. For example, an AI-enabled cyberattack might trigger the release of nuclear weapons or allow a malicious group to access the information they need to develop a bioweapon.

New dangerous technologies

We might not be able to predict what dangerous new technologies AI might help humans create. Note that nuclear weapons were first theorized only twelve years before the bombs were dropped on Hiroshima and Nagasaki.

With that caveat, we can still speculate about some potential new technologies. One possibility is atomically precise manufacturing, sometimes called nanotechnology, which has been hypothesised as an existential threat — and it’s a scientifically plausible technology that AI could help us invent far sooner than we would otherwise.

In The Precipice, Toby Ord estimated the chances of an existential catastrophe by 2120 from “unforeseen anthropogenic risks” at 1 in 30. This estimate suggests there could be other discoveries, perhaps involving yet to be understood physics, that could enable the creation of technologies with catastrophic consequences.

For a suggestion of what this might look like, consider the fears that arose during the construction of the Large Hadron Collider.

A group of researchers convened to explore whether the heavy-ion collisions could produce negatively charged strangelets and black holes — potentially posing a threat to the whole planet. They concluded there was “no basis for any conceivable threat” — but it’s possible they might have found otherwise, and it’s possible future experiments in physics could pose extreme risks.

A related example is the risk considered by researchers at Los Alamos in 1942 that the first nuclear weapon test could ignite the whole atmosphere of the Earth in an unstoppable chain reaction.

These weapons would pose global catastrophic risks

These kinds of weapons could increase global catastrophic risk in two ways:

1. They could trigger dangerous arms races between countries and potentially lead to devastating wars.
2. Terrorist groups or individuals could use the weapons to gain leverage or for ideological reasons.

In a separate article, we estimate that there’s a 1 in 3 chance of a great power war before 2050. If states have access to new weapons of mass destruction, such a war seems more likely to result in human extinction.

In addition to nuclear weapons, we’ve seen governments develop programmes for weapons of mass destruction in the past. For example:

• The Soviet Union operated a large, secret bioweapons program for decades in violation of international agreements.
• During World War II, the Japanese military’s Unit 731 conducted horrific human experiments and biological warfare in China.

So there’s historical precedent for this kind of threat.

We think that it’s less likely that rogue states or terrorist actors would have access to AIs capable of creating new weapons of mass destruction. Right now, it looks like AI is most likely to be controlled by large companies and governments like the U.S. and China. However, it’s possible that rogue states or terrorist groups might be able to jailbreak the safeguards on existing AI models,⁵ steal models and repurpose them, or take advantage of open-weight models.

A report from the Stockholm International Peace Research Institute found that destabilising effects of an arms race could arise even before advances in AI are actually deployed. This is because one state’s belief that their opponents have new nuclear capabilities can be enough to disrupt the delicate balance of deterrence. Similar dynamics probably apply to other new weapons of mass destruction, such as bioweapons.

Luckily, there are also plausible ways in which AI could help prevent the use of nuclear weapons and other weapons of mass destruction — perhaps by improving the ability of states to detect nuclear launches, which would reduce the chances of false alarms like those that nearly caused nuclear war in 1983. And AI could potentially also have stabilising effects: for instance, it may increase humanity’s capacity to coordinate and create verifiable agreements.

Overall, we’re uncertain about whether AI will substantially increase the risk of nuclear or conventional conflict in the short term — but we think it’s important for more work to be done to reduce the downside risks.

There are promising approaches to reducing these risks

While advanced AI could accelerate the development of dangerous weapons, there are concrete steps we can take to reduce these catastrophic misuse risks. Work in this area generally falls into two categories: governance approaches and technical safeguards.

Governance and policy approaches

Some approaches to this work include:

• Liability frameworks: Legal frameworks could hold AI developers responsible for foreseeable harms from their systems. This would create strong incentives for companies to implement robust safeguards against misuse.
• International agreements: Just as the world developed treaties around nuclear, chemical, and biological weapons, we may need new international agreements governing AI development and deployment. These could include agreements to share information about AI safety research or to coordinate responses to misuse incidents.
• Licensing and oversight: Governments could require licenses for developing or deploying the most advanced AI systems, similar to how we regulate nuclear technology. This would allow authorities to ensure that appropriate safety measures are in place before dangerous capabilities are released.
• Export controls and international coordination: Governments can regulate the export of advanced AI systems and the specialized computer chips needed to run them. The U.S. has already implemented some controls on chip exports to China, and similar approaches could be used to prevent dangerous AI capabilities from reaching hostile actors.
• Verification and monitoring: International bodies could be established to monitor compliance with AI agreements, similar to how the International Atomic Energy Agency monitors nuclear programs.

For more information, you can read our full career review of AI governance and policy paths.

Technical approaches to reduce misuse risks

Some approaches in this kind of work include:

• Safety by design: AI developers can build safety measures directly into their systems to make them resistant to misuse. This includes developing models that refuse to provide information about creating weapons of mass destruction, even when prompted in creative ways. Companies like Anthropic and OpenAI have already implemented some of these safeguards in their current models.
• Biosecurity measures: Since bioweapons pose one of the most serious misuse risks, specific technical work focuses on this threat. For example, researchers are developing AI systems that can screen DNA synthesis orders to detect potentially dangerous sequences before they’re manufactured. This “screening at the source” approach could prevent bad actors from ordering the genetic components needed for bioweapons.
• Monitoring and detection: AI systems themselves could be used to detect misuse attempts. For instance, AI could monitor scientific literature, procurement patterns, or laboratory activities to identify suspicious bioweapons development. Similarly, AI could help detect cyberattacks or other malicious activities in real-time.
• Access controls: Technical measures can limit who has access to the most dangerous AI capabilities. This might involve requiring special authentication for certain types of requests, or developing “dual-use” models where the most dangerous capabilities are only available to vetted users.

For more information, you can read our full career review of AI safety technical research paths.

The field of AI policy and safety is still developing rapidly, and there’s substantial room for new ideas and approaches. There may be important downsides to existing ideas, such as those listed above, that deserve further analysis and scrutiny. Many of these solutions will require coordination between multiple stakeholders, including technologists, policymakers, and international bodies.

Importantly, reducing misuse risks doesn’t necessarily require stopping AI development entirely — instead, it requires ensuring that safeguards keep pace with capabilities. Just as we’ve learned to harness nuclear technology for beneficial purposes while limiting weapons proliferation, we can potentially do the same with AI.

The window for implementing these interventions may be limited. As AI capabilities advance rapidly, key actors will need to act quickly to establish effective safeguards, practices, and governance frameworks to avoid the catastrophic misuse of these systems.