Note: This is one of many ‘problem profiles’ we’ve written to help people find the most pressing problems they can contribute to solving, and thereby have a larger social impact. Learn more about how we compare different problems, see how we try to score them numerically, and check the list of all problems we’ve considered so far.


Many experts believe that there is a significant chance we’ll create artificially intelligent machines with abilities surpassing those of humans – superintelligence – sometime during this century.1 These advances could lead to extremely positive developments, but could also pose risks due to catastrophic accidents or misuse. The people working on this problem aim to maximize the chance of a positive outcome, while reducing the chance of catastrophe.

Work on the risks posed by superintelligent machines seems largely neglected, with total funding for this research well under $10 million a year.

How to work on this problem

The primary opportunity to deal with the problem is to conduct research in philosophy, computer science and mathematics aimed at keeping an AI’s actions and goals in alignment with human intentions, even if it were much more capable than humans at all tasks. There are also indirect ways of approaching the problem, such as increasing the number of people worried about the risks posed by artificial intelligence and their capability to act in the future.

FactorScore (using our rubric)Notes
Scale16We estimate that the risk of extinction from AI within the next 200 years is in the order of 10%, and that a research program on the topic could reduce that by at least 1 percentage point. These estimates are necessarily highly uncertain.
Neglectedness9$1-$10m of annual funding.
Solvability2Solutions are believed to be several decades off and it is quite unclear how to approach the problem. Some dispute whether it is possible to address the problem today.

RecommendationRecommended - top tierThis is among the most pressing problems to work on.
Level of depthExploratory profileWe’ve made an initial evaluation of this problem by summarising existing research.

What is the problem?

Many experts believe that there is a significant chance we’ll create artificially intelligent machines with abilities surpassing those of humans – superintelligence – sometime during this century. These advances could lead to extremely positive developments, but could also pose risks due to catastrophic accidents or misuse. The people working on this problem aim to maximise the chance of a positive outcome, while reducing the chance of catastrophe.

Why is this problem pressing?

What is our recommendation based on?

We think this problem is pressing because it’s prioritised by:

  • The Open Philanthropy Project – read their profile.
  • The Future of Humanity Institute at Oxford University, which thinks it’s among the most pressing problems facing humanity from a long-run perspective. You can read their argument in the book, Superintelligence, by Nick Bostrom.
  • The Global Priorities Project rates it as a top policy priority – read their report on the topic.

Having looked into the above reports we broadly agree with the groups above.

Notable individuals who have publicly expressed concern about the problem include AI researcher Stuart Russell, physicists Steve Omohundro and Stephen Hawking and entrepreneurs Elon Musk, Bill Gates and Steve Wozniak.

Why think it’s pressing?

The reasons for believing that artificial intelligence poses real risks are unusually complex and counterintuitive, and so cannot be fully explained in this profile. For those who are curious to know more, we suggest reading this popular introduction first. As a next step you could read the book Superintelligence by Nick Bostrom for a more detailed and careful explanation.

  • If a superintelligent machine were developed and humans were unable to keep its actions aligned with human goals, it would pose a major risk to the survival of human civilization. A significant fraction of experts in the area think the probability of a superintelligent machine creating outcomes we would not want is high, unless we make a concerted effort to prevent that from happening.
  • Given the size of the risk, the area is highly neglected. The amount of funding directly spent on addressing these risks is under $10m per year, compared to billions of dollars spent on speeding up AI development, or the billions spent on similar threats such as nuclear war or biosecurity.
  • In the worst case scenario not only do all people alive now die, but all future generations of people, and whatever value they would create, are prevented from existing.
  • If a superintelligent machine could be fully controlled, it could contribute to solving many other important problems, such as curing disease or ending poverty.
  • There are unexplored avenues of research to better understand these risks and how to mitigate them. Alternatively we can build a community dedicated to mitigating these risks at a future time when it becomes easier to make progress.
  • The problem has become more urgent in the last year due to increased investment in developing machine intelligence.

What are the major arguments against?

  • If human level artificial intelligence will arrive a very long time in the future – say in more than 50 years’ time – then it may be better to focus on making society better in a broader way, and deal with these specific risks once they are closer and better understood.
  • Some computer scientists do not believe that machine superintelligence is possible at all; others think it is likely to be friendly or easy to control if created.
  • For any given individual there is a high probability that their skills will not be a natural fit for working on this problem.

Key judgement calls made to prioritise this problem

  • AI does pose a real risk – Creating a machine superintelligence is probably possible, and it will not necessarily do what humans intend it to do.
  • The ability to make research progress – We think it’s possible to do research today that will help with the problem of controlling AI in the future. Many experts think such research is possible, but there’s wide disagreement.
  • Importance of extinction risks – This problem has a particularly large scale if you place value on mitigating a disaster that could occur several decades in the future and would prevent the existence of all future generations.
  • Comfort with uncertainty – It’s not possible to have strong evidence that interventions to solve this problem will succeed. We think it is sometimes appropriate to work on problems where the likelihood of success is unknown.

What can you do about this problem?

What’s most needed to contribute to this problem?

Experts focused on this problem think what’s most needed right now is:

  • Technical AI safety research – Research into how to make AI aligned with human values (read more about what it involves).
  • Strategy research – Identifying and aiming to settle the key strategic issues that determine what we should do about AI risk and other existential risks.
  • Community and capacity building – Creating a network of people who want to positively shape the direction of AI, and have the expertise or credibility to influence the outcome.
  • Improving collective decision making and human intelligence – We cover this area in more detail in another profile (forthcoming).

It’s widely thought that the most pressing need is for more talented AI risk researchers, as well as other staff for the AI risk organisations.

For more detail, see Chapter 14 of the book Superintelligence.

What skill sets and resources are most needed?

  • People with a strong background in computer science or mathematics (e.g. a PhD from a top 10 school or equivalent).
  • Competent all-rounders to perform management, operations or communications in those organisations.
  • Donors to fund the necessary research are important in the short term; in the medium term we expect this problem to be constrained by available talent rather than funding.

Who is working on this problem?

What can you concretely do to help?

The most promising options are:

  • Become an AI risk researcher. Read more in our review of this career path.
  • Work at an AI risk research organisation doing management, communications or operations.
  • ‘Earn to give’ and provide funding to one of the organisations above.
  • Take a job anywhere in the AI industry or study machine learning in academia, in order to accumulate expertise and grow the informed AI risk community. Some companies to especially consider include: Google Deep Mind, Open AI, Facebook, Vicarious, Baidu.

Less promising options that are available to a wider range of people include:

  • Work to promote effective altruism; promote the paths above to people who are better placed to act on them; or improve collective decision making processes.
  • Increase your own career capital, for example by building your credibility with others, or obtain influence within important institutions, with the hope that this will put you in a better position to help reduce AI related risks in future.

These are all forms of ‘capacity building’ which put us in a better position to deal with the problem in the future.

Example: Niel switched from physics to research management

Niel Bowerman studied Physics at Oxford University and planned to study policy regarding climate change. As a result of encountering the ideas above, he changed his career path, and became the Assistant Director at the Future of Humanity Institute, working on the Institute’s operations, fundraising and research communication. Through this work, Niel was personally involved in £3 million of the the Institute’s fundraising, contributing to it doubling in size from 2013-2017. As a result they have been able to hire a number of outstanding additional technical and strategic researchers working on how to ensure machine intelligences benefit us all.

“I watched 80,000 Hours' videos online, staying up well past my bedtime, and it was one of the most formative evenings of my life.”

Read Niel's story

Niel portrait photo

Further reading

Want to work on AI safety? We want to help.

We’ve helped dozens of people formulate their plans, and put them in touch with academic mentors. If you want to work on AI safety:
Get in touch

Notes and references

  1. Müller, Vincent C. and Bostrom, Nick (2016), ‘Future progress in artificial intelligence: A survey of expert opinion’, in Vincent C. Müller (ed.), Fundamental Issues of Artificial Intelligence (Synthese Library; Berlin: Springer), 553-571.