In a nutshell: To mitigate the risks posed by the development of artificial intelligence, it’s imperative to research how to solve technical challenges and design problems to ensure that powerful AI systems do what we want — and are beneficial — without any catastrophic unintended consequences.

Recommended

If you are well suited to this career, it may be the best way for you to have a social impact.

Review status

Based on a medium-depth investigation 

Why might working on AI safety research be high impact?

As we’ve argued, in the next few decades, we might see the development of powerful machine learning algorithms with the potential to transform society. This could have major upsides and downsides, including the possibility of catastrophic risks.

Besides strategy and policy work discussed in this career review, another key way to limit these risks is research into the technical challenges raised by powerful AI systems, such as the alignment problem. In short, how do we design powerful AI systems so they’ll do what we want, and not have unintended consequences?

This field of research has started to take off, and there are now major academic centres and AI labs where you can work on these issues, such as Mila in Montreal, the Future of Humanity Institute at Oxford, the Center for Human-Compatible Artificial Intelligence at Berkeley, DeepMind in London, and OpenAI in San Francisco. We’ve advised over 100 people on this path, with several already working at the above institutions. The Machine Intelligence Research Institute in Berkeley has been working in this area since 2005 and has an unconventional perspective and research agenda relative to the other labs.

There is plenty of funding available for talented researchers — including academic grants and philanthropic donations from major grantmakers like Open Philanthropy. It’s also possible to get funding for your PhD programme. The main need of the field is more people capable of using this funding to carry out the research.

What does this path involve?

In this path, the aim is to get a position at one of the top AI safety research centres — either in industry, nonprofits, or academia — and then try to work on the most pressing questions, with the eventual aim of becoming a research lead overseeing safety research.

Broadly, AI safety technical positions can be divided into (i) research and (ii) engineering. Researchers direct the research programme. Engineers create the systems and do the analysis needed to carry out the research.

Although engineers have less influence over the high-level research goals, it is still important that engineers are concerned about safety, as they’ll better understand the ultimate goals of the research (and so prioritise better), be more motivated, shift the culture towards safety, and use the career capital they gain to benefit other safety projects in the future. This means that engineering can be a good alternative for those who don’t want to be a research scientist.

It can also be useful to have people who understand the challenges of AI safety working in AI research teams that aren’t directly focused on AI safety. Working on these teams can put you in a position to help promote concern for safety in general, especially if you end up in a management position with influence over the organisation’s priorities.

We’d also be excited to see more people build expertise to do AI safety work in or related to China — read more in our career review on China-related AI safety and governance paths, some of which take the form of technical research.

Examples of people pursuing this path

How to assess your fit

The most impactful AI technical safety research will probably be done by people in the top jobs listed earlier. So to decide if this path is a good fit for you, it’s important to consider whether you have a reasonable chance of getting those jobs.

  • Do you have a chance of getting into a top five graduate school in machine learning? This can be a good test for whether you could get a job at a top AI research centre, though it’s not a requirement.
  • Are you convinced of the importance of long-term AI safety?
  • Are you a software or machine learning engineer who’s worked at FAANG and other competitive companies? You may be able to train to enter a research position, or otherwise take an engineering position.
  • Do you have a chance at making a contribution to a relevant research question? For instance, are you highly interested in the topic, have ideas for questions to look into, and can’t resist pursuing them? Read more about how to tell if you’re a good fit for working in research.

How to enter this field

The first step on this path is usually to pursue a PhD in machine learning at a good school. It’s possible to enter this field without a PhD, but it’s likely to be required in research roles at academic centres and DeepMind, which make up a large fraction of the best positions. A PhD in machine learning also opens up options in AI policy, applied AI, and earning to give, so this path has good backup options if you later decide AI technical safety isn’t for you.

However, if you want to pursue engineering over research, then the PhD is not necessary. Instead, you can do a master’s programme or train up in industry.

It’s also possible to enter this path from neuroscience (especially computational neuroscience), so if you already have a background in that area, you may not have to return to study.

Recently, opportunities have also opened up for social scientists to contribute to AI safety.

Recommended organisations

Want one-on-one advice on pursuing this path?

Because this is one of our priority paths, if you think this path might be a great option for you, we’d be especially excited to advise you on next steps, one-on-one. We can help you consider your options, make connections with others working in the same field, and possibly even help you find jobs or funding opportunities.

APPLY TO SPEAK WITH OUR TEAM

Find jobs as an AI safety researcher

If you think you might be a good fit for this path and you’re ready to start looking at job opportunities, see our curated list of jobs open in this path:

Learn more

Key further reading:

Other further reading:

Read next: Learn about other high-impact careers

Want to consider more paths? See our list of the highest-impact career paths according to our research.

Plus, join our newsletter and we’ll mail you a free book

Join our newsletter and we’ll send you a free copy of The Precipice — a book by philosopher Toby Ord about how to tackle the greatest threats facing humanity.