AI safety syllabus

This page was written by Jan Leike, with contributions and comments by David Krueger, Jelena Luketina, Victoria Krakovna, Daniel Dewey, Laurent Orseau, and others. It is intended as a guide to working on technical aspects of AI safety. See our guide to working in AI policy and strategy for another approach.

This is a syllabus of relevant background reading material and courses related to AI safety. It is intended as a guide for undergraduates in mathematics and computer science planning their degree, as well as people from other disciplines who are thinking about moving into AI safety. It includes tips how to design your degree, how to transition into research, and the relevant conferences. This is not intended as a general guide of how to become a researcher.

Reading List

This is a list of the most relevant reading topics and the appropriate material. The chapter recommendations are indicative of what you should know. If you find the topic interesting, read more! As an undergraduate student, you can plan these courses into your degree. As a graduate student, you can use the provided material to extend your knowledge into areas that you do not have much background in. Focus on the textbooks and lecture notes and use the video lectures as supplementary materials. Doing plenty of exercises is usually a good idea to make sure that you actually understand a topic instead of just thinking you understand it.

Some of the relevant areas might not be offered as courses by your university. You can always read the listed books in your free time, or try to find a MOOC on Coursera.

More remotely related are the following areas.

Degree

Undergraduate Degree

Ideally your undergraduate degree would be mathematics and computer science (for example, a bachelor’s degree in math and a master in computer science). But this does not mean that an undergraduate degree in a different related discipline like neuroscience or physics would be wasted. Make sure you have a solid handle on the relevant mathematics (linear algebra, calculus, statistics, …)!

For your undergraduate thesis, find someone who supervises well and who has time for you (not the most famous/cool professor). Work on a topic that your supervisor finds interesting (to get lots of feedback). Pursuing your own ideas at this point is risky and usually means that you don’t get much supervision. Do something theoretical, preferably in computer science. Find an interesting research group and start doing research early in your degree (it helps a lot if you have clever things to say about their research). Ideally, you should get out of a Master’s degree with at least one publication at an international conference. It’s not a big deal if this delays your degree.

Other tips:

  • If you find a topic interesting, take more classes even if they don’t seem related
  • Choose harder classes over easier ones (favor math courses and theoretical computer science courses over applied computer science courses)
  • Choose your thesis by supervisor, and not necessarily by topic
  • Publications are great, they are a considered a good predictor of your academic potential (even if you are not the first author). As such, they are very helpful when applying for PhD programs
  • Read general advice on whether a PhD is for you and how to approach it
  • Attend MIRIx workshops if they exist in your area

PhD

Getting a PhD is generally an excellent idea and usually a prerequisite for someone to hire you as a researcher. A PhD will not only put you at the cutting edge of research, but also teach you the relevant soft skills (how to write papers, communicate complex ideas, etc.).

Your PhD should be in machine learning, reinforcement learning, statistics, or another discipline related to artificial intelligence. Focus on getting the required expertise first. If you feel comfortable in your area, shift your focus on to AI safety (e.g., in your final 1-2 years). For relevant problems, see:

Research Groups

The following is an non-exhaustive list of research groups where you could apply for internships and PhD candidacy. Make sure you look at their research and see how it relates to your interests. Needless to say, it is not a good idea to mass-email everyone on this list.

Conferences

For your publications, always aim for the best conferences, even if you think your work will be rejected. Even if it is rejected, you will likely get more valuable feedback than in other places.

Attend major conferences even if you don’t have a paper there. You will get a sense of what researchers are interested in, and you can connect to potential supervisors and collaborators related to your interests. Read some of their papers beforehand so that you have a good conversation starter.

Major: ICML, NIPS, COLT, AAAI, UAI, IJCAI, AAMAS, ICLR
Minor: AISTATS, ECAI, ECML, ALT
Applications: ICCV, CVPR (Computer vision), ICASSP (Speech), ICRA (Robotics), EMNLP, ACL (NLP)

Want to work on AI safety? We want to help.

We’ve helped dozens of people formulate their plans, and put them in touch with academic mentors. If you want to work on AI safety:

Get in touch