80,000 Hours impact survey evaluation

To fill out our current impact survey please click here.

Introduction

To understand our impact and learn how to improve as an organisation, we recently ran an open survey of our users.

We released the survey on 7th January through social media, our blog, newsletter emails and some individual emails. The following post analyses the 206 responses we had received by the 7th February.

Summary

• The survey identified sixty three people who said engaging with 80,000 Hours significantly changed their career plans. These people could specify the changes and how they came about. We know from other sources of a further forty people who changed plans, bringing our total to over one hundred.
• About a third of the changes resulted only from reading online content. This is the first systematic evidence that our online content can change plans without one-on-one contact with the team.
• One-on-one coaching, discussion with people in the community and attending events were all significant in changing plans.
• We also collected evidence of impact beyond plan changes. We found for every three plan changes, there was a ratio of 1.5 people introduced to effective altruism for first time who now identify as supporters, and two people who changed their attitudes towards careers.
• Giving What We Can, Less Wrong, word of mouth and Peter Singer’s TED talk also bring people to the effective altruism community.
• Important sources of promotion for 80,000 Hours seem to be word of mouth, Less Wrong, our Oxford and Cambridge events, online search, social media and Peter Singer’s TED talk.
• The rate at which we caused people to change plans roughly doubled when we became a full-time rather than voluntary organisation. This rate has been roughly steady since.
• After seeing what kind of help people want, we decided to increase the priority put on bringing back some simple member networking tools.
• We identified several themes in the feedback, detailed later in the post.
• We received thirty very positive testimonials, which we take as a strong show of support.
• We collected data on the careers and causes that supporters of 80,000 Hours commonly pursue, detailed later in the post.

Significant plan changes

Introduction – what counts as a plan change?

The survey mainly aimed to learn more about how many significant plan changes we have caused. We define a significant plan change as follows:

An individual makes a significant plan change if they change their credence in pursuing a certain mission, cause or next step by 20% or more, they attribute this change to 80,000 Hours and it’s plausible that 80,000 Hours caused it.

For instance, if someone anticipates pursuing med school with probability 55% and law school with probability 45%, then they read an article on our blog and switch to 75% med school and 25% law school, that counts as a significant plan change.

This metric doesn’t fully capture our impact by any means, but we think it’s a useful indicator. If we’re not causing significant plan changes, we’re probably not having much impact. If we’re causing many significant plan changes, then we’re probably having significant impact.

What we learned from the survey

In the survey, we asked:

Has engaging with 80,000 Hours caused you to significantly change your career plans? Please be brutally honest.

We gave the option of answering “yes” or “no.” The answers were:

Yes: 66 people, 32%
No: 139 people, 68%

We also asked:

What did you intend to do before?

What were your intentions after?

Most people offered one to three sentences for each question. We used these answers to judge which plan changes might count as a ‘significant plan change’ according to our definition. This was just a judgement call on our part, and is liable to bias. Please regard the following as preliminary results, before we do our full impact evaluation.

We eliminated five responses, because the plan changes didn’t seem significant enough or the person hadn’t stated a clear plan change. We added four responses. This included two people who said they significantly increased the amount they intend to donate, one who said they changed the cause they support and one who became more likely to pursue earning to give. In the majority of cases, people reported pretty significant changes (e.g. they said before they were planning one career, and after there were planned another). This gave a total of sixty five survey responses that seem to meet our definition of significant plan changes. Note this doesn’t include all the significant plan changes we know about. There’s another nineteen here, and another twenty or so from our recent round of coaching.

What were these plan changes, how valuable were they and were they really due to 80,000 Hours? Because these questions are so crucial to us, in the next two months we will post a separate analysis of all the plan changes we know about. In this further analysis, we’ll summarise what the plan changes involved and what caused them by analysing twenty in depth.

Still, for this document we quickly analysed the careers the changers plan to pursue, and the extent to which they switched from conventional ethical careers to the types of careers commonly recommended by 80,000 Hours. One discovery was that twenty four – over a third – are intending to pursue earning to give. Another discovery was that the majority (63%) were pursing conventional careers before, but now identify as effective altruists and are pursuing the kinds of options we typically recommend. See more on that below.

But more important than changed plans is changed behaviour, so we also asked:

If enough time has passed, did you follow through on your new intention?

This was an open response question.

• 28 (44% of the significant plan changes) responded indicating they had already taken significant steps to follow through on the plan change. Examples included increasing donations, applying for a job and succeeding, or already starting the job.
• 17 (27%) didn’t respond to the question (presumably meaning not enough time had passed).
• 8 (13%) said that not enough time had passed.
• 8 (13%) said they have made some small steps towards following through.
• 2 (3%) suggested they ran into difficulties while making the plan change, and may not be able to follow through.

This suggests that a majority of the plan changes will result or have resulted in changed behaviour.

Evidence of further impact

Some people changed their attitudes about career choice, but haven’t significantly changed their plans yet. This may be because they haven’t faced a career decision recently, or the influence of the attitude change on their plans is unclear.

We didn’t explicitly ask about other influences on thinking, which was a mistake. Even so, forty three people (21%) who didn’t report plan changes mentioned other significant influences. This is a further two people for every three plan changers, and it’s likely to be an underestimate.

We roughly categorised these comments about changed attitudes. Here we list the attitude changes mentioned more than once:

1. Broadly persuaded by the 80,000 Hours philosophy. That is, primarily aiming to make a difference in an effective way with your career. (13)
   Some example responses: “My awareness of 80,000 Hours made me want to be more effective.” “Made me think about how having a particular job can have an impact.”, “80,000 Hours did make me look at various career, paths…in a different light.”,
2. Have considered Earning to Give, but don’t intend to pursue it. (6)
3. More open to options (beyond Earning to Give), in particular the new ones we suggest. (6)
4. More likely to donate to effective charities, like those recommended by GiveWell. (5)
5. More confident in their existing choice. (4)
6. More concerned to promote effective altruism in their career. (3)
7. More proactive in planning their career (e.g. spending more time gathering information). (2)
8. More concerned to have a social impact with their career. (2)
9. Intend to use our coaching when they make their next decision. (2)

This suggests we’ve had a significant influence beyond the people who have already changed plans.

Evidence of impact in building the effective altruism community

In the survey, we asked:

Do you consider yourself an active supporter of effective altruism?

165 people (80%) said yes.

To find out what this meant, we also asked:

Do you work for, donate to, or volunteer for any organisations working on the following issues?

Options included global health, prioritisation research, existential risks, promoting effective altruism, ending factory farming, improving decision making, working for/support-us to/volunteering for 80,000 Hours.

Twenty seven (16%) of the people who said they were supporters of effective altruism didn’t answer this question about causes. The remaining 138 (84%) of effective altruism supporters selected at least one of the above causes. See a breakdown below.

Has 80,000 Hours introduced new people to effective altruism?

We asked the active supporters of effective altruism:

How did you first start engaging with the effective altruism community?

We categorised the responses and put them in the chart below:

Where did the supporters of effective altruism in the survey first discover it?

This suggests that 80,000 Hours is introducing a significant number of people to the effective altruism community. Thirty four people now identify as active supporters of effective altruism and discovered it through 80,000 Hours. This is a ratio of about one per two plan changers. Note that we don’t expect this to capture the full extent to which we’ve introduced people to effective altruism, since we targeted the survey at people who had changed their career plans, rather than our supporters more generally.

Of the effective altruists engaged with 80,000 Hours, significant initial points of contact were:

• 80,000 Hours itself
• Giving What We Can
• LW/CFAR/MIRI (Less Wrong, Center for Applied Rationality and Machine Intelligence Research Institute), where LW was the most important.
• Word of mouth
• Peter Singer’s TED talk

We don’t expect this distribution to fully reflect the distribution of how effective altruists in general first found out about effective altruism. Rather, it’s the distribution of how effective altruists who are engaged with 80,000 Hours first found out about effective altruism. So, we should expect 80,000 Hours itself to be over-represented. We also expect Giving What We Can to be over-represented, due to the close collaboration between the two organisations.

This data also doesn’t tell us much about the process which led people to become effective altruists, though the comments gave hints. People who found out about effective altruism through Less Wrong or Peter Singer’s TED talk often said that they became fully involved by finding a link through to 80,000 Hours, Giving What We Can or GiveWell.

Has 80,000 Hours led to more people pursuing ‘effective altruist’ careers?

We wanted to understand whether we just told people about effective altruism or persuaded them of its value. We also wanted to measure whether we’ve mainly helped people who are already heavily engaged in effective altruism, or introduced new people to the community. To do this, we looked at whether we have convinced people to change from conventional careers to supporters of effective altruism pursuing careers we typically recommend.

We defined conventional careers as those not aiming at impact, and conventional ethical careers as those in the charity sector (in organisations not aiming to maximise impact), social sector or teaching. We took our typical recommendations to include earning to give, working in effective altruist organisations, and policy or academic careers focusing on impact.

Of the sixty four people who made significant plan changes, we estimate about forty (63%) identify as supporters of effective altruism, and switched from a conventional or conventional ethical career to one we typically recommend. Of the remainder, nine (14%) don’t identify as effective altruists, and fourteen (22%) were people already leading effective altruist careers but who improved their plans. This suggests that we’re also increasing the extent to which people engage with effective altruism, as well as informing people about it.

Has 80,000 Hours improved the careers of people in the effective altruism community?

The majority of plan changers seem to have switched from conventional careers to effective altruist careers. However, there were a further fourteen (22%) supporters of effective altruism that switched between paths we recommend. This suggests we’re also, though to a lesser extent, helping effective altruists to improve their career decisions.

Summary

For every 3 significant plan changes in the survey, we have also:

• Told 1.5 people about effective altruism for the first time, who have turned into active supporters (about 34 in total)
• Changed the career attitudes of over 2 people (about 46 in total).
• Caused 2 people to change from a conventional or conventional ethical career to an effective altruist career (over 43 in total).

How did the plan changes come about?

We asked the people who reported plan changes:

How did these changes come about?

Options included reading online content, one-on-one coaching, discussions with people met through 80,000 Hours, attending an event and other things.

The most common answers were as follows. Note that it was possible to select several pathways.

Percentage of plan changers who mentioned each pathway as a significant cause of their change of plan.

In addition:

• 45% had received one-on-one coaching at some point.
• 18% have worked for, volunteered for or donated to 80,000 Hours, suggesting they know us personally.
• About half the plan changers don’t know us personally and haven’t received coaching.
• 37% (22) listed reading online content as the only significant pathway to their career change.

We conclude that all of the pathways played an important part in changing plans.

We also conclude that scalable pathways, like just reading online content, can change careers. This is useful and promising, because we were concerned that one-on-one discussion was required.

Who filled out the survey?

Which causes do they support?

The 80% who said they’re active supporters of effective altruism said they had worked for, volunteered for or donated to organisations that support the following causes. Note that it was possible to list multiple causes.

Which causes do the supporters of effective altruism support? (% who said they worked for, donated to or volunteered for organisations working within each cause)

For those who also changed career plans, the proportions of causes are the same, though the percentages are generally higher because the people who changed plans listed more causes. Perhaps this reflects higher engagement.

Note that this distribution may not reflect the distribution of causes among the effective altruism community in general; rather, it reflects the distribution among those who are also engaged with 80,000 Hours.

Of the remaining 20% who didn’t identify as supporters of effective altruism, most didn’t list any of these causes, though a significant proportion said they support global health.

Which career paths are they intending to pursue?

From the descriptions of the plan changes, we were able to estimate which career 79% of respondents intend to pursue over the next couple of years. The rest didn’t provide enough information. This was a mistake on our part – we should have explicitly asked what they’re planning to do next.

14% reported being undecided or listed multiple options. The remaining 65% intend to pursue these career paths:

Which careers paths are respondents with clear intentions planning to pursue?

In addition, from the descriptions of their plans we estimated how many people plan to pursue earning to give. The following doesn’t include people who listed earning to give as one option among several.

• Of the survey respondents, about fifty (24%) are planning to pursue earning to give.
• Of the plan changers, about twenty four (37%) are planning to pursue earning to give.

For the survey respondents, the distribution is similar to the normal destinations of Oxford graduates. However, the following careers are significantly overrepresented:

1. Software engineering – possibly due to the influx of people from the Less Wrong community.
2. Working in nonprofits – possibly reflecting stronger ethical concerns among our audience.
3. Working in effective altruist organisations.
4. Entrepreneurship.

On the other hand, education (as in teaching, education policy and related occupations) is significantly underrepresented: zero in this sample, compared to 14% normally. Accountancy, consumer goods, publishing, media and culture are modestly underrepresented:

In studying, the most common subjects were economics (6), physics (2) and philosophy (2). This is very different from the normal distribution of subjects, which humanities dominate.

We’re unsure how the distribution differs from Oxford students who particularly want to have a social impact. Our impression is that the common ‘ethical’ careers are teaching, nonprofits and social work. Some people also go into policy and academia.

How was the distribution different among those that changed plans?

We were struck by the following:

1. Finance is about twice as common.
2. No-one mentioned conventional nonprofit careers, although 9% mentioned an intention to work at effective altruist organisations.
3. Academia is much less common (16% versus 3%).
4. Consulting, party politics, software engineering and business were slightly more common.
5. Law, actuary, entrepreneurship and medicine were slightly less common.

This hints at our aggregate influence. For instance, it suggests we’re causing people to take finance jobs and work at effective altruist organisations. We’re not encouraging many people to go into academia or nonprofit jobs.

How did the survey respondents discover 80,000 Hours?

Answer to: How did you first find out about 80,000 Hours? In %

For the survey respondents:

• Media coverage, excluding Peter Singer’s TED talk, hasn’t been a significant source of supporters.
• We were surprised how many came from the Oxford and Cambridge student groups and from online search. The Oxford group is likely to be overrepresented, since we sent email reminders about the survey.
• The groups which have sent us the most people are: the LW/CFAR community (Less Wrong and Center for Applied Rationality), Giving What We Can (GWWC) and Peter Singer.
• We asked people to name who introduced them through word of mouth. Only three people were mentioned more than once: Will MacAskill, by six people including five people who made plan changes;Jacob Trefethen, by three people including two plan changers and myself (two, including one plan changer).

How was the distribution different among people who made plan changes?

The key differences are:

1. Word of mouth is more important.
2. 80k:Cambridge doesn’t appear. This is probably due to 80k:Cambridge starting in October 2013, so it hasn’t had time to cause significant plan changes except through coaching, which is covered elsewhere. The group also didn’t receive email reminders.
3. Online search and 80k:Oxford are less important but still significant.

When did they discover 80,000 Hours?

We asked:

Roughly, when did you first find out about 80,000 Hours?

We categorised the responses into:

• Unanswered
• Founding – knew about us in early 2011, before we were a professional organisation.
• During the the first and second six month periods we existed as a volunteer organisation, from July 2011 to June 2012.
• During the first, second and third six month periods we have existed as a professional organisation.

In some cases, we used our knowledge of the individuals to make a better estimate. Excluding the roughly 20% who didn’t answer, the number of respondents for each category is given below.

When did the survey respondents say they discovered 80,000 Hours, excluding people who didn’t answer?

Note that the 6-12 month period was when Peter Singer’s TED talk was released, boosting the figure by about ten. The 0-6 months voluntary period was boosted by our official launch.

• We roughly doubled the number of respondents discovering 80,000 Hours each period after we started full-time.
• Over the last eighteen months, the number of respondents discovering us and changing plans has been fairly steady.
• The number of plan changes drops off in the last six months. This is probably because people take time to change career plans after discovering us, at least because most people only make career decisions every one to three years.

What help do respondents want?

We asked:

Would you like more help planning your career?

We let people give a score from 1, meaning not at all, to 7, meaning they really want more help.

The mean score was 4.5, which was close to the middle. However, the variance was large at 2.9. Fifty seven people (28%) gave an answer of 6 or more, showing that a significant fraction of our supporters really want more help with their careers.

We then asked:

What could 80,000 Hours do, if anything, to best help you have more social impact with your career?

We roughly categorised the responses, and counted how many times they were mentioned. We weighted each mention by the score given by the person for how much they wanted more help with their career. We normalized by dividing by seven. E.g. If one person said they wanted ‘X’ and gave a ‘5’ in response to how much they want help with their career, then we added a score of 5/7 to ‘X’.

The following categories were mentioned more than once. We’ve grouped them by theme and ranked by score within each theme.

The most significant request was for more research. Eleven people asked us to continue what we’re already doing. The other forty people requested a specific research topic. Unfortunately, the majority were only mentioned once, so it was difficult to find themes. But the following five topics were mentioned two or three times:

1. Provide more advice on having an impact without changing job. (We’ve just done this).
2. Provide career profiles. (We’ve begun progress towards this here).
3. Offer more advice on how to be high-achieving with a career, rather than how to pick a career. (We don’t see this as our niche. A lot of advice already exists about this).
4. Help compare research and earning to give. (We’ve recently made progress on this in a case study).
5. Provide information on the best opportunities within medicine. (We have exciting news on this front coming soon!)

Besides research, people most wanted additional coaching for themselves or some other kind of regular, personal contact.

There also seems to be significant demand for:

1. Ways to make connections with other members.
2. Ways to make connections with employers.
3. More examples and stories of high impact people.

What feedback did the respondents have?

We asked:

Is there any other feedback you’d like to give us, about any aspect of what we do?

We roughly categorised the responses, and list those that appeared more than twice below.

By far the most common response was general praise encouraging us to continue what we’re doing. Among the criticism, we perceive the following themes to be relevant to action:

1. Foster diversity to enable us to expand our appeal beyond the affluent, white and highly educated.
2. Expand by hiring more staff, going to more universities and doing more coaching. In particular, do more outreach using the content we’ve already got.
3. Restore the networking features of the site.
4. Improve our systems to respond better to coaching requests. We annoyed several people by not responding and we apologise for this.

The research received a variety of somewhat conflicting criticism. Several people asked for more concrete, actionable and personalised advice. Others suggested that our research is overconfident. Three people asked for a greater focus on providing data rather than opinion, which could help on both of these fronts.

We won’t fully respond to this feedback here. We’ll include it in our annual review. However, we’ll briefly say:

1. We’re also highly concerned about diversity, especially of perspectives and experience, and take this into account when selecting candidates to work with us.
2. We continually reflect on the balance between research and outreach, and between expansion and consolidation. In the review, we’ll explain why we’re focusing on research and consolidation.
3. The networking features of the site were costly to maintain. They also distracted focus from our key priorities, which are developing research and coaching. However, this feedback has persuaded us to restore a simpler version of these features sooner than planned.
4. We think the research is becoming more concrete, useful and action-oriented, since it’s now driven by case studies. We would also like to provide more empirical data going forward.
5. Two people requested greater transparency. We think we’ve addressed this, but the measures taken aren’t widely known. You can follow our weekly progress in depth. Like GiveWell, we’re also publishing in-depth six month reviews.

Testimonials

At the end of the survey, we asked for testimonials. We received thirty two detailed, highly positive testimonials, which we’re very grateful for. After receiving permission, we’ll add these to the website.

Survey methodology

The survey was promoted through Facebook and Twitter posts, an email to members, an email to the newsletter (and mentions on two other newsletter entries) and a blog post. We also contacted about twenty people individually asking for responses. The most successful method was emailing members, which quickly yielded about fifty responses. To further incentivise responses, we offered a $100 random prize draw in the form of Amazon vouchers or a donation to a charity of choice.

Ben performed the analysis as described. He sought feedback from others in the team and at the Centre for Effective Altruism.

Changes for next time

While analysing the results, we realised the following additional questions would be useful:

1. People mentioned being influenced but not changing their career plans. However, we didn’t explicitly ask about influence beyond changing plans, so we don’t have a good idea of the true extent of our impact. It would have been helpful to add a question like: “besides changing your career plans, in what other ways has 80,000 Hours helped with your career?”
2. In the descriptions of plan changes, most people focused on how their decisions about jobs or their donation plans have changed. But few people mentioned changing the cause they support. This may be because we didn’t cause anyone to change their cause. But it may also be because people don’t normally include their cause in their career plan. It may be useful to explicitly ask how views about causes have changed.
3. It’s useful to know what careers our supporters plan to pursue, but it was difficult to estimate from the information about plan changes. Next time, it might be worth asking for best guesses about which careers people plan to pursue.
4. We’re particularly interested in finding the most dramatic career changes, since we suspect these will be considerably more high impact than average. In the survey, it was difficult to spot these cases. It’s worth brainstorming questions that could flag them.

In the future, we plan to collect evidence of plan changes on an on-going basis, rather than in a one month survey. We’ll do this by having several requests for information on the website. We’ll produce an annual report on how many plan changes we have caused, but may not do a full analysis of our supporters for more than year.

Conclusion

The survey gave us a far better picture of our impact than we’ve had before. Overall, we’re encouraged that our research is changing careers, we’re building the effective altruism community and we have a strong base of supporters.