New definition of a significant plan change

We’re changing how we define “significant plan changes”.

The “significant plan change” is the key metric we track month-to-month to measure our impact.

It’s related to our total impact as follows:

(Number of significant plan changes) multiplied by (the value of the average significant plan change)

We evaluate the value of a significant plan change once every year or two (see our last evaluation). This means the more difficult judgement calls concerning our impact (such as whether plan changes are really improvements, and the extent to which they would have been caused by other groups anyway) are isolated into a separate evaluation process that we perform less frequently.

Our new definition for a significant plan change is:

A sig plan change =df Someone tells us that 80,000 Hours caused them to change the career path they intend to pursue, in a way that they think increases their lifetime impact.

In practice, this means:

• They fill out one of our surveys saying “yes” to “has your engagement with 80,000 Hours caused you to significantly change your career plans?”
• They tell us what path they changed from and what path they now intend to pursue.
• They tell us what sort of engagement with us caused the change (e.g. reading an article vs. coaching vs. speaking to someone in our community).
• They believe this will result in greater impact (rather than just higher job satisfaction or other benefits).

Here are some examples of changes that would count:

• They had no idea what they wanted to do, but have now settled on a specific path.
• They were intending to go into academia, but now want to enter politics.
• They stay within academia, but change field (e.g. maths to economics).
• They still intend to pursue the same career path, but now give an extra 10% of their income. (Update 30th Oct ’15: Changed from 20% to 10%)
• They change where they intend to donate, and are giving at least 10% of their income. (Update 30th Oct: Changed from 20% to 10%)
• Edit Sept ’15: They changed what degree course they intend to study. (Initially we decided not to count this, but we decided to add it back because on reflection it does significantly effect the rest of your life).

Here’s some examples of changes that wouldn’t count:

• They undertake an internship in a different area but then go into the same field as they were planning to beforehand.
• They feel more confident in the decision they have already made.
• They consider a new option but ultimately reject it.
• They change which cause they think is most pressing, but don’t change what they’re doing within their job, which job they have or where they donate.

Why this change?

Our old definition was:

An individual has made a significant plan change if they have changed their credence in pursuing a certain career path, cause or next step by 20% or more; they attribute this change to 80,000 Hours; and there’s a plausible story about how engaging with 80,000 Hours caused this change.

The old definition has a number of problems, including:

• It is difficult to measure a 20% credence shift, and “a 20% shift” doesn’t line up well with the intuitive notion of a “plan change”.
• Technically a 20% shift included going from 80% confidence to 100% confidence in one option, but we never counted these cases.
• It confused causation with attribution.

The new definition is operational, so it much more closely tracks what we actually mean by “significant plan changes” day-to-day.

The new definition is based on people’s belief that their impact has increased. Of course, what we ultimately care about is the actual amount of additional impact which results from our advice, impact which wouldn’t otherwise have occurred (i.e. counterfactual impact). However this is difficult to evaluate, so we plan to perform separate evaluations of the counterfactual value of plan changes once every year or two. Month-to-month we’ll just track the total number of significant plan changes, which under the new definition is relatively unambiguous.

What about significant plan changes counted under the old definition? We’re not going to change the figures in our last impact evaluation, because we don’t think the new definition will alter the figures by more than about 10%. The main reason the figures would differ is that under the old definition we could technically have included people who had only increased their confidence in their top option, but in practice we never included these cases.

Our 2015 evaluation is performed with the new definition in mind.