Our mistakes

These are some of the most important ways things could have gone better and mistakes we have made running 80,000 Hours, taken from our self-evaluations.

Annual review Dec 2017

People misunderstand our views on career capital

In the main career guide, we promote the idea of gaining “career capital” early in your career. This has led to some engaged users to focus on options like consulting, software engineering, and tech entrepreneurship, when actually we think these are rarely the best early career options if you’re focused on our top problems areas. Instead, it seems like most people should focus on entering a priority path directly, or perhaps go to graduate school.

We think there are several misunderstandings going on:

  1. There’s a difference between narrow and flexible career capital. Narrow career capital is useful for a small number of paths, while flexible career capital is useful in a large number. If you’re focused on our top problem areas, narrow career capital in those areas is usually more useful than flexible career capital. Consulting provides flexible career capital, which means it’s not top overall unless you’re very uncertain about what to aim for.

  2. You can get good career capital in positions with high immediate impact (especially problem-area specific career capital), including most of those we recommend.

  3. Discount rates on aligned-talent are quite high in some of the priority paths, and seem to have increased, making career capital less valuable.

However, from our career guide article, some people get the impression that they should focus on consulting and similar options early in their careers. This is because we put too much emphasis on flexibility, and not enough on building the career capital that’s needed in the most pressing problem areas.

We also enhanced this impression by listing consulting and tech entrepreneurship at the top of our ranking of careers on this page (now changed), and they still come up highly in the quiz. People also seem to think that tech entrepreneurship is a better option for direct impact than we normally do.

To address this problem, we plan to write an article in January clarifying our position, and then rewrite the main guide article later in the year. We’d also like to update the quiz, but it’s lower priority.

We’ve had similar problems in the past with people misunderstanding our views on earning to give and replaceability. To some extent we think being misunderstood is an unavoidable negative consequence of trying to spread complex ideas in a mass format – we list it in our risks section below. This risk also makes us more keen on “high-fidelity” in-person engagement and long format content, rather than sharable but simplified articles.

Not prioritising diversity highly enough

Diversity is important to 80,000 Hours because we want to be able to appeal to a wide range of people in our hiring, among users of our advice, and in our community. We want as many talented people as possible working on solving the world’s problems. A lack of diversity can easily become self-reinforcing, and if we get stuck in a narrow demographic, we’ll miss lots of great people.

Our community has a significant tilt towards white men. Our team started with only white men, and has remained even more imbalanced than our community.

We first flagged lack of team diversity as a problem in our 2014 annual review, and since then we’ve taken some steps to improve diversity, such as to:

  1. Make a greater effort to source candidates from underrepresented groups, and to use trial work to evaluate candidates, rather than interviews, which are more biased.
  2. Ask for advice from experts and community members.
  3. Add examples from underrepresented groups to our online advice.
  4. Get feedback on and reflect on ways to make our team culture more welcoming, and give each other feedback on the effect of our actions in this area.
  5. Put additional priority on writing about career areas which are over 45% female among our target age ranges, such as biomedical research, psychology research, nursing, allied health, executive search, marketing, non-profits, and policy careers.
  6. During our next round of board reform, we’ve found a highly qualified woman who we have asked to join.
  7. Do standardised performance reviews, make salaries transparent within the team, and set them using a formula to reduce bias and barriers.
  8. Have “any time” work hours and make it easy to remote work.
  9. Implement standard HR policies to protect against discrimination and harassment. We adopted CEA’s paid maternity/paternity leave policy, which is generous by US standards.

Our parent organisation, CEA, has two staff members who work on diversity and other community issues. We’ve asked for their advice, and supported their efforts to exclude bad actors, and signed up to their statement of community values.

However, in this time, we’ve made little progress on results. In 2014, the full-time core team contained 3 white men, and now we have 7. The diversity of our freelancers, however, has improved. We now have about 9 freelancers, of which about half are women, and two are from minority backgrounds.

So, we intend to make diversity a greater priority over 2018.

In particular, we intend to make hiring at least one candidate from an underrepresented group to the core team a top priority for the next year. To do this, we’ll put more effort (up to about 5-10% of resources) into improving our culture and finding candidates. We hope to make progress with less investment, but we’re willing to make a serious commitment because it could enable us to hire a much better team over the long-term, and talent is one of our main constraints.

Set an unrealistic IASPC target

As explained earlier, we set ourselves the target of tripling impact-adjusted significant plan changes (IASPC) over the year while also focusing on rated-10 plan changes. However, the IASPC metric wasn’t set up to properly capture the value of these changes, the projects we listed were more suited to rated-1 plan changes, and we didn’t properly account for a 1-3 year lead time on generating rated-10 plan changes. This meant we had to drop the target half-way through the year.

We could have anticipated some of these problems earlier if we had spent more time thinking about our plans and metrics, which would have made us more effective for several months. In particular, we could have focused earlier on specialist content that is better suited to attracting people who might make rated-10 plan changes, as opposed to improving the career guide and general interest articles.

Going forward, we’ll adjust the IASPC metric to contain a 100 and 1000 category, and we’ll think more carefully about how easy it is to get different types of plan change.

Not increasing salaries earlier

This year, we had in-depth discussions with five people about joining the team full-time. Four of them were initially concerned by our salaries, but reassured after they heard about the raise we implemented in early 2017. This suggests we might have missed out on other staff in earlier years. Given that talent is a greater bottleneck than funding, this could have been a significant cost.

It’s not obvious this was a mistake, since we weren’t aware of as many specific cases in previous years, but we were encouraged by several advisors to raise salaries, so it’s possible we could have corrected this earlier.

Looking forward, we expect there are further gains from raising salaries. After living in the Bay Area for about a year, we have a better sense of the cost of living, and our current salaries don’t easily cover a high-productivity lifestyle in the area (e.g. living close to a downtown office). Rent costs have also increased at around 10% per year, which means that comparables we’ve used in earlier years (such as GiveWell in 2012) are out of date. Our salaries are also arguably in the bottom 30% compared to other US non-profits of our scale, depending on how you make the comparison.

Rated-1 plan changes from online content not growing

Even though web traffic is up 80%, the on-going number of people reporting rated-1 and rated-0.1 plan changes from the career guide didn’t increase over the year, and the same is true of newsletter subscribers. This is because traffic to key conversion pages (e.g. the decision tool and the article about the GWWC pledge) has not increased.

This is not surprising given that we haven’t focused on driving more traffic into these pages. Instead, we’ve recently focused on driving people into the coaching applications. However, we had hoped that new traffic would spillover to a greater extent, driving extra growth from rated-1 plan changes.

Accounting behind

The CEA ops team (which we share) was short of staff over the year. This meant that our financial figures were often delayed by 3-6 months.

One problem this caused is that we only had delayed information on what our reserves were through the year, though this didn’t cause any issues this year since we maintained plenty of reserves.

Another problem was that it was hard to track spending, which meant that we didn’t catch overspending on AWS until we had incurred $5,000 of unneeded expenses (though we received a partial refund), which was about 0.7% of our budget. We also mis-paid a staff member by about the same amount due to a confusion about their salary over several months, which we decided not to recoup (in part because we wanted to raise their salary anyway).

To address this, CEA has made several operations hires over the year, increasing capacity (though illness on the team has temporarily reduced capacity again). What’s more, all our accounts have been transferred to new software (Xero), are now up-to-date within 1-2 months, and are easier to check, and we can continue to make improvements to systems. We also intend to allocate more time to checking our spending.

Not being careful enough in communication with the community

Quick comments on the EA Forum or Facebook by staff members will be taken as representing the organisation, creating problems if mistaken or misconstrued. Some comments by Ben and Rob this year ended up causing controversy. Even though the criticism was significantly based on a misunderstanding, and many people defended our comments, others didn’t see the defences, so our reputation was still likely harmed.

The most obvious solution is to raise our bar for commenting publicly, and submit this commenting to more checking, moving in the direction of Holden Karnofsky’s policies. The downside is reduced communication between us and our community, so we don’t intend to go as far as Karnofsky, but we’ve taken a step in that direction. As part of this, we updated our team communications policy and reviewed it with the team.

Poor forecasting of our coaching backlog

In November, our coaching backlog suddenly spiked to over 6 weeks, as we received a large number of applications and had reduced coaching capacity. This meant that in Sept-Oct we spent more time getting coaching applications than was needed, recent applicants had to wait a long time before starting, we’ve committed to coach people with different criteria from what we’d now use, and we’ve had to temporarily close applications.

We could have predicted this if we had made more thorough forecasts of how many hours of coaching time we’d have, and how much time it takes to coach each person.

Going forward, we’re making more conservative and detailed estimates of capacity.

Annual review Dec 2016

Challenges hiring

We had several disruptions to the team in 2015, but we put this behind us in 2016.

Over 2016, the team has worked really well together, morale has been high, and everyone has been highly productive. New hires have all said the quality of the team is a major reason for joining.

Besides hiring Peter and Brenton, who are mentioned in the section on workshops above, we hired Jesse Avshalomov. He was previously the Director of Growth and Product at Teespring, one of the most successful Y Combinator startups, where he led a team of 20, conducted hundreds of marketing & product experiments, and oversaw the growth of the company to 19 million products sold. Before that he ran SEO for the North American Apple Online Store… and did professional opera.

Our main challenge has been finding a freelance web engineer. We did a recruitment round and a trial, but didn’t end up finding someone long-term. Fortunately Peter Hartree, our former developer, is still able to give us 1-2 days per week of support. We intend to try again in 2017, taking advantage of our larger audience, advertising a higher salary, and aiming for someone full-time rather than freelance. We also learned that if we hire a part-time engineer, we should (i) make sure they spend several days talking to our existing engineer to on-board (ii) have experience in remote work and WordPress.

It also took longer to on-board Jesse than we expected, in part due to working remotely and spending 20% time on other projects. This again suggests spending longer on-boarding right at the start, and making it a priority that new staff work in the same place as everyone else for the first month. We’ll also make further efforts to avoid part-time staff in the future.

The book launch was delayed over a month, and might have had a smaller reach

The delay was due to it taking longer to on-board Jesse than we expected (as covered above), spending more time improving marketing for the workshops (which paid off, as covered above), and doing too many things at once (covered below).

Produced fewer high-value career reviews and problem profiles than planned

Roman spent more time on the plan change tracking systems than planned, while Rob spent more time on outreach. We also switched priorities several times (as covered below), which probably hurt research output.

Too many competing priorities

We make a lot of effort to create focused plans, and are much more focused than we used to be, but we still probably switched priorities too many times over 2016, and there were also times when we had too many priorities at once. For instance in spring, the research team switched from career reviews, to supporting articles for the guide, to problem profiles. In October, we were doing campus sign ups, workshops and book promotion at the same time. All this creates switching costs, is less motivating and leads to unfinished work, which contributed to many of the other problems. Some of this was made worse by being in different offices. We’ll continue to emphasise focus when creating our priorities. The plan is also for everyone full-time to be based in the Bay Area, in the same office.

Growth of high-value plan changes slower than medium-value

High-value plan changes grew only 50% compared to 360% growth in medium-value plan changes. This wasn’t a mistake but could be a worrying trend. However, the fact that it’s happening isn’t too surprising since high-value plan changes take several years, so our growth over 2016 depends on efforts made in 2014 when we were much smaller. We expect that a significant fraction (perhaps 10%) of the medium-value plan changes will eventually become high-value plan changes. At the start of 2017, we also intend to especially focus on getting high-value plan changes.

Many people didn’t get responses from [email protected] for 6 months, affecting about 70 emails.

This was due to an error with a new inbox client, which was hard to notice, though could have been found earlier with better testing.

Annual review June 2014 – April 2015

Mistakes concerning our research and ideas

We let ourselves become too closely associated with earning to give.

This became especially obvious in August 2014 when we attended Effective Altruism Global in San Francisco, and found that many of the attendees – supposedly the people who know us best – saw us primarily as the people who advocate for earning to give. We’ve always believed, however, that earning to give is just one strategy among many, and think that a minority of people should pursue it. The cost is that we’ve put off people who would have been interested in us otherwise.

It was hard to avoid being tightly associated with earning to give, because it was our most memorable idea and the press loved to focus on it. However, we think there’s a lot we could have done to make it clearer that earning to give isn’t the only thing we care about. Read more.

We presented an overly simple view of replaceability, and didn’t correct common misconceptions about it

We think many of our previous applications of the replaceability argument were correct, but we don’t think it means that you shouldn’t take jobs with direct impact (e.g. working at a nonprofit) or that it’s okay to take harmful jobs for indirect benefits.

Unfortunately some of our early content suggested this might be the case, and we didn’t vigorously correct the misconception once it got out (although we never made replaceability a significant part of our career guide). We’re concerned that we may have encouraged some people to turn down jobs at high-impact organisations when it would have been better to accept them. Read more.

Not emphasising the importance of personal fit enough

We always thought personal fit – how likely you are to excel in a job – was important, but (i) over the last few years we’ve come to appreciate that it’s more important than we originally thought (most significantly due to conversations with Holden Karnofsky) and (ii) because we didn’t talk about it very often, we may have given the impression we thought it was less important than we in fact did. We’ve now made it a major part of our framework and career profiles.

Released an interview about multi-level marketing

We asked users to send us interviews about careers they knew about. One sent us a favourable interview about multi-level marketing, which we released. We were quickly told by a reader that multi-level marketing is highly ethically dubious, and took the post down within an hour. We should have better vetted user-submitted content before release.

Operational mistakes

Allowing a coaching backlog to build up in late 2014

We allowed a large backlog of over 100 coaching applicants to build up at the end of 2014, with the result that many had to wait several months for a response. This happened because our head of coaching was repeatedly on sick leave over the last half of 2014, and I didn’t step in quickly enough to close applications. To make it right, we apologised to everyone and gave email advice to about 50 of the applicants. When we set up our new coaching process in early 2015, we closely monitored the number of applicants and response times, closing applications when our capacity became stretched.

Not improving the online guide earlier

In September 2013 we took down our old career guide, going for a year without a summary of our key advice outside of the blog. I was aware of the problems this caused – most readers don’t visit useful old posts, and it was hard to find our most up-to-date views on a topic. I could have made a minimal replacement (e.g. a page listing our key articles) back in September 2013, which would have resulted in thousands of extra views to our best old content. Instead, we focused on coaching and new research, but in retrospect I think that was lower priority.
We should also have added a newsletter pop-up earlier. We were wary of annoying readers, but it dramatically increased our conversion rate from 0.2% to over 1%. In the end, we added a more complex appeal that just slides down under the header rather than popping up, and is only shown to engaged readers, with the aim of making it less annoying. However, we could easily have added a simpler pop-up a year ago, which would have resulted in 1000-2000 extra newsletter subscribers.

Simultaneously splitting our focus between the online guide and coaching

Perhaps an underlying cause of the previous two mistakes was that we attempted to push forward with both coaching and improving the online guide at the same time, despite only having the equivalent of two full-time staff working on them. We did this despite knowing the importance of being highly focused.
With more focus, we could have had clearer and shorter development cycles, better metrics, and generally better management, which would have helped the team to be more productive.

The reason we didn’t focus more was that we were reluctant to close the coaching, even temporarily, but in hindsight this wasn’t a strong consideration compared to the benefits of focus.

Not focusing more on hiring people we’ve already worked with

It’s widely seen as best practice in startups for the first couple of team members to be people who have worked together before and can work together really closely. We were aware of this advice but pushed forward with trying to hire new people. This mostly didn’t work out, costing significant time and straining relationships.

I think it would have been better either not to hire, or to focus on doing short, intense, in-person trials, since that’s the best way to test fit quickly. Instead, we did longer but less intense trials that were often remote and spread out over the year.

See the full review of progress

Annual review April 2013 – May 2014

Mistake: Team too large and not sufficiently focused on strategic progress. If the team had been smaller, more permanent, higher quality and more focused, we probably would have had less immediate impact, in the form of changed careers, research and outreach, but would have probably had more fundamental strategic progress, such as developing product plans or prototypes, testing the impact of our programs, recruiting staff and raising funding. Ultimately, it’s strategic progress that’s important for our chances of becoming much bigger in the future. In particular, a key cause was having too many interns. Interns allow us to have more immediate impact, but take up core staff time, reducing long-run progress, especially in the face of relatively complicated team plans. We analysed the issue of how many interns to hire in our last six month review, concluding that we should aim to have fewer in the future. In hindsight, we should have been even more aggressive in reducing the number.

What we did: First, we reduced the number of interns to one working on tech (Ozzie Gooen) and one on central CEA. After Ozzie left, we switched replaced them with a part-time paid web developer. Ozzie also significantly simplified the website, making it easier to maintain, and taught us more about how to edit it ourselves (more detail in the website review). We replaced the most useful functions of non-tech interns with long-term professional freelancers, including an oDesk editor, a volunteer editor, a virtual assistant and a contract researcher. We decided to only aim to have one or two interns over the next year, and to restrict these to people we are strongly considering hiring, or who can help with our strategic priorities. Besides reducing the number of interns, we raised the bar on hiring, and decided to focus on building a team of staff who are around to stay, and can take 80,000 Hours to scale. We decided to aim to make the team plans even more focused, by working on fewer activities at once and always having a clear top priority.

See our full list of mistakes.

Six Month Review Dec 2012 – March 2013

Mistake: We ran out of capacity in the operations team around March. This resulted in delays of over a month to the arrival of two interns, since we were unable to complete their visa applications in time, and a meant the Executive Director of 80,000 Hours had to spend a significant amount of time on operations (around 50% at peak). This happened because (i) we didn’€™t make a sufficiently detailed plan for operations at the beginning of the period, so we didn’€™t recruit enough interns to meet demand (ii) we were overoptimistic about how much time operations required (iii) The Director of Operations wasn’€™t given enough authority to make the decisions himself.

What we did: We decided to create a new role – the Executive Director of CEA – which was filled by Rob Wiblin around August 2013. The role was to (i) oversee the operations team (ii) take over fundraising from Will MacAskill and (iii) act as a single point of contact for issues that affect the whole of CEA. It was given equal status to the Executive Directors of 80,000 Hours and Giving What We Can, who decided to meet weekly as a three. The aim was to increase decision making capacity covering central issues, like the office, legal risks, the relationship with other organisations and recruitment, while also freeing up Will’s time from fundraising. We also decided to: (i) Ask the Director of Operations to draw up more detailed plans (ii) Hire additional interns for the central team (iii) consider hiring a second staff member for the central team (though we ended up deciding against). By our next annual review in May 2014, we were happy with the operation of the central team and Rob Wiblin’s performance in the new role.

See our full list of mistakes.

Six Month Review June – Nov 2012

Mistake: We could have been ahead of schedule if we had focused more on testing, product research and strategy from the start, rather than working as much as we did on outreach (although outreach had substantial benefits).

What we did: We addressed this by having two major strategy reassessments – one near the end of the summer and one in November – in which we assessed our competitive niche and analysed our success to date. Going forward, we’re making sure to include more time for strategy in our plans. We’ve designed an iterative product development process where we collect feedback early and use it to constantly adjust our content.

Mistake: CEA Central Operations (shared with Giving What We Can) had success in registering CEA as a charity and dealing with all the admin required to take on staff – sharing the department saved us months of staff time. However, it also had a number of failures, which wasted management attention.

What we did: We changed the role of the Director of Operations to officially answer to the Executive Director of 80,000 Hours and removed their responsibilities to work on Giving What We Can (which was dividing attention). The Executive Directors of Giving What We Can and 80,000 Hours started to meet weekly to discuss central issues, and we reviewed our allocation of interns to the central team.

See our full list of mistakes in this review.

Read our full evaluations