Evaluation of 80,000 Hours as Project

By · Published August 31st, 2013 ·

NOTE: This piece is now out of date. More current information on our plans and impact can be found on our Evaluations page.

This is the third post (of a series of six) on our six month evaluation

In this report, we answer a set of critical questions for 80,000 Hours stakeholders. These questions are inspired by GiveWell’s self-evaluation process, but also include further questions that we think are important for 80,000 Hours, including several we have been asked by our stakeholders.

Summary

  • We believe it’s likely but highly uncertain that our current business model and content is cost-effective relative to GiveWell’s top recommendations, and we continue to rapidly improve our business model.

  • We continue to learn about how to function effectively as an organisation (as evident in Our Progress). One significant challenge ahead is learning how to effectively run a larger, three tier organisation.

  • We continue to rely heavily on our key staff, though we’re making progress towards reducing this reliance.

  • We’re not financially weak, but there is significant room to become more secure.

  • Our team is highly able and reports high levels of satisfaction, and we’re attracting very talented applicants, especially to our internships. Nevertheless, we are being somewhat slowed down by not being able to find enough talented staff.

How well is our current business model working?

This is evaluated in Nick Beckstead’s separate performance evaluation of 80,000 Hours. I think the main conclusions are:

  • We’re successful in our aims to engage people in our programs.

  • My best guess is that we’re yielding a good return on our costs at the margin through raising donations to cost-effective charities (which we do via promoting Earning to Give). Our marginal cost per member seems to be under a thousand dollars. It seems likely (but with high uncertainty) that the counterfactually-adjusted, future, time-discounted donations raised per new member is well above this figure.

  • There is good evidence that people gain new information from our career coaching and find it useful.

  • We only have patchy, anecdotal evidence about how we change career plans, which may demonstrate proof of concept but is not strong enough to conclude our model is effectively changing careers. We need to focus on gathering more data about this.

  • We also need to develop a process for evaluating the quality of our recommendations.

We made two rounds of improvements to our strategy, as listed in Our Progress.

How successful are we in improving our business model?

I think the two significant updates we’ve made to our content strategy (as explained in Our Progress) in the period show that we remain very flexible in response to the evidence and new ideas. We believe these changes were improvements due to the reasons listed.

Moreover, we expect them to continue to improve because we think we have a good planning process. First, we prioritise tracking our performance at every level. Second, we have a variety of ways to keep the team in sync about our performance, including short daily meetings, regular open strategy discussions, weekly impact reports and team meetings to discuss other issues. Third, we cultivate a candid culture by encouraging feedback. Fourth, we build the team with people who have the insight and inclination to help us improve. Fifth, we have a large number of advisors who we respect, and we seek out their feedback on our plans and performance (including the trustees of CEA and the 80,000 Hours Advisory Board, who formally review our six month reviews). As an example, in this period Nick Beckstead performed an evaluation of our impact to date, in which he made several recommendations that were incorporated into our team plans.

How can we improve?

The main weakness seems to have been not having enough experienced, full-time management capacity devoted to improving the business model. We’re dealing with this as quickly as we can by (i) hiring new staff (ii) training existing staff into manager positions. We would also like to seek more experienced people to act as mentors or to join our advisory board.

We may also be able to improve by sharing our plans even more widely in order to get more feedback and encourage an even greater culture of transparency. We plan to start doing this by releasing this review to the blog and improving our tracking of key metrics.

How good are our recommendations about which career opportunities have the most impact?



We’ve decided to delay this evaluation until we have a larger body of research. I plan to include it in the next review.

Do we justify our costs?

I think we can investigate a lower bound by estimating the donations we raise for the best charities. Our current best expected value estimate, primarily based on a trustee evaluation, is that the 400 members we recruited over this period will result in future, time-discounted, counterfactually adjusted donations to top charities of $4.4mn (range $0.28mn – $68mn). This would place our cost-effectiveness ratio on the basis of donations on the order of 40x GiveWell’s top recommendations (range 2x to 570x).

Note that this estimate has not been reverted to the prior, so if you’re sceptical that the kind of thing 80,000 Hours does will be high impact, then you should use a significantly lower estimate for practical purposes. Note, however, that you would need to revert by two orders of magnitude for us to become less cost-effective than GiveWell’s top recommendations. My impression is that this is pessimistic, since I think that it’s relatively easy to raise more than 1x your costs through fundraising, but there is considerable room for debate and I do not rest my case for our effectiveness of this figure.

Rather, it would be highly unfair to focus on this figure for evaluating our effectiveness, since we think the main benefits of our current activities are: (i) gaining valuable information (ii) investing in future outreach.

To expand on (i), I think it’s likely there exists an extremely high impact model in our area, so exploring this space has very high value. The key question for evaluating our effectiveness, therefore, is “are we effectively exploring this space?” I think we are currently making good progress in exploring this space, as shown by the rapid improvements to our business model listed in Our Progress. I expect this to continue so long as we remain committed to monitoring our impact and open to the evidence.

Moreover, we have done this with relatively low costs. We have operated with a team of about 7.5 full-time workers for under £100,000 annually. Our team are usually highly motivated, high performing people in their mid-twenties, so I think this represents exceptional value. Our full-time staff receive below market rate wages even compared to non-profits, and could have earned several times as much in the corporate sector. Moreover, each has thought carefully, often for months, about whether 80,000 Hours represents the career opportunity with the highest expected impact for them.

This estimate also doesn’t include activities like creating and spinning off Effective Animal Activism.

How well is 80,000 Hours run?

Can 80,000 Hours be continued without relying on its head staff?

Where we stood in November 2012

I was personally managing the entire team and we relied almost entirely on Will for fundraising.

Progress since then

We developed a clearer business model and became more solidified in our plans, which would make it easier to replace me. We increased our full-time management capacity to two people after Niel was hired in January 2012. We made progress in developing new leaders for the organisation.

On the recommendation of the last advisory board, we continue to improve the ways of transferring knowledge within the team, again reducing the reliance on myself.

We plan to hand fundraising over to Niel in September and a trainee intern, with support from me. This will reduce our reliance on Will for fundraising.

What we can do to improve

Continue to develop our model, and to train and attract new talented leaders.

Is our team effective, motivated and well managed?

Where we stood in November 2012

  • Team satisfaction was high. The main dissatisfactions concerned (i) the poor office (ii) not enough training (iii) atmosphere not professional enough.

  • The team was highly able. Productivity was high but I thought we could develop more motivation, excitement and focus among the team with a better plan.

  • After a round of improvements in the previous period, we had good processes for line managing and giving feedback.

Progress since November 2012

  • Team satisfaction remains high and we’ve taken steps to address the dissatisfactions (see Our Progress). We also received feedback asking for greater transparency concerning strategy, which has been acted on.

  • The team remains highly able, especially at the top end, though average ability might have slightly reduced as we’ve grown. The case study model is easier to communicate and seems to have increased motivation.

  • I’ve continued to learn about how to be a good manager based on feedback from my team, and learning about specific skills (e.g. coaching).

What we can do to improve

The main area to focus on going forward is learning to manage a three tier team. Managing managers involves different skills to managing people doing direct work. We also need to work more on maintaining a strong organisational culture and good communication of our vision.

In particular, we need to focus on developing an effective role for the shared operations team. Niel will be addressing this as ED of CEA from September.

Can we attract enough talent to meet our growth plans?

The recruitment process in this period revealed that we couldn’t find anyone willing to work for us who we were comfortable with hiring immediately. We’ve also moved Niel and Rob into central CEA, losing two potential staff members for core 80,000 Hours work. This raises the worry that we’re being slowed down by a lack of talented staff.

We have space for about two staff members to join the team almost immediately – a replacement for Jess or Roman, and another person. Unless the next recruitment round is more successful than the last, it’s unlikely we can fill both of these spaces for at least six months (the time to hire new people then trial and train them as interns). Therefore, lack of talent seems to be delaying our team growth by ~6 months.

Is gender balance a problem?

Members of 80,000 Hours are only 35% women, and the proportion within the team has mostly been even lower than this. I think this is a problem because it suggests that we’re not appealing to women, the team is less diverse, and it’s difficult to reverse (since a male heavy team seems to attract more men).

I think the reason for the imbalance is at least in part the feedback loop that a male heavy community tends to appeal to other males. This means it can be altered by improving how the community appears by encouraging women to take more public facing roles. In the last six months, new members have been 38% women, a slight shift in the right direction, but not nearly good enough.

What can we do to improve?

Continue to seek out women leaders. Write more content about careers that have a higher proportion of women, rather than our historical focus on earning to give via tech and finance.

Is our use of interns effective?

Should we have hired more full-time staff rather than interns?

In terms of compensation and operations time, interns cost less than 50% of full-time staff. In most cases, they seem just as able as the people we could have otherwise hired at the margin. In fact, in our last recruitment round, our top intern applicant ranked higher than the top person who was only willing to become staff. This 50% cost saving seems worth incurring the training time loss from higher turn-over.

Besides immediate costs, there are two other considerations in favor of maintaining a higher ratio of interns to staff relative to what’s immediately operationally efficient. First, it has been difficult to find people we’re confident in hiring immediately. Rather than rush into hiring, which can be difficult to reverse, I prefer to get by with interns and then hire later as we find more people.

Second, our training of interns creates wider benefits for us and supports our ultimate cause of promoting effective altruism. Having lots of good people in Oxford is turning us into a hub for effective altruism. Our talent pool has attracted the interest of two potential corporate sponsors. Many former interns continue to promote or fund 80,000 Hours, run spin-offs and pursue high impact careers in part aided by their time with us.

Overall, I think it was the right decision to have a particularly high ratio of interns to staff so far, and expect the ratio to remain higher than many comparable organisations in the future. The main mistake was in execution: we could have reduced the training loss costs of interns by encouraging more people to take 8-10 month placements.

Should we have kept the team smaller?

We have, however, had a slight problem with our staff spending too much time on management and not enough time on strategy and content, which seems to have slightly slowed down our long-term impact in favour of more immediate impact. I don’t think this problem was caused by hiring interns instead of staff, rather it was caused by letting the overall team grow slightly too large.

I’m monitoring this issue more closely in the future, and already slightly reduced the size of our team this summer after becoming aware of this mistake.

Conclusion

I think our heavy use of interns relative to staff was the right decision, but in the future we should (i) attempt to hire more long term interns (ii) control team growth so that we don’t overburden the long-term staff with management duties. In the future, I expect the ratio of interns to staff will decrease from 2-3:1 down to 1-2:1, as we find more people we are comfortable with hiring.

Are we operationally efficient?

We’ve built a team of people who are smart, motivated and believe in the project, at the cost of having less experience. I think this was the right decision, but it means we need to learn rapidly as we go about how to build an organisation. As the mistakes section shows, this has led to some problems, though I don’t think they have been serious.

We’re starting to address this weakness partly by gaining experience, improving monitoring, and by starting to hire more experienced people.

Do we live up to our team values?

I think the main weakness is that our culture doesn’t always appear professional enough. We’ve given a bad impression to several interns and third parties, which has made it harder to retain staff and deal with these parties.

One cause has been having a bad office. This happened due to delays far beyond expectations with the FHI office. Another cause has been having a large number of interns, who tend to feel less like professional staff and turn over quickly. I think the decision to hire these interns was not a mistake at the time – it allowed us to expand much faster given our budget and there were not enough people we were willing to hire full-time.

Going forward, we plan to (i) move into a better office as soon as possible and emphasise developing a good office culture (ii) hire longer term interns (iii) keep increasing the number of full-time staff, especially in operations (iv) work more on fostering the value of professionalism in the team.

We also had feedback on not enough transparency, which we’ve addressed (as explained in Our Progress).

Are we financially robust?

What’s the situation now?

Fundraising has generally gone well and we continue to build up a strong base of donors. Over the period, we have generally been able to cover our budget more than four months in advance. We’ve also started to build up some partners who have offered to help us in emergencies and we’ve developed better tracking of our financial situation.

How can we improve?

Besides continuing to build our donor base, we can increase security further by building more reserves. Currently, our reserves are low at only two months, in part due to having a lumpy annual income stream and in part due to not enough investment in fundraising. Our target is to reach six months (see our upcoming finance report). We believe we can make this target over the next budget period by investing more time in fundraising. In the short-term, we will seek to move some donations forward.

We believe we can further improve the monitoring of our financial situation by producing monthly income reports and improving the usability of our budgets. Creating the ED of CEA position should also help to increase the coordination and management of the people involved in finance.

After this, we will pursue greater diversity of funding sources. First, we will increase our fundraising from grants and foundations, then we will start to re-investigate various forms of corporate sponsorship. Second, we will consider various ways to charge for careers advice.