Our Progress
NOTE: This piece is now out of date. More current information on our plans and impact can be found on our Evaluations page.
This is the first post (of a series of six) on our six month evaluation
The purpose of the evaluation is to explain to key stakeholders our progress, plans for the future and how we think we’re performing as an organisation. The main model for our evaluation process is GiveWell.
In this report, we outline how we see our main achievements and mistakes over the last six months.
Summary
- Our key priority was further developing our business model and content strategy, and we’re pleased with our success in this area. We made two rounds of improvements, culminating with adopting the case study model.
Our next major priority was further building 80,000 Hours and the Centre for Effective Altruism as robust, effective organisations. We made mistakes in this area, but overall I think we made strong progress. We successfully coped with a doubling of the number of people working at the Centre for Effective Altruism.
While prioritising developing our business model and organisation building, we’ve also performed strongly in outreach. Our web traffic increased by 47% and we recruited about 400 new members, which was well ahead of expectations.
We made many mistakes this period, but I’??d classify most as minor mistakes. The worst mistake was that the operations team ran out of capacity in March, slowing down our overall progress.
Overall, I think we’re in a strong position to carry out our plans over the next six months.
Our Achievements
Learning about and improving our business model
I continue to see our main priority as developing a proven, scalable business model that cost-effectively creates impact through improving careers. This period was a success for this aim, and resulted in two significant improvements to our content strategy, with the result that we will take research forward with the case study model??.
In November 2012
We had a long list of content ideas (career profiles, cause reviews, reviews of high impact ‘??activities’?? like Earning to Give, comparisons of activities, decision processes, material on job satisfaction, etc.). We didn’??t have a systematic advising process. Our plan was to (i) work on clarifying which career problems we should focus on (ii) develop demo versions for each of the content types (iii) test out these demos in one-on-ones and on the blog.
In March 2013
The new plan: After preparing demo content for each type, further thinking about how to organise our content, further thinking about how to measure our impact, interviewing about 40 people about their career problems, and carrying out more surveying of other providers of career advice, we decided to create the following content: (i) rankings of 10 careers in terms of impact, based on a framework (ii) a similar ranking of causes (iii) a step-by-step career planning process based on a standardised form of career plan.
We decided to deliver this content through (1) a series of three intro lectures to introduce the rankings and planning processes (2) blog sequences (3) a one-on-one coaching process.
Why?: We think this plan was an improvement, since (a) it was much more focused on answering ‘??in which career can you have most impact?’?? which seems to be the issue we should focus on in order to have the most impact (b) it provided a list of concrete jobs to take, which was one of the most widely desired content types we uncovered (c) it provided an individualised process for narrowing down these jobs, which was also widely desired (d) it has a strong structure that people could follow on the blog (e) we suspect the idea of ‘??career rankings’?? could become widely shared.
We chose these delivery methods because (a) the videos and blog sequences are highly scalable, and the one-on-one coaching is somewhat scalable because we could make it more and more systematic over time (b) after testing out the coaching, we realised some standard intro content would be useful, and we suspected videos would be particularly popular (c) we also needed a general intro video to explain 80,000 Hours (d) blog sequences are a relatively quick way to communicate research and can build up a following over time (e) we included the coaching because there is strong demand for one-on-one help and it allows us to connect with very high impact people.
In late May 2013
What we learned: We decided to discontinue the rankings, because after doing more coaching and attempting to work on the rankings (i) we realised the research might not be decision relevant (ii) the research would likely be too superficial to appear credible. Similarly, we decided to deprioritise developing the planning process since it seemed we didn’??t know enough about successfully choosing careers for further work to be justified. We also decided that we need to spend more time understanding precisely what career problems are the most pressing for people who care about making a difference in an effective way. Although we had identified some broad areas, we were still highly uncertain about what research would have the most leverage.
The new plan: Instead, we decided to: (i) write up our existing thoughts on frameworks for comparing careers and career planning for our findings page (ii) carry out further research using the case study model. In the case study model, we speak to 1-2 coachees each week, spend ~4 days working on their career decision, write the study up on the blog, and over time spot general patterns that we write up for the findings page.
Why?: We’??re pursuing the write up because we’??re fairly confident the frameworks we developed are useful for comparing and planning careers, having tested them with lots of people one-on-one, and they’??ll form a good starting position for our research that we can develop through case studies. The case study model addresses the problems above because the extra time spent on each person means we can do our best to ensure our research is relevant to their decision and carry out the research in enough depth to be credible. Over time we end up with a sampling of the career problems of our target audience, so our research time is spent in proportion to how widely held each problem is.
The model offers some additional advantages: (i) it will be easier to track our impact (ii) we suspect the case studies will be perceived as highly valuable, increasing our reach with the most able people (iii) we will be more helpful to the most high potential people who care about making a difference in an effective way (iv) having lots of stories of high impact people could be inspiring and engaging. Another reason we adopted this model is that we realised we didn’??t need to do as much work to gain broad appeal. Publicity by Peter Singer, the media and the general growth of effective altruism is bringing in a significant number of people who already want to engage deeply with our ideas.
The main disadvantage is that we wi??ll initially speak to fewer people one-on-one and produce less widely applicable written content. This could decrease our short-term impact and web traffic, in return for better content in the long-term. We can slightly offset this disadvantage by preferring to work on questions that are more generalizable.
What else did we learn?
- We gained further evidence that there is enough demand for our programs. Despite minimal outreach effort, we were able to increase web traffic 40%, gain major press coverage and gain 400 members.
Nick Beckstead performed a trustee evaluation of our impact to date. Our key conclusions were (a) our best guess is that our current programs already justify our costs on the margin solely due to the additional individual future donations they raise for effective charities (though with large uncertainty) (b) we are part of the way towards having a model that creates impact by changing careers, but we need more evidence concerning how we change careers. This suggests we should focus future evaluation efforts on working out whether we have programs that actually improve careers.
Further organisation building of 80,000 Hours and the Centre for Effective Altruism
We strengthened the team since (i) Jess and Roman took on management roles, freeing up my time (ii) we attracted a very strong pool of interns in our recruitment process, often very dedicated top students (iii) we improved our staff training (iv) Niel successfully ran 80,000 Hours while I took time off, increasing the robustness of the organisation (v) we continued to learn about management and leadership.
Made progress on improving central CEA and freeing up my time from operations by making plans for Niel to become the ED of CEA in September, improving planning in operations and improving reporting in operations.
Coped well with growth. Despite problems applying for visas for American volunteers, operations successfully dealt with CEA having twice as many staff.
Increased transparency by (i) starting weekly strategy lunches (ii) creating the 80k_updates mail list to expand the weekly impact report beyond the team (email [email protected] if you would like to be added) (iii) making more effort to explain strategy to the team (iv) starting more explanation of our plans on the blog. More efforts will be implemented after the review.
Invested in content
Created the three intro videos filling a need for engaging, summarised intro content.
Developed and tested some key frameworks: how to compare careers and causes, and how to structure a career plan. We think these are immediately useful and will provide a structure for future content.
Wrote over 60 blog posts and did research on the causes of job satisfaction, how biases effect career decisions, intelligence/personality/experience as predictors of success, models for expected earnings, a process for picking a thesis, how to estimate, a process for picking a major, the potential to improve research careers, GiveWell as a career option and the best productivity boosting techniques.
Made significant improvements to the online profiles which significantly increased completion rates
Outreach Impact
As with the last period, we’??re more focused on developing our business model than impact; nevertheless, it was a very successful period, which exceeded our expectations.
- Unique visits to the site increased 47% on the last 6 months, despite low Christmas traffic. Around half of this was due to three rounds of press coverage, including TED, The Washington Post, Quartz, The Daily Mail and NPR. I don’??t expect to continue growing at this rate unless we attract further coverage or ramp up our investment in outreach.
This led to 400 new members. That’??s over twice as many as the previous period, and beat our expectations. We’??re on course to reach 1000 members by the end of August, a stretch goal we only gave 10% odds last summer. The increase was faster than the increase in web traffic, because we improved conversion in November. In the future, I expect member growth to be in line with traffic growth.
We gave over 40 people one-on-one coaching, out of about 150 requests.
Of new members, 53 reported anticipating a “significant” or ‘??complete’? career change. It seems that we are causing some significant career changes, particularly among one-on-one coachees, but we do not yet have good enough evidence the typical scale and type of changes.
We expect this to result in future, time-discounted, counterfactually adjusted donations to top charities on the order of a million dollars. For more on this estimate and our cost-effective, please see our Evaluation of 80,000 Hours as a Project.
We continue to have success in assisting with starting new projects. Our first spin-off, Effective Animal Activism, hired a full-time Executive Director, and has now attracted funding pledges of $60,000. We’??re advising Alex Hoye about assisting in setting up an incubator for effective altruist for-profit start-ups.
Our community is also starting to help people to find jobs and we now have engaged members in Google, McKinsey, Jane St, Rothschild and other high profile firms.
We’??re starting to have success as a headhunter. Our blog posts about GiveWell resulted in a new trial hire and a temporary hire at Giving What We Can discovered the job through our mailing list.
Mistakes and ways things could have gone better
- We think largest mistake was that we ran out of capacity in the operations team around March. This resulted in delays of over a month to the arrival of two interns, since we were unable to complete their visa applications in time, and a meant the 80,000 Hours ED had to spend a significant amount of time on operations (around 50% at peak). This happened because (i) we didn’??t make a sufficiently detailed plan for operations at the beginning of the period, so we didn’??t recruit enough interns to meet demand (ii) we were overoptimistic about how much time operations required (iii) Tom wasn’??t given enough authority to make the decisions himself. We think we’??ve fixed this by: (a) Making Niel the ED of CEA from September. In this role, he will spend about 50% of his time running central CEA. This should increase the decision making capacity covering difficult central issues, like the office, legal risks, the relationship with other organisations and recruitment. The role will also create a single point of contact for these issues, and the other one-off issues that arise for CEA. (b) Asking Tom to draw up more detailed plans (c) Hiring more interns for operations. In addition, we may need to hire a second full-time staff member for operations, but that decision is still under review.
The following are ways we could have been slightly more effective over the period. They are not in order.
- We invested too much time in the intro videos. This was partly due to under-estimating how long they would take (particularly recording and editing the audio), and we’??ve updated our estimates. It was also due to not having a clear enough brief of the purpose of the videos. We should have started the process by ensuring that everyone involved really clearly understood their purpose. Finally, we think we made the videos too long. Although a 3 minute intro video is very useful, it’??s not clear that the second and third videos were worth the additional time compared to putting the same ideas in a blog post. Although video content may be more popular, for this type of content it doesn’??t seem to justify spending more than 10x as much time to create.
Over-frugality. We think we could have been in a better position if we had been prepared to invest more in website design and office space. In the past, we have done website design in house for very little money. This cost significant time that could have been used on research and resulted in lower standards. We think it would have been worth spending several thousand pounds to receive professional web design. We partly avoided this because we were unsure of how much money we could fundraise, but it seems like we were slow to update on new data. I think a similar thing is true for office space.
At the start of the period, I think I was spending too much of my time on management, and not enough on taking the lead with our content. We’??ve made progress on fixing this by training Jess and Roman as managers, and slightly reducing how many interns and volunteers we have. I still have more to learn about how to manage managers.
Not being focused enough with content. Although we’??ve been becoming more focused over time, I have sometimes approved content that seemed valuable, despite not being core to our plans. Although the decisions in isolation seemed good, in aggregate, they contribute to a lack of structure on the blog and delayed our core mission. In addition, it’??s easy to make the planning fallacy and underestimate how much time they take from the mission. In the future, additional research will all be tied to case studies, which will force it to be relevant to choosing between careers.
Didn’??t have enough narrative and structure on the blog. This was because I wasn’??t sure how to structure our content or what was best to work on, and also because I wanted to be flexible with what content the interns worked on in order to attract talent. The case study model will allow us to have a much clearer narrative in the future, and we’??ve improved our ability to attract interns.
Spent slightly too much time on outreach rather than accelerating the development of our content and model. In particular, I think we spoke to slightly too many people one-on-one rather than focusing on provably changing the careers of the most promising people, and Niel spent too much of his energy on events in Oxford. We’??re addressing this by adopting the case study model and making the university chapters more autonomous.
Not enough transparency about strategy. Several team members and outside supporters felt they didn’??t understand our plans during the period. We worked on this by taking the steps listed in ‘??organisation building’?? above.
Not offering longer internships. We have in the past avoided offering internships longer than 3-4 months out of caution. In practice, we have almost always been happy for interns to stay longer than 3-4 months, which we knew by autumn 2012. By not promoting long stays and recruiting for them, I think we’??ve had higher intern turn-over than necessary, which is a significant cost. We’??re addressing this by advertising 8-10 month internships in the next recruitment process.
Not fundraising more from outside the effective giving community. It would have significantly reduced the opportunity cost of our funds and increased our financial robustness. We think it would have taken somewhat more time, but not as much as our original estimates.
Significantly underestimated how long it would take to get clearance to move into the FHI from the university. This cost us in having a significantly worse office for seven months. We wi??ll be more conservative about estimates involving university bureaucracy in the future.
Financial reporting not clear enough. Will wasn’??t clear what his fundraising goals should be far enough in advance. Our reports were difficult to understand. We’??re working on this by making Tom the finance officer and giving more feedback on the reports.
Was it a mistake not to start the case study model earlier?
Nick suggested an approach similar to the case study model in summer 2012. Now, almost a year later, we’??ve adopted it. Was it a mistake not to adopt it sooner?
I don’??t think it was a mistake not to immediately start the case study model once we started to prioritise content and strategy development in November 2012. At that point, we thought it was likely to be more effective to create less in-depth content that was more widely applicable, so it was worth experimenting further with this approach. Moreover, there were other strategic issues that I wanted to work on at the time, like understanding conventional careers advising. And finally, we didn’??t seem to have anyone on the team with the time and inclination to do case studies.
The mistakes I think we did make were (i) spending too much time on outreach rather than content and strategy development and (ii) not always being focused enough with content. These slowed down our arrival at the case study model.
How did we Perform Relative to our Stated Goals?
Overall, I think we performed well relative to our stated goals in the last review and to our updated goals set in March. Our main failures were modest delays due to the planning fallacy. The additional differences are due to the mistakes listed above.
In November 2012
Initially our main priorities were carrying out (i) further research into which customer problems we should address (ii) the plans to develop a package of demo content based on our best guess content (iii) the annual impact survey (iv) the recruitment process.
Progress up to March 2013
We were largely successful in these aims in the short term. The recruitment process was finished late by about a month, which was due to (a) waiting two weeks on a third party (b) underestimating the time required. (i)-(iii) precipitated a shift in strategy in early March, which meant that we didn’??t continue with the rest of the goals in the plan.
The strategic shift led to a new set of goals for the end of May, which most importantly included (i) creating new impact tracking plans (ii) developing a series of three intro lectures as our main intro content on the site and (iii) creating a first version of a systematic one-on-one advising process. In operations, the key goals switched to finding housing and doing visa applications for the influx of staff starting in June.
Progress up to June 2013
We were successful in (i) and (ii), except the site was only fully updated in early June. (iii) was about 60% completed. We also fell behind on several more minor goals. This was because creating the intro videos took more than 4 person-weeks longer than planned. In particular, we underestimated how long the audio and editing would take. In addition, Niel had to spend much longer on operations than planned, due to the capacity crunch noted in ‘??Mistakes’.??