In complex jobs, the best 10% of workers produce 8 time as much as the bottom 10%.
Imagine you’re offered the top job at SCI – one of the world’s most cost-effective charities – should you take it?
SCI’s historical annual budget has been just under $10m (1). With $10m, SCI is able to provide 20 million children with medicine that prevents neglected tropical diseases each year (2). We’d expect this to lead to 250,000 years of extra healthy life per year (3), as well as to yield long-term education and economic benefits to the millions of people in the affected area (4).
A 10% improvement to SCI’s efficiency means treating an extra 2 million children and creating an extra 25,000 years of healthy life per year. That’s roughly equivalent to saving 800 lives per year (5). So, a small improvement to SCI’s efficiency is a big deal. The same would go for someone who could fundraise an extra 10% for SCI.
Whether you could increase SCI’s impact by 10% is a tough question, and given the organisation’s extreme efficiency, we have reason to be sceptical. But one issue that bears on the question is the following: how much do different employees typically differ in output?
There have been many studies looking at this very question, across a wide range of jobs, which are summarised in a meta-study by Hunter, Schmidt and Judiesch (6). Output is measured in a variety of ways. For salespeople, it’s the dollar value of what they sell. For doctors, it could be the number of patients seen and treated. Other studies have been done with standardised tests, supervisor ratings and many other metrics (7). I should flag that it’s not clear how well these metrics correlate with the real value produced by jobs, but I’ll run with them for now.
What they found is that in low complexity jobs, workers’ outputs do not vary much, and the best worker is usually not much better than the average worker. As the jobs become more complex however, there’s more and more variation, and the difference between the best worker and the average grows. For example, in low-complexity jobs the top 10% of workers produce 25% more than the average, and 75% more than the bottom 10%. For high-complexity jobs, such as professional and sales jobs, the difference is much larger. The top 10% of workers produce 80% more than the average, and 700% more than the bottom 10% (8).
Taking these findings literally suggests that the bottom 3% of workers in professional jobs have negative output. Hunter, Schmidt and Judiesch believe this is unlikely, and interpret it to mean that the distribution is not a perfect bell-curve, but instead stops around zero output (it’s probably log-normal instead). I agree that it’s unlikely that someone’s direct output could be negative for long before they were fired. Imagine if a doctor killed more patients than he treated. But if we think about all the indirect effects someone can have on an organisation, like decreasing team morale and consuming lots of supervisor time, then it’s not implausible that some people have an overall negative contribution.
The methods used to measure output often don’t take these indirect effects into account. They tend to focus on what each employee did directly, e.g. the number of insurance contracts sold or the number of patients treated. This would suggest the studies significantly understate the differences in output that really matter.
The crudeness of the metrics will be another reason why the figures might tend to underestimate the true variability in output. For instance, ‘the number of patients treated’ gives some indication of the output of different doctors, but some treatments are better than others. Curing 100 patients of sore throats is not as good as curing 100 patients of cancer. So, there another important type of variation that’s being missed.
So, among charity managers and fundraisers, it would be normal for a good one to be about twice as good as an average one, and many, many times better than a bad one. Being 10% better than your replacement at the top of a charity, therefore, could be quite achievable. And very high impact.
You might also be interested in:
References
(1) The total budget from 2002-2010 was $68m. http://givewell.org/international/top-charities/schistosomiasis-control-initiative#Overallspendingbreakdown
(2) The annual cost of treatment is about 50p/person http://www3.imperial.ac.uk/schisto/whatwedo
(3) Using $40/DALY http://blog.givewell.org/2011/09/29/errors-in-dcp2-cost-effectiveness-estimate-for-deworming/
(4) http://givewell.org/international/technical/programs/deworming#Developmental
(5) Using the standard conversion: preventing 30 DALY = saving one life
(6) Hunter, J. E., Schmidt, F. L., Judiesch, M. K., (1990) “Individual Differences in Output Variability as a Function of Job Complexity”, Journal of Applied Psychology http://psycnet.apa.org/index.cfm?fa=fulltext.journal&jcode=apl&vol=75&issue=1&page=28&format=HTML
(7) Note that the study does correct for the fact that we can’t perfectly measure output, because we can only take a limited sample of what the worker does
(8) More accurately, output follows a normal distribution (though sometimes with positive skew). For low-complexity jobs, the standard deviation of the distribution is 15%. For high-complexity jobs it raises to 45%. The most variation is found in life insurance sales, where output has a standard deviation of 110%!
Comments
So what we are really saying here is that assuming that if the interviewer picks you over the other guy, you are probably 10% better than him rather than 10% worse and that research is our justification that the gap is probably going to be around 10%.
okay - here’s is a premise: A talent market for High Impact. You can get better paid doing the same thing for private companies than you can at charities - just as Google get better talent than they pay for because they are an awesome place to work charities get better talent than they pay for because working for a charity has value to an employee in itself. just like working for Google. (lets ignore any long term effects working in the charity industry has on your ‘talents’)
so it stands to reason that working for a high-impact charity also has intrinsic value to an employee - be it logical or prestige based. The more motivation people have to apply to a firm the more people will apply and the lower the talent gap between candidates - so I would argue that as High Impact becomes sexier ~HIEL GIVEWELL!!!~ so will the potential to make a difference in high impact charities. is this diminishing marginal returns?
So i’m not pursuaded by this argument - I would be inclined to class working at well-known charities similarly to research. In a charity there are rain makers and rain collectors.
Funelling talent that does want to work at charities towards high impact charities seems as meritable a cause as GiveWell’s but it seems to me that if you want to be high impact by working for a charity you need to go to a high impact charity where the guy you replace wasn’t motivated by it being a high impact charity.
Hey - I would love to read an argument on High Impact Career Arbitrage - i.e. looking for high impact careers that are poorly staffed for some reason and then taking advantage of that… starting to sound like a hedge fund? great!
Why does it universally seem to me that articles on 80,000 hours are simply not long enough? This article seemed interesting to me, and featured some interesting facts - but it didn’t seem to really contain the trajectory of a rich argument or explanation. The end seemed rushed, and the topics are simply far too deep and complex to be brought up, considered and set down in such a short piece. I am not sure what I am supposed to walk away from this piece with - output is difficult to measure, many common metrics are incomplete, and the idea that “being 10% better than your replacement” could be “quite achievable” (which doesn’t seem to really follow from the argument and data presented)? I mean, I understand a replacement could be a lot better, but how would we see that? And what sort of things should we strive for to actually achieve being 10% better? The article doesn’t seem to answer these questions in a thick, meaningful way.
Hi Austin, we get lots of other feedback saying the articles are too long! Our approach is to build shorter posts into sequences. For instance, we’ve now got around 7000 words on the complex relationship between happiness and making a difference with your career, which is summarised here: http://80000hours.org/happiness-and-your-career This post is going to work with many more in describing replaceability.
That said, I think your comments area bit unfair. Rather than arguing that output is difficult to measure, I argued that there are some commonly used ways to measure it, which are likely to underestimate the true variability. So, the 45% standard deviation figure for high-complexity jobs found by the meta-study is an underestimate of the true variability. Being 10% better than your replacement is achievable (or perhaps it would be better to say ‘realistic’) in the sense that if the standard deviation is 45%, then a large fraction of people are 10% better than the mean. (and the true variability is even larger). In other words, you should expect 10% differences between people to be common. The topic of how to actually achieve a 10% increase in your output is huge, and will vary from job to job.
“But if we think about all the indirect effects someone can have on an organisation, like decreasing team morale and consuming lots of supervisor time, then it’s not implausible that some people have an overall negative contribution.”
There are large costs in firing people (it’s unpleasant and time-consuming for managers, lowers morale for those who stay, risks lawsuits, etc) and hiring methods are quite imperfect, so it’s not surprising that people with zero or negative marginal productivity can hang around for a while.
Did the study correct for differences in pay? It could make sense to hire someone who is less productive if you could pay them less.
If the study did account for pay differences, and the data were gathered from the US or some other nation with a flexible labor laws, it seems that there are a lot of twenty-dollar bills lying on the ground. If some outside researchers can look at a company and figure out that some employees are reliably seven times as productive as others, markets are a lot less efficient than I thought. I wouldn’t be surprised if there were such large differences in the non-profit sector, since the incentives are different.
The studies just looked at the differences within positions. They didn’t adjust for pay.
As you say, however, there are rarely 7x differences in pay for people doing the same job, so it seems like the market is not efficient in this instance. I would guess this is because of employment law, issues surrounding team morale, uncertainty about how much value different people are producing, and other distorting factors.
Hello!
If you'd like to comment then please sign in if you are an 80,000 Hours member, or fill in your name and email below.
Take me back to the blog
Take me to the homepage