Should you go into a research career? Here’s one striking fact about academic research that bears on this question: in most fields, the best few researchers get almost all the attention.
Most scientific articles get little to no attention (Van Dalen and Klamer 2005). One study found that 47 percent of articles catalogued by the Institute for Scientific Information have never been cited, and more than 80 percent have been cited less than 10 times (Redner 1998). Articles in the median social science journal, on average, get only 0.5 citations within two years of publication (Klamer and Dalen 2002). The mean number of citations per article in mathematics, physics, and environmental science journals is probably less than 1 (Mansilla et al. 2007).
By contrast, the top 0.1% of papers in the Institute for Scientific Information have been cited over 1000 times (Redner 1998). Citations per paper are basically distributed by a power law, which means that only a few papers dominate. This trend seems to hold across fields, even when the average number of citations per article varies widely (Radicchi, Fortunato, and Castellano 2008), and a similar distribution holds across individual researchers, not just articles (Petersen et al. 2011a).
What does this skewed distribution imply for the prospects of academic research as a high-impact career? You might think it means that becoming a researcher is a sure way not to have a significant impact. But what matters is the expected outcome of becoming a researcher. That the best researchers get all the attention only implies that the expected outcome of becoming a researcher may be dominated by your chance of becoming a giant in your field. The expected value of a research career could nonetheless be very high if you could have an astronomical impact as a leader in your discipline, even if the chances of reaching that position are low. So it would be too hasty to infer that you shouldn’t enter research because of the skewed distribution of output and reward.
The above only applies to the extent that (a) citations are a good measure of attention, and (b) attention is a good measure of impact. There may be problems with citation measurements (Seglen 1998), and there are different ways of using citations to assess attention (Bornmann et al. 2008). But citations, publications, and reputation seem to be the currency for influencing policymakers, university officials, students, and peer scholars (Dalen and Henkens 2004). If your potential to make impact through research in your field is directly related to the amount of credit and recognition you receive (Cash et al. 2003), then the skewed distribution of citations is highly relevant to your chances of making the most difference through academic research (Faria and Goel 2010). You might, however, aim to make an impact in other way, like teaching, or your contributions might be valuable regardless of uptake by your peers.
In the next post, I’ll discuss one possible cause of this skewed distribution.
Bornmann, L., Mutz, R., Neuhaus, C., & Daniel, H. D. (2008). Citation counts for research evaluation: standards of good practice for analyzing bibliometric data and presenting and interpreting results. Ethics in Science and Environmental Politics(ESEP), 8(1), 93–102.
Cash, D. et al. 2003. “Salience, credibility, legitimacy and boundaries: Linking research, assessment and decision making.”
Costas, R., & Bordons, M. (2007). The h-index: Advantages, limitations and its relation with other bibliometric indicators at the micro level. Journal of Informetrics, 1(3), 193–203.
Dalen, H.P., and K. Henkens. 2004. “Demographers and Their Journals: Who Remains Uncited After Ten Years?.” Population and Development Review 30(3): 489–506.
Faria, J.R., and R.K. Goel. 2010. “Returns to networking in academia.” Netnomics 11(2): 103–117.
Klamer, Arjo, and Hendrik P van Dalen. 2002. “Attention and the art of scientific publishing.” Journal of Economic Methodology 9(3): 289–315.
Laband, D.N. (1986) Article popularity, Economic Inquiry 24: 173-80.
Mansilla, R. et al. 2007. “On the behavior of journal impact factor rank-order distribution.” Journal of Informetrics 1(2): 155–160.
Petersen, A. M., Wang, F., & Stanley, H. E. (2010). Methods for measuring the citations and productivity of scientists across time and discipline. Physical Review E, 81(3), 036114.
Peterson, G. J., Pressé, S., & Dill, K. A. (2010). Nonuniversal power law scaling in the probability distribution of scientific citations. Proceedings of the National Academy of Sciences, 107(37), 16023–16027.
Radicchi, Filippo, Santo Fortunato, and Claudio Castellano. 2008. “Universality of citation distributions: Toward an objective measure of scientific impact.” 105(45): 17268–17272. http://www.pnas.org/content/105/45/17268.short.
Radicchi, F., Fortunato, S., & Vespignani, A. (2012). Citation Networks. Models of Science Dynamics, 233–257.
Redner, S. 1998. “How popular is your paper? An empirical study of the citation distribution.” The European Physical Journal B 4(2): 131–134.
Seglen, P. O. (1998). Citation rates and journal impact factors are not suitable for evaluation of research. Acta Orthopaedica, 69(3), 224–229.
Van Dalen, H.P., and A. Klamer. 2005. “Is Science A Case of Wasteful Competition?.” Kyklos 58(3): 395–414.
Comments
That said, there’s a big difference between one citation and none - if that one citation allows a discovery that wouldn’t have been made, or means it occurs a year earlier, that shifts all science which will depend on that discovery forward a year. If you believe science has accelerating returns, that’s still potentially a huge gain over humanity’s lifespan.
Another thought - a lot of publications are written under duress from the academic system, so it’s possible that the distribution of success is less skewed than you’d think from papers alone. Worth looking into if there’s any way to find similar data by researcher rather than paper?
Last thought - as you say, expected value is what matters rather than prominence in your field. It’s not too hard to imagine that some fields have hundreds or even thousands of times the expected value of others (cognitive enhancement vs cosmetics maybe?), in which case you could afford to be proportionately less successful in those fields to still contribute the same benefit.
Hey Zander, great comments.
I wonder if there is a citation index that includes the citations of publications that cite you. That might help to determine how important the one-citation articles are, since they may be cited by works that are a lot more important. But if most one-citation articles lead to a dead end, then getting a citation wouldn’t be such a big difference. Maybe that’s irrelevant though.
I think (Petersen et al. 2011a) shows a similar distribution over individual researchers.
Great point about different fields having different expected values. Was going to discuss that in the next post, but thanks for bringing it up now.
I think the ‘perverse appeal of beating the long odd’s’ (as my valuation (banking) lecturer used to say with regards CEOs and creating values through mergers) is perhaps better visualised by looking at careers with similar earnings distributions - say film acting or being a pro footballer.
The other day I was talking to a particularly sexy lady who was gunning for actress-style fame. “Interesting!” , I said, - “well do you feel you are talented enough to be the 0.1% who make it?”. “Talented? Unless you want to be Emma Thomas its not about talent - you need to be beautiful, super-hard-working and bloody lucky.” - I think she was inspired by Kierra Knightley here, she did look quite a bit like her.
And she has a very interesting point - many actor’s with banker-level salaries really are… crap. There are loads of well paid, talented actors - no argument, but that’s not the point.
So you’re top of your year at Cambridge and you’re research papers all got huge accolade - you’re the top 0.001% in terms of talent - so to what degree are you likely to be that 0.01% that write a well cited paper?
How good is the research idea? What % of the most promising research ideas actually bear fruit?
This is a set of question’s I went over and over when deciding between entrepreneur / banker and they’re FASCINATING! - If you are exceptionally talented in one area does it make more sense to go into a related field with a steeper ‘power-law’ distribution than a field with a better paid average? That’s a huge question! Should we start recommending our best looking, most outspoken and hard-core members go into earn-to-give film acting!? THINK OF THE INFLUENCE OPPORTUNITIES!!!!
I have a feeling the answer has very much to do with the degree that luck and contacts effect a field’s success distribution. In acting I would have thought it was extremely high. In banking it feels like its very low in the medium term even if it feels high-luck in the short term. In research … yeah I have no idea.
Jacob writes this paragraph didn’t show up the first time I tried to post this comment, “I wonder if there is a citation index that includes the citations of publications that cite you. That might help to determine how important the one-citation articles are, since they may be cited by works that are a lot more important.”
One thing to be aware of here is that, at least in the parts of physics that I was familiar with, a citation does not usually mean that the citing paper relies on the cited paper. A large chunk of the references are made in the introduction, when you put your paper into the current broader research context (“Related experiments have recently been described in Refs. 7-35”), and your peers might get angry if you leave out their work.
Biology papers, from memory, tended to have about twice as many references as physics papers. In general I’d be very sceptical about a paper with one citation being influential.
There’s something I don’t understand here–it seems like being a “giant in your field” is being equated with publishing high-impact papers. But is that necessarily the case? Suppose you publish one or two papers that “strike gold” during an otherwise lackluster career. They alone have thousands of citations. Are you therefore considered a giant or just a mere mortal that got lucky?
Hello!
If you'd like to comment then please sign in if you are an 80,000 Hours member, or fill in your name and email below.
Take me back to the blog
Take me to the homepage