A meta-analysis may not mean much

Scott Alexander recently posted an interesting and provocative article: “Beware the man of one study” (and see the follow up post here).

In the post, he points out that it’s not uncommon to find two meta-analyses with opposite results on the same question.

Indeed, especially when it comes to a politically divided issue, both sides can sometimes produce apparently overwhelming evidence in support of their case. He uses the example of whether minimum wages decrease employment. He points out that the political right, if selective about which studies they pick, could correctly claim:

Economic theory has always shown that minimum wage increases decrease employment, but the Left has never been willing to accept this basic fact. In 1992, they trumpeted a single study by Card and Krueger that purported to show no negative effects from a minimum wage increase. This study was immediately debunked and found to be based on statistical malpractice and “massaging the numbers”. Since then, dozens of studies have come out confirming what we knew all along – that a high minimum wage is economic suicide.

Systematic reviews and meta-analyses (Neumark 2006, Boockman 2010) consistently show that an overwhelming majority of the research agrees on this fact – as do 73% of economists. That’s why five hundred top economists recently signed a letter urging policy makers not to buy into discredited liberal minimum wage theories. Instead of listening to starry-eyed liberal woo, listen to the empirical evidence and an overwhelming majority of economists and oppose a raise in the minimum wage.

While the left could correctly claim:

People used to believe that the minimum wage decreased unemployment. But Card and Krueger’s famous 1992 study exploded that conventional wisdom. Since then, the results have been replicated over fifty times, and further meta-analyses (Card and Krueger 1995, Dube 2010) have found no evidence of any effect. Leading economists agree by a 4 to 1 margin that the benefits of raising the minimum wage outweigh the costs, and that’s why more than 600 of them have signed a petition telling the government to do exactly that. Instead of listening to conservative scare tactics based on long-debunked theories, listen to the empirical evidence and the overwhelming majority of economists and support a raise in the minimum wage.

What does this mean?

Our thought is that you shouldn’t update too much based on a meta-analysis or data on expert consensus, unless you’ve also done a comprehensive survey to check for similar evidence with opposite results, making sure the view does actually reflect consensus. You should be especially cautious when dealing with a politically sensitive issue, or using sources who may be biased. And you should probably be generally more cautious about how much weight to put on the results of a single study, even if it’s a meta-analysis.

At 80,000 Hours, I think we may have sometimes been too quick to update based on a single meta-analysis or data on expert consensus in the past, and will consider adding a clause to our research principles page about the need to survey the evidence more comprehensively before strongly updating.