…if when the President went around the room and he asked his advisors how likely is Osama to be in this mystery compound, if each advisor had said 0.7, what probability should the President conclude is the correct probability? Most people say well, it’s kind of obvious, the answer is 0.7. But the answer is only obvious if the advisors are clones of each other. If the advisors all share the same information and are reaching the same conclusion from the same information, the answer is probably very close to 0.7
But imagine that one of the advisors reaches the 0.7 conclusion because she has access to satellite intelligence. Another reaches that conclusion because he access to human intelligence. Another one reaches that conclusion because of code breaking, and so forth. So the advisors are reaching the same conclusion, 0.7, but are basing it on quite different data sets processed in different ways. What’s the probability now? Most people have the intuition that the probability should be more extreme than 0.7. And the question then becomes how much more extreme? … You want to extremise that in proportion to the diversity of the viewpoints among the forecasters who are being aggregated.
the John Cleese, Michael Gove perspective that Expert Political Judgment somehow justified not listening to expert opinion about the consequences of Brexit struck me as a somewhat dangerous misreading of the book. It’s not that I’m saying that the experts are going to be right, but I would say completely ignoring them is dangerous.
It’s very hard to strike the right balance between justified skepticism of pseudo-expertise, and there’s a lot of pseudo-expertise out there and there’s a lot of over-claiming by legitimate experts. So justified skepticism is very appropriate, obviously – but then you have this kind of know-nothingism, which you don’t want to blur over into that. So you have to strike some kind of balance between the two, and that’s what the new preface is about in large measure.
There’s an active debate among researchers in the field about the degree to which calibration training generalizes. I could get you to be well-calibrated in judging poker hands; is that going to generalize to how calibrated you are on the weather? Is that going to generalize to how well-calibrated you are on the effects of rising interest rates?
The effects of transfer of training are somewhat on the modest side so you want to be really careful about this. I would say, oh gosh. You really want to concentrate your training efforts into things you care about. So if it’s philanthropic activities, I think you want to have people make judgements on projects that are quite similar to that to get the maximal benefit.
I’m not saying that transfer of training is zero, although some people do say that. I think it’s too extreme to say that transfer of training is zero and, I think the transfer of training is greater if people not only get practice at doing it but if people understand what calibration is…