Transcript

Cold open [00:00:00]

Ezra Karger: We defined a cluster of bad outcomes related to AI, and this included AI-caused extinction of humanity. It also included cases where an AI system, either through misuse or misalignment, caused a 50% or greater drop in human population and a large drop in human wellbeing.

What we found is that the AI risk concerned group thought there was a 40% chance that something from this cluster of bad outcomes would occur in the next 1,000 years, but the AI risk sceptics thought there was a 30% chance that something from this cluster of bad outcomes would occur in the next 1,000 years.

So if we connect that to the forecasts we’ve been talking about, about what will happen with AI risk by 2100, what we’ll see is that both groups are concerned about AI risk, but they have strong disagreements about the timing of that concern. People who are concerned in the short run remain concerned about the long run and get more concerned about the long run if you accumulate those probabilities. But the people who are sceptical about AI risk in the short run are still concerned if you look at a broader set of bad outcomes over a longer time horizon.

Luisa’s intro [00:01:07]

Luisa Rodriguez: Hi listeners. This is Luisa Rodriguez, one of the hosts of The 80,000 Hours Podcast. In today’s episode, I chat with Ezra Karger about the Forecasting Research Institute’s work on forecasting catastrophic risks to humanity. I was personally really excited to hear about their Existential Risk Persuasion Tournament (or XPT) — because I remember how shocked and disappointed I was to realise there were really no good estimates of the probability of various existential risks back when I was doing research in this area.

The XPT is a huge step forward on this front. It’s a survey of hundreds of subject matter experts, superforecasters, and the general public using some really innovative methods to help the participants think carefully about low-probability events, and better understand why some participants think catastrophic events are much more likely than others.

We talk about:

• Why superforecasters’ estimates of catastrophic risks seem so much lower than experts’ estimates, and which group Ezra puts the most weight on.
• The specific underlying disagreements participants had about how likely catastrophic risks from AI are.
• The science of forecasting and the areas Ezra is most excited about exploring next.
• Plus just loads more.

Before we jump into the interview, I wanted to quickly flag that the Forecasting Research Institute is hiring for a bunch of roles right now, so if you’re excited about this work, you can learn more and apply for the roles at forecastingresearch.org/participate.

OK, without further ado, I bring you Ezra Karger.

The interview begins [00:02:54]

Luisa Rodriguez: Today I’m speaking with Ezra Karger. Ezra is an economist at the Federal Reserve Bank of Chicago, where he works on quantifying the effect of government policies on children and households. He’s also a research director at the Forecasting Research Institute, or FRI, where he works with collaborators to develop forecasting methods and run forecasting tournaments.

Thanks so much for coming on the podcast, Ezra.

Ezra Karger: Thank you for having me.

Luisa Rodriguez: I hope to talk about how you became a superforecaster and what the latest forecasting research is in general. But first, what are you working on at the moment, and why do you think it’s important?

Ezra Karger: So my work is split across a few related areas. As you mentioned, I’m an economist at the Federal Reserve Bank of Chicago. And there I do a combination of policy work and research, so I write internal and external memos and papers trying to understand how the economy is doing right now, and how it compares to recent and historical time periods.

Then on the research side, I have two interests. The first is applied microeconomics: I work on trying to understand topics in labour economics and public economics, which are two subfields of economics. And I try to quantify the effects of government policies on children and households.

And then another big area of research that I think a lot about relates to forecasting. So I, along with Josh Rosenberg and Phil Tetlock, run a research group called the Forecasting Research Institute, where we try to advance the science of forecasting in practical directions. We’re working on a series of projects to improve the quality of forecasting and to understand how better forecasts can improve decision making.

So that may sound like a lot of unrelated things, but I feel very lucky to have a job where I can focus on three domains that I think are quite connected: working to understand how historical policies affect people, thinking about current events and current policy debates at the Federal Reserve Bank of Chicago, and then exploring people’s beliefs about the future and trying to understand consensus and disagreement about events that haven’t happened yet — and forecasting those events and trying to understand why people disagree about their forecasts of what would happen if we did policy A or policy B.

I think the connections between those three things teach me a lot about how the world works, and that’s what excites me about my work.

Luisa Rodriguez: Cool. Yeah, it is true that I would have found it slightly hard to draw the connection, but that’s really nicely done.

The Existential Risk Persuasion Tournament [00:05:13]

Luisa Rodriguez: OK, let’s talk about one of your projects with the Forecasting Research Institute: the Existential Risk Persuasion Tournament, which you helped run. Just to give some colour and context, what’s the motivation behind the tournament?

Ezra Karger: So going back several years, in the summer of 2021, Phil Tetlock and I applied for a grant to run a large forecasting tournament, where we were going to ask experts and accurate forecasters to forecast the likelihood of these short- and long-run existential catastrophes.

I think we were interested in this topic for a few reasons. First, Toby Ord had just published The Precipice, which is a book that lays out his personal forecasts of risks to humanity from nuclear war, artificial intelligence, pandemics, and other sources. And right around when the book came out, the COVID-19 pandemic made risks from pandemics particularly salient.

So as with most sudden events, I think Phil and I noticed that some people claimed to have predicted in advance that something like the COVID-19 pandemic was going to be a catastrophe. And we realised that we couldn’t find a systematic attempt to survey a large group of people about these long-term risks, their likelihood, the precursors to those risks. We thought there would be a lot of value in creating a dataset of what several hundred people — ranging from experts to accurate forecasters to the general public — thought about these long-run outcomes. We can use data like this to understand what people agree or disagree about when it comes to these risks, and whether experts studying a specific risk are more concerned about that risk than other people or than other experts.

And then from a scientific perspective, this opened up several cans of worms that we were excited to dig into. So we wanted to figure out how we should elicit forecasts about very long-run questions; how we should incentivise high-quality debates about these important topics that people have a lot of trouble arguing about; and how should we elicit forecasts in low-probability domains — when most of the evidence on how we should forecast comes from forecasting questions where the true answer is between 20% and 80%.

I think that’s what excited us about this tournament, and that’s why we decided to spend several years running it and trying to design it.

Luisa Rodriguez: Yeah, I agree it’s very exciting! Can you give some context for what numbers existed for these kinds of risks before the tournament?

Ezra Karger: Yeah. So I mentioned The Precipice by Toby Ord. Toby Ord is a philosopher at Oxford, but he thinks a lot about these issues. And he wrote a book, and he just put in a table saying, “This is what I think my forecasts are.” But a lot of people reference those forecasts — and they’re the forecasts of one person without as much domain knowledge about maybe each of the domains that he’s talking about.

So while it was a really good survey, we wanted to understand if the numbers that Toby Ord put in his book were similar to numbers that other experts would give. And there are other people who’ve gone out and produced their own personal forecasts of existential risk from different sources, in academic publications, in op-eds, in blogs. But I don’t think there’s been this systematic attempt to understand what many experts, across domains, think about all of these types of risks together.

So there are some papers which say, “On biorisk, this is what a set of experts think,” or, “On risks from nuclear war, this is what people forecast” — but I think trying to do this all at once gave us a unique dataset that let us compare what experts in different domains forecasted about risks across these domains, and we could see strong patterns emerging where experts in a specific domain had very correlated beliefs relative to experts in other domains. So I think that’s new and something that hasn’t been done before.

Luisa Rodriguez: Right. Yeah, that does seem new and important. I still feel like you’re slightly underselling how big of an improvement it is.

Like, I remember I wrote a very scrappy blog post on the probability of nuclear war that was just trying to pull together other probabilities from anything I could find basically on Google Scholar that was like, what is the risk of different kinds of nuclear exchanges? I think I found maybe a handful of things. But it felt like there were these little dots of data, and nothing remotely systematic but also large.

Now there’s this dataset with well over 100 people all at once thinking about the same group of risks with the same kind of wordings and questions, talking about actually the same thing — like nuclear war between these countries at this scale — as opposed to these super differing, incomparable forecasts that I at least was trying to cobble together before. So I feel extremely grateful and excited that this exists.

Ezra Karger: Thank you. Yeah, I had the same experience. So when I was looking at the literature and trying to figure out what people thought about these topics, I found this survey done at a conference where people had really quickly written down their forecasts on a variety of topics. And I found a few public reports, policy reports, think tank reports, which said, “The risk, according to this person, is this.” But I don’t think there was anything systematic.

And I do think there’s a lot of value in saying, you know, we only had 10ish nuclear experts, but here’s what they thought. And let’s start with that as a baseline and then work from there to try to understand how nuclear risk is changing over time.

Luisa Rodriguez: Yeah, for sure. The other thing is just that, when someone says “AI risk is high,” what does that mean? Does it mean 2%, which is high given the consequences? Or does it mean 40%, which is closer to what I think of when someone says “relatively high” as a probability? Actually, what I actually think of is 80%, and probably some people think that too. But who knows, because we hadn’t systematically asked them.

So it feels like that’s the other thing that this gave me. Like, lots of people talk about these risks, but now I know what they mean.

Ezra Karger: Yeah, I completely agree. I think that gets at a really key point about this report, which is that we spent probably six months coming up with very detailed definitions of what we were talking about for each of these questions — because when we looked in the literature at what people said about risks from artificial intelligence, for example, what we found was a huge number of definitions that spanned from people would be annoyed about artificial intelligence, to artificial intelligence had a bad effect, to artificial intelligence caused human extinction.

And when you’re forecasting things that range so much in terms of definition or scope or severity, comparing these forecasts is very difficult. And so we thought it was really important to say, “Here are three pages about what we mean when we say ‘an AI-caused existential catastrophe.’ Based on this very precise definition, can you go out and tell us what you think? and then talk to other people who are in the tournament as well and try to get to a consensus about what you all think?”

And when you’re starting with different definitions, disagreeing is really hard. When you’re starting with the same definition, we at least have some hope of getting people to agree on what they think, from a forecasting perspective.

Why is this project important? [00:12:34]

Luisa Rodriguez: Cool. So with those, I think we’re getting at basically some of the reasons this is a huge improvement on what we had. And also, some of the reasons that this kind of framework of forecasting is helpful — because it helps kind of define things, it thinks about the best way to ask people for specific forecasts. Are there things we haven’t talked about yet around why forecasting was the right approach for getting these probabilities from experts and forecasters?

Ezra Karger: I don’t think forecasting is the only approach you should use to address these problems. And I want to make sure people don’t think that because we’re talking about forecasts, that forecasts are the be-all and end-all to solving problems that are really hard and complex.

But what I would say is that when decision-makers or normal people are thinking about complex topics, they are implicitly making and relying on forecasts from themselves and forecasts from others who are affecting their decisions. So if you look at recent discussions about existential risk, congressional hearings on risks from artificial intelligence, or workshops on artificial intelligence and biorisk, in many of these recent debates about policies, there’s an implicit assumption that these risks are nonzero, that these risks are big enough to matter for policy discussions.

But it’s very hard to find examples where people say, “I’m starting from this point. I’m starting from this belief.” So we wanted to make that very legible to people. We wanted to say, “Experts think this; accurate forecasters think this.” They might both be wrong, but we can at least start from here and figure out where we’re coming into a discussion and say, “I am much less concerned than the people in this report; or I am much more concerned, and I think people in this report were missing major things.” But if you don’t have a reference set of probabilities, I think it becomes much harder to talk about disagreement in policy debates in a space that’s so complicated like this.

And let me just make a quick analogy to inflation. So governments, researchers at the Federal Reserve, we carefully track expectations about inflation. So we have multi-decade surveys. The Survey of Professional Forecasters does this, where we ask forecasters for their beliefs regularly about what inflation will be, what GDP growth will be, what unemployment will be. And that’s been happening since the 1960s.

So if we’re going to continue to have discussions about existential risks, it seems useful to have forecasts that we in the future will track over time that tell us how people’s beliefs about risks are changing, and how people’s expectations about what policies might work well or poorly in this space are changing. And that’s the type of research we hope to do and build on in this report.

Luisa Rodriguez: Cool. Is there anything else you want to say about why this matters? I think I’m asking because it’s a big report. Reports are a little bit boring. It has a bunch of probabilities in it. Probabilities are also kind of boring. But I think this is so important. So yeah, is there any other kind of pitch you want to make for why people should actually pay attention to what this project has created?

Ezra Karger: Yeah, I think that’s a great question. And we thought a lot about how to explain our findings to people, and we decided to put out an 800-page report with 2,000 footnotes. So I don’t recommend that anyone read this report all the way through. I think that would be a bad idea, unless you’re really excited to learn more about everyone’s forecasts on 50 to 100 questions. But what I would say is having long reports like this that other people can look at, can use as references, can cite, I think improves discussion about these topics.

And just producing this report isn’t the end of this project. We’re writing a series of academic papers that are based on the findings of this technical report, we’ll call it — and those will hopefully be peer-reviewed. And we’ve heard from people — policymakers, decision-makers, academics — who say, “We would love to have a peer-reviewed survey of experts that we can cite in our own work.”

So we think what we’re doing is kind of setting a set of reference forecasts out there that other people can rely on when working on their own research; other people can understand what the key mechanisms are from the set of people in our tournament. And by doing that, we want to improve the quality of debate about these topics; we want to improve people’s understandings about why folks disagree about existential risks.

And while a large brick of a report might not seem like the best way to do that, I think it often is the best way to start doing that, and then we can learn from that going forward. So I do think reports can be boring, but I think if you find the interesting parts of the report, and you figure out how it relates to your own work, that’s a success in my book.

Luisa Rodriguez: Totally. Yes. And I was not trying to imply that you shouldn’t have written a report. I was just trying to imply that if someone were to find a report off-putting, there are reasons they should go engage with it anyways, which I think we’ve now covered.

Ezra Karger: I’ll just recommend that people read the executive summary, which is a few pages at the start of the report. If you want to get a sense of the results of this forecast exercise without reading an 800-page report, I think you can do that as well.

Luisa Rodriguez: Absolutely.

How was the tournament set up? [00:17:54]

Luisa Rodriguez: OK. Let’s talk more about how the tournament was set up. Who participated? What was the format? What kinds of questions were there? Maybe just starting with who actually gave forecasts.

Ezra Karger: So we wanted to get forecasts from three groups of people. The first was experts. So these are people who publish reports in their domains of expertise, who work in a specific domain — like they study nuclear risk or biorisk or artificial intelligence, or they’re working in industry, but have a really strong track record of digging into key frontier topics in AI research or biorisk. So we ended up with about 100 experts. It was between 80 and 100.

The second group we were really excited to survey was superforecasters. So these are forecasters identified in some of my collaborator Phil’s prior work to be very accurate forecasters on short run geopolitical questions. So these are people who, over the course of a year or two, gave very accurate forecasts to questions about who would win an election in a foreign country? Or what would the price of gold be? Or would there be protests in a certain country?

And while these were short-run questions, we can be quite confident — and there’s research to support this — that these “superforecasters,” we call them, are accurate over time at forecasting on these short-run questions. And we ended up with about 100 superforecasters.

So we were very curious to see how the forecasts of superforecasters and experts compared.

And then the last group was the public. We wanted to know how the forecasts of experts and superforecasters compared to people who we just found on the internet and gave forecasts on similar questions. And we think that was a useful reference class to bring in to compare these forecasts from these maybe slightly strange groups of people who think about these topics.

Luisa Rodriguez: Nice. And it’s worth keeping those three groups in our heads, because they actually all had noticeably different forecasts as groups. What was the overall format? How did you elicit forecasts from these groups?

Ezra Karger: We had a four-stage process for eliciting forecasts. And I won’t go into full detail — you can read that in the report — but the first stage involved getting initial forecasts from each person. And this was before they saw the forecasts of the people who were in the tournament, besides for themselves.

Then we put these forecasters on teams, and we asked them to work with people like themselves: so superforecasters worked with superforecasters, experts worked with other domain experts across all of the domains, and we asked them to revise their forecasts, to maybe forecast on a couple more questions.

The third stage involved pushing these teams together. So taking superforecasters and experts, sticking them on the same team, and again asking them to update. And then the last stage involved asking each team to look at the forecasts of other teams, and updating based on whether that provided them with new information.

And at each of these stages, we asked them not only for their quantitative forecasts, but also for their rationales, for their explanation of why they thought their forecast was correct. And that gave us this great dataset of I think 5 million+ words that they were really digging into trying to understand with each other these very complex questions about a variety of topics.

Luisa Rodriguez: That’s awesome. Do you want to give some examples of the questions?

Ezra Karger: Yeah. So the set of questions that everyone had to answer, because we wanted forecasts across domains, were ones about existential risks themselves. So we asked all forecasters to answer questions about the likelihood of a nuclear catastrophe or the likelihood that artificial intelligence would cause human extinction by several dates: 2030, 2050, 2100. And this got at the core questions that have been discussed in related work: what are the risks from nuclear war, biorisk, artificial intelligence, climate change, and other sources?

But in addition to these longer-run questions about risk, we also asked forecasters to answer a random subset of 45 shorter-run questions. And they were also welcome to answer more. Some people answered all of the questions.

Luisa Rodriguez: Legends.

Ezra Karger: And these shorter-run questions, they range in complexity and in timing. So we asked people questions about what would happen in the next couple years with AI progress, what would happen to the total amount of money spent on the largest run of an AI experiment, for example. Then we also asked people over the long run to think about what would happen to democracy in the world. So we were really trying to get at a cross-section of short-run questions that you might think would be associated with those long-run risks that were the core purpose of the tournament.

Results from the tournament [00:22:38]

Luisa Rodriguez: OK. With all of that in mind, let’s talk through the headline numbers. Maybe let’s start with what these groups thought about all of existential risks combined. So not individual risks, but what is the risk to humanity?

Ezra Karger: Great. So when we thought about existential catastrophe, we split it into two types of existential catastrophe. We asked a set of questions about “extinction risk”: the likelihood that these domains would be responsible for human extinction at various dates; and then we also asked about what we called “catastrophic risk”: the likelihood that each of these risks would lead to the death of at least 10% of the world’s population within a five-year period. And we asked about these numbers over many time horizons. But let me focus on the numbers by 2100, which was the last date we asked about.

Focusing on total extinction risk, this is what people in this project said was the risk of human extinction by 2100 from any source. Domain experts — this is averaged across all of the experts in the project — said there was a 6% chance of human extinction by 2100. Superforecasters said there was a 1% chance of human extinction by 2100. So we can already see that there are major differences in beliefs about extinction risk.

Now, maybe we should pause there for a second and say these numbers seem very big, right? That is a large probability to put on an extinction risk event happening in the next 80 years.

So I do want to say, and maybe we can come back to this later, that we don’t know how to elicit forecasts in low-probability domains. It’s possible that these numbers are high or low, relative to the truth, but we think it’s very important to document what these numbers are and how they compare to each other.

Luisa Rodriguez: OK, sure. So with that caveat in mind, maybe these numbers are inflated because we’re talking about very hard-to-think-about things — like the probability of human extinction. But still, it’s a group of over 100 people who have thought some about these risks, and superforecasters put it at 1% and experts put it at 6% — so 6% chance that by 2100, humanity has gone extinct. How does that compare to other preexisting estimates of human extinction risks?

Ezra Karger: If we look at the academic literature, there have been some attempts to elicit forecasts about extinction risk. What we see is that for experts, this is roughly in line with what we’ve seen in previous work. No one has looked at what superforecasters thought, so we don’t have a good comparison. But superforecasters are on the lower end of forecasts that have been discussed in the academic literature before. And again, this could be because the superforecasters maybe don’t know enough about this topic, or it could be because experts are biased and maybe think that the risks are higher than they actually are.

Luisa Rodriguez: Yeah. OK, so that’s extinction risks. What were the forecasts for catastrophic risks? Which, again, are 10% of the population dies in a short period of time, all by 2100.

Ezra Karger: So domain experts thought there was a 20% chance of this catastrophic event happening — of at least 10% of the world’s population dying within a short period by 2100. And superforecasters thought there was a 9% chance of that happening.

Those are large numbers. They’re larger than extinction risk, which makes sense. And they’re also maybe more similar: if you look at extinction risk, you see that experts were six times as concerned about extinction risk as superforecasters. Here, we see that experts are maybe twice as concerned as superforecasters. So in terms of relative risk, if we’re thinking about ratios, that really interested us.

Luisa Rodriguez: Yeah, there’s less difference. So maybe we can be a little bit more confident that they’re closer to being on the same page about catastrophic risks being in this range, as opposed to something wildly different.

OK, so those are the estimates for all of the existential risk put together. What were the headline numbers for individual risks? I think you looked at nuclear war, artificial intelligence, climate change, and biological risks.

Ezra Karger: Yeah. So let’s go back to maybe extinction risk, just so that we’re talking about one type of risk.

If we look at risks from artificial intelligence: domain experts — so people who are experts on artificial intelligence — gave a 3% chance that AI would cause human extinction by 2100. And superforecasters gave a 0.38% chance that AI would cause human extinction by 2100. Those were the largest forecasts we saw for a specific domain on extinction risk.

If we look at nuclear risk, domain experts on nuclear risk gave a 0.5% chance of nuclear extinction risk by 2100, and superforecasters gave a 0.07% chance of nuclear [extinction] risk by 2100. And so what we’ll see is that, like with AI, it was almost 10 times as high, if we look at domain experts relative to superforecasters.

Maybe one more specific risk to talk about is the source of “non-anthropogenic extinction.” So this tried to clump together all of the things that were not human-related.

Luisa Rodriguez: So like asteroid impacts on the Earth or something?

Ezra Karger: Exactly. So you can think about asteroid impacts, or solar flares, or other things that are outside of human control. And when we asked forecasters and participants in this tournament about that risk, one of my favourite findings is that they largely agreed. Domain experts thought there was a 0.004% chance of extinction from these non-anthropogenic causes, and superforecasters gave virtually identical forecasts.

And so one thing that showed me is that disagreements between experts and superforecasters did not persist across all of the domains. On the domain where maybe it’s easiest to estimate a base rate — because we know what the likelihood is of asteroids hitting Earth, or at least there are plausible estimates of that based on what’s happened over the last several million years and the data we have on that — on these risks that might be easier to understand, superforecasters and domain experts gave very similar forecasts.

Luisa Rodriguez: OK. Do you want to say more about exactly what you felt like you should take away from that?

Ezra Karger: So, coming into this project, one pattern I thought might happen is we might see that domain experts were more concerned about everything than superforecasters. Maybe domain experts are just less well calibrated, less accurate when thinking about low-probability forecasts.

And what this forecast of non-anthropogenic extinction risk tells me is that it’s not the case that, uniformly across topics, that is true. At least on something where you can arrive at a more reasonable set of forecasts based on historical data, we do see that superforecasters and domain experts agree. So this implies that something different is going on when we think about those large differences in forecasts of AI extinction risk, and forecasts of nuclear extinction risk.

Luisa Rodriguez: Yeah, that makes sense. Let’s come back to that in a bit. Are there any other high-level headline numbers you want to point out, or just a story you have about what to take away from the headline numbers taken together?

Ezra Karger: Maybe one more point on headline numbers is that when we look at catastrophic risk, we also still see differences between the domain experts and the superforecasters. These differences weren’t only there when we think about extinction; they were also there when we think about catastrophe. But the differences were maybe a little more muted.

So on the likelihood of a catastrophe caused by AI — so this is greater than 10% of the world’s population dying because of AI by 2100 — domain experts were at 12% and superforecasters were at 2%. So the difference, in ratio terms, is smaller. Now, the difference in percentage point terms is still very large: we’re seeing a ten-percentage-point gap in those forecasts of catastrophic risks caused by AI.

So the general pattern we saw across these headline numbers is that experts in a specific domain were more concerned about risks than superforecasters. And the amount of disagreement really varied depending on the topic area, but there were consistent patterns.

Luisa Rodriguez: Right. OK, cool.

Risk from artificial intelligence [00:30:59]

Luisa Rodriguez: So those are a bunch of meta-level takeaways from this tournament, which are very interesting. Let’s talk about just one of the specific risks that the forecasters made forecasts about. There are others covered in the report, so this is just a plug: If you’re interested in other risks that we’re not going to talk about, you should really go look at the report.

But for now, let’s just talk about AI. What were some common stories for how AI caused extinction, according to these forecasters?

Ezra Karger: So let me start, before we dig into maybe the qualitative explanations and back and forth, by caveating and saying that it’s very hard to quantitatively analyse text. What we wanted to do is pull out key trends or patterns from all the debates people were having about AI risk. This involved sending some incredibly good research assistants and analysts into the millions of words that people had written about this, and asking them to summarise what was going on here, to summarise the key patterns.

So we have in the report a description of what those trends were, which we can talk about here. But there is this important caveat, which is that this is filtered through the researchers and the research assistants. We’re not just giving you a dump of the one million words people wrote about this topic; we’re trying to summarise it. So I think this is useful colour, but I also want to flag that this is filtered through our understanding of what was happening.

So if we look at all of the discussion that was happening on the platform during the course of the tournament, I think discussion that focused on AI-caused extinction or AI-caused catastrophe often centred on discussions about alignment and whether AI systems would be aligned with human values. It also centred on this question of AI progress: how fast AI progress would continue over the next 50 years. And it centred on this question of whether humans would choose to deploy or to use these advanced AI systems.

So you can think about how the people who are more concerned about AI thought that there would be more progress, there were more concerns about alignment, and that humans would continue to employ these powerful systems — and then employ them more and more as the systems got better.

On the other side of things, you have the people who are less concerned about AI risk, and they were more confident that these scaling laws would slow down, or there would be things — whether regulatory or restrictions on data use or other things — that would stop the scaling laws from continuing, that would stop this exponential growth in AI progress. They were also somewhat more optimistic about alignment and the ability of people to figure out how to align AI systems with human values. And they were also more confident that humans would not employ or use advanced AI systems that were risky — whether that was through regulation or collective action or other decisions that were being made.

Luisa Rodriguez: Cool. And then did the forecasters make predictions about AI timelines in particular?

Ezra Karger: Yeah. One of the pieces of this work that I found most interesting is that even though domain experts and superforecasters disagreed strongly, I would argue, about AI-caused risks, they both believed that AI progress would continue very quickly.

So we did ask superforecasters and domain experts when we would have an advanced AI system, according to a definition that relied on a long list of capabilities. And the domain experts gave a year of 2046, and the superforecasters gave a year of 2060. So we do see an important difference there in when these groups thought that AI systems would become advanced, according to some technical definition, but they both thought that this would happen in the near future. I found that really interesting.

Luisa Rodriguez: Yeah, that is interesting. How did those timelines compare with other timelines out there in the world?

Ezra Karger: On the timelines front, I think it’s hard to compare it to timelines that people have forecasted on in the past. There are a few reasons for this. One is that there’s no clear definition of when this measure of advanced AI system should be met. So we drew from other definitions that have been given in the past, and you can find estimates that are all over the place. You can find estimates that this will occur in the next few years; you can find estimates that this will take centuries. A lot depends on the group of people who are forecasting, and then also the definition of what it means to get to a point where you have an advanced AI system. So I don’t think there’s a good way to compare that to prior estimates.

Luisa Rodriguez: OK. I guess, given that, I know that we can’t meaningfully and rigorously compare them. But do you have some sense of like, “The forecasts from this tournament are kind of sooner or kind of later than the forecasts that I hear people make in the wild,” or does that feel farther than you want to go?

Ezra Karger: If I had to compare this to the recent discussions that we’ve seen in op-eds or papers where people have written about forecasting AI progress, I think this is pretty consistent with what people are saying. People think there are going to be significant improvements in AI capabilities over the next 20 years — and that’s true whether or not you’re worried about there being risks associated with AI progress. So this is reassuring to me in some sense, in that the forecasters in this tournament I think are very consistently reflecting the views of either AI experts, or just smart, thoughtful people who are thinking about this question.

Now, where they disagree: so we’ve talked about timelines; we asked other questions about AI progress, and there was really remarkable agreement. There was some disagreement. One of my favourite questions was: “When will an AI system have written at least three New York Times bestselling books?” And we had clear resolution criteria in there, where we said it has to actually do most of the writing with minimal human input; we have to know about it or have a panel of people who decide that this happened. The domain experts said this would happen by 2038, and the superforecasters said this would happen by 2050. That’s 12 years apart, but it’s within the realm of, like, AI progress is going to be impressive and AI systems are going to be able to do things that they cannot do now that look more like what humans do.

So one of my biggest takeaways from this project is the disagreement exists when you look at some measures of AI progress. But where people really disagree is whether there will be regulatory interventions, or whether there will be risks from that AI progress.

Luisa Rodriguez: Right, right. Yeah, that is really fascinating. Did they make predictions about economic impacts of AI? I guess you could consider these a risk because enormous impacts on the economy could be very strange and very weird, or you could consider it a benefit or not a risk. But either way, did these two groups end up having at all comparable beliefs about how AI would affect economic growth?

Ezra Karger: So this was one of the places where we saw the strongest levels of disagreement, beyond the risk questions. I think you’re getting at a really important point here, which is the beliefs about capabilities are really interesting. And there’s, I would argue, a lot of agreement in this data when we think about whether AI systems will be able to perform well on specific benchmarks or whether they’ll be able to write New York Times bestselling books.

We also asked superforecasters and domain experts what the probability is that annual global GDP growth will increase by more than 15% year over year before 2100, or relative to a base year. What we see is that AI domain experts thought there was a 25% chance of that happening, and superforecasters thought there was a 2% to 3% chance of that happening.

Luisa Rodriguez: That’s pretty different.

Ezra Karger: Yeah, that’s different. And what I like about that is it’s different than risk. It’s associated with risk, but it’s different. Because you might think that these capabilities are improving, and then those get baked into economic growth, and those cause a large increase in economic growth. Or you might think that these capabilities are increasing, the world figures out how to harness these capabilities and that happens gradually, and it happens with regulation, and it happens in a way which is very controlled. And therefore, as the superforecasters forecasted, there’d be a much lower chance that there’s a sharp increase in growth in one year. So I think this gets at a key mechanism underlying the differences in belief about risk.

Luisa Rodriguez: Right. So that mechanism, just to make sure I understand, is about the pace at which these increasingly impressive capabilities are incorporated into the actual means of production that make economic growth happen?

Ezra Karger: Exactly.

Luisa Rodriguez: Yeah. Cool. That does seem really important. Was there any convergence here?

Ezra Karger: On the short-run questions, we did not have enough data to look at convergence from the start of the tournament to the end of the tournament. And that was, I think, a flaw of this project. We were asking questions about so many different domains. There were 45 shorter-run questions, so getting people to give their initial forecasts and then follow up didn’t work as well.

So what we’re reporting here is their final forecast, because over time, people forecasted on more questions. I would love to do followup work where we focus on a few of these AI-related questions. We’ve done some work, that we can talk about later, on that. I would love to try to understand, is there agreement or disagreement about different components of growth, AI progress, AI capabilities, and general economic outcomes related to artificial intelligence?

Luisa Rodriguez: Are there any short-term forecasts coming due at the end of 2024 that you’re excited to see the outcome of?

Ezra Karger: I think one question that’s worth talking about briefly is about how much compute and how many dollars will be spent on the largest AI run, the largest experimental run of an AI system. This is a case where we do see that, by 2024, superforecasters thought there would be $35 million spent on that largest run, and domain experts thought there’d be about $65 million spent on that run. I think both of these estimates are going to be off and too low, but I think the superforecasters will be more off.

So this is a case where, at least on one question, I think you are seeing that the people at the frontier of thinking about artificial intelligence are going to be a bit more accurate. Now, this really is a function of how you measure accuracy, because I think the domain experts are also going to be off — but we won’t know how off until the end of the year, when we can see what the next months bring from AI progress.

I think we’re already getting hints that there might be something interesting going on with forecasts of AI progress and use of AI systems, where some groups are better than other groups. But something interesting that’s happening is I don’t think those are that correlated with beliefs about risk. So even among the experts, I think the people who are more optimistic or pessimistic about trends in the cost of compute for the largest AI run, I don’t think they disagree that much about risk.

And we did an analysis where we took beliefs about risk, and we looked at the top third and bottom third of our samples in terms of their beliefs about AI risk. And if you try to find questions where the median forecasts for those two groups are very different, you just don’t really see much in the short run. So even if, by 2024, some people are more accurate than others, at the moment I don’t have a good way to figure out or to think about whether that’s going to be correlated with beliefs about risk.

Luisa Rodriguez: Yeah. And the reason they don’t seem that correlated is the kinds of beliefs and reasoning and facts that we have about the questions that are resolvable in the next few years — like how much will be spent on compute — are just pretty different from the kinds of beliefs you have to have about arguments about how technology gets incorporated into the economy over decades and decades, and how we’re able to address risks from novel technologies?

Ezra Karger: Exactly.

Luisa Rodriguez: And those differences mean that it’ll just be really hard to update on these short-run forecast results either way.

Ezra Karger: Yeah. I could be surprised, because we can’t look right now at the joint distribution of who is more accurate or not. But my expectation is we will get a muddy picture about whether the people who are more accurate were more concerned about risk or not. And I think that is for exactly the reason you said: on these short-run questions, we’re looking at very specific indicators of AI progress or AI risk. They’re not that correlated with these longer-run questions about how AI systems will advance over the next 50 to 100 years, and how risk will change, and how regulation will respond.

So I think when we’re thinking about all of these things working together in a complex system, these short-run questions don’t have that much to say.

Luisa Rodriguez: Yeah, that makes sense. When I’m trying to like, wrap my head around why this difference exists, it feels like for the kinds of questions that are these longer-term, large-scale outcomes of these new technologies or of AI in particular, it feels like even if I think about questions that have in some sense resolved — like how economic growth has happened over the last 100 years, and how different technologies have affected economic growth in different countries — there are so many narratives that analysts have put forward for like, what has caused economic growth in Asian countries?

And that’s all so complicated, and we already know what’s happened. So I guess it just makes sense that the kinds of questions that are much harder to think about even now, with all the data we already have on them, are the kinds of questions that we’d need to have answers to in order to make good predictions about the longer term, kind of more high-level impact questions about what will be the effect of AI on society.

Ezra Karger: I like that analogy a lot, because I feel like there are thousands of economists trying to figure out why countries grew over time.

Luisa Rodriguez: Yeah, exactly.

Ezra Karger: And now we’re asking the much harder question of, “Why will countries grow over time?” or “How much will they grow over time going forward?” where we don’t actually see the data. So if we don’t know why this happened in the past, figuring out why it happened in the future feels very difficult.

And you can make analogies — and people have made these analogies — to technological progress coming from the internet or the Industrial Revolution. I think at the beginning of those changes in how the economy worked, there was still a lot of uncertainty about where on the exponential growth curve or on the non-exponential growth curve we were living. And now, when we don’t actually see the data, we certainly don’t know what the answer is.

So if you’re optimistic or pessimistic about AI progress and how that will translate into economic growth, we can see that we still don’t know how important the internet was to economic growth over the past 30 years. How are we going to forecast whether progress in AI over the next 30 years is going to be important for economic growth?

So the short-run resolvable questions, where we’ll know the answer in the next two years, I feel like it is completely wishful thinking to say, we’ll just know who was right after two years. And again, I don’t think that’s too upsetting to me, because the world’s a complex place. We’re asking people these questions; we’re trying to establish some basic patterns of who is more accurate, and how that’s correlated with risk beliefs.

But it also might be the case that the policies that people think should be put in place — or the beliefs people have about actions that should be taken or philanthropic grantmaking that should be done, or what kinds of debates should be happening now — maybe everyone agrees on those things. Maybe everyone takes this as evidence that there’s a lot of uncertainty in the world, and we should think more about it and work to figure it out more together. But this focus on disagreement I think can often cloud the fact that this uncertainty is really important, and we should all figure it out more.

Luisa Rodriguez: Yeah. OK, cool. Let’s leave that there for now. Again, we don’t have time to talk about the other risks, but again, if listeners are interested in those, I really recommend looking at the whole report. It is super interesting.

How to think about these numbers [00:46:50]

Luisa Rodriguez: A thing you’ve mentioned a few times is that there’s a difference between the absolute difference, in percentage points, between the estimates from the superforecasters and also the estimates from the domain experts. But then there’s also these ratios. So in some cases, it’s the case that the estimates might differ by like a single percentage point, but that equals something like 10x the risk. Which of those should we be paying attention to when thinking about how different their views are on these risks?

Ezra Karger: We wanted to make sure not to make that decision for the reader, so we do present a lot of raw data in the report so that you can look for yourself at these differences. But I think there are really interesting reasons you might care more about ratios or percentage point differences in levels.

To give an example of that, on AI extinction risk, domain experts had a 3% number, and superforecasters forecasted around 0.3%. That 10x difference means that if you’re looking at something like an expected damage calculation, and you’re multiplying these probabilities by some cost, those numbers will be 10x apart, because you’re multiplying those probabilities which are 10x apart by some number. What that means is if we were looking at larger percentage point differences, but maybe at the middle of the probability distribution — like a 50% number versus a 60% number — that might matter less for something like a cost-benefit analysis.

So I was really interested by these disagreements if you think about ratios, but I also think percentage points are really important. And the reason for that is percentage point forecasts tell you something about absolute risk. So if you’re trying to come up with policies to reduce risks to maybe under 1% or under 5%, where these forecasts are in probability space matters a lot more.

So let’s say we look at the risk of an AI-caused catastrophe by 2100: the domain experts were at 12% and the superforecasters were at 2%. You might think that trying to get to a risk of under 5% is important. And if that’s the case, then you may want to focus on catastrophic risks from AI and try to halve the risk, or halve the domain experts’ beliefs about the risk, if you’re starting from that starting point.

So I think the question of whether you should focus on percentage point differences or ratio differences is really going to be a function of what you’re doing with the forecasts.

Luisa Rodriguez: Right. That makes a lot of sense.

Should we trust experts or superforecasters more? [00:49:16]

Luisa Rodriguez: Pushing on, I’m interested in how to think about whether to trust the superforecaster forecasts or expert forecasts more. On the one hand, I think if you told me you were running this tournament, and told me you were going to have this superforecaster group and this domain expert group, I’d have been really excited in general.

Partly because I’m like, great, we’ve only had these domain experts making these estimates in these weird biased contexts. And I would love to know what the superforecasters think, because I’ve read the one or two books on superforecasters, and I know that they’re the kinds of people who are much better at thinking about things probabilistically and putting numbers on things in a way that is actually well calibrated and not doing the weird bias things brains do when they’re not trained in thinking in a calibrated way. So that would have been my kind of initial starting point.

On the other hand, when I read the report, and was like, “Interesting. The superforecasters think these kinds of risks are much lower than the experts. They’re also lower than my personal views, at least on some of these risks,” that made me want to backtrack and put more weight on the domain experts’ views — which feels very strange and yucky and weird to me. So I’m suspicious of myself there.

So that all made me want to ask you, did you have anything like this? Did you have something like, based on kind of priors about these groups and their knowledge about these things, which group did you endorse putting more weight on, before you saw the results?

Ezra Karger: Yeah, that’s a really good question. I think I have pretty strongly held beliefs about these risks that weren’t affected too much by the report.

Maybe I can talk a little bit about why you might want to trust one of these groups, but then also why they both might be wrong in either direction. Because I think one problem with a tournament like this, or an elicitation exercise like this where you ask a bunch of people for forecasts, is there’s this implication that the true answer is somewhere in the range of the forecasts you got. And I don’t actually think we know that, so I want to talk a little bit about the tradeoffs there.

So, just like you said, you may trust the superforecasters more because they are the forecasters: they are the people who were quite well calibrated, quite accurate in forecasting on important questions, important geopolitical questions. They may have been over short time horizons, but they have some track record of accuracy. So maybe we should update based on the beliefs of the superforecasters.

On the other hand, if you think that the world is changing very fast, if you think that new technological trends, artificial intelligence, that’s changing so fast that only the people on the frontier of those new changes in artificial intelligence — like large language models, scaling laws — if you think that is something that you need a lot of domain expertise to understand, then you may want to defer to the experts on a topic. You may say the world’s changing a lot. The superforecasters have shown that they’re accurate in these regular times, these regular geopolitical questions. But now that things are changing fast, we really need to go to the frontier experts and see, what do they think?

You also may say you want to put some weight on each, and that will put you somewhere in the middle.

But there are arguments for why both of these groups might be wrong, and I want to talk a little bit about that. So we have no research, or very little research, on whether people are accurate at forecasting in low-probability domains. There’s some kind of fundamental work in psychology from Daniel Kahneman and others about low-probability forecasting.

But we don’t have a lot of empirical evidence because it’s very hard to test. If you have a bunch of events that have a one-in-a-billion chance of occurring and you ask people to forecast on them, odds are that none of them will happen. So you won’t know who is accurate, right? Someone could have said 1%. Someone could have said one in a million. You’ll never really know who was right.

That’s a problem for empirical tests of this type of question. It might be the case that people are just biased when they’re forecasting in low-probability spaces, and they don’t really forecast well when trying to differentiate one in a trillion from one in a billion from one in a million from 1%. And if that’s the case, the true answer might be lower.

On the other hand, on the expert side, we can talk about selection into who the experts in this project were. The people who are quite concerned — who are even more concerned than the experts in this project — about these risks may have decided that they didn’t have time to work on a project that was a long-lasting forecasting elicitation exercise that would take them 50 hours where they had to think about these questions. They want to actually think about how you can mitigate risks from nuclear disaster, AI, pathogens.

So we may have gotten a select group of experts who are less concerned about these risks than the experts who didn’t participate in the tournament or some experts who didn’t participate in the tournament. I’m a little bit less concerned about that because our estimates are so similar to the literature on expert elicitation from these domains.

On the superforecaster side of things, I got some really interesting responses when recruiting superforecasters. We reached out to many of them personally and on the forum where the set of superforecasters coordinates on what they’re going to do next. And I had three people tell me that they didn’t think these risks were risky. They didn’t think they’d enjoy forecasting on them because they just thought the experts were wrong, so they didn’t want to participate. They just would rather do some crossword puzzles, read some books, go about their lives. What that means is it’s also the case that the superforecasters might be biased towards the ones who are more concerned about risk from the sample of accurate forecasters.

So I want to push back a little bit against the idea that we should put weights on these groups that put our answer somewhere in the middle. I think what we’re trying to do with this project is produce a baseline set of beliefs from groups who are very interesting. But we’re not trying to say that the true answer is one of them or the other one. We’re saying that the answer is somewhere out there, and understanding what people think and what their uncertainty is about the topic is really important if you care about these topics.

Luisa Rodriguez: Yeah. I mean, I’m definitely very sympathetic to that. I feel like sometimes we have to make choices about numbers to use. And one really concrete example is I recently interviewed Matt Clancy on whether the returns to science are good. And he had a billion nice things to say about this work because he found it so valuable, but it made really big differences to his results whether he picked the forecast by the superforecasters versus the experts, which makes sense because they were so different.

So I think to some extent, I really, really want to at least understand more about how to think about which group to put more weight on — given that just really practically speaking, sometimes people will want to use these estimates to think about other things, and they’re not going to either form their own views on exactly what the risk number should be, so they’ll want to defer somehow. So thinking about how to defer and at least thinking through some of the considerations still seems really valuable to me.

Ezra Karger: So that’s a really useful example. I love Matt’s report, and if you look at his paper, he shows that a lot of his results, it depends on what forecasts you use, or you can look at the range or the robustness of the results to the forecast you use. And I think that in and of itself is the most exciting thing you can do with the forecast.

So instead of saying we should trust this group or that group, what I would say is, if you’re trying to make a decision that relies on forecasts of these topics, and your results don’t change when you use the forecasts of either of these groups, that means that they’re pretty robust to whether you trust the superforecasters or the experts.

But if you’re trying to decide something — whether it relates to policy or the returns to science or other questions that people think about in the academic literature — if your results are not robust to whether you stick in 0.3% or 3%, then that means you should be uncertain about it. You can decide how uncertain to be, but I think that type of uncertainty is my biggest takeaway from this report, which is: if you’re trying to put together a set of decisions, make policies based on numbers like the ones we presented in this report, you should be thinking about uncertainty surrounding that number. You should be understanding whether, if this group is right or this group is right, then the policy recommendation shifts or the answer to the research shifts.

And if that’s the case, maybe it’s worth taking a step back and thinking about how confident you should be about the results from your report. That’s, I think, the most useful thing to do with these numbers.

Luisa Rodriguez: Yeah. It sounds like even if there are reasons to think that both groups are biased in different ways — and it sounds like they probably are — we can’t learn so much from that that we should then go be like, we should use the superforecaster estimates and all of the back-of-the-envelope calculations we ever do from here. We’re just actually still way too uncertain about how [good] these groups are with these kinds of probabilities and these kinds of topics.

Ezra Karger: Yeah, and I will say that’s a research agenda that we’re pursuing. We want to do a lot more research on trying to understand how well people can forecast in low-probability domains, trying to understand how you should incentivise people to forecast on these unresolvable questions. So I hope to have better answers for you in a year or two.

Luisa Rodriguez: Nice. Yeah, I find it really exciting that I think you said that by the end of this year there’ll be at least some kind of short-term questions resolving in a way that will mean you’ll get at least some colour on which groups are doing well — in the short term, at least.

Ezra Karger: Exactly.

Luisa Rodriguez: You might think that you’ll still have problems there, because the kinds of things that are going to resolve in the short run might be the kinds of things that superforecasters will do better at, even though there still might be the issue that everyone’s just bad at predicting low-probability events, and we should still be relying on expert forecasts because they just know more about the things that can happen over 50 to 100 years, given the way technology is changing.

Ezra Karger: Yeah, that tradeoff will still exist, but we’ll at least be able to give you some more evidence about which groups were more accurate. And then also within the group of experts or superforecasters, we can look at who is more accurate and then ask whether those subgroups were more concerned or less concerned about risks from various sources.

So I think it’s going to be a personal decision of how much you want to update on your own forecasts of these things when you see which groups were more accurate, or when you see that the more accurate forecasters had these higher or lower levels of risk. But I think it is useful data to have so that you can make a better decision for yourself about how much to weigh up these groups’ beliefs.

Luisa Rodriguez: Yeah. You might just refuse to answer this, but do you have a guess at, either at the end of the year or in five or 10 years, if you just had to make a bet about which group was going to come out looking like they made more accurate predictions, which group it was going to be? Even though I know they’re not clearly distinct groups; they’ve got some overlap, and it’s complicated. But if you’re willing.

Ezra Karger: So I’m expecting the results to be muddled, unfortunately. And not in a bad way.

Luisa Rodriguez: Sounds like life.

Ezra Karger: Yeah, I think it’s like life in that I think that the superforecasters are going to be more accurate on some questions and then the experts will be more accurate on others. We may find interesting patterns where the experts are more accurate on the AI-related questions, where things were moving more quickly, and I think that would be a very interesting result. But we’ve only looked at a couple questions where the answer seems more obvious now.

So we need to go in and do a really careful, large-scale analysis of who is more accurate in what domains. What can we say about average accuracy, and what can we say about differences in accuracy across groups?

Maybe one point on that is we did look, in this project, at whether people who were more concerned or less concerned about risks had different short-run forecasts. And we found some differences, but surprising similarity in the forecasts from people who disagreed about risk. So another thing we might find in eight months, when we resolve these forecasts on the short-run questions, is that it doesn’t tell us much because there wasn’t that much disagreement in the short run.

And that might be because when you have either a flat line or an exponential curve that you’re trying to predict, if you’re at the start of it or if you’re towards the beginning of it, it’s very hard to know which world we’re in. I think that might be happening with the experts and the superforecasters. Whoever’s right, we may just not be in a world right now where we can differentiate accuracy.

Luisa Rodriguez: That makes sense. OK, well, we will eagerly await those results.

The effect of debate and persuasion [01:02:10]

Luisa Rodriguez: Another interesting part of the tournament was this persuasion aspect. Can you explain how that worked in a bit more detail?

Ezra Karger: Yeah. We directly incentivised forecasters to produce high-quality rationales. We had their teammates vote on who had given the most useful comments and who had given the most useful explanations, and then we paid cash prizes for people who had done a good job according to their teammates.

Our goal was to see if we could really incentivise high-quality debate. I think the debate varied a lot. I was following along with the tournament in real time to make sure nothing was breaking, so I was reading the back and forths, and we saw a really interesting combination of argument styles.

We had some arguments where an expert would say, “I am basing my forecasts on Toby Ord’s book, and he says that the risks are this.” And then a superforecaster responded and said, “Well, that’s one random philosopher’s beliefs. Why should we trust them? That’s not evidence; that’s just an opinion that you’re deferring to.” And this would go back and forth and they wouldn’t really make any progress.

And then on other questions, we saw some really interesting convergence where, at least anecdotally, someone would say, “I think you’re forgetting about this. I think you’re forgetting about this key fact.” And then someone would respond and say, “I think you’re right. I think I need to update” — whether it’s down or up on a forecast of a short-run indicator or of risk — because they had not thought about a key consideration.

What I would say is that, overall, persuasion was largely a wash. When we tried to do this, we saw very little convergence in some quantitative ways on the short-run indicators and on some of the long-run risks.

But we did see some convergence. We’re working on some academic papers that come out of the results of this report. If we look just at AI extinction risk by 2050 or 2100, we do see that the experts became less concerned over the course of the tournament about risks from AI. This could be because of persuasion, or this could be because of the news that was happening at the time that was causing them to be less concerned because people were more focused on AI safety.

But when we look at stage one — the initial forecasts that were given in private by the superforecasters and experts — we saw that by 2100, the superforecasters had a 0.5% chance on AI extinction risk, and the experts in AI had a 6% chance. And then, by the end of the tournament, these answers had converged to 0.38% and 3%. So we did see about a halving of the difference.

But while the p-value on this is under 0.05, if you think about these forecasts as being independent, a lot of other stuff was going on that might make you mistrust the statistical significance of that result. So I don’t want to put too much weight on whether or not there was convergence. I think there wasn’t that much convergence.

Even if you look at that one question where we saw some convergence — and we also saw convergence by 2050 and some convergence in total extinction risk — if you look at those convergence numbers, they still leave very large gaps. We went from maybe a 10x difference to a 10x difference, but in percentage point terms, it dropped from maybe 5.5 percentage points to 2.5 percentage points. So this gets back to this question of do we care about differences in ratios or percentages? And if all you care about is the ratio difference, then I don’t think there is much convergence.

Luisa Rodriguez: OK, this seems really important to me too. It sounds like people spent hours and hours debating each other. And I know there’s kind of a range of engagement. Some people were really engaged in having these discussions, and some people probably sat them out. But still, some people invested a lot of time in trying to express their reasoning, at least considering other people’s reasoning. And the fact that there was almost no, at least meaningful, convergence seems important.

Do you think there’s anything that we can learn from that? Do you think there’s some underlying thing that explains it that is worth pulling out?

Ezra Karger: I think my biggest takeaway is that when people have these strongly held beliefs, it’s very hard to cause them to change their minds. And that’s not going to be surprising to people who think a lot about beliefs and forecasts. But I was hoping for more convergence. I think I was coming in and saying, maybe we can get people to agree if they think about these topics together.

Luisa Rodriguez: Right.

Ezra Karger: It’s possible that the structure of what we did wasn’t well set up for that. We were trying something very strange. We just made a decision that getting people together in this way would cause them to converge. We didn’t put them in a room: they were on an online platform. We didn’t give them specific hour requirements of, “You must be on Zoom for two hours to try to get to the bottom of this key source of disagreement.” We didn’t try to really guide the disagreement — so we didn’t have a structured adversarial collaboration where people would come in and say, like, “I’m noticing that these two people are not converging. Can we dig into the points underlying their disagreement?”

So I updated to thinking that this is harder than I thought originally. I think a more optimistic take would be people had strongly held beliefs. The arguments for why these risks are high or low are not particularly persuasive to people who disagree with you.

But maybe that means we should just stop disagreeing about whether these risks are high or low, and try to figure out whether these groups agree about other things. It’s possible that the actions that people in these groups would want to take — to, you know, do policy A or policy B — it’s possible we’d see a lot more agreement when it comes to those decisions about actions than these forecasts of underlying beliefs.

So I’m always really curious if we’re focused on the wrong thing here. We’re focused on quantifying uncertainty about forecasts about risk. And in fact, if you went to these groups and said, “Do you think that there should be a liability regime for AI systems?,” they would just say, “Yeah, of course. We disagree by an order of magnitude in how much risk there is from AI or nuclear risk. But overall, we really agree that this set of common-sense policies are things that we should do.”

And so I think there’s a pessimistic take — which is that we couldn’t get people to agree on these underlying forecasts — but the optimistic take is, well, maybe that means we should just move on and figure out where they agree.

Luisa Rodriguez: Yeah.

Forecasts from the general public [01:08:33]

Luisa Rodriguez: So you actually did a whole project that follows up from the persuasion part of this tournament, and we’re going to talk about that more in a bit. But before we do, another element of the tournament was that you actually asked members of the public about their probabilities on the same existential risks, which is something you mentioned earlier. Were there any interesting takeaways from that part of the exercise?

Ezra Karger: Yeah so this was probably my favourite piece of this project, even though I didn’t expect that in advance. We wanted to have a comparison group of normal people — people we found online who were going to answer a similar set of questions — and we wanted to ask them for their estimates of risk.

But we also wanted to explore what we call “methods of elicitation”: whether we ask a question in one way or another way is going to affect their forecasts. And I was particularly interested in trying to understand, in these low-probability domains or these potentially low-probability domains, whether giving people access to known low-probability events and their probabilities would cause them to change their forecasts of these risks.

So what we did is we started off with a survey of several hundred people from the public. We recruited them from a platform called Prolific, and we asked them to forecast on the same existential risks as the superforecasters and experts had forecasted on.

And we saw surprisingly similar answers. On AI extinction risk, the public said there’s a 2% chance by 2100. On total extinction risk, they said a 5% chance. And these numbers are very usefully in between what the superforecasters said and what the experts said. So my first thought was, did we learn anything by asking the superforecasters and the experts? We could have just asked random people on the internet.

Luisa Rodriguez: We could just ask random college students.

Ezra Karger: Yes. But we then went back to these people who had given us these forecasts that were very similar to what the superforecasters and the experts had said, and we said, “Here are some examples of true probabilities of events that could occur with low probability.”

So we have this idea that when people say a low-probability number, they often don’t know the difference between 2% and one in 300,000 or maybe even 5% and one in a million. And giving them access to some comparison classes we thought might help to improve their forecasts.

When we did that — when we gave people in the public access to a set of eight or 10 reference probabilities — the public changed their beliefs a lot. So this is the same group of people who said that 2% and 5% number I just told you about. And when we gave them these reference classes, they gave a median probability of human extinction by 2100 of one in 15 million, and a median probability of AI-caused human extinction by 2100 of one in 30 million.

Let’s just compare those numbers for a second: one in 30 million is very different from 2%, right? If we’re thinking about a cost-benefit analysis or expectations, those are many orders of magnitude apart. And the same thing is true for total extinction risk.

So this is maybe a place where I think we need a lot more research and understanding of how people forecast. If you’re thinking about what the superforecasters and the experts said, and you’re thinking that this 0.3% or this 3% number are correct, well, it should give you pause to know that among a survey of people who gave roughly the same number, if you give them a set of reference classes in low-probability space, they then reduce their forecasts tremendously.

I think there’s a useful anecdote here. There was a New York Times podcast with Lina Khan, who’s the chair of the FTC. And the New York Times journalist asked Lina Khan, because this is now a common interview question, “What do you think the probability is that AI will cause human extinction?” And I’m paraphrasing here, Lina Khan said that she has to be an optimist about this, so she’s going to hedge on the side of lower risk. So she’s saying, “I’m optimistic. I don’t think there’s a high chance this is happening.” And the New York Times says, “So do you think there’s a 0% chance? Do you think there’s no chance that this happens?” And Khan said, “Not zero. Maybe like 15%.”

So what does this mean? Well, I think that very smart people can be miscalibrated on low-probability events. But if you have someone saying that they’re not at all concerned about a risk, and then giving a number that’s 15%, in some contexts that might be very low, but in some contexts that feels very high — when you’re talking about AI causing human extinction.

So if we look at the results from this public survey, my understanding of these results is that if we gave people access to a precise set of reference classes, it would cause them to give us lower forecasts of risk. I think Lina Khan is a good example of this, where she thought that a good number, an optimistic number, was 15%. But as we’ve been talking about, you have this feeling that the numbers we were looking at here of 1% and 6% were pretty high already. And so you disagree with that; you don’t think 15% is optimistic at all.

There are several questions that come out of this. One is: If we did this with superforecasters or domain experts, what would happen? And we haven’t done that yet. We may do that in the future.

Another followup question you might have as well: You’re anchoring the public, you’re giving them a set of low probability numbers; of course they’re going to choose a low-probability number, and we do see that’s the case. So we’ve done some followup work where we went back to this public survey and we gave them a different set of reference classes. We cut out the lowest of the reference classes, and it turns out that several of them are just choosing the lowest number we give them. So several of them are just saying, “You give me a set of reference classes. I think that AI causing human extinction has the lowest number you gave me chance of happening.”

Now, that could mean that they’re very miscalibrated. It could also mean that they’re just trying to tell you that they think the risk is very low. So how should we update on that group? I’m not really sure, but it opens up a whole set of puzzles that I’d like to think about more.

Luisa Rodriguez: Yeah. Does it ever make you just pessimistic about this whole thing? Like one reframing of the question totally changes the kinds of probabilities you’ll get on very important questions. And maybe you’d like to think that most of what is driving variation in the probabilities people are giving is nuances in their arguments, but maybe all of the variation that you’re getting in these probabilities is just like, how do people think about low-probability stuff, and what kinds of low probabilities do they understand? What are their very unrelated-to-arguments biases that they have when thinking about numbers? That just feels very unnerving to me.

Ezra Karger: Yeah. I think the pessimistic take is we should just throw out all these numbers. No one’s really good at forecasting low-probability domains.

This may be because I think about this in research, but this just makes me very excited — because people are making decisions based on low-probability forecasts all of the time. So this means there’s a significant amount of room to try to understand how we can help people make better forecasts in low-probability domains, how different modes of elicitation affect their forecasts. And maybe their beliefs about actions vary a lot or don’t vary at all, depending on how you elicit these forecasts.

One of the projects we’re working on that I’m most excited about is a large-scale experiment where we’re trying to understand how people forecast in low- and mid-probability domains. And there are ways to test for how calibrated people are and how accurate people are. This project is led by Pavel Atanasov, a collaborator of ours, and we are seeing that you can get more accurate forecasts in low-probability domains by eliciting forecasts in some ways versus other ways.

Now, we’ve only collected a few hundred people’s forecasts so far, so I don’t know if that’s going to hold up, but I’m very excited to explore whether we should have elicited these forecasts differently from superforecasters, from experts — whether we should have gone to the public and said, “We now have evidence that this is how we should ask you for your beliefs about low-probability forecasts.” And I think the fact that we have very little empirical evidence of what to do there is just a gaping hole in research that we should try to fill.

Luisa Rodriguez: Yep. I guess it still sounds really daunting to me, because the kinds of probabilities we care about that are very low, we won’t resolve with any confidence. Is the thing that gives you hope that there are some kinds of low-probability forecasts people can make, without knowing the answers in advance, that are resolvable?

Ezra Karger: Yes.

Luisa Rodriguez: Can you give just an example?

Ezra Karger: I think the lightning example is a good one. Let’s say I take you, and I put you in a room with a piece of paper, and I ask you to write down the probability that a human will be killed by a falling fridge or by lightning or by other sources. I can experiment whether asking you that question one way versus another way produces better forecasts, because I know roughly what the true answer to those questions are.

So I do think there are ways for us to evaluate whether we can elicit low-probability forecasts well, without asking about these very unresolvable, long-run questions. And just knowing mechanically how we should do that can then tell us how we should be using forecasting and eliciting forecasts in these long-run domains.

So I think it’s a really interesting exercise, if you think a lot about low-probability forecasts. Like, you’ve interviewed several guests about existential risk. It might be good to just sit down with 10 events that people think are pretty low probability, and see if you have well-calibrated forecasts of what those probabilities are. And if not, what could have caused you to make better or worse decisions when forecasting? How could you have come up with more accurate forecasts in those spaces?

But yeah, I think the reason I’m optimistic about research is that we have these other types of questions where we can really analyse accuracy and calibration that aren’t these questions about existential risk, and we can then extrapolate from that to figuring out how we should ask these questions.

But to put another optimistic take on it, we talked about how one of the benefits of this project was trying to understand people’s uncertainty about these probabilities. So this is just another source of uncertainty. And it means that if you’re modelling something — if you’re doing what Matt did in his paper, where you try to understand the returns to science research — maybe you should add some more uncertainty to your estimates that you’re putting into your model, because we’re not sure if these results are driven by choices of elicitation or true beliefs that people had. It still gives you a starting point where you can make very explicit assumptions about how much uncertainty you want to inject into your models.

How can we improve people’s forecasts? [01:18:59]

Luisa Rodriguez: Going off that, actually, I’m curious if there’s a way to help make that more concrete? So you’ve had some preliminary success getting people to make better low-probability forecasts by changing the elicitation method. Can you give an example of that? Either by you explaining the different ways you could elicit something and how that changes people’s reasoning, or by helping me try to make a forecast about some low-probability event?

Ezra Karger: Yeah, definitely. And I want to caveat that we don’t yet know what works well, so we’re still exploring this in a big experiment. But I’m excited to think about how providing people with reference classes, for example, can improve their forecasts.

So maybe just to stay on the theme — which is a little bit negative, but maybe useful in comparison to what we’re talking about — of death, let’s think about the probability that a single person in, let’s say, the US, will die from various sources. Maybe you can start by just giving me your belief about the chance that someone dies from a bee sting or a bee-type sting in the US.

We can all agree that, ex ante, we think that’s a somewhat low-probability event of happening. But before I looked at the numbers, I didn’t really have any idea about what that might be.

Luisa Rodriguez: Totally, yes. Should I try to give an off-the-cuff number, or should I try to do a little bit of reasoning?

Ezra Karger: Totally up to you. I think this actually demonstrates well what people do. Some people are going to give an off-the-cuff answer to a low-probability forecasting question; some people are going to think about it for a while. And we actually don’t even know if off-the-cuff is better than thinking hard about it or not.

Luisa Rodriguez: Right. That’s really funny. OK, if I were to give an off-the-cuff answer… And this is like, dies ever in their life?

Ezra Karger: Yeah. So let’s say there’s data on, in a specific year, how people die. As a rough approximation, we can think about these as if we take a given person in the US, what is the chance that they die from a variety of sources, and we’ll ignore the fact that that changes over time.

Luisa Rodriguez: Yep. OK, I’m gonna be horrible at this. So how should I give my answer? As a percentage of all deaths?

Ezra Karger: Yeah, let’s go with “one in x,” where x is some larger number. So do you think there’s like, I know we’re not going to get a precise number here, but, like, a 1-in-100, 1-in-1,000, 1-in-10,000, or 1-in-100,000 chance that a given person in the United States would die because of a wasp or bee sting? Which I think are clumped together in the data.

Luisa Rodriguez: I would guess it was lower than one in a million.

Ezra Karger: Great. So then I think what can be nice is we can give people who are producing these forecasts some other types of deaths, some other data on deaths that aren’t the thing that we’re asking about. We can say things like, “The probability that someone dies because they get hit by lightning is on the order of one in 300,000. Does that change your forecast?” Do you think the odds of someone dying from a bee sting are higher or lower than that? That might cause you to update, or it might cause you to be happy with your forecast. So does that piece of information change your belief at all?

Luisa Rodriguez: Yes. Yes, it does. I suspect that, at this time in history, we mostly don’t die from lightning strikes anymore. And I would guess that allergies to bees and wasps are going to end up slightly more common. I don’t know that much about how many stings are deadly, but I still think it’s probably going to be more likely that you die of a sting now than of a lightning strike. So I’d put it more at, like, one in 70,000.

Ezra Karger: Awesome. And then we could go a bit farther, and we could say, what’s the chance of someone dying from a bicycle accident? Then you could say, “I think that’s more likely than dying from a bee sting.”

Luisa Rodriguez: Yep.

Ezra Karger: But you basically got the right answer. I think the answer is around one in 50,000.

Luisa Rodriguez: Nice!

Ezra Karger: So what we can see is you started off at one in a million, but then giving you a piece of information and just having you calibrate your belief about a low-probability event to that piece of information may have improved your forecast. In this case, it did. We don’t know if, on average, it would.

Luisa Rodriguez: Yep.

Ezra Karger: Now, this is obviously a different situation than when we’re thinking about existential risks, but I think there are a lot of similarities. So one thing we’re going to do in followup experiments is try to understand whether giving people reference classes causes them to be anchored to worse outcomes, or causes them to produce more accurate forecasts in these low-probability domains.

And my prior, my belief is here, is that most people don’t really know how to think about probabilities that are one in a million, or one in 10,000, or 1%. Things that happen in a normal day are not usually that rare, and so they’re just calibrated to a set of events that happen in their lives that are pretty common. Based on that, I think giving them some access to reference classes across the probability space will improve their ability to forecast in low-probability domains. But that’s a type of research that I think we just don’t know the answer to that I’m very excited to dig into.

Luisa Rodriguez: Cool. Yeah. Just that tiny experiment alone did make me a little bit more optimistic that it’ll yield some good stuff.

Ezra Karger: Amazing.

Luisa Rodriguez: Yeah, I guess before we move on from that though, I have the intuition that there are important differences between risks that we know the probabilities of — like lightning strikes killing people — and risks that are kind of unresolvable — like the probability that nuclear war kills 10% of the population by 2100. How important do you think those differences are?

Ezra Karger: I think there’s an important parallel, and then also an important difference.

Before we started this example, you didn’t know what the risk of being killed by lightning was, and you also didn’t know the risk that AI would cause human extinction by 2100. So at some level, I think you had similar levels of uncertainty about both of these pieces of information. And without looking up what the odds were that you would die because of a lightning strike, I think that type of similarity is really useful and really parallel.

But in another sense, we know that there is a base rate for lightning strikes. We know that we can look at the data from last year and learn something about the data from this year. And that isn’t the case when you think about AI causing human extinction, so I do think there’s a difference there. I think making that leap from information cases where we do have base rates to cases where we don’t is a leap that’s very hard to understand and very hard to puzzle over.

So I do think there are important differences. I think one thing that makes me somewhat optimistic is there are things in between “What is the likelihood that lightning causes a death this year?” and “What is the probability that AI causes human extinction?” As an example: “What is the probability that in 50 years a random person will be killed by lightning?” Now we’re adding some uncertainty to this, right? We don’t know how technological change will affect the probability that people are hit by a lightning strike, but it doesn’t get us to a point where we don’t have base rates. We have a base rate; it’s just less relevant.

So I think by scaffolding up to this level of uncertainty that we have about some of these very hard-to-understand risks, we can learn a lot about how hard or easy it is to forecast in those spaces. So I’m still optimistic. I think we can learn a lot from these places where we have base rates.

Incentives and recruitment [01:26:30]

Luisa Rodriguez: OK, that all makes sense. Circling back to a few more niche details of the setup of the tournament, I’m interested in understanding how exactly you got relevant people spending so much time on making all these forecasts?

Ezra Karger: Yeah. Let me talk about two things. One would be incentives, and the one would be how we got the experts.

On the incentive front, we wanted to make sure that people had an incentive to tell us the truth and also an incentive to be accurate. And this creates a lot of problems when you’re thinking about questions over long time horizons. So, to incentivise accuracy, we held money in escrow from our grant to give people bonuses in 2024 and 2030 based on how accurate their short-run forecasts were. And we used what’s called a proper scoring rule. This is a rule that incentivises accurate forecasts under some conditions.

For longer-run forecasts, we were a bit stuck, because asking forecasts about what will happen in 2100, it’s very hard to get people to care about whether you give them $10 for being accurate or not. So we wanted to think about what’s called an intersubjective metric. And an intersubjective metric is one where you’re not predicting what will actually happen, but you’re predicting what another group of people or another person will say. And if you structure these intersubjective metrics well, they can also elicit true responses, according to a large academic literature.

And so what we did is, on the long-run questions about existential risk, we asked forecasters for their own unincentivised belief, and then we also asked them to predict what superforecasters would say and what other experts in the domain would say. This gave us several comparison points for thinking about their beliefs: one was unincentivised, and two were incentivised to be close to a group whose forecast they couldn’t see. So they had to think about what those groups would say. In the report, we mainly focus on the unincentivised beliefs, but we do think a little bit about those incentivised beliefs.

Luisa Rodriguez: OK. So that part feels a little surprising to me, but potentially important and interesting. The idea is that there’s an academic literature that says if you ask a group to make predictions about what another group will predict, those predictions are close enough to the likely true outcome that you can just reward them on their predictions about other people’s predictions, and that has the right incentive? Or am I misunderstanding?

Ezra Karger: Yes, but with a very important caveat: it depends on what group you’re predicting. So if I asked you to predict what a random person who you see in New York City thinks about a topic, and you’re thinking about a very complex topic, you’re not going to be incentivised to report the truth. You’re going to be incentivised to report 50% maybe, because that’s what you expect some random person who we run into to say about a specific forecast.

But we didn’t ask people to forecast what a random person would say. We asked people to forecast what a set of accurate forecasters would say, and then also what a set of experts in a domain would say. So by doing that, we are incentivising you to think about what other people would say who have these characteristics that make for interesting forecasts.

But to your point, this is why we also wanted to ask for people’s unincentivised forecasts. What’s been done so far when eliciting forecasts about existential risk is incentivise: people just ask experts what they think. So we did that as well, and those are the forecasts we’ve largely focused on in this project. But we think it’s useful to also have this data where we directly ask people to forecast what another group thinks — both because of this academic literature on how that creates incentives to tell the truth in some cases, and also because we can then see who is best at understanding what other groups think, which is in and of itself a measure of accuracy that we can look at.

Luisa Rodriguez: Right. OK, cool. I think that makes sense. It feels a little weird to me still. It feels like you’re then making an assumption about how good the superforecasters and experts will be at making predictions in aggregate. But is it just kind of empirically true that if you pick a group you think is likely to make sensible-ish predictions in aggregate, and you ask someone to make predictions about that group’s aggregate predictions, you’ll just get closer to the actual correct outcome than you would otherwise?

Ezra Karger: So we have a paper where we do this. So we took 1,500 people from the public and we put them into three conditions. There was an unincentivised condition, so 500 of them just told us what they thought about a set of short-run resolvable forecasting questions. Five hundred of them were asked to predict forecasts and given a proper scoring rule. So they were told, “We will score you based on how accurate you are relative to the truth, and we’ll pay you an incentive if you’re very accurate.” And then 500 people were put into what we call the “reciprocal scoring condition,” where they were forecasting what a set of superforecasters would say.

And what we saw is that the unincentivised group did relatively worse, and that both the incentivised group using a proper scoring rule and the incentivised group using this comparison to superforecasters did better than that unincentivised group, and were pretty equivalent. So we do have some evidence that getting people to forecast what another group thinks can improve accuracy relative to unincentivised forecasts.

But I think your concern here is a good one, and I think this is still a domain we want to do a lot more research in. So in the report, we largely focus on the unincentivised forecasts, but we wanted to gather these incentivised forecasts as well so that we could dig in and understand who was more accurate on this incentivised task.

Luisa Rodriguez: OK, and then was there also something you wanted to mention about how you got the experts?

Ezra Karger: An “expert”: that’s a very amorphous concept. We don’t really know what an expert is. We don’t have a good definition of it. So what we did to recruit experts is we reached out to people who worked in academic labs, in industry, and think tanks who had experience thinking about artificial intelligence, climate change, biorisk, or nuclear risk, and we asked them to participate in this tournament. We also reached out to our professional networks and said, “Can you please forward this around to people you know who are experts in this space?”

What we ended up with is 500 people who applied to be in this project. And we then had research assistants go through and try to understand whether they were experts or not in a specific topic. And we wanted to get experts for each of the topics in the tournament, and we wanted to make sure they were weighted roughly in proportion to the number of questions we would ask about each topic. So we ended up with more experts on AI than experts on, let’s say, climate change, just because we had more questions in the tournament about artificial intelligence.

But this is in many ways a convenience sample. It’s a sample of people who were interested in doing a four-month project where they gave us lots of forecasts about these topics. So I want to make sure to say upfront that what this group is representative of is a very complicated question. We’re not claiming to have forecasts from a random set of experts in the US or a random set of experts in the world. We have experts who wanted to participate, so we want everyone to keep that in mind when thinking about these results.

Luisa Rodriguez: That does seem important.

Criticisms of the tournament [01:33:51]

Luisa Rodriguez: For now, let’s talk about some of the criticisms that the report has gotten more broadly.

One worry people have had is that the predictions are generally really correlated across topic areas. For example, people who thought that biorisks were more likely also thought that nuclear war and AI risks and climate change risks are more likely, even though those things aren’t obviously correlated — at least in the really core parts of the risks. Maybe they’re correlated in the sense that people’s involvement in making sure nuclear war doesn’t happen is also a little bit related to making sure certain kinds of biological risks don’t happen. But broadly, they’re pretty uncorrelated.

So how worried should we be about the results being coloured a lot by whether people have optimistic or pessimistic views about the world or humanity’s prospects or something?

Ezra Karger: I don’t think we should be worried about that, but I do think it’s one of my favourite or most interesting empirical results or findings from the report.

What I would say is what the report shows is that your concerns about the world being fragile from many of these sources are correlated. So there’s this underlying belief about the resilience or fragility of the world in response to risk — whether that’s nuclear risk, biorisks, risks from AI — that are very important to someone’s own understanding of how they think about the long-term future of humanity in this context of existential risks.

So I don’t think it’s a bad thing that we found that some people think that risks from all of these sources are high, and some people found that risks from all of these sources are low. It might just say that people believe that the world is correlated in ways that mean that each of these risks are not independent.

Luisa Rodriguez: Yeah, right. And that get at real things, and not just like some people are inherently deeply pessimistic and some people are inherently deeply optimistic, and that maybe a little bit of that is colouring some of this. But that isn’t necessarily the explanation for why these things ended up clustered like this.

Ezra Karger: Yeah, it’s not necessarily the explanation, but it could be. Part of this might be that some people are way too optimistic about the world and some people are way too pessimistic about the world. I think it’s impossible to differentiate that view from the idea that the world’s risks are correlated. So that’s something we want to dig into in future work, but I don’t view that really as a criticism of this exercise. I think it’s a finding that needs to be explored more.

Another reason you may not be that concerned that these risks are correlated is we weren’t trying to independently look at the risks from each of these sources. So if you believe that the key change happening in the next 10 years in terms of technological progress relates to artificial intelligence, but that improvements in artificial intelligence’s capabilities will lead to more biorisk or more nuclear risk because of how those systems interact, then you may have these correlated beliefs about risk, even though it’s all driven by one underlying source.

Luisa Rodriguez: Yeah. OK, that makes sense. Moving on to another one. Another criticism that came up from people who were participating in the tournament is that they felt there was, at least in some cases, low engagement from other participants, and that those participants who were engaging weren’t being that thoughtful or open-minded. Which is a thing you’ve already alluded to a little bit, but how strong do you think the evidence is for that? And do you think it was a big enough deal that it should shape how we think about the results?

Ezra Karger: I think this is always a great criticism of research that involves asking experts for their opinions, because experts are often very busy, and having them give answers to these complex questions over many hours is not something that they often have time to do. So when you look at a paper or a report that says, “We asked experts or superforecasters for their detailed opinions about a topic,” you should make sure you agree that they actually went in depth on these topics. You should make sure they’re not just giving off-the-cuff answers in ways that are miscalibrated or that you wouldn’t necessarily trust.

But I also have a criticism of this criticism, which is that you can always apply it to any study. You can always say that people didn’t spend enough time thinking about a topic, or people weren’t engaging in a high-quality way. So my view is that what we did here involved the most engagement that has been gotten from experts and superforecasters, relative to other studies about these topics. You might think that it would have been better if they’d engaged for twice as many hours or three times as many hours. And I think it’s worth exploring whether spending more time on a topic changes people’s beliefs about the topic, or spending more time on a topic leads to higher-quality engagement.

I would say that I was following along with the conversations on this online platform. I was reading in admin mode what people were saying in back and forths. And as with any study like this, the quality of conversation varied. So you had some conversations which I thought were incredibly high quality, and you had some conversations which felt like two 10-year-olds arguing about something they didn’t really understand. And what fraction of conversations were A versus B is very hard to figure out.

We tried to do some quantitative analysis of rationales, but what I’ll say is that when people are arguing in the real world, there also tends to be the range of argument styles that you get depending on people’s moods, how their days are going, whether they know a lot about a topic. So it wasn’t surprising to me that the quality of engagement varied a lot. And when I looked at the results that pointed me towards trusting one group over another, we really saw high-quality engagement from all of the groups in the study, and we saw evidence of low-quality engagement from some participants at some times.

Luisa Rodriguez: Yep. How about this question of the high dropout rates among participants? How big of a problem was that?

Ezra Karger: I think that was a problem. I would say that was a big problem for thinking about changes in beliefs from the initial parts of the tournament to the last part of the tournament. Because if, over time, you have differential attrition — many more experts dropped out during the study than the superforecasters — you might think that we’re underestimating how much updating there would have been if we had all of the participants responding during every stage of the tournament.

But in terms of analysing the overall numbers, I’m not too worried about that. People gave independent forecasts. The people who stuck around didn’t change their minds much. So the fact that people left, and I think the people who left tended to be people who wouldn’t have updated that much anyways, means that when I look at the overall results at the end of this process, I think those forecasts from the groups are very representative of the forecasts we would have gotten if we hadn’t seen attrition.

Now, that’s not to say that we don’t learn anything from the attrition. I think experts didn’t have time to engage with all the parts of this process in as much detail as we would have liked. So in a followup we’re doing, we’re running a big study on nuclear risk now where we’re surveying 100 experts, we’re asking them for their feelings about many questions related to nuclear risk. And one thing we learned from this differential attrition is, let’s not bring people together for four months and try to get them to discuss things in an online platform. Let’s give them a survey, and let’s try to understand their beliefs, and then maybe we’ll give them another survey later. But just doing that means we can avoid the attrition process.

And because we saw in the XPT that people weren’t really updating from the start to the finish of this process, we think that those initial forecasts are very valuable to gather, and so we’re going to do more of that going forward.

Luisa Rodriguez: Cool. Yeah, that makes sense to me. It sounds like you basically think some of the criticisms are valid, but that this is still basically at least among probably the best source of information we have on these kinds of risks and their probabilities. Plus you’re like, “And we learned from them. And now when we do these kinds of studies again, we’ll do them slightly differently.” And that is a good and reasonable thing to happen.

Ezra Karger: Exactly. And if you think of this as a set of reference forecasts, I think this is the best set of reference forecasts that people have gathered about this topic from people who disagree. I think the criticisms are great, and I love seeing why people think that these results should be different in various ways.

I have my own criticisms that I think are important to think about as well. For example: did we ask these questions in ways that we’re going to get high-quality forecasts? We talked a little bit about low-probability forecasting. When you’re forecasting in low-probability domains, what kind of scale should you use? Should you give people reference classes? Should you try to get them to update after giving an initial forecast in a specific way?

So there are all of these questions about how to ask people for their beliefs and how to elicit high-quality forecasts that this study largely papered over. We went with the standard approach, and I’d like to improve that going forward. What I’m really excited to do though, is once we have new methods we want to test, we can go back to these participants and to other participants and see, did the results change?

Luisa Rodriguez: So cool. Do you have any other critiques that you yourself are like, “Man, when we do this again, if we do this again, I’d really want to change X.”

Ezra Karger: I think I’m moving more towards thinking that we should have asked fewer questions. Most people didn’t answer all of the questions. We didn’t require it. We asked people to do a random subset of the questions, as well as answering all of the questions about the key existential risks. I think focusing people on either a topic or a smaller set of general questions would have led to better engagement and discussion.

I also think structuring the debate a little bit better would have helped arguments get to the truth faster. You can imagine, knowing that two people are going to disagree, putting them on a Zoom call or putting them on an online platform: how should you get them to engage? Should you just leave it open and have them talk to each other? That could be great, depending on the people. It could also lead to terrible conversation. So I do think there’s value in digging in, and trying to figure out how people should be having structured disagreements when they disagree strongly about quantitative values.

Luisa Rodriguez: Yeah. When I think about how it would go for me to talk to a random person about a thing that I believed and that I knew we disagreed about, I’d like to think that I’d try my best to be reasonable and open-minded and to explain my reasoning well. But would it go better if someone helped facilitate that, and helped us figure out exactly why we disagreed, and pointed out when we were failing to be open-minded? Yes, 100%. When I imagined going on a website, and trying to type up my arguments and then change my mind, I’m like, yeah, that was not gonna happen.

Ezra Karger: Exactly. And I think one other thing I would change is the way we recruited experts was somewhat ad hoc. I think a very good criticism of this is, who were your experts? We had experts who were more associated with the effective altruism community. We had experts who are maybe younger on average than what people generally think of as an expert. And I think there are reasons to get forecasts from the set of people we got forecasts from, but I would also love to have a methods section of the report where I could say that we started with 2,000 people selected in a very specific way, and we sampled from them, and then we got them to participate, and then we talked about how that represented a bigger group that we care a lot about.

And I think there’s other work eliciting expert views. You can think about Katja Grace’s work at AI Impacts, where they start with a sample of a bunch of computer science authors, and then they try to get them to answer questions about AI. That means that when you’re looking at the answers, you can think of it as a selected subsample of a group you might care about. And that’s something that we can’t do in this project because we really did have a convenience sample.

That leads to a lot of related problems, like what about selective attrition? And what about lack of effort? A lot of the things that we talked about today apply to work like that as well. But I think starting with a baseline set of people who are well defined can help give readers an understanding of who we’re talking about when we talk about groups like “experts.”

Luisa Rodriguez: Makes total sense.

AI adversarial collaboration [01:46:20]

Luisa Rodriguez: OK, let’s turn to another topic. You’ve been working on a really interesting project, AI adversarial collaboration, which basically aimed to get forecasters who were worried about AI risks and forecasters who are not worried about AI risks to figure out what their disagreements were, and whether there was anything that could change their minds. Do you want to, again, say what the motivation for this was?

Ezra Karger: Definitely. And let me start by saying that this project was really led and spearheaded by Josh Rosenberg, one of my collaborators. I worked with him on it, but I want to make sure he gets a lot of credit for the fascinating results we’re going to talk about today.

Luisa Rodriguez: Cool. Thank you, Josh.

Ezra Karger: The motivation behind this project was that the XPT showed large differences in these beliefs about AI risk. So when we compared the beliefs of people who are AI experts to super forecasters who have this track record of forecasting well in these short-run geopolitical questions, we saw these large differences.

And one of the responses to the XPT was to say that people were thinking about all of these domains; they weren’t engaging that much with any one specific topic. So is it really the case that this disagreement would hold up if you have these groups talk more and engage more with each other’s arguments? So, while the XPT was focused on analysing these patterns across the risk areas, we wanted to take one of these risk areas and go really deeply into it, to dive into the complex arguments underlying why the people who are more concerned about AI risk were thinking the way they were thinking, and why the people who were less concerned about AI risk were also thinking the way that they were thinking.

We were trying to answer, I would say, two general followup questions: What happens if we take thoughtful people — some of whom are sceptical about risks from AI, and some of whom are concerned about risks from AI — and we let them focus on that disagreement? And can we identify shorter-run indicators — forecasting questions where we will actually know the answer to these questions in the short run, say, five years — which can quantitatively explain disagreements between these two groups, these AI risk sceptics and this group of AI risk concerned people?

Luisa Rodriguez: Awesome. OK, so that’s the motivation. And it does sound, again, just like such a good project to exist in the world. Do you want to just super briefly say a little bit more about what the setup was?

Ezra Karger: Yeah. Because we wanted to do a deeper dive into this topic, we wanted to start with a much smaller group of people than we had in the XPT. So we drew together 11 people from each of these sides: 11 people who are sceptical about AI risks, and 11 people who are concerned about AI risks.

To gather these people together, we took sceptical folks from the XPT — so these are people who gave very low forecasts of AI extinction risk. Nine of them were superforecasters, and two of them were not superforecasters; they were domain experts from the XPT. And for the concerned side of things, we wanted to partially address these criticisms that the experts in the XPT maybe weren’t the best representatives of concerns about AI risk. So we asked staff members at Open Philanthropy — which was the organisation that funded this project — and people within the broader effective altruism community, who spend a lot of time thinking about AI risk, to recommend people to us who could thoughtfully debate these issues from the perspective of an AI-concerned person.

The setup was, we asked these participants to work together for around eight weeks on an online platform, and to really dig into the arguments that each group had for why AI risk was high or low. Just thinking in terms of how much time these groups spent on the project, the median sceptic spent 80 hours on this project, and the median concerned person spent about 30 hours on this project.

And the participants were reading background information that we gave them, they were writing down a series of forecasts specifically about AI risk, they were engaging in these online discussions, and also structured video calls where we would bring in experts and also have them talk to each other on Zoom.

So we asked them to forecast both on these longer-run questions about risk, and also on dozens of shorter-run questions — where we asked them to tell us what they thought the probability of these shorter-run questions resolving positively was — and then also, if each short-run question resolved positively, how their concerns about risk would change. That told us kind of how important these cruxes were.

Luisa Rodriguez: Right. Cool. Before we talk more about that, did you get some kind of baseline measure of how different these two groups were in their beliefs about AI?

Ezra Karger: Yes. Because we focused on people who disagreed strongly, at the beginning of the project, the median sceptic gave a 0.1% chance of existential catastrophe due to AI by 2100, and the median concerned participant forecasted a 25% chance. So if we think about what this means, in the XPT, we saw these major differences in belief about AI extinction risk by 2100: I think it was 6% for AI experts and 1% for superforecasters. Here we’ve accentuated that disagreement: we’ve brought together two groups of people, 22 people in total, where the concerned people are at 25% and the sceptical people are at 0.1%. So that’s a 250 times difference in beliefs about risk.

Hypotheses about stark differences in views of AI risk [01:51:41]

Luisa Rodriguez: Yeah. So really wildly different views. Then I think that you had four overarching hypotheses for why these two groups had such different views on AI risks. Can you talk me through each of them?

Ezra Karger: Definitely. We developed these hypotheses partially as a result of the X-risk Persuasion Tournament. The four hypotheses were the following.

The first was that disagreements about AI risk persist because there’s a lack of engagement among participants. So, we have low-quality participants in these tournaments; the groups don’t really understand each other’s arguments; just the kind of whole thing was pretty blah.

The second hypothesis was that disagreements about AI risk are explained by different short-term expectations about what will happen in the world. So if hypothesis two is right, then we can hopefully find really good cruxes for why these groups disagree, and really good cruxes that will cause each group to update.

Luisa Rodriguez: Right. Just to get super concrete, number one is like, these groups are kind of talking past each other; they don’t fully understand each other’s arguments. Number two is more like — if I just conjure up something randomly — they have really different views on whether, in the next year, GPT-5 is going to be wildly different from GPT-4. If one group thought there was only going to be a tiny improvement and there was actually a big improvement, that might be a big update for them about how quickly things are changing.

Ezra Karger: Yeah. And that might then cause them to update on risk forecasts. They might then get more concerned about risk because AI was progressing faster than they expected. Exactly.

Luisa Rodriguez: Right. Yeah. So the key thing there is that there are short-term expectations that could cause them to update their beliefs, that are kind of knowable in the next year. OK, so those are the first two. What were three and four?

Ezra Karger: Great. The third hypothesis was that disagreements about AI risk are not explained necessarily by these short-run disagreements, but there are different longer-run expectations. This may be more of a pessimistic hypothesis when it comes to understanding long-run risk, because it might say that we won’t actually know who is right, because in the short run, we can’t really resolve who’s correct, and no one’s going to update that much.

Luisa Rodriguez: OK. So that one would be something like, it doesn’t actually matter how different GPT-5 is from GPT-4. What matters is: over the next several decades, is AI going to be integrated into the economy in a certain way? Or like, are we going to fundamentally, at a governmental level across the world, implement certain policies that keep us safe or not safe from AI? Or fundamentally, is AI actually a thing that can be made to be really unsafe or to be made really safe? And those are the kinds of disagreements that you just won’t know until you basically know the outcome of what happened.

Ezra Karger: Exactly. And another option would be that people do disagree about the short run, but those short-run disagreements aren’t related to long-run disagreements about risk. So it could be the case that even though they disagree about progress on GPT-4 versus GPT-5 versus GPT-6, it doesn’t matter; they’re not going to update on their long-run beliefs about risk.

And then the last hypothesis, the fourth hypothesis, was that these groups just have fundamental worldview disagreements that go beyond the discussions about AI. And this gets back to maybe a result from the XPT, where we saw that beliefs about risk were correlated. You might think that this is just because of some underlying differences of belief about how fragile or resilient the world is. It’s not AI-specific; it’s not about beliefs about AI capabilities; it’s not about risks for misalignment — it’s about a belief that, like, regulatory responses are generally good or bad at what they’re doing.

Luisa Rodriguez: Yeah. What are some other examples we can think of? Like, humanity is bad at resolving global commons problems, or…

Ezra Karger: Yeah. Coordination is hard, and so humanity is bad at coordinating on complex issues. Or regulation won’t necessarily have the effects people expect. Or individuals can have big effects on the world or not. These are fundamental, maybe, differences of opinion that extend beyond artificial intelligence, and might just be a function of fundamental worldview disagreements.

Luisa Rodriguez: Right. And they might have answers, but we already — in contexts where we’ve seen things play out — just have massive disagreements about them, despite the fact that we’ve seen some things empirically that would inform our views on them.

Ezra Karger: Exactly. One more example on that. One of my favourite examples of what might be a worldview disagreement that I’m hoping to dig into some more in followup work is: Do you think the world was close to a nuclear catastrophe or not during the Cold War?

Luisa Rodriguez: Right, right. That’s a great example.

Ezra Karger: It’s possible that the people who are very concerned about AI also had beliefs that the world was closer to disaster during the Cold War, and the people who are not concerned about AI think that it was actually not going to happen — like, we didn’t get close to disaster in the Cold War. And that might just reflect some fundamental beliefs about the fragility of the world that tell us something about AI risk, but aren’t related to AI specifically.

Luisa Rodriguez: Nice. Cool. OK, so those are the four hypotheses. These are such interesting hypotheses, and the fact that we might learn anything about them at all through this study feels really, really exciting to me. Because it does just feel like, when I’m talking to people about these kinds of problems, the fact that it’s really unclear whether we’re disagreeing about empirical things or just these weird, fuzzy worldview things — and maybe we’ll never agree, because our disagreements about worldviews are just kind of unresolvable — that all just feels like it makes it really, really hard to understand where these disagreements come from, and whether there are productive ways forward. So yeah, I’m just excited that you did this.

OK, so which of those hypotheses ended up seeming right?

Ezra Karger: So I think hypotheses one and two did not turn out to be right, and I think hypotheses three and four have significant evidence behind them. And so I can maybe go through the evidence. That may be less exciting, because it would be great if hypothesis one or two had been right. But I was really excited to be able to differentiate these hypotheses, and figure out which ones had more evidence behind them.

Luisa Rodriguez: Totally. Yes. I do think that that is an exciting result, even if it’s a shame that it means that we probably won’t be able to resolve disagreements that easily.

Ezra Karger: Agreed. So, to talk about hypothesis one for a second: this was the idea that these disagreements about risk persisted because there wasn’t that much engagement among participants, or people didn’t disagree well. I think we can reject this hypothesis, but readers may disagree. This is very much a determination you should make after seeing how the disagreements went in our long descriptions of the arguments that people had. I think participants spent a lot of time understanding each other’s arguments, and people largely understood each other’s arguments, and engagement was pretty high quality.

There’s a criticism that was levelled at the XPT in a very interesting way, which is that these people aren’t engaging in a high-quality way. And you could just bring that criticism to this project as well, and say that people who were concerned or not concerned about AI risk weren’t really engaging in a way that was useful.

I think that criticism always applies to research projects like this, but I want to know what the limiting factor is. People in this project spent maybe 50 to 100 hours thinking about these topics. Is it the case that you think if they had spent 1,000 hours, they would have agreed? I don’t think there’s any evidence of that. I think they were really understanding each other’s arguments by the end of this project, and we saw very little convergence.

Luisa Rodriguez: Interesting. OK, so you saw very little convergence in that these two groups didn’t move that much toward each other at the end, which suggests that it’s not that they weren’t engaging. What was the evidence against hypothesis two?

Ezra Karger: Hypothesis two was the one I was saddest not to find strong evidence for. This was: can we find short-term disagreements or short-term differences in expectations that explain these long-run disagreements about AI? Much of this project involved giving these forecasters short-run forecasts to do and asking them to tell us how they would update if those short-term cruxes resolved positively or negatively.

And what we saw is that of the maybe 25-percentage-point gap in those initial beliefs, only about one percentage point of that was closed in expectation by the best of our short-term cruxes.

Luisa Rodriguez: Wow.

Ezra Karger: So what that means is, even if the sceptics and the concerned people had the best evidence from a specific question that they expected to have by 2030, they wouldn’t change their minds that much, and they wouldn’t converge that much.

Now, it is possible that we could just have done a much better job of developing cruxes — that we could have found some short-run crux that would have caused convergence, that would have caused tremendous updating by either of these groups. And I do think that there’s still a place to do better research there, and I’m excited to see what people come up with when it comes to developing better cruxes.

But we gave people dozens of cruxes that tried to span the space of what these concerned people and the sceptical people thought would update them, and I don’t think we saw huge evidence that by 2030, we’ll know information that will cause these groups to update a lot on their beliefs about risk by 2100.

Now, maybe one caveat to that: the way we dug into these cruxes, it’s very easy for us to figure out how one crux is going to affect beliefs, but it’s very hard to figure out how a combination of cruxes can affect beliefs. So it is still possible that enough will happen by 2030 on a variety of dimensions to cause one or both of these groups to update significantly. That’s not something that this project could have really dug into.

And it’s also possible that people are wrong about how they’ll update, and that actually we’ll get to 2030, a couple of these cruxes will have resolved, and then suddenly we’ll see major swings in beliefs.

Luisa Rodriguez: Yeah. OK. So the reason for the first thing, which is that we can’t really know yet from this project alone whether a bunch of these cruxes updating, maybe systematically in one way — either in favour of AI moving really quickly, or in favour of AI being really slow and limited and not progressing as quickly as people expect — maybe if a bunch of those updates happen in that kind of systematic way, people will change their beliefs.

But you didn’t ask, “How would you change your beliefs if X and Y and Z and A and B and C cruxes all update in a specific direction?” You were like, “How much would this specific question change your beliefs?” Which does seem like a much more tractable question to answer for now.

Ezra Karger: Yeah. The problem with taking 30 questions and asking how people will update on any possible combination of those 30 questions is you end up with this exponential space of possibilities that people have to think about. So we did something much more restricted here. But I am excited to explore whether we can tractably get at some of these questions about, “What if the world changes along a lot of these dimensions at once?”

Luisa Rodriguez: Yeah, yeah. And just thinking about it a little bit more, it could go either way. It could be the case that if all of these things update systematically in one direction, that’s just huge news for someone, and they’re like, “Whoa, OK, my underlying beliefs clearly were wrong. It turns out progress isn’t going to be nearly as capped as I expected. And so I should expect AI to be much more impactful than I would have guessed.”

Or it could be that you’re like, “Well, I think all of these things were correlated and have to do with one specific belief of mine, something about AI progress being fast or not. But the fact that they all ended up pointing in the same direction, which is toward fast, is only one part of my belief, and I still have all the other parts of my belief that are not about that thing. And so maybe it only moves me a little bit.”

Which makes it frustrating, but also interesting that we don’t know exactly what will happen when we get lots of resolutions to these forecasts.

Ezra Karger: Exactly. And I will say that I think it’s possible that one or both of these groups are just wrong about their forecasts of these short-run cruxes, and their beliefs about how they’ll update. And when we think about the ways we can measure the quality of a crux — these question quality metrics that we describe more in the paper — it is the case that they rely on someone reporting how they will update, and then using that information to figure out the value of information to the forecaster themselves, and how they will both update themselves and then also converge or diverge from someone else.

So another option is maybe we just should assume that people are biased when it comes to their beliefs about these short-run cruxes. And if that’s the case, some of these short-run cruxes might cause bigger updates in expectation than we expect.

Luisa Rodriguez: Cool. And I guess if I’m just trying to empathise really hard with what it would be like to be the people participating, it is really difficult for me to even start to think about how my beliefs would change if a particular thing happened. With how the costs of compute are declining, I think I’d be pretty bad at just thinking about that as a question in and of itself. So it doesn’t seem totally out of the question that that’s just too hard a thing for many people to do super well.

Ezra Karger: Yeah. I think it’s an open research question: Can people forecast how they themselves will update when given new information? That’s something I haven’t seen good data on.

Luisa Rodriguez: Cool. So many open research questions. I’m glad FRI exists.

So then you did find evidence for the third and fourth hypotheses, which are: disagreements are explained by long-term expectations about how AI plays out, and also just fundamental worldview disagreements. What did that evidence look like?

Ezra Karger: On hypothesis three, we found a lot of evidence that these disagreements about AI risk decreased when we looked at much longer time horizons — so when we looked at forecasts about how AI risk would change over the next maybe 1,000 years. And when we think about whether these groups fundamentally disagree that AI is risky, versus disagreeing about the time span of that risk — whether we should be worried about that in the next 80 years — I think we found substantial evidence that both groups were concerned about AI risks over much longer time horizons.

Luisa Rodriguez: That’s interesting.

Ezra Karger: Hypothesis four was this question of, are there just fundamental worldview disagreements? That’s much harder to get at with data, I would say. But we do have a lot of text and conversation that we captured in this online platform between these groups, and I would say we saw strong evidence that these two groups disagreed in the sense of their worldviews.

The sceptics felt anchored — and that’s not using the word “anchored” in a good or bad way — but they felt anchored on the assumption that the world usually changes slowly. So rapid changes in AI progress, or AI risk, or general trends associated with humanity seemed unlikely to them. And the concerned group worked from a very different starting point: they worked from the starting point of, if we look at the arrival of a species like humans, that led to the extinction of several other animal species. That happened pretty quickly. If AI progress continues, and accelerates, and has this remarkably fast change in the next 20+ years, that might have really negative effects on humanity in a very short timeframe, in a discontinuous way.

So I think we did see evidence, when we had these conversations going back and forth between these groups, that the sceptics thought the world was more continuous, and the concerned AI risk people thought that the world was more discrete.

Luisa Rodriguez: Cool. OK, so there’s a lot there.

So let’s actually dig into the evidence for hypothesis three. What were the long-term outcomes from AI that sceptics expect?

Ezra Karger: This maybe gets at a source of agreement that I didn’t expect: both the sceptics and the concerned people believe that “powerful AI systems” — and we define this as “AI systems that exceed the cognitive performance of humans in at least 95% of economically relevant domains,” so this is a big change — both groups thought that this would be developed by 2100. The sceptics thought there was a 90% chance this would occur, and the concerned group thought there was an 88% chance this would occur.

Now, that’s a lot of agreement for people who disagree so much about risk. And I think there are a few things going on there. First is that we tried to define these questions really carefully, but what does it mean for AI systems to “exceed the cognitive performance of humans in greater than 95% of economically relevant domains”? We can both agree that this is a big deal if it happens, but it’s possible that the sceptics and the concerned people disagree about the extent to which that means that AI systems have really accelerated in ability.

One other place where the AI risk sceptics and the AI risk concerned groups really seem to agree is in what would happen with AI risk over the next 1,000 years. We defined a cluster of bad outcomes related to AI, and this included AI-caused extinction of humanity. It also included cases where an AI system, either through misuse or misalignment, caused a 50% or greater drop in human population and a large drop in human wellbeing.

What we found is that the AI risk concerned group thought there was a 40% chance that something from this cluster of bad outcomes would occur in the next 1,000 years, but the AI risk sceptics thought there was a 30% chance that something from this cluster of bad outcomes would occur in the next 1,000 years.

So if we connect that to the forecasts we’ve been talking about throughout this conversation, about what will happen with AI risk by 2100, what we’ll see is that both groups are concerned about AI risk, but they have strong disagreements about the timing of that concern. People who are concerned in the short run remain concerned about the long run and get more concerned about the long run if you accumulate those probabilities. But the people who are sceptical about AI risk in the short run are still concerned if you look at a broader set of bad outcomes over a longer time horizon.

Luisa Rodriguez: That does feel really, really huge. Because it feels to me like often when I either talk to people or hear people talk about why they’re not that worried about AI risk, it sounds to me like they sometimes have beliefs like, “We will do AI safety properly,” or, “We’ll come up with the right governance structure for AI that means that people won’t be able to misuse it.”

But this just sounds like actually, that’s not the main thing going on for even the sceptical group. It sounds like the main thing is like, “No, we’re not confident things will go well; we just think it’ll take longer for them to potentially go badly” — which does actually feel really action-relevant. It feels like it would point to taking lots of the same precautions, thinking really hard about safety and misuse. Maybe one group doesn’t feel like it’s as urgent as the other, but both think that the risks are just very genuine. So that’s really cool. Also, terrible news. I just so prefer that the AI sceptics believe that AI poses no risk, and be correct.

Ezra Karger: Yes, I think there’s optimistic and pessimistic news, if you agree with the sceptics’ general framework here. The optimistic news would be they are much less concerned about the near term, so that implies that there’s a lot more time to figure out what to do with these new AI systems that both groups think will become much more capable over the next 100 years. The maybe bad side of this is that both groups think that AI progress is going to accelerate, and that we are going to have AI systems that can do a lot of things, can do a lot of tasks that humans currently do.

We actually asked a question about this. So let me talk a little bit about one more question, which is: what will happen by 2100? We’ve been focused on extinction risk, but your question gets at this idea of, will the progress in AI capabilities be good or bad? Like, if progress accelerates, will it be good for humanity or bad for humanity if it doesn’t cause human extinction?

We divided up the potential outcomes for humanity due to AI by 2100 into 11 categories. And I won’t go into all the categories, but two categories where we saw a lot of disagreement were whether powerful AI would be deployed and not cause extinction, but median human wellbeing would be high; or whether powerful AI would be developed and deployed, it wouldn’t cause human extinction, and median human wellbeing would be low.

What we saw is that the group of people who were concerned about short-run AI risk were much more confident that if humanity did not go extinct, then human wellbeing would actually be high. And the AI risk sceptical people thought that if powerful AI systems were developed and deployed and they didn’t cause human extinction, human wellbeing would be lower.

So what that tells me is there’s this really interesting relationship between your beliefs about risk in the short run and your belief about overall variance. The concerned group thinks AI might be terrible, it might cause human extinction. If it doesn’t cause human extinction, there’s a pretty high chance that things will be great, that humans will be in really good shape, be really happy, human wellbeing will be high by 2100. And the AI risk sceptical people, maybe consistent with their concerns about risk over a longer run time horizon, think that even if AI does not cause human extinction, if powerful AI systems are developed and deployed, then it is likely that human wellbeing won’t be that high. So I thought that was really interesting.

Luisa Rodriguez: That is interesting. It’s a source of disagreement that I didn’t realise would be a particularly big thing, like, is there going to be this moment? Are we in this time where we can either resolve this thing and make the future of humanity and hopefully the AI systems that might be living alongside us go really well? Or is there just going to be this drawn-out period of kind of higher risks that are not going to be this discrete moment that’s all or nothing; it’s just going to be like a bunch of changes because of AI, that might make our lives worse — in this way that is consistent with their differences in thinking that things could be discontinuous or continuous, I guess.

Ezra Karger: Exactly. And I do keep using the words continuous or discontinuous. That might be a little imprecise. Maybe what I should say is fast-moving or slow-moving, because it’s hard to figure out what we’re looking at continuous things over.

Luisa Rodriguez: Sure.

Ezra Karger: But I think that’s exactly right.

Luisa Rodriguez: Fascinating. So that’s one kind of fundamental disagreement about the long-term outcomes from AI. Were there any other disagreements in this category?

Ezra Karger: Yeah, and this goes back to the question where we divided up the space of possible outcomes by 2100 into 11 categories. I always love pointing people to appendices of things we write, so this is Appendix 5 if anyone’s interested. We also see what we’ve talked about previously, which is a belief that humans can coordinate on these big, complex problems.

The concerned group thinks there’s only a 4% chance that powerful AI will be developed, but not widely deployed because of coordinated human decision-making or other prohibitive costs to deployment, and the sceptical group thought there was a 20% chance this would happen. In other words, there’s strong disagreement not only about the likelihood of risk, or the likelihood that powerful AI would be deployed and it would have some effect on wellbeing, but also whether it would be developed but not deployed — either because of regulation, coordinated human decision-making, or these additional costs of deployment that are imposed by humans.

So this gets at hypotheses three and four: are there these underlying disagreements about the effects that humans can have on AI progress and risks?

Luisa Rodriguez: OK, yeah. That is also super interesting.

Cruxes and different worldviews [02:17:15]

Luisa Rodriguez: Coming back to this idea of cruxes: cruxes are this thing where if you had the answer to some short-term question — like how much is going to be spent on compute in the next two years — it might really change your mind about what you expect is going to happen with AI in the long term, in this case. Were there any that were just actually successful or good cruxes for people?

Ezra Karger: Yeah. We asked about around 40 cruxes. And we asked forecasters to forecast on those cruxes, and also how they would update based on whether that crux resolved positively or negatively.

Let me maybe just talk about a few that stood out when you think about what we call “value of information,” which is how much each crux would cause the forecaster themselves to update positively or negatively based on whether that crux resolves. Two of the cruxes we asked about, and these will end up being the best cruxes by value of information for each of the groups, were: Will alignment researchers change their minds about existential risk by 2030? And then: Will superforecasters change their minds about existential risk by 2030?

So I want to talk about these cruxes because they’re kind of interesting. They’re very meta.

Luisa Rodriguez: They are! They’re so meta. That’s not what I was expecting you to say at all.

Ezra Karger: So they were, in some sense, trick cruxes, because we were asking a group whether they would update if people like themselves had updated in the next five years. And I think usefully for the validity of this project, those were the best cruxes. Those were what people updated on the most in expectation.

Luisa Rodriguez: That makes sense.

Ezra Karger: Yeah. The people who are concerned about AI risk, if alignment researchers change their minds by 2030, then they will update their beliefs about risk by 2100 more in expectation. The AI risk sceptical group, which consisted primarily of superforecasters, they said that if superforecasters from the XPT changed their minds, if they forecasted at least a 5% chance of extinction due to AI by 2100 — when we ask them that question from the XPT in 2030 — they said that if that happened, they’d update the most in expectation.

And even more interestingly, this was not a top crux for the other group. In other words, the people who are concerned about AI don’t care as much if the sceptics update on AI risk by 2030, and vice versa. So I think this gets at one of these fundamental differences of belief, which is who people are deferring to, or whose beliefs people respect. And I found that really interesting.

But let me maybe talk about some of the more substantive cruxes that we asked about, besides those metacruxes. The two other cruxes that stood out for the concerned group were whether there would be a major powers war: by 2030, would at least two major superpowers declare war officially and go to war for at least one year? This maybe gets at beliefs about instability of the world system. So if that happens or doesn’t happen, it would dramatically cause the concerned group to update on AI risk. This may reflect the fact that if major powers declare war on each other, the concerned people think that this will accelerate people’s investment in AI systems and will cause increases in risk from a variety of AI-related sources.

Luisa Rodriguez: Cool. So it’s like if people had been making predictions about nuclear war, they might have put them lower until World War II started, and then they might have all increased them because they were like, now we’re going to invest a bunch in this technology.

Ezra Karger: Exactly. Or another thing you could be worried about is — and there have been some recent policy reports on this — if AI increases biorisk, then investment in AI systems might increase biorisk. And if you think that a large-scale war will lead to a Manhattan Project–style effort by major powers to improve AI systems, and that then causes increases in AI-related biorisk, then that might cause you to update on risk overall.

Luisa Rodriguez: Got it.

Ezra Karger: The last crux that I want to mention for the concerned group was this question about whether an independent body like METR, which was previously called ARC Evals, would conclude that state-of-the-art AI models have the ability to autonomously replicate, acquire resources, and evade deactivation. This is a type of concern that the AI risk concerned people are very concerned about, so if this happens, or if it doesn’t happen, it will cause relative updates for the concerned group.

Luisa Rodriguez: Makes sense.

Ezra Karger: I also want to mention that this was what would cause the concerned group to update the most. It was also, interestingly, something that if it happens, would cause the sceptical group to become more concerned about AI risk. Now, the sceptical group doesn’t think this is likely to happen. They gave something like a 1% chance that this happens. But if it happens, their concerns about risk went from 0.1% up to one percentage point.

Luisa Rodriguez: So that is actually a thing that for both of them would make them much more worried — which is interesting, because it sounds like that means they kind of agree on one of the really scary mechanisms by which AI could end up causing really bad outcomes. A big component of it is that the sceptics just think that’s very, very unlikely, and so it would move them some.

Ezra Karger: Exactly. So it didn’t have what we call “high value of information” for the sceptics, because they thought it was so unlikely to occur, and so they don’t expect to update on it by 2030 because they don’t think it will happen. But if it does happen, they will update a lot. And I thought that was fascinating.

Luisa Rodriguez: That is fascinating.

Ezra Karger: To briefly talk about the AI risk sceptic group’s highest VOI [value of information] cruxes: the first was whether those superforecasters would change their minds, which we talked about. But the second and third ones relate to, I think, a difference in worldview that’s really interesting.

The sceptics said they would update more in expectation if there was progress in lethal technologies — so if a panel of superforecasters in 2030 thought there was a very high chance that by 2100 a technology would exist that was capable of causing human extinction. So what does that mean? The sceptics think that the world is more resilient — this is my interpretation — and so they don’t think that there’s currently an easy way for a large-scale event like human extinction to occur. If they become more confident that that’s possible, then they will become more concerned about AI risk. And that technology could be AI-related, or it could be something biorisk related that then increases the interaction of bio and AI risk.

And then the third crux that was highest on this VOI scale for sceptics was this question of influencing democracies: By 2030, will AI systems have been found to have autonomously and intentionally changed or heavily influenced the results of a democratic election? This also gets at this question of instability. Sceptics are less concerned about these fast-moving, immediate risks, but they’re more concerned about AI systems having these long-run negative effects on humanity through effects on elections or other types of resilient systems that they think might be affected negatively by AI.

Luisa Rodriguez: Right. So their worldview is something like, the world is pretty resilient. It’s pretty complex, but that complexity kind of creates a robustness, because there are a lot of moving parts. And that means it’s really, really hard to kill everyone, or to radically change the way the world geopolitically works, and what people’s lives are like.

But if the thing that AI does is make it possible to kill everyone — which seems really hard to them now — or makes it possible to really undermine democracy, then they find it much more plausible that AI is going to be the thing that causes actually very significant outcomes for what people’s lives are like. That is a fascinating result.

Ezra Karger: Yeah, I found this fascinating. And I want to maybe mention two final things on these cruxes.

Luisa Rodriguez: Great.

Ezra Karger: The first is that none of these cruxes caused large changes in expectation for either group. That was one of the hypotheses we talked about. So it is the case that even though these were the “top” cruxes from the perspective of causing people to update in each group, they still only explained a small fraction of the beliefs about risk by 2100. I don’t think we have a coherent narrative about worldviews that I can just tell you and be confident about. But this does hint at future research that should be done to understand why these groups disagree.

And then lastly, we’ve talked here about the top three cruxes for each group from the perspective of those groups updating in expectation. But one thing we haven’t talked about is what will cause these groups to converge, what will cause their forecasts to converge when we have information by 2030?

And the best crux along that dimension was this crux related to METR or ARC Evals or other independent bodies: When will an independent group conclude that the state-of-the-art models have these abilities to autonomously replicate, acquire resources, and evade deactivation? This, if it happens, makes the sceptical group more concerned, and if it doesn’t happen, makes the concerned group less concerned. So in expectation, that caused the biggest convergence in beliefs by 2030.

I think that points to a really interesting avenue for developing additional benchmarks and other short-run information we could gather about these AI systems — that we’ll only be able to gather if groups invest in these tests of AI systems.

Luisa Rodriguez: Right. So evaluations by METR and others could really change people’s minds, tentatively.

Ezra Karger: Exactly.

Luisa Rodriguez: OK, so those are the cruxes. So, so interesting. Then there was also this discussion of disagreements. After discussing all of their disagreements for eight weeks, for many, many hours, it sounds like the groups didn’t converge that much, but maybe they did a little bit. Do you want to say exactly how much it was?

Ezra Karger: I would say there was very little convergence. The sceptical group moved from around 0.1% to 0.12%. That wasn’t a statistically significant change. And the concerned group’s median fell from 25% to 20%. That may seem like a lot, but I don’t think it is, because from April to May of 2023, which is when this project was run, a lot of other things were happening in the world of AI. So several participants said that their updated probabilities of extinction, their maybe more optimistic takes, were actually driven by these other developments that were going on in the world, and not updates they made during their work on this project. And also, we’re talking about a very small number of people, so that difference is maybe driven by a couple people who changed their minds, not by everyone in the group changing their minds.

Luisa Rodriguez: Right. A systematic convergence. Yeah, OK. So just really not that much convergence, which I guess isn’t shocking given that they weren’t able to identify that many empirical things that were grounding their disagreements, or at least empirical things that we know very many answers to right now. Genuinely, I just think that’s such an interesting project, and I hope you do more like it.

Ezra’s experience as a superforecaster [02:28:57]

Luisa Rodriguez: For now, I want to ask you some questions about forecasting more broadly. You’re actually a superforecaster yourself. How did you get into forecasting?

Ezra Karger: Yes, and I was one of the earlier research subjects in Phil Tetlock’s research, and we now work together, which is very fun.

Luisa Rodriguez: So fun.

Ezra Karger: After college, I participated in ACE as part of the Good Judgment Project team. This was one of the first large-scale attempts to understand the accuracy of crowdsourced forecasting. It was a government-funded project where they were exploring whether getting forecasts from the public and aggregating them in high-quality ways could improve on forecasting accuracy.

So when I was working in consulting after college, I got very excited about this, and signed up and just spent a lot of my year forecasting on random geopolitical questions. I then kept participating as a research subject in followup government forecasting projects. And around maybe 2018 or 2019, I started doing research with Phil. We were emailing back and forth and talking about different ideas, and then we started working together on some projects.

Luisa Rodriguez: Awesome. What do you think made you so good at it?

Ezra Karger: I think partly it helped that I was a news junkie, and just listened to and read a lot of news and tried to understand what was going on in the world. And I don’t think I was really good at it to start. I think I needed a lot of practice. It was definitely the case of, when I first started forecasting, I didn’t have a great sense of what a probability was. I didn’t understand whether a 5% number or a 10% number were that different. I think just doing a lot of forecasts helped me improve there.

I think also being a fan of baseball, which involves a lot of games where people win and lose and rare events happen, was probably helpful in understanding how to stay well-calibrated, and not get too upset when the Red Sox lost. That helped.

Luisa Rodriguez: OK, nice. That’s unfortunate. I am neither of those things. I’m neither a news junkie nor am I a baseball fan, so I must be doomed.

Forecasting as a research field [02:31:00]

Luisa Rodriguez: On the field of forecasting, you’re very good at forecasting itself, but you’re also doing kind of meta forecasting research. I read Phil Tetlock’s books on forecasting years ago, but since then I haven’t really kept track of what’s going on in the research field. How do you think forecasting research is going, generally speaking?

Ezra Karger: Now that I’m spending a lot of my time thinking about forecasting research, I think it’s a really exciting field. There have been academics thinking about forecasting for a very long time — for definitely decades, probably centuries. But Phil’s work maybe 10, 15 years ago largely changed how people are thinking about forecasting and forecasting research, and it opened up a whole new set of questions that I’m now really excited to pursue with Phil and other collaborators.

What I would say is, if you just look at the projects we talked about today — this X-risk Persuasion Tournament, this project on adversarial collaboration — I always think to myself, as I’m talking about these projects, that there are 20 or 30 other questions I want to answer now that I have these initial forecasts.

I can’t point to experimental evidence on how you should do forecasting in low-probability domains. So we’re working on a project right now where we try to figure that out, where we bring in thousands of people, ask them to produce forecasts in low-probability and mid-probability domains, and figure out how we can help them make better forecasts.

We talked a little bit about intersubjective metrics. How should we incentivise accurate forecasts when we’re asking questions over the very long run? And there is some really interesting academic evidence on that, but I’m excited to do more, and to try to compare these intersubjective metrics, and figure out how they can be used to produce accurate forecasts from normal people, experts, and policymakers.

I’ll also say that Phil’s work on the Good Judgment Project and earlier work on forecasting, I think it did take the first steps of trying to link forecasts to decision-making. But there are still a lot of open research questions about how people should be using forecasts, on how government agencies, experts, policymakers, philanthropic grantmakers should be relying on forecasts when making the decisions that they make on a daily basis.

And I think trying to produce high-quality forecasts is maybe the first step. There are lots of people doing that now. We’ve talked today about some work trying to produce high-quality forecasts about longer-run outcomes, and now there’s the question of what should people do with these forecasts? We could just keep asking people for their forecasts, but if you’re making a decision about whether to implement policy A or policy B, is there a way that we can take these forecasts and help you make a better decision?

So I’m really excited to do research on that question as well. I just think there are so many open questions in this space that I would love for people to be digging into more.

Luisa Rodriguez: Yeah, cool. It sounds like, maybe more than I would have guessed, there are not only open questions, but it sounds like they’re just really answerable questions. Not all of them, I’m sure, as much as you’d like, but you’re doing just many, many empirical studies that are teaching you real, concrete, robust things about how to get people to give accurate forecasts, and that’s really exciting to me.

Ezra Karger: Yeah, I think that’s exactly right. We’ve been talking a lot about who we should trust when it comes to these forecasts of existential risk, and that’s a really big, hard-to-answer question. But there are all of these smaller empirical questions — like how should we be eliciting forecasts in low-probability domains — where I think we can just spend a few months running a well-powered experiment to get some preliminary answers about that on resolvable short-run questions. And that then has implications for how we want to ask about these longer-run questions, or these longer-run questions in low-probability domains.

And we’re lucky to have some funding to run those experiments. But I think there are so many other experiments that should be run, and I can’t wait to dig into more of them.

Can large language models help or outperform human forecasters? [02:35:01]

Luisa Rodriguez: One area that you’re looking into that piqued my interest is related to large language models. You’re looking into both whether LLMs can augment human forecasting, and also whether LLMs might be able to outperform human forecasters — both of which are really interesting questions. Have you learned anything about either of these questions yet?

Ezra Karger: Yeah. On the question of whether large language models can improve human forecasts, we have a working paper that we put out, so it’s not yet peer-reviewed, which shows in an experiment that giving people access to a large language model does improve their forecasts. So that, I think, is a really interesting result.

Now, I will say this was on normal people who we gathered through one of these online platforms for recruiting subjects, so we still have a lot of questions about whether this will extend to people who are already accurate forecasters, or to experts or to policymakers. But I’m really excited, as large language models get better — and become better at retrieving information and organising it, as we get better at prompting and fine-tuning that can help the large language models act as assistants — whether we can start to improve human forecasting in noticeable, measurable ways in experiments.

I will say there’s also a risk here that I’m excited to dig into in future research, which is: once you have large language models that are being used as assistants to people in terms of forecasting, there’s also the possibility that they’ll be misleading: that they’ll give people information that will cause them to be maybe more accurate forecasters when it comes to some topics, but less accurate forecasters when it comes to others. And as people rely more and more on these systems, I’m also very excited to dig in and to try to figure out how that type of misinformation or misleading information from the large language models can affect human forecasts, even if we’re incentivising the humans to be accurate.

Luisa Rodriguez: Yeah. Cool. That does seem worrying. How about LLMs actually outperforming human forecasters? Do we know anything about that yet?

Ezra Karger: There’s a group of researchers at Jacob Steinhardt‘s lab in Berkeley who have produced a couple of papers where they have AI-based systems based on large language models do forecasting. I love these papers. There was one that was released recently, and what they show is that humans are still better forecasters than AI-based systems, but these AI systems are improving. So if you look at the accuracy of these AI systems relative to humans, from an earlier paper that they produced that I believe relied on GPT-2 as the underlying large language model, it was not doing very well relative to humans. In their more recent research, what you’ll see is that these AI-based systems are maybe even approaching human performance on some questions.

Now, on average, they’re still noticeably worse than humans, but I’m really excited to track that over time.

Luisa Rodriguez: Do you know what kinds of questions they’re relatively better at?

Ezra Karger: So it finds that maybe on these questions where people are more uncertain — so where the human forecast is more in the 50% range — large language models might be better, while on questions that are maybe in the lower-probability space, these AI-based systems are not necessarily doing as well. They also find that when there’s enough news to actually train a large language model to understand the question, you might see different results than when there isn’t any news. But that might also apply to humans.

I think there’s this completely open research question of how should we be using large language models to forecast, and how should we be using AI systems, fine-tuned, using these better and improving large language models to do forecasting in an automated way. And we’re currently working on a project, which we’ll hopefully release this summer, where we’re going to produce this dynamic benchmark of forecasting questions that updates every day or every week, where people can submit automated forecasts, and we’ll then have a leaderboard that tracks AI progress over time.

So as these systems become better, we want to explore whether we can actually measure AI progress, and whether AI systems are going to get better than humans, or maybe whether humans will remain more accurate than AI systems for a significant time period.

Luisa Rodriguez: Super cool. That’s awesome.

Is forecasting valuable in the real world? [02:39:11]

Luisa Rodriguez: Zooming out a bit on the kind of broad usefulness of forecasting, I feel like I’ve gotten the sense that at least some people kind of think forecasting isn’t actually that valuable in the real world. I have this sense that there was a lot of excitement about Phil Tetlock’s books, and then people were like, it’s actually not that practical to use forecasting. It’s like a fun game, but not useful in the real world. First, have you heard that argument? Second, do you think there’s any truth to that critique?

Ezra Karger: Yeah, I think I partially agree and partially disagree with that critique. So, first of all, I’ll say government agencies are using forecasts all the time, and people are using forecasts all the time. So I think this idea that forecasts themselves aren’t being used or aren’t being used well, I don’t think that’s right. If we look at improvements in weather forecasting, I think that’s just clearly saved lives in the past few years, relative to 100 or 200 years ago, when you saw these really costly natural disasters because people didn’t know when hurricanes were coming, for example.

Now, what we may be talking about more here is these subjective forecasts from random people. Like should we be using forecasts that people online have given about geopolitical events, or should we be using forecasts that people, or even experts on a topic, have given about events? And I do think there’s less evidence that those are useful yet.

What I would say is, Phil’s work in the Good Judgment Project, in these government-funded forecasting tournaments where we tried to understand how crowdsourced forecasts could improve accuracy relative to experts, showed that normal people could come up with forecasts that were as accurate or maybe more accurate than experts in some domains.

But they didn’t look at things like quality of explanation, for example. So if you’re a policymaker trying to make a decision, it’s very hard for you to say, “I’m going to rely on this black box number that came out of this group of people who we recruited online.” It’s much easier to say, “I have some analysts who think that these are the important mechanisms underlying a key decision I’m making.” And relying on that to make a decision I think feels more legible to people who are actually making decisions.

So I would partially agree and partially disagree with the criticism in your question. I think that government agencies are using forecasting. I’m involved in producing this short-run index of retail sales, where we just track retail sales, try to forecast how the economy is doing, and that gets used in our discussions at the Federal Reserve Bank of Chicago about how the economy is going. So that’s an example of a forecast being useful because we can very clearly state how the forecast is constructed using a model based on underlying data that we understand.

When you’re talking about these forecasts that are coming from people who aren’t also explaining their reasoning in very coherent ways or aren’t necessarily being incentivised to write detailed explanations that show that they have knowledge about a specific topic, I think we haven’t yet seen those forecasts being used.

Maybe one last point on this: after Phil’s work and other people’s work on these crowdsourced forecasts, there were attempts within the intelligence agencies in the US — and this has been documented publicly — to use forecasts, to try to use systems like the ones that Phil and others worked on. There’s this great paper by Michael Horowitz and coauthors arguing that the US intelligence community didn’t incorporate these prediction markets or these forecasts into their internal reporting, even though this research shows that those systems generated accurate predictions.

And the reasons were partially related to bureaucracy, partially related to incentives. So people didn’t really have incentives to participate to provide forecasts. If you provide a bad forecast, then maybe you look bad. If you provide a good forecast, maybe no one remembers. And also, the decision-makers were really trying to dig into underlying explanations and rationales, and they weren’t really ready to just take a number and run. And that might be a good thing, but I think that explains why some of these methods haven’t taken off in certain policy domains yet.

Luisa Rodriguez: OK. Which is why it sounds like you’re interested in figuring out how to actually, in the real world, help decision-makers use forecasts to make decisions in a way that feels good to them. Like it actually touches on reasons for things as opposed to black-boxy numbers.

Ezra Karger: Yeah. A project we’re working on in that space that’s trying to make this leap, at least in a small way, is a project about nuclear risk. What we’ve realised is that coming up with questions about nuclear risk requires a lot of technical knowledge. So we’re working with experts — with people at a think tank called the Open Nuclear Network, with other expert consultants — on constructing a series of questions about nuclear risk that people who are experts in this space think would be very valuable to get forecasts on, and think would cause them to maybe update their beliefs about nuclear risk.

And we’re then going to a panel of 100 experts that we’ve put together on nuclear risk specifically, and we’re also going to bring in some superforecasters and we’re going to ask them to forecast on these questions. And as a last step, to try to link this maybe a little bit to decision-making, we’re going to come up with a menu of policies proposed by the nuclear experts, and we’re going to ask the experts and the superforecasters to tell us what they think would happen if these policies were implemented or not.

Luisa Rodriguez: Oh wow, that’s so cool!

Ezra Karger: And we want to write a report about this. This gets to maybe my theory of change here, which is that writing these reports can be helpful, but we think just documenting what experts, superforecasters, and the public think about nuclear risk, about underlying mechanisms related to nuclear risk, and then proposing a set of policies that experts can then evaluate in a quantitative way, might improve the discussions about nuclear risk.

So I’m curious if we can go from forecasts on random geopolitical questions from random people online, to forecasts from experts and people who are accurate forecasters of causal policy effects that can then maybe help people in the decision-making space, thinking about nuclear risk, make better decisions.

Ezra’s book recommendations [02:45:29]

Luisa Rodriguez: Neat. OK, we’ve only got time for one more question. What are some books you recommend and why?

Ezra Karger: I am an over-book-recommender, so I love giving book recommendations. Related to our discussions today, I might start by recommending a book called Moving Mars by Greg Bear. This is science fiction, but it has a very interesting discussion of what happens when there’s technological progress, when there’s a new technology that people are very uncertain about, and what happens when you have geopolitical groups who disagree about how to use that technology, or who disagree about who should have that technology?

And while it’s set in a world where there’s Earth and there’s Mars and there’s tension, I think it just very nicely plays out a scenario for how that might go that has some risks attached to it. I won’t spoil what happens at the end, but I find it very interesting reading this book and then thinking about the forecasts of existential risk from these tournaments.

Luisa Rodriguez: Nice.

Ezra Karger: So in addition to Moving Mars, let me recommend two other books. One is called The Second Kind of Impossible, and this is a book about scientific discovery. It’s by Paul Steinhardt, who’s an academic, and it’s about his search for quasicrystals, which I knew nothing about going into the book. But just seeing how he thinks about scientific discovery, how he thinks about trying to figure out whether something that we’re not sure exists actually exists, was really fascinating to me.

Maybe the last book I’ll recommend is more in my economic history wheelhouse: The Rise and Fall of American Growth. This is a book by an economic historian. It’s a really big book, but it has all of these great anecdotes about how GDP growth and productivity in the US changed from the Civil War to the present.

What I love about this book is I spend a lot of time thinking about what people’s forecasts are for the next 10 or 20 or 100 years, and it’s very interesting to see what mattered a lot for economic growth and other key outcomes over the past 100 or 150 years. I think that gives you maybe useful base rates — or maybe not useful base rates, if you think change is going to be more discontinuous now — but useful information about what was important historically.

Luisa Rodriguez: Nice. Thank you for those. And thank you for coming on. My guest today has been Ezra Karger.

Ezra Karger: Thank you so much for having me.

Luisa’s outro [02:47:54]

Luisa Rodriguez: Before we go, I wanted to mention again the Forecasting Research Institute is hiring! At the moment, they’re hiring for a research analyst, part-time research assistant, part-time content editor, and data analyst. You can learn more and apply for the roles at forecastingresearch.org/participate.

If you liked this episode, and want to learn more about forecasting, I recommend going back and listening to our interviews with Phil Tetlock:

• There’s episode #60: Accurately predicting the future is central to absolutely everything. Phil Tetlock has spent 40 years studying how to do it better
• And episode #15: Phil Tetlock on predicting catastrophes, why keep your politics secret, and when experts know more than you

All right, The 80,000 Hours Podcast is produced and edited by Keiran Harris.

Audio engineering by Ben Cordell, Milo McGuire, Simon Monsour, and Dominic Armstrong.

Full transcripts and an extensive collection of links to learn more are available on our site, and put together as always by Katy Moore.

Thanks for joining, talk to you again soon.