Transcript

Cold open [00:00:00]

Matt Clancy: Say you have a billion dollars per year. You could either write it to science, or you could spend it improving science: trying to build better scientific institutions, make the scientific machine more effective. And just making up numbers, say that you could make science 10% more effective if you spent $1 billion per year on that project. Well, we spend like $360 billion a year on science. If we could make that money go 10% further and get 10% more discoveries, it’d be like we had an extra $36 billion in value. And if we think each dollar of science generates $70 in social value, this is like an extra $2.5 trillion in value, per year, from this $1 billion investment. That’s a crazy high ROI — like 2,000 times as good.

And that’s the calculation that underlies why we have this Innovation Policy programme, and why we think it’s worth thinking about this stuff, even though there could be these downside risks and so on, instead of just doing something else.

Luisa’s intro [00:00:57]

Luisa Rodriguez: Hi listeners, this is Luisa Rodriguez, one of the hosts of The 80,000 Hours Podcast.

In today’s episode, I speak with Matt Clancy, an economist and research fellow with Open Philanthropy. I’ve been excited to talk to Matt for a while, because he’s been working on what seems to me like a really important issue: one, how to accelerate good scientific progress and innovation; and two, whether it’s even possible to accelerate just good science, or whether by accelerating science, you accelerate bad stuff along with good stuff.

We explore a bunch of different angles on those questions, including:

• How much can we boost health and incomes by investing in scientific progress?
• Scenarios where accelerating science could lead to existential risks — such as advanced biotechnology being used by bad actors.
• How scientific breakthroughs can be used for both good and bad, and often enable completely unexpected technologies — which is why it’s really hard to forecast the long-term consequences of speeding it up.
• Matt’s all-things-considered view on whether we should invest in speeding up scientific progress, even given the risks it might pose to humanity.
• Plus, non-philosophical reasons to discount the long-term future.

And at the end of the episode, we also talk about the reasons why Matt is sceptical that AGI could really cause explosive economic growth.

Without further ado, I bring you Matt Clancy.

The interview begins [00:02:34]

Luisa Rodriguez: Today I’m speaking with Matt Clancy, who runs Open Philanthropy’s Innovation Policy grantmaking programme and who’s a senior fellow at Institute for Progress. He’s probably best known as the creator and author of New Things Under the Sun, which is a living literature review on academic research about science and innovation. And I’m just really grateful to have you on. I’m really looking forward to this interview in particular. So thanks for coming on, Matt.

Matt Clancy: Yeah, thank you so much for having me.

Is scientific progress net positive for humanity? [00:03:00]

Luisa Rodriguez: I hope to talk about why you’re not convinced we’ll see explosive economic growth in the next few decades, despite progress on AI, but first, you’ve written this massive report on whether doing this kind of metascience — this thinking about how to improve science as a whole field — is actually net positive for humanity. So you’re interested in whether a grantmaker like Open Phil would be making the world better or worse by, for example, making grants to scientific institutions to replicate journal articles.

And I think, at least for some people, myself included, this sounds like a pretty bizarre question. It seems really clear to me that scientific progress has been kind of the main thing making life better for people for hundreds of years now. Can you explain why you’ve spent something like the last year looking into whether these metascience improvements that should bring us better technology, better medicine, better kind of everything, why might it not be worth doing all that?

Matt Clancy: Yeah, sure. And to be clear, I was the same when I started on this project. I maybe started thinking about this like a year and a half ago or so. I was like, “Obviously this is the case. It’s very frustrating that people don’t think this is a given.” But then I started to think that taking it as a given seems like a mistake. And in my field, economics of innovation, it is sort of taken as a given that science tends to almost always be good, progress in technological innovation tends to be good. Maybe there’s some exceptions with climate change, but we tend to not think about that as being a technology problem. It’s more like a specific kind of technology as bad.

But anyway, let me give you an example of a concrete scenario that sort of is the seed of beginning to reassess and think that it’s interesting to interrogate that underlying assumption. So suppose we make these grants, we do some of those experiments I talk about. We discover, for example — I’m just making this up — but we give people superforecasting tests when they’re doing peer review, and we find that you can identify people who are super good at picking science. And then we have this much better targeted science, and we’re making progress at a 10% faster rate than we normally would have. Over time, that aggregates up, and maybe after 10 years, we’re a year ahead of where we would have been if we hadn’t done this kind of stuff.

Now, suppose in 10 years we’re going to discover a cheap new genetic engineering technology that anyone can use in the world if they order the right parts off of Amazon. That could be great, but could also allow bad actors to genetically engineer pandemics and basically try to do terrible things with this technology. And if we’ve brought that forward, and that happens at year nine instead of year 10 because of some of these interventions we did, now we start to think that if that’s really bad, if these people using this technology causes huge problems for humanity, it begins to sort of wash out the benefits of getting the science a little bit faster.

And in fact, it could even be worse. Because what if, in year 10, that’s when AGI happens, for example. We get a super AI, and when that happens the world is transformed. We might discuss later why I have some scepticism that it will be so discrete, but I think it’s a possibility. So if that happens, maybe if we invented this cheap genetic engineering technology after that, it’s no risk: the AI can tell you, “Here’s how you mitigate that problem.” But if it comes available before that, then maybe we never get to the AGI, because somebody creates a super terrible virus that wipes out 99% of the population and we’re in some kind of YA dystopian apocalyptic future or something like that.

So anyway, that’s the sort of concrete scenario. And your instinct is to be like, come on. But you start to think about it and you’re like, it could be. We invented nuclear weapons. Those were real. They can lead to a dystopia and they could end civilisation. There’s no reason that science has to always play by the same rules and just always be good for humanity. Things can change. So I started to think it would be interesting to spend time interrogating this kind of assumption, and see if it’s a blind spot for my field and for other people.

Luisa Rodriguez: Cool. Yeah. There’s a lot there, and I want to unpack a few pieces of it. I guess one: I think when I first encountered the idea that someone out there as an individual might realise that they could use some new genetic engineering technology to try to create some civilisation-ending pandemic, I was like, “Who would want to do that? And surely one person can’t do that.” But then I interviewed Kevin Esvelt and got convinced that this was way more plausible than I thought. So we’ll link to that episode. If you’re just like, “What? What are you guys talking about?,” I recommend listening to that.

Maybe the other thing is just that I do find the nuclear weapons example really helpful, because it does just seem really plausible to me that, had science been a little bit further behind, that all of the work that went into making nuclear weapons happen when they did might not quite have been possible. It was already clearly a huge stretch. And clearly some countries did try to make them, but failed to do it at the same pace as the Americans did — and maybe a year of better science would have made the difference for them. And all of these possible effects are really hard to weigh up against each other and make sense of, and decide whether the world would have been better or worse if other countries had nuclear weapons earlier, or if Americans had nuclear weapons later.

It does kind of help me pump that intuition of having science a year faster than we might have otherwise can, in fact, really change the way certain dangerous weapons technologies look and how it plays out. Does that all feel like getting at the right thing?

Matt Clancy: Yeah, I think that’s the kind of core idea, that science matters. There are technologies that cannot be invented until you have the underlying science understood.

Luisa Rodriguez: And some of them are bad.

Matt Clancy: Yeah. And science is, in some sense, morally neutral. It just gives you the ability to do more stuff. And that has tended to be good, but it doesn’t have to be.

Luisa Rodriguez: Yeah. I think there are some reasons people still might be sceptical of this whole question, but maybe before we get to them, I’m also just a bit unsure of how action relevant this all is. Like, if it turned out that you wrote this whole report and it concluded that actually accelerating science is net negative, what would Open Philanthropy do differently? I imagine you wouldn’t stop funding new science. Or maybe you would. Or at the very least, I imagine you wouldn’t go out and start trying to make peer review worse.

Matt Clancy: Yeah. So I joined Open Phil in November 2022 to lead the Innovation Policy programme. And there were a number of people who were concerned about this, and it affects the potential direction that the Innovation Policy programme should go. We’re not going to, as you say, try to fund thwarting science in any way. But there’s finite resources, and there are different things you can do.

So one option would be we don’t pursue stuff that’s going to accelerate science just like across the board, like finding new ways to do better peer review or so on. Instead we just, instead of making grants to those organisations, we make them to other organisations that might have nothing to do with science at all — like farm animal welfare or something like that. So that’s one possible route you could go.

Another route is you could say that we take away that accelerating science is not good, but that’s only one option. There are other options available to you, including trying to pick and choose which kinds of science to fund. And some you might think is better than others. So maybe we should work on developing tools and technologies or social systems or whatever that align science in a direction that we think is more… So you could basically focus less on the speed and more on how does science choose wisely what to focus on and so forth.

So those are different directions you could go. It could be just stuff outside of science, or it could be trying to improve our ability to be wise in what we do with science.

Luisa Rodriguez: OK, fair enough. Let’s then get back to whether this is even a real problem, because I think part of me is still kind of sceptical. I think the thing that makes me most sceptical is just that it does seem like there’s clearly a range in technology. Some technology seems just really clearly only good. I don’t actually know if I can come up with something off the top of my head, but maybe vaccines, arguably. And some technology seems much more obviously dual use — so has some good applications, but also can clearly have some bad ones — like genetic engineering.

Why are people at Open Philanthropy worried that you can’t get those beneficial technologies without getting the dangerous ones by just thinking harder about it?

Matt Clancy: I think that to an extent we do think that is possible. We think, and we have programmes that work on this in specific technology domains, like biosecurity or AI, that make decisions that are the kind you’re talking about — where we think giving a research grant to this would be good, and giving a research grant to that would maybe not be good, so we’re not going to do that kind of thing.

But there’s, as we’ll talk about, a lot of value on the table from just improving overall science. Like, if you could make overall science a bit more effective, and if science is on average good, then you should totally do that. I think that’s the core argument: just that that’s always an option to just pick and choose, but we want to explore whether interventions that can’t be targeted are also worth pursuing, even though they can’t be targeted.

So like high-skilled immigration: it’s not going to be at a level of granularity, like, if there’s some kind of change to legislation that’s like, “If your research project is approved by the NSF as good, then you can get a green card.” It’s just going to be like, “If you meet certain characteristics” — and that’s going to sort of improve the efficacy of science across the board.

Luisa Rodriguez: OK. Yes, you can try to primarily fund safe and valuable science, but if you want to do any of this really high-level, general make science as a field be better, be more efficient, create more outcomes at all, you can’t make sure all of those things trickle down into the right science and not the bad science. That’s just not the level you’re working on.

Matt Clancy: Yeah. And I think that also at a higher level, there are limits to how well you can forecast stuff. So there’s this famous essay by John von Neumann called “Can we survive technology?” And he’s writing in kind of the shadow of the nuclear age and all this. He has this interesting quip, where he’s talking from first hand experience in a way that’s really interesting, where he’s like “It’s obvious to anyone who was involved in deciding what technology should be classified and not” — and I was not involved in that, but presumably he was during the war or something like that — he’s like, “It’s obvious to those of us who’ve been in that position that separating the two is an impossible task over more than a five to 10-year timeframe, because you just can’t parcel out.”

So science is like, really, I think it’s going into the unknown, and trying to figure out how something works when you don’t know what the outcome is going to be. And it’s an area where it’s very hard to predict what the outcome is going to be. It’s not impossible, and we try to make those calls, and that’s good. But I think it’s also important to recognise that there are strong limits to even how well that approach can work.

So for example, consider video games. Video games seem harmless, but the need for video games created demand and incentive to develop chips that can process lots of stuff in parallel, really fast and cheaply. So it seems benign. Another thing that seems benign: let’s digitise the world’s text, put it on a shared web, make it available to everyone. Create forums for people to just talk with each other — Twitter, Reddit, all this stuff — maybe it doesn’t seem as benign, but it doesn’t seem like it’s an existential risk, perhaps. But you combine chips that are really good at processing stuff in parallel with an ocean of text data, and that gives you the ingredients you need to develop large language models, we probably accelerated the development of those things. And the jury’s still out on if these are going to turn out to be good or bad, as you guys have covered before in different episodes.

Luisa Rodriguez: Got it. OK, that makes a bunch more sense to me. So yes, you can forecast whether this very genetic-engineering-related science project is going to advance the field of genetic engineering. But if you go far enough out, the way different technologies end up being used really does become just much more unpredictable. And I don’t know, there was a time, 30 to 40 years before we had nuclear weapons, that probably there was some science that ended up helping create nuclear weapons that no one would have said, “This is probably going to be used to make weapons of mass destruction.”

Matt Clancy: Yeah. And science is full of these sort of serendipity moments where, you know, you find a fungus in the trash can, and it becomes penicillin. Teflon was discovered through people working on trying to come up with better refrigerators. Teflon was also used in seals and stuff in the Manhattan Project. I’m not saying the Manhattan Project wouldn’t have been possible without refrigeration technology, but it contributed a little bit.

The time of biological perils [00:17:50]

Luisa Rodriguez: That’s interesting! OK, let’s talk about the main scientific field you think might end up being net harmful to society, such that we might not want to accelerate it: synthetic biology. What kind of evidence have you found convincing to make you think that there’s some chance that synthetic biology could get accelerated by broad scientific interventions in a way that could make science look bad in the end?

Matt Clancy: So we can focus on kind of two rationales. And just as an aside, there’s lots of other risks that you could focus on. But life sciences is a big part of science, like the big part of science, so that’s one reason we focused on that. AI risk is the thing that’s in the news right now, but I think what’s going on with AI is not primarily driven by academic science; it’s sort of a different bucket. So that’s why we didn’t focus on it in this report.

Luisa Rodriguez: Right. OK, that makes sense.

Matt Clancy: But anyway, turning back to synthetic biology: biology is sort of this nonlinear power where if you can create one special organism, you can unleash it. And if one person could do that, it would have this massive impact on the world.

Why hasn’t that ever happened? Well, one reason is because working with frontier life science and cutting-edge stuff is really hard, and requires a lot of access to specialised knowledge, and getting training. It’s not enough even to just read the papers. Today, you know, you typically need to go to labs, collaborate with people, get trained in techniques that are not easy to describe with text. So this need to collaborate with other people and spend a lot of time learning has been a barrier to people misusing this technology, at least one guardrail against this. And sometimes it gets supplanted, but it’s been a thing that makes it hard for people to work.

And this is kind of a general science thing too, is that frontier science requires working as a team more and more. So it’s harder to do bad stuff when you need a lot of people operating in secret with specialised skills, and how do you find those people and so forth.

The trouble is, AI is a technology that helps leverage access to information, and helps people learn, and helps people figure out how to do stuff that maybe they don’t necessarily know how to do. So this is the concern. It’s not necessarily that AI today is going to be able to help you enough, because as I said, a lot of the technology, a lot of what you need to know is maybe not written down. But it gets better all the time. People are going to maybe start trying to introduce AI into their workflows in their own labs. It seems not at all surprising that we would develop AI to help people learn how to do their postdoc training better and stuff, so maybe this stuff will get in the system eventually. And then if it leaks out, instead of needing lots of time and experience and to work with a bunch of people, maybe an individual or a small group of people can begin to do more frontier work than would have been possible in the past. So that’s one scenario.

Another line of evidence is drawn from a kind of completely different domain, or a different area. There was this forecasting tournament in 2022, called the Existential Risk Persuasion Tournament, which turns out to be a really important part of this report.

We’ll go over it; I bet we’ll talk about it more later, but in short, there’s like 170 people involved. They forecast a bunch of different questions related to existential risk. Some of those questions relate to the probability that pandemics of different types will happen in the future. They ask people, what’s the probability a genetically engineered pathogen will be the cause of the death of more than 1% of humans before the year 2030? Before the year 2050? Before the year 2100? And because they give these different timelines, you can kind of trace out what people think is the probability of these things happening.

And there’s two major communities that participated in this project: there are experienced superforecasters who’ve got experience in forecasting and have done well in it before; and then there’s domain experts, so people with training in biosecurity issues. And both of those groups forecast an increase in the probability of a genetically engineered pandemic occurring after 2030 relative to before. And they both not only see an increase, but an increase relative to the probability of just a naturally arising pandemic occurring. So you could have been worried that they just think maybe in an interconnected world, more stuff is going to happen. For example, the superforecasters actually think that the probability of a naturally occurring pandemic will go down, but the probability of a genetically engineered pandemic will go up.

That kind of suggests that this is a group that also sees new capabilities coming online that are going to allow new kinds of hazards to come along. And this is a really big group: 170. They were incentivised to try to be accurate in a lot of different ways. They were in this online forum where they could exchange all their ideas with each other. So I think it’s probably the best estimate we have of something that is inherently really nebulous and hard to forecast. But I think this is the best we’ve got, and it sort of does see this increase in risk coming.

Luisa Rodriguez: Cool.

Matt Clancy: Yeah. This whole report, I think, would be way worse if this project had not existed. So thanks to the Forecasting Research Institute for putting this on.

Luisa Rodriguez: OK, so that aside, the headline result is basically that these different communities that did these forecasts do think that there’s going to be some period of heightened risk, and it’s different to just increasing risks of natural pandemics because of the fact that we fly on planes more and can get more people sick or something. It seems like it’s something about the technological capabilities of different kinds of actors improving in this field of biology.

So I think in your report, you give this period the name “the time of biological perils” — or “the time of perils” for short. So all of that is basically how you’re kind of thinking of science as obviously being good, but then also having this cost, and that’s this time of perils.

Matt Clancy: Right. The short model is basically we’re in one era, and then the time of perils begins when there’s this heightened risk. And how long the time of perils… It could last indefinitely. Well, there’s a discount rate that kind of helps, is one way you can get out of it. But the short answer is like, there’s two regimes that you can be in.

Luisa Rodriguez: Yeah. Just to make this idea of the time of biological perils a bit more intuitive, it’s like, I think maybe we’ve talked enough about why we might enter it, why it might start. And obviously it’s not discrete — it’s not like one day we’re not in the time of perils, and the next day we are — but we’re going to talk about it like that for convenience. And then we think it might end because of various reasons. Maybe we create the science we need to end it by just like solving disease, so we can no longer use biological weapons to do a bunch of damage. Maybe it ends in some other way. But it could end, or it could just be like, now we have this technology forever and there’s no real solution. So just an indefinite time of perils might start at some point, and then from that point on, we just have these higher risks. And that’s really bad.

Matt Clancy: Yeah. This isn’t the first time that people have invented new technologies that bad actors use, and then they just become this persistent problem. Bombs, for example.

Luisa Rodriguez: Yeah. Arguably, we’re just in the time of nuclear perils now.

Matt Clancy: Right.

Luisa Rodriguez: Cool. OK, so that’s how to kind of conceptualise that.

Modelling the benefits of science [00:25:48]

Luisa Rodriguez: Let’s talk about the benefits of science. You take on the incredibly ambitious task of modelling the benefits of science. How do you even begin to put a number on the value of, say, a year’s worth of scientific progress?

Matt Clancy: We were inspired by this paper by economists Ben Jones and Larry Summers, which put a dollar value on R&D. And to clarify, like, not all R&D is science. Science is like research about, how does the world work? What are the natural laws? And so forth. A lot of other research is not trying to figure that out; it’s trying to invent better products and services.

And so this paper from [2020] by Larry Summers and Ben Jones was like, what if you shut off R&D for one year? That’s the thought experiment they imagine: what would you lose, and how much money would you save? So you would save money from what you would normally spend on R&D. It’s a little more complicated than that, because it also affects the future path of R&D. But as a first approximation, you lose that year’s R&D spending, or you save that year’s R&D spending, but you lose something too. So they want to add up the dollar value of what you lose.

And the thing that’s clever about it is economists think that all of income growth comes from technology — like the level of income of a society is ultimately driven by the technology it’s using to produce stuff. And the technology that we have available to us, especially in the USA, comes from research and design that is conducted. So if you stop the research, you stop the technology progress, you slow your income growth. So in the US, to a first approximation, GDP per capita income grows about 2% every year, adjusted for inflation. And so if you pause R&D, you don’t grow by 2% for one year at some point — maybe with a delay — and then every year thereafter, you grow by 2%. But because you missed that one year, you’re always 2% poorer than you sort of otherwise would have been.

So we’re going to borrow this idea, but apply it to science. So we’re going to say, if you pause science, you don’t lose all of economic growth — because unlike R&D, some technological progress doesn’t depend on science. So a big problem we’re going to have to solve is what share of growth comes from science. But you lose some, and then you’re a little bit poorer every year thereafter.

But we’re also going to have this flip side, which is since we have this mental framework of this “time of perils,” which begins at some year — and we’re assuming that the time of perils begins because we discover scientific principles that lead to technologies that people can use for bad things. If we delay by one year the development of those scientific discoveries, we’re going to also delay by one year the onset of that time of perils. And that’s going to tend to make us, in this model, spend less overall time in this time of perils. — basically, to a first approximation, one year less time.

So we’ve got kind of benefits — we would get income from a year of science — and we’ve got costs, which is that we get into this time of perils a bit faster. And then we add one third factor, which is: the idea behind the time of perils is that there’s these new biological capabilities that people can use for bad reasons — but obviously, the reason people develop these biological capabilities is to try to help people, to try to cure disease and so forth. So we want to capture the upside that we’re getting from health too, so we also try to model the impact of science on health. And all of this is possible to sort of put together in a common language, because at Open Philanthropy, we’ve got grants across a bunch of different areas, and we have these frameworks for how do we trade off health and income benefits.

And a couple of useful caveats: one, we’re going to be using Open Philanthropy’s framework for how do we value health versus income. And other people may disagree. So that’s one thing. And then of course, you could also argue that there are benefits to science that don’t come from just health and income. There might be value to just knowing things about the world — like intrinsic value of knowledge — and we’ll be missing that in this calculation too.

Luisa Rodriguez: Right. OK, so we’ll keep that in mind. So that’s kind of the framework you’re using. So then from there — and this is incredibly simplified — you make several models with different assumptions, where you first consider how much a year’s worth of scientific progress is worth in terms of increased health and increased incomes for people. Then you consider how big the population is, I guess both at a single time, but also over time, because it’s not just the people alive now that are going to benefit from science; it’s also generations and generations to come. But then you also have a discount rate, which is basically saying that we value the benefits people get from science in the very near future more than we value the theoretical benefits that people in the future might get from science.

And basically, that’s a lot of things. And I want to come back to the discount rates, because that’s actually really important.

Before we move on and talk about how you modelled this all together, I think there’s still a piece that at least hasn’t been hammered home enough for me, because if I’m like, “What if we’d skipped last year’s science and we were stuck at the level of science of a year ago?” then I’d be like, meh. So I think there’s something about how the benefits of science over the last year kind of accrue over time. Can you help me understand why I should feel sad about pausing science for a year?

Matt Clancy: Yeah, sure. And as well, you know, it’s an open question. Maybe you’ll feel happy.

Luisa Rodriguez: Right, right.

Matt Clancy: But no, spoiler alert, I think… Well, it’s complicated, as we’ll see. Anyway, you pause science. Yeah, you don’t miss anything, right? You maybe didn’t get to read as many cool stories in The New York Times about stuff that got discovered, but what’s the big deal?

The impact happens like 20 years later. So what is this now, 2024? We lost all of, say, 2023 science. So we’re working with 2022. That means in 2043 or whatever, we’re working with the 2022 science instead of the 2023 science. And so when finally the tech, not the science, bears fruit and turns into mRNA vaccines or whatever the next thing is going to be in 20 years, that gets delayed a year.

So it’s not that in the immediate term you really notice necessarily the loss to science, but a lot of technology is built on top of science. It takes a long time for those discoveries to get spun out into technologies that are out there in the world. It takes a long time to spin them out. It takes even longer for them to diffuse throughout the broader world, we try to model all this stuff, but that’s when you would notice it. You wouldn’t notice, but you would maybe notice in a couple of decades. You’d be like, “I feel a little bit poorer.” No, you wouldn’t say that.

Income and health gains from scientific progress [00:32:49]

Luisa Rodriguez: Got it. OK, that makes sense. Let’s talk more about your model. How did you estimate how much a year’s worth of scientific progress increases life expectancy and individual incomes?

Matt Clancy: So to start, US GDP per capita, as I said earlier, grows on average like 2% per year for the last century. Some part of that is going to be attributable to science, but not all of it. Some of it is attributable to technological progress. But again, actually not all of it. Even though economists say that in the long run, technological progress is kind of the big thing, in the short run — and the short run can be pretty long, I guess — there’s other stuff going on. For example, women entered the workforce over the 20th century in the US, and a lot more people went to college — and those kinds of things can make your society more productive, even if there’s not technological progress going on at the same time.

So there’s a paper by this economist Chad Jones, “The past and future of economic growth,” where he tries to sort of parcel out what part of that 2% growth per year that the US experiences is productivity growth from technological progress and what’s the rest. He concludes that roughly half is from technological progress — so say 1% a year.

So we’re going to start there. But again, technological progress, part of that comes from science, part of that comes from other stuff — just firms, Apple doing private research inside itself. So how do you figure out the share that you can attribute to science? There’s a couple different sources we looked at.

There was this cool survey in 1994 where they just surveyed corporate R&D managers, and they were like, “How much of your R&D depends on publicly funded research?” They didn’t ask specifically about science, but the government funds most of the publicly funded science, so it’s like a reasonable proxy. And this is 1994, 30 years ago. And they were like, “20% of our [R&D] depends on it.”

Another way you can look is like more recent data. You can look at patents. So patents describe inventions. They’re not perfect, but they are tied to some kind of technology and invention. And they cite stuff; they cite academic papers, for example. About a quarter, actually 26%, of US patents in the year 2018 cited some academic paper. So this is sort of in the same ballpark.

But there’s reasons to think that maybe this 20% to 26% is kind of understating things, because there can also be these indirect linkages to science. So if you’re writing software, if you’re using a computer at all, it runs on chips. Chips are super complicated to make and maybe are built on scientific advances. That’s not going to necessarily be captured if you ask people, “Does your work rely on science?” It relies on computers, but I’m not going to tell you it relies on science.

So one way you can quantify that is again, patents: some patents cite academic papers, and patents also cite other patents. So you could be like, how many patents cite patents that cite papers? Maybe that’s a measure of indirect things. And if you have any kind of connection at all, then you’re up to like 60% of patents are sort of connected to science in this way.

So those are two ways: surveys, patents. And then there’s a third way, which is based on statistical approaches, where you’re basically looking to see if there was a jump in how much science happened — maybe it’s R&D spending, maybe it’s measured by journal publications. What happened to productivity in a relevant sector down the road? Do you see the jumps matched by jumps in productivity later? Those find really big effects — almost one to one: a 20% increase in how much science happens leads to a 20% increase in productivity or so on.

Luisa Rodriguez: Wow.

Matt Clancy: So this indirect science stuff and this other stuff sort of suggests to me that if you just stopped science — we’re done with science; we’re not doing it anymore — I think in the long run, that would almost stop technological progress. That’s just my opinion: we’d probably get a long way from improving what we have without any new scientific understanding, but we’d hit some kind of wall.

But that’s actually not what we’re investigating in this report. We’re not going to shut down science forever; we’re just going to shut it down for one year. And that gets into some of these things that I think you alluded to earlier about maybe it’s not such a big deal; maybe we can push on without one year of science.

And there’s a couple of reasons to think that maybe that evidence I cited earlier is sort of overstating things. Like, you cited a paper, but does that really mean you needed to know what’s in that for you to invent the thing? And we don’t really know. We know that they’re more valuable, and there’s sort of other clues that they were getting something out of this connection to science, but we don’t really know.

And then also, you could imagine innovation is like fishing out good ideas: we’re fishing out of a pond new technologies, and science is like a way to restock the pond, because we fished out all the ideas, we sort of do everything. But science is like, “Actually, you know, there’s this whole electromagnetic spectrum and you can do all sorts of cool stuff with it” or whatever. And maybe if we don’t restock the pond, people just overfish, like fish down the level that’s in there, and there’s some consequence, but it’s not as severe.

So anyway, on balance, you kind of have to make a judgement call, but we end up hewing closer to the sort of direct evidence — so like the surveys where people say it relied directly on evidence, or the patents that directly cite papers. And our assumption is, if you give up science for one year, you lose a quarter of the growth that you get from technological progress. So instead of growing at 1% a year, you grow at 0.75% for one year. And again, that all happens after a very long delay, because it takes a long time for the science to percolate through.

Luisa Rodriguez: Right. In that first year, they’re still mining the science from two decades ago because of how that knowledge disseminates. How do you think about the effects on health?

Matt Clancy: Yeah. So to start there, how are you going to measure? What’s your preferred way of measuring health gains? Health is this sort of multidimensional thing. We’re going to use life expectancy, with the idea that a key distinction of health is like alive or dead.

And there was this paper that Felicity C. Bell and Michael L. Miller wrote in 2005 that collected and tabulated all this data in the US on basically life expectancy patterns over the last century. So they’ve got actually the share of people who live to different ages for every year going back to 1900. So if you’re born in 1920, we can look at this table and say, how many people live to age five, 10, 20, 30, et cetera. We can look again at 1930, if you were born in 1940. And as you would expect, the share of people surviving to every age tends to go up over time.

And because they did this report for the Social Security Administration, and the Social Security Administration’s goal is, “We want to know how much are things going to cost? How much are our benefits going to cost in the future? How many people are going to survive into old age?” — so this report forecasts into the future out through 2100. So they have other tables that are like, if you’re born in 2050, what do we think? How many people will survive to age five, 10, 15, 20?

So I’m going to use that data, rather than trying to invent something on my own. And the thing that’s kind of nice about this is, because they’ve got these estimates for every year, we can have this very concrete idea about if you lose a year of science, you could imagine — we don’t actually do this — but then you lose a year of the gain. So if you would have been born in 2050, but we skip science for a year, instead of being on the 2050 survival curve, you’re on the 2049 survival curve.

So we don’t actually do that because, again, I don’t think you lose everything If you lose a year of science. We end up saying you lose seven months. So how do we come up with that? This is a little bit of a harder one. I don’t think there’s quite as much data as we could draw on. But you can imagine that health comes from scientific medical advances plus a lot of other stuff that is related to, for example, income. Like you can afford more doctors, more sanitation and so on.

And there are some studies that try to portion out across the world how different factors explain health gains. And they say a huge share, 70% or 80%, of the health gains comes from technological progress. But if you dig in, it’s not actually technological progress. It’s more that we can explain 20% to 30% of the variation with measurable stuff, like how many doctors there are or something. The rest, we don’t know what it is. So we’re going to call it technological progress. So we want to be a little cautious with that.

But on the other hand, I think it makes sense that science seems important for health. And you do see that borne out in this data too. Like, if you look at the patent data, medical patents cite academic literature at like twice the rate, basically, as the average patent.

So where we come down is like, we’re going to say 50% of the health gains come from science; the rest comes from income. But a little bit of the income gains also come from science, and that’s how you end up with seven months instead of six months.

Luisa Rodriguez: OK, got it.

Matt Clancy: I guess the last thing to say is that you can tinker with different assumptions in the model and some of this stuff and see how it matters. I think that, are these exact numbers right? No. Are they like in an order of magnitude? Probably, yes. I would be very surprised if they were more than twice or half.

Discount rates [00:42:14]

Luisa Rodriguez: Great. Then let’s talk about discount rates. So usually, when I hear people discount the value of some benefit that some future person I don’t know, living centuries from now, will get, I have some kind of mild moral, “That’s not a good thing to do. Who says that your life is that much more valuable than someone living in a few centuries, just because they’re living in a few centuries and not alive yet?” Why are you discounting future lives the way you are?

Matt Clancy: Yeah, yeah. And this is a super important issue, and the choice of this parameter is really important for the results. And as you said, it’s standard in economics to just weigh the distant future less: it counts for less than the near future, which counts for less than the present. And there’s lots of justifications for why that’s the case. But you know, I take your point that morally it doesn’t seem to reflect our ethical values. Like if you know that you’re causing a death 5,000 years in the future, why isn’t it as bad as causing a death today?

So the paper ends up at the same place, where it’s got a standard economic discount rate. But I think I spend a lot more time trying to justify why we’re there, and giving an interpretation of that discount rate, and trying to more carefully justify it on grounds that are not like, “We morally think people are worth less in the future.” Instead, it’s all about epistemic stuff: what can you know about the impact of your policies? The basic idea is just that the further out you go, the less certain we are about the impact of any policy change we’re going to do.

So remember, ultimately we’re sort of being like, what’s the return on science? And there’s a bunch of reasons why the return on science could change in the distant future. It could be that science develops in a way in the future such that the return on science changes dramatically — like we reach a period where there’s just tonnes of crazy breakthroughs, so it’s crazy valuable that we can do that faster. Or it could be that we enter some worse version of this time of perils, and actually science is just always giving bad guys better weapons, and so it’s really bad.

But there’s a tonne of other scenarios, too. It could be that just that we’re ultimately thinking about evaluating some policy that we think is going to accelerate science, like improving replications or something. But over time, science and the broader ecosystem evolves in a way that actually, the way that we’re incentivising replications has now become like an albatross around the neck. And so what was a good policy has become a bad policy.

Then a third reason is just like, there could be these crazy changes to the state of the world. There could be disasters that happen — like supervolcanoes, meteorite impacts, nuclear war, out-of-control climate change. And if any of that happens, maybe you get to the point now where like, our little metascience policy stuff doesn’t matter anymore. Like, we’ve got way bigger fish to fry, and the return is zero because nobody’s doing science anymore.

It could also be that the world evolves in a way that, you know, the authorities that run the world, we actually don’t like them — they don’t share our values anymore, and now we’re unhappy that they have better science. It could also be that transformative AI happens.

So like, the long story short is the longer time goes on, the more likely it is that the world has changed in a way that the impact of your policy, you can’t predict it anymore. So the paper simplifies all these things; it doesn’t care about all these specific things. Instead, it just says, we’re going to invent this term called an “epistemic regime.” And the idea is that if you’re inside a regime, the future looks like the past, so the past is a good guide to the future. And that’s useful, because we’re saying things like 2% growth has historically occurred; we think it’s going to keep occurring in the future. Health gains have looked this way; we think they’re going to look this way in the future. As long as you’re inside this regime, we’re going to say that’s a valid choice.

And then every period, every year, there’s some small probability the world changes into a new epistemic regime, where all bets are off and the previous stuff is no longer a good guide. And how it could change could be any of those kinds of scenarios that we came up with. Then the choice of discount rate becomes like, what’s the probability that you think the world is going to change so much that historical trends are no longer a useful guide? And I settle on 2% per year, a 1-in-50 chance.

And where does that come from? Open Phil had this AI Worldviews Contest, where there was a panel of people judging the probability of transformative AI happening. And that gave you a spread of people’s views about what’s the probability we get transformative AI by certain years. And you get something a little less than 2% per year is the probability, if you look in the middle of that. Then Toby Ord has this famous book, The Precipice, and in there he has some forecasts about x-risk that is not derived from AI, but that covers some of those disasters. I also looked at, there’s been sort of trend breaks in the history of economic growth — since there’s been one since the Industrial Revolution, and maybe we expect something like that.

Anyway, we settle on a 2% rate. And the bottom line is that we’re sort of saying people in the distant future don’t count for much of anything in this model. But it’s not because we don’t care about them; it’s just that we have no idea if what we will do will help or hurt their situation. Another way to think of 2% is that, on average, every 50 years or so, the world changes so much that you can’t predict, you can’t use historical trends to extrapolate.

And one last caveat — which we’ll come to much later I think — is that extinction, when you think of discounting in this way, extinction is sort of a special class of problem that we will come back to.

Luisa Rodriguez: OK, I actually found that really convincing. I know not everyone shares the view that people living now and people living in the future have the same moral weight, but that is my view. But I’m totally sympathetic to the idea that the world has changed drastically in the past, and the world could change drastically again — and there are all sorts of ways that centuries from now could look so different from the present that we totally shouldn’t be counting the benefits that people living 1,000 years from now could get from our science policy, if there are all sorts of things that could mean our science policy has absolutely no effect on them, or has wildly more effects on them for the reasons you’ve given. So I didn’t expect to be so totally persuaded of having a discount rate like that, but I found that really compelling.

Before we move on, I’m a little sceptical still of 2% in particular. Like, something really weird happening once every 50 years was really helpful for making that very concrete to me. But it also triggered a, “Wait, I don’t feel like something really weird happens every 50 years.” Like, I feel like we’ve done something like move from an epistemic regime to another a couple of times based on kind of how you’re describing it and how I’m imagining it — where one example would be going from very slow economic growth to going to very exponential economic growth. Maybe another is like when we came up with nuclear weapons. Am I kind of underestimating how often these things happen, or is there something else that explains why I’m finding that counterintuitive?

Matt Clancy: No, I think that if you look historically, 2% seems like something changing every 50 years. That seems too often. I think that’s what you’re saying, right?

Luisa Rodriguez: That is what I’m saying.

Matt Clancy: Yeah. It’s rare that things like that happen. And I think that I would share that view. I think that the reason we pick a higher rate is because of this view that the future coming down the road is more likely to be weird than the recent past. So that’s kind of embedded in there. That’s implicit in people’s views on, is there going to be transformative AI? Transformative AI, by definition, is transformative.

I also think that, remember, this is a simplification. I think more realistically what’s going to happen is that there’ll just be a continuous, gradual increase in fuzz and noise around. You’re not going to move from one to another. It’s very convenient to model it as just like we move from one regime to another, and that’s sort of captured by that.

And then lastly, I’d say all else equal, a conservative estimate where you’re a little bit more humble about your ability to project things into the future aligns with my disposition rather than… But I guess other people would say, well then, why not pick 5%? So you still have to pick a number at some point.

Luisa Rodriguez: Yeah, fair enough.

How big are the returns to science? [00:51:08]

Luisa Rodriguez: OK, so that’s kind of the framework behind the model. Let’s talk about how big the returns to science are, if we kind of ignore those costs we talked about earlier, under what basically feel like realistic assumptions to you. So in other words, for every dollar we could spend on science, how much value would we get in terms of improved health and increased income?

Matt Clancy: Yeah. So setting aside that whole bio time of perils, the dangers from technology — so if somebody doesn’t think that’s actually a realistic concern, I think they’ll still find this part useful — our current preferred answer is that every dollar you spend on science, on average, has $70 in benefits.

And that’s subject to two important caveats. One: this is ongoing work, and since the original version of the report went up, we improved the model to incorporate lengthy lags of international diffusion. So when something gets discovered in America, it’s not necessarily when it’s available in Armenia or whatever. And so, the version of the paper with that improvement is the version of the report that’s currently available on archive. Anyway, $70 per dollar.

The other interesting caveat is that a dollar is not always a dollar. A dollar means different things to different people. What we’re talking about, what we care about, is people’s wellbeing and welfare. And a dollar buys different amounts of welfare. If you give a dollar to Bill Gates, you don’t buy very much welfare. If you give a dollar to somebody who’s living on a dollar a day, you buy a lot more welfare with that dollar. So at Open Philanthropy, we have a benchmark standard that we measure the social impact of stuff in terms of the welfare you buy if you give a dollar to somebody who’s earning $50,000 a year — which is roughly a typical American in 2011, when we set this benchmark.

Luisa Rodriguez: Cool. So spending $1 giving you a $70 return seems like a really, really good return to me.

Matt Clancy: Yeah, it’s great. Yeah, it is good. The key insight is that, when you talk about $70 of value, you can imagine somebody just giving you $70. And the benefits we get from science are very different than that kind of benefit. So if I gave the typical American $70, that’s like a one-time gift. And economists have this term, rival, which means like one person can have it at a time. So if I have the $70 and I give it to you, now you have the $70 and I no longer have $70. It’s sort of this transient, one-off thing.

The knowledge that is discovered by science is different. It’s not rival. If I discover an idea, I can give you the idea, and now we both have the idea. And lots of inventors and practice can build on the idea and invent technologies. And technologies, you can think of them as like blueprints for things that do things people want — and that is also a non-rival idea. So when science gives you $70 in benefits, it’s lots of little benefits shared by, in principle, everyone in the world and over many generations. So the individual impact per person is much smaller, but it’s just multiplied by lots of people.

Sort of the baseline is, we spend hundreds of billions of dollars per year on science. And what’s that getting us? It’s getting us a quarter of a percent of income growth in this model, plus this marginal increase in human longevity. But to a first approximation, after lots of decades, everybody gets those benefits. And that’s what’s different about just like a cash transfer.

Another way to think about if this is a lot — and this is my plug for metascience — 70x return is great, but that actually wouldn’t clear Open Phil’s own threshold for the impact we want our grants to make.

Luisa Rodriguez: Interesting.

Matt Clancy: What that means is, we would not find it a good return to just write a cheque to science. We pick specific scientific projects because we think they are more valuable than the average and so on.

But imagine, instead of just writing a cheque to science… Say you have a billion dollars per year. You could either write it to science, or you could spend it improving science: trying to build better scientific institutions, make the scientific machine more effective. And just making up numbers, say that you could make science 10% more effective if you spent $1 billion per year on that project. Well, we spend like $360 billion a year on science. If we could make that money go 10% further and get 10% more discoveries, it’d be like we had an extra $36 billion in value. And if we think each dollar of science generates $70 in social value, this is like an extra $2.5 trillion in value from this, per year, from this $1 billion investment. That’s a crazy high ROI — like 2,000 times as good.

And that’s the calculation that underlies why we have this Innovation Policy programme, and why we think it’s worth thinking about this stuff, even though there could be these downside risks and so on, instead of just doing something else.

Luisa Rodriguez: Right. Cool. That was really concrete. So that seems just like a really insane return on investment, as you’ve said. Does that feel right to you?

Matt Clancy: So it’s tough to say, because as I said, the nature of the benefit is very alien to your lived experience. It’s not like you could know what it’s like to be lots of people getting a little benefit.

But I did do this one exercise to try to check, like a sense check if this is right. And I was like, let’s think about my own life. Spoiler, I’m 40. So I’ve had 40 years of technological progress in my lifetime. And if you use the framework we use to evaluate this in this model, it says if progress is 1% per year, my life should be like 20% to 30% better than if I was my age now, 40 years ago.

So I thought, does it seem plausible that technological progress for me has generated a 20% to 30% improvement? So I spent a while thinking about this. And I think that yeah, it is actually very plausible, and sort of an interesting exercise, because it also helps you realise why maybe it’s hard to see that value.

Like one is: it affects the amount of time I have to do different kinds of things. And when you’re remembering back, you don’t remember that you spent three hours vacuuming versus like one hour vacuuming or something; you just remember you were vacuuming. So it kind of compresses the time, and so you lose that. And then also there’s just so many little things that happened that it’s hard to… Like, it’s easy to evaluate one big-impact thing, because you can imagine if I had it or I didn’t, but when it’s just like 1,000 little things, it’s harder to value. But do you want to hear like a list of little things?

Luisa Rodriguez: Yeah, absolutely. Tell me.

Matt Clancy: All right. So start with little trivial things. Like, I like to work in a coffee shop, and because of the internet and all the technology that came with computing, I can work remotely in a coffee shop most days for part of the day.

I like digital photography. These are just trivial things. And not only do I like taking them, but I’ve got them on my phone. The algorithm is always showing me a new photo every hour. My kids and I look through our pictures of our lives way more often than when I was a kid looking at photo albums.

A little bit less trivial is art, right? So my access to some kind of art is way higher than if I’d lived 40 years ago. The Spotify Wrapped came out in November, and I was like, oh man, I spent apparently 15% of my waking hours listening to music by, it said, 1,000 different artists. Similarly with movies: I’m watching lots of movies that would be hard to access in the past.

Another dimension of life is like learning about the world. I think learning is a great thing, and we just know a lot more about the world, through the mechanisms through science and technology and stuff. But there’s also been this huge proliferation of ways to ingest and learn that information in a way that’s useful to you. So there’s podcasts where you can have people come on and explain things; there’s data — there’s explainers, data visualisation is way better, YouTube videos, large language models are a new thing, and so forth. And there’s living literature reviews, which is what I write. So like a third of what I spend my time doing didn’t exist like 40 years ago.

Another dimension that life is worth living and valuable is like your social connections and so on. And for me, remote work has made a big difference for that. I grew up in Iowa; I have lots of friends and family in Iowa. And Iowa is not the hotspot of economics of innovation stuff, necessarily. But I live here, I work remotely for Open Phil, which is based in San Francisco. And then remote work also has these time effects. So I used to commute for my work 45 minutes a day each way. And I was a teacher, a professor, so that was not all the time — I had the summers off and so on — but anyways, still saving 90 minutes a day, that’s a form of life extension that we don’t normally think of as life extension, but it’s extending my time.

And then there’s tonnes of other things that have the same effect, where they just free up time. I used to, when I was a kid, drive and go shop — walk the shop floors a lot to get the stuff with my parents that you need. Now we have online shopping and a lot of the mundane stuff just comes to our house, shipped, it’s automated and stuff. We’ve got a more fuel efficient car; we’re not going to the gas station as much. We’ve got microwave steamable vegetables that I use instead of cooking in a pot and stuff. We’ve got an electric snow blower; it doesn’t need seasonal maintenance.

Just like a billion tiny little things. Like every time I tap to pay, I’m saving maybe a second or something. And then once added up with the remote work, the shopping, I think this is like giving me weeks per year of stuff that I wouldn’t be doing.

But I can keep going. So there’s like, it’s not just that you don’t have to do stuff that you wouldn’t normally do. There’s other times when it helps you make better use of time that might otherwise not be available to you. So like all these odd little moments that you’re waiting for the bus or for the driver to get here, for the kettle to boil, at the doctor’s office, whatever, you could be on your phone. And that’s on you, how you use that time, but you have the option to learn more and do interesting stuff.

Audio content, the same: for like a decade, half the books I’ve “read” per year have been audiobooks and podcasts. And I’m sure maybe there’s people listening to this podcast right now while they’re doing something that they otherwise normally would not be able to learn anything about the world — so they’re driving or walking or doing the dishes or folding laundry or something like that. So that’s all the tonnes of tiny little things.

And this is just setting aside medicine, which is equally valuable to all that stuff, right? I’ve been lucky in that I haven’t had life-threatening illnesses, but I know people who would be dead if not for advances in the 40 years, and they’re still in my life because of this stuff. And then I benefited, like everyone else, from the mRNA vaccines that ended lockdown and so forth.

So, long story short: it seems very plausible to me that the framework we’re using, which says this should be worth 20% to 30% of my wellbeing, seems plausible over a 40-year lifespan. I’m luckier than some people in some respects, but I’ve also benefited less than other people in some respects. Like, if somebody had a medical emergency that they wouldn’t be alive here today, they could say that they benefited more from science than me. So if this is happening to lots of people now and in the future, that’s where I start to think that $70 per dollar in value starts to seem plausible.

Luisa Rodriguez: I was all ready with my follow up questions that were like, “I’m not so sure that you’re 20% to 30% better off than you were in 1984…” — but I think I’m just sold, so we can move on.

Matt Clancy: Well, thanks for indulging me, because I think it’s hard to get the sense unless you really start to go through a long list, and it’s just so many little things. And if you think of anyone individually — like, “Oh, this guy’s happy that he can go to coffee shops, woo-hoo” — but that’s just like a tiny sliver of all the different things that are happening.

Luisa Rodriguez: Yeah. I think I still have some… It’s closer to a philosophical uncertainty than an empirical one, that’s just like, humans seem to have ranges of wellbeing they can experience. And it might just be that a person that grew up kind of like you or I did — in the US, without anything terrible happening to them — can’t be that much better now than they were 40 years ago, just because of the way our brains are wired. But it also seems like there are some things that clearly do affect wellbeing, and a bunch of those — especially the health things — clearly do.

And I definitely put some weight on all of these small things that add up also making a difference. It does fill me with horror that I couldn’t have listened to the music and podcasts I would have liked to have listened to while doing hours of chores at home — that now, one, I get to do while doing fun stuff, and two, get to do with a Dyson. My Dyson is awesome.

Matt Clancy: Yeah, we have an iRobot that we use on some of our things. But actually, this concern, this is part of what motivated me, is this like, does innovation policy really matter for people who are already living on the frontier? And I came away thinking that it may be hard to see, but I think it does.

Forecasting global catastrophic biological risks from scientific progress [01:05:20]

Luisa Rodriguez: OK, so let’s leave the benefits aside for now and come back to the costs. So the costs we’re thinking of as the increase in global biological catastrophic risks that are caused by scientific progress. And to estimate this, you used forecasts generated as part of the Forecasting Research Institute’s Existential Risk Persuasion Tournament. Can you just start by actually explaining the setup of that tournament?

Matt Clancy: Yeah. This tournament was hugely important, because the credibility of this whole exercise hinges on how are you going to parameterise, or how are you going to estimate these risks from technologies that haven’t yet been invented.

Anyway, the tournament was held in 2022, I think with 169 participants. Half of them are superforecaster generalists, meaning that they’ve got experience with forecasting and they’ve done well in various contests or other things. The other half are domain experts, so they’re not necessarily people with forecasting expertise, but they’re experts in the areas that are being investigated. They got people from four main clusters: biosecurity, AI, nuclear stuff, and AI risk. There might have been climate, but I can’t remember. I focused mostly on the biorisk people. There are 14 biorisk experts. Who they are, we don’t really know. It’s anonymous.

Anyway, the format of this thing: it’s all online, and it proceeded over a long time period. People forecast I think 59 different questions related to existential risk. First they did them by themselves as individuals, then they were put into groups with other people of the same type — so superforecasters were grouped together, biorisk experts were grouped together — and then they collaborate, they talk to each other, and then they can update their forecasts again as individuals. Then the groups are combined. And so you get superforecasters in dialogue with the domain experts, they update again. And then in the last stage, they get access to arguments from other groups that they weren’t part of. And it’s well incentivised to try and do a good job. You get paid if you are better at forecasting things that occur in the near term: 2024, 2030.

But they want to go out to 2100. So it’s a real challenge. How do you incentivise people to come up with good forecasts that we won’t actually be able to validate? And they try a few different things, but one is, for example, you get awards for writing arguments that other people find very persuasive. And then they tried this thing where you get incentivised to be able to predict what other groups predict. There’s this idea, I think Bryan Caplan maybe coined it, of the [ideological] Turing test: can I present somebody else’s opinion in such a way that they would say, yep, that’s my opinion? So this is sort of getting at that. Like, do you understand their arguments? Can you predict what they’re going to say is the outcome?

Fortunately for me, in this project they asked a variety of questions about genetically engineered pandemics and a bunch of other stuff. The main question was like: what’s the probability that a genetically engineered pandemic is going to result in 1% of the people dying over a five-year period? And they debated that a lot. At the end of the tournament, they asked two additional questions: what about 10% of the people dying, and what about the entire human race going extinct? So those last ones, it’s not ideal; they weren’t based on the same level of debate, but these people had all debated similar issues in the context of this one.

Luisa Rodriguez: Right. OK, so that’s the setup. What were the bottom lines relevant to the report? What do they predict? Did they think there’ll be a time of biological perils?

Matt Clancy: So one thing that was super interesting was that there was a sharp disagreement between the domain experts and the superforecasters. And this sharp disagreement never resolved — even though they were in dialogue with each other and incentivised in these ways to try to get it right, they just ended up differing in their opinions in a way that never, the gap was never closed.

And it differs quite substantially. So if you start with the domain experts, the 14 biosecurity experts: they thought that a genetically engineered pandemic disaster is going to be a lot more likely after 2030. The superforecasters thought the same thing, that it’s going to be more likely, but their increase was smaller. And because there’s the sharp disagreement between them, throughout the whole report, I’m basically like, “here’s the model with one set of answers; here’s the model with the other set of answers.”

And the key thing that we have to get from their answers is: what is the increase in the probability that somebody dies due to these new biological capabilities? And like I said, there’s a sharp disagreement. The superforecasters gave the answers that, if I put it in the terms of the model, implies like a 0.0021% probability per year. The domain experts: 0.0385. The numbers don’t matter…The numbers do matter, but they’re not going to be meaningful to somebody who’s never thought of it… [The key thing is that] the domain experts think the risk is 18 times higher than the superforecasters do.

Luisa Rodriguez: Is there a way to put that in context? I don’t know, what’s my risk in the next century or something of dying because of the risk, according to each of those groups?

Matt Clancy: So we all just lived through this pandemic event, so we all actually have some experience of what these things can be like. And The Economist magazine did this estimate that in 2021, the number of excess deaths due to COVID-19 was 0.1% of the world. I think that’s like a useful anchor that we’re familiar with.

So if you want to take the superforecasters, they’re going to say something like that happens every 48 years. And I should say it’s not that a COVID-like event happens every 40 years — it’s very specifically that there is an additional genetically engineered-type COVID every 48 years, as implied by their answers. The domain experts, it’s like this is happening every two and a half years, is their forecast.

And I think another important thing to clarify is that these answers are also consistent with them thinking, for example, that it could be something much worse less often, or it could be something not as bad more often, that kind of thing. But that’s a way to calibrate the order of magnitude that they’re talking about.

Luisa Rodriguez: OK, yeah. It could be something like, at least according to the domain experts, you have something half as bad as COVID once a year, or it could be twice as bad every four years. But that’s the kind of probabilities we’re talking about. And they do seem even more different to me once you put them in context like that. So maybe we should actually just address that head on: how do you think about the fact that the superforecasters and domain experts came up with such different results, even though they had months to argue with each other? And in theory, it would be really nice if they could converge.

Matt Clancy: I think this is a super interesting open question that I’m fascinated by. I didn’t participate in this tournament; I’m not one of those sort of anonymous people. But I did engage in a very long back-and-forth debate format with somebody about explosive economic growth. And we’re on opposite sides of the position. And again, I think there was like no convergence in our views, basically. It’s interesting, because the biorisk people in this are not outliers. There was this disagreement between the superforecasters and domain experts just across the board on different x-risk scenarios.

And I could say more, but I think the core thing, my guess is that people just have different intuitions about how much you can learn from theory. I think some people, maybe myself included, are like, “Until I see data, my opinion can’t be changed that far. You can make an argument, I can’t necessarily pick a hole in it, but you might just be missing something.” Like theory, theoretical speculation has a bad track record, maybe.

But I think that it’s actually an open question. We don’t actually know who’s right. We don’t know if I’m right that people have different intuitions about how much you can learn from theory. We don’t have, as far as I know, good evidence on how much weight should you put in theory? So hopefully we’ll find out over the next century or something. You know, I have views on who’s right, but through the report… I’m also not a biologist, and I didn’t participate. Like I said, I just present them all in parallel, and try to hold my views until the end.

Luisa Rodriguez: OK, so as we talk about the results, we will talk about the results you get if you use the estimates from both of these groups, and we’ll be really clear about what we’re talking about. But then I do want to come back to who you think is right, because people are going to hear both results, but the results in some cases are pretty different and have different impacts, at least in the degree to which you’d believe in one set of policy recommendations over another. And I want to give people a little bit more of something to have when thinking about which set of results they buy. But we’ll come back to that in a bit.

What’s the value of scientific progress, given the risks? [01:15:09]

Luisa Rodriguez: Let’s talk about your headline results. You estimated the net value of one year of science, factoring in the fact that science makes people healthier and richer, but also might bring forward the time of biological perils. What was that headline result?

Matt Clancy: Yeah, so like you said, it depends on whose forecasts you use. Earlier I said that if you just ignore all these issues, the return on science is like $70 per dollar. It was actually $69 per dollar. And that only matters because if you use the superforecaster estimates, the return drops by a dollar to $68 per dollar.

If you use the domain experts, you see it drops from $69 to roughly $50 per dollar. Which, getting $50 for every dollar spent is still really good. But another way to look at it is like these bio capabilities, if they’re right, erase like a quarter of the value that we’re getting from science. Because dropping from $70 to $50 is almost 25% of the value, basically.

I think this whole exercise, you know, we’re not physicists charting with precision if we’re going to see Higgs boson particles or something. This is more like quantitative thought experiments, right? If you quantify the benefits based on historical trends, you quantify the potential costs based on these forecasting things, you can kind of do a horse race.

And my takeaway from this exercise is basically the benefits of science historically have been very good, even relative to these forecasts. So you should keep doing it. But also, if these domain experts are right, the risks are very important, and quantitatively matter a lot. And addressing them is super important too.

Luisa Rodriguez: Yeah. OK, let’s see. I feel like there are a couple of takeaways there. One is just like, in particular if the domain experts are right, the risks are really eating some of this value. And that’s a real shame, even if it still points at science being net positive. I guess also, just to affirm: even though you’ve done this super thoroughly, it’s just such a massive, difficult question that your results are going to have really wide error bars still.

But I love this kind of exercise in general, because I feel like people could make really compelling arguments to me about science being net negative because of engineered pathogens, and people are going to have more and more accessibility to the ability to create a new pathogen we’ve never seen before that could have really horrible impacts on society. And I could be like, “Wow. Yeah, what if science is net negative?” And then I could also be convinced, I guess through just argument, that the gains to science from science are really important. And yes, they increase these risks, but we obviously have to keep doing science or society will really suffer.

And it just really, really matters what numbers, even if they’re wide ranging, you put on those different kinds of possibilities, and you just don’t get that through argument.

Matt Clancy: Yeah, 100%. And when we started this project, that was the state of play — just like, there are arguments that seem very compelling on either side. And I think that in that situation, it’s very tempting to be like, “Well, maybe we’ll just say it’s like 50/50” — but the quantities really matter here. And so when we started this research project, I was like, we have to see if, best-case scenario, one of the benefits way outweigh the costs, or vice versa. Because as you said, there’s going to be these error bars.

But I was like, “That’s not going to happen. I’m sure that we’re going to end up with just this muddled answer, where it’s like, if you buy the arguments that say risk, you get that it’s bad, and if you buy the arguments that say it’s not risky…” So anyway, I’m very glad we checked, and I’m super glad that there was this Existential Risk Persuasion Tournament that I think gave us numbers that were more credible for what reasonable people might forecast — if they’re in a group, they’ve got incentives to try to figure it out and so on. It’s not just, you know, me trying to figure it out.

Luisa Rodriguez: Asking around. Yeah, yeah, yeah. I completely agree. And yeah, I see what you mean. It could have been the case that you play around with a few assumptions that people really disagree on, and that changes the sign of your results, so you still don’t get any clarity. But in fact, this just did give you some clarity. And using even both of these estimates, you get positive returns from science.

OK, so there are these superforecaster estimates that you use, and then you also use domain experts, but then you do this third thing that doesn’t rely on either of those. Can you explain that?

Matt Clancy: Sure. So suppose you think that both these guys are really wrong. Like you think it’s maybe worse or better or something. Another kind of useful thing this model can do is say, what would the forecast need to be for the average return on science to flip from positive to negative? So we saw that if it’s 0.0021, it barely makes a difference. That’s the superforecasters. If it goes up to 0.0385, which is 18 times as high, it chops off like a quarter of the value. How much higher does it need to go before you’re actually like, science has flipped from being a good thing to a bad thing?

Luisa Rodriguez: Right, OK. So how likely do global catastrophic biological risks, because of these advances from science, have to be before you’re like, “Let’s pause science.”

Matt Clancy: Right. So there’s lots of different assumptions you can make in the model, but the preferred one I do, which I think is the conservative one, is that break-even forecast is around 0.13% per year probability of dying from one of these new capabilities.

Luisa Rodriguez: That sounds really high.

Matt Clancy: Well, yeah, it is actually really high. It’s three times higher than the domain experts, and they were the more pessimistic group. And if you compare that to COVID, remember, that’s actually worse than 2021 COVID. The Economist estimated excess deaths due to COVID were 0.1% in 2021. So this would be like every year, the worst parts of COVID just sort of all the time. Or something 10 times as bad once every decade or something like that. So that’s not what anyone is forecasting in this tournament, or at least not what the group medians are. But that’s what it would need to be for the model to say that now you want to avoid that; the benefits of science aren’t worth all of this.

Factoring in extinction risk [01:21:56]

Luisa Rodriguez: OK. But importantly, those results are only considering catastrophes that don’t permanently affect the future of humanity. So in particular, it doesn’t consider the possibility that biological catastrophe could cause human extinction. What happens when you account for the risk of human extinction?

Matt Clancy: Yeah. Everything we’ve done so far has been kind of in this normal economic framework where you’re discounting the future. So stuff that happens in the distant future doesn’t matter, and that’s because we can’t predict the impact of science or policies that far out. But extinction is kind of a different class of problem, as we alluded to earlier. It doesn’t have the same uncertainty about long-run forecasts: if we actually go extinct, then we know we can predict that a billion years from now, we’ll still be extinct. We can’t predict a billion years from now if our little science policy will be very good or very bad, but we can predict that.

And so in that sense, if you think that faster science can affect the probability of something like extinction — or even just a collapse of civilisation: that we are no longer having this advanced technological civilisation that’s making scientific progress, something like a sci-fi movie like The Stand or The Last Man on Earth or something like that — in any of those cases, the discount of 2% per year doesn’t seem to make sense to do.

The way I handle that in the report is basically build a second model that now kind of has two branches. One branch is the same as everything we’ve talked about: the benefits of science we’re discounting because we don’t know what’s going to happen in the long run, stuff like that. But then there’s this other branch with a very small probability — very remote, but not zero — where the human population, we all go extinct, basically. And if we’re extinct, now we know for sure what happens as far out as you want to go.

But then the question becomes: what is it worth not to be dead? We don’t know what the value of the policy will be, but we might expect that there’s some value of not being dead that we can forecast out longer. You can then ask in the model: is the very remote chance of losing all that worth the benefits that we get from science? Which comes from this other branch in the tree, all the modelling we’ve done.

So I don’t try to estimate what the long-run future value of the species is. Instead, the model can spit out some guidance for what you might think it is and how you might think about these questions. The first problem we have though is that, whereas the superforecasters and the domain experts disagree a lot about the probability of genetically engineered pandemics causing all these problems, they really disagree about the probability of extinction. So if you recall, there was like a 20x, I think it was 18x, but close to 20x difference in the annual probabilities of dying from one of these diseases that I brought up before. But the difference in the probability that the species goes extinct from a genetically engineered pandemic is like 140 times different between the two.

Luisa Rodriguez: That’s pretty huge.

Matt Clancy: Yeah, yeah. Anyway, you can basically say, if there’s this very small chance that we go extinct, but we get these benefits if we don’t, how bad does extinction need to be for this to not be worth it? And if you think the biosecurity people are right, the more pessimistic group, then you only really need to believe that the long-run future is worth more than about 100 years of today’s global wellbeing — so 8 billion people at this standard of living persisting for another 100 years. If you think the superforecasters are right, you need to have a much bigger number: it’s like 20,000 years-ish of that range.

And this is interesting for a couple of reasons. One is that you guys have covered longtermism before on the podcast and so forth: the idea that we should think a lot about our long-run impact on the species. And one critique you sometimes hear of longtermism is that actually you don’t really need that baggage of the long future distant population to affect your decision making: if you think transformative AI is going to be a big deal, a lot of people think it’s going to happen in our lifetime, and so all you have to care about is yourself and your kids and you make similar decisions.

But science might be a situation where this framework does actually give you different decisions. If you are not putting much weight on that very long-run future, you’re going to say the benefits of science clearly outweigh the cost. And if you do put a lot of weight on it, then it’s plausible that you’ll have the opposite conclusion.

And I’m personally not 100% sure what to make of all this. Also, it sort of seems weird to make big decisions based on these parameters that we have the least information about. It’s really hard to know what these probabilities really should be. But I do think one bottom line I take away is that these issues are worth taking seriously, and they actually do affect some decisions we made in the Innovation Policy programme.

Luisa Rodriguez: OK, just because those are kind of complicated but important conclusions, I want to make sure I understand them. So if you buy into the estimates by the domain experts, then you think that extinction risks are so plausible from these risks that science kind of pushes along or brings forward, that all you have to do is think that there’s a plausible another 100 years of human existence that’s roughly as good as the current human existence to think that it’s just not worth risking the next 100 years for the benefits of science?

Matt Clancy: I mean, to be really precise, what the model is saying is if you had the choice to pause science for one year — which of course you don’t, but that’s the thought experiment the model chooses — you would give up some benefits, but you would also reduce your risk by a small amount, like one year worth of being in the extinction risk zone. And if you think the chances of extinction are sufficiently high — which the domain experts think it’s higher — then basically they would be happy to make that trade, because they’d be willing to give up the benefits because they’re not as high as the expected value of what you might lose. Very remote chance you lose it, but if you do, it would be a big deal. And they don’t think it’s that remote, I guess you would say, compared to other people. So that’s how you think about it in the context of this model.

Luisa Rodriguez: OK. Which I think makes it actually even a more striking conclusion, because I think I was kind of glossing over and saying, you might not think accelerating science in general is worth the risk — but this is actually just like, you might not think that the benefits of science over just one year are worth the risk of it bringing forward this time of biological perils, which seems even more extreme and worrying to me.

Matt Clancy: Yeah, it is. I don’t know. In some sense it’s a surprising result. But also there’s a sense in which maybe it’s not surprising, because… I mean, it’s surprising that it’s 100 years. But I guess the thing that maybe is not surprising is that if you think the human race is going to go on for a long time, then that’s a big number that can swamp lots of other considerations.

Luisa Rodriguez: Right. And so the superforecasters don’t think it’s that likely. So then you have to think that the future might have 20,000 years’ worth of value to be willing to pause science, which seems like a way more difficult position to stand behind. I really don’t know how to think about whether I think we’ve got 20,000 years of good stuff ahead of us.

Matt Clancy: That’s right. And there’s other kind of complicating things. Because you have to think about things like: does what the species turn out to look like in 20,000 years, is it the kind of thing that aligns with our values or not? I don’t know. It’s weird stuff.

Luisa Rodriguez: It’s weird stuff.

How science could reduce extinction risk [01:30:18]

Luisa Rodriguez: OK, so does that mean that if you put some weight — maybe you have to put a lot of weight — on these domain experts’ views, you should just be really in favour of pausing science now for a year?

Matt Clancy: I think that the model sort of says, if you just take that part of the model, then maybe, yeah. But as we’ll say, there are other stories you can tell that we also try to look at in the paper. So the two countervailing stories that we look at are basically ideas that better science could reduce extinction risk — because if you think extinction is the whole ballgame, and improving science can reduce that, then suddenly things flip.

So there’s kind of two ways that could happen. One is it could be that better, faster, nimbler science — like that is really well-targeted at important problems — that’s going to accelerate science, but maybe it’ll also reduce the dangers in this time of perils because we’ll be able to quickly identify and develop countermeasures; we’ll be more prudent maybe in what we fund, and reduce risks in that way. So that’s one thing I look at.

And then the other is maybe there’s some endgame where you can reach a sort of stage of technological maturity, where you invent technologies that render this whole period of time of perils and everything a much less salient problem. Like maybe you invent a miracle drug that can kind of cure anything. Or maybe more realistically, you develop far-UVC lighting that can sort of sanitise the air, and you embed that all over your built environment or something like that. And maybe if we can get there, we don’t have to worry so much about that. And in that case, if you accelerate science, maybe you can get there quicker. So those are two scenarios that we investigate.

Luisa Rodriguez: Yeah, let’s talk about those, actually. Let’s first talk about whether we could do better science. I guess you’ve already said a version of this, but it seems totally plausible to me that under your remit could be something like, “help institutions that fund science or help decide what science gets done” — figure out how to distinguish between technology that’s very likely to be purely good versus technology that’s reasonably likely to be dual use. It does seem like you can at least make some distinctions and get better at noticing which is which. Or maybe it’s like, “make institutions better at governing and regulating how science gets used” or something.

Is it possible that throwing a bunch of that makes this whole question less important, because you just kind of get rid of this bad risk part of the calculation?

Matt Clancy: Yeah, it is possible. I was really interested in this idea, and I think there’s some good theoretical arguments. One interesting theoretical argument is that if you discover some kind of really advanced scientific solution to problems, a huge group like governments can coordinate around it — you can get the world to coordinate around making this thing happen, like making mRNA vaccines or something, and faster science could basically tilt the balance so that we have more of those frontier things available to these big actors. And presumably that kind of cutting-edge stuff, that takes the coordination of hundreds or thousands of people, wouldn’t be available to sort of bad actors.

But I ended up coming away a little bit pessimistic about this route, just because the reduction in risk that you need to hit seemed really large. So the way I looked at this was, I thought rather than pausing science for a year and evaluating that, it was sort of like, what if you’ve got an extra year of science somehow — through efficiency gains, the equivalent of an extra year — and that allowed you to reduce by some percentage or some amount the danger from this time of perils. How big would that need to be?

And I thought a reasonable benchmark is however much we reduce all-cause death per year. That’s what science is trying to figure out now, we have a bunch of scientists working on it, and we can look at how much life expectancy tends to go up each year. So that’s just a ballpark place to start. And if that’s what you get from an extra year, like you reduce the annual probability of death by some small amount, and now you apply that to these estimates from the superforecasters or the domain experts, it’s just not big enough to make a difference. It doesn’t move the needle very much in this model.

Luisa Rodriguez: Is that really the best we could do though, if we tried really hard to do better science? Like, if we had some sense of what the risks were, and we have at least some sense, could better science just be really, really targeted at making sure those risks don’t play out?

Matt Clancy: I mean, I think it is possible, so I don’t say that this is definitive or anything, but I say it would need to be better than we’re doing against the suite of natural problems that are arising.

So I’ll give you two reasons: I’ll give you a reason to be optimistic and a reason to be pessimistic. A reason to be optimistic is that we actually do have pretty good evidence that science can respond rapidly to new problems. So when COVID-19 hit, prior to COVID-19 the number of people doing research on that kind of disease was like all but zero. Afterwards, like one in 20 papers — this is based on a paper by Ryan Hill and other people — was in some way related to COVID. So basically, 5% of the scientific ecosystem, which is a huge number of people, pivoted to starting to work on this.

You can see similar kinds of responses in other domains too. Like, the funding for different institutes in the National Institute for Health tends to be very steady, just goes up with inflation or whatever, and doesn’t move around very much. But there were like three times when it moved around quite a lot in the last several decades. One was in response to the AIDS crisis, and another was after 9/11, when there were a lot of fears about bioterrorism. So these both are examples of something big and salient happened, and the ecosystem did change and pour a lot of resources into trying to solve those problems.

The counterargument though, like why you might want to be pessimistic, is that it took a long time to sort out AIDS, for example. And in general, even though science can begin working on a problem really quickly, that doesn’t mean that a solution will always be quick in arriving. Throughout the report, we assume that the benefits from science take 20 years before they turn into some kind of technology — that’s under normal science, not any kind of accelerated crisis thing — and then they take even longer to spill out over the rest of the world. So, you know, the mRNA vaccines that ended COVID quickly, the research that was underlying that had been going on for decades earlier. It wasn’t like COVID hit, and then we were like, we got to do all this work to try to figure out how to solve it.

Luisa Rodriguez: Right. Make it up from scratch. Yeah, that seems balanced. I guess overall, it sounds like you’re not that optimistic about this, but you wouldn’t rule it out as a thing worth trying or aiming for.

Matt Clancy: Sure. Yeah, I think that’s certainly something that you would try. But it’s a big lift. But remember, that’s only one of the two ways that we think about science reducing risk: just moderating how bad things are during this time of perils. And it’s kind of based on this assumption that there’s not a lot of forward-looking planning — like 20 years in advance, we’re trying to figure out how are we going to deal with this when it comes. It’s more like, when it’s here, how quickly would we be able to respond to whatever the novel threat is?

Luisa Rodriguez: OK. So the other counterargument that might make science look less clearly bad, if you’re super worried about extinction risks in the way domain experts are, is this argument made by Leo Aschenbrenner, which is that we might be able to rush past the time of perils, or rush through it. So, by actually accelerating everything a lot, we can both enter this time of biological perils sooner, but also sprint through it to get to the other side faster. That seems pretty intuitively plausible to me. How do you think about it?

Matt Clancy: Yeah, I think it’s really easy to understand. Imagine bad times start in 10 years. Good times will start after 30 years — that’s when we discover, say, a miracle cure. If you have science go twice as fast, the bad times start in five years, but the good times get here in 15 years. In the first example, you’re 20 years in this period where every year bad people are using this technology, you don’t have a good countermeasure for it, and they’re trying to hurt you. So that’s 20 years that you have to try to beat them to the punch by investigating and foiling their plans before it happens. And if you have faster science, maybe you only have to do that for 10 years, and maybe that’s better. So that’s the basic argument.

And notice that, if this is true, then this also reduces extinction risk. It’s not just this conventional, non-civilisation-ending kind of bad things happening; the argument would apply equally well to extinction risk.

And I think that this is definitely a plausible argument. I would say, I don’t know, 50/50 that that will work.

Luisa Rodriguez: Yeah, it does feel very theoretical to me. It feels like maybe you could also argue that if the time of perils is brought five years earlier, we are less prepared for it, and so it goes worse overall per year during those 15 years than it could have if it had been 30 years long. But we spent the next 10 years thinking seriously about these risks… So it definitely has always felt to me like a “yeah, maybe” kind of argument, but not at all like there’s some iron law that says that if we just accelerate everything, it becomes shorter.

Matt Clancy: Yeah. I think that you wouldn’t want to bring things forward if you trust in non-technological solutions being an important part of how we deal with this. For example, maybe we actually already know how to make PPE, and it’s just about building large stockpiles and getting a good monitoring system in place that uses existing technology. And if that’s the case, then yeah, you might want more time to build that infrastructure and you would prefer that route.

On the other hand, maybe you think that stuff doesn’t work that well. Like you can compare the efficacy of lockdowns and social distancing to mRNA vaccines, and you think the end solution is going to be technological, just because we’re bad at the other stuff, maybe.

Luisa Rodriguez: The social stuff, yeah.

Matt Clancy: In which case, maybe you want to trade getting it here earlier for the possibility of getting a technological solution.

I think one other interesting wrinkle is just that if you look at the forecasts in the Existential Risk Persuasion Tournament, they do seem like the domain experts do seem to think the risks will fall in the long run. But we don’t know if that’s for one reason or the other.

Luisa Rodriguez: Oh you don’t? They haven’t said something like, “We think it’s going to go up because of X, but then we think it’ll go down because we’re going to solve this somehow.”

Matt Clancy: Not that I could see in the tournament results. Obviously if you asked them, they would have views. So I don’t want to lean too hard on the fact that they… I think it’s notable that they do seem to think that there is going to be something like an end to this, or like a reduction in this peril risk at some point in the future. But I don’t know if they think that’s because of better preparation by society or because of technological fixes.

Are we already too late to delay the time of perils? [01:42:38]

Luisa Rodriguez: Yeah. That’s just really interesting. I guess when I’m thinking of other questions that maybe affect your bottom line results, one question you could ask is whether there’s a point at which we’re actually just too late to pause the time of perils. So in theory, the time of perils might be decades from now, and pausing science would in fact meaningfully delay its start. But I guess it also seems possible, given the time it takes for technology to diffuse and maybe some other things, that we could just already be too late.

Matt Clancy: That’s right, yeah. That’s another additional analysis we look at in the report: that, as you said, we could be too late for a lot of reasons. One simple scenario is that all of the know-how about how to develop dangerous genetically engineered pathogens, it’s sort of already out there, it’s latent in the scientific corpus, and AI advances are just going to keep on moving forward whether or not we do anything with science policy. It sort of is driven by other things, like what’s going on in OpenAI or DeepMind, and eventually that’s going to be advanced enough to sort of organise and walk somebody through how to use this knowledge that’s already out there. So that’s one risk: that actually it hasn’t happened yet, but there’s nothing left to be discovered that would be decisive in this question.

Luisa Rodriguez: Would that mean we’re just already in the time of perils?

Matt Clancy: The time of perils is this framework of how you estimate the probability and if you would want to bring that forward. Technically we wouldn’t be in it, but we would be too late to stop it. It’s coming at a certain point. And I guess this is also a useful point to say that it doesn’t have to actually be this discrete thing where we’re out and then we’re in; it could be this smooth thing, but it gets riskier and riskier over time. And this whole paper is sort of like a bounding exercise. This is a simple way to approach it that I think will tend to give you the same flavour of results as if you assume something more complicated.

Luisa Rodriguez: Sure. I guess just being in this time of heightened risk doesn’t guarantee anything terrible. In theory, we still want to figure out what the right policy is. So if we’re in that world where we’re too late to pause science, what exactly is the policy implication from your perspective?

Matt Clancy: I mean, I think it’s like, basically there’s this quote from Winston Churchill, “If you’re going through hell, keep going.” So we’re in a bad place, and there’s no downside to science in this world, right? If you think things can’t get worse, if you think that the worst stuff has already been discovered and it’s just waiting to be deployed, well, then you’ve got to kind of bet on if we accelerate science, there’s all these normal benefits that we would get. It’s not going to get worse, because that die has already been cast, and maybe there’s a way to get to the other side and so forth. So I think that’s the implication. And that’s it’s a bad place to be, but at least it’s clear from a policy perspective what the right thing to do is.

Luisa Rodriguez: Yeah. Do you have a view on whether we are, in fact, already too late?

Matt Clancy: I’m not a biologist; I’m an economist. But it’s another one where I’m like, I don’t know, 50/50.

Luisa Rodriguez: OK, Jesus. All right, well, that’s unsettling, but we’ll leave that for now.

Domain experts vs superforecasters [01:46:03]

Luisa Rodriguez: We’ve actually touched on this a bit, but I want to address it head on now. Whether you use those forecasts from domain experts or superforecasters feels like it could make a really big difference to my own personal bottom line takeaway. Which do you put more weight on, domain experts or superforecasters?

Matt Clancy: So again, not a biologist, but I think that there are at least some kind of outside-view reasons that I trust the superforecasters a little bit more than the domain experts. I talk about three in the report.

One is that the [tournament] report had this problem: how do you try to provide incentives for people to do good forecasting for events that are going to occur way in the future beyond what they care about? And one thing they tried was rewarding people for being able to predict what other forecasters would think: if you’re a superforecaster, what are the biosecurity experts going to think about this problem? And vice versa.

And one interesting finding that they got from this report is that people who are better at that task tend to be more optimistic by actually quite a lot when you compare the extremes about existential risk stuff. So that’s one reason why I lean towards the superforecasters, because they’re the more optimistic group of the two. It seems that being able to forecast what other people think, which we think might be a proxy for understanding the arguments well, is correlated with being more optimistic. So that’s one reason.

Luisa Rodriguez: OK, so that’s one.

Matt Clancy: A second reason is that there’s this weird optimism or pessimism correlation across lots of domains. So if you’re worried about existential risk from biology, from extinction stuff, people who had that view tended to also be more worried about AI risk, nuclear risk, but even what they called “non-anthropogenic” x-risk — which is stuff not caused by people, so supervolcanoes or meteorites hitting. So basically it just sort of seems like some people are optimistic about our future, and they just think all of those risks are smaller, and some people are more pessimistic and they think all of the risks are greater than they are.

So that doesn’t tell you who’s right and who’s sort of biassed. But are the superforecasters excessively optimistic or are the domain experts expertly pessimistic? I can see reasons why I would lean towards thinking the domain experts are more pessimistic.

So the superforecasters: why would they be disproportionately optimistic? Maybe all of society is, and they’re pulled from society. But this is a group that receives feedback on forecasts. If they have a general optimism bias in their life and on other forecasts or whatever, they’re a group that’s going to get feedback and an opportunity to at least correct that bias.

Domain experts aren’t necessarily in the forecasting game all the time. They’re not necessarily going to be getting that kind of feedback. And there’s this plausible selection story, where if you are worried about these things, you might choose to learn about them, and then become an expert to try to stop them. To be clear, I think it’s good that people try to stop them, and it makes sense that the people who are worried about them would be the people who do. But that’s a reason to maybe think that, you know, why I trust the superforecaster forecasts a bit more.

And the third reason is this really scant evidence: we have two studies that are kind of related to the track records of these two different groups. One of them was this 2014 report by Gary Ackerman that described this research project to get domain experts — who in this case were futurists and biosecurity experts — to forecast the probability of biological weapon attacks over the period 2013 to 2022, so this 10-year period. So 2022 has now resolved, and we can see how did they do.

The problem is that there was basically like one attack in that period. And if you add up their lower bound — they gave a range of estimates: lower bound, upper bound, middle, and so forth — their lower-bound estimates add up to 1.23 attacks, forecast over 10 years. So their lower-bound estimate is about right. The median midpoint probability is closer to three attacks they would expect over that period; their lower bound is more correct than their median value.

And it just so happens that in domain experts, the domain experts also have bounds. And the lower-bound estimates for the domain experts that were asked in the Existential Risk Persuasion Tournament are close to the median superforecaster estimates. So that’s one reason.

But you know, it’s like one attack. If there had been one more attack, suddenly the results would change. So you don’t want to put too much weight on this.

Luisa Rodriguez: Sure, sure. But some weak evidence.

Matt Clancy: Yeah. Then the other one is one of my colleagues, actually a new colleague, David Bernard, did a dissertation chapter where they looked at how do superforecasters and domain experts forecast stuff out over several years? And in this case, the domain experts are not biosecurity — we’re not talking about x-risk, anything like that — we’re talking about economic development. But that’s the data he had: there are all these different economic development interventions, and they ask academics who are experts in that field, “What do you think is going to be the impact of this three years later, five years later?” I think maybe up to nine years later is the farthest out they go.

And they compare that to superforecasters. And basically the superforecasters score a bit better by sort of standard scoring measures. But the domain experts don’t do terribly either. Their midpoint is accurate, but they’re just too confident about things, whereas the superforecasters are a little bit more, they have a wider range — maybe because they’ve got more experience.

So anyway, that’s the third. And like I said, not a tonne to lean on, but it is something we looked into. And I think you add that to the intersubjective probability stuff and this correlation of pessimism or optimism, and that leans me towards thinking that if I had to pick one, I think the superforecasters are more right. I don’t feel super confident about it. I wouldn’t bet the farm on it or whatever. But you know, three-in-four chance that they’re the ones I go with.

Luisa Rodriguez: OK, three-in-four chance. I wish I’d put a number on it in advance to check what I would have done, but I guess we don’t have to pick. It seems like we can use them as a range, or people could come up with their own range that puts some weight on domain experts and some on superforecasters.

Matt Clancy: That’s true. One interesting thing is that some of the superforecasters are more risky. And so if you go to the 90th percentile — the 10% who think the risks are highest — they’re pretty close to the biosecurity average. And if you look at the lower bound on the biosecurity experts, like I said, they’re not too far off from the superforecaster median.

Luisa Rodriguez: Right. So it’s not even a perfect distinction. But yeah, interesting.

What Open Philanthropy’s Innovation Policy programme settled on [01:53:47]

Luisa Rodriguez: OK, so you lean towards superforecasters. This just seems like such an important consideration that if someone, maybe like me, is still pretty divided, I’m really sympathetic.

I think overall I’d put more weight on the superforecasters too. But I’m also like, I don’t know, maybe kind of a lot of what makes them really good comes from the fact that they’re predicting things within the window of things we’ve seen recently. And maybe the kinds of things we’re talking about are just not the kinds of things that we’ll see recently. But that doesn’t mean they don’t happen; it just means that they happen so seldom that if we make predictions based on the recent past, we won’t see them coming. That seems plausible to me.

So what should we do if we’re just really uncertain about which to pick? Because in some cases they’ll point to really different outcomes.

Matt Clancy: Yeah. So the answer I come up with is sort of like a cheat, because it’s like a secret third option, which is, where do I come down on this? I’m actually very confident that faster science is a good thing. I think that’s because, one, I think the superforecaster forecasts are probably more likely. But also, even if the other guys are right, the biosecurity experts, which I think is still reasonably likely, I think there’s a decent shot that faster science would actually still be good — either because we’re too late, or more optimistically because they can develop these solutions faster.

So for those reasons, I think I’m really pretty confident that faster science is a good idea. But also the secret third option is that we can do stuff to mitigate downside risks, and those are very important. So we can kind of have a two-pronged strategy. And in fact, that’s sort of what I’ve settled on.

And what we settled on in the Innovation Policy [programme] is that we should, in general, try to accelerate scientific progress. Keep an eye out for projects that seem like they have extreme downside risk, because those are the ones that are really bad. But also, you can have a separate set of projects that are trying to reduce technological risk.

So an analogy is you can think about climate change: you could try to mitigate climate change by slowing economic growth, because the two are maybe connected; or you could try to reduce climate change through targeted renewable energy policies or carbon capture stuff, and in another basket of your policy space, have stuff that you think makes economic growth better, like R&D policy, for example, and so forth. So that’s kind of the approach that I think is my ideal when I come out from this report.

Luisa Rodriguez: Is that kind of close to what you would have thought was reasonable before you’d done this?

Matt Clancy: Yeah, I think that before we had started this project, I really was unsure what we would find. You know, when you’re building a model and you’re going to put parameters into it, they could give you different answers. So I think it’s possible we would have ended up in a space where Open Phil is full-on on differential technological development, trying to focus on that. I think it was never super likely. There’s always a small chance, but it’s not that likely we would just completely put these issues to bed, because when we started, there were these sort of compelling arguments that people had and so forth.

But anyway, an outcome from this report is — and not just this report, but other discussions — that everyone at Open Phil, for example, is not necessarily confident like me that faster science is net positive. But I think most people now think that faster science plus doing some work to try to mitigate risks is a net positive. So that bundle is a good thing.

Luisa Rodriguez: Cool. Actually, that leads to my next question, which was just: how has this work been received by your colleagues? Have people tended to agree with you? Have they been sceptical?

Matt Clancy: Like I said, disagreements remain, but I think people don’t think that accelerating science is probably a huge negative, if it is negative, and if you add in some of this stuff to try to make stuff safer… So now our mantra, I think, is sort of like “safe acceleration.” So you check the crosswalk for pedestrians and then you hit the gas.

Luisa Rodriguez: OK, nice. Are there any other ways the report has kind of influenced your strategy, or is the core thing safe acceleration?

Matt Clancy: I think that’s the core idea, I’d say. But while we were writing this report and before we had finished it, we had this very cautious kind of approach: we were going to, by default, assume grants could be bad just while we were working on this, and had to be convinced that they couldn’t be. So, for example, we focused a bit of work on social science research, because I’m a social scientist and nobody’s that worried that people will misuse social science for evil. After the report, like I said, I think we now are more by default, kind of not worried. Faster science is probably good. Send us your ideas if you have ideas for doing that.

But we basically now consult with the biosecurity/pandemic team about what we’re thinking, where we’re going, and give them a chance to flag stuff that might have a big unexpected downside risk that I’m missing because that’s not my area of expertise. And like I said, we also have this new sort of substrategy that we launched that we haven’t done too much in yet, but it’s going to try to look for opportunities in this space too.

Luisa Rodriguez: Cool. Given all that, what kinds of levers do you have available to you for meaningfully improving science? What’s going wrong when people try to do science that could go better?

Matt Clancy: Yeah. This is going to sound stereotypical coming from an economist, but think about the contrast between how we choose what science to fund versus how we choose other sectors and businesses to receive funding. In other sectors, you’ve got a whole advanced banking system, there’s an entire field and discipline of finance, people’s bonuses are tied to how well they make investments — and tonnes of people pour tonnes of energy into figuring out how to do this.

Science is super high return and super valuable, and lays the foundation for a lot of future technological progress. But the way that we choose what science to fund is comparatively less developed, we might say. You know, it’s anonymous peer review panels, people whose performance is not necessarily tied to how well they do; there’s lots of concern among people that they’re excessively risk averse, and maybe things can get captured.

So basically, one area is: are there better ways to do this? It’s not a given. Maybe we actually have landed on the best you can do for a really hard problem. But we can try alternative ways, and we can try to study how well they work. That might be one class of problems.

Another class of problems that I kind of alluded to earlier is that science is increasingly specialised, and getting bigger and bigger all the time. So it’s hard for people to keep up on the literature; it’s hard for people to keep up even on their own narrow discipline. And that could be a problem, because a lot of the best ideas are maybe cross-pollination, where somebody develops an idea and it gets used in an unexpected context, and you get kind of a breakthrough that way. And if everybody is just struggling to keep up with their own specialty, maybe it’s hard to understand and keep up with what other people are doing. That’s one reason we’re sort of experimenting with these living literature reviews, which try to synthesise the literature and make it accessible to non-specialist audiences. But there’s lots of other things you could potentially try.

A third topic is maybe the incentive to produce reliable, replicable research. There’s the whole replication crisis. Science provides really strong incentives to push the frontier, discover things that nobody has known before. You get a lot of sort of social credit and prestige, and maybe you can get grants and publications from doing that. It provides less strong incentives to reproduce and make sure other people’s work sort of stands up and to publicise when it doesn’t, and so forth. So we make some grants that try to make progress on those issues too.

A final one that’s often not thought of as a science policy issue, but is actually closely related, is immigration policy — especially high-skilled immigration, because the talent pool for science could come from anywhere in the world, but the resources to do science and to learn and to sort of network with scientists is not as evenly distributed. So you can either try to distribute the scientific resources more equitably — which is another thing I’m interested in looking at in the future — or you can try to reform immigration policy to let people with the aptitude and interest in science go to where the science can be done.

Luisa Rodriguez: Cool. What are some examples of things that you think might make it work better? You’ve just listed a bunch of problems, and I imagine the answer is to solve those problems. But are there any specific ideas for how, or things that people are already doing maybe?

Matt Clancy: Yeah. So one grantee that I like is the Institute for Replication, which is looking to replicate a bunch of economics articles that are published in top journals or a set of journals. And actually, they’ve expanded to lots of other disciplines beyond just economics, but I’m biassed because I’m an economist.

Anyway, the thing that’s interesting about them is, rather than cherry-picking and having this ad hoc replication system — where somebody, when you get replicated, suddenly your heart freezes up because you think, “What do they have against me? Why are they coming for me?” — it’s just like, if you publish in these top journals, we want to make it the norm that somebody’s going to try to replicate your work. And replication in this context basically means take your data and think about what is the right approach, and then make sure they code it up and try it themselves, and variations on that. We’re not going to rerun experiments, because we’re usually using government data in this context.

But anyway, it’s making it a norm. It’s not something that they’re out to get you. So you get a better baseline of what’s normal and what’s not normal. And also, if you’re a policymaker and trying to devise better policy based on the latest research, if you know that everything that’s in this set of top journals gets replicated, then you can go look for it and see did it replicate? And that just becomes a norm too. So that’s one example of a solution that we like.

I think they also have a nice system for moderating the interaction between the replicator and the original authors, which could be contentious if it’s in this previous kind of thing. And that could make it a disincentive to do, because the only people who want to be involved in replicating stuff are people who have kind of an appetite for having a conflict or something. So that’s one example of a grantee we like.

Luisa Rodriguez: Cool. Yeah, that is really exciting.

Matt Clancy: Shout out to the Institute for Replication.

Luisa Rodriguez: Huge shout out. Now you’ve just piqued my interest. If you had another example, I’d be super curious.

Matt Clancy: Yeah. So I don’t have another example as crisp, but I think a general approach that we’re very excited about is finding partners who fund or in some ways run the infrastructure of science. Maybe it could also be like journals or so forth. And finding partners who are interested in experimenting with the way they do things, and pairing them with social scientists and economists so that they can learn carefully how it works.

I mean, science has this problem where it’s hard, I think, to casually infer whether something is a good idea or not — because science takes a long time for results to be clear how they’re viewed by the community. Outliers matter a lot, where sometimes maybe you only care about getting one home run, and you’re willing to take a lot of losses — but that means you need a very big sample to sort of tease out the home runs. So I think you kind of need to partner with social scientists and think about this in a bigger-picture way than just like, “We’re just going to try something and see if it works.” You might have to actually work with people, and then we can share that information with other people. And if we aggregate up a lot of other people trying similar stuff, we can get the big sample sizes we need to improve the system. So I think that’s the approach that I would love to promote.

Explosive economic growth [02:06:28]

Luisa Rodriguez: OK, let’s leave that there for now. So we’ve been talking about metascience, and metascience is kind of about trying to accelerate scientific and technological progress. But maybe the biggest thing that we can do to accelerate scientific progress is to solve artificial intelligence. Then we could have basically a bunch of digital scientists doing all the science we want.

This idea is actually related to explosive growth, which is the subject of an interview we did with Tom Davidson, one of your former colleagues, where he made the case that developing AGI — so AI that can do 100% of cognitive tasks at least as well as humans can do — could cause explosive economic growth, leading us to a honestly unrecognisable world. But I think you’re more sceptical of this. I think that you think that there’s a 10% to 20% chance that this happens, and that’s from a writeup that you’ve done that we’ll talk more about.

Just for context, we’re talking really specifically about a world where AI can do most or all the things humans can do. So things like running companies and all the planning and strategic thinking that comes along with that, designing and running scientific experiments, producing and directing movies, conducting novel philosophical inquiry. And the argument is that AI at that level will cause GDP to grow at least 10 times faster than it has over the last few decades. So potentially 20% — which would mean we’d get 90 years of technological progress in the next 10 years, which really is kind of mind-boggling.

A big reason that you’re sceptical of this is that the people who think explosive growth is likely are putting a lot of weight on the fact that economic models predict explosive growth if you just plug in the fact that when you have a much bigger AI labour force, then those models predict that we just get a bunch more growth — because right now we’re kind of bottlenecked by not having more people that can do more things, but the AI labour force could be huge. Can you explain that core argument in a bit more detail, in case I’m missing anything?

Matt Clancy: Sure, yeah. To clarify a couple of things at the beginning: one, this is nothing to do with the report anymore. This is now a different interest I have at Open Phil, and I find it pretty interesting. And I’ve chatted with Tom, for example, a lot about this, and it’s great. I think it’s really interesting.

Also, just to clarify, I’m not like, “AI doesn’t matter. It’s no big deal.” I’m like, my modal guess, or my average, is that yeah, it will improve productivity, but I think I’m a lot more sceptical than some people about this explosive growing at 20% per year type thing. I’d be like, 5% per year. Wow, this is crazy. But 20% per year, I’m a lot more sceptical.

So let me lay out an economics model that I think is really helpful for thinking about this. It’s mathy, but I’m going to give you the simplified, mostly non-mathy version that I think gives you the intuitions for the different ways that AI could matter.

Just imagine that we’re trying to invent something new. And inventing has 1,000 steps that need to be completed. There’s 1,000 different tasks — everything from discovering the fundamental scientific theories that underlie the principles that this thing relies on, to a bunch of testing, and then there’s also stuff like fine-tuning it, figuring out how to manufacture it, and even diffusing it to people. Because productivity growth is when people are using technology; it’s not when the technology exists. It’s when people are using it to do things in the world.

So we’ve got 1,000 tasks to get there and we’re going to assume that you’ve got to do all this stuff, but you can maybe substitute to some degree some of them. So you can do maybe the scientific theory really well, and then you don’t have to search as much, or you don’t have to do as good a job on downstream parts. But there’s some degree to which all of this stuff has to get done, but maybe to different extents. For simplicity, we’ve got 1,000 different tasks. Let’s imagine we’ve got 1,000 scientists and inventors, and each one of these takes about a year on average. So we’re going to get it all done in a year, because they can all work in parallel at the same time.

So that’s our first step. But now we’re going to add one wrinkle, which is that invention gets harder and harder. And this is a pretty robust finding, that it takes more and more researchers to increase productivity by a certain amount. So let’s just, for simplicity, say that every time you invent something, the next invention takes 10% more like cognitive effort. So you’re going to either need to spend more time thinking about each of these tasks, or you’ve got to put more people on it, and then they can team up together and so forth. So if we wanted to invent one invention every year, we actually have to increase our labour force by like 10% per year to sort of offset this “things getting harder” stuff.

But there’s another route we could take besides just getting more people, which is that we could get robots to do some of the jobs. So suppose that we’re able to figure out how to automate 5% of the jobs that humans are doing, and we can put robots on those tasks instead. That has an interesting effect, which is that 5% of the jobs people were doing, robots are now doing. So those people are kind of like free agents; now they can do something else. In particular, they can go work on the other stuff. So that gives us 5% extra people to work on the other tasks.

And now, if we add 5% to our overall labour force, we’ve automated 5% of the tasks. That’s another way that we can get to an extra 10% of people working on the stuff that only humans know how to do. And that again, gets us to one invention every year. So in the first step, it’s because we’re just increasing the population of scientists by 10% a year. In the second, we increase the population by 5% per year. But we keep shifting people away from stuff that they used to have to do — like maybe they had to do their regressions by hand and the statistical analysis, and now a machine does it and so forth.

So that’s actually, I think, a pretty good description of what has been happening for most of the 20th century. We have been steadily figuring out how to hand off parts of our cognitive work, or actual work, to machines. Like we invent Excel, Word, Google Scholar, lots of stuff, and people focus on different parts of the invention process, and they don’t spend as much time doing regressions as maybe they used to do. And that is consistent with steady growth and also increasing the population of researchers working on stuff and so forth.

So now you can say, this is your kind of core model. How does AI fit into this? Why would AI in this framework lead to much faster growth?

And there’s kind of three options you’ve got. One is, you could just flood the zone with lots of robots on the tasks that robots can do. And remember, we assumed that there’s a bit of substitutability before these tasks. So maybe you can put 1,000 times as many digital minds working on the things digital minds are good at. And maybe that means you only need to do the other stuff 10% as well. Well, that would mean the humans could do their part in one-tenth the time, and they’d invent at 10 times the pace. And maybe this works because AI is really scalable: once you train a model, you can copy it and deploy a lot of effort on this. So that’s one option, this flooding AI effort onto tasks they can do.

A different route you can go is that you could have this big surge in what share of the tasks the AI can do. So instead of figuring out how to automate 5%, maybe they automate 90% of the stuff humans are doing, and that frees up 90% of the people to work on the last 10% of tasks. It’s kind of like you’ve 10x’ed the labour supply for that small share of things that only humans can do. Again, maybe they finish 10 times as fast, and that’s how you can get 10 times the speed of growth or invention. Maybe this works because we think AI is just super general: it can figure out anything, and so it can do lots of tasks.

And then the third option is if you can actually just eventually automate all of those tasks, so you don’t need humans at all. Then you can get these arguments where this can lead to sort of a positive feedback loop. So imagine the thing that we’re inventing — I’m adding an extra step in our thought experiment — is a robot factory. And every time we make a new factory, we discover a new way to do it, and the factory is 20% more efficient than the last one. And that means basically now we can build 20% more robots than we had. We only needed 10% more because invention gets harder. So we’ve got more robots than we need, and we can get it done even faster than we did last time. And then we get to the next step, and we have another 20% and we can get it done even faster. And so you just get this, like, acceleration until you reach 10%, or like 10 times the speed you were at and so forth.

So those are the three main options: flood the zone on the tasks robots can do with just tonnes of cognitive labour, a surge in the share of tasks that are automated, or just doing everything.

And the last thing is just that they can kind of interact with each other. So maybe the thing that they get better at is automating; maybe you’re inventing ways to automate better. And then there’s also maybe stronger incentives to finish up the last rump of tasks that humans can do, because then you could get this positive feedback thing that would be really valuable. So that’s the argument for explosive growth.

Luisa Rodriguez: Right. And then my understanding is that you, at least, you’re more sceptical of it. You don’t put no weight on it, but you think it’s less likely than at least some of your colleagues. Why is that?

Matt Clancy: I basically think that it’s a possible scenario, but I think that people underweight frictions that would cut against it. And remember, I think that it is a useful direction for the direction we’re going, but less useful for predicting the magnitude of like 10 times growth.

Luisa Rodriguez: OK, so you’re like, yes, AGI is going to be a big deal. People need to update toward it’s going to change the pace of economic growth in some way. And it’s going to get a little bigger. But the claims that it’s going to be explosive are the thing that you’re like —

Matt Clancy: Yeah, it’s possible, but I think underrating a lot of things. So to take one, this first option, which was like, you flood the zone with cognitive labour on certain tasks. We kind of have run that experiment. There were tasks that computers can do. Moore’s law led to just this astronomical increase in chips and essentially cognitive power on those kinds of tasks. It didn’t lead to a noticeable, large increase in the rate of economic growth.

And that kind of implies that there are really strong limits to how much you can substitute between these tasks. So that means you get bottlenecked on the things that only humans can do, even if you throw 1,000 times or a million times as much labour at the other stuff. And a million or a billion are kind of the magnitudes we’re talking about with this. In terms of Moore’s law, it doesn’t translate into less need to do the other stuff. So that’s the first option.

But there were other routes to explosive growth, and one of them was this surge in automation that we can replace humans in 90% of the tasks or something. One challenge there is if it continues to be the case that you can’t substitute very well, and the stuff that humans need to do really is a bottleneck, well, even though you have all this free labour or freed labour that can go work on that, it might take them time to be retrained and go work on that. Tasks are very different, especially if you’re imagining moving from a scientist to moving to more like the entrepreneur stage of the innovation set of tasks. Those are pretty different skills.

But I think, more importantly, the world is just really full of details and frictions that are hard to see from 30,000 feet, but they’re just omnipresent throughout the economy — and they’re not all the kind of things that can be bulldozed through with just more intelligence.

Luisa Rodriguez: Yeah. Can you give some examples?

Matt Clancy: Yeah. So factors that can matter is like, it might take time to collect certain kinds of data. A lot of AI advances are based on self-play, like AlphaGo or chess. They’re sort of self-playing, and they can really rapidly play like a tonne of games and learn skills. If you imagine what is the equivalent of that in the physical world? Well, we don’t have a good enough model of the physical world where you can self-play: you actually have to go into the physical world and try the thing out and see if it works. And if it’s like testing a new drug, that takes a long time to see the effects.

So there’s time. It could be about access to getting specific materials, or maybe physical capabilities. It could be about access to data. You could imagine that if people are seeing jobs are getting obsoleted, then they could maybe refuse to share and cooperate with sharing the data needed to train on that stuff. There’s social relationships people have with each other; there’s trust when they’re deciding who to work with and so forth.

And if there’s alignment issues, that could be another potential issue. There’s incentives that people might have, and also people might be in conflict with each other and be able to use AI to sort of try to thwart each other. You could imagine legislating over the use of IP, intellectual property rights, and people on both sides using AI to not make the process go much faster because they’re both deploying a lot of resources at it.

So the short answer is, if there’s lots of these kinds of frictions, and I think you see this a lot in attempts to apply AI advances to any particular domain. It turns out there’s a lot of sector-specific detail that had to be ironed out. And there’s kind of this hope that those problems will sort of disappear at some stage. And maybe they won’t, basically.

One example that I’ve been thinking about recently with this is: imagine if we achieved AGI, and we could deploy 100 billion digital scientists, and they were really effective. And they could discover and tell us, “Here are the blueprints for technologies that you won’t invent for 50 years. So you just build these, and you’re going to leap 50 years into the future in the space of one year.” What happens then?

Well, this is not actually as unprecedented a situation as it seems. There’s a lot of countries in the world that are 50 years behind the USA, for example, in terms of the technology that is in wide use. And this is something I looked at in updating this report: what are the average lags for technology adoption?

So why don’t these countries just copy our technology and leap 50 years into the future? In fact, in some sense, they have an easier problem, because they don’t have to even build the technologies, they don’t have to bootstrap their way up. It’s really hard to make advanced semiconductors, because you have to build the fabs, which take lots of specialised skills themselves. But this group, they don’t even have to do that. They can just use the fabs that already exist to get the semiconductors and so forth, and leapfrog technologies that are sort of intermediate — like cellular technology instead of phone lines and stuff like that. And they can also borrow from the world to finance this investment; they don’t have to self-generate it.

But that doesn’t happen. And the reason it doesn’t happen is because of tonnes of little frictions that have to do with things like incentives and so forth. And you can say there’s very important differences between the typical country that is 50 years behind the USA and the USA, and maybe we would be able to actually just build the stuff.

But I think you can look at who are the absolute best performers that were in this situation. They changed their government, they got the right people in charge or something, and they just did this as good as you can do it. Like the top 1%. They don’t have explosive economic growth; they don’t converge to the US at 20% per year. You do very rarely observe people growing at 20% per year, but it is like always because they are a small country that discovers a lot of natural resources. It’s not through this process of technological upgrading.

Luisa Rodriguez: Is that true of Asian countries that I feel like have had really fast economic growth, and it seems at least partly driven by technology, like adopting new technologies?

Matt Clancy: Yeah, I think these are examples of these countries that are performing at sort of the best that you can do, and they’re growing at 10% per year. Maybe one year at like 15%, but not a continuous 20% per year kind of thing. So in fact, if you wanted to say 10%, I’d be like, well, you know, we’ve got some precedent for that. Maybe this is semantics, but 20% is the number that is sort of thrown around, and it’s just not… I think it’s very ambitious, let’s say that. So that’s another example of just that I think these frictions matter.

And one last point is that back in the early days of economics, like Bob Solow and other people were coming up with sort of the first models of technological progress. They had these models that made predictions like this. They basically said countries that are really capital poor, like there’s a declining marginal product of capital. And so these countries are going to get a lot more investment and they’re going to converge and grow at a faster rate.

And you could have said at that time, you could be Matt Clancy at that time, and be like, “Actually, there’s all these frictions that make that kind of thing hard to pull off.” And somebody could have said, “But Matt, you’re underestimating that the order of magnitude difference in the capital levels of these countries is so massive that this is going to overwhelm any little frictions.” But it turns out that was wrong. The frictions really did add up, and a small number of countries did grow fast, but they didn’t grow necessarily as fast as being predicted might be the outcome from AI — which is also, I think, based on a simplified economic model of the world.

Luisa Rodriguez: OK, so let’s say that there are these frictions, and they’re the kinds of things that meant that even though there are some countries in Asia that got to learn about a bunch of new technologies and implement them kind of all at once, they saw big growth, but not huge growth, and they didn’t see it super sustained.

I think the Tom Davidson on my shoulder — who has made this big case for explosive growth on the show before — is like, if that’s the case, then the priority of AI systems, or humans making AI systems to optimise for different things, is going to be to iron out those frictions. So I think Tom argued that for experiments that involve humans, we’ll do a bunch of them in parallel. They’ll be so good at optimising these processes that we actually can do much better than you’d imagine based on how long things take and the fact that there are just biological processes that have to happen.

If tonnes of resources are just thrown at the bottlenecks in particular, do you still feel really sceptical that those bottlenecks will just really slow everything down?

Matt Clancy: I mean, I’m unsure. That’s why I said in one interview, 10% to 20% chance that this all happens, which I think is like this weird middle ground between people who are really enthusiastic people, and really sceptical.

But I do think I have the experience of running a programme where we think there are these important problems with how things are going and it’s hard to just convince everyone to fix them, you know? I think that we have long had incentives to solve various problems that are very important and that the market is really aligned on, and sometimes these are just hard problems; we don’t instantly solve things that there is a strong incentive to solve. That is not a natural property of the world, I would say.

Luisa Rodriguez: OK, so if you just wait long enough, do you eventually think we get this? Because maybe part of my belief is that these are really hard problems to solve, but having this much bigger labour force with a higher ceiling on how intelligent the kind of minds working on these problems can be, at some point there will be breakthroughs that mean that we can solve bottlenecks that we’ve not been able to solve before. And so I don’t have that much confidence in any particular timeline, but I feel like I have more confidence in the… Obviously, I don’t think it’s an inevitability, but some high likelihood on this thing happening eventually.

Matt Clancy: So I think that I share the view that it seems very plausible that eventually we figure out how to automate lots of this stuff, and machines are doing it all and doing it maybe better than people. And I think that if we get into a world like that, we’re going to be in a world that’s like, probably a lot better off and richer and so forth.

I think the thing that I disagree a little bit about is whether that will imply we end up with explosive growth at that point. Because if it takes a long time to get there if these are hard problems, by the time we get there, we’re a rich society. And rich societies have the luxury to put a lot of value on things that are not survival relevant or so forth. We may think that lots of the characteristics of human beings are the things that we care about. Like, I want to be talking to something that I know is sentient. You know, think about tonnes of people who went to go see Taylor Swift in concert. They can stream all of her stuff anytime they want, but they like the in-person, human experience, and they paid a lot of money for that. And a society that gets richer might think like —

Luisa Rodriguez: We still value that.

Matt Clancy: Yeah, that’s true, but we just value the stuff that the robots can’t do so well. And that’s a good place to be. It means you’re very wealthy and have the luxury of indulging in these kinds of preferences that are obscure. But it can still mean that you don’t get explosive economic growth, because maybe it’s just that the kinds of stuff that explosive economic growth could give us is not the kind of stuff that people are that interested in getting. They don’t want to build islands that look beautiful. Or maybe they want to a little bit. They want to be on the island where all their friends are, and there’s limited space or something. I’m not sure that’s a great example, but that’s kind of the flavour of it.

Luisa Rodriguez: Sure, yeah. So let’s say you’re thinking about the next two to five years. What do you imagine the world looking like if we were headed toward more explosive growth? Maybe you don’t even think the next two to five years is a plausible time span to start getting indicators of that. But what would be early signs, in your view, that the explosive growth thing is more plausible than you think now?

Matt Clancy: I think that my theory is based on frictions to adoption. So if it turns out that we’re just seeing adoption happen really quickly in lots of places — not just in one place really quickly; I imagine that there are going to be some places where there’s not much fiction and then those places might adopt it quickly — but if you see it across a wide range, that’s something that I’d be like, quantitatively, maybe these things are there, but they just don’t matter that much. So that would be one view.

Another thing is that a general trend in the economics of innovation is just that science is getting harder. So people are assembling bigger and bigger teams of specialists to solve problems, and you can just see the number of coauthors on papers keeps going up. If AI is substituting for an increasing share of tasks, one very concrete indicator is the size of teams going down in science, because people are just offloading more of their coauthor duties, basically, to an AI.

Those are two examples of things that I might see that I’d be like, maybe this is coming. The first one — adoption across lots of sectors — is maybe a better indicator than the second, which is sort of specific to idiosyncrasies of science.

Luisa Rodriguez: OK, so those are things that you might see going forward that might make you think explosive growth is more likely. Is there anything that you’ve learned since you first started thinking about the topic of explosive growth that’s kind of changed your thinking about it in general?

Matt Clancy: So I’ll give one example that’s made me more confident in my original views, and one thing that’s made me less confident.

So on my original views, one example that’s often given is that we have a precedent for these rapid accelerations in the dawn of agriculture and in the Industrial Revolution. So let’s focus on the Industrial Revolution, where we went from growth plausibly increased… Like there was a long run average over the preceding several hundred years, maybe it’s like 0.1%, and after the Industrial Revolution, it went 10 times faster, to 1%. Actually, I think I looked this up, and it went to 1.6% on average over a longer time period. So this is often given as proof that these kind of discontinuities can happen.

I think one thing that should give you a little bit of hesitation about that is if you look at our best estimates for what was economic growth back before the Industrial Revolution, from 1200 to 1600 or so — and I’m using Angus Maddison data, who’s this economist who had these really long runtime series — how often did you observe growth above 1% per year, 1.6% per year? Which is the stuff that we observe in the era of the Industrial Revolution. The answer is it’s like one-third of the time. It happened all the time that you saw this fast growth. You even saw, with relative frequency, very long runs where average growth for maybe 30 years averaged Industrial Revolution levels. It’s just that it gets knocked back; they didn’t persist for hundreds of years. You’d get fallback and things would go up and down a lot. So it’s noisy. Today it’s not the same, where sometimes people grow at 20% but then it gets offset years later by 5% or 1% or -10%. So I think that’s one reason to think that this is a more unprecedented prediction than the past, so that’s one thing that’s updated me in favour.

Now on the other direction, I don’t know if this is so much about explosive economic growth, but just the advance of large language models. Before I joined Open Phil I was working full-time on writing New Things Under the Sun, which is this living literature review, summarising and synthesising literature. I am glad that that is not my full-time job anymore, because I don’t know where that’s going in the next couple years. This grants programme that we run does give people financial support to write them, but with the general idea that people are going to do it part-time and they’re going to have another role. Because, you know, it seems like that job is maybe getting automated in the near future. So yeah, I’ll keep my eyes on it. It’s not there yet, but coming down the road I think you can kind of see it.

Luisa Rodriguez: Makes sense to me. I also worry about my job.

OK, so if you want to learn more about different sides to kind of the debate on explosive growth, I really recommend this article that you wrote with Tamay Besiroglu in Asterisk magazine called “The great inflection? A debate about AI and explosive growth.” It’s kind of written as a dialogue between the two of you. And if you prefer audio, Matt and Tamay recorded themselves performing the parts that they wrote for themselves as part of this dialogue, and that’s available on New Things Under the Sun.

Matt’s favourite thought experiment [02:34:57]

Luisa Rodriguez: For now, we’ve got time for one final question. Matt, what is your favourite thought experiment?

Matt Clancy: I’ll stick with one that is generally related to our themes about technological progress. So there is this paradox that comes up in time travel called the bootstrap paradox. They kind of do this in Bill & Ted’s Excellent Adventure, if you’ve seen this, where he’s like, “I’m going to in the future, find this key and put it here” — and then he pulls up the rug and the key’s there, and then he just has to remember in the future to put it back. So you could imagine that as a general thing, where you could say, “I intend in the future to give myself this object, and then I’m going to show up right now and give it to myself,” and then there you have it. Then you just have to make sure you give the object to yourself in the past.

And there’s this question of, where did that object come from? And so the variation that I think is kind of interesting about is time travel possible is: could you do this with information about time travel itself? So could you form this intention, like, “I’m going to first learn how to build a time machine after being tutored by my future self, then I’m going to devote my life to implementing the instructions that I have been told by my future self. I’m going to build this time machine according to the specifications. I’m going to take notes, and then maybe there’s some tinkering, experimenting, whatever. I’m going to figure it out and then I’m going to use that to go back and give myself the specifications about it.”

In principle, I could do that right now. I could just form this intention that this is what I’m going to do. And nobody’s coming through the door to tell me how to build the time machine. So does this imply time travel is not possible, or is there some other hole?

Anyway, that’s one that I thought of. And where does technological progress come from? Hopefully not just like from the void like that, but maybe sometimes.

Luisa Rodriguez: Yep. I really love that. Interesting. I wonder if you’re not setting the intention hard enough.

Matt Clancy: Like, you just have to really, really want it.

Luisa Rodriguez: You’re being too cheeky with it.

Matt Clancy: Yeah, that’s what it is.

Luisa Rodriguez: You’ve got to actually set that intention. I don’t believe you’ve done it yet.

Matt Clancy: Somebody out there, please try it with the serious effort that is required to make it work.

Luisa Rodriguez: Seriously. Cool. Well, that’s new to me. Thank you for that. My guest today has been Matt Clancy. Thank you so much for coming on. It’s been really interesting.

Matt Clancy: Thank you so much for having me.

Luisa’s outro [02:37:23]

Luisa Rodriguez: If you enjoyed this episode, you might be interested in speaking to our one-on-one advising team. At 80k we measure our impact by how many of our users report changing careers based on our advice. One thing we’ve noticed among plan changes is that listening to many episodes of this show is a strong predictor of who ends up switching careers.

If that’s you, speaking to our advising team can help massively accelerate your pivot into a new role or career. They can connect you to experts working on our top problems who could potentially hire you, flag new roles and organisations you might not have heard of, point you to helpful upskilling and learning resources — all in addition to just giving you feedback on your plans, which is something we can all use. You can even opt-in to a programme where the advising team recommends you for roles that look like a good fit as they come up over time — so even if you feel like you’re already clear on your plans and have plenty of connections, it’s a great way to passively expose yourself to impactful opportunities that you might otherwise miss because you’re busy, not job hunting at one given point in time, or underestimating your fit for a particular role.

And, like everything we do at 80,000 Hours, it’s all totally free — the only “cost” is filling out an application! Applications often seem daunting, but our advising application form only takes about 10 minutes: just share your LinkedIn or CV, tell us a little bit about your current plans and which top problem areas you think are most pressing, and hit submit.

You can find all our one-on-one team resources, including the application, at 80000hours.org/speak. If you’ve been telling yourself for a while you’d apply for 80k advising, or you’ve been on the fence about it, now is a really good time to apply. This summer we’ll have more call availability than ever before.

All right, The 80,000 Hours Podcast is produced and edited by Keiran Harris.

The audio engineering team is led by Ben Cordell, with mastering and technical editing by Milo McGuire, Simon Monsour, and Dominic Armstrong.

Full transcripts and an extensive collection of links to learn more are available on our site, and put together as always by Katy Moore.

Thanks for joining, talk to you again soon.