Carl Sagan, when talking about this stuff in the ’80s, when he realized these risks, he had some great conceptual work on this that’s very similar to the work done by Derek Parfit. He attributed this to humanity growing powerful without growing commensurately wise. I think that’s a pretty good way to look at it. That it’s not the case that it’s just an argument that we shouldn’t have high technology because high technology will doom us, so we should be Luddites. But rather, you need to be aware of these fresh responsibilities that come with these new levels of power. And that we’re capable of growing our power exponentially on a lot of metrics. But there aren’t many metrics where we’d say humanity’s wisdom, or institutional ability has been growing exponentially. We generally think it’s faltering, if at all. So, he sees this as this problem, and I kind of agree. He calls it a “Time of Perils”. I call this time, “The Precipice”. A time where we’ve reached the ability to pose existential risk to ourselves, which is substantially bigger than the natural risks, the background that we were facing before. And this is something where I now think that the risk is high enough, that this century, it’s about one in six.
So I think that there’s a time period, The Precipice, that can’t go on for all that long. One of the reasons it can’t go on for all that long is if the risk keeps increasing, or if it stays the same, then you just can’t survive that many centuries with a risk like this. That changes a bit depending on how much risk you think there is now. So you might think that the case for existential risk is still really strong and important, but there’s only 1% risk. But therefore the time period could go on for a very long time, perhaps. I think that it could only go on for a handful of centuries. So it’s a bit like thinking about something like the Enlightenment, or the Renaissance. Some kind of named time period like that is what I’m talking about. I think that the risk is either going to go higher, or that we fail out of this time period. Or, that we get our act together and we lower these risks.
Estimating total natural risk
We have this catalog of risks that we’ve been building up: these things that we have found that could threaten us. A lot of which we just found in the last hundred years. So you might think, “Well, hang on, what’s the chance we’re going to find a whole lot more of those this century or next century”? Maybe the chance is pretty reasonable. If you plotted these, which maybe some enterprising EA should do, over time and when they were discovered to see if it looks like we’re running out of them. I don’t think that there are particular signs that we are, but there is an argument that can actually deal with all of them, including ones that we haven’t yet discovered or thought about, which is that we’ve been around for about 2000 centuries: homosapiens. Longer, if you think about the homo genus. And, suppose the existential risk per century were 1%. Well, what’s the chance that you would get through 2000 centuries of 1% risk? It turns out to be really low because of how exponentials work and you have almost no chance of surviving that. So this gives us a kind of argument that the risk from natural causes, assuming it hasn’t been increasing over time, that this risk must be quite low. In the book I go through a bit more about this and there’s a paper out that I wrote with Andrew Snyder-Beattie, where we go into a whole lot of mathematical detail on this. But, basically speaking, with 2000 centuries of track record, the chance ends up being something less than one in 2000 is what we can kind of learn from that. And this applies to the natural risks.
So, when I first looked into it, one thing I wanted to be able to say when writing a section on it is, well, some people talk about the Earth becoming like Venus. So having this runaway climate change where the oceans start evaporating, creating more and more water vapor in the air and that this can run to completion and basically we’d have no oceans and the temperature goes up by hundreds of degrees and all life ends. Or at least all complex life. So I wanted to be able to at least say, “Don’t have to worry about that”.
It turns out that there is a good Nature paper saying that that can’t happen no matter how much CO2 is released. You’d need brightening of the sun, or to be closer to the sun for it to happen at any CO2 level. But normally one Nature paper saying something, you know that’s enough to say, “Yeah, probably true”. But there’s a limit to how much epistemic warrant can be created by a single Nature paper on something. But it still seems like that probably isn’t going to happen and no one’s really suggesting it is going to happen. There’s another thing that was a bit alarming there with something called a ‘moist greenhouse effect’, which is similar, but doesn’t go quite as far, but you could still get something like 40 degrees Celsius extra temperature. And the scientists are like, “Oh yeah, I mean you can’t get this runaway, but you might be able to get this moist one”. And from a lay person’s perspective, you think, “Well hang on a second, why didn’t you include that in the other category? I thought when you were giving reassurance as the other thing wasn’t possible, that you weren’t saying there’s a thing that’s, for all intents and purposes, identical, which is perhaps possible. And that one also probably can’t happen, but people are less sure and there are some models that suggest maybe it can.
Arden Koehler: So it seems like you think that natural pandemics, like the ones you’ve listed, although extremely serious, pose a pretty tiny chance, one in 10,000, of causing extinction in the next century. Whereas you think engineered pandemics pose a dramatically higher risk, one in 30. Why the huge difference?
Toby Ord: Yes. So the main reasons are that there is this natural risk argument that we discussed earlier whereby the total amount of natural risk can’t really be much higher than about one in 10,000 per century. And so part of that comes down to how much that argument applies to these natural pandemics. I suggested earlier that it doesn’t quite apply because they may have gotten less safe in some ways, but there’s also many ways in which we’ve gotten more safe. We understand diseases much better with the germ theory of disease. We have antibiotics, we have quarantine ideas and so on, and we’re spread much further across the world and so forth. So we have a whole lot of reasons why we’re actually more safe or less safe. And it’s hard to be sure how that all balances out. I think it leaves things somewhere in the ballpark that the natural risks is in the same level. Whereas when it comes to the engineered pandemics that we could have, there are several different ways that could happen when I’m talking about engineered pandemics to start with. There are cases of gain-of-function research where scientists make diseases more deadly or more infectious in the lab. So that’s a case where they’re being engineered for these bad qualities.
The idea is obviously to help us overall by better understanding what genetic mutations need to happen in order for this to become more lethal or more infectious so that then we can do better disease surveillance in the wild and check for these mutations. Things like that. But they pose their own risks. But I don’t think much of the risk comes from that though, partly because even if it did escape, it’s still again very difficult for it to kill everyone in the world because it’s not that different from the wild types of these diseases. It’s somewhat worse, but they’re not making it thousands of times worse or something. And then there’s also cases of diseases that are engineered for use in war: so biowarfare. And that is quite alarming because they’re trying to make them much worse. And we have a lot of known cases of that. And then there’s also possibilities of what we often think of as terrorist groups, but perhaps cults that are omnicidal and have some plan to kill everyone in the world or something like this. And they actually are at least attempting to make the thing to exactly design the thing that could kill everyone, but they’re much less resourced than a military is. So there’s a few different kinds of ways they could be engineered. But the thing that they have in common is that they’re kind of working towards an actual objective of widespread destruction, potentially aiming to kill everyone in a way that the natural things are not. They’re trying to maximize inclusive genetic fitness is what they’re being optimized for, which is not the same as killing your host. They only kill the host inasmuch as that helps them spread. So it’s quite different to what’s happening there and that’s the agency or the fact that someone’s actually trying to make something that would wipe out humanity is what makes the chance so much higher.
I think that a useful bird’s eye view of it all is actually helpful. I think that you can divide it into these two parts. One of them is, “What’s the chance that we will develop something more intelligent than humanity in the next 100 years”? And then, if we did, as I point out, humanity has got to this privileged position, where it’s humans who are in control of our destiny, and control of the future of the Earth-based life, in a way that, say, blackbirds or orangutans aren’t because we have unique cognitive abilities. That’s both what we think of as our intelligence and also our ability to communicate, and to learn and so forth, and to work together as teams, both within a generation, and between generations.
It’s something broadly like our intelligence that put us in this unique position. We’re talking about creating something that knocks us out of that position. So, we lose what was unique about us in controlling our potential and our future. Then, one question is what’s the chance that we develop something like that? Then, the other question is, “If we develop it conditional upon that, what’s the chance that we lose our whole potential”? Basically, you can look at my 10% as, there’s about a 50% chance that we create something that’s more intelligent than humanity this century. And then there’s only an 80% chance that we manage to survive that transition, being in charge of our future. If you put that together, you get a 10% chance that’s the time where we lost control of the future in a negative way.