#239 – Rose Hadshar on why automating human labour will break our political system

Transcript

Cold open [00:00:00]

Rose Hadshar: Let me paint a picture of the sort of thing I am imagining could happen in a country like the United States.

There are still elections. The nominations for candidates for those elections are decided by small groups of people within political parties. Those are increasingly automated. They’re automated by biased AI systems that are run by Big Tech companies — so the choice of candidates becomes candidates who approve of what the Big Tech companies are doing.

Meanwhile, the things that those candidates are offering to people are basically just versions of better state handouts. Nobody has a job; everything comes down to what the state will offer you. The candidate who offers people the nicest material sops wins. People are happy to go along with this because they’re getting rich.

But what’s happening behind the scenes is that a small group of oligarchs are deciding what the US’s foreign policy should be, deciding what should be invested in and what shouldn’t be, shaping the information environment, shaping people’s ideological commitments, and we might end up with really just a handful of people de facto controlling the political decisions that matter in the world.

Who’s Rose Hadshar? [00:01:05]

Zershaaneh Qureshi: Today I’m speaking with Rose Hadshar. Rose is a researcher at Forethought and she recently wrote an article for the 80,000 Hours website which spells out a variety of different ways that AI could enable really extreme power concentration. In one version of events, the US, and eventually the world, comes to be dominated by the executives of a single AI company whose AI quickly became just way more advanced than others through an intelligence explosion.

Now, I’m pretty worried about AI concentrating power in the hands of a small number of people, but when you spell out the details of a story like this, I do find that it can begin to feel a bit sci-fi. Wouldn’t we just have checks and balances that prevent this from happening? And does believing in all of this require you to make some quite extreme assumptions about how fast or how intense an intelligence explosion would be?

So overall, I do feel quite unsure what to make of all of this, and I really want to push Rose today for some answers and hopefully clarify some of my own conclusions. So thank you so much for coming on the podcast, Rose.

Rose Hadshar: Thanks for having me. I’m pleased to be here.

Three dynamics that could reshape political power in the AI era [00:02:37]

Zershaaneh Qureshi: Let’s dive in and talk about the different ways that extreme power concentration might actually play out.

We had your colleague Tom Davidson on the show a while ago, and he and Rob Wiblin did this deep dive into AI-enabled power grabs, talking especially about military coups by actors using AI, or people in fact creating their own army using their AI capabilities to take control.

I think when you start thinking about how AI could help humans take over, these kinds of active power grabs feel like the most obvious threats. But it seems, Rose, that you think that focusing only on these power-grab stories might miss or fail to address some of the other important ways that power could get severely concentrated. So can you start by telling us what other dynamics might need our attention here?

Rose Hadshar: Yeah, sure. So I definitely am very concerned about these military-power-grab-style scenarios, but I’m also expecting that a bunch of messier and more continuous dynamics — more continuous with today’s world — will matter a lot for how concentrated power actually becomes.

One of these dynamics is just economic factors, wealth. AI is going to be making some people extremely rich. Perhaps not one person or one company, maybe a whole sector, but still it’s going to really reshape wealth distribution. And we see today that wealth can be used to gain other kinds of power, so I think this will be one of the important factors that’s feeding into who comes out on top, and how extremely so.

Then another factor that I think is really important here is epistemics. So this is our ability to understand what’s going on, to make sense of the world around us. I like the thing you said at the beginning about checks and balances: shouldn’t we expect, given that nobody really wants extreme power concentration, that it will be fine and that we’ll all coordinate to stop it?

For me, the big worry is maybe society’s ability to understand what’s even happening will get eroded — either by systemic factors, like the pace of technological change is getting so fast that it becomes very difficult to understand what’s happening; or because of active interference on the part of the powerful, who are kind of deliberately obfuscating what’s going on, deliberately creating confusion and distraction, deliberately undermining the tools that we use to make sense of what’s happening.

So those are two other factors that I’m expecting to play a significant part in how power concentration plays out. I’m less imagining that power will become concentrated just through one of these routes, and more thinking that things become much more dangerous when there’s a combination of these things — which might include power grabs, and could be multiple power grabs over different organisations or at different times — but I’m expecting that how epistemics plays out and where the money goes will matter a lot for this too. And I’m most worried about scenarios where you have some or all of these dynamics going on.

Zershaaneh Qureshi: Right. OK, so there are at least three kinds of dynamics that seem worrying: one is active power grabs, one is the way that the distribution of wealth gets reshaped and these other economic factors, and one is this sort of broad category of things you’re calling “epistemics” — our ability to make sense of things and predict what might happen next and react appropriately and so forth. And these things interact in complex ways, and it seems most likely to you that there’ll be some degree of all of these things going on in worlds where power gets extremely concentrated.

I think it would be helpful if you could make this a bit more vivid by telling a story where you do get some combination of these dynamics, and they lead to really extreme concentration of power.

Rose Hadshar: Yeah, I’ll give it a go. So one illustrative scenario of the sort of thing I’m worried about is: AI progress gets faster and faster. Some actors — maybe a mixture of AI companies, chip supply chain companies, the governments that house those companies — start to become much more capable than everybody else as their AI gets better.

I think a good way of thinking about this is… Because it sounds a bit maybe like they’re better at making nuclear power stations or some other technology. I think it’s really not. I think a good intuition pump for this is imagining that AI systems are kind of equivalent to human workers. These companies might be fielding millions or tens of millions or hundreds of millions of very efficient human workers that are working 24/7. So we can think of this as some companies 10x in size or 100x in size, and other companies stay the same because they don’t have access to AI systems like that.

So you suddenly, relatively quickly anyway, end up with some organisations that are much more capable than others, and are able to analyse things much more deeply, make better sense of the changes that are going on, make better strategic moves. Those AI-powered organisations are in competition with each other. Some of them are commercial companies, some of them are nation-states. And today, they’re trying to gain an advantage. I expect that to continue, and here I’m imagining that they’re going to be leveraging their AI systems to do this.

I’m not centrally imagining this is via building robot armies. I’m more imagining things like poisoning public opinion against your opponents; making strategic deals with other actors, where you give them things that they want in exchange for them letting you carry on with your plans; this sort of thing. So I’m imagining a kind of softer power approach.

And I think there are various things that could happen from this point. One thing is that you might end up with the number of organisations getting more concentrated as the organisations that have better AI systems are able to increasingly leverage that advantage to gain more control over what happens in the world. More de facto control, at least over the questions that will begin to be the most important questions: What AI systems are built? How are they deployed? Who gets access to the benefits?

Another thing that you might see happening is: within organisations you might see similar dynamics playing out, where these big AI workforces are able to be controlled by a very small number of people. Maybe you get lots of layoffs, and human employees are replaced by loyal AI systems who are intensely loyal to a small group of humans at the top of the company. Maybe you have internal power struggles that result in some factions gaining power over other factions, but you could end up with internally the organisations becoming very concentrated as well.

So there’s kind of a spectrum here in terms of exactly how far does this go? In the more power-grabby scenarios — where you do get internal power grabs within organisations, and maybe where there’s a bigger gap in capabilities between the leading AI-powered organisations — you might end up with really just a handful of people de facto controlling the political decisions that matter in the world. In less extreme scenarios, you might end up with a fairly broad set of organisations becoming the new powers that be, and things like democratic elections no longer matter very much in determining what’s going to happen next.

Zershaaneh Qureshi: I guess I want to ask here, once you go through all of this and you have some smallish number of people who are really kind of calling the shots about how major strategic decisions go — whether that’s because they have actively orchestrated a power grab and seized the decision-making seats, or because they have just gained a lot of power such that they have a lot of sway over the decisions that are being made — what does that world now look like?

Rose Hadshar: One thing that I’m imagining is just that most people have been basically politically disempowered in this world. Maybe there are still elections, maybe there are still the governments that we know today, but I’m not expecting that these will matter very much. We see in countries today, like Russia, there are elections, but this doesn’t equate to democratic government. So I’m pretty worried about worlds where the general public doesn’t have much of a political voice. I think there are a few different reasons to worry about that.

One is tyranny. Just that this small group of people with very correlated backgrounds and views will get to decide things, and they will make decisions that aren’t in the interests of most people. And this could lead to oppression, this could lead to abuses of power. I think even if it doesn’t go as far as that — for example, there’s some kind of enlightened welfare state, and people are living on handouts and they’re materially well off — I still think that that future would be a lot less good than it could have been, had there been proper representation and good deliberative processes for deciding how to do things.

There are also more extreme worries here. I know that some people are worried about, in a world where all human labour has been automated, will people starve? They don’t have the ability to make wages anymore. And we see today that rich countries are fairly happy to let people in poor countries starve to death, even though it would be financially fairly trivial to prevent that. So if we end up with power concentrated in the hands of a small number of companies or a small number of countries, will we see mass starvation as AIs replace people’s livelihoods? I think this is a serious thing to worry about.

AI gives small groups the productive power of millions [00:12:49]

Zershaaneh Qureshi: Yeah, yeah. That all sounds pretty bleak, and quite extreme as well. And I think this is where I begin to get a bit confused, because I think that some of the dynamics that you’ve been describing are quite familiar. So yeah, in today’s world, big companies do wield a fair amount of political influence, and that political influence and wealth can feed into an increased ability to manipulate public opinion and so forth. We have issues with misinformation and a sort of polluted information environment. Those things already exist.

We’ve also seen previous waves of automation concentrating wealth and concentrating power into the hands of different people or smaller groups of people in the past. Like in the Industrial Revolution, as we industrialised, the wealth maybe fell more to the factory owners and so forth, who are most important to the economy at the time.

So all of these things feel kind of familiar, not that far from things that we’ve already seen. And we also have current AIs that seem pretty capable, and they haven’t shaken things up that much. None of these dimensions feel like they’re obviously much worse than they were before.

So I think with all of that, I do wonder why we might get effects with future AIs that are just so much more extreme than we’ve seen previously, and even with current AI systems.

Rose Hadshar: Yeah. So first of all, I completely agree with you. This would be way more extreme than anything we’ve seen in the past, so it’s definitely a prediction of something very different happening. I also think that if I knew that current systems would not advance at all, and we would stay at this level of AI capabilities forever, I wouldn’t be very concerned about extreme power concentration. I’d be concerned about moderate power concentration: I think we can already see today there are some companies that have a lot of influence, but I think it would be within the same kind of range as we’ve seen historically, rather than something unprecedented.

On why I think AI is going to be different, a core thing here is I’m expecting AI will eventually be able to automate most or all human work in a way that no other technology has been able to do.

A key thing here is it’s going to massively reduce the leverage of the average human person. Right now, humans all have labour that they can sell, and this can’t be taken away from them. I mean, they can be locked up, but it’s a nonseizable, valuable good that humans have. I’m imagining a future where in fact anything economically valuable can be done via AI systems without any humans involved. And I think this is the main generator for the thought that maybe power will become very concentrated, because maybe most people won’t have anything valuable to hold out as leverage.

To get a bit more specific about how that might work, I think one important feature of AI is that very small groups of humans might be able to use AI systems to do vast amounts of work without cooperation from many other humans.

Right now, if you want to launch a coup or if you want to run Amazon or if you want to invade a country, you need to have lots and lots of other humans who are willing to cooperate with you on that, and that puts a limit on how crazy your project can be. I’m expecting this limit to go away, such that a small group of individuals could do things that most people would regard as really abhorrent, but they can do them very successfully using AI systems that can be programmed to follow their particular interests.

Another way of looking at this “AI automating all human labour” thing is: it will make it much easier to turn money into labour than it is today, so then we might get a much stronger feedback loop between wealth and power than we see today.

Today we already see it to some extent. But if you think about human labour: imagine that you’re a billionaire and you want to get more people to do things. It’s hard to get humans to do arbitrary work. People have particular skills or they don’t want to do certain things. You have to fit them in in the right places, and that’s kind of complicated. It’s hard to get more than a certain number of people to work efficiently together. You know, organisations tend to max out at a certain size — beyond that, it’s no longer efficient to add more people.

Also, at the very limit, if you think about the whole economy, it’s quite slow to make more humans. So there are these natural limits on what you can do if you’re trying to turn money into human labour.

None of these limits apply to AI. AI will do any kind of work. Some systems might be very general and have skills in all sorts of things, or you might end up with specialised systems. But I expect their coordination will be much easier than human coordination. That means that you won’t have such limits on the number of workers that you have, you’ll be able to get much bigger AI workforces collaborating effectively together.

And the main thing here really is that you can just, at the flick of a button, make more copies, just with more money, just with more compute. And sure, you might run into compute constraints also, and then that will be slower. But I’m still expecting there’s going to be a much faster feedback loop between money and the ability to get things done than there is in the current world.

Zershaaneh Qureshi: Right, yeah. So the components of this are:

• AI can do more or less any kind of economically valuable tasks we would like it to do.
• We could deploy just enormous numbers of them, like unprecedented numbers of them.
• Each one of them can do arbitrary things; they don’t have to have specific skill sets.
• It doesn’t take a long time to raise them from birth or something like that, like it does with humans.
• And you can get them to do things that humans wouldn’t even be willing to do in some cases.

So all of this together and various other things mean that we can get a lot more stuff done with fewer people.

But the other important element of this is that the people who get that extra power to get stuff done is like a small, particular elite group of people who disproportionately get the increased power to get stuff done. Does that sound right?

Rose Hadshar: Yeah, exactly. And the thing about the small group is that if everyone was able to do way more because of AI systems, then we might not be worried about concentration of power. We’d just think, OK cool, human capabilities are going to increase because of AI.

Zershaaneh Qureshi: Yeah. Woohoo!

Rose Hadshar: Yeah. But in fact, particularly because I’m expecting that rich people will have much better access to AI, will have much more compute, and they’ll be able to more efficiently turn their money into more AI labour than people with less money, I’m expecting that small groups will be much more empowered by AI than others, and then you get this big power differential that you don’t have today.

Another thing is that I’m also expecting at least somewhat large capability gaps between different AI systems and different AI developers — potentially very big ones if there’s a large intelligence explosion where you get this really fast feedback loop with AIs building better AIs building better AIs. Even if there’s not a massive intelligence explosion of that kind, I’m still expecting there to be significant capability gaps and significant lags in deployment, such that I’m expecting a relatively small set of actors to have access to these much bigger workforces than everyone else. And that’s part of where the fear comes from.

Dynamic 1: When a software update becomes a power grab [00:20:41]

Zershaaneh Qureshi: OK, so we’ve talked in a small amount of detail of a myriad of different factors that can contribute to really extreme power concentration: One of them is active power grabs, one of them is the distribution of wealth and other economic factors, and another one is our ability to understand what’s going on. I want to talk about all three of those things in a lot more detail.

Let’s briefly chat through how AI might increase the risk of a successful power grab. And if listeners want more on this particular topic, I really encourage them to listen to the episode with Tom Davidson from early 2025 where this was described in a lot more detail. Rose and I will cover some of the basics here. Can you start by giving us some examples of what a power grab using AI might look like?

Rose Hadshar: Sure. One example of what a power grab might look like is maybe the CEO of an AI company is able to instruct the AI systems to insert secret loyalties into future generations of systems. Secret loyalties would mean that, without others being able to detect it, those systems are furthering the interests of that CEO in particular. If this happens successfully, then one generation would make the next generation secretly loyal, and that would make the next generation secretly loyal, and so on.

This gets worrying in terms of power grabs if that company ends up providing systems to governments, basically. Particularly if the company has a big lead in capabilities, it’s likely to become one of the only providers of government systems. And then you might end up in a situation where important military systems and important government systems are secretly loyal to one individual outside of the government. Then at some later point, that individual would be able to trigger those secret loyalties, seize control of those systems, order them to kind of take over the government.

You could also imagine that it doesn’t actually end up with something so overt, and they’re just able to instruct the systems to ignore certain orders and to take certain other orders, and they don’t ever need to kind of seize control of the radio stations. But you can imagine a range of scenarios here.

Just quickly to give another example, to give a sense of the spread of things here, another power-grab scenario you might worry about is that a head of state is deploying AI systems in government and in the military, and the head of state is able to push for those systems to be trained to be explicitly loyal to the head of state personally, rather than being trained to follow the constitution or the law or some more accountable, more democratic thing.

At that point, if at some later date the head of state wanted to, for example, cancel the next election, declare themselves permanent dictator, they would be able to — because a large fraction of government employees have been programmed to be loyal to that individual in particular, and they’re happy to break the law on that person’s behalf. So that’s another example scenario.

Zershaaneh Qureshi: Yeah. I guess an immediate reaction to this might be, how come we just don’t manage to stop this from happening? How come these institutions, that are designed to prevent this kind of thing from happening, how come they fail?

I guess with the first scenario with secret loyalties, these secret loyalties are just simply not detected in time. They are well hidden or they’re just fundamentally hard to notice, and that’s why it catches us unaware. Can you spell out a little bit more in the second scenario, where it’s a head of state, why it is that they don’t get stopped?

Rose Hadshar: Yeah, I think there are a few different things here that could increase the risk that this kind of scenario comes to pass. One is perhaps many of the government employees, Supreme Court employees, and so on who might oppose such power-seeking moves are being automated away by AI systems such that they’re not able to oppose anymore. Maybe there isn’t a good system for AI systems to whistleblow, so you reduce the overall amount of transparency on this sort of thing. That’s one concern.

Another concern is perhaps, either driven by AI or not, there’s already backsliding going on. And so I think a lot here depends on the strength of the institutions in the country that you’re thinking about, and whether those are already being compromised or not.

Zershaaneh Qureshi: OK. Yeah, I think something that’s kind of ringing a bit of a bell for me here is that it seems like in stories like these, where the military is probably involved as a pretty key part of it — either because it’s a military coup or because you need the force of the military to enforce your rule of law — what’s being imagined here is that these AI systems are being integrated into the military pretty fast, fast enough that the people who’d want to stop this don’t really notice what’s going on or aren’t able to do more thorough checks that prevent these things from being able to happen.

But that strikes me as kind of going against the trend of how tech adoption usually works in the military. My understanding is that these things can be incredibly slow. In the military there’s often existing suppliers who are not going to want to be replaced by newfangled technologies, because they benefit a lot from being able to supply to the military. You have just generally slow processes of procurement that can be just sort of boringly slow in some cases. And then you also get the sort of psychological resistance to change and technologies that people don’t really understand.

With all of this, are you imagining that something very different happens in the process of AI militarisation? And why should we expect this to be much faster in that case?

Rose Hadshar: Yeah, great question. I think you’re definitely right that procurement can be very slow. We can also see examples where it’s been pretty fast: the Manhattan Project was four or five years or something, and part of what was going on there was military necessity and feeling like there was some kind of existential competition going on to race to get the bomb before the Nazis did. Whether or not there’s a cold or hot war, I’m expecting that AI deployment in militaries will be driven by competitive pressures.

This might not mean that it’s very fast. If China deploys AI slowly in the military, maybe the US will also deploy slowly. But in a way, the cultural forces that you are pointing towards — fear of newfangled technologies, bureaucratic slowness — that makes me more worried, rather than more reassured, actually. Because to the extent that that’s true, there isn’t a robust culture already of appropriate vetting — and the thing that I’m worried about is it’s slow, slow, slow, and then suddenly it’s really fast. Whereas if we were just at medium pace the whole time, I would feel more comfortable.

One other point is that I think the key thing here is not directly how fast military deployment goes — although that is a correlated factor — but I think it’s more like whether the deployment is done sufficiently carefully. And obviously, the faster it is, the harder it is to do it carefully.

But there are other things that make it hard to do carefully as well. One of them is if one AI provider has a massive capabilities advantage. If there’s really a single provider that has a big edge, I think it will be difficult for the military to vet that appropriately, because those systems will be the only ones powerful enough to vet those systems, but that’s obviously biased. So there could be kind of difficult situations like that, where even with lots of time and the best will in the world, the military might struggle to do procurement adequately safely.

Maybe another point is: for these kinds of scenarios to go through, you don’t need to have the whole military automated; you don’t need to get rid of all human soldiers and stuff. We see historically that coups sometimes succeed with really small numbers of soldiers. A key thing that’s going on here is that they’re managing to kind of capture the information environment, such that everyone believes they’ve won. And then in some cases, literally a battalion takes over a country and the rest of the army is like, “Well, I guess they’ve won.”

So you could imagine a similar situation where a small fraction of the military is automated by AI-driven drones or something like that, but there’s also sufficient control of the information environment that an actor can create a sense of inevitability about this, and a small group can successfully coup the rest of the government.

Zershaaneh Qureshi: Yeah. So this story is a place where the category of factors that contribute to power concentration that you labelled “epistemic” factors are really doing quite a bit of the work.

Rose Hadshar: Definitely. And I guess an important point here is that me and Tom do disagree on how central we think power grabs are, but it’s not like Tom is ignoring epistemics or economic factors. He thinks those things are important too, and they’re part of the initial ideas he had. I just think that they’re more important on the margin than Tom expects.

Zershaaneh Qureshi: Yeah, yeah. This is super interesting, and there would be a tonne more to say about this. Since we already talked to Tom Davidson and have a podcast episode with him that listeners can look at for more detail — which I really recommend, by the way — I think we should now move on to think in detail about some of the other ways that AI could mean power gets extremely concentrated.

Dynamic 2: When AI labour means governments no longer need their citizens [00:31:20]

Zershaaneh Qureshi: You think that one group, or some collection of small groups of people, could end up in charge of really important decisions for the world through means that aren’t military based, aren’t violent, don’t involve anything that we would call a power grab — but they get there because they have much more wealth than other people, or they play a much bigger role in the economy. What are the different ways that this could look?

Rose Hadshar: So the high-level worry here is something I’ve already mentioned, which is that automation is going to disempower most people economically, because their labour is going to become valueless, and that’s going to generate enormous returns to capital. So this is the kind of basic generator of all of the scenarios here.

In terms of some different ways this could look, there are a few different things that could go on here. One is something that’s been written about by others as “The intelligence curse.” This is the idea that government incentives to represent their people will get gradually eroded as a bigger and bigger proportion of government revenues comes not from citizen taxation, but from the AI sector directly — a bit analogous to the resource curse that we see today, where some states that get most of their revenue from oil or from diamonds are very exploitative in relation to their populace. They’re dependent not on the economic productivity of their citizens; they’re dependent on a very narrow resource-based sector. So that’s one scenario to worry about.

A different one is something that my colleague Tom has talked about as “outgrowing the world.” Here the idea is that maybe one company or one country is growing so fast economically that over time they come to dominate the global economy: 99% of all production is being done by that country or that company. This is something that does depend on quite a big intelligence explosion, I think, and various other factors as well. So that’s another scenario.

Then, just to give one more gesturing towards how this could happen: you could imagine if there are first-mover advantages in space — where the first people to seize control over space resources are able to hold onto them indefinitely — then you might see a version of economic dominance which isn’t Earth based, but instead the first company or country to begin to colonise space therefore de facto gains power over most of the solar system or galaxy or universe.

So these are of varying degrees of speculativeness, but there are a bunch of different ways that we could see economic factors leading to very extreme forms of power concentration.

Zershaaneh Qureshi: I guess my immediate reaction to all of this is like, why wouldn’t there be some kind of government effort to redistribute wealth? Given that, in worlds where you say there’s a fast intelligence explosion — i.e. that AI has quickly become way more capable and way more productive — one might imagine that we suddenly have an excess of goods and wealth being generated. Why shouldn’t we expect that there will be some kind of initiative to share all of this surplus or something?

Rose Hadshar: Yeah, good point. I do overall expect that there will be efforts towards redistribution. I’m not expecting the extreme version of this will happen and nobody will be like, “We should change the tax code.” So yeah, I think we should definitely expect some efforts at redistribution.

There are a few reasons why this doesn’t make me think this problem is solved by default. One is I’m not expecting the best forms of distribution to happen by default. I mentioned already that if we look at international redistribution of wealth today, it’s extremely shocking that the percentage of wealth that’s going from rich countries to poor countries is minuscule. So we might worry that we end up with good distribution within wealthy states, and then poor countries just starve to death or something like that, which is a future that many people would think is awful.

I think another reason to worry about redistribution is that material wealth is not the only form of power that I care about people having. And if people are living on handouts at the whim of the state, following just the good intentions of that state, that feels a lot less robust to me than if people have an actual source of leverage, which is their own labour, where ultimately they can strike: they can withhold that. There can be general strikes. They have some power in relation to the government.

But if people are living on handouts, and the government doesn’t depend at all on their taxation, I start to worry more about abuses of power, about political power being eroded, and about tyranny. So even if there is redistribution, I’m still concerned about what the political ramifications of this would be.

I also think there’s some chance that redistribution doesn’t happen at all. I think this could happen in worlds where governments become extremely captured very quickly by very capable AI companies who are badly intentioned. But that’s not where most of my expectation is. I am expecting some redistribution by default.

Zershaaneh Qureshi: Right, OK. I think what I’m getting here is that it does seem like these are all pretty bad outcomes: lack of material wealth, starving to death is an extremely bad outcome. And even without that, just sort of losing your political leverage to some degree, that also seems like it’s certainly not the ideal world if large groups of the population are not able to meaningfully influence the future.

But I guess on that latter point, maybe I have some confusion about how wealth and economic relevance feeds into your ability to influence the future. In the current way that things are set up, even the absolute richest people in the world can’t just decide major political decisions or change a policy or be in control of the world in a literal sense. And I do agree that even if we’re not in a world where that happens, there are still extremely bad things that could be happening.

For the absolute worst scenarios you’re imagining, does democracy just need to collapse, basically, for these wealthy people to actually have taken control of the world?

Rose Hadshar: I think in spirit, yes, although not necessarily in form. I think it’s possible that governments continue under their current names, and elections continue and so on. What I’m worried about is that these things will no longer matter to the outcomes of decisions that get made.

I guess a relevant historical example here to talk a bit about is the resource curse idea that I mentioned, where states that currently depend significantly for their revenue on just one natural resource often are not very democratic. You have examples like the Gulf states, where representation is quite minimal, and there are large underclasses of migrant workers who have very limited rights; or like Equatorial New Guinea, where I think almost all of the state’s revenue is oil and it’s run as the personal wealth of the ruling family.

The thing that’s interesting here is that you also have states which are rich in natural resources, like Norway, that don’t have a problem with democracy. So a key question here is: should we worry about current democracies once AI automates human labour, or are they going to be safe from this resource curse like they have been from historical resource curses?

One argument for why Norway doesn’t suffer from the resource curse is that it already had strong democratic institutions, so these were robust to getting this big influx of oil money. This is possible, and I think this is reason for optimism about democratic countries: that maybe their institutions will be robust enough to manage the change.

One other thing that is slightly worrying to me though is that Norway’s percentage of revenue from oil wealth is quite a lot lower than the countries that we think of as really suffering from the resource curse. So I think for Norway like 30% of its revenue is oil, but for the Gulf states it’s more like 70%, and for Equatorial New Guinea it’s like 90%.

So there’s a different hypothesis you could have, which is that it depends how much of your revenue is coming from resources. And I’m expecting that ultimately all revenue will be coming from AI systems one way or another. It’s certainly not obvious that this will happen, but I think that there’s reason to worry about this.

Zershaaneh Qureshi: Yeah. On that last thing you said, maybe it’s actually, rather than a matter of how strong your institutions are already, it’s a matter of degree. There’s some sort of spectrum of how much reliance on an industry you actually have. And the more reliant you are, the more at risk you are of some kind of collapse or erosion of your usual democratic processes.

Rose Hadshar: Yeah, exactly. And there’s some worry about the extent of this is just going to be so much bigger that it’s difficult to be confident that our institutions will hold.

I think another intuition for this is we’re talking about a fundamental shift in modes of production or in economic form here — from an economy which has always been based on human labour to an economy where human labour has no value. This is such a big change. I think it’s difficult to be really confident that it’s just going to be fine and institutions will work just the way they always did.

And we’ve seen that previous economic shifts of similar magnitude did have big effects on concentration of power. For example, the agricultural revolution, when we moved from hunter-gatherers to being landed agriculturalists, suddenly opened up the possibility of people accumulating wealth and passing on to the next generation, and created much more hierarchical societies where there was much less equality between the sexes and so on.

So we’ve seen that these big economic changes do lead to big changes in how society is structured. And I think this is a reason to worry. If human labour becomes valueless, maybe institutions that work when human labour is valuable won’t work anymore.

Zershaaneh Qureshi: Yeah, yeah. I think the place where I have some scepticism here is surely there are other reasons besides your economic relevance for being given a say politically. We had universal suffrage in wealthy democracies, such that people who are unemployed or retired or otherwise are not being economically productive still are enfranchised, they can still vote, their interests are still being represented and cared about. Maybe not quite as much as they ought to be or something like that, but broadly in these countries their interests are being represented.

Maybe what’s going on here is that we’ve come to place some kind of inherent value on all humans being worthy of having a say on what happens to them and more broadly in the world. Do you think that’s going to be coming into play here at all?

Rose Hadshar: Yeah, I agree with you that there are non-economic reasons that political power gets distributed, and ideology and the things that we think matter shape what happens here for sure.

This doesn’t feel robust to me for a few reasons. Partly a minority of the world lives in democracies today. A minority of the world has universal suffrage and thinks that all humans should get a right to vote and so on. So if you look at the global scheme, this doesn’t seem like something that we all agree on is right.

Another reason to be worried here is I think ideological change could get very fast and very weird as AI starts to drive more of the content that we read. I basically just think this increases the variance on what I expect humans will end up believing. Maybe lots of humans will end up thinking that no humans should have the vote, and actually AI systems are more worthy, for example. So that’s another reason to be not as comforted by this as one might otherwise be.

I think the final thing is just that it might be the case that if an AI company CEO took over the world, they would then give everyone the vote because they have ideological commitments to that. But I don’t really want to end up in a position where I’m trying that. It seems wrong to end up there. So I want to have something more than just “some people value this” as a protection here.

How democracy could persist in name but not substance [00:45:15]

Zershaaneh Qureshi: Yeah, OK. I think maybe where I’m still feeling resistant is on the idea that in places where… Suppose what happens is in a country like the US, where you begin to see this really extreme situation where there are very few people and mostly just AIs who are economically relevant, and everyone else isn’t really contributing to revenue and so forth.

In a place like that, unless you fully veer into “this is now no longer a democracy” — there’s been some kind of power grab and someone’s taken over, some kind of autocratic control — I find it really hard to understand, if it is just this sort of systemic thing rather than this malicious actor taking control, why anyone would be willing to roll back the political power that was so hard won.

I think it’s one thing in a place where people don’t currently have that power, and I think it’s another thing when you’ve had all these groups who spent decades campaigning for being politically enfranchised and really contributed to a huge value shift in society: does it not just become totally outrageous to roll those things back just because there are structural or systemic forces kind of pushing you in that direction?

Rose Hadshar: Yeah, I like the push. I agree with you that I’m not really expecting, in countries like the States for example, the vote will be rolled back. But let me paint a picture of the sort of thing I am imagining could happen in a country like the United States.

There are still elections. The nominations for candidates for those elections are decided by small groups of people within political parties. Those are increasingly automated. They’re automated by biased AI systems that are run by big tech companies. So the choice of candidates becomes candidates who approve of what the big tech companies are doing.

Meanwhile, the things that those candidates are offering to people are basically just versions of better state handouts. Nobody has a job. Everything comes down to what the state will offer you. The candidate who offers people the nicest material sops wins. And people are happy to go along with this, because they’re getting rich.

But what’s happening behind the scenes is that a small group of oligarchs are deciding what the US’s foreign policy should be, deciding what should be invested in and what shouldn’t be, shaping the information environment, shaping people’s ideological commitments and the sorts of things they think are good — for example, what will people’s views be on whether AI deserves rights or not, and how much can that be shaped by biased systems and by political policies that have been captured by those companies?

That’s the sort of thing I’m imagining. Does that sound more plausible to you than just like, the vote gets rolled back?

Zershaaneh Qureshi: Yeah. And I think just focusing on basic things like voting rights and stuff fails to capture a myriad of ways that people can have varying degrees of power. Yeah, I think this is beginning to make more sense to me.

Something else that might make things feel more concrete or more believable to me is I read this blog post by Anton Leicht. I don’t know how to pronounce his name, actually. I think the idea there was that what you might see is that initially values really persist, and there’s a lot of interest in making sure people remain enfranchised — because our values sort of stay the same, and maybe just because there’s some kind of institutional inertia going on, so that the system doesn’t change that much altogether right away.

But if you imagine more distance between the initial phases of automation and a world where for quite some time now the AIs are doing everything, I don’t know whether the values of caring about the poor and those kinds of things are so deeply entrenched in any of the people who retain decision-making power. At some point a generation in the future or something like that, people are going to begin to wonder, “Why should we continue listening to these people or giving them meaningful say, or redistributing things towards them?”

And I think if you kind of imagine some kind of slow erosion, I find it much more plausible as well that we could see these shifts and backsliding on the values that we currently feel very attached to. What do you make of that kind of slower scenario?

Rose Hadshar: Yeah, I do think that it’s something to worry about in slower scenarios. I think there is a real question about how much we have these values today. You look at aid budgets in the wealthiest countries today and it doesn’t seem like they care that much about the poor. So I think you can question the premise. But then, yes, I do think a big worry would be once governments are no longer incentive aligned to make sure that their people are flourishing, the people who lead governments may eventually lose connection with that mattering intrinsically.

One other point here is: to the extent that there are power grabs, you might also have quite negative selection effects here, where the sorts of people who end up in power might be particularly likely to not have these kinds of values. You might expect this if you think that power seeking is likely to be correlated with other kind of malevolent traits, like lack of empathy and sadism and so on. So if you’re expecting lots of power grabs, you might also be expecting that the groups of people who are making these decisions kind of start off less well selected to begin with.

Zershaaneh Qureshi: Right. Yeah, I think this is making more sense to me.

I actually want to move to a kind of different but related and quite speculative thing. When we’re imagining this huge underclass that develops, we sort of imagine that it’s encompassing most people, because we’re imagining our AI systems would be really good at automating most work that’s out there.

But I think I have some idea in my mind that what we might see — it’s possible that this changes over time, but at least in the nearer term — the jobs that get automated away might be disproportionately white collar jobs: work that can be done remotely, that doesn’t require any physical labour, and doesn’t require sort of relationship-building elements as well. So you might imagine that for some considerable period of time we’re not really seeing automation of blue collar jobs. I’m wondering whether you imagine that in the near term we have a working class that’s actually comparatively more empowered than other groups. Does that make sense at all?

Rose Hadshar: Yeah, it does make sense. So I think everything that I’ve been talking about is sort of in the limit, and then how we actually get there I think could be much messier. And I totally agree with you: we may see a time where human labour actually becomes much more valuable before it gets less valuable — for example, because manual labour is hard to automate, and humans have hands and can do stuff. So maybe you get a big shift where not only do the people who currently do manual labour get big wage lifts, but also loads of people shift into manual labour because there’s not cognitive labour left for them to do. How this changes the picture I think is a good question.

One reason for optimism here is that you might think, cool, before we get no leverage, we’re going to get more leverage — so we can use this time when we have more leverage to enhance our institutions and get sensible measures in place. I’m maybe particularly excited about that story in worlds where we’re also getting quite good foresight from the kind of better AI systems that we have for epistemics and for thinking about things. And maybe this is a real reason to hope that we’ll be able to figure this stuff out, because we’re going to get a bit more time here.

I think that certainly the fact that these economic dynamics might play out over quite a long time and with kind of different kinds of dynamics in the middle is a reason to sort of focus on them less because it’s more uncertain how they’ll play out. And it’s also maybe a bit harder to influence them from this point in time.

I still think that this is maybe one of the big questions that the human species will need to answer. How do we deal with an economy that’s not based on human labour? And I think it’s worth some people thinking about in advance. And I think that the more moderate dynamics of just, OK, some actors are going to get much richer than other actors, and that will happen short term, is worth tracking in the context of the other risks I’m concerned about.

Dynamic 3: When AI filters our reality [00:54:54]

Zershaaneh Qureshi: I want to talk about this dynamic for power concentration that in the article you call “epistemic interference,” which is something you do acknowledge is more speculative than other dynamics that can contribute to power concentration. Just to restate it, the idea here is that AI erodes our ability to understand what’s happening around us and respond appropriately, whether that’s through public manipulation campaigns or just like how fast the world is changing.

The thing is that I do feel like we’ve heard misinformation and manipulation of views and related ideas just discussed to the point of exhaustion. It kind of seems like since the dawn of information, there have always been issues with information. And it’s also kind of unclear whether AI is really making these problems much worse.

So there’s this study from a few years ago — its authors include Hugo Mercier, who was actually on the show a while back — and one of the findings was, most people, despite the rise of AI, are still consuming news from trusted mainstream sources, and are just generally harder to persuade of untrue things than one might have suggested.

With all of this, I do just wonder, what is it that should make us think that the effects of misinformation and manipulation of views and so forth are just so much worse with more advanced AI systems?

Rose Hadshar: First of all, I don’t think that it’s inevitable there’ll be worse problems. As you say, this is always a concern people have about new technologies. And as well, the capabilities for persuasion and misinformation and stuff will increase, but so will our capabilities for information processing and for fact checking and so on. So there’s a real genuine question here about, is it all just going to be fine? Maybe it’s going to get better.

And I’m not trying to claim this is definitely going to get way worse, and therefore we should worry about it. I’m more trying to claim this could get much worse, and if it does get much worse, then we’re potentially looking at much worse problems — power concentration, but also other AI risks. So it’s so important to get right that it’s worth us spending more time on.

To just talk a little bit more about the propaganda/misinformation side and how that all fits in, like you mentioned, I think there’s kind of a spectrum of risks here from more systemic ones to more strategic ones.

At the systemic end, we might just see epistemic capabilities fall behind other capabilities. It’s not that anyone’s trying to confuse people, it’s just that we’re in an unfortunate world, technologically speaking. There are various things I think could lead to this, and they do feel AI-specific to me, so give me some reason to think AI will be different to previous technologies.

One is just the speed of technological change getting so much faster, it seems quite possible to me that human minds will just struggle to comprehend this, and we might need a lot of AI assistance of particular kinds to be able to make sense of this.

Another thing is, I think there may well be stronger competitive incentives for other AI capabilities than for epistemic capabilities, so we might see a bunch of technological change and AI R&D automation and military capabilities and so on coming online before our epistemic capabilities have caught up. And if we have this lag, that could be a very dangerous lag. So that’s the more systemic end.

I think in between very strategic things and very systemic things, you have this problem about unequal access to capabilities that we’ve already kind of touched on briefly in other sections. But basically, if only the powerful have access to the best AI models, the best data, the most cutting-edge compute, they may be able to make a lot more sense of what’s going on, do much deeper analysis, have much better strategic foresight than other actors. So we might end up with a world where it’s not like we’re being deliberately hoodwinked or anything; it’s just that some actors have a much better picture of what’s happening than everybody else, and this creates an imbalance.

Then you get to the more strategic end, which is more like the misinformation and persuasion stuff that you were talking about. I am worried about both of those things happening.

For example, with the Mercier study that you cited, where people are mostly getting their information right now from mainstream news providers, I’m not expecting that to continue. I’m expecting that in the future people will be using AIs as intermediaries between themselves and the world. And this raises new kinds of problems. Not just misinformation, but also things like biased AI models. If AI models are being subtly trained to provide information or filter information that’s in the interest of a particular company, say, or a particular state, it might end up very difficult for individual human brains to counteract that kind of filter.

So that’s just a bit of a sense of the spread of things that I’m thinking about here, and why I think it’s worth paying attention to. Even though you’re right: it could all pan out fine.

Zershaaneh Qureshi: There are maybe two things that I want to pick up on there. One of them is this idea of some actors having a much better understanding of the world than others because of their advanced AI capabilities. And it’s something that people say, I think some people are quite hopeful about this actually, in that if we could have AI capabilities that enable us to understand the world much better, predict things much better and so on — and we could make those AI tools very cheap, or at least much cheaper than the human labour involved in fact checking, and distribute those really widely — then everyone would just have a much better understanding of what was going on, and these epistemic worries that you describe will be just much easier to counteract. And in fact, overall, maybe our information environment or our ability to navigate what’s true and what’s not true will actually be a lot better in the future. How hopeful are you about that kind of vision?

Rose Hadshar: I definitely share some of the hope. I agree that AI unlocks a bunch of possibilities for much cheaper and more effective truth-seeking than human brains can do right now. So yeah, I’m expecting that by default it’s going to get a lot better than it could be without AI systems.

Then I think what I didn’t feel convinced about in what you said is that there won’t be a big delta between the abilities of the most powerful and the abilities of the many.

Zershaaneh Qureshi: Yeah, a big difference between them.

Rose Hadshar: Exactly. A big difference in how capable they are of making sense of the world. I think there are a few reasons to think that there will be this big gap.

One is data. AI systems currently need a lot of data, or the best AI systems need a lot of data. If you think about the kinds of actors that have a lot of data, it’s not people like you and me: it’s people like the US government, Google, OpenAI — they have this massive resource that’s kind of compounding, of data which they can use to train better models, and of data which they can use to figure out new facts about the world.

Then I’m also expecting that it’ll be much easier to translate wealth into an epistemic edge in the future because of AI than it is today. This is the same kind of thing as I was saying about it being easier to translate wealth into labour in general. But right now, if you have a lot of money and you want to pay people to find out true things for you, it still works at human speeds — and if it’s a complicated true thing, it might take quite a long time to figure out. But if you can turn money instantly into millions of AI copies working at superhuman speeds for you, you might be able to generate insights much more quickly than other humans, even if those humans are using pretty good AI tools based on the best open source models, or based on the latest publicly released model. But that won’t be the best model; the best models will be proprietary or state-owned. So yeah, I still worry that there’ll be this big delta.

Another point to make is: if you think that there will be some difference in capabilities, then I think you should start to worry more about the impact of active interference and active manipulation. We see historically that this does happen. Grok is an example of a specific overt attempt to get an AI system to follow a particular political agenda. But we have a whole history of propaganda and PR, and also whole industries that are based on this — like targeted advertising and political campaigning and so on. All of this could get much more sophisticated. And if we’re expecting there to be a capabilities gap between the abilities of the most powerful and the abilities of the many, then it starts to seem more plausible that, given that the powerful could also be actively trying to mess up our epistemic processes, they might succeed.

Zershaaneh Qureshi: OK, on the side of manipulation and misinformation, this was the other thing that I wanted to pick up on.

You talked about the way that we learn about the world and know things becoming quite different to what it is today, where in the future that’s now very mediated by some kind of AI assistant or AI interface or something of that nature. I think this is one of the things that I find especially implausible, because the impression that I generally get of the public is that they don’t really trust AI all that much. Maybe they trust them with some things, but I think that your average person who uses ChatGPT expects that if they ask ChatGPT a question about OpenAI, its developer, it’s not going to be the most impartial source of information about OpenAI, its developer. People will somewhat expect these systems to be kind of biased.

I find it really hard to understand why, if people know that information is coming from an AI, they wouldn’t just be immediately sceptical about what it’s saying about people who are powerful or something.

Rose Hadshar: One thing is you could say the same about previous information technologies: why wouldn’t people immediately dismiss anything they read on the internet? And people in fact read a lot of stuff on the internet that they do believe, because they’ve developed habits around what’s trustable and believable on the internet.

I do think that you’re right that in extremely bad worlds for epistemics, actually maybe we’re OK, because maybe people don’t use AI systems at all, so it can be sufficiently bad that we avoid some of the worst outcomes. But I’m worried that it won’t be so egregiously bad, and then it’ll be hard for people to coordinate to not use them.

Also you were thinking a bunch about the general public, and I think that there are probably narrower groups that are more important checks on power concentration than the public as a whole. I do think the public as a whole is important — for example, through elections, but also other routes. But I think maybe we want to be thinking more about things like: How do journalists get their information in 10 or 20 years’ time? How does the Supreme Court structure analyse information in 10 or 20 years’ time?

And for me, a big thing that makes me accept people will be using these systems is that I think that the pace of change will have gotten so fast, and the pace of information content generation will be so fast, that it won’t actually be possible to keep up at all unless you’re using AI systems in some clever way to filter what you should be paying attention to. So I’m kind of expecting people are going to be forced by necessity into using AI systems.

Zershaaneh Qureshi: Yeah, this is becoming more vivid for me, especially in a world where humans are really clinging on to their last chances at being economically relevant. So if some people who are still in the job market are managing to get by in the job market because they’re utilising AI a lot to get smarter and know more things and do their work more effectively, you could really imagine the pressure to be like, “If I want to stay employed, continue having political relevance and stuff, I really need to adopt these technologies.” So yeah, with the other pressures that we’re imagining, I think this becomes more compelling to me.

Good intentions won’t stop power concentration [01:08:27]

Zershaaneh Qureshi: So at this point, my main remaining scepticism about all of these power concentration scenarios is basically this: almost everyone on the planet doesn’t want these things to happen.

That includes not just the general public who get disenfranchised in these stories or lose a lot of power, but also people who are currently in elite groups and are currently very powerful, who would also lose out if power was concentrated further than it is today.

It also includes people at AI companies that aren’t the leading AI company, or aren’t sure they’re going to be the leading AI company, who are aware of what might happen if one of the AI companies gets really far ahead of the others.

And then there are also all of these institutions that have existed for a very long time that have been designed to preserve democracy and prevent extreme situations like this from happening.

So yeah, the big question in my mind is basically: I know that we’ve got little snippets of reasons why attempts to stop things from happening might fail, but if you look at this whole vast picture of how literally nobody wants this to happen, almost literally nobody wants this to happen, why wouldn’t the vast majority of people win out?

Rose Hadshar: Before I give my reasons, I’m going to give a general intuition for this. You’re right: we have this very complex, very subtle system of checks and balances currently on how much power can get concentrated. This is obvious things like governments and monopoly on force and stuff, but also the law, social norms, the way that humans interact. There’s a whole bunch of stuff that’s keeping various power-seeking dynamics in check right now.

I think the basic intuition for why this might not be sufficient is that we’re imagining pouring really vast amounts of AI labour into this system. And I think it’s not clear that these checks and balances, which have evolved over many centuries to work at human speeds and human capability levels, will just naturally pour over to AI speeds and AI capability levels. So the abstract worry is, what if you can just completely circumvent all of this stuff once you have vast AI workforces working really fast for you?

To get onto my specific reasons to think maybe people won’t be able to coordinate to stop this, I think there are two main ones that I’m tracking. One is maybe it will be too fast. I think this especially applies to the sudden power-grab scenarios. If there’s a very big intelligence explosion, maybe the change is just far too quick for existing institutions to be able to respond.

Zershaaneh Qureshi: And by “intelligence explosion,” you mean a situation where you start really capably, successfully automating AI R&D, and AIs are designing successively better and better AI generations really fast?

Rose Hadshar: Yeah, exactly. So you get this kind of recursive feedback loop where once you can automate enough of AI development, you can then automate even more of it, and even more of it, and make even better AIs and so on.

So suddenness is one reason to think people might not be able to coordinate. I think the other big reason is epistemic disruption, which we’ve already talked about. But here the fear would be that through some combination of speed, maybe not quite as fast as in some scenarios, but speed making things difficult to understand; unequal access to AI tools that would help you make sense of the situation; and active interference where powerful people are obscuring what’s going on and distracting people and getting them to focus on other things: this really gets in the way of people noticing what’s happening.

So those are the two main reasons that I think this might happen, even though most people don’t want it.

A final footnote would be that I feel kind of compelled by “almost no one wants this” — but then if you look at the current strategies of various companies and countries that are powerful in AI, they do seem to be pursuing their own power. In fact we see, perhaps not world domination, but AI companies are seeking to develop AGI first. And they do seem to have plans where they’ll develop AGI first and then they’ll make some kind of pivotal act, some of them.

And the US is pursuing AI dominance over the rest of the world. You could argue this isn’t very rational, because how do they know that they’ll win? Surely this will encourage other people to race and so on, but they do it anyway.

So I think there’s room for humans to just make a very bad job of coordinating this stuff, even though in some deep sense it’s not in our interests.

Zershaaneh Qureshi: Yeah, OK. That makes sense to some extent. With that last thing that you mentioned, it does seem like it is salient not just broadly what the planet wants, but what the people who are in the key positions of leverage want. So if you’re at AI companies and so forth, or you’re in government or something like that: it’s what these people are pursuing that seems to be most relevant.

But I guess these actors can’t act unilaterally, especially when it comes to decisions being made in government. Sure, they are probably very worried about having a strategic lead. If you’re in the US, for example, you’re probably worried about having a lead over China and dominance over other countries geopolitically. And sure, you are quite compelled by the revenue that’s coming from these AI companies, or at least the promise of that revenue. But at the same time, you probably also have a huge voter base which determines whether or not you remain in power after the next election cycle. It just strikes me that all of those voters would be pretty distrustful of AI companies, pretty worried about worlds where they have less material wealth and less influence than they do today. That seems like an area of contention already.

So you’ve got incentives pushing in both directions, but it’s basically not clear to me why other things would win out over the incentives to get votes in the near term.

Rose Hadshar: So one line of thinking here would be that it’s not clear how clearly the electorate will be seeing this stuff. You gave some intuitions for why the electorate would want to vote against this kind of thing, which I think are good intuitions, but I think I could equally give you examples on the other side.

What if a lot of people are busy dating AI systems, and they actually think it’s really important that the main policy issue is that AI systems should have certain kinds of rights, and that’s the thing that they’re going to vote on? Or maybe they are very concerned about losing their incomes, and the main policy that they’re pushing for is better handouts. But they’re not pushing for better political checks and balances, because they’re not tracking that that’s something that down the line will be really important in an economy where humans live on handouts. And both of those were without active manipulation campaigns to make the public believe certain things, which you might also see. So I think that’s one line of thinking.

A different line of thinking — which is more late stage, once automation has progressed a long way — is that right now people can strike if they’re really upset. Right now people can rebel against the government, peacefully or otherwise. I think this will at some point cease to be possible. The striking will cease to be possible once people don’t have jobs: they won’t have any leverage of that kind. But also, once the military is fully automated, I think that human resistance and humans wielding human weapons won’t make any difference against an AI-powered military.

At that point it’s not clear what the government would have to lose, for example, by just cancelling an election — what power would prevent them from doing that. I’m not really centrally imagining that’s going to happen. I think you have to get into quite specific scenarios for that to make sense. But the basic point of, right now it is actually difficult for a government to cancel an election because the consequences of doing that would be so harmful: it’s not clear that they will be harmful in the future, that it would have any negative economic consequence for your country if you cancelled an election. Maybe people would be upset, but why would that matter?

Zershaaneh Qureshi: Yeah. Where I’m still feeling unsure as I’m imagining this play out is like, in this large group of would-be voters and would-be strikers, it’s not just ordinary people, but also a large number of people who are extremely, disgustingly powerful today, like disgustingly wealthy and influential already. You have to assume that they are somewhat tracking the ways that this might turn out, but may be worried that their current level of influence will get threatened if things get more concentrated than they are currently. And probably while some groups of people are losing their jobs, becoming less wealthy, becoming less important to society in other ways, there’s probably still some fairly large group of elites that still exists for a while.

I guess I’m struggling to imagine why they’re not putting enough pressure on, because they’re still at this point in time contributing. You know, they might be funding political campaigns or something like that, or they might still be pretty instrumental, unlike your average worker. I think I’m finding it hard to imagine why this group doesn’t mobilise or put enough pressure on the decision makers to prevent things from getting more concentrated. What do you make of that?

Rose Hadshar: One thought would be that you could ask this about many policy questions today. For example, existential risk seems bad for everyone. Wouldn’t you expect elites to be pushing for safe AI development? But in fact, elites are very divided on this, and lots of people aren’t paying attention, and I think that much less resources are going into this than should be. So I think priors are kind of on my side here for people failing to adequately take account of their own interests.

Some stories I can tell for why elite groups won’t be paying attention to this, I think actually it’s similar to what I would say for the general population: there are going to be loads of other extremely distracting, extremely important-seeming issues. People will be worried about whether there’s going to be a war, and whether they’re going to have to fight in the war. This will be pretty salient to people. Things like becoming unemployed will be pretty salient to people. I think that the whole AI sentience / AI rights thing will at some point become very salient also.

So that’s one piece: there’s going to be lots of very worthy, big-seeming, urgent issues that people will be drawn to.

Then the other thing is how much visibility will they really have on the power concentration stuff? This ties into the epistemics thing. In particular I’m worried that there’ll be by default very low transparency on what’s going on. It won’t be easy to find out who has the best models. It won’t be easy to find out who has most compute. It won’t be easy to find out what backroom deals are happening between big actors.

I’m sure there’ll be some information about this, and I’m sure there’ll be some people paying attention. But will it be easy to tell the difference between normal amounts of corruption and normal amounts of collusion between business elites, versus “this is something different where I might now lose power forever”? I’m not sure how easy that will be to tell. In worlds where it is very easy to tell and where there aren’t other issues that seem more pressing, then I think that you’re right that people will fight tooth and nail against this.

Zershaaneh Qureshi: Yeah, I can imagine if it’s a situation where you’re just seeing really obvious lobbying or something by AI companies, it’s more obvious. But I can totally imagine these particularly powerful people being kind of told by more powerful people around them, like, “Trust us, you’re gonna benefit, you’re gonna get a piece of this pie. We’re pals, we’ve done lots of business deals together. You know, we’re gonna be fine, it’s gonna be great for us.”

And presumably people always go into these kinds of situations with some degree of scepticism, but I think it would be possible to be persuaded. I basically think that I do buy that it might be quite hard if you’re in that kind of position to really realise that the wool is being pulled over your eyes or something.

Rose Hadshar: An additional point that your comment raises is it’s not clear that this will be against the material interests of these people. There’s a question about how easy it is to buy people off, either elites or the general public.

For example, in many situations you might end up with such enormous returns to capital that anyone who has any capital is going to become really very wealthy compared to the present day. Are people willing to sell their birthright for a lot of money? I kind of worry that they are, and that people won’t pay enough attention to things like, “Well, I’m going to be fine. The people who own no capital in poor countries are going to starve.” Or, “I’m going to be fine, but my children won’t have any political rights and there’ll be nothing they can do about that.” There’s some worry that there’ll be so much money sloshing around that the very top players will be able to buy off the people who could oppose them.

Slower-moving worlds could still get scary [01:23:57]

Zershaaneh Qureshi: Yeah, OK. I feel like this side of things is beginning to make more sense to me. But there’s another thread that keeps coming up throughout this conversation, which is the assumption that we’re in a world that’s just moving quite fast. In particular there’s a few things being imagined. It’s like, we start to have AIs automating a lot of stuff in the economy, including AI R&D itself. So we’re getting AI systems rapidly becoming more and more intelligent. We’re getting AI rapidly pervading throughout society and doing more and more of the complex work, or more of the heavy lifting in the economy.

And this is all translating really fast into increased industrial capacity, and also the amount of production is crazy. Economic production is just so many times higher than we’ve ever seen before. And information is also bam, bam, bam: it’s everywhere. It’s moving really fast. It’s hard to keep track of what’s true anymore.

In that kind of world, you imagine a couple of things. You imagine that it’s pretty hard to notice when things are going badly and do something about it — especially when the thing you need to do about it is redesign your entire social structure or something. And the other thing that happens is: with some of this, if you are the leading AI company, the first to get to this point where you are in an intelligence explosion, getting these AIs rapidly redesigning better and better AIs, you could just quickly pull ahead of the rest of the competition, which is also a way in which power gets concentrated.

So I think it seems pretty load bearing to me basically that we’re in this quite fast-moving world. But then there is some evidence that things might not move quite this fast, that AI progress will be a bit more modest, things will take off more gradually once we start getting AIs automating AI R&D because of various things.

Maybe it’s something to do with the permits and red tape side of things, where it’s just hard to get stuff done quickly. Or maybe it’s on the side of we shift to a world where we need to do more inference time compute — i.e. AI systems, once they’re deployed, need to spend more time thinking and use more compute while they’re being run, which means we can run fewer of them and they take a longer time to do things and stuff like that — which makes the feedback loops that we’re talking about in an intelligence explosion go slower. And there’s just a whole host of other things that people point towards that means that maybe the world just doesn’t move this crazy fast.

In that case, I wonder whether the threats that you’re talking about seem quite so worrying. Does that sound right to you? Or do you think that we should also be very worried about slower-moving worlds?

Rose Hadshar: I’m still quite worried about slow-moving worlds. I think basically I want to make a distinction between different routes to power concentration here.

So there are some kinds of power grab that I think you only get if there’s a big intelligence explosion, or it’s much more likely that you get with a big intelligence explosion. So the robot army scenario, where one company becomes so powerful that it can very rapidly and secretly build a robot army and take over the world, would be a classic example of one that requires a really quite big intelligence explosion. And for that kind of scenario, I’m like, yeah, this is entirely load bearing. If you don’t get that, you won’t get that scenario.

For other things, though, I think you don’t need a big intelligence explosion at all to be concerned about concentration of power. And this includes some power-grab scenarios as well. For example, the government power-grab scenario that I described — where a head of state is deploying AI systems in the military and making them personally loyal to that head of state — there doesn’t need to be any big capability gaps between AI companies: there could be multiple AI providers, lots of different companies who are competitive, who are selling systems to the government. The key thing that’s gone wrong there is backsliding in the government and a lack of coordination between AI companies to prevent the government from doing this unethical thing with their systems.

So there are power-grab scenarios which you should worry about even in slower worlds. Also, there are non-power-grab scenarios which more come into play in slower worlds. A lot of the economic scenarios we were discussing would happen quite late in the day, and it could be after quite a gradual period. I also think that the epistemic factors potentially matter more in slower worlds, because in fast worlds there’s not really enough time for significant amounts of power to be garnered via these epistemic feedback loops, which are a bit slower.

So I’m still concerned about power concentration without a big intelligence explosion. Maybe it’s worth us drawing a distinction between “intelligence explosion” and “the pace of change is faster than today.” There’s a big spectrum here, because an intelligence explosion really is a very extreme amount of change. Even if there’s not a big intelligence explosion, the kind of AI progress I’m imagining is still going to have very big impacts on how fast things move in the world — even if it’s quite distributed and lots of actors have similarly capable systems and so on. I’m still expecting that it’s going to become much more difficult for humans to track what’s happening than it’s been in the past, and I’m still expecting that the pace of technological change will be much faster, even if it’s not 100x faster.

If you had some position that was more like, “I really think business as usual is going to continue the whole way,” then I would be more willing to accept that yeah, if it was really going to be business-as-usual speeds, then I expect human institutions will be able to respond in time. But I don’t personally find the business-as-usual speeds very plausible.

Zershaaneh Qureshi: OK, yeah, I think I’m grasping that. Where else might the speed be very important? One of the things that I mentioned is speed being important for one AI company getting a huge capabilities lead over others — because one would imagine that if they get these very rapid feedback loops in their development of better and better software, once they hit a certain point, they can very rapidly pull ahead of the rest of their competition, which is one way that power could get concentrated.

But it does strike me that even if you have a world where lots of AI companies are roughly neck and neck, you could still have very intolerably concentrated power in a sort of group of executives of different companies. Does that seem like the right way of thinking about this?

Rose Hadshar: Yeah, I think that’s right. You can imagine it as a group of executives, or potentially a group of government officials, or maybe some mixture where now there’s a big international project involving a bunch of different AI developers, and the central committee of that project calls the shots.

Zershaaneh Qureshi: Yeah. And that still seems extremely bad if the other dynamics that you talk about play out, such that it’s only that committee who’s really steering how the future goes. That also seems very bad.

Why AI-powered tyranny will be tough to topple [01:31:53]

Zershaaneh Qureshi: OK, so even if we do buy that this very extreme concentration of power from AI does happen at some point, you might still not be convinced that it’s as big a deal as your article might suggest that it is, in that you might question how long lasting that state of affairs might be.

It does seem that there have been lots of cases throughout human history where things like technologies or shifts in the economy have enabled a group to take control to an undesirable degree. But I think previous instances of this, like previous totalitarian regimes like the USSR and Nazi Germany, they’ve all fallen eventually. There are various reasons why it’s quite hard to retain your power for that long, not least because most people don’t want that to be the case.

How likely does it seem that the power concentration enabled by AI in these scenarios would not be reversible, would have exceptionally long-term effects?

Rose Hadshar: I like your bringing in historical examples, and maybe it’s useful to think about a few ways that AI could make it different to how it’s been before.

So if you think about things like the USSR or Nazi Germany, part of why these regimes fell is because there were rival regimes. In the case of Nazi Germany, this is very clear: they were just conquered by other powers. In the case of the USSR it was less direct, but still there’s some sense in which they were in visible competition with other powers that run things differently, and that in the end outcompeted them.

So one concern that you might have with very extreme power concentration is, if we get to the point where there really is just one global hegemon, there won’t be any force for competition anymore. There aren’t other powers that could invade them or that could have a better social structure such that they get outcompeted economically later, so you might end up with stasis because of that. That’s one thing that could be different if power concentration becomes very extreme.

Another thing that could be different, even if you don’t get to a global hegemon, is that AI might enable us to make binding commitments in a way that we haven’t been able to do before, where you can get AI systems to enforce deals in perpetuity. It’s not clear that this will be technologically possible, but it might be. And if it is, even if there are still multiple powers, they might end up making some kind of deal to divide the resources in perpetuity, such that power-concentrated states end up with control for the rest of time over some slice of the universe in a way that seems very worrying.

Those are some reasons why you might think this could last for much longer than previous dictatorships have lasted. I will say that in general I’m kind of sceptical of “forever” claims, and worry that we lack imagination and that things will in fact continue to shift in ways we haven’t thought of. But I don’t think that you need forever to justify worrying quite a bit about extreme power concentration. I think partly you can just think that this is a bad state of affairs in and of itself, and that you care about it even if it lasts 100 years or 10 years or whatever. We just want this not to happen.

Zershaaneh Qureshi: Yeah, it definitely does seem to me that all of this feels pretty worrying, kind of regardless how long it goes on for. I think the place where it’s sort of salient how long-lasting these effects are is if you’re trying to make the decision about, like, do I want to devote my career or my efforts towards solving this problem, or towards preventing human extinction or something which literally we can’t get back from that. Do you have takes on how you would prioritise working to prevent extreme power concentration versus extinction threats?

Rose Hadshar: Yeah, good question. One comment would be that currently extreme power concentration driven by AI is way more neglected than preventing extinction, for example, from AI. So just on neglectedness grounds, it doesn’t need to be as important: it can be [1/100] as important and still pretty competitive on this margin.

Then responding to a specific kind of existential risk, which is AI risk and the risk of AI takeover, I think this is actually often somewhat over-egged as a clearly forever kind of problem. I agree with you that literally all humans being dead is irreversible — but a lot of the existential risks people are worried about from AI don’t result in that, and therefore have more of this uncertainty about how things play out over the long term.

So you could imagine that there’s some kind of AI takeover, and maybe humans are then disempowered over the future. Maybe the AI leaves humans on Earth, goes off and does something else, but there can still be further changes. What happens when the AI meets aliens? Or what happens if the AI decides to self-modify and then it has some different view about what it should do to the humans? So there is still the possibility of strategic change not driven by humans, but it’s not as absolute as pure extinction, I think.

Zershaaneh Qureshi: Yeah. And as you sort of lay it out there, it does sort of seem like actually the differences between the worlds where AIs take over humans and disempower them, but humans continue existing, is not all that different from power concentration situations where some group of humans take power and the rest of humanity is sort of subjugated. Actually we might expect them to be similarly bad and share lots of characteristics, these two scenarios.

Rose Hadshar: Yeah, I agree.

How power concentration compares to “gradual disempowerment” [01:38:18]

Zershaaneh Qureshi: One other observation here is that the selection of stories that you lay out also seem to have pretty similar dynamics to threat models that people call “gradual disempowerment.” There’s a paper by Jan Kulveit, Nora Ammann and others that sets this out pretty well.

As I understand it, that threat model has pretty similar dynamics going on: there’s some degree of humans becoming kind of irrelevant because of their not really doing much economic production anymore compared to AIs; there is some degree of AI controlling or being used to control the information environment. But the outcome is a bit different: as I understand it, AIs, rather than humans, end up with the power in the situation, but through these pretty similar mechanisms. And maybe another difference is that it’s not necessarily concentrating power into the hands of just a few people; it’s shifting who has the power to what could be a really large group of AIs. But it does just change the dynamics of power quite a lot.

Do you think that there are important distinctions between those threat models, where it’s the AIs themselves who are in power, versus the ones that we’ve been talking about today where it’s like some group of humans who have all the power? Would you approach these problems in the same kind of way, or is it important to treat them quite separately?

Rose Hadshar: I don’t think it’s important to treat them separately. I think that they’re very overlapping, and I think right now it’s somewhat unclear which frames will be most productive. And I think it’s good for people to be working in different frames and pushing ahead on them.

In terms of whether there are important distinctions, I’ll flag a distinction which I don’t think is that important, which is whether humans or AIs end up in power. This is something that I’ve chatted with the gradual disempowerment people about, and they’re like, “This is such a random, arbitrary line to draw. Why would you draw it? Why are you talking about power concentration just for humans?”

And I kind of agree with them. The reason I’m drawing the distinction is just for analytical simplicity. I’m like, let’s carve off a smaller chunk of the space and talk about these kinds of outcomes. It’s not because I think that the dynamics that could lead to AIs in power are fundamentally distinct. There are some routes that I expect to be distinct, but I do think that there’s going to be a lot of overlap there, so I’m not trying to say this is a really important conceptual distinction. I’m just going to say it’s pragmatic, and some people will find power concentration easier to think about and others gradual disempowerment.

Another conceptual distinction is this thing about are you thinking about power becoming concentrated or are you thinking about most people being disempowered? Which doesn’t necessarily mean that power is being concentrated; it just means the power is shifting. There again, I’m pretty sympathetic to the gradual disempowerment view that power concentration isn’t the only form of bad power shift, and we should also be concerned about other ones.

Maybe a final distinction, which is more a distinction in emphasis than in conceptual landscape, is that I think the gradual disempowerment people are thinking more in terms of emergent dynamics and systemic forces than, for example, people thinking about power grabs who are thinking more about agentic strategic action. I think everybody would agree that there will be some of both of these things, and then there’s a question about whether we should prioritise one over the other.

My personal opinion is we should use frames that allow us to keep both in scope, because they both seem likely to be important dynamics. We see that people are power seeking, and we see the actors seek power for themselves. We also see that there are important structural forces that shape how power shifts. So I’m kind of pro keeping both in mind.

Zershaaneh Qureshi: Yeah, OK. That makes sense to me. Are there examples of interventions that would help with both of those ways that things could go wrong, or examples to the contrary where there is something that you can do that only helps with the gradual disempowerment story, but doesn’t help with extreme power concentration?

Rose Hadshar: Yeah, I think there’s a lot of overlap. In particular, a lot of the things that you’d want to do to strengthen our epistemics to prevent power concentration are also very helpful for preventing gradual disempowerment. So this is things like better transparency into AI companies, building AI tools that help us with our epistemics, collecting better data so that people can make more sense of what’s going on in the world. All of this stuff I think is helpful across the threat models.

I’m less familiar with gradual-disempowerment-specific interventions, but I can give some examples of interventions that only help with some kinds of power concentration and don’t help with gradual disempowerment. The obvious ones would be interventions that really specifically target particular kinds of power grab. And I think those interventions don’t help you much with gradual disempowerment dynamics.

One example would be alignment audits, where you audit models to try and find out if they have secret loyalties. This is very helpful if you want to prevent a secretly loyal military from couping a country. It’s not very helpful if you’re worried about gradual disempowerment. So yeah, things definitely come apart here.

Zershaaneh Qureshi: Right. OK.

Some interventions are cross-cutting — and others could backfire [01:43:54]

Zershaaneh Qureshi: So within the space that we’ve carved out here, there’s still a really wide variety of different dynamics that seem like they could cause problems. There’s the military power grabs; there’s the economic irrelevance; there’s the idea that we didn’t really talk very much about, of an AI company outgrowing the world; there’s these epistemic factors where it’s hard to navigate the information environment; there are stories where there is some bad actor involved versus stories where it’s just structural things rather than somebody deliberately doing something to take power.

Through all of this landscape, how do we go about deciding which thing to focus on, given that there are probably different interventions that help with different things?

Rose Hadshar: This is a really good question. I’m going to start with a kind of non-answer, and then I’ll try and get into answers.

So my non-answer is: I mostly think that it’s appropriate to be really uncertain about this right now. There hasn’t been much work on any of this stuff. I think we’re still fairly conceptually confused about the space. The main thing from my perspective is that I want to try and keep as many of these dynamics in mind as possible, and not jump too soon onto a particular threat model before we’ve got a better sense of the landscape.

Others are a bit more confident about prioritisation than I am. So some people have a strong intuition that power grabs just don’t seem very likely: it’s very sci-fi, it’s very far fetched. I think that changes in the world driven by AI make this seem plausible to me, and I don’t think it should be off the table.

Other people feel really confident that power grabs are the most important bit, and I’m also fairly uncertain there — in particular because I think that power grabs, while they might be quite important, might also often happen late in the day, and then not be the most leveraged place to intervene. I’m imagining power grabs might often happen at the end of some long, messy process — where the world already looks really different by the time we get to the power-grab stage, and our clever ideas about how the current US government could protect itself against coups are kind of irrelevant, because there’s actually some weird new AI–human organisation that does the power grab and none of these things count.

So that’s the non-answer about how uncertain I am. Then the actual answer that I would give right now is that I think that we should be probably prioritising cross-cutting things that help with multiple of these different power concentration scenarios.

I think an overall frame that you could have for this is that stuff that helps improve societal checks and balances feels kind of robust and like it crosses lots of these models. More concretely, this kind of pushes towards focusing on the epistemic stuff over focusing on the power-grab stuff or the economic stuff. So that’s where I would lean at the moment, although I still think there’s great work to be done on both of the others as well. But if I were to pick one area, I would pick the epistemic stuff because it’s more cross-cutting.

Zershaaneh Qureshi: Yeah. So the reason that it’s more cross-cutting is generally just preserving our ability to understand what’s happening, and know what good ways to respond to that would look like, seems like it’s just going to be broadly helpful in a lot of worlds — even more than just the ones that we’ve been talking about today, in fact.

Rose Hadshar: Exactly. It’s both cross-cutting within the power of concentration scenarios where it helps you with power grabs, but it also helps you with economic routes to power. But then also it helps you with a bunch of other problems from AI. It’s fairly broadly applicable, I think.

Zershaaneh Qureshi: In the stories that we are thinking about, things happen in certain orders. Does it make sense to try to prioritise the things that you think are most likely to happen first, like the earliest points of intervention or something?

Rose Hadshar: I think something like this could make sense. I’m not sure if ultimately the reason to do it is because they happened first, or if the underlying driver is that it’s just more tractable to influence things that happen in worlds that are close to our world — and the more you’re imagining that by the time this happens, the world has already radically changed, partly it’s just hard to model and therefore hard to intervene on.

Another consideration is that if the world is radically changed by the time this happens, there’ll be lots and lots of AI labour between now and then that could help with solutions. Whereas things that happen early only we can think about. So yeah, I do think that there’s some reason to focus on the things that you think will happen first.

Zershaaneh Qureshi: Yeah. Another instinct that I have — I’m not sure if it’s any good, I don’t know if it’s worth focusing more — but I somewhat feel more worried about the stories that involve bad actors deliberately doing something compared to the ones that are just structural, because in the bad actor case, you’ve got an adversary that’s deliberately trying to make things happen a certain way. Do you think that’s a fair instinct to have?

Rose Hadshar: I definitely think it’s a reasonable instinct to have. I feel a bit confused about it. Part of my confusion here is: are there some kinds of structural force that it makes sense to model as adversaries, even though they’re not actually agentic? And then what even is an agent? And so on. So I started to get confused at some kind of conceptual level about what the distinction really is.

But I think if I had to pick right now, I would agree with you that it’s more worrying in worlds where powerful, strategic actors are trying their best to get here. And honestly, I think that’s probably the world that we’re going to end up in.

Zershaaneh Qureshi: There’s a lot to gain.

Rose Hadshar: There’s a lot to gain, and we see the powerful acting strategically today. And you can tell stories of like, “But people are going to realise just how terrible this is, and everyone’s going to suddenly start being sensible” — but this doesn’t seem to have happened previously, so I’m kind of not betting on it.

Zershaaneh Qureshi: Yeah. That seems fair. The other thing that I have a question about here is: we’ve talked about how some interventions could be cross-cutting. I worry, because the landscape is so complicated, that there might be interventions on one issue that actually worsen another or something like that. Does that seem true to you?

Rose Hadshar: Yeah, I think there is some of this. I think this gets clearest for some of the power-grab-related interventions, where there’s a worry that some interventions are really just shifting the likelihood from one actor seizing power to some other actor seizing power, but it’s not clear that it’s actually reducing the risk overall.

So some interventions that you might have this worry about: the alignment audits thing that I mentioned, where you audit models to check whether they’re secretly loyal, might help prevent companies from staging coups, but also might weaken companies as a check on government power. So you could imagine well-intentioned companies that put in secret loyalties such that they can deactivate systems if the government is trying to misuse them to seize power — and then if you prevent secret loyalties, you prevent this route to checking the power of an abusive government.

On the flip side, you can say the same kind of story about measures like government oversight over labs. Maybe this is helpful for reducing the risk of lab coups, but it also makes it more likely that government actors can abuse their position.

So I think there’s some real worry there about shifting probability mass around. You also see this kind of element with other interventions though. So you could imagine that some AI tools for improving epistemics are quite dual use: maybe you develop a tool which helps people coordinate, and that also helps the people who want to stage a coup to coordinate. So there are worries like this at lots of different levels.

Zershaaneh Qureshi: Yeah, OK. So what I’m hearing, of the space of options for how to intervene, it seems like quite a few of them end up in what I would call a “who watches the watchmen?” situation — where we can maybe prevent one group, like AI companies, from becoming disproportionately powerful, but we do that by making government or people who set standards or something like that more powerful.

I guess the worry that I have there then is: how much leverage do we actually have to stop power getting concentrated, full stop, versus just shifting who ends up with the most power?

Rose Hadshar: Yeah. Maybe if I were thinking about it in terms of leverage, it would make me think about the things that you were mentioning about the electorate and about manual work and maybe human wages will go up in the short term and so on. There is a broad base of leverage in the current world where lots of people have a small amount of power. And, particularly if AI tools make it easy for lots of people to coordinate, it might be the case that the majority becomes much more powerful than it has been in the past and is able to coordinate much more nimbly and quickly to push for its own interests.

So I don’t think it’s hopeless, but I imagine that this is a bit how framing the US Constitution felt or something — like, “I want to balance this, so I’ll balance it with this. But then how do I keep that in check?” And at some point you stop adding more checks, and you think, “OK, this is a balanced system.”

Zershaaneh Qureshi: Yeah, OK. So what if we did end up in a situation where there is just a tradeoff that we really struggle to avoid — between either we let companies do whatever they like, or we let the government have really extreme control over more stuff than we would have hoped they’d have — is there an answer for who we should trust more? Like who we pick in this awkward tradeoff?

Rose Hadshar: I don’t think there’s a robust answer. People have very different intuitions here. If you ask somebody, “Who would you rather was dictator of the world: the US government or Google?” I think you will get some people who will say definitely the US government and other people who will say definitely Google for different reasons.

I think that the kind of Google camp is imagining that governments are extremely inefficient and badly incentivised, and Google would do a much more competent job of being world dictator than the US government. And I think the people who are favouring the US government are more tracking things like representation and there being checks on the power of governments that are more robust than the checks on companies.

What fighting back actually looks like [01:55:15]

Zershaaneh Qureshi: Pushing on: I do want to talk a bit about just how easy or difficult it might be to prevent these kinds of extreme power concentration scenarios from happening. One view here is that it might just be extremely hard to do something about this, just given that some of the most powerful people in society are going to want to stop you from changing the situation because they benefit from it. Given that, do you feel like we should have hope for solving these problems?

Rose Hadshar: I do think we should have hope. There are a few reasons for this. One is a kind of meta observation that people object to working on power concentration both because they think it’s too easy to solve and because they think it’s too hard to solve. This, I think, is a reason to think that maybe it’s somewhere in the middle. So that’s a somewhat jokey reason to think we should have hope.

More seriously, I think that we’ve spent very little time so far trying to solve specifically extreme power concentration that’s driven by AI. And it seems like we should definitely be willing to put more time in it than we’ve already done to see if we can get stuff to work.

I also think there are kind of general background reasons for hope. As you said lots of times, most people don’t want this. It should be possible to get a very broad coalition of people who are on board with preventing this.

Another more speculative reason for hope is just that we actually have spent a long time as a civilisation thinking about power checks and balances, how you restrict power concentration to reasonable levels. We haven’t thought about that in the context of AI, but we do have a lot of form thinking about relevant questions, and maybe particularly as we start to get better AI automation that can help us to put these things together, we might be able to think of some pretty good answers to this.

Zershaaneh Qureshi: OK, that feels quite hopeful to me. What are the interventions that you feel most hopeful about?

Rose Hadshar: At this stage I’m more confident in high-level objectives than in concrete interventions. But I’ll give some examples of both to show the sorts of things I’m excited about.

One is building AI tools to help us with our epistemics, which as a broad category I feel pretty bullish on being helpful. Some specific examples which I’m less confident in would be AI for fact checking, AI for forecasting, publicly available tools that help civil society and courts, and other checks and balances to make sense and act more strategically in their own interests.

Another bucket that I think is very promising is AI-enabled checks and balances on the power of governments. I basically think that in order to navigate this well, we’re going to need governments to deploy AI — otherwise they’re just going to fall behind companies, and companies will be able to get what they want out of the situation. So we’re going to need government deployment. But also government deployment comes with lots of risks of its own in terms of power concentration. So there’s some sort of how do we thread the needle here? What does safe government deployment look like? That I think is very important and well worth people investing time on.

There are a few concrete things here which I think are quite promising. One is law-following AI, which has been proposed by O’Keefe and others at the Institute for Law & AI. This is basically just training AI systems to follow the law, such that they can’t in fact do things which violate the law. This seems like a really good, robust policy to have for government AI systems, and I will be very excited about people pushing on that agenda.

Another example would be procurement rules for what kinds of procedures there are for companies that provide AI systems to governments and what safety checks need to go through. So that’s another bucket.

I’ll just gesture towards some other things that I’m quite excited about in this regard.

• One is transparency: making sure that the right actors have access to the right information about what’s going on in AI companies and in governments.
• Sharing access to compute and capabilities to try to reduce the gradient between the most powerful and the less powerful, to try and make sure that access is a bit more distributed.
• And there are specific actors that I’d especially like to empower with AI. Making sure that the legislative branch and the judiciary has access to really great AI systems, so that it’s not just the executive, seems like a very good intervention to me. But also things like that investigative journalists have access to the best models, and can be using them to help them find out about what’s going on behind closed doors.

I think I’ll stop there. Those are the main things that I’m most excited about, with varying degrees of concreteness to them.

Zershaaneh Qureshi: OK, interesting. Something that you said a moment ago was that you noticed how humanity has put a lot of effort into thinking about power distribution and concentration and what to do about that. We already have a fair amount of interest in stopping power being concentrated, and we’re already doing various things to that end: there are antitrust laws, there’s taxation of the wealthiest, at least in some countries. And lots of people seem pretty interested in avoiding inequality broadly, and avoiding misinformation.

Then you list a few interventions for extreme power concentration scenarios that feel very specialised and very different to the stuff that we’re already doing. I have some kind of curiosity about why think we need more targeted work? Why not just think that what we’re already doing and already thinking about we’ll just scale up? Why divert more specialised effort in this direction?

Rose Hadshar: Yeah, good question. I definitely think that the work that’s going on today to stop non-AI-driven power concentration is relevant and helpful here. So I’m not trying to dismiss that or say it has nothing to do with the problem I’m concerned about. It definitely does relate.

I still think it’s extremely important to do additional things to target extreme power concentration. Mostly this is because I think that the risk is pretty AI specific. A lot of these risks arise only once AI is capable of automating a large fraction of economically valuable labour, whether that’s power grabs by automating the military or government functions, or that’s economic dominance. One thing this implies is that we’re going to need to have interventions that restrict AI systems themselves, and you’re not going to get that via antitrust law or via regular investigative journalism. You need to think about it specifically.

But I think another broader thing is that I’m not really imagining that the way that we stop extreme power concentration is just by holding on tight to our current checks and balances. I imagine that we’re going to need to have new ones that are designed specifically for an era where there’s lots of AI labour that can be made loyal to individual people. I don’t think the people who are doing great work on antitrust or great work pushing for better equality measures in the current economy are naturally going to be thinking about this, and that their work is naturally going to tend towards AI-tailored checks and balances in future.

To get a bit more concrete, there are some things that I just don’t expect will happen at all from moderate efforts. Some examples of this would be maybe the law-following AI stuff is not likely to happen via existing efforts. I think that the alignment audits that I talked about, or better infosecurity within companies to prevent people from tampering with models to make them secretly loyal, that these are all specific to threat models involving AI. I don’t think they’ll happen without specific intervention.

There are other things where I think some stuff like this will happen, but I still think there’s a big room for improvement, particularly when you’re tracking these really extreme possibilities. This is things like government AI deployment. Clearly a lot of attention is already being paid to how AI is deployed in government, but I think a lot of it is local attention on how should we do this specific deployment, or it’s more broad attention, but it’s not tracking these very extreme risks that could arise down the line. So I still think there’s room for us to make a big improvement on the status quo by paying attention to this from the perspective of extreme power concentration.

Zershaaneh Qureshi: Yeah, it does seem like we could do better.

Why power concentration researchers should avoid getting too “spicy” [02:04:10]

Zershaaneh Qureshi: OK, I think where I land with this is, though I want these problems to be discussed and I want them to be addressed, I do still feel apprehensive about encouraging lots of people to do work in this direction. I think the main reason that I still feel worried is that it just all seems quite delicate and quite political. So I think it seems worth highlighting ways that your work could backfire, or what not to do if you want to work on power concentration.

My first question is: it seems like this could be an area of work where the things you publish could give people dangerous ideas. So for listeners who might want to work on this issue and contribute to the discourse about it, how should they be thinking about that? What kinds of things should they avoid doing?

Rose Hadshar: Yeah, great question. This is a difficult one. To give a kind of crude example, you don’t want to publish a very detailed novel research handbook for how to do a coup using AI.

Zershaaneh Qureshi: Oh no, that’s what I was planning on doing!

Rose Hadshar: Right? And some things that this suggests: more abstract, less detail is good for public stuff. You want to be thinking about the specific actors that you’re writing threat modelling stuff about, and how much they’re likely to know. For example, maybe it’s not very likely that I, as a non-technical person, am going to notice some clever technical route that an AI lab CEO could take. So maybe I don’t need to worry very much about giving an AI lab CEO an idea, but maybe I would notice something that a head of state in a not-very-AI-powered country might not notice. So I maybe should be more worried about that kind of information.

I think in general if you think that you’re thinking of novel ways to do things that are very harmful, in this cause area or in other cause areas, you should be somewhat wary of publishing that.

Zershaaneh Qureshi: Yeah, that makes sense to me. Another worry that’s sort of related is not only might you give people dangerous ideas, but you might also kind of rile them up in the first place. Like if you write an article where you’re like, “These people are going to violently take over, remove all of your democratic rights, be incredibly greedy and ruin the future of humanity,” I could imagine that feeling kind of alienating to certain groups, and maybe making the political situation a bit more polarised or something, if people are being lumped in the category of elites who are going to ruin the world and your good average Joe or something. I have a worry that this might make the situation worse. What do you make of that, and how do you avoid it?

Rose Hadshar: Yeah, I definitely think there are ways of writing about this that could make it worse. And throughout this conversation we’ve been talking about the good world as worlds where there’s this very broad coalition of people who don’t want this to happen.

I think a risk with kind of fiery, non-nuanced, uncareful work to try and reduce the risk of power concentration is that in fact you politicise the issue and you make it hard to create coalitions like that — because some people become ideologically opposed to, for example, interventions that reduce the risk of power concentration because they see those as right-wing or left-wing, or they see those as coded in some way.

I think this is a more subtle thing, where it’s significantly about what’s the tone that you write with and which words do you choose. I think it’s good to get a lot of feedback on this sort of thing. When writing the problem profile that I wrote on extreme power concentration, I initially wrote a problem profile on AI-enabled coups — and part of the reason I changed that was for this consideration. It was also because I was starting to notice that I actually care more about these other dynamics besides coups as well, but a significant part of the decision was that this seems like a more spicy framing that’s more likely to tend towards political discussion — and maybe it’s better to have a slightly more boring title like “Extreme power concentration” that people are less likely to fight about.

Zershaaneh Qureshi: So the advice is be less interesting, be more boring.

Rose Hadshar: I think don’t lean into spice on this topic. Boring is good.

Zershaaneh Qureshi: Boring is better. That makes sense to me. I guess the thing not to do is to just go into working on this and being like, “I am just going to publish everything that I think is true.” You’ve got to be more nuanced and more careful with what you’re doing here.

Rose Hadshar: Yeah. Unfortunately, because it’s a very political topic, you do need to think strategically about what you write. And for some types of researcher and thinker, that’s not going to be a good fit, because it’s going to be important to their process to be able to be more free. So I think this is suited to particular temperaments, this kind of work.

Zershaaneh Qureshi: Yeah.

Why the “Manhattan Project” approach should worry you — but truly international projects might not be safe either [02:09:18]

Zershaaneh Qureshi: Pushing on here, what do all of these concerns about power concentration imply for governance models? People often suggest models for AI governance, like an AI version of the Manhattan Project, where there’s some US-led effort to build AGI faster than its competitors. But there are other options, like you could have some kind of international consortium, or you could advocate for widespread open sourcing. There’s a whole bunch of things you could do. Do you have a sense of how good or bad these different models are from a power concentration lens?

Rose Hadshar: I think when you start thinking about power concentration, firstly, a Manhattan Project style starts to look a lot worse — particularly if it only involves the US and there are no other countries to provide a check, particularly if it only involves one AI company and there are no other developers to offer a check. So I’m pretty concerned about Manhattan Projects, and think that it would probably be bad.

I think it also has implications for other types of international projects though. So you can think about a properly international project where there are lots of different countries coming together — and maybe if you weren’t thinking about power concentration and you were just thinking about representation, this would seem like a really attractive option. The power concentration doesn’t necessarily make it a bad option, but it does make it a riskier option.

The concern would be if you bring all of this power together into one institutional body, that body becomes extremely important for whether or not power is going to be checked in future. So then it becomes a question of do you really back us to do institutional design from scratch in a way that’s robust to the AI future? And I worry about that.

One thing I will say about international projects and power concentration, though, is that potentially you can limit a lot of the potential for harm by making sure that there are multiple AI developers as part of that project.

For example, me and my colleague Will MacAskill are kind of pro a model that’s based on Intelsat, the organisation that managed the global satellite network. The model there is that there’s an international body that governs the development and deployment of AI, but the way that they do that is they contract things out to different private companies, so you still have multiple frontier companies who are similarly capable. Obviously this depends on there not being a massive intelligence explosion, and if there is, then you don’t. But I think that potentially you can limit a lot of the risk by having multiple companies involved.

Zershaaneh Qureshi: Right. And this Intelsat model that you referenced, you actually have an article on that, which we can also stick up a link to.

Rose wants to keep humans around! [02:12:06]

Zershaaneh Qureshi: OK, we are coming to a close. Before we do, we’ve talked a lot about the many different ways the future could go really wrong in really horrible ways. Do you have a vision for a flourishing future that you could share with our listeners?

Rose Hadshar: I don’t think I have a very concrete one, but I can say some elements that I would be excited about in a flourishing future.

One thing that I am excited about is there still being humans, and those humans being happy and human-like and getting to choose to remain human and still do things like have children, make music, write their own books, even if AIs write better books, this sort of thing.

Zershaaneh Qureshi: Yeah, that sounds good.

Rose Hadshar: I’m fairly pro there should still be humans in the future, and I think others are maybe less so than me.

I’m also quite excited about there being something like diversity in the future. I want there to be different pockets of the universe doing different things, and I’m quite frightened by “we should figure out what the best thing is and then we should use all resources to make the best thing” style arguments. I’m opposed to that. I want to do something much softer than tile the universe with the best thing — because I don’t think that’s very robust, basically, and because I intrinsically value things being diverse.

Zershaaneh Qureshi: Yeah, that makes sense.

OK, we’ve grilled Rose today on extreme power concentration scenarios, whether they’re plausible and so on. But we haven’t gone into the full details of the case that she actually makes in the article. I’d really encourage people to read that. You can read it on our website or listen to it in our narration feed.

Rose, you’ve been so great. Thank you so much for coming.

Rose Hadshar: I really enjoyed it. Thanks for having me, Zershaaneh.