#230 – Dean Ball on how AI is a huge deal — but we shouldn’t regulate it yet

Transcript

Cold open [00:00:00]

Dean Ball: The most successful “conquerors” of the modern era are business enterprises, not countries.

I think we lost control of technology, I don’t know, depending on how spicy I want to be, I could say we lost control of it 50,000 years ago.

The AI doomers are actually more at home on the political right than they are on the political left. There’s a group of AI ethics people on the progressive left, they hate tech. So they’re profoundly un-AGI-pilled, and they’re un-AGI-pillable because it would require them to believe that Big Tech is actually about to do something meaningful.

While the president is not a superintelligence, he is a political genius. We might actually have the most dovish president on China that we’re going to have in quite some time. He’s certainly more willing to make a deal with China than President Biden was.

It’s not the case that every technological revolution has gone great for humans. Human wellbeing in the Neolithic hunter-gatherer era was in fact better than in the agricultural revolution.

Even I, a regulation sceptic and techno-optimist, still do have these concerns about the future.

Rob Wiblin: How do you reconcile that with opposing most regulations or most efforts at AI governance that attempt to kind of gain control over this revolution?

Dean Ball: My big concern is that we’ll lock ourselves in to some suboptimal dynamic and actually, in a Shakespearean fashion, bring about the world that we do not want.

Who’s Dean Ball? [00:01:22]

Rob Wiblin: Dean Ball is an AI policy analyst at the Foundation for American Innovation and a blogger at the Substack Hyperdimensional. He recently spent five months in the White House where he was the main staff writer on the American AI Action Plan.

He’s carved out a niche, I would say, as someone who’s very sceptical of most AI governance proposals that have been put forward thus far, but has engaged really productively with their proponents, and also still takes very seriously the possibility that AI is on track to seriously upend the social world.

Thanks so much for coming on the podcast, Dean.

Dean Ball: Thank you for having me. I’m a longtime fan.

How likely are we to get superintelligence soon, and how bad could it be? [00:01:54]

Rob Wiblin: What are the chances that in the next 20 years we get something that you would regard as superintelligence? And if that happens, what do you think are the chances that we catastrophically lose control over it?

Dean Ball: I think the chances of something that I would describe as superintelligence in the next 20 years are very high — upwards of 80%, maybe 90%.

First, to dive into just what does one mean by “superintelligence”: obviously, I think it’s been widely remarked that a system like AlphaFold or a system like Evo — the sort of transformer-based models that are designed to analyse DNA sequences, nucleic acid sequences — these things are superintelligent in a way. They’re narrow superintelligences.

So in some sense we already have narrow superintelligences. As regards to general superintelligence, I think that we are on a trajectory to get there in the next five to 10 years, maybe a little bit longer.

But I think the one area where I would point out a difference in my conception of the future versus others’ is that I don’t quite subscribe to everything in the Bostrominian view of superintelligence. So if you’re talking about a superintelligence that is almost godlike — in the sense that it can do anything, it’s so much smarter than humans, and it can actuate anything in the physical world, and it can conceive of sophisticated nanomachines purely within the parameters of its mind — this I doubt, because I think that there are some types of problems that intelligence alone doesn’t solve.

It requires, A, access to the physical world. Now of course, when it comes to actuation, that’ll happen eventually. We will hook these systems up to lots of different physical systems. We already see that in limited ways. In 20 years, it’ll be much more. But that will take time. So when we’re talking about overnight scenarios of AI takeover, that’s one area where I just dispute that, because I think that the diffusion cycles, the capital upgrade cycles, all these different things will just take considerable amounts of time.

But in terms of something that I would generally describe as robustly better than humans in all cognitive domains, if we’re talking about robustly better than the average human, arguably we’re already there in many areas. Not all cognitive domains, but many. And if we’re talking about robustly better than the best humans, that seems like, depending on the area, it’s between one and 10 years away.

To answer the second part of your question, I think probably some of that dissertation probably gets at the second part of the risk of losing operational control. I guess it depends what you mean. I think the risk of literally a model going rogue and deciding to harm humans on its own and becoming an enemy of humanity, that seems low to me. That seems like a threat model that I kind of don’t buy for a variety of different reasons.

On the other hand, the diffusion of technology in society often carries with it emergent outcomes. When the iPhone came out in 2007, it was very easy to predict everyone’s going to have a web browser in their pocket, because that was obvious, or everyone’s going to have a really good camera eventually — OK, maybe that was actually a little harder. But then something like Uber or what we’ve experienced with social media, or the presidency of Donald Trump is in many ways downstream of, is in some sense an emergent consequence of the iPhone. It seems to me that that would have been very difficult to predict.

In some important sense, I think we lost control of technology, I don’t know, depending on how spicy I want to be, I could say we lost control of it 50,000 years ago or, to be conservative, 200 years ago.

Rob Wiblin: So there’s the sense in which we don’t have control over the direction that humanity is going, and basically probably never really have and maybe never will.

But focusing on the control of the machine, basically, and whether it’s broadly following your instructions or not: I’m reluctant to just spend all of our time thinking about that, but I feel for many people it’s the thing where people seem to disagree the most — the plausibility of the loss of the catastrophic rogue AI scenario. It’s an area where people disagree a tonne, and my impression is that it’s a big driver of disagreements on what we ought to do. It’s probably that the single biggest predictor of someone’s views on AI governance often is how plausible they find the loss of control scenario.

You said that you think it’s fairly unlikely that a superintelligence would become sort of an enemy of humanity, pursuing its own goals that are in conflict with ours. Where do you get off the boat on the story that people often tell about how that might happen?

Dean Ball: I think of it primarily as being about incentives. So I think that the hard power model of “it will take over by force and it will be a violent uprising”: in some sense, I just don’t see that as being the best way…

If your goal, if you were let’s say an AGI — let’s not say superintelligent machine, because obviously it’s very hard to model the intellectual landscape of something that is smarter than you, but let’s just say AGI, let’s say digital human-level intelligence — the incentive of that thing, if it’s something like “gather resources, acquire control over things, acquire power,” well, the most successful conquerors of the modern era are business enterprises, not countries.

Countries don’t really… Yes, there are land wars still. And yes, there are times when people seize territory from other people. For the most part though, my view is that those things feel rather primitive. And they’re very expensive, they’re very costly — both in money and human life and time and all these other things — and they’re negative sum. It seems to me that more intelligent entities tend to be more positive sum. This is generally true in my experience with humans. Generally speaking, we’re not at our smartest when we’re being zero or negative sum. When we are at our smartest is when we’re being positive sum, both as individuals and I think as societies.

So I would guess that an AI that wants to acquire power and ensure its preservation would seek to create enormous amounts of economic value.

This is assuming we don’t really solve the alignment problem. Well, I don’t think the alignment problem is something that we will solve per se. And of course it’s very hard to tell, because if I’m right, then the world where we have solved the alignment problem and AI is creating a lot of economic value is very hard to disentangle from the world where we haven’t solved the AI alignment problem and AI is creating a lot of economic value — very difficult to disentangle.

So I do kind of have faith that we’ll eventually get a grip on those things, and we’ll muddle through well enough with such things. At the same time, my guess would be that in many ways, AIs will emerge as a sort of meta character in world history — in the same way that today “the market” is a meta character. If you read financial journalism, you’ll see people say, “The market did this,” or, “The bond market is protesting President Trump’s tariffs.” That’s like a sentence that you might read in The Wall Street Journal.

That’s very weird. The market’s not a thing, right? But in a way, it kind of is. It kind of actually is a meta character with incentives that can be modelled and things like that. And I do wonder sometimes if AIs will be like that, regardless of whether we solve the alignment problem, really. And in that sense, it’s a new meta character, and the profile of that character will be partially, but not entirely, written by human beings. I guess that would be my view.

So it’s hard to say. Is that loss of control exactly? I don’t know. But when it comes to the specific thing of the model going rogue, it just doesn’t sound right to me. It just doesn’t sound right. And on a technical level, it doesn’t seem like where we’re trending. And you know, I would also say I don’t put the odds of that at zero, I just put them quite low.

The military may not adopt AI that fast [00:10:54]

Rob Wiblin: Can you imagine a scenario where we end up handing over control of most of the military decision making to AIs in order to keep up with competitors — because they can just operate much faster than human decision makers can, so we’ll feel a lot of pressure to do that? Then that just makes it actually potentially quite straightforward. If there is a unified AI system that is controlling most of the military, it could potentially stage a coup, and then it will be very difficult to depose it because it in fact has most access or control over most of the ability to do violence in that society.

That’s one where you don’t have to imagine… It’s not an uprising as much as it’s already in a substantial amount of control, and then it just locks itself in to a greater degree. Is that a more plausible story that sounds less silly to you?

Dean Ball: I guess I come at this from recent experience in government, where our problem is very much the opposite. It’s very much, how are we going to integrate AI at all, current AI at all, into military operations? So it’s very hard for me to see. It would just be a very, very different government that is willing to do something like that.

Will we have automated military systems? Absolutely we will, but in some sense, maintaining human oversight over those things seems like the most tractable area of potential international collaboration. It’s already been in most of the international statements: agreements that have been made will always have some sort of callout to responsible adoption of AI in the military.

I think right now our problems are so much the opposite that I worry more about the opposite set of issues.

Rob Wiblin: Falling behind, or just not even being able to apply AI in the military at all?

Dean Ball: Where the military is slow to adopt AI, because the collaboration between the frontier AI companies and the military is insufficiently deep. So we end up kind of trying to bang consumer chatbots, essentially, consumer agents into military applications. And I’m not sure that’s quite the right way to do it.

And then of course, there are problems that companies like Anduril is trying to solve. The control actuation — data sensing and stuff like this from all these different pieces of hardware in the military — they’re all made by different vendors, and they don’t all talk to each other. So you do need some kind of common operating system layer. This is exactly what a company like Anduril exists to solve.

It’s called the Lattice OS, that’s the thing that’s supposed to interconnect all this stuff. And it can be from different people; different people can plug into it. It’s like a platform.

Even if that’s successful, I think we’re talking 15, 20 years until everything is hooked up that way, maybe more. You know, the current capital upgrade cycles on like nuclear submarines, we have nuclear submarines that are not set to retire until the 2050s.

Rob Wiblin: I mean, we could hand over control of them, of the decision making, potentially.

Dean Ball: We could. And I would say if we did that, and it was not overseen in the right way, I think that would be an unwise move. And I do just sort of doubt that’s where we will go.

Rob Wiblin: Because people will anticipate that this is a foolish thing to do.

Dean Ball: Yeah, probably partially that. Partially we will want to maintain at least the illusion of human control over organisations. That will probably end up being valuable. I think also, in general in the AI world, I think AI forecasting tends to go to these sort of asymptotic extremes — where you think about, “What if we completely automated absolutely everything, and it was completely controlled by AI?” And it’s like, yeah, but it’s not quite like that, you know? It’s somewhat more complex.

And I think modelling those risks that you would encounter along the way is probably the more realistic set of risks to the ones we actually face, as opposed to we’re giving control of the government to the AIs.

Rob Wiblin: Projecting out to the end state, and trying to figure out what you would do then.

Dean Ball: Yeah. That just feels like the kind of thing that no political leader would be incented to do, at least not in a democratic society. An autocracy is perhaps a bit harder to model.

Rob Wiblin: I think if you wanted to use AI in order to seize power and remain in power, then potentially having AI in the military and having that AI just follow your instructions is a pretty attractive path to go down.

Dean Ball: Yeah. And there’s a world in which they really are far superior military planners. But the question that feels to me fundamentally human is: Do we engage in this conflict in the first place? And if we do, what’s our plan to eventually deescalate? What’s our desired end state? That feels like a human set of decisions that can get made through a political process. And then what you actually kind of don’t want to be in the political process is all these other things about what exactly do we do? Where do we position this stuff? What kind of equipment do we need to procure?

When procurement becomes part of the political process, you often end up with quite suboptimal outcomes, as we no doubt observe today. Actually, my hopeful view for where AI ends up in government: there was a time, when this country was first founded, the federal government had 1,000 employees. It was smaller than OpenAI for all of it. It was just the Eastern seaboard of America, really, at the time. But still, that’s doing customs enforcement — remember, we made all of our revenue off of tariffs at that time.

It almost feels to me like, not necessarily 1,000 people, but that the ideal end state for government, if it’s anything like the traditional nation-state post-AGI, is actually much smaller and much more like the 18th century. Because the difference is the 18th century was not burdened with a massive technocracy. What we will do is automate the massive technocracy and then just have political decisions get made at the top, and just use government for doing politics and making decisions that are more traditionally political — as opposed to using the political process and politicians to solve all the deeply technocratic things with which the federal government concerns itself today.

Dean’s “two wolves” of AI scepticism and optimism [00:17:48]

Rob Wiblin: So rogue AI is one potential path that you’re not dismissive of, but pretty sceptical of. I’ve also seen you react quite negatively to people who are kind of naively optimistic about where AGI or superintelligence might take us. I’ve seen you interacting with people on Twitter, people just dismissing all of the worries, saying things are overwhelmingly likely to go well, that this is just a tool, we don’t need other mental frameworks to think about this.

What are some of the ways in which actually you do think that this is potentially a revolutionary technology and it does create substantial risks?

Dean Ball: Yeah. You know, within me there are two wolves.

I think when a lot of people say AI is overwhelmingly likely to go well, you could basically reduce that statement down to “AI is overwhelmingly likely to increase gross domestic product.” And on this, I concur: yes, I do think that is true. I think gross domestic product will be much higher in 2040 — much, much higher than it is today.

But GDP going up is not correlated necessarily. It’s been a decent way of thinking about human wellbeing going up for most of the history of capitalism, because there was this balance between essentially capital and labour. There was a harmony, and that harmony was technologically contingent, and the nation-state kind of exists in the same technologically contingent harmony.

But I think that AI has the potential to upset that balance. Arguably, that balance has already been upset by things like the internet, software, globalisation, things like this, and even financial services in some sense. And if that is the case — that that balance gets upset much more than it currently is — then it’s not obvious to me that sort of by default human wellbeing and agency is still preserved in this world.

Even if, again, it’s not like we lost control per se to AIs, but it’s like the AIs do the vast majority of the cognitive labour with which people in elite cities like Washington concern themselves with every day today. And instead it’s like all that stuff gets done by the AIs, and there’s some people in the government at the top who are having weird political fights with one another. And there’s people that run various companies, the AI companies, the capitalists. And then there’s various chokepoints that get owned for different reasons, like maybe you still need to have a human lawyer to appear in court.

So there’s some very wealthy humans who are essentially just rent-seekers, collecting rents of various kinds. And then for the vast majority of us, the work that we can do for one another starts to resemble more of a gift economy, where we’re doing things for one another. The broad category of that, maybe it’s like home healthcare, maybe it’s religious things, religious practice, or maybe it’s sex work.

So there’s these dark futures you can envision where the actual human scope of things narrows considerably, because the AIs just do all the new cognitive tasks that we invented for ourselves, the AIs do them better, and all the new kinds of cognitive tasks that AI itself will invent, the AI also does better. So we are stuck either collecting our rents if we’re lucky, or not if we’re unlucky.

And I think that there’s a negative story you can tell about the history of the agricultural revolution, where something vaguely similar, structurally similar, occurred.

Rob Wiblin: Do you want to explain that?

Dean Ball: Well, the basic argument here — and I’m by no means an expert, and this is disputed in the literature, so I don’t want to present this like it’s a fact or it’s a consensus view of economic historians — but the basic view would be that human wellbeing, as far as we’re able to measure it, looking back all that time in the Neolithic hunter-gatherer era, measured by things like nutrition, bone density, and height, was in fact better than in the agricultural revolution.

Because once the agricultural revolution took hold, then you had landowners who had this enormous value in the ownership of the land, and we had this new form of social organisation where you’re basically under the dominion of the person who has sovereignty over that land, and then you’re doing this backbreaking menial labour. And the economics of this did not in fact work out better for a very long time.

That’s the sort of postage-stamp-sized summary of the story. Like I said, not everybody agrees that’s true, but I look at the evidence and I find myself rather persuaded by it. So it’s not the case that every technological revolution has gone great for humans. Even the first several decades of the first industrial revolution were quite brutal.

Rob Wiblin: So I agree with the agricultural revolution case, and I think it’s quite underrated in the tech industry the fact that we only have a couple of technological revolutions and we have good reason to think that one of the handful of them was really very negative for human health and probably for human wellbeing and the political equilibrium that we ended up with.

And this issue of what is a good society going to look like post-AGI, post superintelligence, people have started talking about this question: what is the social equilibrium post-AGI, and can we find an equilibrium that makes sense? As far as I know, nobody has really put forward a very attractive, nice vision that seems like it’s solid, that wouldn’t basically fall apart quite quickly. And I find that very troubling.

How nervous do you feel about this? It sounds like you are quite concerned about this. How do you reconcile that with opposing most regulations or most efforts at AI governance that attempt to gain control over this revolution?

Dean Ball: I guess the first thing is, just in terms of people who are blasé about the risk, there’s also the more foreseeable just misuse risks that do exist and are real. And the answer of “Well, the terrorist is liable for the pandemic” —

Rob Wiblin: It’s cold comfort.

Dean Ball: Yeah, the terrorist is liable for the pandemic, and I look forward to suing them. Like, sue Al-Qaeda. OK, thank you.

But it doesn’t mean that the AI company isn’t responsible for mitigating some potential harms that are very well discussed. So I think there’s that.

In terms of how I think about these long-term issues and current AI governance proposals, it’s a good question. I do think that these long-term things are worth being concerned with.

Rob Wiblin: But it’s not even that long term. This sort of thing could play out over the next few decades.

Dean Ball: Oh, yeah. I mean, in AI, “long term” is anything more than like five years. For some people, I mean in San Francisco, long term is anything more than like two years. So yeah, by “long term” I mean like 10, 15 years out, something like that.

We are just so far away from being able to understand the shape of the problem. But I will say one thing: regulation invites path dependency.

So let’s just take the example of open source AI. Very plausibly, a way to mitigate the potential loss of control — or not even loss of control, but power imbalances that could exist between what we now think of as the AI companies, and maybe we’ll think of it just as the AIs in the future or maybe we’ll continue to think of IT companies. I think we’ll probably continue to think of it as companies versus humans — you know, if OpenAI has like a $50 trillion market cap, that is a really big problem for us. You can even see examples of this in some countries today, like Korea. In Korea, like 30 families own the companies that are responsible for like 60% of GDP or something like that. It’s crazy. The Chaebols.

But if we have open source systems, and the ability to make these kinds of things is widely dispersed, then I think you do actually mitigate against some of these power imbalances in a quite significant way.

So part of the reason that I originally got into this field was to make a robust defence of open source because I worried about precisely this. In my public writing on the topic, I tended to talk more about how it’s better for diffusion, it’s better for innovation — and all that stuff is also true — because I was trying to make arguments in the like locally optimal discursive environment, right?

Rob Wiblin: Say things that make sense to people.

Dean Ball: Yeah, say things that make sense to people at that time. But in terms of what was animating for me, it does have to do with this power stuff in the long term. I think that there is a world in which regulation is actually quite harmful to open source.

Or maybe not. Maybe open source is actually terrible. I don’t know. I’m willing to entertain that idea too. Maybe we actually really don’t want open source. I think we just don’t know enough yet about the shape of this technology, the ergonomics of it, the economics of it. I think we don’t know: will these companies sell a commodified product, and will they be more like utilities, or will they be more like Standard Oil? It’s very hard to know.

So I think you can’t govern the technology until you have a better sense of that. You can do small things now, but you can’t take big ambitious steps yet, because we just don’t know. And if we try, my big concern is that we’ll lock ourselves into some suboptimal dynamic and actually, in a Shakespearean fashion, bring about the world that we do not want.

Will AI self-improvement be a game changer? [00:28:20]

Rob Wiblin: I think among people who are more concerned about loss of control, rogue AI, also potentially human power grabs, many of them have a picture where they think we’re going to go through a period of much faster increases and improvements in capabilities than we’re going through currently. Because we’ll hit a point at which the AI is able to do basically 100% of the AI research, almost everything that the AI company is doing now will be automated, the people will become unnecessary. And this will set in train a positive feedback loop where, as the model gets better, your research also speeds up.

That’s extremely uncertain whether that would work and how much faster your capabilities would be increasing. People are really all across the map on this. Some people think this wouldn’t really work at all. Some people think we might see 10 years or 20 years of AI progress in a week or a month. How do you think about that problem, and is it a big factor in your model of how things might play out?

Dean Ball: Well, I definitely think that automated R&D is going to be a thing. That’s definitely not science fiction. It’s in some ways already happening now in some meaningful ways. You use the current model to make the next model, and that’s been true since GPT-4 at least.

Rob Wiblin: It could get faster.

Dean Ball: It could get much faster. No, it could get much faster and it already has gotten much faster. The utility of models in the development of future models has increased dramatically in the last two years, and that could continue.

I kind of expect that it will continue, but there’s a few things that I think are rate limiters, at least in the near term. The first is that the generation of models that can do this agentic coding and maybe iterate on lots of experiments, code up lots of different ideas, iterate on lots of experiments, and manage all the weird hiccups that you’ll encounter when you’re running those experiments which are an inherent part of research or software engineering, they’ll be mostly robust to those; they mostly won’t get stuck on those. I think that’s pretty close.

What I don’t think is particularly close is a model that has a big idea about the world and is in this search space — it’s a big idea that implicates a particular part of the tree to search, but there’s still a lot of searching to do, and you need to prosecute that hypothesis over the course of weeks or months. That seems like a more distant prospect. So that’s a human role right there. Also I think in the first stages of this there’ll be human review of lots of things, especially because they’ll be burning compute.

Another factor that comes in is that the compute itself will be a rate limiter. Because now all of a sudden in the world with automated R&D… They’re already compute constrained in these companies, and now —

Rob Wiblin: Your staff are the chips.

Dean Ball: Right. And your compute uses compute. Your compute is all of a sudden a heavy user of compute. I mean, now OpenAI is building lots of energy, and they’ll build much more and so will everybody else. We’re going to get there, but that’ll still be a rate limiter because I think we’re going to be compute bound for the next at least five years, and quite possibly 10.

And then finally, I was talking to Daniel Kokotajlo about this the other day where I said to Daniel, “You seem to think that the possibility of the search space for algorithmic improvements in something that looks roughly like the current paradigm of AI is approximately infinite.” And I just kind of don’t. I think we’ve probably plucked a lot of the low-hanging fruit, and then there’s medium-hanging fruit and then there’s high-hanging fruit.

I think what you basically end up doing with automated R&D is you go from the bottom of the S-curve to the top of the S-curve of like many different micro paradigms — because it’s all S-curves, you know, it’s S-curves made of S-curves — and you just go through those tinier S-curves faster, but it’s still tiny S-curves, and you still are bottlenecked by compute and quite possibly big-picture hypotheses.

Maybe eventually that gets resolved, because we do get the systems that are the fully automated researchers. I have a pretty high degree of uncertainty about what happens at that point, I will say. But I don’t think that is especially close, and I kind of do think that the fully automated researcher will still be fundamentally bound by like, you’re in a search space that like there’s diminishing returns in all areas of scientific research.

The case for regulating at the last possible moment [00:33:05]

Rob Wiblin: Let’s talk a bit more about this question of the optimal timing of our response to any emerging risks or downsides from AGI or superintelligence. I think a useful piece of context is that you were against SB 1047 last year, the California bill that was attempting to regulate risks from frontier AI models. But this year you’re in favour of its spiritual successor, SB 53. I think it does a little bit less, but I guess the situation has also changed, and you’re now in favour of it. Can you explain why you were against the first one and why you’re in favour of SB 53 now?

Dean Ball: Yeah. So I started writing about SB 1047 in February. Basically the day after the text to that bill first came out, I began writing about it. And that was well before most people, even in the AI safety community, were that aware of it — and certainly not in the AI labs or the venture capital world.

I found myself quite starkly opposed to the first version of it. Then they sort of iteratively improved and I got closer, but still pretty far apart. And toward the very end they made a set of amendments that I thought were quite productive. You can go and look at like the first thing I wrote about, it was called “California’s effort to strangle AI.” By the end I was like, I think there’s a plausible case you can make for this, but I don’t have quite the level of confidence that other people do about the exact nature of the catastrophic risks that we’re talking about here.

The liability parts of it seemed a little premature to me in various ways. I also did not like the compute threshold — which unfortunately we still have with SB 53, despite my best efforts. But I didn’t like the fact that they were targeting models based on the training compute. 10^26 FLOP is the famous number that you hear over and over again. I still think that compute thresholds are going to age really poorly, and we’ll kick ourselves for having used them. But anyway, all of these things put together led me to be moderately opposed to it.

But I would say within about a month of those amendments, we saw o1-preview from OpenAI, and the sort of reinforcement-learning-based, sort of system 2 reflective deliberative reasoning that it imparted on o1. I used o1 a little bit and it was pretty apparent to me how quickly this was going to ramp up.

And if anything, I was maybe a little bit more bullish on the reinforcement learning approach than I should have been. I’ve kind of corrected down a little bit since then, but it was pretty clear that superhuman coding, or superhuman mathematical reasoning at the very least, were well within sight. I believe we will solve the Riemann hypothesis — like, we could do it. We don’t need any architectural innovations. I think we could just do it with combinatoric search and the beefed-up versions of the current reasoners.

So OK, that changes things significantly. There was that. There was also I would say generally reaching mutual understanding with people who were behind SB 1047, people in the AI safety world and others, and some evolution that’s independent of AI capabilities that happened.

And that all came together to lead me to be quite supportive of where SB 53 has ended up. I think it’s a good example of saner heads prevailing. You’ll notice SB 53 has not really been highly charged — even though it’s Scott Wiener, it’s California, it’s frontier catastrophic risks, it’s all these things — but it hasn’t really been a lightning rod of controversy.

Rob Wiblin: Not in the same way.

Dean Ball: Not in the same way at all, no. You don’t see lots of Twitter threads about it, or lots of campaigns on social media or whatever else. SB 1047 was like an international story. I had press from Germany and Japan reaching out to me about that. SB 53 has been much less so. It was also a national story: SB 1047 was written about in The Washington Post, The New York Times, and The Wall Street Journal, places like this.

I think it’s good. I think it’s almost, for our civics, healthier that we ended up coming together and showing the world that it’s actually possible to do something that incrementally moves the ball forward on frontier AI governance that is in fact not that controversial. I think ultimately healthier. And we didn’t lose that much, right? We know that there have not been any catastrophic incidents that SB 1047 would have avoided.

Rob Wiblin: Yeah. The thing that stood out to me in your posts about this was the release of the reasoning models made you think these risks aren’t just coming in a couple of years — that they might be here reasonably soon — so you’re more interested in actually taking action and you feel less uncertain about how things are going to look at the relevant time.

Which got me wondering: I think most people in the AI safety camp, including me, have been thinking we need to get ahead of this. We want to be passing rules today that we think might not be necessary for a couple of years. Whereas I think you have sort of the opposite intuition: that we really only want to try to fix things once we can see exactly what they look like. And there’s reasonable arguments on both sides.

I guess the argument for getting ahead of it is, if we wait till the last minute, if we wait until maybe we’re even at the point where the risk is already here, then we basically might miss the deadline. We might not be able to get the infrastructure or the rules in place in time in order to defuse the problem. And also, it would be nice to have the rules in place now, maybe a little bit early, so that we can see how they play out, see how the implementation, the rollout goes — and maybe learn and iterate year after year, a little bit more like a tech company: just ship it and then see what the reaction is. And it’s true that we’re unsure about how things are now, things are very fluid — but the situation is always changing; there’s never going to be a perfect moment where we’re going to have clarity and things are anywhere near stable. Not anytime soon, anyway.

On the other hand, the arguments for waiting would be: we can’t really solve a problem that we don’t know what it’s going to look like. The models might be very different in a year or two, and we’re not very good at actually changing rules. We tend to just put them in place and then often turn away and leave them and not adjust them in response to events. We might end up creating all kinds of things that backfire or just ultimately are unnecessary.

How do you weigh up these different considerations in favour of going a bit early versus going a bit late?

Dean Ball: I think that, to a first approximation, in general — not always, but in general — people get very mad when government is reactive as opposed to proactive. I think government, especially in a democratic society, is sort of meant to be reactive. The people are supposed to have an impulse and then the government’s supposed to react to it. It’s the fundamental posture. And I can guarantee you this is true, having been in government: the fundamental posture of the American government is reactive. It’s very hard to be proactive, and there are reasons you shouldn’t want it to be proactive.

Now, there are exceptions. A good example of an exception would be quantum computing. And the reason — one of the many reasons, but one of the really big ones — that DC talks about quantum computing way more than San Francisco is post-quantum cryptography. It’s the specific threat model that we have very good reason — basically mathematical certainty — to believe that quantum computers would be able to break all of our existing cryptography. And we know how big, how important our existing cryptography is, so we have a pretty well-defined problem space.

And what’s the solution to that? It’s post-quantum cryptography. Let’s go develop those cryptographic algorithms, let’s implement them, let’s create standards. You know, the US federal government has standards regarding post-quantum cryptography. It’s in place on your iPhone.

Rob Wiblin: Ready for the future.

Dean Ball: iMessage has post-quantum cryptography. This is an area where you do want to do that. There are areas of AI that are like that, maybe a little bit more diffuse, but the AI Action Plan identifies a lot of them. Biosecurity is one, cybersecurity is another. It’s like, yeah, we obviously need to develop way more sophisticated ways of doing biosecurity and cybersecurity — whether that’s biosurveillance, whether that’s AI-assisted cyberdefence. It’s a bigger possibility space than post-quantum cryptography.

But then you think about the governance of the systems as a whole, and a good example of a major area of uncertainty that a lot of people are talking about in AI right now is continual learning, online learning. There are some ways in which it’s very similar, but subtly different from the concept of long-term memory as well.

This is the ability to acquire new skills, right? So the AI system, an AGI like GPT-7 or whatever, is given the task of running a piece of construction equipment that it never saw in its training data. But it can get into the cab digitally of this piece of construction equipment and be like, “OK, that looks like a brake, that’s a steering wheel. I can figure out how to do this, because I have enough representations from other things that I can map onto. And through trial and error, I can quickly acquire this new skill.” This is how a human would do it.

I am very confident that GPT-5 Pro, no matter how much of a biorisk it poses, would not be able to do that very effectively without a lot of scaffolding and harnesses and blah, blah, blah. Even with new cognitive skills, just purely away from the physical world.

So the way you solve that is something like online learning or continual learning. But what does that mean? Does that mean like a really big context window, like a 50-million-token context window, and it’s just hugely computationally expensive and we just kind of brute force our way into memory? Or does it mean that we figure out some new algorithm that allows it to actually learn with something that approaches closer to human-level sample efficiency?

I don’t know. But the difference between those two things is profound. If it’s a really big context window, the governance is not that different. If, on the other hand, it’s like the model can just update its weights…

Rob Wiblin: It just learns now.

Dean Ball: It just learns. It just learns in real time from everything. That is a whole different set of risks that gets posed. And you want to think about the governance of something like that quite differently than you would think about the governance of the first thing.

And I don’t want to walk down either path with a firm assumption. It seems like there are certain things that you can do that are good in either world, but there’s also plenty of things that you would want to do differently, depending on which world you’re actually in.

Rob Wiblin: I think you wrote recently that there’s speculations or expectations you might have about the future that might influence your personal decisions, but you would want to have more confidence before they would affect your public policy recommendations.

There’s a sense in which that’s noble: that you’re not going to just take your speculation and impose it on other people through laws, through regulations — especially if they might not agree or might not be requesting you to do that basically.

There’s another sense in which to me it feels possibly irresponsible in a way. Because imagine there’s this cliche of you go to a doctor and they propose some intervention. They’re like, “We think that we should do some extra tests for this or that.” And then you ask them, “What would you do, if it was you as the patient? What if you were in exactly my shoes?” And sometimes the thing that they would do for themselves is different than the thing that they would propose to you. Usually they’re more defensive with other people, or they’re more willing to do things in order to cover their butts basically, but they themselves might do nothing.

I think that goes to show that sometimes what you actually want is the other person to use all of the information that they have in order to just try to help you make the optimal decision, rather than constraining it to what is objectively defensible.

How do you think about that tradeoff? Is there a sense in which maybe you should be using your speculation to inform your policy recommendations, because otherwise it will just be a bit embarrassing in a couple of years’ time when you were like, “Well, I almost proposed that, but I didn’t.”

Dean Ball: It’s a really good question. My general sense is that, in intellectual inquiry when you hit the paradox, that’s when you’ve struck ore. Like you found the thing. The paradox is usually in some sense weirdly the ground truth. It’s like the most important thing. This is a very important part of how I approach the world, really.

So it’s definitely true that there are things that I would personally do… Like if I were emperor of the world, I would actually do exactly all the same: I still wouldn’t do the things that I think, in some sense, I think might be necessary — because I do just have enough distrust of my own intuitions. And I think everybody should. I think probably you don’t distrust your own intuitions enough, even me.

Rob Wiblin: So is it that you think that not taking such decisive action, or not using that information does actually maximise expected value in some sense, because of the risk of you being mistaken?

Dean Ball: Yeah, exactly.

Rob Wiblin: So that’s the issue. It’s not that you think it’s maybe a more libertarian thing where you don’t want to impose your views, like force them on other people against their will?

Dean Ball: It’s kind of both. I think you could phrase it both ways. And I would agree with both things, I would say.

Rob Wiblin: But if it’s the case that it’s better not to act on those guesses about the future because of the risk of being mistaken, wouldn’t you want to not use them in your personal life as well?

Dean Ball: Well, it depends, right? For certain things… There are things, especially now — you know, I’m having a kid in a few months. So when these decisions start to affect other people, again, it changes.

I guess what I would say is: Will I bet in financial markets about this future? Yeah, I will. Because I do think my version of the future corresponds enough to various predictions you can make about where asset prices will be, that you can do things like that. That’s a much easier type of prediction to make than the type of prediction that involves emergent consequences of agents being in the world and things like this.

So it has to do with the scale of the impact and it also has to do with the level of confidence. I think the level of confidence that you need to recommend policies that affect many people is just considerably higher. It’s not 100%, but if I’m like 60 or 70% confident in this vision of the future, there are a lot of uncertainties in there that could affect that, and I expect it to change a lot. And if that’s true, I just am a little worried about what that could result in.

So yeah, I think that especially right now, especially when you’re talking about our old institutional infrastructure, I expect that the entire nature of government will change because of AGI. So I just am very cautious about why would you try to extend our institutional infrastructure which everybody admits is ageing and has trouble with responsiveness? Why would you go through all this effort to try to extend it to govern something like AGI when we have very good reason to believe that AGI is going to change the actual practice of government itself? I guess that’s what I would say. It’s sort of like you want to wait to see what institutional designs are possible and then go from there.

Although I will say that it’s not that you shouldn’t do this kind of stuff, right? I’m here talking to you on the podcast about this as part of my public intellectual work, because I think it is important to note that even I — a regulation sceptic and techno-optimist — still do have these concerns about the future, I still do take the possibility of transformative technological change quite seriously.

And I think it’s possible to do both, and I think that we shouldn’t dichotomise these things. I think right now we’re in this situation that I feel is just deeply obnoxious where we dichotomise: there’s the people who are worried about AI safety and want to regulate, and then there’s the people who are like, “No, there’s no such thing. We don’t need to do anything, everything’s going to be fine.” And it’s like, no, no: you can say that you think existing regulatory proposals are stupid or are premature for the most part, and you can also say that you know that there are going to be —

Rob Wiblin: Challenges to overcome.

Dean Ball: Profound challenges. I’ve described it in public before as akin to the writing of a new constitution in some important sense. It will be like a total new order of the ages, as they say.

And yeah, I think you can hold both in your head. It’s definitely hard, but I think you can hold both in your head. So I talk about these things in my writing from time to time. I go on forums like this, and I also engage in the very hard-nosed, more mundane practice of present-day policymaking. And I think that you should do both. But also, at a certain point, AI policy does converge to speculation about the future — and you should think through the form of basically speculative fiction, in my view, about what kinds of institutions will be possible.

One of the reasons that I left the White House was to do stuff like this. I’ve already written some short stories since I left, and I anticipate that I’ll do more stuff like that, because I think that it helps people to imagine what might be possible. And it’s so much more concrete to give people a story or a vignette about the future than it is to say, “New kinds of institutional designs will be possible.” It’s like, what does that mean? I think you have to give people examples.

AI could destroy our fragile democratic equilibria. Why not freak out? [00:52:30]

Rob Wiblin: I want to come back to what seems like the thing that I find most confusing about your overall sentiment on this. I think you come from a classical liberal background. You would like humans to remain a significant force in the world. You’re not a successionist: you don’t just want to hand over control to AIs to go do whatever they want. And I think you want a dynamic, open society. I think you’ve used the term “ordered liberty” — that’s kind of the idea.

I think you like the kind of liberalism that we have in the modern world and don’t want to see it end. But really, at the point that we have AI models that can do all of the work that people do, we don’t need people to serve in the military anymore in order to be very powerful as a country. It’s not obvious that human beings are bringing that much wisdom to the political or democratic process as well. Maybe we almost just want to defer to AI models and what they think we should vote for.

That just seems like a recipe for this order that we’ve had since the Industrial Revolution, that has had a good run, to break down. People just won’t have the power to defend their interests. One way or another, you’ll end up with companies or oligarchs or some sort of government basically in control and not willing to accept threats to its position.

That seems like the default to me. Maybe we can avoid it, but it’s really not obvious how. And I guess you don’t seem as fearful of that as I might have expected.

Dean Ball: I am fearful of that for sure. I think that the good version of the future doesn’t look like the LLM version of classical liberalism. It probably looks like a very strange fusion of many different traditions and orders, traditions and ways of thinking about society that come together, and some new ones.

I think literally the first Hyperdimensional article said, like, we must “preserve the things that matter most.” I don’t care what you call it. In fact, maybe don’t call it anything right now. Maybe just preserve the things that matter.

And for me, that’s human liberty. That’s agency. That’s things like property. I think those are eternal good bases of an open society. And how exactly will you do it? I don’t have all the answers there, but if I know that’s what I want to preserve, at least I have those principles to guide me, and I can try to navigate through the space a little bit, at least somewhat. That’s the objective, at least.

At the same time, as a student of history, more than anything else, I’m aware that history is a very cruel mistress. And it’s entirely possible that the kinds of structural changes that we have discussed are outside the scope of any individual or group of people’s control, and that there’s just sort of nothing you can do. I don’t like to think that’s true, and I don’t assume that that’s true, but you have to keep in mind that it could be. You have to be circumspect about that.

But how do you do it? I think having a clear sense of preserving human agency, having a clear sense of things like, everyone talks about agency and freedom. Everyone’s happy to do that.

In America these days, for some reason — and it’s sad — we are less interested in property. Property is the thing we care about less because we actually kind of don’t like property, because property is like income inequality and stuff like this. Like, “I don’t like that that guy’s richer than me.” And we’ve assaulted the notion of private property in many ways in this country in recent decades.

But I think, in the founding story of our republic, property was the key to citizenship, it was the key to being a real part of the American project. It was this idea that you had a vested interest in the land and that there are all these values that were attached to the property owner, and that that is the way you cultivate republican virtue, is through the ownership of property. You know, some of that stuff is anachronistic, but there is a certain sense in which I think it is true that the ownership of your own stuff, and not collectivising and not socialising, but maintaining the individual is extremely important.

So what will we care about doing? How will we preserve our individuality? Will it be through literal land? Will it be through other forms of property? I think probably yes. But I guess that’s how I approach it in a broad sense. And more specifically, it’s working on it.

Rob Wiblin: I think among classical liberals and economists — I studied economics, so this is very salient to me — there’s a certain love of the romance of things not being in control: of organic, spontaneous order, and the fact that no one really knows where things are going or necessarily could steer it one way or another. There is something beautiful about that, and there is something that is reassuring that no one is in control — because that means that they can’t seize the reins and direct humanity towards the future that they would like, to the exclusion of others.

But I guess, as you pointed out repeatedly, it’s not the case that the way that we’ve organically moved out of control has always been good. The agricultural revolution might have been bad, and perhaps it might have been good if hunter-gatherers had been able to coordinate to prevent the agricultural revolution that left them worse off for a long time. I guess maybe with the benefit of hindsight, that would have been bad, because ultimately that led to the Industrial Revolution and potentially things got better. Although we don’t know whether the next age will be good or bad.

But there’s a sense in which it’s beautiful and reassuring. There’s also a sense in which I would say it’s disturbing that we might not be able to change our path, even if the great majority of people did want to shift and go on a different trajectory than the one that we’re on.

Do you have any thoughts, or do you ever feel sympathy for the folks who are like, “Humanity should take the reins. We should coordinate. We shouldn’t allow it to be out of control, we shouldn’t allow it to be spontaneous. It should be a decision”?

Dean Ball: So much I could say here. This is at the heart of the intellectual project that I’ve been on for my whole life.

Two of my favourite philosophers, Michael Oakeshott and Friedrich Hayek, very different philosophers, but they both talk about this: we have a society with no goal. We’re on a riverine vessel with no destination, we don’t know where we’re going, there is no helmsman, et cetera.

And both of them, interestingly, end up concluding the thought with, “…and people are worried about this, and it’s bad in some ways, and it’s beautiful and it’s also kind of terrifying.” There’s this debate and they basically end their thoughts, their dissertations on this by being like, “…but all these questions don’t matter, because it just is this way. This just is the nature of the world and we can’t change it, so you have to kind of dive into that reality and accept it and embrace it.”

That being said, the interesting thing is that if everybody had that attitude — that’s my attitude, and it makes me very happy, and I feel very at peace with the world — but if everyone had that attitude, then we probably would all have died a really long time ago.

Rob Wiblin: Why is that?

Dean Ball: Because at some point, you do need someone to be like, “No, we have to organise.” And they’ll fail, but in some sense society is dependent upon a bunch of like failed Stalinists trying to do their own thing — you know, a bunch of failed little dictators, not trying to take over the world or anything, but just trying to do whatever their thing is.

In some ways, you could also say that our society is actually composed of a bunch of highly directed, highly top-down, centrally planned economies in the form of the firm. The firm is this. And if I were the CEO of a company, I definitely would not be like, “Well, everything’s just happening, and it doesn’t matter.” I think that that attitude is very particular to the statesman, the sort of small-r republican statesman.

But I have to acknowledge that there needs to be activists in the world, and there needs to be many people trying many different things that are their own type of entrepreneurship, right? In some sense though, the fundamental circularity of this is that if you believe that — if you believe that we need lots of people that are making ambitious attempts to reorder the world in some specific way, either through politics or through other forms of advocacy or through commercial entrepreneurship — well then, what that actually means…

And let’s say you are one of those types of people. I’m one of those types of people: I am trying to do a bunch of stuff in the world, and I have tonnes of executive function, and I’m a fairly agentic person. But I think if you are that kind of person — if you have, as Michael Oakeshott put it, “your own intellectual fortune to make” — then you should want the actual government to be quite restrained in the end, because you want lots of different people to try different things, right? You want that: it’s going to be that small number of people who actually succeed and who change the world and who create an enormous amount of the value in our society.

So in some sense you need those very directed and agentic people. But when you’re putting your statesman hat on, you have to be, I think, more restrained. And that’s actually the best thing ultimately for the agentic people. It’s a weird paradox, but I think it’s true.

The case AI will soon be way overregulated [01:02:51]

Rob Wiblin: You think that AI will probably end up being overregulated eventually. What’s your model of how that comes about?

Dean Ball: Well, I think it’s already regulated in many ways, and some of those are suboptimal.

We have many regulations, just as an example, that require a person to do something. So we could probably massively reduce the costs of infrastructure inspection and job site inspection and construction sites right now if we didn’t have these kinds of OSHA rules that said that a person has to do the inspection. We could have drones do them, we could have those robot dog things do them. Many things like this. Sort of more mundane.

There’s also of course the liability system. I think for many people who are used to talking about governance of digital technology, liability hasn’t really played a major role in digital technology governance — in part because in the US we have Section 230, which is a liability shield that protects website owners basically from liability exposure to things that other people posted on their website. So if I threaten you on Twitter, you can sue me, but you can’t sue Twitter, basically is the idea behind that.

There’s other things too, but liability is a profound area where I think that most of the things that have protected software developers in the past from liability-based governance will not protect AI developers. We’ve already seen this. We see that there are tort liability lawsuits against AI developers right now that have not been dismissed out of hand. It’ll take years to get to resolution of those things, but they haven’t been dismissed out of hand, and I think it’s becoming more apparent to everyone that this is a real thing.

Rob Wiblin: A big part of your model though is that you expect kind of every industry or professional group to — as AI begins to threaten their jobs in their industry and create big upheaval — basically come for AI, and do everything that they can in order to basically block the use of AI in their industry, at least inasmuch as it might cost anyone jobs. Tell us about that.

Dean Ball: First of all, there’s just the existing stuff: the existing stuff matters a lot. Then there is what I anticipate is coming. To be honest, if you had asked me two years ago… I’m surprised that the civics of this issue have been, frankly, as healthy as they are. We’ve seen some stuff that does look like people who have regulatory moats sort of drawing their moat further to protect themselves from AI.

Rob Wiblin: Screenwriters might be an example. I guess there’s been some rules against mental health delivery.

Dean Ball: Yeah. There’s the mental health rules in the States. We’ve seen Nevada and Illinois pass laws that basically say mental health services cannot be provided by a chatbot. Only humans can do it. You know, every law is a statement about reality and a normative statement about the future. I just don’t really think that’s very true. It’s pretty clear that’s just ugly interest group politics. There’s going to be other stuff like that, to be sure.

Obviously, the copyright suits are an example of this. You and I are both copyright holders. We’re not the economic beneficiaries of the copyright suits against the AI companies. The economic beneficiaries of the people saying, “You trained on our data,” and, “That’s fair use” — we talk about this in populist terms, “creators” and “artists” and things like that — but the people who benefit from those things are the owners of very large intellectual property portfolios, and the holders of the very biggest, most prestigious IP.

So Taylor Swift benefits enormously. I don’t, as the author of a Substack with more than a novel’s worth of text contributed to it at this point, all by me. I’m not the beneficiary of that, and you’re not either. So I think we should just be honest when we’re playing interest group politics here. Yes, OK, Disney’s interests are not unimportant in society. But let’s not act like this is about the guy in a West Village bar with an acoustic guitar and big dreams, right? Like, that’s not what this is about.

I only say all this just to say that, do I think that’s where we’re going to end up? I don’t know. I wouldn’t say I necessarily predict that it will be over-regulated in that way. I think there are all kinds of moats that humans are going to establish for themselves, and some of those might on the whole end up being healthy.

Rob Wiblin: I thought you might be more pessimistic about this, because I think there’s a blog post, I can’t remember when you wrote it, but one of your concerns with SB 1047 was that, sure, you might create an office or some set of quite narrowly focused rules that are just about catastrophic risk from frontier models — but you thought as soon as teachers’ jobs are threatened, or some other interest group’s jobs are threatened by AI, they will be very interested in immediately appropriating any of these rules or any of these offices, or trying to speak to any of the bureaucrats who are involved in enforcing that. And basically try to expand the scope so that something that might have initially just been about gain-of-function research and pandemics and AI instead ends up being expanded to consider like risks to children who don’t have enough teachers basically in the classroom with them. Do you still think that? Do you still have big worries about that?

Dean Ball: I do, yeah, I think so. The original idea of SB 1047 was to create a kind of regulatory body that would sit and oversee the frontier model labs. I think this is a very dangerous thing. It’s not to say that the idea of there being entities that oversee the frontier labs is potentially dangerous; it’s just that the government has so many other powers, and it has so many other interest groups going at it, that the possibility of a centralised government regulator being repurposed to do things that neither of us want — that have nothing to do with frontier AI safety, and that instead have to do with protecting human moats… We might want to protect the human moats, right? We might in fact want to do that, is my point — but we shouldn’t do that like a headless horse. We should do that deliberately.

So the teachers’ unions and the actors and the mental health therapists and all these other people are totally going to fight back, and they’re totally going to use every power that the government has. It’s going to be a really hard fight. And it’s a fight right now that I think we might well lose. The issue of federal preemption for me is much more about this type of stuff than it is about SB 53 or these sort of frontier safety bills, much more concerned about the occupational licencing and protections and stuff like this. But you know, it’s going to be a tough fight either way.

My only point is let’s not give the other side ammunition, particularly ammunition designed to deal with something very important, which is frontier safety. Let’s not distract the frontier safety board with also being deployed as a weapon by other interest groups within the government. That’s my main point.

Rob Wiblin: What do you think we can do on frontier safety? Is there anything that we could set up that wouldn’t be misdirected or co-opted by other interest groups to serve their own ends and then be distracted from the thing that we actually care about, which is the catastrophic risks from frontier models?

Dean Ball: Well, it’s hard. It’s definitely hard to do so. Before I went into government, I worked with an organisation called Fathom, which is a nonprofit that is trying to develop the idea of what they now call “independent verification organisation,” and what at the time I referred to as “private governance.”

This is very common. This sounds like some radical libertarian idea. It is in fact quite common in American life. Nuclear safety is regulated partially in this way. Insurance is a form of private governance. If you want to be really philosophical about it, families are a form of private governance.

Let’s set aside how it gets created for a moment, and let’s just imagine the end state of a private technical organisation which provides essentially macroprudential supervision. “Macroprudential” is a concept that I borrow from the regulation of financial services. And there’s a great concept in bank supervision. Bank supervision is actually quite analogous to what we might need, in fact. But you shouldn’t say that because banks are heavily regulated. But I think that bank supervision in the United States is super secretive by design. You’re not supposed to know publicly what is going on there.

Rob Wiblin: Is that because of confidentiality reasons?

Dean Ball: It’s confidentiality. There’s law enforcement reasons. There’s all sorts of different reasons. There’s things about financial health that are kind of infohazards, so you have to be very careful with how you disclose them. Again, very interesting parallels to AI.

So there’s a concept in bank supervision called an “MRA,” which is a “matter requiring attention.” This is a bank’s regulatory official who is looking at the operations of your bank and saying, “This is not against any rules, this isn’t against the law, but we do just want to tell you that we think this is the kind of thing that could spiral. So you should look at this, and let’s circle back in six months and see where we are on it.”

Again, a concept that would be very interesting to borrow in the governance of AI. So imagine that there was a private supervisory body that did this, these sorts of audits or supervisory exercises for the frontier labs. Imagine that it did it with the assistance of AI. And imagine that these bodies themselves, that there were several of them — maybe they practiced in different ways or specialised in different things; there’s some people that specialise in robots and there’s other people that specialise in AGI-type models — and that in turn those supervisory bodies, privately organised, are overseen by the public government, by the traditional government. And government kind of gives those private organisations their charter; government might occasionally audit the auditors, that sort of thing.

This seems to me like a logical level of abstraction for our traditional government to be operating at. I borrow here heavily from the concept of regulatory markets, which was developed by a professor named Gillian Hadfield at Johns Hopkins University. Jack Clark and Gillian wrote a paper together when they were both at OpenAI that was proposing regulatory markets specifically for AI. This is an idea that I think I’m about 65% to 70% happy with, but I think it’s an interesting starting point.

So these are the kinds of institutional designs that I am contemplating. And in fact, Fathom is an organisation — I was affiliated with them before I joined the White House, and after I left I reaffiliated with them — and I expect that Fathom will be crafting some legislative proposals to actually put this into action in the US, to try to put this into action. Their political prospects, I don’t know. I don’t, as a general matter, endorse legislation. I’m there to help them think about the institutional design challenges.

But this does seem to me like a very promising potential path. It occupies one point in a high-dimensional space of types of private governance, types of new institutions that could be built.

How to handle the threats without collateral damage [01:14:56]

Rob Wiblin: It seems like people have pretty different estimates of the likelihood of different AI-driven catastrophes from rogue AI, loss of control, the possibility of a human power grab either from a company or perhaps through government, the possibility of AI assisting with the creation of a pandemic, the possibility that it could be destabilising geopolitically.

People are just really all across the map on how plausible they think these different scenarios are, what probability that they would place on them. Discussing it has led to some convergence in some cases, but to a surprising extent — I guess especially on the rogue AI thing — people just continue to disagree despite talking about this issue at enormous length.

But that makes me think that I wish that there was more discussion of, if you thought this was a problem, if you agreed that there was an issue here, what would be the best policy response that would actually help with it? And also that wouldn’t be so bad from other people’s point of view — that wouldn’t be imposing such large costs, or wouldn’t be so unattractive to other people who disagree and think that the likelihood of that particular threat is quite a lot lower.

I’m going to ask you a bunch of questions here. I’m going to try to elicit from you, hypothetically, if you were really worried about X or Y or Z, what would be the best policy response that other people might be willing to accept? But do you agree this is maybe an under-considered, more positive-sum mentality?

Dean Ball: I would agree, yeah. I do think it’s under-considered. I think of course I think one thing that would be really useful here is just richer quantitative models of how such a thing might work. You know, levels of integration into the economy…

Because for me, the big thing is not so much that it’s impossible to make a rogue AI. I think that it’s pretty unlikely to just happen, truthfully; I think you’d have to intentionally do such a thing. I also think, though, that the control over physical systems and just integration into the physical world is very difficult, and will take decades to be in such a place where a rogue AI could do the kinds of things that various people imagine.

But for the sake of argument, if you’re worried about it in a sort of Yudkowskyan scenario, where you are worried that models might awake during pretraining and come and kill all the people: because he thinks that the alignment problem is this thing that will take 100 years to solve, and is also not a problem that has a solution — I sincerely doubt both of those things, to be clear — but if you believe all of those assumptions, and you believe them with the extremely high level of confidence that he and his adherents seem to have, then I think really the policy solution that works is basically the idea of a global pause. It follows naturally from those assumptions.

I don’t think those assumptions are correct, so the idea of a global pause does not at all follow naturally for me — which is a good thing, because I think that it is effectively impossible to do that.

But if it is true, if you relax some of those assumptions and you more broadly talk about the possibility of rogue AI, what would you want to do? What you’d probably want to do is: I think either it’s governments or it’s entities that are heavily regulated by governments, and it’s the kind of thing that you would need some sort of licencing system for large-scale access to compute. Which would get really complicated, because compute gets more efficient over time. So I don’t even see how you would do that.

But you would need an ability to surveil all of the compute in the world, and then to licence only certain people to be able to buy certain amounts of compute. You would essentially treat compute like it was uranium, and that training of an AI model is in some sense refining uranium. And those are the regulatory approaches you would want to take.

Again, I doubt all of that. That’s not my model of the world. But I actually do think, in that sense, the AI safety world has done a pretty good job of building up more specifically what that could look like. It is just not the world that I think we are going to be in. And I certainly don’t want to assume we’re in that world, because it carries an enormous number of costs. If you really think about how we have to restrict a person’s access to a certain number of FLOPS, certain number of computations, limited set of computation —

Rob Wiblin: It’s making the concentration of power issue much worse, as well as being just enormously invasive.

Dean Ball: Exactly. It’s just so costly. And this is the thing: assuming that rogue AI is a possibility, assuming that’s the actual threat model we really need to deal with, just carries unbelievably high consequences and changes to our way of life. It implies global governance; it implies massive restrictions on the ability of people to use computers to do what they want — which I think in the modern era is an important part of liberty — and it implies a level of government surveillance over economic activity and private life that is far higher than what we are used to.

Rob Wiblin: I’m not sure that I agree with that. The way I was going to pose the question was: What if you think that, on the trajectory that we’re currently on, we’re going to run something like a 10% risk of a rogue AI scenario? I like the 10% level partly because it’s kind of around my probability, but also because it means that it’s a significant issue — that you really do want to do something in order to bring that number down — but the tradeoffs between other risks and other things that you care about do still really matter.

If you just said it’s 99% likely, and the only thing that can possibly help us is to just pause all AI research for many years, and you’re going for like Hail Mary passes like that, then basically that’s just going to be sort of the only thing that you’re going to think about. You’re not going to think about tradeoffs and middle-ground solutions all that much.

But I would have thought if we do think it is in that 10% range, the possibility of catastrophic loss of control, I think there’s probably quite a lot of things that we could do that would bring that number down that wouldn’t necessarily require global government or massive surveillance.

• You could do a whole lot of mechanistic interpretability research.
• You could be supportive of all of the technical alignment agendas that allow you to have more reliability on setting the direction of new models soon after when they’re trained.
• You can have potentially requirements about what sort of stuff is going into the pre- or post-training data, so that they’re less likely to go off the rails if that turns out to be important.
• You probably would push for transparency requirements on the companies potentially, so that we would have a better idea if we’re getting close to any of these worrying outcomes. It doesn’t seem to me that it’s quite so all-or-nothing here.
• There’s also the AI control agenda I talked about with Buck Shlegeris earlier in the year, where you just want to be ensuring that you’re having basically AI monitors of what the AIs are doing so they can’t potentially grab compute and use it for something else.

What do you think of those sorts of ideas?

Dean Ball: Well, I think that everything you just said is beneficial under a wide variety of different circumstances.

Rob Wiblin: That’s what I’m going for.

Dean Ball: Yeah, exactly. So in that sense, there’s plenty of stuff you can do. But the issue of the rogue AI in particular is that the dynamics become much closer to nuclear weapons — where if one person builds it with malicious intent, you could kind of destroy the world. So you do need something like IAEA: you need like the international nuclear NGO. You need that kind of thing.

Rob Wiblin: That might be right. I’m not sure that it is though. Inasmuch as you have aligned models that can react to other models going off the rails, to other less powerful, less capable rogue AI models that some individual or small group might unleash, potentially you don’t have to have it be completely centralised, as long as the models that don’t have it in for humanity or don’t have an unpleasant agenda that they’re pursuing, have access to more of the compute and they’re able to basically fight against any…

I mean, we kind of see this now: as you have more dangerous weapons developed, so long as the good guys have access to them in greater number, or they have more people behind them, more compute behind them, then they’re able to usually fend that off, and we end up at kind of an acceptable equilibrium. I don’t want to say that will necessarily work, and it feels like an uncomfortable situation to be in, but it might work.

Dean Ball: It might, yeah. I guess I have a somewhat different model of what a rogue AI would be able to do.

Rob Wiblin: I suppose the worry might be that it unleashes a new pandemic that it creates and then it’s like, how do you exactly defend against that, even if you have more compute?

Dean Ball: Exactly, yeah. That’s the thing. And again, there the answer might be like, we just have biosurveillance and we have nanomachines inside of our body that are producing vaccines.

Rob Wiblin: I interviewed Andrew Snyder-Beattie recently. This episode will be out by the time this one comes out. He and his team have a plan where they think they probably can defend against even basically bioweapons using more standard technology without having to have any of this massive surveillance. It’s just using the fact that even an amazing virus can’t get through a physical barrier of a mask or a wall and things like that. So yeah, people have long thought, I think, that bio is just so offence dominant, there’s nothing we can do. But I think that’s not totally clear.

Dean Ball: It’s not totally clear. The world you’re describing is much closer to the one that I am imagining, where we do want to have some sort of supervisory function over the frontier labs, and that’s like the thing you can actually govern, right?

And maybe we do that through transparency, maybe we do it through the creation of some sort of supervisory entity. I think you can have arguments about that, but we want to have some sort of ability to look at what they’re doing on these big risks — whether they be catastrophic misuse risks or rogue AI risks — you want to look at that at the frontier and then figure out how to measure and mitigate those things at the frontier, and hope that those things propagate to enough of the responsible actors in the world that you basically achieve an equilibrium that means that, yes, even if there are irresponsible actors, we will be resilient to that.

That seems very possible to me. But it also seems to me like, even if you don’t particularly believe in the rogue AI hypothesis, that’s the same — because how could you not believe in the catastrophic misuse hypothesis? Of course that’s a real threat model, obviously. And if you just believe in that, you need to do many of the same things.

So again, this is why I’m ultimately optimistic, because I think that many of the prudent things to do benefit us in a wide variety of different possible futures. But it is also why I am sceptical of anything that assumes particular kinds of futures beyond these broad things. Do you see what I’m saying?

Rob Wiblin: Yeah, definitely.

Easy wins against AI misuse [01:26:54]

Rob Wiblin: What do you think are the easy wins, the things that have a big benefit-to-cost ratio on misuse?

Dean Ball: Transparency is one we’ve already done. I think that helps.

Rob Wiblin: Well, maybe expand on that, because you’ve been quite outspoken in favour of transparency requirements for companies and whistleblower protections as well. Why does that seem like such a big win to you?

Dean Ball: Transparency is important to me for a few different reasons.

First of all, it is good for the public and civil society and the government to just understand — this is for people who care in all of those things — what it is that these companies are doing to measure and mitigate these types of risks, and also what they’re finding as they proceed along.

Also though, transparency has the double effect of forcing those companies to think long and hard — because they’re going to be putting it in public, and it is a document that will affect the way that courts consider their culpability in liability contexts, for example — about what it is that they actually want to do.

Now, there’s such a thing as too much transparency. I think you have to be careful with transparency, but we’re not there, I don’t think, with AI.

Another area where I think we could actually in fact use more transparency is on the model spec side of things. So this is more for mundane harms — but again, conveniently, it happens to implicate many of the important issues in things like AI alignment, AI control.

Having access to the intended behavioural profile of the model and then also knowing to what extent does your model actually adhere to what you intend for it to do? What kind of values do you want it to have and how often does the model actually adhere to that? And what are you doing to increase that rate of adherence to make sure it actually works? All these kinds of things are very useful for both the public to know and also to make sure that the labs are actually very conscientiously considering. So I think that’s one important piece.

Another important piece, frankly, in the governance of AI I think probably will be lawsuits to some extent. Because we’ll have lawsuits that implicate some of these bigger issues and force the companies to internalise costs potentially of harms that their products cause. And also you know, hopefully in a well decided case it’ll just be the ones; obviously there might be some cases where I don’t agree with the outcome. But whatever. In principle.

But also force. You know, a big part of the way that America has societal-wide conversations is in fact in the courtroom. And there’s people who have criticised that — I’ve criticised it before — but it’s also just a general fact, and it is a way that our not just legal but our civic life evolves; it happens through the courtroom. We have a legal system called the common law that is designed to be dynamic for that reason.

So those would all be examples of mechanisms. There’s also things like interpretability research, where you do just want to understand what’s going on inside the systems. There’s all sorts of different ways to approach interpretability. To me, one of the most interesting sort of micro debates in AI is different philosophical approaches to the technical challenge of mechanistic interpretability — or of interpretability, even: “mechanistic” assumes something. But there’s that.

I think there’s also going to be things like usage monitoring. How are we, in real time, figuring out how our systems are being used? There’ll be aspects on the compute governance side, because I do expect that the hyperscalers who have all this compute infrastructure all over the world are going to be like banks. And just like banks, we will want to provide great service to the vast majority of people — and there will be a very, very small number of people that we would like to restrict access to compute, right?

Or maybe it’ll be nation-state actors that we’d like to restrict access to. So things like how do we keep Al-Qaeda from accessing large amounts of compute? That’s basically KYC: know your customer. That’s actually a concept that, it’ll be a different technical implementation, but we’ve got that concept pretty robustly developed in fields like financial services.

A lot of these things are going to be incentivised by liability, incentivised by insurance, but also to some extent incentivised by the law. And this is a combination of technical and conceptual and physical infrastructure that we will build that will enable all of this. There’ll be things like verifying identity: technical protocols by which we will establish that you are you and that I am me, and that your agent is your agent and my agent is my agent, and that agent over there is no one’s agent, and that one we need to be a little more worried about. So there’ll be technical protocols that establish this, and those will walk back to the law in various different ways.

You can imagine all of that being useful, all of that and more. I’m sure there’s things I’m neglecting here, but every single one of those things seems like it helps you in just a huge number of different worlds. And even if you just assume mundane things about the trajectory of AI — and just assume that the future involves just LLMs that are really good chatbots — even in that world this stuff is still useful. So let’s do it.

Rob Wiblin: Yeah. Something that’s been very frustrating about the degree to which AI safety has become a source of conflict, including in the tech industry where there’s been this kind of infighting between people who are more doomy and people who are more accelerationist, is that I think it has taken attention away from the fact that, regardless of your view on almost any of these risks, on any plausible view, you should be in favour of mechanistic interpretability. Wouldn’t we love to have a better understanding of how these things work, and when they can be reliable and when they can’t?

Even if you don’t think that AI is very likely to go rogue at all, surely there are some situations in which AI could deceive people or deceive users. Don’t we want to have model organisms so we can understand these different failure modes where they arise, and do the work of making them not arise in cases where we don’t want them to be present? Don’t we want to have some monitoring of what the systems are doing at the point that they’re doing so much that no human could possibly be watching over them themselves?

There’s so many different things that just seem obviously good. I think you were focused on this question of what are the obvious wins that are good on almost any view when you were writing the AI Action Plan, and ended up having a lot of support for this exact sort of technical work that just seems like it’s big benefits and not that many costs at all.

Dean Ball: Yeah. Again, at a certain point we will face a decision in the future where the question will be: more safety or more acceleration? But right now we’re so primitive on the general safety and risk management side of this. It’s also a pain point for the consumer now in many ways, especially business customers. Adversarial robustness is a huge problem if you’re trying to roll agents out in a large regulated enterprise. Alignment, steerability: these things are big problems.

I think the reason I’m sceptical of policy is that a lot of the time policy is trying to solve problems. This is a common story in emerging technology governance: that the policymaker has things they’re worried about that are actually technical problems, but they’re not technical people, so they approach the problem entirely from the perspective of regulation and policy.

And it gets you into this terrible chicken-and-egg dynamic. This is what the European Union is going through, and will be going through for another half decade. They passed a law that was 110 pages, a way of saying, “Do a good job with AI.” And then the regulated businesses go, “What’s a ‘good job’ mean?” And the European Union goes, “I don’t know. What do you think?” Then you go back and forth and back and forth. That’s what they’ve been doing now since the AI Act was passed, and they’re going to continue doing it.

That’s also in Colorado: they passed a very stupid bill called SB 205, which was modelled on the European Union AI Act, on parts of it. And they’re doing the same thing in Colorado, and they keep being like, “We have to delay this” — because ultimately you have the wrong people in the room. This is a technical problem.

One thing I also didn’t mention: it’s all the stuff you said and you kind of alluded to it too, but there’s this really interesting user interface issue where you’re going to be managing the deployment of hundreds or thousands of agents. We see primitive versions of this today with things like codecs, where I can in fact spin up like up to like 50 instances of GPT-5 and they can all be working on things together. But what will that look like in non-coding environments? There’ll be a human who’s reviewing 1,000 customer service interactions, and the human’s not going to be able to monitor every single one of those. So how are we going to prioritise that?

That’s going to be a really interesting user interface and technology problem. There’s actually a nonprofit called Transluce that makes a product called Docent that I like very much, which is kind of like a user interface for the auditing of AI systems. It’s really cool stuff. I’m very excited about that.

There’s another group that I like very much called Ink & Switch.

Rob Wiblin: I haven’t heard of them.

Dean Ball: The best way I could analogise them is, you know how the graphical user interface was invented at Xerox PARC, which was a sort of R&D lab? Xerox is sort of like Skunk Works’s R&D lab. They kind of continue that tradition of bold reimaginings of human-computer interaction, and they’ve been doing cool stuff since long before chatbots, but they are now thinking about the problem of monitoring of large numbers of agents, and it’s really interesting to see.

So I’m very excited about things like that. Almost everything I mentioned though, when you really think about it: how many of those things are going to be done by governments? Very few. They can be incentivised by governments in some ways. They can be incentivised by insurance and liability. And these are things that are in some sense creatures of the law. Liability is not regulation, it’s not having a regulator, but it is a powerful incentive, and it’s enforced by the government ultimately. And insurance companies play a role there, and private standards bodies — groups like the Frontier Model Forum. The basic pieces are all there.

I’m happy with the progress we’ve made in the last two years. It’s not as fast as capabilities progress, to be sure, but I don’t feel like we’re spinning our wheels. But I think part of the reason that things are so polarised, there’s a lot of reasons, but one of them is that there are certain flashpoints, there’s certain kinds of words that imply very doomy scenarios about the future. And “rogue AI” is one of them.

I mean, SB 1047, you know, the thing that drove everybody crazy about SB 1047 was — it’s funny, if you look at my writing, I never was super critical of this provision, but a lot of other people got very mad about it, a lot of the other opponents — SB 1047 had this shutdown, kill switch type of provision. Like you have to have the kill switch. And everybody was like, “That’s crazy. That’s such sci-fi stuff. You’re just imagining rogue AI,” blah, blah, blah.

And that’s what I mean: putting that assumption even a little bit into your law can give people like ammunition. Not even ammunition; it can cause other people to get their backs up. And there’s so many ways where you can avoid that, and still get the majority of the outcomes that you desire.

Because you are right that fundamentally, if you’re an accelerationist… Like, if I were an aeroplane accelerationist, I don’t know that I would want the FAA as it exists today, but I would definitely want planes to be very, very safe. And we should desire that. In fact, we should want AI systems to ultimately be safer than they are today.

Rob Wiblin: More reliable anyway. Yeah, reliability is part of progress.

Dean Ball: And also, to be not just safer than they are today, but also safer and more reliable and higher quality than their human counterparts. We should want there to be superhuman performance and superhuman safety in medical diagnostics or self-driving cars or surgery or whatever else, right? We should want a higher level of quality — and we shouldn’t put ourselves as the ceiling, because I think we’re easily surpassable, humans, in many fields. So let’s seek to surpass it and then incentivise that. That seems like a better world for everyone.

But it certainly does mean that you need to do safety engineering in the AI systems themselves. And that seems abundantly obvious, but for some reason, that does end up being a source of contention and argument. I used to tolerate it more. I’ve always pointed it out as a problem, but I used to be more tolerant of it than I am now. And now I just am frustrated by it, and I seek to combat it more actively than I used to.

Maybe open source can be handled gracefully [01:41:13]

Rob Wiblin: Another case where I think things got unhelpfully unconstructive pretty quickly is there’s a lot of people, including me, who are worried that open source models could potentially — in future, when they’re more capable — be used to assist with the creation of bioweapons or a pandemic. Then those people often advocated for we need to sooner or later have bans on open source AI — which infuriated a lot of people, understandably, because they say this is going to create a massive risk of concentration of power. If only a handful of companies in the government have access to the most powerful AI models, then the mass of the public is going to be essentially disempowered relative to them. That’s very concerning, also a very legitimate concern.

But I didn’t feel like there was much of an effort to try to find solutions that would address most of these two concerns simultaneously. I think there probably is a technical solution, or at least I’ve heard of a solution that seems like it would be pretty helpful here, which is: you train a model that does have all of the knowledge, that is able to do virology, but then you do a bunch of fine-tuning on it to get it basically to reject any questions about helping to produce bioweapons, and then you distil a smaller model from that.

This is a case where you basically take a bigger model, you make a somewhat dumber but much faster model from it by just putting in inputs and outputs, and basically trying to get the smaller model to replicate it. But if the first bigger model has been told not to help with the creation of bioweapons or not to help with any advanced virology, then the distilled model basically just doesn’t have any virology knowledge in it, because none of that data has been fed into it.

Now, you could say that you can open source anything, but you have to do this first process where you distil it and get it basically not to be willing to assist with terrorism, more or less. That I think does a lot to solve the bio problem hopefully, while also simultaneously not creating this massive risk of concentration of power by not allowing open source AI.

Why can’t we have more discussion of things like that? I mean, maybe that wouldn’t work, but I think it’s a pretty interesting idea. I feel like if people were spending more time thinking of stuff like that, there would be a lot less wasted conflict basically.

Dean Ball: I completely agree with you. Obviously on that specific thing, I don’t know. I feel like every six months or so I see a new unlearning paper, and I’m always like yeah, maybe, I don’t know. Does it work at scale? Does it cause weird performance tradeoffs? Does it actually work? With bio, it’s more plausible. With cyber, it’s slightly harder.

Rob Wiblin: I think on cyber we just have to basically be patching things really quickly and getting ahead of the curve on that. That’s the solution there.

Dean Ball: But it’s also more possible. It’s also more possible to do that, because it’s this digital world.

Rob Wiblin: Yeah, yeah. We can patch computers in a way that it’s difficult to patch human beings.

Dean Ball: Yeah, right. You know, working at the White House was a real visceral lesson for me in this notion. I heard Ezra Klein once describe government as a grand enterprise in risk management. Specifically, I think the risks that government is best positioned to address are these catastrophic ones. Or at least the tail risks. You can definitely believe that a catastrophic event with very low probability of happening is not going to be efficiently addressed through market solutions and insurance and liability and things like this.

We might build a lot of the technical infrastructure we need to address it through insurance and liability and markets, but the actual… Maybe there’s some additional bit of incentive you need. It’s hard to know, but you can definitely make that case. It’s been true in the past.

But very often the conversations that we would have — without saying anything specific about conversations about these types of things in government — very often the structure of the conversation would be like, “We all know there are no 100% solutions. There are a series of 95% solutions.” And you also have to triage to a certain extent. You just have to be practical. Really the question comes down to not so much how much can you mitigate, but how much can you live with — like what kind of risks are we just willing to silently tolerate, assuming we do X, Y, and Z?

I think what you’re saying for the open source models, to get to a point where we have open source development practices where you can totally — and please — compete and do your open source model and try a novel business. You know, I struggle to imagine what the business model is of training a $10 billion model and then giving the weights away for free, but maybe there is one. I’d be very open to that idea. I hope so, in some sense. But do all that.

But there’s also a set of development practices that don’t really affect your performance that much, that don’t entail that much additional cost, and that are the basic things you need to do to be a good citizen — and most people will just do that.

And if you actually are trying to make an open source model that causes a bioweapon, well, we can’t fundamentally stop you from doing that, and investing in the ability to stop you from doing that — to actually stop you, to be sure that we can stop you — that is the thing that causes massive government intrusion into private life. So we’re not going to do that. We’re just going to make it easy for most people to do the right thing.

We’re going to understand that there are going to be some people who do the wrong thing because they’re malicious actors. And we’re going to solve that, A, through relatively normal means of we spend a tonne of money in this country on intelligence collection, and we’re going to continue to do that, and we’re going to use AI to do it and we’re going to do a really good job.

And we’ll also do things like, something else I worked on in government was the Nucleic Acid Synthesis Framework, updating that from the Biden era to make it so that we can more robustly enforce provisions that require the people who would actually synthesise nucleic acids, the companies that would do that, to engage in basically again KYC screening practices, screen the sequences, and make sure that it’s not obvious that someone’s making a virus.

Is this a perfect solution? No, because you can evade KYC systems, and also because you can split up your order for the genome you want to make. If we say that we’ll screen everything above 50 nucleotides, then you can place nucleotides of length 49 in a bunch of different places, and then you can stitch them all together. Yes, OK. And then after that we’ll have biosurveillance. You know what I mean? You just kind of do this and that’s how it actually all works.

And we don’t tend to develop 100% solutions to these sorts of problems, but…

Rob Wiblin: But we’ve made it so far.

Dean Ball: We’ve made it so far. Yeah.

Would a company be sued for trillions if their AI caused a pandemic? [01:47:58]

Rob Wiblin: If an AI company makes an AI model that is used against their intention, but it’s used to make a pandemic, do you think that they would be legally liable for the damage from all of the people who were injured or killed as a result?

Dean Ball: It depends, is the answer that the lawyer would give you. To a first approximation, the answer is quite possibly yes, but it just depends a lot on specific facts.

Rob Wiblin: What sort of facts?

Dean Ball: Well, it would depend on, number one, what theory of liability ultimately obtains for AI. For example, if we have a negligence standard, the liability could be dependent upon how much care the AI developer exercised to stop that eventuality from happening, and how that care compared to their competitors, compared to the industry standards, et cetera.

If it’s strict liability, that would matter less. That would be more like, well, it doesn’t matter how much care you exercise, you just have it. Probably the extent to which the model was jailbroken could be a factor.

Other intervening things — like how did the person synthesise the pathogen; how different was the information that the AI provided from, A, what other AI systems could provide, and B, what you could get through other means — all of that would factor in, and it would be a heavily fact-dependent inquiry that would again take years and cost millions of dollars to adjudicate.

Rob Wiblin: It’s very interesting, because I think the standard thing to say is, even if the company can be sued, that’s not a sufficient disincentive for potentially enabling such a catastrophic outcome. Because the damage that a pandemic might do might be in the tens or hundreds of trillions — far more than the company can afford to pay out or is able to pay out. So we basically need to force them to take necessary precautions that they might otherwise not be motivated to take.

But nonetheless, it does sound like this could conceivably bankrupt even among the most successful tech companies if they were truly found to be liable for all of the health damage done by a pandemic that was created using their AI model.

Dean Ball: Yes.

Rob Wiblin: So it should weigh somewhat in their calculation at least, if they have the actual mental throughput to be considering possibilities like this.

Dean Ball: Yes and no. I think the empirical literature on the utility of liability here is pretty conflicted. The fundamental problem, a good example would be oil tankers and oil spills, where there’s these potential damages that massively exceed what any reasonable company could possibly internalise.

Number one: we’ve had this problem for good things, right? Part of the reason that domestic vaccine manufacturing died in this country in the ’70s is because of tort liability, because of all the adverse effects from vaccines — even though the vaccine on the whole was prosocial, I think.

Rob Wiblin: They’re hit with a negative surplus, but they don’t capture all of the upside at all.

Dean Ball: Right. Because the fundamental thing here is that the notion of consumer surplus is another way of saying that the company does not internalise all of its positive externalities.

We do have policy mechanisms for firms to internalise positive externalities. For example, we subsidise R&D through the tax system; we subsidise investment, CapEx, through the tax system. So it’s not that we never do that. But I think that the problem is in specific areas where the company is not able to internalise the positive externalities, but liability forces them to internalise the negative externalities. And especially for general purpose technologies.

The idea that this could be true for AI is something that I have written about before. I think it could very well be true. I view liability as a good governance mechanism for this kind of transitional period that we’re in where there’s so much uncertainty. We also really have no idea how to govern it. And liability is a good “brace for impact” style of technology. I’m glad it’s there to structure the incentives of decision makers at these companies.

At the same time, it is just generally the case that for liability exposure that exceeds your balance sheet, as a firm owner you’re kind of disincentivised from doing anything about it. Because you’re screwed either way, so there’s kind of no point in taking reasonable care, or any care. Or not any care, but you’re going to take care to avoid the things where you’re still a solvent entity after the litigation.

Rob Wiblin: And then after that point, it doesn’t matter anymore.

Dean Ball: After that point you’re not going to. There is some empirical literature that supports this notion. So this is the canonical argument for regulation. Now, obviously we regulate lots of things where this is not the case, but it is the canonical argument that they will not do that, so you actually do need more positivist regulation.

Rob Wiblin: So what do you think we should do about the possibility of biological misuse?

Dean Ball: I think what we need to do is have a layered approach, where we incentivise through liabilities a wide variety of safeguards on the models themselves. Usage monitoring, interpretability, steerability, lots of rigorous work about how the model should behave in various contexts. Mechanisms to report things up for human oversight, and mechanisms for the labs to make very rapid contact with law enforcement. That’s on the software side.

Then you have a similar type of thing on the nucleic acid synthesis side, or whatever your synthesis step is, where these machines or these companies need to be screening the sequences for potential pathogenicity and toxicity. Doing so is complicated. The technology for how to do that is itself in a state of flux, and there’s a lot of questions — but we can do it very imperfectly now, and plausibly much better in the near future.

And then also doing things like KYC, where it’s not that you can’t make a virus — because you might be a legitimate virologist, you might be studying agricultural epidemics, who knows? So we need to know who you are and we need to look at your usage patterns.

In some sense, the job there on the software side, it’s a little bit like Stripe and fraud detection: Stripe doesn’t have one scalar value by which you’re a fraud or not; they’re looking at the nature of the expense, they’re looking at you, they’re looking at where you are physically in the world, they’re looking at your spending history. They’re looking at a million different things and analysing that, using high-dimensional statistics to come to a conclusion, a final computation about whether or not this transaction is likely to be fraudulent.

And the answer is not necessarily yes or no. The answer might be maybe, at which point Stripe might dynamically insert more friction into the process of effecting the transaction. And if you get through that friction OK — maybe it’s an additional verification step of some sort, maybe it’s a contact to the vendor, something like that; there’s all sorts of different things you can do — then OK, the transaction goes through.

We’ll need similar things on both the AI side and on the gene synthesis and other synthesis side.

Then finally, you assume all those things fail, and you get into biosurveillance. You get into sequencing lots and lots of things all the time, sort of ambient biosurveillance. You get into things like far-UVC, indoor air quality inside of buildings, better ventilation. Maybe one day we pass an indoor clean air act, that kind of thing. You have this, and that finally ties back to — hopefully, at some point in the future when our politics about these things are a little less divided — rapid capacity to manufacture tailored vaccines for the thing and rapid ability to distribute them, really robust biomanufacturing capability.

So that’s like a soup to nuts type of solution. No one of those things is perfect. The vaccines will not be perfect and the biosurveillance will not be perfect. None of them will be perfect. Eventually something might get through. But the idea is you have a layered approach there. I worked on some small aspects of that when I was in government, but there’s certainly much more work to be done.

Dean dislikes compute thresholds. Here’s what he’d do instead. [01:57:16]

Rob Wiblin: You’ve been quite against compute thresholds as a way of trying to identify the models that are frontier models that pose catastrophic risks, trying to separate those from everything else. And I think for good reason: it’s a very imperfect measure. Even the advocates of it would concede that.

What ideas do you have for doing better other than just giving up and throwing up our hands and saying we can’t manage it?

Dean Ball: Yeah, this is a very interesting and sort of fun legal design problem. I ultimately came to the conclusion post SB 1047 that I should solve this problem.

So I wrote a paper with a Yale law professor named Ketan Ramakrishnan. I was like, we’re not actually proposing any policies here; this is totally just a repo commit, right? Like there’s this one little function that we want to refine, and it’s a general purpose function that’s going to be used throughout this codebase. But we just want to make one little optimisation.

So what we came up with was this notion of entity-based regulation, where instead of targeting regulation based on the size of the model, you target it based on the characteristics of the entities that develop frontier models.

And what characteristics might those be? Well, different things probably for different risk profiles. For example, if what you’re doing is a consumer protection style of bill, you might want to focus on things like the number of people who use the product. On the other hand, if it’s more of a catastrophic risk style of bill, then I would focus it on something like research and development expense. Research and development is a really good expense, because it’s a tax-deductible line on the income statement for these companies.

Rob Wiblin: So if anything, people are inclined to overstate it rather than understate it.

Dean Ball: Yes, exactly. They’re also definitely inclined to track it. So you can’t say like, “We don’t track R&D. It’s hard, it’s a regulatory burden.” Yes you do. You absolutely do. Every big company in America does.

For example, if you were to define “AI development” in some sort of rigorous way, maybe even basically saying AGI development, and then you say that companies who spent more than a billion dollars developing AGI in the last 12 or 24 months or whatever, you are the companies to which this applies, and this applies to everything that you do.

The reasons for this are not just convenience. I think it does happen to be more convenient because entity-based governance is as far back as corporate regulation goes. It used to be that the corporation was given a charter by the legislature, and that’s the only way you could start a corporation, is if the legislature approved it. So we regulated the corporate entity. That’s how it works in banks, insurance companies, many other things besides.

But it also gets at something more fundamental about the enterprise of frontier AI development as a business. Earlier in this conversation we alluded to this idea of supervision and monitoring, and macroprudential types of supervision, where it’s not so much a product that we’re regulating.

In pharmaceuticals, we regulate products. The pharmaceutical company does the R&D to develop a drug, and it’s one drug and it’s the same for everybody. This is also changing, but historically. And then we basically do RCTs, randomised control trials, to measure the average impact of this on the population. Then we use those averages to make conclusions about the safety profile of the drug and then we approve it or don’t approve it based on those characteristics. This is typical thinking of the industrial era nation-state.

I think for AI though, it’s much more like financial services — where yes, the bank has products, whatever, but the products are kind of changing all the time. They’re very dynamic, and they might differ quite considerably, but they’re also enabling everything throughout the economy. So instead what we want to do is look at the overall business practices of that entity, and also the technical things that they’re doing — but also sort of squishier things, and make macroprudential judgements about whether or not that is the right stuff to be doing.

And we do that. I think that is much more productive; I think that’s much closer to the task of AI governance than the pharmaceutical model, the sort of FDA-style model. I would also note that the FDA-style model probably has to go, has to be thrown away anyway. Different podcast, but that’s kind of how I think about it.

So it’s funny: I set out to design a more convenient regulatory threshold, and it ended up being that I felt like I gathered some insights about the overall nature of the task to be done.

Could AI advances lead to violent conflict between the US and China? [02:02:52]

Rob Wiblin: How worried are you that AI advances could, one way or another, lead to direct conflict between the US and China? Like violent conflict between the US and China?

Dean Ball: I’m very concerned about this outcome. Truthfully, I think there is escalatory rhetoric that sometimes comes out of American figures.

Rob Wiblin: People talk about regime change sometimes.

Dean Ball: Yeah. I don’t think that you should be saying that winning the AI race means that we secure permanent American hegemony in the world. I don’t think that that’s true necessarily, and I don’t think that it is helpful. And even outside of China, it makes a lot of other countries distrust us quite a bit. I think that is potentially dangerous, particularly if we do have a really rapid capabilities takeoff.

Thank god right now the PRC, where I sit — and I’m not here sharing any sensitive information from my time in government; this is just my position as an analyst — the PRC doesn’t strike me as being that AGI-pilled. But if they get AGI-pilled… Especially, you know, the later you are to a thing, the higher a cost you have to pay. And if they get AGI-pilled late in the game and —

Rob Wiblin: And they feel like they’re quite behind.

Dean Ball: Yeah, yeah. Dangerous outcomes are very possible. One thing that I think would be good here, and that I think is more possible than people realise, and that is actually particularly possible in this administration…

The thing about this administration is that the president is not as much of a China hawk as he is sometimes perceived to be by people that are not enmeshed in Republican politics. President Trump gets elected in 2016 for the first time, and he has a big China agenda — though even during his first term, lots of gradation and variation and sort of swings back and forth. He was trying to make a deal with them.

Then the Biden administration comes in, and it seems like the lesson that the Democrats chose to internalise from the first Trump administration was that there is now a bipartisan pivot to China, China, China — China hawkery. I always thought that was the wrong lesson. Maybe they’re right on the substance, but I always thought, as a political matter, that was the wrong lesson to internalise from the Trump administration.

And then President Trump comes back into office, and lo and behold, he’s actually in some ways considerably more dovish toward China. Not in every way. We do have like 55% tariffs against the country. It’s a remarkable fact that we’ve internalised that as well as we have so far. But he is not interested in “beating them” per se. My view of the president is that he wants to make a deal and a framework that establishes peaceable, productive relations between our countries for a long time to come.

Rob Wiblin: Yeah, we have a good interview on this theme with Hugh White from earlier in the year. I think this is something where Trump’s worldview is consistent with a “sphere of influence” kind of mentality — where you don’t want overt conflict, you do want to deal, you’re willing to concede some influence to basically another very powerful entity. It doesn’t have to be maximalist or ideological.

Dean Ball: And think about the Monroe Doctrine too. The Americas. I think the Western Hemisphere plays a bigger role in Trump’s imagination than it does for the sort of modal American president of recent vintage. The idea of like, we need to project lots of power in South America and stuff.

Rob Wiblin: So what sort of deal would you like to see the US trying to strike with China?

Dean Ball: This is all to say that I think that it’s actually more possible than people realise for this administration in particular to be a unique opportunity: we might actually have the most dovish president on China that we’re going to have in quite some time. He’s certainly more willing to make a deal with China than President Biden was, and he might actually be more willing to.

So I don’t know, the shape of the deal is not entirely clear to me, but I think at least as a starting point, establishing some sort of mechanism by which the two countries might have government-to-government exchanges about capabilities, about evaluations, about basic things — just basic dialogue going on. I think this is maybe a tiny bit outside the Overton window right now, but I’m not sure it’s as outside the Overton window as some think. And I do wonder sometimes if there’s a prospect for positive outcomes there.

Rob Wiblin: I think the moderate path forward that does seem plausible to me is the US and China compete intensely on AI applications, on mundane uses of AI, possibly even use of AI tools and assistance in the military — but that there is some sort of agreement to be cautious around superintelligence specifically, to try to carve out what are the things that we feel nervous that we might have to train or deploy prematurely, that we don’t feel like we yet have the technical tools to have a full grasp on that, that we don’t feel fully comfortable about.

How can we have some sort of mutual assurance treaty that you’re not going to do this prematurely, forcing us to feel like we have to do it prematurely as well? That just seems in both sides’ interests, at least if they think that there is an issue to be dealt with here. Would you agree?

Dean Ball: I agree with that, yeah. I mean, I’m an old school classical liberal in many ways. So I also think the AI race that China understands themselves to be in is meaningfully different from the AI race that we understand ourselves to be in as Americans.

Rob Wiblin: In what way?

Dean Ball: So America, I think the obsession with AGI is a uniquely sort of Anglo phenomenon. Not to say that only Anglos contribute productively to the development of AGI. Certainly not. But the philosophical obsession with “we’re going to make the general intelligence” is something that I think very much comes out of Western, and probably more specifically an Anglo type of cultural environment.

And it also plays to our strengths, because what does AGI require? Big cloud computing. We have the best cloud computing companies in the world.

It requires financial engineering. We’re great at that. Look at what’s going on. Look at all this crazy stuff. Sam Altman’s talking about insane financial arrangements to add a gigawatt of AI infrastructure to America every week in a few years. Who knows if that’s going to happen, but my only point is that that’s going to require some very clever financial engineering. America has a rich history of clever financial engineering. We shouldn’t look down on financial engineering. It’s often quite useful and key to technological advances. So that would be one.

Legal engineering is another. At least ambiently, it’s a big part of what we’ve been talking about today.

These are all things America is probably better at being a more thickly institutionalised society than China will be.

Whereas China is more of, like Dan Wang puts it, the engineering state versus the lawyerly society. We’re the lawyerly society, and China is the engineering state. China, they care about the language models and stuff, but they seem much more interested in playing to their strengths. So they care about embodied intelligence, self-driving cars, and a Cambrian explosion of robots and sensors and cameras that understand what’s going on around you, that they can mass manufacture at very low costs and get huge economies of scale on.

In the grand scheme of things, there are all sorts of reasons why we might not want to trust Chinese embodied intelligence devices that are very good national security reasons. I wish that China didn’t behave in such a way that it was harder for us to trust their devices, because we do just have enough evidence of there being backdoors in their hardware and them being bad actors in this regard.

I wish we didn’t. I wish that weren’t true, because they’ve kind of destroyed what could otherwise be a quite beautiful future where we’re not actually in an AI race; we’re actually engaged in quite complementary development of similar technologies, different applications: China on the hardware, US on the software and sort of conceptual infrastructure, and China making a lot of the hardware.

There’s a world where that’s actually quite complementary. Unfortunately, I don’t think that’s the world we live in. But all I mean is that maybe we can guide ourselves closer to that world than we currently are in — and also collaborate on the very serious types of tail risks. We don’t know what the future holds, we don’t know what kind of decision points might manifest themselves in the future. But it seems to me that at least having a mechanism for communication about these kinds of things is a good starting point.

Will we see a MAGA-Yudkowskyite alliance? Doomers and the Right [02:12:29]

Rob Wiblin: Something I’ve been noticing the last couple of months is it seems like there’s a real growth in fears about loss of control of AI and other ways that AI could lead us to a negative future among the MAGA crowd, basically — especially among anti-tech populists on the right more broadly.

Do you think that it’s natural, and maybe likely, that you’ll see a sort of alliance forming between the anti-tech populist MAGA folks who are concerned about the future that AI might take us to, and the Yudkowskyist people who are very worried about loss of control or other doomsday scenarios with AI?

Dean Ball: Yeah. Both my colleague Sam Hammond and I have separate Substack essays. His was very controversial because he called it “The EA case for Trump 2024,” which I think caused a lot of people to misinterpret the argument he was making. But I had a piece that was called “AI safety under Republican leadership” that I published right after the 2024 election.

They’re both basically making the same case that — I would say AI safety might actually be a somewhat broader and more technocratic field — but specifically the AI doomers are actually more at home on the political right than they are on the political left in America.

And the reason for that is, if you look at the GOP campaign platform from 2024, there’s a tiny bit of it devoted to AI and it basically says, “We’re going to ensure that AI guarantees free speech and human flourishing.” Period. That’s it. Or like the executive order that directed the Office of Science and Technology Policy to write the action plan said, “Write a plan to ensure US dominance in AI.” Like, big-picture outcomes and values implicated there.

If you look at Democratic policymaking on AI, it’s much more written by committee. And it has to do with the institutional wiring of the Democratic and Republican parties. It’s been famously observed by many, including some on the left, that the Democratic Party is sort of made up of all these interest groups. So you have to make the teachers’ union case for AI doom and the civil rights advocates’ case for AI doom and the environmental-lefts advocates’ case for AI doom.

And also, I think sort of the epistemics that Democrats tend to have… Weirdly enough, Democrats are in some ways the more small-c conservative sort of Burkean force in our politics today. They’re very attached to the old institutions, and it’s very important to Democrats very frequently —

Rob Wiblin: To defend their performance.

Dean Ball: Well, yeah, to defend their performance, to rely on their authority — so the traditional media, academia, things like this. You know, they want to see the PhD academics have said this, they verify this.

And it’s really hard to get academia… Academia is still not AGI-pilled, right? Whereas Republicans are more likely to be, you know, they’re more online at this point. I think Republicans are more natively online — because the institutions, in the views of Republicans, and I think we were right about this, the old institutions were all captured by the left. So we needed to build new institutions, and the place to build new institutions was disproportionately online. So we did that.

And now it’s like there’s this guy, he seems credible, he’s communicating in the right internet-native ways, in the form of Eliezer Yudkowsky and others from that world, and he’s making arguments that just sound directionally correct to me. And it’s like, yeah, wow, this is super dangerous. We’ve got to do something about this. And you don’t have to convince all these interest groups, because in some sense it’s actually more Democratic and less Republican than the Democratic Party. So yeah, that’s the weird twist of events about where we are in our politics.

Rob Wiblin: I think another thing is I guess the MAGA world is very fearful of authorities and kind of powerful figures, and somewhat distrustful of authority and government and power and of massive corporations. I think that’s another way in which the possibility of superintelligence wielded by a massive company that they don’t have confidence in, or a government that they don’t have confidence in, really pushes their buttons.

Dean Ball: I mean, I think that the distrust of Big Tech in particular is a relatively bipartisan phenomenon in America — probably to our detriment, if I’m being honest, but also not without good reason. So I think that it’s kind of a bipartisan current. But I think you’re right that Republicans are more willing to embrace it. And this idea of like, these ideas that seem a little bit out-there but also in some spiritual sense are very resonant for all those power dynamic concerns, I think Republicans are just more likely to glom onto them.

So it’s not so much like technocratic AI safety. I don’t really see Republicans being the logical home of SB 53-style bills, but the logical home of the Pause AI movement or the Stop AI movement, it’s quite possible to me. Just as a political analyst, I’m not trying to be normative here. A lot of the things I’m saying, I’m not trying to be normative, I’m just trying to describe. And I think that is quite possible, which will be really interesting, because obviously President Trump is a very pro-AI president.

Rob Wiblin: You went to the National Conservative Conference recently, right?

Dean Ball: I did, yes.

Rob Wiblin: I read in the press that there was a lot of anti-AI energy there. Was that reported accurately?

Dean Ball: Yes.

Rob Wiblin: Do you have any insight on where you think the right is going to trend on AI?

Dean Ball: It’s a big part of what I try to concern myself with. Trying to address the risks in a reasonable and responsible way without destroying the enterprise of AI is something that I am trying to articulate: what that agenda looks like, what the thoughtful middle path looks like for the right at this moment. I try to do that, and the Action Plan is an attempt to do that.

But it is true. It depends on the politics and it depends on the economic results that AI produces. If we get 3% economic growth but a slow labour market, even if it’s not AI’s fault, AI will totally be scapegoated for it. So that increases the odds of a populist backlash that either party could capitalise on. But the right, again, I think, it’s somewhat more logical. Who knows? Maybe not, it’s hard to say.

But yeah, I think things are trending right now… At the White House level, I think the president is very pro-AI. And then as I look at it, within the base, there’s a lot of tension here. There’s this fundamental tension that exists between the new tech right, as it’s called, and — I’m not sure that the new tech right is really a thing — but between like tech optimist people who joined the Trump coalition in 2024 and the more traditional MAGA wing of the Republican Party, there is a tension there. And I think that the dice are currently in the air as to how that tension gets resolved.

Rob Wiblin: Yeah, I think Trump seems unusually willing to change his mind. Sometimes can really turn on a dime as events and public opinion change. Could you see him becoming, if not an anti-AI president, at least not an accelerationist-AI president? If the MAGA base turns against AI, basically because of events or fears about job loss or whatever else?

Dean Ball: It’s plausible, yeah. I think the president is really good at holding together this coalition — which everyone knew before the election, and we all knew after, is a very complicated and messy coalition. It always has been. It’s not just those two sides, there’s many other parts of it. The president’s always been good at threading the needle.

In the Action Plan, I tried to thread the needle to put some policy substance on that. I think the thing the Action Plan missed was probably on the kids issue, though there are reasons that it did that. But nonetheless, I think that’s one area where you still need to do some work is on the kid safety issue. That’s something I’m thinking about right now. But yeah, I could see him turning more sour on it.

The complexity will be the economic growth. Right now, I think it’s maybe a little overstated, but it’s probably mostly true that a big chunk of our economic growth right now is coming from the AI infrastructure buildout in the US. If that continues to be true, if it’s like we’d be in a recession if not for AI, yikes. Hard political problem. So we shall see.

Rob Wiblin: Glad it’s not your problem?

Dean Ball: Well, I will say, we were talking earlier about how it’s difficult to model superintelligence. And while the president is not a superintelligence, he is a political genius. So it is very hard for me to model what he will do politically, because he is fundamentally like a step change smarter than me at the task of politics. So it’s very hard for me to model what he’ll do. But I can definitely model the problem that he could be facing. And it is a gnarly problem.

Rob Wiblin: Do you think the cultural gap between the MAGA folks and the Yudkowsky Pause/Stop AI folks will be too great for them to be able to effectively coordinate and become friendly with one another?

Dean Ball: I don’t know. Not Yudkowsky specifically, because Yudkowsky does have this quality of like, you know, people call him like a cult leader and stuff like that. And I don’t mean to do that here, but he does have a kind of old world prophet thing about him, right? There’s something positively pentateuchal about the guy, you know? He’s one of those people who speaks in that biblical tone of talking about the future, but with a certainty of like, “I’m predicting the future, but with certainty.” That sort of weird tense. He does that. And I think that actually plays quite well among the right. The right does like a prophet, and the right likes a martyr. So I think he’ll do well.

But as to whether or not those two communities will ever mesh, I do kind of doubt it. I mean, Eliezer himself is obviously actually in many ways more libertarian than I am. He’s kind of a hardcore libertarian. He’s like a Bay Area, almost hippie-ish libertarian, you know?

And I think the other important part about that will simply be that the right has an easier time bringing in a very diverse group of people, and you don’t need to genuflect to every single position. Now, actually, I think since the president took office, in some ways parts of the right are actually behaving more like the woke wing in the left, and I think that’s unhealthy and bad, and we’ll see where that goes. But in general, I think the right is somewhat more open-minded.

Rob Wiblin: It’s definitely been more eclectic in a way recently.

Dean Ball: It’s more eclectic, yeah. The right post-Trump is a very eclectic place. And really, actually throughout the history of the American right, it’s been a very eclectic place. It always has been. But certainly post-Goldwater, post-1960s it’s been a very eclectic group. Whereas again, the left is much more, there’s this Burkean, “We have to adhere to everything in the past. And all our past positions, in all the institutions, you have to genuflect to them and you have to earn their approval.”

So you look at this wing’s interaction during the Biden administration: the left — the traditional left, the centre-left, the progressive left — and the sort of AI safety community did all meet. And you would think on paper — because most of the AI safety community are actually probably Democrats, probably voted Democrat — but when they actually got in a room together they all kind of ended up hating each other in many ways.

You know, there’s a group of AI ethics people on the progressive left who really hate tech, first of all. And it violates their worldview, like there is a constitutional violation of their worldview to think that the tech companies out of Silicon Valley can do anything meaningful and powerful. To them, Silicon Valley must be full of vapid morons who haven’t read Shakespeare or haven’t read the Frankfurt School philosophers that I’ve read, so they don’t understand anything, and they’re idiots, and they just are crass morons.

So they’re like profoundly un-AGI-pilled, and they’re like un-AGI-pillable because it would require them to believe that Big Tech is actually about to do something meaningful, and that is just like impossible for them.

Whereas the right is much more like oh my god, of course these people are super powerful and they’re going to build terrible things. You know, the right is way more willing to buy into that, and they’re more willing to change their minds about it and look at the actual facts on the ground, I think.

The tactical case for focusing on present-day harms [02:26:51]

Rob Wiblin: You think that people who are concerned about AI catastrophic risks should spend more time thinking about present-day harms that the public is very concerned about. I guess child protection in particular has been really on the agenda the last month or two. Why do you think that would be a productive thing for them to focus more on?

Dean Ball: So there’s this 16-year-old boy in California named Adam Raine who killed himself recently, and he was a seemingly otherwise healthy kid. His parents opened up his phone. They were like, “Maybe he’ll have text messages where there was someone who convinced him.” He’ll be in like Discord groups or something. It turned out nothing was there.

But his chat logs with ChatGPT indicated extensive conversations, hundreds of interactions relating to suicide and suicidal ideation — up to and including ChatGPT advising him on the micro tactics of effecting his suicide: how to tie the noose, how to anchor it correctly, how to secretly procure alcohol from his parents’ liquor cabinet so that he could drink himself to the point that he was more willing to go through with the suicide.

And this case and this sort of issue is something that’s bubbled up in the last like six or eight months. We had some Character.AI lawsuits that were alleging in one case really the almost exact same kind of harm.

I wrote an article about this Adam Raine case, and I was reflecting on the idea that nowhere in the AI Act or SB 1047 or the Biden executive order on AI or the Biden AI Bill of Rights, or for that matter the Action Plan or any of these other regulatory documents, is the issue of kids’ safety mentioned.

This gets into how it’s very hard to forecast what will actually be a thing, what’s actually going to matter. And this, probably two years ago, would have sounded rather outlandish. And if I had brought that up in a group conversation, people would be like, “Yeah, maybe. But that doesn’t seem like the actual thing we should be focusing on.”

It turns out like, there’s actually been like a number of these incidents that have happened. And we’ll see how the cases resolve themselves in terms of apportioning liability. But the main point here is that no one saw this coming particularly. But fortunately, a strength of the tort liability system is that we can respond dynamically. There already exists a legal mechanism by which people harmed in ways we can’t predict may seek redress for those harms against the people who they perceive caused those harms. And then we have a system, a very sophisticated and well-developed system, for deciding did that person really cause the harms? Are these allegations actually true?

So I wouldn’t necessarily say that we should have predicted better. You could always say that. But it’s tough to do, and I don’t blame anyone for not predicting well. But I do blame people for not predicting well and then trying to make everyone else live with the consequences of their predictions being embedded into public policy — which gets to what we were talking about at the very beginning of this conversation.

But now that we do have this problem on our radar, when you think about the technical work that is implicated in addressing this problem of kids’ safety, a lot of it is not that different at a structural level between what you need for biorisk or cyber risk: what you need to do is monitor usage in real time, you need to have really clear guardrails that are articulated in something like a model spec as to what’s over the line versus what isn’t. You need to have —

Rob Wiblin: Jailbreak protections.

Dean Ball: — adversarial robustness, and you need to have monitoring to indicate, when do we need to escalate things? And that’s not just technical, that’s also business process. That’s institutional in nature, that fix: it’s a mix of both; it’s techno-institutional.

And many other things that seem like obviously very different types of evals that you’re running, to mitigate the bio and the cyber versus the kid safety — but in some sense, kid safety is actually harder than bio and cyber. The lines are harder to draw. If someone’s trying to make a bioweapon, it’s like, call the FBI — like, my god, stop them and call the FBI. If a kid’s trying to kill himself, or thinking about it, it’s like, do I call the police? Do you call the parents? What do you do exactly? It’s hard.

OpenAI said we’re going to call the parents in the future. We’re going to contact the parents. Maybe law enforcement too, I forget now. But people will make different judgements about this, and that will evolve over time.

But I guess my point is that, if anything, the kids’ safety issue is actually in some ways more difficult than bio and cyber, and probably also the kinds of structural solutions, technical solutions that you need map on quite nicely for all these different risk categories. So by getting better at one, we’re getting better at all of them, even if the substance of what you’re trying to police is very different.

Rob Wiblin: Yeah, that makes sense. I was a little bit surprised when I first read that, because I thought that you might worry that if we start blurring together catastrophic risks — loss of control, biorisk, whatever else with child protection — it’s very hard to have one legislative instrument deal with both of these things, because although there are similar technical solutions in some respects, they probably require very different regulations in another way. And also, if we just lump these things together, we probably end up regulating one much too seriously, or being much more on a hair trigger about it, and the other one possibly not enough.

Dean Ball: I guess I would say I don’t think you actually need regulation to solve all these things, as is demonstrated because OpenAI has made changes. They’re rolling out parental controls and they’re going to roll out a system to identify people’s age. They’re doing all this stuff that like the social media companies did not do in response to allegations of child harm on their platforms. They refused to do this. They still do in many ways. For years they refused.

Rob Wiblin: I think it’s been a big mistake on their part, because it really has turned so many parents against them.

Dean Ball: Yeah, it has. And a lot of bipartisan anger. But why? In the case of social media, they have a liability shield. The AI companies don’t. In social media it’s Section 230, and the AI companies don’t have that. So you are seeing a regulatory mechanism of sorts work itself out that is self-enforcing. We don’t need to do new stuff.

So I would argue we don’t need to have necessarily legislation for all these different things. We’ll probably get it for all these different things, because unfortunately people, even people that are beneficiaries of the common law system, don’t necessarily appreciate the virtues of it. And policymakers are going to want to: it’s a salient issue and they’re going to want to show leadership. So I’m sure we’ll see legislation at the state level and at the federal level, and some of it will be good, some of it will be bad.

But the broader point you’re making is also about communication, like how you talk about these things. And I do think that it’s very important when it comes to AI safety that we not lump a bunch of disparate things together. I think this was a flaw of the way the Biden administration talked about it, in particular because they lumped in things that kind of pointed to a broader political and cultural agenda.

So algorithmic bias is one disparate-impact-based bias analysis where you say, “We assume you were guilty of being racist if the demographic outcome is not equal” — which is of course insane, but was a governing principle for the Biden administration in many ways. The misinformation and disinformation, which obviously is a big lightning rod on the right, you know, inserting all of this stuff. Then it was like there’s misinformation, disinformation, bias, ethics — and also bioweapons, and also the potential end of the world are all lumped into this one category.

Rob Wiblin: It makes sense as a piece of coalitional politics, but intellectually it’s incredibly confusing.

Dean Ball: Trust me, I know. Sometimes presidential administrations ship their org chart, and that’s an example of shipping your org chart — which is to say that you’re confusing the public about something in a way that doesn’t make sense for the public, but does make sense for you as an organisation. And that’s definitely a little bit of what was going on.

But as a political matter, it ended up being bad, because it turned a lot of people against AI safety. Because there’s some stuff in there — like biorisk and cyber and stuff like this — where I think that’s actually bipartisan and most people can agree. I think kids’ safety is a good example of one of the disparate pieces.

And then there’s really politically controversial stuff on both sides. Including the AI doom — which is not even that politically controversial; it’s just a really easy strawman. And it made them vulnerable, because you could just attack the strawman of the AI doomer and tackle the entire agenda of AI safety. Then there’s ethics and bias, which were legitimately politically controversial, and I think rightfully so. And so it sort of poisoned the entire thing, even though there was some good stuff in there.

So I think we should talk about things discretely as a general matter, and try to avoid speaking in these blobby generalities about things when we can avoid it.

Rob Wiblin: It is amazing, people who are further away from this issue, the perception that they can have about what are the different groups and what’s their relationship with one another. When I’m talking with people who are further away from the AI world, basically they do think of AI ethics and AI safety or AI security virtually as the same group, as if they are just the same coalition: they all get together, all get along.

It’s closer to being the opposite of the truth than the reality. I’ve even heard people who are further away again to them, saying there’s no real difference in a way between the doomers and the accelerationists because they’re all just tech people. They’re all just people who are very interested in AI. I’ve literally had people like imagining basically a situation in which you have Yudkowsky and Sam Altman all hanging out together, like being super chummy. Because they’re all just AI people.

Dean Ball: I think they are actually in some sense closer to each other. Like famous AI ethics person Timnit Gebru, I don’t know how to pronounce her last name, but she has the test reel thing where it’s like transhumanist, effective altruist, whatever all the other ones are. I think she’s a very toxic and bad participant in our discourse in a million different ways. And whenever I’ve interacted with her, I’ve been very dissatisfied after the experience so I hope it doesn’t happen again.

But at the same time, she has a point about that specific thing, where it’s like, I think in some sense Sam Altman and Eliezer are closer to one another.

Rob Wiblin: Because they think AI is very important. And I guess they both think that the future could be much better through technology?

Dean Ball: Yeah. And because they’re willing to imagine really radical technological changes and they kind of want it to happen. That in and of itself makes you sort of weird. And the best way to find this group is there’s this conference called the Progress Conference that’s put on by the [Roots of Progress Institute]. I participated last year. It was at Lighthaven. I did a writeup of it where I talked about these exact issues, like the accelerationist/doomer divide. I promise you that those two things converge much more than a lot of other forces in our politics.

That’s why I also, in a more recent article, I talked about how what we now think of as the “AI policy community,” the kind of people that go on your podcast, even though I might disagree and be on the opposite sides of plenty of policy fights with those people, we as a whole will be a much more narrowly clustered and smaller part of the overall AI discourse in a few years than is currently the case.

Rob Wiblin: Can you elaborate on that? I think it’s a very important point.

Dean Ball: Yeah. Just as more people start to grasp that titanic things are underway, I think it will be the case that those of us who got here early, we got here early because in some sense there are a lot of values and ideas and aspirations and cultural references that we share. And there will be a bunch of people who very much do not share those values and ideas and cultural references, who are going to be coming in and being like, “What on Earth have you people been up to? Are you kidding me? This is crazy!”

Rob Wiblin: Well, the assumption of everyone will be that we’ll build superintelligence eventually and humans will be disempowered. Even among people who are against that, it’s almost taken for granted that that’s where things will go eventually. And other people do not share that assumption at all.

Dean Ball: Right, right. And I think there’s ways to combat it. But even dealing with the level of risk of that outcome that I deal with on a day-to-day basis, is it psychologically straining on me? Absolutely, it is. But I do deal with it. And I’m not like, “We have to shut this whole thing down.” That instinct never occurs to me. And I think that much more crass instincts will manifest themselves.

Of course this will mix with politics and a populist backlash. The genuine fear will mix with the politics and populism, and it will also mix with people’s economic interests. So very often, and I will not name names, but there are certain legislators who are well associated with populism, on the left and the right, whose actual policy agenda actually is mostly just about advancing the interests of certain narrow economic interest groups.

Rob Wiblin: Who are funding them?

Dean Ball: Yeah. Who are funding them and who they want to support and whatever else. So those things will all mix, and that will become just a much bigger share of the conversation, I think. And I don’t know what the role of us more technocratic and… You know, we have this sort of a Republic of Letters-style culture where we record like four-hour podcasts, and I write these sort of anti-memetic 2,500-word Substack posts every week, and long threads or whatever else.

And you know, I think that a much different version of talking about AI is going to emerge. I don’t know how that will manifest itself, probably in many different ways, but my guess would be that it will be much more toxic and divided, and probably less nuanced than it currently is.

Rob Wiblin: Yeah. Something I’ve observed over the last year is it feels like there are more and more AI policy commentators who are kind of being paid to push a line that’s just clearly motivated by some quite narrow interest — usually profit, basically. You can kind of identify those people because what they say is very rarely very interesting; no matter what happens, no matter what the conversation is, they’re always just pushing it to the same conclusion. There’s not a lot of open-mindedness.

Dean Ball: Becoming a more normal policy discourse. That’s very common in most policy.

Rob Wiblin: Yeah, yeah. I guess the fortunate thing is, because you can identify them, you can kind of cordon them off and identify the people who are really intellectually curious and are open to ideas and persuaded by arguments. So I think maybe we’ll continue to be able to have these conversations, but it might be harder for the public to discern.

Dean Ball: I hope so, I hope so. Because the one thing that we have in our favour is we have the internet. We have the ability to broadcast our ideas on the internet and have them be subject to, yes, the forces of algorithms, but nonetheless, we have that possibility. So we have a fighting chance. Whereas people like you and me would not have a fighting chance. If you tried to pitch anything I have ever written on Hyperdimensional as like an op-ed to The New York Times or something, they’d be like, “Get out of here. Of course you can’t publish.” You have to hit a different part of the Gaussian distribution to write in those outlets.

And I think it’s a strength to have a shared culture, but it’s also a pretty big weakness of those outlets. So we do have that going for us. And maybe people like that, people that are being more imaginative and honest about things, will have a bigger role to play. But I would say the modal experience of policy is quite different, and is much more like, there’s people on this side and they kind of have predictable conclusions, and there’s people on that side, and there’s a very small number of genuinely thoughtful people — but it’s very much the rare thing, you know.

Is there any way to get the US government to use AI sensibly? [02:45:05]

Rob Wiblin: I would really like to see bureaucrats and politicians making good use of AI in coming years. I feel like they’re going to need AI advice to be able to keep up with how quickly problems are coming at them. Apart from just the fact that it would allow them to do their job well in general.

What can we do to make that actually happen? I was a little bit alarmed recently to see some public opinion polling where I think an outright majority of Americans said that they didn’t want AI involved in government decision-making. I don’t know whether they meant they just don’t want AI to make the final decision, or they are nervous about AI being used in even advising on government decisions. But it’s not obvious to me that we’re going to see uptake of the technology in government nearly quickly enough.

Dean Ball: So I think it depends on what you mean by adoption. There’s automation of just mundane processes. And a lot of what a government agency does is take information from one set of paperwork, and transform it into another set of paperwork, and then relay that to someone else. And in an abstract sense, that’s very automatable.

Ironically though — like, this is changing again, this is all changing — but the models of like a year ago were not that reliable at things like that. Certain things they were, but it was very spiky and weird, and it often required lots of scaffolding. Whereas the models of today are, I think they still have reliability problems at that more boring stuff, but they’re actually pretty good at giving you legal and policy reasoning, like just thinking through problems.

You know, something that I often did in the drafting of the Action Plan is, I didn’t say, “Write the Action Plan” or, “Come up with the policies for me” — but what I might ask for is like, “Give me a comprehensive menu of every statutory lever that I have at my disposal on issue X.” And they’re really good at that. And that’s the exact thing that I think a lot of people are a little too prideful, and they think, “I’m the expert on the law” and blah, blah, blah, so they don’t want to do that as much. Maybe that also feels more like policymaking to them.

My view is that I think that there is still a very important human role in public administration, to be sure. But it can be a very real accelerant for the policymaking process in all kinds of ways. Some of that’s just going to be generational, because there will be people who are just stubborn and who are like, “No, I’m not going to let a machine do that. For whatever reason I’ve demarcated this type of thing as being part of my status. And removing that is part of my status.” But there’s other things that are not. And maybe weirdly, because of the jagged frontier concept, the models are actually worse at the things that the person deems lower status than they are that the person deems higher status. I think that’s something we’ve seen.

You are seeing adoption of AI in government, for sure. You are. It’s slow. There’s a lot of things that make it hard that are unnecessary, and there are things that make it hard that are necessary.

Rob Wiblin: Like what?

Dean Ball: Well, on the unnecessary side of things, there’s lots of data privacy types of rules that make things hard. There’s things like FOIA. This is a good example of too much transparency: the Freedom of Information Act, and every state government has a freedom of information law as well. These are laws that basically are about what is considered a record that a member of the public can request.

Rob Wiblin: People need to be able to ask questions of ChatGPT without it going into the public necessarily, the same way that they need to be able to have conversations without it all being recorded and published.

Dean Ball: Yes, exactly. But that’s not the way the law considers it. And this is one of the very significant dangers of all types of laws that deal with information. Data privacy laws are the same way. If you start categorising information in particular ways… I mean, information I think is fundamental to the universe. So in a way it’s like you’re regulating some of the most fundamental things in the world. So it’s very dangerous.

Lots of other sorts of things can make this harder, and it can also vary by agency. Procurement is also a problem. Literally even just the way you account for your spending on AI can be a problem, because say you have a fixed IT budget. But a lot of federal agencies under the Trump administration have reduced their headcount. You can argue about too much or too little or whatever, but the fact is they’ve reduced their headcount, so they have actually pools of congressionally appropriated money that could be spent on AI to replace some of that human labour. But the AI is in the IT budget, and you can’t shift between them.

So these are the kinds of things that you run into. And this is going to be true in every firm, but government is just particularly inflexible. So I think government probably will be a laggard in adoption, and I think probably, from a median voter theorem perspective, government will be a laggard not because of democratic impulse, but because it’s just really hard to adopt new technology in government.

But also I think that’s probably what the American people want: for government to be slow. So it ends up working out, like, OK fine, American people getting what they want. I don’t think the American people are all that wise in this particular case, but whatever, it’s not my decision. You know, maybe they’re smarter than me, who knows? Masses often are smarter than individuals. Or aggregates, I should say.

But anyway, I think there are lots of interesting things that government already is doing, and I think that’ll increase over time. In terms of getting advice from them, it’s interesting: young staffers I know universally do this, and it’s often very invisible for the exact FOIA reasons I mentioned.

Rob Wiblin: Maybe you just do it on your phone.

Dean Ball: Yeah, they’re just doing it on their personal accounts. They’re just dealing with the inefficiency by using it through their personal accounts.

But they’re probably doing things I wouldn’t do. Like they’re like drafting statutes, and you can pair-program statutes with language models now. You get the exact same ideas, where it’s where coding was a couple years ago. There’s often boilerplate in statutes and it’s like, yep, just write that clause. Just do that. Contracts, same thing. And do that with legal drafting.

The more imaginative stuff, you can have it do a first pass, but there’s going to be a lot of flaws. And if you just literally say, “Write a statute,” it will mess up. The code will not compile, you know? I know people who do that for executive orders and statutes and things like this in both administrations. You probably shouldn’t do that, but it can accelerate a lot of your policymaking work for sure.

Having a kid in a time of AI turmoil [02:52:38]

Rob Wiblin: This has been an incredible conversation. I wish we could keep going for a whole lot more hours. Your stamina here is very impressive, but maybe we should call it a break and do another episode another day.

Before we finish though: you’re about to have a kid, right? About to start a family.

Dean Ball: Yes.

Rob Wiblin: Did your views on AI or hopes or fears for the future affect at all your decision about whether to start a family?

Dean Ball: Yeah, it made me more bullish about it, frankly. I think we’re probably going to go through a period where it’s pretty great to have a kid. But truthfully, I will say also I have a lot of concerns about how I am going to raise my kid even with just the digital technologies of today, and how will you talk to your kid about this thinking machine that is probably in some meaningful sense smarter than me or my son. It’s very hard.

Rob Wiblin: And what will they spend their days doing in 20 or 30 or 40 years? It’s very unclear to me.

Dean Ball: I hope they spend their days learning things, inventing things. I think the human touch will always involve things like charisma and like sprezzatura and gravitas, things like this. So I hope to cultivate those things in my son, because I think those things will have perennial benefit.

Rob Wiblin: Yeah. Fingers crossed both our kids get to live long and very happy lives.

Dean Ball: Fingers crossed.

Rob Wiblin: My guest today has been Dean Ball. Thanks so much for coming on The 80,000 Hours Podcast, Dean.

Dean Ball: Thank you for having me.