#247 – Anton Leicht on how middle powers avoid losing everything in a post-AI world

Transcript

Cold open [00:00:00]

Tom Reed: It feels to me that you could pitch this to the US as, “We’re going to create fast follower open source models, such that the best open source model in the world is not Chinese. This is a good thing. We’re not trying to eat your lunch.” Why not go for that strategy?

Anton Leicht: If you think the fast follower can actually steadily stay like four months behind, you do two things to the Americans.

You put them on a clock for every defensive operation because you tell them, “Whatever crazy, dangerous stuff you come up with, in four months, everyone in the world is going to have it — every terrorist, every non-state actor, every rogue nation in the world.”

Also, you have these like $1 trillion companies that base their entire sort of revenue strategy on selling these like exclusive frontier models, and you only ever get four months to sell them because after that, we’ll just undercut them with an available open source model.

I’m not quite sure you could sell the Americans on that being a great idea.

Who’s Anton Leicht? [00:00:43]

Tom Reed: Today I’m speaking with Anton Leicht, who is a fellow at the Carnegie Endowment for International Peace and writer of the excellent policy blog Threading the Needle.

Among many different strengths, I think one of Anton’s core appeals is that he has this gift to write things that somehow sound both secret and obvious. Thanks for coming on the show, Anton.

Anton Leicht: Thank you so much for having me.

Most countries face bleak AI futures [00:01:06]

Tom Reed: Why do you think that, by default, countries that are not the US and China look like they have a pretty bleak future?

Anton Leicht: I think when we talk about AI policy, usually we talk about sort of minimising the risk while capturing all the benefits. I think the default outcome for basically most countries in the world that don’t build their own frontier AI models is the exact opposite: they capture all the risk that comes from these models going through society, and yet they minimise all the benefits they could potentially have from AI happening.

Tom Reed: OK, what can middle powers do to avoid this future?

Anton Leicht: Well, I think find some stake and find some participation in not only in this broader AI supply chain, but in what you expect the world order with advanced AI to look like. Because, you know, Anthropic didn’t promise us just geniuses in a data centre; they said they would cure cancer. So how do you do that? You actually have to test the cures you make, you actually have to build production plants to actually make the medicine, and so on. The same thing goes for robots, the same thing goes for all the military assets that you think you might want to produce with that for US–China competition.

Like, lots of middle powers, especially the Europeans and South Koreans and Japanese, are pretty good at occupying these bottlenecks. And I think if you have something like that, and if you can integrate it into the AI supply chain sufficiently effectively, then that is just a sort of long-term asset that gives you some economic contribution, and that creates some minimal mutual dependency.

There is a very realistic second-best/worst equilibrium where the middle powers are sufficiently useless allies that don’t really build up the capabilities that would feed into this broader US AI model supply chain that would give them this edge, but instead decide to focus on whatever — their own models, or shutting off their markets from American models, or just moving away from this mutually beneficial arrangement.

And I think on a very high strategic level, what the goal here should be is to avoid the second-best equilibria and just move toward the actually best broad, alliance-wide supply-chain integration: the middle powers do what they’re good at, the US does what it’s good at, and then we sort of combine that. But it’s not obvious, and I think there are a lot of factors pulling away from that and into the worst outcome.

Tom Reed: What policies do you pursue right now to not waste that window? It’s kind of easy for, I don’t know, Japan, South Korea, or the Netherlands. But what if you’re not one of those countries? What are you doing right now to make sure you’re moving towards that partnership future?

Anton Leicht: I think there are a lot of narrow policy interventions that are very country specific. You want to think about very specifically integrating parts of your industry with American development in a way that is mutually beneficial.

The American labs want close cooperations with legacy industries for a bunch of reasons — between financing, and data, and ability to very deeply integrate into broader supply chains and actually deliver on the things they said they’d build. I think these partnerships are attractive to the American labs.

Then I think there’s the much broader question of: what are the actual input assets you need as a middle power right now to enable going down that path in the future as AI models become more important? And I think the most important thing for that is just access to the frontier models: you just need sufficiently good access to sufficiently good AI to then think about how to structure your administration and your business and even your consumer market around that.

For securing that access, there are not that many options. I think the best general version we’ve come up with is compute-for-access: you build out data centres to alleviate the inference crunch and meet the demand for inference that these leading labs have, and in return, you ask them for guaranteed access to frontier models — at least at parity with the US commercial market.

Tom Reed: Has anyone done this yet? Like Norway and the UAE, do you think they’ve gotten anything in return for their domestic compute buildout?

Anton Leicht: Those are different cases that were in a position to ask for different things.

I think the UAE deals were in many ways an uphill battle for the UAE where they couldn’t ask for much. It was already quite a stretch for the UAE to even convince the administrations that negotiated this deal to even let them export the chips to the UAE as-is. That was already pretty contentious, so I think they weren’t in a position to ask for much more.

I think Stargate Norway — which is now like a Microsoft buildout instead of an actual Stargate, because OpenAI has partly gone back on some of that — had the beginnings of that. But I also think that the negotiations around that also preceded this acute need for frontier access and guaranteeing it, so I think this is not yet part of any of the deal frameworks that are around. I do expect this to be part of the deal frameworks that come up as bigger data centre buildouts in middle powers get negotiated.

But it is an open question whether that works, because the draw for the frontier labs in building these data centres anywhere else that isn’t the US is: can you get them up faster, and can you get them up at least somewhat efficiently? And then the geopolitical considerations of: maybe the labs would like to reduce political concentration in the US; maybe they would like to reduce the geopolitical risk that comes with completely having all their inference supply in just one country; maybe they’d like to reduce the domestic political risk of local political backlash against data centres, like national political backlash against the AI industry. They’d probably like to diversify out of that a little bit, but they can really only afford to diversify out of it if it doesn’t take like four years to build a data centre.

How middle powers can strike AI deals [00:06:10]

Anton Leicht: I think the benchmark here is something like 18 to 24 months, realistically, for a big data centre. So if countries can offer something like a gigawatt built in something like one to two years, I think that’s a great basis. I think if you pitch that to a lab and say, “We’d like access parity in response,” you can probably make that deal work.

Tom Reed: Are they striking these deals with the labs directly or through the US government? How do you envision this playing out? Is it George Osborne they’re calling up?

Anton Leicht: This is all somewhat hypothetical. Generally what we’ve seen is that direct deals with the US government are both somewhat volatile and somewhat susceptible to spreading out horizontally in a way that isn’t great for the host countries.

Like we’ve seen the UK–US Technology Prosperity Deal that was in theory a model for some deepened tech cooperation that also included data centre buildouts as one pillar of the deal. That didn’t work very well, all things considered, because then last December the US brought food safety standards into the negotiations, and then OpenAI sort of reneged on the Stargate aspect of it, citing copyright and energy price reasons. And so that all didn’t really work out the way they would have hoped.

And I think there’s the more general risk of, if you strike a deal with the US government, then the US can sort of go back on that deal, if you do things that the US government doesn’t like in any other domain — and that doesn’t sound that great if you’re a middle power.

I think making the deals with the labs is easier — both because the lab demand for diversifying the inference compute is more acute, and because I think the better way of political influence in some way, and of making sure that the access restrictions aren’t as hard and making sure that US government doesn’t intervene in your access parity, is incentivising the labs to themselves lobby the US government, if you have a lab that has guaranteed you access parity and the lab has put in writing: “You gave us this data centre, we are now giving your market the access to the same models that the US market gets.”

I think that’s the realistic case, right? I think government contract parity is a lot harder to negotiate for because that’s national-security-relevant. But if we’re just talking about market access, this is a perfectly reasonable ask, right? Like if Fable 5 is on the US market, it also comes on the UK market, also comes on the European market, it comes on any market that has also come up with a data centre for Anthropic, right? That’s the basic deal structure.

And then if the US government decides to come up with some access restriction, then the American firms that are involved in these deals, and that have a stake in these deals and that have invested a lot of their money into these deals, now have an incentive to tell the US government. You’d have Nvidia and Anthropic, and ideally other labs as well that have maybe even better standing with the US government, going to the government, saying, “We actually have all these contracts and all these gigawatts of inference compute online other countries. Can you guys not restrict access in this way? Can you at least give a carveout to this country that we have a data centre in, or can you make a carveout for allies only or something? Because we’d really like to not default on our contracts with them, and we’d really like to honour these agreements so we keep access to this inference.” So you’d get a much stronger advocate in DC.

Tom Reed: But does that count for that much though? Because I feel like Wired were reporting that the Fable export control was in part prompted by them giving Mythos access to SK Telecom, the South Korean company, which the admin alleged had ties to China or something. It just feels like, especially in the scenarios where things go crazy and where you most want access to the inference compute, the labs will be powerless if the US government decides that they should just pull the compute. How do you make sure that deal is not reneged on?

Anton Leicht: Well, first of all, I think the Fable thing is a red herring in this conversation. Just because there is no restricted access for Fable, right? There is just no access for Fable. So I think Fable doesn’t violate access parity in a strict sense: the UK doesn’t have any less access to Fable than the US market; no market has access to Fable, so this is perfectly compliant with access parity in some way.

I think the specific scenario that you want to avoid is the US saying, “We’re fine with you guys giving access to the US market, but not with giving access to other markets.” And I think these are largely not the scenarios where they’re super strong and super restrictive security concerns, because then why would you even give access to the US market?

This is mostly insurance against the economic divergence specifically, because what you are worried about on the economic divergence is one market has access to abundant frontier AI and another market doesn’t have that. And I think for these cases, the security concerns are never all that high. Because if you are giving access to the entire US market, to all US firms that you aren’t really doing super intense KYC [know your customer] and very high defence-contractor-style security checks for, then it can’t be that bad. So that’s not the situation where things are going crazy. That’s not the thing that commercial access parity insulates against.

If we’re talking about Glasswing-level access parity or even government access parity, I think for that, only the inference compute incentive doesn’t work out quite as much. I think for that you also need cooperation on security architectures. Which is to say, if I’m South Korea, for example, and I want to be included in something like Project Glasswing, then the firms that I propose my Project Glasswing is extended to have to pass some security standards that will probably be handed down from the US government. These security standards will be about cybersecurity architectures; they will also be about standard general security of the firms; and they will also be about China ties, inevitably. And I think that will have to be part of the cooperation.

If we think about closed-access parity or even government-level access parity, you just have to come up with some shared amount of security standard, and only firms and only governments and only agencies that clear the security standard get access parity. I think for that, the inference leverage still matters a little bit, because giving and expanding access once the security is guaranteed is not that big of an ask; these carveouts aren’t particularly rare to get, and I think it’s not particularly difficult to lobby US government to come up with narrow carveouts for allies — you just have to create some marginal incentive for anyone in the room being interested in doing that.

So I think the hedge against that is just being like, “Well, the Australians” — or the UK or the Japanese or whatever — “we know that this is a pretty secure country. We’re not that worried about the security implications of exporting to them or expanding to them.” You just need someone in the room to think, “Maybe we don’t screw over our allies in this moment.” And I think for that you just need a marginal incentive of the inference thing, for example. But most of it is about the security cooperation instead.

The $500 billion AI moonshot [00:12:17]

Tom Reed: This still feels like a really fragile position to be in. There’s all these problems where you need to build loads of compute, and obviously your citizens are going to be unhappy with that anyway because that’s going to suck for their life. And you need to depend on the US government being reliable or dependable. Why not just build your own frontier model? Is that not the best option?

Anton Leicht: I mean, why does building compute suck so much? I think it’s perfectly fine for the citizens, especially if we get the hyperscalers to pay for it, and I think there is a realistic chance we do that. A data centre isn’t all that bad for most of the citizens. It’s just like a box with a computer in it that sits somewhere in your countryscape and Amazon paid for it.

Tom Reed: OK, but you’re also going to have to relax a lot of regulations if you want to be at all competitive with time to power and the cost and all these kinds of things. This is going to be hard to sell.

Anton Leicht: Yes, maybe you do like narrow carveouts. And yes, you get some ideological groups opposed to it. But I think you don’t have to give up on your entire regulatory framework. I think there is a way to sell it, is what I’m saying.

But yes, I think that’s right. Building your own frontier model is just really extremely hard and extremely expensive. I mean, if they really wanted to go for it, I think there is a way to do it. I wrote about this recently. I did some educated guesses on how much that might cost. Realistically, for like a four-year project split between different middle powers that would work together to throw in the talent and the compute and the funding and the supply chain leverage and everything you would need to come up with an actually competitive way to build a frontier, frontier, frontier model, it was something like $500 billion over four years.

That’s a lot of money. I think if you go into a room right now, especially if you go into a room in a treasury in a middle power country and not in a digital ministry, and tell them you’d like to have $500 billion to build highly speculative enterprise, invest into American infrastructure, build huge data centres, and then try to train a model on them — and you don’t have a business case for where you’d sell that model, you don’t have a clear way to get the investment back, and this entire approach is highly vulnerable to both domestic political pressure and to US coercion to stop doing this — I think there is probably no treasury in the world right now that would tell you that they’d happily buy their share. So I just don’t quite know how you pay for it. I don’t quite know how you do it. There’s a long list of political challenges to it.

I will say if there was sufficient motivation to do it, and if there was sufficient funding to do it, $500 billion isn’t actually that much money for X of the G7 — Europe plus Canada plus the UK plus Australia plus Japan plus South Korea. If you could get the best version of the allied liberal democracy alliance setup, and you got all of them to pitch in, they could afford $500 billion for that kind of project. It’s not an absurd idea on paper.

And I think if you’re sufficiently bought into this specific idea that the frontier matters a lot, I think this is the only realistic play. Because you’re right: if the frontier matters as much as we’ve speculated it might, and if the Americans are as volatile as some people fear they might be — especially after the Fable incident — then I think any other conclusion really doesn’t work that much. They are all hedges, and they are all bets that at some point the Americans will be wrong, or at some point the commodification will kick in, or at some point the fast followers kind of stay around, or maybe the econ just works out in a way where you capture a lot of the revenue downstream.

This all might still happen. But any other sovereignty strategy that is not building a frontier model is basically a bet on one of these things. I think you can make enough hedged bets that you probably end up fine. But if you actually want to be certain that you don’t get screwed over by frontier model progress specifically, and by super-effective AGI systems taking over the world, then you do need your own AGI, and that’s the only way to do it.

Tom Reed: Why not pitch this? It feels to me that you could pitch this to the US as, “We’re going to create a fast-follower, open-source model such that the best open-source model in the world is not Chinese. This is a good thing. We’re not trying to eat your lunch.” Why not go for that strategy? Why say we’re going to deliberately go for the frontier itself?

Anton Leicht: First of all, I don’t think the Americans will be particularly happy. I mean, this in a very real way does eat the lunch of the Americans, right? Because if you think the fast follower can actually steadily stay like four months behind, you do two things to the Americans:

• You put them on a clock for every defensive operation, because you tell them that whatever crazy dangerous stuff you come up with, in four months everyone in the world is going to have it — every terrorist, every non-state actor, every rogue nation in the world. Everyone is going to get access to however dangerous this capability is.
• If you’re just actually committing to throwing resources into having a fast follower that open sources things, I think that doesn’t sound great to the Americans, not only because of the security part, but also just because of the commercial part. You have these trillion-dollar companies that base their entire revenue strategy on selling these exclusive frontier models, and you only ever get four months to sell them, because after that we’ll just undercut them with an available open-source model.

So I’m not quite sure you could sell the Americans on that being a great idea. I’m also just not sure it actually works, because what does it actually mean to build a fast follower? I think there’s basically two versions of saying this is a “fast follower”:

• One is it’s just code for we are building a worse frontier model: someone who’s just four months worse at building a frontier model, but does all of it independently. So they invest almost the same amount of money, almost the same amount of talent, and they’re just four months slower. So they never get any exclusive window where they get to capitalise on a lead; no one ever has any incentive to go for their models instead of any other models. It’s just a money drain where you just spend a lot of money to be worse at something than someone else. And there is no real niche for that. That I think that you can always do.
• Then there’s “fast follower” in the way that is actually dependent on frontier progress, where you’re following and you’re not only in parallel moving up the same gradient, but you’re following because you know what is happening at the frontier. And because you can emulate some of the things happening at the frontier, you can maybe even distil some of the things happening at the frontier. And that requires the frontier to be available to be able to fast follow it, and I’m just not so sure that’s going to remain the case.

Then I think the final question about the fast follower is: why do you think that helps in this specific scenario? I think what we’ve talked about until now — why you need the compute-for-access strategy, why you need these bottlenecks — was a specific scenario where you were very worried about this four-month gap that the Americans have to use their frontier model very well. Where you said that we also want the same exclusive access to this frontier model, and we think that if you give any other nation a reliable four-month, six-month, I think it might realistically rather be 12-month lead, then they will use that to generate a compounding advantage in R&D, in software development, in AI progress in all these domains that these models might be good at that eventually disempowers us relatively.”

If you think that’s the case, then the fast follower doesn’t actually help you: in the limit, in this scenario, you actually need the frontier model itself. And I think then you actually have to go all-in on buying the resources you need for that, and actually developing the model itself.

Tom Reed: So if the middle powers decide they want to go for building a frontier AI project themselves, what would that project actually look like?

Anton Leicht: I think what you do is you set up the biggest possible thing you can. You don’t start small, you don’t start with giving money to any existing institution. You build an actual coalition that has enough money, that has enough leverage, that has enough talent, and that has enough general geopolitical heft to actually back this up. You get all of them to put in their money right away, and they commit their money on day one of the project: they don’t get it back under any scenario; they can’t defect. You put it all in one pot.

And then what you do with that is you fund a private company-like vehicle that is structured just like a frontier lab. This is the thing that builds your model and that builds your frontier system. And then this system, this vehicle, only corresponds with the funding governments, and only takes instructions through a middle layer of people that get what’s going on. Every country that is involved in this coalition designates sort of, “This is our czar for the project” — and this is the guy who talks to the board of the project or talks to the founders of the project, and communicates what we say in political language to them in technical language, and what the project says in technical language sort of goes back to the coalition in political language, translated by the czars. I think this is the structure you have. This is what you throw all your money into.

Then this vehicle just gets maximum freedom and maximum buy-in to actually do its thing in the way that we know that works, which is the structure of a normal frontier lab: you pay a lot of very, very good engineers from other labs a lot of money so they do work on foundation model research that is powered by a large, large amount of compute. I think you don’t need to reinvent the wheel. In fact, you mustn’t reinvent the wheel in any of this. Because there is only one way we know that works how to build a frontier model, and we just exactly replicate that. We give them all the resources they need, we stay out of their hair as much as we possibly can, and we just sort of let them cook.

We buy in as much as we can. And I think the more people you get into this, the more countries you get into cooperating on this, the broader the coalition you get that actually buys into this, the better that is — just because you will need a lot of geopolitical resistance to American pressure, and you will need a lot of funding to buy in. So I think the broader you go as far as you can get the buy-in, the better that is.

Tom Reed: And do we just expect the countries involved to accept that this will just be a huge cost sink for a couple of years?

Anton Leicht: I mean, what I expect is that this does not happen. And I think this is one of the reasons. I think there are smaller-scale versions of this where basically countries have been OK with making this kind of bet. Airbus is one obvious example. I think framed and understood as, “This is just a critical strategic capability we need to have,” this would not be understood by countries as a money sink or not a money sink.

Like a warplane is also a money sink; a warplane doesn’t make its money back. It’s something you buy so you have it. And I think in some way this AGI project would be the same thing, which is tough to swallow because obviously the frontier labs in the US also make a lot of money, but they have to justify their cost to private-market investors as opposed to governments; it’s just the American capital market that is invested into the frontier labs, and soon the sort of broader US and global stock market that’s going to be invested into them — whereas the project needs to be funded by treasuries mostly, because the private-capital market doesn’t exist in the rest of the world.

So that’s a bit of a disanalogy, so you do have to make the strategic asset pitch first and foremost. But that said, once you have developed a frontier model as a strategic asset, that is also something you can probably sell. You can do just the government procurement of this frontier model — and people buying the frontier model, like governments buying the frontier model and domestic firms buying the frontier model, surely gives you some revenue back.

And then I think there are some arguments to be made why the European project, if successful, would be somewhat competitive in broader markets. I think there’s some concerns about American capriciousness that might make people less excited about buying American models.

Tom Reed: People are still buying from Mistral.

Anton Leicht: Yeah, yeah. Mistral still has a decent American market share. So I think in an international market you could think about competing in some ways. I’m pessimistic about that, because I think you can perhaps brute force your way to building a good frontier model, but building a good software product company that actually manages to sell a lot of its product has been historically almost harder for Europe to do than it has been to build big infrastructure projects. So that’s going to be really tricky. So I think to some extent you should expect this to be a money sink for some time — but so are militaries, and I think they still in some way justify themselves.

Tom Reed: We keep them around.

Anton Leicht: Yeah, yeah. I don’t think anyone ever is like, “You know, this aircraft carrier is just sitting there, and it’s not making any money. Why don’t we rent it out? Why are we so bad at productising aircraft carriers?” They’re not supposed to productise. They’re national security assets, and that’s why we pay for them. And I think you would have to pitch the coalition on a similar logic for why they should go for the project.

Tom Reed: Who do you think should be in the project?

Anton Leicht: Who I think should be in the project is the EU, the UK, Canada, Australia, New Zealand, South Korea, and Japan. And if you can get Taiwan in too, that’s even better, but I think that’s really stretching the limits of how realistic this is. The general logic here is somewhat aligned liberal democracies, like a fairly even value floor for: coalition of the willing, good talent, good compute bottlenecks that will help you in terms of leverage, just enough capital to build the actual thing.

That’s the ideal version. I think realistically you’re not going to get that, because there are strong incentives to defect from this coalition, and I think countries that still have a very good bilateral relationship with the US or that still think that their bilateral relationship with the US is their best play to secure access to models and to secure participation in this economic order in basically the way that we’ve just described — countries that have the best access to this allied-scale play, why would they instead throw in with the European Union that has time and time again failed at building the things it announced it would build?

That’s a pretty hard pitch to make to a country like the UK, to Korea, to Japan — who are currently still feeling pretty good about their bilateral [agreements], all things considered. So I think realistically you have to cut those out, and hope that they maybe join later if things deteriorate further, or maybe they join if the project seems to be going well. But I think for the time being you probably start with Canada, as much of Europe as you can get, try to get Australia and New Zealand in there as well, and then you start from there.

Would the US crush allied AI? [00:24:54]

Tom Reed: How do we keep the US from blowing this up? Why wouldn’t they just be extremely antagonised by this attempt at a project?

Anton Leicht: They would be. This is part of the reason why it’s very difficult to pull this off. I think if you were in a completely free and open and fair market, and you had abundant access to every US asset and no US retaliation to fear, this would not be as difficult.

But the thing is, A, you do risk just broader retaliation from the US on whatever the US might choose to do: tariffs on unrelated things, seizing intelligence cooperation on all kinds of issues, just hitting the security architecture of especially the Europeans hard. For example, in the context of Ukraine, in the context of intelligence sharing, there are a lot of broader things the US could do to just stop you from doing things that have very little to do with the project or with building AI itself, and they just have a lot more to do with whatever other issue the US has leveraged in.

So this horizontal escalation is the first thing to worry about. And I think there is no clever 300-IQ AI policy intervention that deals with that. What you need to deal with that is just, you just need to be generally more strategically powerful. You need generally more geopolitical leverage. This is an open problem for a lot of countries in the world. We’re not going to solve it in the specific AI policy conversation.

But then there are two specific AI-related threats that the US has against anyone trying to build a frontier model outside of the US: the first is you need US chips to build this frontier model, and the second is you probably need at least to start with using US coding agents to build this model.

I think much of the talent base of frontier labs today are already coding agents, and the best coding agents by far in the world are the US coding agents. If you actually poach a bunch of the US researchers, if you put them into a lab in Europe, if you give them all the compute they want and you tell them to start building a frontier model, but you don’t give them like Codex or Claude Code, they’re going to be pretty annoyed and they’re going to be a lot slower at building these models. And as the labs keep using their best version of their internal coding models, and the project still doesn’t have access to even basic versions of the coding agents, the gap just widens even more. And the more pronounced this gap is at the start of the project, the harder it is to ever get it off the ground.

So you do need access to some version of the coding agents, ideally, which is something that both the US and the US labs can restrict. So you have to find a way to maybe do a carveout for some coding agent use at the very start of your project, and then you bootstrap very quickly to your own model and you build your own coding agent scaffold around it. You just have that be your first step and you just hope that works well enough that you’re eventually not that far behind on coding agents.

And maybe you can get a lab to agree to that. Maybe you can get a lab to not take you very seriously, and think, “Well, the Europeans are never going to get anywhere anyway, so we might as well sell them 3,000 Codex seats for like a billion euros or something.” And then you just spend a lot of money on getting coding agent access as long as you possibly can. But it is going to be hard. You could try to build around Chinese open-source coding agents in that case, and try to bootstrap from that, but I think that makes it even more of an uphill battle.

And the second thing, that’s even worse, are the chips you need. My very general guess is something like 3 million Blackwell-class chips for building this thing. That’s a lot of chips. There aren’t even that many chips just to buy at the moment. Nvidia’s supply is pretty limited, so you would actually need to offer sufficient money, and buy out contracts, and give sufficiently good incentives to Nvidia to actually want to sell you these chips. That’s not trivial, but that’s the easy part — because Nvidia is interested in having lots of different customers.

Tom Reed: They love selling chips.

Anton Leicht: Yeah, yeah, they love selling chips, they love money. But what they specifically love is selling chips to a bunch of different people. They don’t actually want to be reliant on just one or two buyers, so I think they’d generally be excited about selling a lot of chips to this project.

I think the US would be somewhat less excited about Nvidia selling chips to this project. There is this argument that they’re selling chips to China; surely they would also be interested in selling chips to the Europeans. I think that’s sadly not really true. Not to get into the export control discussion itself, but I think this specific style of argument only applies to China. Because if you don’t export the chips to Europe, they’re not going to build their own chips in five years; they’re just not going to have any chips. So it’s a lot less geopolitically risky to not export the chips to Europe and it’s a lot more geopolitically risky to export the chips to a project that actually says, “We’re going to reach frontier parity,” right?

I think the basic motivation here is just that they just said that Fable and the cybersecurity capabilities is so scary that they don’t want it on the open market, because they’re scared it’s going to get jailbroken. So why exactly would they look at a European project that says, “We’re going to build our own Fable, and then we’re going to use it for whatever the hell we want, and you guys can’t control it anymore,” and then export the chips to them instead? I think that’s just very, very unrealistic in the long term.

So you do need some leverage to secure access to both the coding agents and the chips. I think there are ways to do that.

I think the main way to do that is to just use the semiconductor supply chain links that these countries currently have. Most obviously, the thing everyone always talks about is ASML in the Netherlands. If you can get Japan and South Korea on board as well, you get some high-bandwidth memory, you get some more semiconductor manufacturing equipment into that, you get all these things that Nvidia and the US are very reliant on to eventually build the chips they need and get the chips online they need.

There’s probably some dealing and threatening to be done around that, because there is eventually a mutually beneficial setup to be had here, right? The Europeans keep expanding, and the coalition keeps expanding its supply of these semiconductor manufacturing equipment and all these upstream inputs, continues giving them exclusively to the US, and the US turns them into chips — and then in return, the coalition gets to buy some of these chips for its own use.

I think that’s basically a pretty positive outcome. It’s less good for the US than the outcome where the coalition gives them all the manufacturing equipment, and then the US just uses all the chips it builds with that just for itself, but I think it’s a better outcome for the US than if the coalition just decides to turn off the supply, to limit the supply, or even in the limit starts threatening to export some of the supply to China.

But I think there are things that you can do as a coalition to threaten around the supply of these semiconductor manufacturing things that don’t only target the US but that also target Nvidia, so they just create incentive on the US and Nvidia’s side to keep the favourable equilibrium of exclusive access to these semiconductor assets, and then in return they give you some of the chips. And I think if you’re sufficiently coordinated to actually play hardball on some of this stuff, there is an incentive and the path of least resistance for the Americans to keep giving you the chips.

And realistically, if you look at what the American appraisal of a sort of middle-power-government-led project to build AGI would be, I don’t think a lot of people in DC or SF would take this very seriously, to the extent that they think, “We have to stop this at all costs.” I think they will realistically think this would probably go the way of all the failed catchup projects before, so they might just not care that much about stopping the chip exports early. And then by the time that the chips are online and the project is going and things are actually going well, it might be too late to claw things back — and then the leverage just needs to get you over this initial phase where you buy and get online the chips.

When to launch the AI moonshot [00:31:57]

Tom Reed: What would convince you that it’s go time? Let’s actually pursue the project now rather than settle for compute access deals or anything else?

Anton Leicht: I think there are two parts here: the first part is political willingness to do this, and the other part is just government and policymaker interest in doing this.

Right now I’m not sold that this is the right thing to do, just because I still am a big fan of the allied-scale-integration comparative advantage. It just appeals to the efficiency-maximising part of my brain very much to just have the US do what it’s good at, and then have the allies do what they’re good at, and then we put all that together. And that’s just the best possible outcome that we could come up with for an alliance-wide post-AGI economic and strategic structure. I’d just very much still like to do that, and I feel like it’s not dead yet.

I think what is needed now is for the allies to make a sufficient contribution to incentivise that, and then for the US to respond rationally to the incentives that are set by actually effective allies that are doing well at building out these capacities. So I would very much still like to try that, and I would very much still like to see whether that works. And sort of cutting off the bilateral before we’ve really even tried that would be pretty sad.

That said, things are going a lot faster than I thought. I thought we’d have another year to set up this compute-for-access stuff at least, and get some of these deals online, get some of these cooperations online. The Fable thing — even if it is a red herring — has shaken up a lot of people, and has made a lot of people more worried about this. And it has at least now very much made this idea of export controlling models, whether that’s weights or whether it’s outputs, a much more realistic prospect in the policy discussion in DC. The conversation on that is moving a lot faster than I would have hoped for setting up this mutually beneficial alliance. I think that’s one of the problems.

Then the other problem is: how much political awareness do you really have to get started on this? I think realistically, the political awareness is always going to lag what is going to be necessary by at least a few months. That’s the problem.

I’ve written this piece from the perspective of this week. I think if we started tomorrow, we’d need the $500 billion. If we started in six months even — if we started a big tour through all the capitals now, and we met with all the treasury ministers and somehow sold them to take $100 billion of sovereign debt each and put it into a project that they don’t know will work, which is pretty big ask — if we only took six months for actually authorising this budget and getting the project started, I think we’d already have to increase the price by at least $100 billion again, because it gets harder and harder.

It gets harder and harder to compete for fewer and fewer chips that are going to be more and more shipped by then; it’s going to be harder and harder to convince the labs to give you access to somewhat leading coding agents to build things as the labs pull further and further ahead; and it’s going to be harder to avoid US retaliation as US political awareness of this risk and of the strategic relevance of these systems keeps and keeps increasing.

So $500 billion is a price tag now. If we convince people to do it, we’re going to have to tell them in six months that now it costs $600 billion. Then we have to do another round of negotiations. And by the time we’ve sold them on getting $600 billion online, we’re going to need like $1 trillion. And I think there is just a very real way in which the political awareness is just lagging the necessary moves too much. That is part of why I think it’s probably just not going to happen. It’s probably not going to work.

What would convince me on the political side is people actually want to do this. And what would convince me on the strategic/tactical side is the compute-for-access, allied-scale type of thing is not actually working: the US doesn’t want to go for it, the US stops these deals from happening, the US restricts the outflow of chips even to allied data centre structures, this kind of thing.

So if our play fails and if political awareness magically fixes itself, we should go for this. Until that happens, I think I’m still fine or at least hopeful to do the alternative play instead.

Why AI dominance is forever [00:35:45]

Tom Reed: OK. One question is: why is this so catastrophic? So the US has frontier AI before the rest of the world does — but the US already outmatches most countries on most domains, and they’re doing kind of fine. Why is this so bleak for France or England or anywhere else?

Anton Leicht: Well, I think the question is: what can you do with a two- or three- or six-month lead time on all the secrets of the world? I think if you just have the cyber capabilities now, the effect is somewhat limited. But you could already see the Europeans freaking out about this, and I think rightly so. And then I think we expect models to be able to do more things well than just cyber, and I think a lot of them could matter a lot economically.

I think there’s an open question here about in which domains specifically the frontier matters that much, but I think R&D is one of the other obvious examples. If you have a few months’ lead time on innovation, and you find out what material you’re supposed to use a few months before the Euros do, and then the Euros find out as well, and they build their thing and it takes 24 months for anything to get off the production line anyway and for anything to get shipped anyway, really what do the few months matter? I think that’s a realistic near-term future where the lag is just not all that relevant for the R&D stuff, for example.

But the faster the world moves, and the more integrated these systems get, the more automated manufacturing gets, maybe there are ways to sort of lock in these innovations in a way — so that, for example, you can register patents for all the innovation that your AI finds before the Euros gets around to them, or you just get quicker and quicker iteration cycles on building things and testing based on them.

And the faster the world starts moving as we integrate these systems and as we automate more and more things, the more you would expect these initial advantages sort of compound — because then you build the better thing, then you check how well it works if you build a better thing, then you feed the data back into the system and you try what the system makes of that, and then you feed in the innovation again and again.

So you just get to start the flywheel, and you get to start the exponential of this innovation just a few months earlier. And you’re always a few months earlier, so your product is always better by a few months. And I think if the world just moves sufficiently fast for that, that does compound into a pretty substantial advantage.

Is AI dependence catastrophic? [00:37:42]

Tom Reed: But the world you paint, if most of the marginal returns on intelligence are in areas like cyber or R&D, then Europeans just get access to better and better products from American firms. Our purchasing power probably goes up if these products are also being able to be produced at cheaper levels. Again, that doesn’t sound catastrophic to me. What’s bad about that situation?

Anton Leicht: I mean, I think this is not catastrophic in the near term. Just general increasing economic growth and general increasing innovation and productivity usually starts out being pretty good for the world in an open market.

I think there are two conversations here. The first conversation is: how does that lead to adverse economic effects? And the other question is: how does that lead to bit-by-bit relative disempowerment?

I think the economic part of it is not as catastrophic, as you say. I think the relative disempowerment stuff is a problem though.

So these frontier AI systems themselves become a really important input into your economy. You would think that they pull away further and further from the AI systems you yourself build in your country — for the same reason that the R&D effects compound, that the software R&D and the AI R&D effects also compound, right? The Americans not only build better models; they also get much better at building their own AI models, because they can use their own AI to both broadly improve their broader AI supply chain — from everything from chips to data to everything — and because of the more narrow recursive self-improvement effects that might start within the labs. So between those, you just expect the American software to pull further and further away.

Tom Reed: Why are the open-source models not compounding at the same time, at the same rate?

Anton Leicht: A, you already have a growing divergence in compute. That just means that the American labs can move faster, can explore more new innovation, can spend a lot more compute on building better models.

And then B, because there are these recursive effects inside the labs, they can reinvest all the revenue they make into building more infrastructure, and then their models get even better, they can reinvest all the usage data they get into making even better models. And of course, they can just use their superior models, deploy them internally, use them to make their models better, but then prevent other competitors from using the models in the same way.

We already see this with Fable, which is limited for Frontier model research specifically — which is, I think, something that we’re going to see more broadly. Because why would you build a tool that helps everyone else make models that disrupt your own business model? I think at some point they will stop doing that. And then I think the question and the challenge is exactly the other way around: why on Earth would you expect this four-month gap in open source to remain in place forever?

I think the specific mechanism that has led to this in the past was there was a lag because you could see where the frontier was, and the frontier was sort of moving four months ahead, and you could sort of see what’s the shape of the frontier, what kind of models are they building at the frontier, how quickly is it moving, what are the capabilities they’re focusing on, what are the specific capabilities they’re throwing all their RL resources on, what benchmark shape comes out when they build a new frontier model?

And from that you can sort of guess, what are we exactly chasing here? What are we going for? What are the next architectural breakthroughs that we have to find as a fast follower? And so on. To say nothing of the fact that I think a lot of fast following is also about outright distillation, which is even harder to do.

So if you get to this point where the frontier, A, pulls away even further, and also, B, becomes a lot less visible because some of it gets restricted, then it’s just a lot harder for a fast follower to even know what they’re supposed to be doing and just find the road to the frontier. And if that’s the case, and they also have less and less compute relative to the frontier, I would just expect that gap to open up further and further.

So I do think we should expect the rest of the world not to have their own fast followers that can really keep up, so the frontier models stay a relevant input to these economies. So you have a lot of reliance from that — because you need at least the frontier or the sub-frontier models from America as just your basic economic input, but you also need all the AI-produced goods that they’re giving you, right?

To the extent that your economic mechanism is, “Well, we’ll be fine because we have something to offer to the Americans” — like, you know, the Amalfi Coast or Milanese suits or whatever you might come up with, r maybe even very sophisticated machinery that integrates the AI very well — there is still asymmetric leverage, right? The US could cut you off at much lower cost to the US than you could be able to threaten the US.

And who’s to say? You can go into pretty crazy futures from there on. I think at the limit there are just very few things that you, as a relatively disempowered country, can really do to stop this power-concentrated country from doing basically whatever it wants. Maybe land on Earth becomes very scarce, and they decide they really want more land on Earth, and then what do you really do about that? Maybe they would like your country as a military base on the way to somewhere else, and then you can’t really say anything about that. Maybe they want a natural resource you have, and then you can’t really say all that much about that either.

I think if the relative leverage becomes pronounced enough, then at some point it’s really difficult to look at these European countries and still think that they’re sovereign nations in a relevant sense. And I think that is a realistic threat if you just have a relative growth divergence. Even if the Europeans are fine on an absolute level as you describe, I think at some point of divergence the relative geopolitical leverage is just bad enough that it’s still a pretty existential threat.

What’s left to sell in an AI-dominated world? [00:42:45]

Tom Reed: Let’s go back to the world where you don’t go for the project, but you have ensured continued access to the Frontier models. In that world, how is your economy capturing value when most of the labour is done by AIs created by American AI firms?

Anton Leicht: Maybe the labour is just sort of commodified in a way that makes the labour not matter all that much. I think the AI agents are just one input into your economy in that world. Maybe it’s a high-revenue input, maybe it’s a low-revenue input; maybe it’s a high-margin input, maybe it’s a low-margin input — but there surely are going to be other things that are also important to contribute to this, right?

The AI agents do the labour, but what do they need? They probably need, at least for the start, a lot of proprietary data that you feed into them. And as they continue to do the work, they probably want to interface with production that generates more data as they try things out.

The very classic loop that you might imagine here is: you have a super modular, super automated factory that builds whatever — let’s just say it builds robots — and it’s connected to a US frontier model. And ideally you have this US frontier data centre here, you have this robotics manufacturing plant here, and they correspond and coordinate very closely. And the model comes up with, “This is how we should ideally build this component,” and then ideas of the AI are implemented in the factory. A lot of sensors and a lot of monitoring in the factory captures how that exactly happens, feeds back into the model, and tells the model, “This has worked, this hasn’t worked.” The model comes up with another way to optimise the supply chain, it feeds back into the factory, the factory corresponds to that and makes the modifications and then feeds back in.

And at the end, you have very good robots coming out of this process, and that’s just obviously a valuable economic asset. The question is: how big is the role that the AI company has played in that, and how big is the role that the manufacturing has played in that? I think it turns out it’s almost as hard to build a factory that can do this as it is to build an AI model that can do that. And I think a lot of the world is going to be bottlenecked on factories that can do this, because you need a lot of inputs for that: you need the factory to be staffed by people that are good enough at making the modifications ad hoc that the AI system is telling you to; you need a factory that is good enough on sensors and all these reporting loops that it can actually give information on how well the AI ideas are working back to the AI.

And I think this is just a very rare asset and a very important asset. And I think if you have that, it probably generates a lot of revenue for you. This is like the idealised version of industry that is highly AGI relevant, that is integrated into a lab.

But I think you can come up with a bunch of other small ways to do just the same thing. Like people that adjust the way that they provide services, the way that they advertise services based on their integration with AI apps in the same way. You can come up with things that are maybe not quite as AGI-pilled as robotics, but they are still just important to build. Maybe it’s drones, maybe it’s just random hardware devices that are used to do whatever you want to do. Basically any industrial output in the world you can integrate into AI in the same way. And it turns out these industrial assets could just turn out to be an important bottleneck.

I think the same story, in a slightly broader sense, can be told for a lot of other downstream bottlenecks that can be integrated with AI, and that maybe just captures a lot of the value.

And then there are a lot of things that just don’t interface with AI that much that also capture a lot of the value. This goes back to the Baumol stuff that we’ve talked about. I think once your economy isn’t super disrupted, once your country is somewhat stable, once your governments are sort of doing fine, you can just come up with things that are scarce as AI becomes important. You can create products that people still want. It goes back to the tourism and the music and the artisanal goods and all these things. All of these things still capture substantial value.

I will say it’s sometimes overrated the extent to which that is important. Even in countries that we think of as very tourism heavy, tourism rarely makes up more than 10% of their GDP. It turns out you just need to do all the legacy things that your economy is still doing in some ways. I think that’s going to be somewhat hard in the limit. There is just a lot of the services industry that is susceptible to being displaced by AI eventually.

And there, the question of saving some of your production and saving some of your economic activity is more about comparative advantage. Why exactly would it be worth it for AI agents to completely replace some random aspect of the services industry in Germany or in Greece or whatever, instead of investing this marginal inference budget you have, this marginal R&D you have? Why are you spending this on displacing this random German services industry instead of doing things that countries that have geniuses in data centre crazy innovation they could come up with? It’s just very unclear that that’s the thing you primarily do.

So I think a lot of this is also just a bet on the division of labour shaking out that way, and some of these activities still staying around. And in the limit, if they all get displaced and if all the service industries just get done by cheap and abundant AI, I think there is no good answer to that. But then there is also no good answer to what the US economy does all day. Yes, they probably get to tax the agents or whatever so it’s not as economically catastrophic for them. But as a matter of economic structure disruption, I think if the AIs just do all the jobs, there is no good answer to what does your economy do? Because currently economies still do hinge on doing all the jobs.

Policies to avoid mass AI-layoffs [00:47:47]

Tom Reed: What policies do you think this nets out in right now for middle powers? If what you want is to own the robot factory, does that mean Germany should be banning Bezos’s new company [Prometheus] from buying out all new German manufacturing firms?

Anton Leicht: Definitely, definitely screen for the FDI [foreign direct investment]. Because I think this is a very risky situation right now, where I think a lot of outside companies, and a lot of US-based actors specifically, are starting to become aware of how important the bottlenecks are, but the countries in the rest of the world haven’t internalised how valuable and how important these things are just yet. So there is a lot of incentive for outsiders to come in and buy up all these bottlenecks and then productively redeploy them once the AI systems have become more important.

The Bezos Prometheus thing that you’ve mentioned is I think one obvious example: they just buy all the industrial capacity and then they redeploy it when AI systems are stronger.

And obviously you shouldn’t sell these things right now. Obviously if you think about selling them now, a price that something like Bezos would be able to offer — or something that basically any investor that would be interested in running this play would be able to offer — sounds great if you’re not sufficiently aware of what is coming in AI. You get like a few billion for an asset that you thought was only worth a few hundred million in the current economy. That sounds crazy good to you. You sell those data. This is just a windfall; your family is set for life. That sounds amazing. But it turns out, alternatively, this would have been your slice of the light cone: this is a super important piece of manufacturing; this would have secured your stake in the AGI economy for decades to come, and it would have been worth so much more than a few billion.

And this awareness gap and how much these assets are worth is a very threatening phase of the AI economy transition right now. So yes, I think you just need to get ahead of that. You need to screen for foreign investments into this. You need to make sure that your important assets aren’t taken over. I think that’s the very baseline of what you should be doing as a country.

I think you can also think about how you can make these assets AI compatible, because as I said before, I think the risk here is it’s just become so hard to work, for example, with manufacturing that is AI compatible that it’s just more efficient to just rebuild this thing in the US to begin with.

And I think to make it efficient, and almost bindingly interesting to the AI labs to cooperate with your robotics factory somewhere else in the world, rather than just rebuild it in the US is to just make it AI compatible in all the ways we just went through:

• Make it sufficiently modular that it can quickly react to what the AI input is
• Have workers around that are sufficiently skilled to adjust the production process to whatever the new idea of how we’re doing this now is
• Actually manage to not regulate yourself out of providing the data to the US developers in a way that creates these fast feedback loops
• Make sure that the plans are sufficiently set up with sensoring equipment and all this kind of stuff that actually reports the data back

There are AGI-compatible production plans for a bunch of things you can build right now that are extremely enticing and attractive for cooperation, both on a geopolitical level for the US government and on a commercial level for the AI labs. If you make these changes now, you have a very good position to then ask for these cooperations and integrate. If you don’t do that right now, and you just have a basically defunct plant somewhere that is still staffed by people that don’t really know what they’re doing, that isn’t really modernised in ways, then the pitch becomes a lot harder. Then the only thing you might be able to do is sell some of your IP you still have, sell the physical plants and then have the US take them over, reintegrate them, eventually indigenise them.

So I think the way you stay alive is just anticipating where this future is going and making sure that whatever your production process is actually becomes compatible with that. I think that, on the specific industrial policy stuff, is the most obvious intervention. This is the sort of micro example for the broader economic structure, because that’s the sort of cleanest and most direct integration example.

One slightly more abstract version of this is what you do about your labour market. Currently labour markets in a lot of middle powers aren’t very well set up to deal with disruption and quick iteration based on what AI systems do to the labour market. To the extent that there is a future labour market that still includes humans in important roles along bottlenecks and things that AI can’t do very well — and you have this idea of you have this very jagged frontier and the humans sort of fill out the spots where the AI isn’t particularly good, and the AI does the things that the AI is very good at — moving into the sort of dips in the jagged frontier of AI capabilities requires your labour market to be extremely flexible.

If your labour market isn’t very flexible, then people just stay in their jobs even if AI is better at their jobs. And then you have huge efficiency losses, people don’t move into the things that they would otherwise be good at quickly enough, your firms become uncompetitive because you’re not doing a particularly efficient distribution of the workforce across the task profiles that your human workers would actually be good at, and eventually you just risk being disrupted in a few ways:

• Either fast competitors that are integrating AI agents entirely sort of vertically just start disrupting you, because you are just not using AI to its full effects. Then at some point, these quick and more agile competitors just start outcompeting you, and your entire business model crumbles and then you actually have to do mass layoffs, your company just goes under, things got really bad.
• Or you just get a scenario where other countries in the world that are better at having a more flexible labour market, that are allocating their workers along the jagged frontier more efficiently, these firms just get more efficient, and then they outcompete you internationally, and then you have the same effect: your firm gets under pressure, maybe it gets outcompeted, maybe you get forced to just put up higher and higher trade barriers to protect that firm.

But either of those are pretty bad outcomes. So what you need to do is you need to make sure your labour market is flexible enough that the workers can move to the sections of the jagged frontier where humans will still be required, so you get the general economic efficiency gains and you don’t expose your workforce to the broader international disruptions from AI agents just being deployed everywhere.

And that’s a pretty counterintuitive finding, right? Because in a world where AI agents get really good and you get really worried about the workforce displacements, the first thing you might want to do is lock up the labour market and make sure people can’t get fired for now, and make sure that people stay in their place and stay in their positions and make sure that the disruption doesn’t get too bad too quickly — because that’s politically really risky, that’s socially really disruptive. Having mass layoffs and having people go to 6% unemployment for some time — 7%, 8%, 9%, 10% unemployment, like some of the labs are calling — that is extremely politically scary.

But what’s even more scary is: we lock this labour market in place, we stay at 5% unemployment for another year or two until our firms go under, then they do mass layoffs, then we end up in 15% unemployment. But it’s also the case that none of these workers can really reallocate into more useful jobs after that.

So I think you have to bite the bullet on the counterintuitive conclusion of making your labour market more flexible as the AI disruption comes in, otherwise you risk going to the second-best, very inefficient structure in the short term, and then get really disrupted in the long term.

Maybe one framing to think about this is that there is this sort of race between automation and augmentation, where you have this threat of this impending automation: the labs are building these agents, you might use these agents to build entirely vertically integrated firms that need very few humans. What you want to do is you want your firms that still employ a lot of humans to stay competitive with these vertically integrated, very agent-heavy firms. And you need to do that by moving towards the most efficient distribution of labour between human workers and AI agents. And if you don’t do that, you lose the race between the augmented firm and the automated firm: the automated firm wins and you have no use for the workers anymore.

So rather than try to win the augmentation/automation race, try to incentivise a flexible labour market that makes humans move into the parts of the jagged frontier that are actually human favoured, and then you can deal with some of the disruption.

Tom Reed: How do we do this without causing too much social unrest? Is there anything else we can offer?

Anton Leicht: Well, I think you just have to buy off the losers of this trend, basically. I think this is going to lead to localised disruptions. There are inevitably going to be people who are currently very good at the things that AI agents are going to be very good at in the future, and that are only good at these parts of the narrow task profile. They’re not very good at selling software, maybe they don’t have lots of great research taste, but they’re very good at executing on building software.

And these people you need to compensate in some way. You just need to buy them off, basically. These are the people that generate your social unrest.

It’s not trivial how you would buy people off. It’s not entirely clear. Something like wage guarantees and wage insurances have been done in past globalisation shocks, digitalisation shocks to economies. That has worked OK, but it still has created these pockets of political resentment. You can’t actually get around it just by giving people money.

Some people have floated the idea of giving them job guarantees instead, and just ensuring that these jobs stay around, but basically just subsidising them into existence. I think in the short term, that is not the worst idea. But I think there are real limitations to how well that works in this specific case, mostly because you can do wage guarantees and you can do job guarantees fairly well if the only thing that people care about is their actual job as they have it right now.

But a lot of the disrupted workers we’re talking about here are in their early to mid-20s. They’re not interested in their job because they want to do their specific job; they’re interested in their career trajectory, and they want to get somewhere with this. And if you just give them their wage and their job forever now, no one who’s like 23 years old and doing software engineering is like, “I just want to do this forever. I just want to sit here and execute doing this code. No, no. I don’t ever want to climb up the ladder. I don’t want to ever make more money. Just give me my wage guarantee.”

It’s very different than a 55-year-old guy who works in an industrial plant that you can just sort of pay off and be like, “You get the wage you got, you can stay at home for most of the day, you can work for 20 hours.” This is the deal that a lot of Western economies tried to make with displaced industrial workers. It also didn’t work great, but it kind of worked because those weren’t upwardly socially mobile workers that really wanted more out of their career.

So the tricky thing here is you don’t only need to buy off people’s current state, you need to give them something that is sufficiently attractive to slot in for the career they used to have. I think one of the things we can do about this is hope that if you just keep them around by force with something like a junior job subsidy — like incentivising firms to keep them around on top of the AI agents they use — that they just pick up enough along the way, they just learn enough along the way that they eventually just become good at other things, and they become good at things that will be valuable in the broader context of this augmented economy. And for that, you just need to get them through these first few years where they probably don’t have a lot of useful skills to offer, and they probably primarily have things to offer that are good for AI agents.

So maybe you can brute force your way through that with some sort of subsidy. I think that is a start, and I think that gets rid of some of the disruption.

Then the other question is how much of the disruption happens as a result of generally bad vibes. I think that you just can’t get around. If you have these AI systems entering the economy at the same time where you just have bad macroeconomic indicators and bad macroeconomic sentiment and you just have high-ish unemployment rate, even if it’s for unrelated reasons, people are just going to hate this trend, no matter how good or how bad the specific safety nets for the AI disruptions are. The vibes are going to be bad, the stories are going to be bad. There’s going to be political incentive to point at AI systems as the reason for these disruptions.

And if that happens, I think there’s just not all that much you can do. You have to try to counterbalance with some social policy interventions. You have to try to counterbalance with some economic [interventions]. At the limit this is just going to take courageous leaders that say, “Yes, this is hard. This is economic hardship. We’re still going to have to go through this, because if we don’t do this, the equilibrium we get to if we don’t do this is just even worse. So just grit your teeth. This is a pretty bad economic situation, but this is the only path to a better economic situation.”

And then you just hope that these leaders don’t get politically outcompeted by someone who has a better message of, “No, this guy’s just selling you guys out to the AI companies. We’re just going to shut down our market. We’re going to keep the AI companies outside. Screw these guys. We’re just going to do the firms as we always have, and we’re just going to put frictions in. We’re going to pass human-in-the-loop laws, we’re going to pass laws against AI agent adoption. We’re going to institute a particularly restrictive token tax that disincentivises AI agent uptake.”

You can come up with all these restrictive policies that ultimately just make you lose this automation/augmentation race and mean you get really disrupted in two or three years. But these are going to be attractive political solutions, and getting around them is just a matter of beating them in the political arena. That’s going to be hard, but I think it’s going to be necessary.

Tom Reed: What’s your preferred way to pay for these junior job subsidies or whatever else it is that you’re offering? I think you said you prefer a corporation tax to something like a token tax?

Anton Leicht: Yeah, I think the general worry here is anything that narrowly targets AI use makes us lose this automation/augmentation race even faster. Because what you want is maximal uptake of these AI agents at the things that they’re good at in the legacy firms that exist right now, because otherwise these firms get less and less efficient and ultimately do get displaced by either international or national competitors. So anything like a token tax that basically tells a corporation, “If you use more AI, we’re going to tax you more,” I think that makes it very hard, and that just disincentivises the kind of adoption we want.

The other thing you might think about is: can you tax the AI companies and the AI developers specifically? I just think maybe that works. The maximum version of that is taking the equity stake in the AI developers that the US government has recently started thinking about; Bernie Sanders has written about it a lot, and then the president has also started looking into this, I suppose, and calling meetings on this, and they’re having conversations on this.

Tom Reed: Is this a good idea?

Anton Leicht: No, I think it’s a very bad idea. A, I think it’s a bad idea on narrow economic grounds, mostly because we’re just not sure whether it works. Even if these AI companies become extremely valuable, like 10x or 100x their value in the next few years, that’s still not even remotely enough to pay for large-scale social disruptions. The economy is a lot larger than the AI labs right now. The economy is still going to be a lot larger than even AI labs that become 10 times as big as they are right now. So whatever equity stake you have in these labs is not actually going to be enough to pay for any of this. If it isn’t enough to pay for any of this, you still need alternative modes of taxation to pay for part of it.

But then by taking the equity stake, you just create a massive misincentive flow for the government in basically doing anything to the AI market and regulating AI in any way. I know we’ve mostly talked about the economic part here, but we also do want AI models to be regulated in a reasonable way for other risk-related factors, obviously.

And if the US Treasury and also just general US economic incentive and maybe like a Trump account filled with AI equity stakes or something is directly tied to the stock of AI providers, what kind of incentive does that place on the US government in terms of regulating models, in terms of making sure these models are safe? You’re going to give a lot of power to these labs to say, “Well, you can do this, but then the Trump accounts are going to crash. And how does that look politically?” And I think you don’t want to create this incentive basically ever.

Tom Reed: What about other countries?

Anton Leicht: Well, I think the question is do you even get the equity stake? You can buy it. That’s one way to do it. I think in the US they’re still thinking about just ceding some of the stakes to the US government. I don’t think there’s any other country where any ally would be interested in giving them stakes for free, so you just have to buy them. At this point, this is just a sovereign wealth fund with extra steps, but a sovereign wealth fund that is much less diversified than any other sovereign wealth fund. So this is like the stupid “I only buy Anthropic stock” version of the sovereign wealth fund.

And yeah, I think in the limit that works. That’s hard to sell politically, right? It’s already hard to sell the idea of “We’ll take on a lot of debt and build a sovereign wealth fund,” even though it is a good idea because it turns out stock markets are pretty good that generate a lot of revenue. But then maybe you just invest into the broader stock market, and you just do an actual sovereign wealth fund instead, and you probably put in the AI labs as a hedge.

But you need a pretty big sovereign wealth fund to pay for widespread social disruptions like that, so I think in the limit you need something that captures wherever the AI growth accrues. And I think it’s a pretty good guess that just like corporate income tax is a pretty broad net that captures a lot of value accretion at all kinds of different steps of the supply chain: maybe it’s the chips and then the tax hits Nvidia; maybe it’s the model development and then the tax hits the model developers; maybe it’s the downstream users and then the tax hits the downstream users. But we get some of it anyway. We just increase the corporate income tax. That is what we have right now. That is how we capture a lot of the value.

I think there are other ideas around that are even more elegant in some ways. A consumption tax, just like a very broad-based consumption tax for whatever consumption, is one of the other things that people have talked about. I just have a political issue with that, which is: it works elegantly, but then people open their DoorDash app and they look at their order and there’s like a $1 like consumption tax on the burrito they ordered.

Tom Reed: That’s kind of regressive anyway, yeah.

Anton Leicht: Yeah, I think that’s just not a great vibe for a new tax, and it makes it feel like the burden is on the consumers, even though most of the consumption tax probably does fall on the companies instead. I think it’s just politically more elegant to campaign on this as “We’ll raise the corporate income tax,” and that’s just the thing.

Tom Reed: Isn’t there a big risk of capital flight also, if you have vertically integrated firms, and they’re not tied to a workforce in a particular country? Doesn’t that just make capital a lot more mobile than ever? How do you deal with that situation?

Anton Leicht: Yeah, as with many of these things, a lot of the basic low-hanging-fruit solutions that we already have for general issues in corporate tax and capital flight just become a little bit more important. You probably need global or at least between alliances you need minimum floors that just work. We’ve got pretty close to that. That seems like a tractable problem, comparatively, to me: just like a 10%, 15% minimum corporate tax rate floor across major allies that could potentially host this.

These firms are mobile, but they’re not that mobile compared to just a postbox somewhere, because they do need a physical anchor. A firm that uses a bunch of AI agents needs a data centre that runs these agents somewhere, needs access to a data centre that runs these agents somewhere. So the fact that the most disruptive firms, we would expect them to have huge compute spends, and the compute spends need to be anchored in a data centre somewhere, you do get a revenue anchor for these firms in more general terms — because they have to be linked to a data centre somewhere, and then they can’t really escape the sort of gravity of this data centre in terms of where they’re located.

Tom Reed: But you also want to incentivise them to build your data centre in your country too. So doesn’t that cut against that somewhat?

Anton Leicht: Well that’s great, right? I mean, you will probably be incentivised to give them some carveout to build a data centre in your country. But it also means that they can never entirely run away from the tax. You get minimal race-to-the-bottom dynamics as you always do with company headquarters, but we currently have that sitting, and the floor is… Yes, corporations don’t pay as much tax as in the ideal scenario where they all sit in the high-tax countries or whatever, but it’s also not a pure race to the bottom. It turns out not every company is listed on the Cayman Islands, because there are a lot of factors that pull against it.

And not that many countries can build a data centre, and not that many countries are willing to muster the political energy and the capital backstops and everything to build huge data centres. So you will have the data centres hopefully diversified across a spread of countries that isn’t entirely incentivised to just put the corporate tax rate to zero, especially if you have global minimum floors, and then you can’t actually escape the pull of the data centres. Having the data centre gives you some minimal tax revenue anchor, and the fact that this is also infrastructure-bound means the capital can’t actually fly away from the minimal tax rate anywhere.

So you would think that the economy just sort of smoothly distributes across the countries that have data centres that can serve as revenue anchors, and I think things probably shake out somewhat fine. It’s still going to be hard to capture a lot of this value, especially if you compare this to labour taxes. It’s just very easy to tax labour, because labourers aren’t particularly mobile and because we have pretty high taxes on labour, all things considered — both direct income taxes, but then also all the costs that come with payroll, especially in some of the European countries.

So if we move a lot of the revenue away from labour and towards basically capital — which is to say AI agents and the infrastructure they run on — we do get much fewer taxes in total. You have rates of 25% maybe on corporate income, and on everything to do with the capital. We also give a lot of tax breaks to capital investments compared to labour investments, so if we just naively move one unit of revenue from labour to capital and AI agents, I think we lose maybe half or more of the tax revenue we get off of that.

So yes, there is going to be a massive revenue crisis for most treasuries in the world. Yes, there is no good tool to think about this right now. And yes, at minimum, corporate income tax changes are probably going to be necessary; at maximum, we have to basically rethink how we do tax revenue on private-sector activity, all things considered. Because right now it all hinges on fairly high taxes on human labour that we are not going to be able to sustain.

Who really governs Anthropic? [01:08:29]

Tom Reed: Within the US, to what extent or what kinds of nationalisation do you think are in some way inevitable? Like, how do you expect Anthropic to be governed by the time Claude is running robot factories?

Anton Leicht: My mainline scenario, especially in the current administration, is there is no strong framework. There isn’t at any point an EO [executive order] or a bill that says, “This is how we’re soft nationalising or hard nationalising anything.” It’s just that you have slightly more embedded observers that are linked to the US government within the labs that get what’s going on. Maybe these are third-party evaluators that report to the US government. Maybe these are just directly placed assets from, for example, the intelligence community that just sit in the labs and monitor what’s going on. And then you just have these feedback loops where the labs communicate more and more of what they’re doing to the US government specifically.

And then there are people in the US government that think about whether they like what’s happening or not, and if they don’t like what’s happening, then the executive calls up the lab and says, “Have you guys considered not doing that? And would you guys not like to do this differently? Because if you don’t, then we have this range of potential threats that we could come up with, but we never quite do” — whether that’s export controls on the models or on the model weights, or whether that’s sort of Defense Production Act–related interventions into the data centre, or compelling the labs to turn over their models, or compelling the labs to give information as to what they’re up to and what they’re doing.

You have all these theoretical policy backstops that are already available to the executive today, and I think most of the nationalisation that happens is just going to be the admin sort of vaguely gesturing at, “You guys don’t want us to pull on any of these levers, right? So maybe just do what we want in some unspecified way.”

And then there’s going to be a push and pull. There are going to be commercial forces pulling against that, and the labs are going to say, “Well, we’d like to, but this is pretty bad for our market position.” And then the admin is probably going to say, “Well then, maybe don’t do the worst version of this.” And we’re going to see a lot of messy negotiations like around the Fable thing happening time and time again.

I don’t think we’re going to see a big framework for any of that; it’s just going to be this continuing push and pull of the admin threatening to use some of these tools, and then the labs sort of acquiescing in some way, and the labs kind of doing what the government wants in some way. And we just try to figure out what the balance of power there is.

And then at some point we’re probably going to come to some sort of law. There are going to be executive orders that come up with some minimal version of this that are going to somewhat standardise some of these processes. But I think a lot of it is just going to be the soft back and forth between the admin and the labs.

Why “pausing superintelligence” fails [01:10:52]

Tom Reed: Under what conditions would you advocate for something like a pause or a slowdown? Are you excited by this idea?

Anton Leicht: I’m not. I think there are two questions here. The first is what is the politically realistic version of a pause that we would see happening? And then there’s the question of what kind of policy intervention do we like in the abstract?

On the politically realistic version, especially if this comes from the general, “This stuff is really dangerous; we’ve got to get a handle on this and we’ve got to get this coalition of people — who all vaguely don’t like AI for somewhat unrelated and uncorrelated reasons — we’ve got to get all of them together and we’ll build this constituency and they’re going to ask and push for the pause and then we’re going to do the pause,” I think this is very likely to not come up with a very good pause idea. And I think you need a very good pause idea for a pause to work right.

If you just pause in the US in a sort of naive way, then of course the first thing you do is you crash the stock market, because that is just super dependent on the infrastructure investments that would run directly counter to a pause. It’s contingent on the IPO expectations of the labs and everything. So domestically, the naive version is pretty harmful policy.

And then you need all this international sophistication for the pause to not actually mean that all the compute reconcentrates somewhere else, that China doesn’t catch up in terms of semiconductor manufacturing in the meantime, and just builds the thing later on. There are all these complications that require you to have a very sophisticated treaty in place and not just the domestic pause.

To their credit, a lot of the most intelligent pause advocates know this, and they have developed treaty architectures that they’d like to see. But I think politically speaking, it’s going to be very difficult to get this kind of treaty done right. You have this broad coalition of everyone who doesn’t like AI you get into the boat: they’re worried about the job impacts, they’re worried about the environmental stuff. And then you get them to the point where they pass a bill or where they have a majority for a bill that says, “Shut it down, let’s stop this” — some vague sense of no more new data centres, maybe let’s tear down some of the old data centres, no more new AI models.

And then the question is, does this coalition stick together to actually also negotiate the treaty, to actually advocate doing the most sophisticated version of this, to keep compelling policymakers? No, no, the domestic policy is not enough. You now have to get under the treaty, you have to get the Chinese on board, you have to fly out there again, you have to do another round of negotiations. And if they’re not going for it now, you have to give up more to get them into this treaty.

Is this coalition really going to be interested in the nuances of the safely-relevant part of the pause treaty, or is the broad coalition only there for the vague symbolic policy of “we’re doing the pause,” and does the rest of the coalition jump off the ship at the moment that they’ve gotten the pause, and then you’re just left negotiating for the treaty? You have to get 60 senators on board for ratifying some crazy treaty.

Tom Reed: Isn’t there a version of a pause which is a bit more like essentially a slowdown, where what you’re really doing is just reallocating compute? So you’re making commitments about reducing the amount of compute that you’re using for R&D, increasing the amount of compute you’re using for inference for state capacity or societal hardening or something like this. Is this something you’d support more readily?

Anton Leicht: I think that’s where we get into that’s less of a pause, that’s more of a slowdown. And I think that is sort of what a sophisticated vehicle would be like in the abstract, which is just like, “Let’s find some elegant way to reduce the pace of frontier capabilities specifically, and increase adoption and increase rollout and societal hardening instead.”

I think that probably works pretty well if you can coordinate around it in some reliable way. I think there is still a geopolitical issue to that, in that a symmetric pause in frontier capability development is not a symmetric pause in US–China competition for AI — because right now, China doesn’t actually do a lot of frontier development, obviously. What they do do is a lot of diffusion and a lot of catching up on semiconductor manufacturing.

So if you pause specifically frontier development symmetrically throughout the world right now, and if you slow it down in some coordinated way, I think most of the Chinese ecosystem is extremely thankful for that, because the slower we play this, the quicker their semiconductor manufacturing catches up, the quicker their diffusion plays work out through all of the economy.

Tom Reed: You have to catch up pretty fast though.

Anton Leicht: The question is like, how long is the slowdown? What does the slowdown give you?

I think that the question is how much would the pause three years ago, when they asked for it for six months, would have really helped you? We would have still been like, “Well, we really haven’t built that much resilience.” Will we really have, in that time, anticipated the kinds of things the systems can do now if we have passed the policies that would have worked now? I think you have to do a pretty long slowdown to actually get on top of all the things, and then also make the predictions of the things you would like to in the future get on top of. So the slowdown has to last for a while.

And I think right now, as it is, it would be a geopolitically asymmetric slowdown — which is an extremely difficult ask to make of the US government in the context where they do see themselves in a geopolitical competition. You might think that AI is not only something that the US needs to sort of narrowly win on, but AI is the advantage that the US has to win the broader great power competition. And if the frontier lead is the one thing they really have ahead of China, plus the compute lead, if you institute a policy that specifically hits the one thing that America is very much in the lead on, I think that’s a very hard geopolitical sell.

So that’s probably something that the US government is going to be very sceptical about. If that’s something the US government is very sceptical about, then you’ll need broader Chinese concessions to make up for that.

For this to be a fair deal, geopolitically speaking, you instead have to ask China to also slow down on semiconductor indigenisation and on AI diffusion — because that’s how you get a geopolitically even version of this case. And good luck getting China to do that, I think. And good luck verifying not only the fact that they’re not using their frontier development very well, but verifying all these unrelated industrial policy efforts.

I think a lot of good work has been done on verification of a narrow pause treaty or a narrow slowdown treaty, where it’s like, do we have some way to figure out whether the Chinese and the US are both sticking to this narrow idea of what you described? Which is like, we put more compute in the deployment, we put less compute in the training runs, we don’t exceed certain sizes of training runs, we don’t put that much into the RL and everything.

I think maybe we can come up with a way of verifying that, but can you really verify what kind of progress the Chinese semiconductor industry is making while the slowdown is happening, and make sure that the geopolitical parity of the deal works out that way? I think that’s a very hard case to make.

I think sometimes the discussion about this basically just makes the case, “Well, the US would still be ahead, and if the US is ahead, then there’s no way that China would catch up in that world.” But I think that misunderstands the geopolitical dynamic here. The geopolitical dynamic is China is better at most other things that matter for great power competition right now. And the one asset that the US has right now is that basically much of its economy, and also some of its strategic environment — and an increasing amount of its strategic environment — is a leveraged bet on AGI development. And if you slow down specifically on the AGI development, that is just a huge asymmetric hit to the US. And if you ask that of the US, I think it’s just not going to work.

Tom Reed: OK, why not rally around a slightly more stupid but maybe still appealing idea of just a ban on superintelligence? So it’s not a pause, it’s just like a permanent thing. And this coalition is broad enough, and the US is interested, China’s interested.

Anton Leicht: I just honestly have no idea what that means. I think that is just an incredibly underspecified thing. Does that mean that systems aren’t allowed to be superintelligent at everything, or anything?

Tom Reed: No AlphaZero.

Anton Leicht: Because systems are definitely superintelligent at some things right now. Obviously they exceed human capabilities at some things. So that narrow superintelligence ban… I mean, this is like the lazy version of this “Do you want to ban calculators?” thing.

But even if you go into slightly broader human domains, there are already important task profiles — even strategically relevant task profiles, like large-scale data and image analysis and this kind of thing — where AI systems are already superhuman. So we have narrow superintelligences even in strategically relevant tasks already.

What we don’t have yet is a broad superintelligence that’s better than humans at everything. But how do you operationalise that? Well, you just keep one thing behind the giant frontier. It never gets quite as good at, I don’t know, colouring colouring books or something. And then it’s not a superintelligence because it doesn’t have the broad capability set?

The operationalisation of this is just very difficult. And I think you can just, even under a general superintelligence ban, build extremely dangerous, very jagged systems. And I’m not entirely sure that you’re in a much better place with that. Maybe the most charitable way to operationalise something like “Is there something we can ban?” is maybe banning closed RSI [recursive self-improvement] loops within labs specifically — like basically ban anything that would realistically lead to an uncontrollable and unsupervisable software-only intelligence explosion — or even ban anything that, even if it’s not software only, just bans closer and closer RSI loops that just makes systems much better, much quicker than we used to have.

I think there’s something about this kind of policy, in that it doesn’t only ban a thing that seems vaguely scary. There’s a more principled reason for why you want to do this, which is to retain visibility into what these systems are doing. Ever-quicker RSI loops are just extremely difficult to monitor. It’s very difficult to come up with a way that a government would actually have reasonable insight into the risks that emerge from this as the RSI is getting faster and faster and faster. So I think there’s a good argument to be made that this should not be something — with our current evaluation infrastructure, with our current oversight infrastructure — that labs should really be doing at the moment.

Again, this is not going to be as obvious and as easy of a sell, because this is one of the theories of victory for the US AI developers and therefore will also become one of the theories of victory for broader US–China competition for the US. Closing this loop and actually making these systems go exponential in their capabilities much faster than they otherwise would is one of the few ways you actually translate vague few-month advances in AI capabilities into very broad geopolitical leverage, by actually getting to super-super-superintelligent systems that can both carry out all these important tasks, but then also maybe by themselves prove to be some version of a decisive advantage.

And convincing the US to not click the “decisive strategic advantage tomorrow” button is still going to be extremely difficult, but I think especially the strong closed-loop versions of RSI are maybe the most easily operationalised ways of thinking about what we should ban or slow down or prevent. And I’m sympathetic to doing that.

Is an American AI monopoly safe? [01:21:08]

Tom Reed: How do you personally feel about the prospects of an American decisive strategic advantage? Whether or not it’s possible, is that something that would actually be net positive for the world, in your view?

Anton Leicht: Yeah, I think I’m still unsure about what exactly a decisive strategic advantage is. I think you can come up with softer and harder versions of it that work or do not work in the real world. But taking the strong framing of it — which is basically just that you get unilateral power over any sort of strategic rival or any other country, basically enforcing your will on them in any potential way — I think I’d feel a lot better about a Western alliance with a strategic advantage, with the DSA, than just an unconstrained US.

I still have a little bit more faith in the ultimate balance that will shake out between the US and the labs, but I think there are a lot of ways that US decision making around how to deploy this could go very badly. We’ve seen the administration is very volatile in how it chooses to deploy its assets. I wouldn’t be entirely sure that it deploys this asset very well. I think we’ve also seen unfortunate tendencies in the securitisation of the labs that would lead to this. I think there are a lot of unfortunate amalgamations between the labs and the US security apparatus specifically that you might come up with that are maybe favourable to some narrow version of American strategic interests as determined by whatever the political mood of the day is in the administration, but that aren’t necessarily furthering the world very much.

That being said, if you think this advantage is something that is slightly constrained within the context of both a separation of power within America — between both the classic constitutional separation of power, also the separation of power between the private businesses that build these models and the government that oversees how they’re used — and then also a broader separation of power within the broader Western alliance that controls its advantage, and there are some more checks and balances on how exactly it’s deployed, and it’s mostly like an ultimate enforcement stick and like a last piece of leverage that just wins geopolitical competition, I think that is a much more tenable future that I’m much less worried about.

Because then you still get all the positive effects: you get AGI deployed or ASI deployed in a way that is ultimately backstopped and backed up by somewhat functional liberal democracies that sort of control how it’s used. That seems pretty good, and that seems like a better way to control how these systems are deployed than basically any other way we can come up with. It also seems broadly geopolitically good and also geopolitically stable for this advantage to exist, but to rest in the hand of a broad coalition that has a decently high bar of adversarially using it.

So the best version, the alliance and separation of power constraint version, I feel good about. The narrow version — where you have a closer and closer amalgamation of labs and US government and then this big public-private partnership or whatever of labs and USG just can use this on a day’s notice — that I’d rather not happen.

Tom Reed: And from the perspective of risks from loss of control, is there a reason to believe that middle powers being involved, having access to compute, is going to be helpful from a safety perspective?

Anton Leicht: I would think so. My general view of this is: what it takes for us to lose control over these very powerful systems is fundamentally some people making stupid decisions. Well, just on the policy front. I know that the technical problems are hard and I think the technical problems don’t only come down to people being stupid; I think we will have to actually figure out a lot of the technical stuff that we haven’t figured out yet. But on the policy side, we need —

Tom Reed: To choose to stop people from releasing MechaHitler.

Anton Leicht: Choosing to deploy a system that’s not safe, giving it abundant access to compute and the internet and all these things, all these classic preconditions to loss of control, they do also come down to just bad policy decisions being made around both the governance of the developers and also the governance of the specific systems.

So the question is: in what environment do you get the least density of stupid decisions that can really screw over other ones? And I think whenever you concentrate power within a single volatile actor, the variance of the decision quality just obviously gets higher, right? It gets less constrained. You get more daring moonshots and cool things you might deploy these systems with. So the upside of just letting one guy run wild with his AIs may be even higher because it’s less constrained by process and whatever, and you can make risky bets. But the risk of just doing something monumentally stupid with a very intelligent system is also much lower. So it’s much higher if you have the crazy guy and it’s much lower if you have the constrained version.

So I think just if we narrow the option corridor a little by involving more people in the decisions of how these systems are deployed — and more people can unilaterally shut them down in some way or unilaterally threaten to blow up the structure that makes these systems possible by withdrawing a bunch of the compute, withdrawing a bunch of the supply chain inputs or whatever — if there is more mutual control over what is happening with these systems, I think the band of outcomes just gets a little bit more narrow, and we avoid the stupid policy decisions that seem to me to make up a lot of the policy risk for loss of control. So I think just broadening the inputs seems robustly good in that case.

And then on a much more general note, I feel like just having middle powers in the room and sort of geopolitically leveraged, more evenly distributed, is a less chaotic and volatile world in a lot of ways. I think if the US pulls ahead, but the US pulls ahead alone and the rest of the world sort of gets disrupted in all the ways that we’ve talked about — the labour markets get disrupted, the strategic position gets seriously threatened, people lose their jobs, people get hit by AI misuse every day, they start protesting on the street, there’s mass migration flows happening because they want to move to countries that actually have to access to the defensive AI that keeps them out of that, countries get into conflicts because one side of the conflict has access to good AI systems and the other side doesn’t have access to these systems, and these temporary technological advantages reopen old conflicts and make for an opportune window for one country to strike against the other — all these scenarios where a world that doesn’t have equitable and good and mutually beneficial access to AI systems just becomes a volatile and chaotic and messy world.

Just from a very general perspective, that doesn’t sound like a world where we’ll be more likely to make good technical and policy decisions than the world where we have pretty complex and well functioning supply chains, people are generally not entirely discontent and disrupted by the situation, there is less AI misuse happening, and the world just generally seems less like it’s going to hell outside the borders of the US.

If you just think about which world would you rather make the ASI policy in and make the important decisions in, in what world do you think the people in the room are going to respond to better incentives? In what world do you think the decisions that are going to be made are going to be more sane and less reactive to a world in crisis? Then I think you just want the more stable world.

And I know that that’s not the sort of finished causal model that people usually like when they talk about these things and when they talk about the impact that work specifically has. You usually want this very step by step by step: “…and we built the compute in Japan and then therefore the ASI rather does this.”

I can’t give you that link, but I can give you an offer to build a much more stable world that does much better on any basic categories of what we would want the world to look like and what world we would think good decisions might be made in. And then I put to you: I think that sounds like a much better world to make these important decisions in, and I think that’s a good enough reason to work on this.

Explaining AGI to the world [01:28:40]

Tom Reed: And what are you doing in this world? Are you just going to be writing all the way up to the singularity? Do you have a plan?

Anton Leicht: If people keep reading. I think there is a real open question that anyone who’s doing writing and work in the outside game right now is facing, which is: how is the outside game changing relative to the inside game?

I think a lot more information that is basically only available within the labs and within the governments is becoming more and more relevant. The labs have exclusive access to the best frontier models and the gap between internal and external models is maybe going to increase. The information siloing is going to get worse and worse, you would expect, as more of these things enter national-security-relevant domains. So I think fewer of the discussions are going to happen on the outside.

Dean Ball had this piece, I think it must have been two years ago now, about the sort of “republic of letters” that we had when SB-1047 was being discussed and everyone was writing their little essays and writing their little tweets, and people were going back and forth, and it was all an open discussion and there was like a battle of ideas — and, you know, you could make a real contribution by writing about this and saying things about this.

And one by one, the people writing the letters are choosing the inside game instead, and the republic of letters is sort of getting starved of both its core input, but also of its core output — in that the decisions are just moving further away from the idealised discussion on the outside. So I think that does ask a real question about the value of the outside game on all these things.

But I think there is also something pulling in the other direction, which is: this still needs doing. All of this needs explaining. All of this needs analysis by people that have sufficient access to the information by having the conversation with the people that are on the inside, but still not looking at this from the biased institutional view of any one specific place.

And it also just needs doing from the perspective of, you need to make these points and these findings credible to the rest of the world. I think especially in the rest of the world, where people are still sceptical of the narrative that might come from inside securitised environments or inside the labs, there is real value in making this case and making this argument in the context of the outside game, and not just in privileged conversation between intelligence officials between different countries, or in a pitch that reads like a marketing pitch made by a lab to a foreign government.

So I think especially for the work I do, there is still real value in doing it from the outside. But yes, I think there is an ongoing trend that pulls against the outside game. And I think that is something to be — not for myself, but for the ecosystem — somewhat worried about, and something to at least look at.

Is Anton bullish or bearish on Germany? [01:31:05]

Tom Reed: Are you long-term bullish or bearish on Germany?

Anton Leicht: I think bullish on the macroeconomic basics, and I think there is also a sort of regression-to-the-mean argument to be made here, in that this has long been a very successful economy and a very — at least in the context of the West — obviously a strategically important country. And maybe you just think that, incidentally, this is not going so well right now. It used to go better 10 years ago, maybe it will go better 10 years in the future.

And maybe this is the very high-level historical view, and then the more you zoom into the details of what is actually happening, the more worried you get about some of the specific policy features. Not to expand this to general domestic politics conversation, but just keeping it to the AI policy conversation, Germany has a lot of great assets for doing well in AI in very general terms: a very skilled workforce; less service-heavy economy than most other comparable economies; still a pretty good industrial base; a pretty good network of both bilateral and organisation-based relationships that I think are the right set of relationships to have if you think that the US is building AGI, but you still need some counter-leverage against that.

I think all of these are a great position to be in, and the only things that Germany needs to do is make all the politically hard decisions to get back to 3%, 4% economic growth, a flexible labour market, all these things. In a way, if Germany only solved all of its other problems, it would be fine on AI, which is a fairly contentious thing. There are a lot of countries that have just bad assets for AI specifically, and if they do well right now, it doesn’t mean they do well with AGI.

Tom Reed: Like who?

Anton Leicht: Say maybe Singapore or Hong Kong or maybe even the UK: if you have a very service-heavy economy that is very susceptible to disruption by AI agents specifically, it just seems much scarier than if you have economic assets that matter in a broad set of AI futures.

A lot of German AI policy and I think a lot of European AI policy in general is just much less about AI and much more about general policy. So in a way it’s very easy to solve, because you just have to do all the things that hundreds of thousands of people are thinking and writing about anyway. But in a way it’s also much harder to solve, because you can’t go there and you can’t give them a two-page memo of, “Here are all the clever ways to pull the rope sideways: come up with all these very specific interventions, and then you fix your AI policy and then things are good.”

No, you actually need to go through the messy politics of doing the labour market reforms and getting the industrial base back on track and all these things. And that’s really hard in a way, but that’s at least a known and well-scoped challenge. And I think if Germany can get a handle on that, then it’s in a great position for doing the AI stuff well as well. TBD. And honestly, way out of my depth on whether that is going to happen and how that should happen.

Tom Reed: Great. Well, it’s been a pleasure having you on, Anton, and we’ll see you next time.

Anton Leicht: Thank you so much for having me.