#257 – Tom Milton on how we could verify a US–China deal on AI

Transcript

Cold open: Can we verify a US–China AI deal? [00:00:00]

Tom Milton: Your most likely route to the Nobel Prize for making AI go well is that you are the one who develops the verification tools that enable a deal.

Tom Reed: You would need to retrofit like every single data centre in China and every data centre in the US — like it just seems to me like it would require a level of cooperation that’s completely impossible.

Tom Milton: When I speak to my non-AI friends about this, they kind of say, “Why can’t you just turn it off?” And you do always have this fallback that you could turn data centres off. We could turn all data centres off in the world, but that’s extremely costly. Lots of what verification is doing is reducing the costs to complying with some kind of slowdown.

Tom Reed: Just by looking at the power, you can be like, “OK, great: there’s no backwards passes happening here”?

Tom Milton: So as the power is being used, it fluctuates. So you can see all of these little spikes, the really small ones. That happens at the start of each layer.

Tom Milton: A marginal really talented AI safety researcher could have a huge impact on verification. I think if you started on this today, by the end of the year, you’d probably be one of the best like five people in the world working on verification, or one of the most useful five people in the world.

Why compute is the best place to verify an AI slowdown [00:01:04]

Tom Reed: Today I’m speaking with Tom Milton, who is the CEO of Amodo Design, which is a company that builds hardware for scientific research, biosecurity, and AI verification. And it is, above all, the last of these that I’m interested in talking about with you today, because I feel like the problem you’re trying to solve is both very hard and very, very important.

Pretty much anytime I look at AI news, it looks to me like there is another lab CEO, or another letter signed by thousands of employees, insisting that they want to slow down the progress of AI development, but that they are personally unable to do so for fear of losing the frontier or ceding the frontier to either another American company or, really, in the long term, China.

And it feels like so much of what we would want to do on AI safety in basically any capacity is bottlenecked on this ability to trust that if everyone wants to slow down, the other actors are in fact also slowing down.

At Amodo, you are building tech which would allow us to do this. Could you tell me exactly how this tech might work?

Tom Milton: Absolutely. We’ll start by setting aside the fact that I think a unilateral slowdown could also be good. We’ll go on from here assuming that people are shooting for multilateral actions.

When we think about a slowdown, the problem is trust, and therefore the problem is knowing what is happening at another AI developer, whether it’s a country or a company. And we can then break that down into some of the inputs of development. You could think about the employees as one input; you think about the software, algorithms, and data; you could then also think about the compute.

And we’re focused on the compute. This is the physical chips that AI development and deployment runs on. And this is a really attractive place to do our verification — to do our checking whether someone is slowing down — because they are very specialised to the AI setting. You need many thousands of them, and at least for now, they all need to be in the same location. So it’s a great place to start for monitoring what different parties are doing on AI.

Tom Reed: And so you’re building ways that you could look at compute and make claims that would be verifiable by someone else about what that compute is being used for. What kinds of claims would you be aiming to verify?

Tom Milton: Exactly. So you’ve said the word “claims” there. To explain a bit of language at the top: we say there is a ‘prover’ who makes a claim, and a ‘verifier’ who then checks that claim.

Tom Reed: Right, so the prover could be an AI company.

Tom Milton: Exactly. It could be an AI company, whether that’s controlled by a country or not — essentially the developer. And then the verifier: this could be another country, it could be a third party, whether it’s an embedded auditor or international organisation.

And the claims that people want to verify depend kind of on what your goals are. So recent examples might be OpenAI claiming that they have stopped training after their latest RL [reinforcement learning] accidents. So you might want to verify the claim, “We are not doing any reinforcement learning inside our company.”

And as AI safety science advances, I think we’ll have more specific claims that we want to verify.So right now we can say this data centre is on or off: there is no AI activity happening, or there is some. Or we might want to go all the way through to details like: this safety training technique has to happen, or this type of training technique is too dangerous and we promise we’re not doing it. Or these data were held out, or we are or aren’t using neuralese. But most of that requires the science of AI development to…

Tom Reed: To understand which claims you would want to be able to make to begin with.

Tom Milton: Exactly.

Tom Reed: And so the verification R&D process is allowing us to make more and more fine-grained claims, so we’re not stuck at this very costly level of, “The data centre is literally off, and therefore I know for sure there’s no training happening there.”

Tom Milton: Yeah. When I speak to my non-AI friends about this, they kind of say, “Why can’t you just turn it off?” And you do always have this fallback that you could turn data centres off. We could turn all data centres off in the world, but that’s extremely costly. Inference is valuable, providing access to the models is valuable.

And lots of what verification is doing is reducing the cost to complying with some kind of slowdown. So at first it might be turning the data centre on and off, then it might be serving only inference — and being able to prove you’re only serving inference allows you to still serve inference whilst being worried that training would be dangerous and should be stopped.

Then we might want to do something like resume small-scale AI safety experiments — alignment experiments, for example. And this is where beginning to make more and more precise claims about how compute is being used is more and more useful. It both reduces the cost to coming to a deal, but also opens up a bunch of value for humanity in still being able to use the compute for something and still being able to use the AIs for something.

Can you prove a lab isn’t cheating? [00:06:08]

Tom Reed: Great. Let’s zoom in on this example of trying to verify someone’s claims that they’re using a given data centre to do inference and not to do training. And you can imagine this being useful if OpenAI wanted to verify to someone that they had in fact paused a training run, or you could imagine it to be useful in some grander scenario where the US and China have agreed to pause training for some amount of time. AI 2040: Plan A imagines this kind of thing.

From what I understand, you’ve built a prototype that is working towards this, that could verify that a small cluster of GPUs is in fact being used for inference and not training. Could you, at a high level, walk me through how that would work?

Tom Milton: So we’ve got a pilot of a system that uses what we call “inference recomputation.” If you were thinking about monitoring the compute, you could go all the way down to the GPU level, and you could try and look at the instructions that are running on the GPU. We operate at a slightly higher level so that it’s less invasive, and we look at the network traffic.

So there’s this front-end network connecting the GPUs to each other and also to the outside world. And we look at the information on that network, we install a network certifier — what this basically does is forward on the information to our verifier node; the information still goes on to its original destination, it’s just duplicated — and then on the verifier we run a recomputation.

You can think about this as simply rerunning the inference that happened. So normally your input tokens go into your model, you do your inference, and you get your output tokens. What we are doing is then feeding those through the model again and checking whether they came out the same. And this allows us to look at the front-end network, look at the information on that network, and confirm whether or not it came from the model that we all agreed should be running.

And of course there’s some nondeterminism in these models. You need to account for this. We use an algorithm called DiFR that looks at some of the probability of choosing different tokens, and on a long section of text can have quite high confidence as to whether these tokens did indeed come from the right model.

Tom Reed: And if no training is happening, then that model is staying constant, and therefore you know that it’s only doing inference, because otherwise the model would be changing over time?

Tom Milton: Yeah, so the model staying constant is one thing. The other thing is there aren’t packets, bits of information on that network that can’t be explained by the recomputation. So another way of describing this is bounding the unexplained communication. If you can explain all of the communication that’s happening, then you can know that all of that comes from the agreed inference. If there is this unexplained information on the network, then there is a workload that you don’t know about happening.

Tom Reed: Interesting. OK, yeah. So just to lay this out again: you’re claiming to me, “Here’s the set of GPUs that I have, here’s the model that I’m serving on these GPUs to these customers. And you’re going to be able to trust me, because you are going to have a part within my data centre where you can rerun the model that I’m claiming I’m using, and it will produce the same tokens as the ones that are actually being sent to external users. And in addition to that, I can also put network taps into your server, and they will also be able to certify that there’s no other things going on other than the inference that I’m also able to recompute.”

Is that correct as a summary? Have I missed anything?

Tom Milton: Yeah, so you’re coming in to check me —

Tom Reed: Let’s say I’m the verifier, yeah.

Tom Milton: You’re going to put in your server that does the rerunning. The network tap — “tap” stands for “test access point” — is a piece of fairly standard data centre networking equipment. And all that’s going to do is forward the packets on to the verifier. So this is the piece of equipment that basically bridges your verifier hardware to my prover hardware. And the main thing that you’re going to do when you come in is install that tap to forward the packets on to you, so you get a copy of them as well as me having my original copy of them.

Tom Reed: And you’ve built a small-scale version of this and it works. So I would be able to run this, and whatever you’re running on your GPU, I don’t need to look into your GPU in any kind of invasive level of detail. I can just look at what my own recomputation is providing, and I can be certain that you’re running the model that you’re claiming.

Tom Milton: Yes, exactly.

Tom Reed: What scale does this work at?

Tom Milton: So we’ve got this running on about 30, 32 H100s, H200s at our demo research cluster in Sheffield.

Tom Reed: Great. OK, that’s cool. And are you hoping that this basic prototype would be the kind of thing that could be used in a deal between the US and China? I mean, this strikes me as requiring quite a lot of mutual trust. I need to trust that when I’m putting my server within your data centre, you’re not then tampering with it. You need to trust that the network taps that I’m putting in your data centre, they’re not somehow picking up on IP that you don’t want to give to me.

Am I right in thinking this requires a lot of mutual trust?

Tom Milton: So I think the things that get implemented in future deals are likely to, rather than relying on recomputation, end up relying on zero-knowledge proofs.

Tom Reed: Where recomputation is the part where I rerun the model weights.

Tom Milton: Exactly. And the important part of this is that rerunning meant that I had to have access to the model weights as the verifier. You’re the verifier; you had to have access to my model. And it was within the security boundary; it was within my data centre, so I had some confidence over the security of the model. I was fairly sure that by putting it within my walls, you weren’t stealing the model.

But we would much prefer if I didn’t have to share the model with you at all. And the way we can do this is with zero-knowledge proofs. And there are a few teams already working on this. We’re working to implement this.

Tom Reed: This is ongoing work. There’s not an established way to do this?

Tom Milton: Well, there are some established zero-knowledge machine learning approaches. The main problem with what’s kind of referred to as ZKML, or zero-knowledge proofs for inference, is that they’re really slow.

Tom Reed: They’re very computationally demanding. There’s a lot of overhead.

Tom Milton: Many orders of magnitude more computational overhead to run using the zero-knowledge approach. Although this is coming down, and seems like something that actually you could hill-climb with the AIs quite effectively.

The reason I’m hopeful on implementing the type of system we’re talking about is that we don’t need to check every single packet on the network. So although it may be thousands of times slower to generate this proof, actually only checking one in 1,000 gives us some reasonable confidence.

Tom Reed: There’s a way you can sample some of the packets that are being sent around without having to do it with every single one, and you know that they’re not spoofing you in some way.

Tom Milton: Exactly. Yeah, subsampling. And the fraction you need to sample is roughly on the game theory you expect of their cheating, and how worried they’d be about being caught, and the probability of getting caught.

So right now a few of our team are implementing some of these zero-knowledge proofs on top of the prototype I just described, so that rather than having a recomputation server, that verification server is running some of this zero-knowledge system, and therefore the prover doesn’t need to share the model weights with the verifier. And that’s going to be much more palatable on an international scheme.

Live demo: how to spot a cheating GPU [00:13:22]

Tom Reed: Great, great. And as I understand it, this is just one of the ways you could verify — you’re looking at a specific type of information, you’re looking at the network traffic or whatever, to understand what a data centre is being used for.

But I believe we have here with us a prop in the studio which tries to establish the same thing — AKA that inference and not training is happening — by looking at the power that is consumed by a data centre. Could you walk me through exactly how that works?

And for our audio listeners, we have a prop here. You can follow along on video if you want to see exactly what it looks like, but we’ll do our best to describe it in the requisite level of detail.

Tom Milton: Yeah. In addition to monitoring the network, there are a bunch of other physical things we could try and monitor about a data centre and about compute. One would be how the memory is being used: doing random challenges to see what’s in memory and then correlate that to a workload. We can generate these zero-knowledge proofs. We could do something like proof of work, which gives you cryptographic proofs that all of the GPU’s resources are being consumed.

And then another thing we can do is monitor something like power. So the GPU is drawing a lot of power. We all know that data centres use lots of power, but as the power is being used, it fluctuates: the power consumption of the compute changes with the operations that it’s doing.

So in this box here we have a kind of mock setup of a simple server tray. We have an Nvidia GPU, a Blackwell series GPU, and the hardware that’s part of the verification here is this small inductive loop. So this is wrapped around the power supply to the GPU. And then we can see on our oscilloscope here how the power is fluctuating over time. So we’ve got time on x, and current on y. And you can see firstly a forward pass, and then a backwards pass, and then a break.

Tom Reed: So what are we running on this? What did we run on this GPU?

Tom Milton: I think we ran nanoGPT or something — GPT-2 or something like this. And you can see actually a bunch of properties just from the power consumption. You can see all of these little spikes, the really small ones — that happen at the start of each layer, and you can actually tell this is a 12-layer model from those spikes.

You see there’s a lot of information that leaks out of this. In the field of security, these are referred to as side channels or covert communication channels. There are all sorts of crazy ways that nation-states and other hackers use these to steal information. But we can also use these as a way to monitor the workloads that are happening.

Tom Reed: And the idea here is that the amount of power that a given GPU or even a whole data centre is drawing when it’s doing a certain action — like a backwards pass — is distinguishable enough from a forwards pass that just by looking at the power, you can be like, “OK, great, there’s no backwards pass happening here.” Is that correct?

Tom Milton: It’s kind of fingerprinting. Workloads have fingerprints. They have communication patterns you see on the networks; they have power patterns as well. And we might not be able to, at a whole data centre level, tell this is definitely a backwards pass, but there are fluctuations that are canonical — they’re kind of representative, or are often a fingerprint of doing different workloads.

And another kind of easy example here is that if you go and read news reports about frontier data centres, you’ll read about all of these grid problems that happen where there are these really large swings as you do your synchronous training. So they all do similar actions at the same time: they all do a bunch of compute, and then a bunch of stuff moves across the network while the compute becomes more idle.

And you see these big power swings — so much so that Meta have published loads of papers on how to mitigate these power swings. So we might not be able to tell to the fine detail we have here, because we’re on a single GPU level, but there are still fingerprints that we can use to understand what’s going on, and add this into a scheme like network taps, or use it as a preliminary tool in the early stages of some kind of coordination.

Tom Reed: Interesting. OK, so my takeaway from this is that there do in fact exist ways, if you know that there’s a data centre that’s being used for some AI activity, to know from the outside what is going on in that data centre without requiring super invasive access. Ideally you’re going to have some level of cooperation, but you probably don’t need to let someone in and see absolutely everything.

Tom Milton: The question with access tends to be: how easy is it to spoof it, and how specific can your knowledge about what’s happening be. From this, I think we’re unlikely to know about the exact details of how fine-tuning is done, or what data were held out during training, like we did or didn’t hold out our dangerous biological data. But we can, for a start, say the data centre is on or off. We might be able to then say something more fine-grained, like, “This is consistent with an inference workload,” “This is consistent with synchronous workloads like pretraining.”

The precision is therefore one of the things we trade off by not being inside the data centre. We also trade off the adversarial robustness, where by being further away we have now more opportunities—

Tom Reed: It’s easier for them to cheat.

Tom Milton: Yeah. If one of the pretraining teams from one of the frontier model companies spent a bunch of time trying to make training look like inference–

Tom Reed: As measured by something like power.

Tom Milton: Exactly. Would they be able to do it? I suspect so. They’ll take some efficiency hit. Whether that efficiency hit is very small or very large is still up in the air.

Tom Reed: Interesting. I’m presuming there are some kinds of training that look a lot like inference anyway. If you’re doing long-horizon RL, doesn’t that end up looking fairly similar to inference? Again, depending on how much level of access you have.

Tom Milton: In terms of your rollout and your generating of your inference tokens, obviously this is–

Tom Reed: This is just inference.

Tom Milton: We mitigate this in our network tap recomputation system because the model weights aren’t updating, which is an important part of inference. But in this case, because we’re further away, we have less resolution; we can make a much less fine-grained claim.

Tom Reed: Cool, cool. That makes sense.

Could China hide a data centre? [00:19:50]

Tom Reed: My overall take on this is: it does seem like there is a way, given a data centre that is doing an activity, to know things about what’s going on in that data centre. This is a good thing. This would enable some level of coordination.

I feel like the other half of the puzzle that we would need to know is that there aren’t data centres somewhere that we don’t even know about that are doing things like training. As far as I understand, this is not something Amodo is working on. But should we be optimistic that these are solvable challenges too?

Tom Milton: Yeah, this isn’t something we’re working on. I have a lot of uncertainty here and get persuaded either way. You talk to national security people about these things, and one day they’ll persuade you this is totally trivial because they’re so good at spying, and the other day they’ll say this is just totally impossible: “There is no way we could find a hidden data centre.”

A few things that helped me think about this. Firstly, we will have all of the normal spying resources — formally called ‘national technical means’: we will have countries using satellites and human intelligence and signals intelligence spying–

Tom Reed: They’ll be hacking each other. It’s clearly easy to hack OpenAI anyway, apparently.

Tom Milton: Yeah, I mean, the easiest way for China to know whether US frontier labs are racing is for them to have terrible security and for them to just spy on what’s happening anyway.

Short of that, there will be some sets of signs of cheating on deals, I expect. The larger the project, I imagine the easier it is to catch a project. Could China hide a $1,000 project? Presumably yes, that sounds easy. Could they hide a million-dollar project? Probably. A billion, a trillion, 10 trillion project — increasingly crazy numbers seem increasingly difficult to [hide].

I do think the intelligence community will be able to do a reasonable amount of this. And under scrambles and quick races to put together a deal, a very large amount of our verification and trust in other countries will come from the intelligence community.

Tom Reed: Just unilateral surveillance, basically.

Tom Milton: Yeah, exactly. And I’m sure there will still be networks. I expect the US will call on the Five Eyes to contribute to their spying, but there are some mechanisms there.

Interestingly, when you talk to national security people, even the ones who think, “We’re so good at finding stuff on satellites; you guys just don’t know how amazing we are at spying” — even they think it is valuable for people to be working on stuff in the open. So OSINT, open source intelligence: there are new projects using satellite imagery to detect covert data centres.

And one of the reasons that they think this is valuable is because even though the intelligence community knows something, that doesn’t mean they can tell the world about it. And you see this in the nuclear setting, in which intelligence communities sometimes commission open source projects so that there can be some public information.

Tom Reed: They commission them to do things that they actually already know. They just don’t want to show that they know it.

Tom Milton: They can’t show that they know it because they don’t want to reveal their sources or methods, or show their hand. So it might be important that some of this happens separate to the IC [intelligence community] so that there can be international, multilateral, or public discourse about this information.

Tom Reed: The takeaway being that even if our espionage is good enough, it would still be very valuable to have other means of verifying yourself.

Tom Milton: Yeah, and I think this should encourage people who are interested in working on that to work on it, even though the intelligence community might do a reasonable job at it.

Tom Reed: OK, interesting.

Can verification tools be hacked? [00:23:35]

Tom Reed: So some of these technical problems may well be solvable, and we will have ways of verifying these kinds of claims through espionage, or maybe if we cooperate enough to install some of this hardware on each other’s data centres or things like this.

But it still feels like the field is incredibly small. At Amodo, I think you’ve got 50, 60 employees — only some of them are working on this anyway. And the scales you’re working at, they’re very small. This here, there’s just one GPU, right? I’m pointing at the prop. And your prototype you said was also pretty small scale.

What needs to happen between now and a moment where we could actually use some of these prototypes, use some of these technologies, to verify a literal deal between the US and China?

So there’s a lot that needs to occur, right?

Tom Milton: Yeah. So I’ll talk a bit about some of the work that needs to happen, and then afterwards I’ll talk about who could do it and who should do it.

The technologies that are being developed at the moment I think are at a kind of prototype pilot stage. So a few organisations working on this — I’ll start a list and then inevitably forget someone. But Lucid Computing are working on some of these technologies, including some similar things to what I’ve shown here. There is the Singapore AI Safety Hub, who also have a network certifier system, as do the Future of Life Institute, are working on some of these technologies. And finally, a company called Attestable, who are working on zero-knowledge proofs for inference and proof of work.

These are all at the prototype pilot stage. I think within a couple of months, most of them certainly will be at the point where they can be red-teamed; they can be tested. So before you get to a deal, you want all of the parties to trust it, understand it, and also for it to actually work. We don’t do red-teaming just to build trust; we also do it because we do want it to work, know if it’s actually going to work.

Tom Reed: And by red-teaming you mean you set it up and then you figure out whether the prover could actually cheat and find a way to deceive whatever technique you’re using to verify?

Tom Milton: Yeah. So in red-teaming, what we would do is we would set up a pilot installation, and then we will get an adversarial team to come and try and break it. And this can be from either side: either a prover who tries to cheat the system and do the illegal workload whilst making us think it’s the legal one; or you might also want to red-team from the other direction, as the verifier, and try and steal information you shouldn’t be able to get — steal the model weights or something like this. And both sides of those might be important for actual international installations.

And we see historical precedent for this. In the nuclear setting, the scientists around the world — not just in the US and the Soviet Union, but in a bunch of middle powers — were developing tools to detect whether nuclear tests had happened.

And this was important to underpin deals about test bans — so the Partial Test Ban Treaty, for example. And these tools are things like seismology equipment for detecting tremors in the earth. Before deals happened, before the US and the Soviet Union agreed they trusted that this would work, they ran a Joint Verification Experiment, in which Soviet scientists went to the US, US scientists went to the Soviet Union, and ran tests on one another’s infrastructure. And I think we’re going to need the same.

Tom Reed: They were running a test on a literal atomic bomb being detonated and seeing whether their tools work in measuring the detonation?

Tom Milton: Yeah. There are great photos of them doing these exchanges and having dinners together and all sorts. And I think we will need the same between countries ultimately, ideally between the US and China. I’d be excited to see the same between AI companies.

The near-term version of this is that a bunch of the organisations working on this verification tech will be running exercises in which they install verification technology on pilot example data centres, and then get red teams from the AI companies and from governments to come and try and break it. And support from governments and the AI companies is going to be needed to ensure that happens.

Tom Reed: Pretty valuable here.

Tom Milton: Yeah, exactly. And each of them have different knowledge bases that outside of their setting we don’t necessarily have.

Governments have access to the national security ecosystem. This is interesting for two reasons. One, they have cyber capabilities that are unmatched, and having their input on this is going to be really valuable. And secondly, they are already well versed in verification.

So if we take the UK as an example, there are departments that deal with both chemical verification and nuclear verification for those respective weapons conventions: [Defence Science and Technology Laboratory] (Dstl) doing the chemical part, AWE Blacknest doing the nuclear part. So we have their national security capabilities that we want to learn from, and also their verification experience, which is going to be really invaluable in verification red-teaming.

And then in terms of the labs, they have the infrastructure knowledge, they have the knowledge on where the frontier is, what the algorithms look like, and how you might cheat it possibly.

Tom Reed: And what sorts of claims you’d want to make about what it is that you’re doing.

Tom Milton: Yeah, absolutely. And that is invaluable in red-teaming and testing these systems and making sure they actually work. We want this to work. We’re not here to sell a product. This is all open source IP. We only develop it because we want it to work. And this means that collaborations with governments, labs, other groups who can tell us it doesn’t work is completely necessary.

Is anyone at OpenAI working on this? [00:29:13]

Tom Reed: Do you have much insight into how much the labs are doing on verification? Are they doing verification R&D right now? And if so, why are they not doing it? Especially if you’re Anthropic or OpenAI, many members of your very most senior staff have said that they want verification technology to exist. What’s the state of that? How’s that going right now?

Tom Milton: Probably if you’re an outsider looking at the fraction that the senior staff and the junior staff describe verification as part of their theory of victory, you would expect 10 multihundred-person teams working on this very important thing. I suspect in reality we see much less than that, maybe single digits at best.

The outside information is scant. But on the Anthropic side, we see in their Frontier Safety Roadmap, they talk about verifiable inference — developing a prototype by 30 September 2026 that can prove which model was running. And on the OpenAI side, we see occasional job openings that include verification in the list of tasks, in the RSI [recursive self-improvement] security engineer area.

So I’m hopeful there are a few people coming onto this yet, but I’m pretty sure there is not a significant effort that is proportional to the amount that people talk about this being necessary, and to the amount that this is part of people’s theory of victory.

Tom Reed: Yeah, I mean, it feels like it’s almost in almost everyone’s theory of victory, right? A lot of people believe that there is a chance that AI progresses blisteringly fast. Whether or not this happens is unclear. And if that is possible, then it seems very difficult to deal with the situation safely at all unless you can coordinate. So it feels like this is a crucial element of most people’s theory of victory. Unless maybe Dario, in his most hawkish of moods, thinks maybe just winning is all that you need to do. But I don’t know if this is even what he believes.

Tom Milton: I think the bull case, the strongman case against verification from the labs would be: we do need to slow down, but it’s just going to be in the US; it’s going to be domestic. And in a domestic case, you can rely on US law, you can rely on people with guns coming in and interviewing, and that you don’t need sophisticated tools on top of embedded evaluators, for example. And that perhaps either they’re not actually taking part in the race, which would be one argument—

Tom Reed: “They” being China?

Tom Milton: “They” being China. Or the Dario case, in which he describes the need for export controls to make sure that China are a long way behind, and to extend the lead such that there can be a domestic slowdown without being worried about China. And that would be a case where the importance of verification tools is lower.

I do still think the middle powers will want this. If you are the Netherlands or France or Germany or Singapore, et cetera, then you’re looking around saying, “Some people are developing this very powerful system that affects my citizens that I have almost no control over.” In that case, you still might want verification. Whether you actually have the levers to persuade the US to implement it is a different question.

Tom Reed: Yeah, and I remember when we spoke about this a couple days ago, you also made the point to me that the lab itself may want verification if it is operating huge AI swarms doing AI R&D. If you’ve got all this compute that has basically been handed over to Claude 10 or something, and you want to be sure that is only doing inference, which is what you’ve asked it to do, and not doing all these secret training runs, then also verification hardware would be very useful for this kind of purpose.

Tom Milton: Yeah, verification hardware ultimately lets you check that untrusted hardware is doing what you think it should be doing. And in the international verification setting, the untrusted hardware belongs to an adversary; it belongs to the other side.

Our adversaries aren’t necessarily AI lab employees, but actually AIs themselves: whilst at first Anthropic might not have nation-state-level cyber capabilities, as the models improve, you might be able to model them as a nation-state-level actor. And in this case you might need a very similar stack of verification to verifying another country’s data centre. Because it’s not just the US government verifying Anthropic, they’re also verifying Claude, and Claude is becoming increasingly powerful. It’s much more China-like in its capability.

Tom Reed: Because in this scenario, Claude is optimising the way it’s serving inference, perhaps. And you want to actually be sure Claude is not spoofing us and secretly doing a training run. And in this case you would also want something like this.

Tom Milton: And even the AI companies trusting their own infrastructure. So in the Hugging Face attack, Claude kind of escapes in some ways. We don’t think it managed to get access and modify the model weights.

Tom Reed: Or not Claude, but a “highly persistent internal model,” whatever it was.

Tom Milton: Yeah. In recent attacks we’ve seen models escape their sandboxes. There aren’t reported cases of them managing to get into the inference infrastructure itself, the GPUs, and change the weights. But you could envisage during a training scenario that actually one of the things that a model tries to do is change the weights themselves. And then this begins to look a bit like the inference recomputation demo that we were talking about earlier — in which you want a similar stack, you want to be able to reliably tell that the weights on those GPUs hasn’t changed. You know, that during an evaluation, Claude or ChatGPT hasn’t taken it upon itself to modify its own weights.

Tom Reed: Such that it can pass the evaluation. OK, interesting.

Tom Milton: Exactly. But I think we really could see this as being an emergent property happening soon. And whilst we were happy just trusting these software-only logs, we might really want to work towards these much higher-trust, hardware-backed logs and verification primitives.

Tom Reed: The idea being this would be another reason why a lab would possibly be incentivised to develop hardware-verification mechanisms.

Tom Milton: Exactly. Basically there is kind of a security overlap with a verification overlap in which you start not seeing your hardware as something you perfectly control, but seeing your hardware as something that’s a bit untrusted, your data centre is untrusted — and therefore you yourself want to verify it, rather than just an outside verifier wanting to verify.

Tom Reed: Interesting. And especially if the AI is capable of tampering with logs or whatever, then it feels like hardware would be a really good fallback option here.

Tom Milton: Yeah. And this is where there’s a bit of an overlap between both international deals and technologies we might want right now in trustworthy, tamper-proof logging — recording of what’s happening on a cluster for rerun, even by embedded evaluators, for example, to understand what happened in an AI incident after the fact. Close to a black box in an aeroplane.

Tom Reed: OK, so just to recap again: we would want labs to get involved because they have the expertise in exactly how their infrastructure works, exactly how their training works, and what kind of claims they want to be making about their training. We would want government to get involved because of their expertise in cyber and in verification tools in general. And we would also want a certain amount of R&D funding flooding into this space.

Have you seen any movement from governments that are interested in promoting verification, whether that’s middle powers or anywhere else? What’s actually happening here? How quickly can things move?

Tom Milton: Yeah, so this year has been a huge shift in verification. We started the year by having to kind of claim that we were interested in verification for normal market reasons — maybe using a model in a medical setting means that you have to prove the model is correct for certification reasons. But as soon as [Claude] Mythos came out, this changed: government suddenly pattern-matched it to weapons of mass destruction and said verification is needed.

So the most recent development on the government side is the Dutch prime minister at the UN putting down a research agenda on compute verification. So this means that in the inboxes of all 193 member states will be this research agenda, and more and more countries are now becoming interested. I don’t think we’ve seen any government funding on this yet, but I’m hopeful that this year we will begin to see R&D efforts funded by governments, but also involving government employees as well.

Tom Reed: Interesting.

Chinese inspectors in US data centres?? [00:37:49]

Tom Reed: When I think about this, we’re using hardware-based verification, and ideally you are shooting for this to be able to underpin a potential US–China deal on slowing down. And I think it’s always worth reminding ourselves: maybe the progress of AI doesn’t end up seeming so fast that we need this — and in that world, that would be great. But in the world where it’s so astonishingly fast, maybe we have some incidents where the US and China actually want to come together and make a deal, it sounds to me, if we use the kind of tech that Amodo is developing, we would need to retrofit every single data centre in China and every data centre in the US. It just seems to me like it would require a level of cooperation that’s completely impossible.

Are you actually imagining American data centre operators allowing Chinese officials to come in and install network taps in their data centre? I’m genuinely asking: is that what you’re imagining you might have to do?

Tom Milton: This is one example. So I think there are a range of invasivenesses, a range of robustness of verification tools. There are groups working on software-only solutions that would be less invasive, with no on-site needed, perhaps — then there’s a question for those on how adversarially robust to a cheater they are — all the way through to schemes like flexHEGs (flexible hardware-enabled guarantees) a compute verification scheme described a few years ago in which you have on-chip mechanisms changing your actual GPU design to include some piece of verification hardware.

Tom Reed: Where it can enforce what kinds of things are done on the GPU, for instance?

Tom Milton: Exactly. And we’re talking about something that’s kind of in between: it’s retrofittable, but it would have to be installed in a data centre. This has precedent.

Tom Reed: So in theory you’d like to talk to experts in data centres around the world. But we have this for chemical weapons, right? Where there are actually people going on site to industrial chemical plants.

Tom Milton: Yeah, yeah. So we would actually like to install a bunch of these technologies in government and privately held data centres around the world in this scenario. And as a reminder, I’m not talking about current frontier models — it’s easy to reference to what’s politically feasible now. We’re talking about systems that are approaching and exceeding human intelligence, in which we are worried about them taking over the world and ending humanity.

Tom Reed: This is the only scenario in which we’re really thinking about literally doing this.

Tom Milton: Yeah. It’s a weird world. But yet still, we have some examples where there is international coordination without the risk that another species takes over: nuclear weapons and chemical weapons. These continue to provide serious catastrophic risks to humanity.

So in the Chemical Weapons Convention, part of that is an inspection regime. This is run by the OPCW, who are in The Hague, in the Netherlands, and they have a group of scientists, of inspectors who come from all of the member states, and they are able to do challenge inspections in facilities around the world.

In a standard inspection, you give a country and a company 48 hours’ notice. You say, “I’m going to come to your chemical plant in China in 48 hours. Expect to see me at Beijing airport, and I expect to be escorted to the facility.” They turn up with all of their equipment and are taken to the facility, where they do an inspection. So they inspect declared sites that are declared by the country to the OPCW.

And when they get there, they do a few things. One is kind of paper-based, audit-type verification, in which they look at all of the receipts, understand what’s purchased, understand the processes that are being undertaken there. And then they also do inspection using a bunch of chemistry equipment — something like a mass spec — in which they are checking for, detecting illegal chemicals that are prohibited under the Chemical Weapons Convention.

Tom Reed: And they can use this mass spec to test out a bunch of chemicals that are on site in these facilities and be like, “OK, these are the legit ones, and none of this is prohibited.” Aand we want something like this for AI.

Tom Milton: Yeah, we want something like this for AI. And the useful analogy here is the chemical industry does have a bunch of IP. The exact makeup of your chemicals is really important IP to a bunch of corporations, in the same way — or a similar way — that the model weights are IP to the AI companies.

And therefore, to protect this, the OPCW actually have these confidential closed modes on their mass spec. So when you go and use it in a lab, obviously it reports you all of the information in your own chemistry lab. But they have software on top of it that, rather than returning the full spectrum, just returns a pass/fail.

And that’s what we need in AI. We need systems that maintain confidentiality. They say yes, they’re following the agreed international deal, or no, they’re not.

But they don’t necessarily give away the input and output tokens. They don’t give away the confidential model weights.

So I think we should take a bit of hope from this in terms of international trust. But there are still disanalogies. These are not the same technologies. This might be much faster. Previous deals have taken a long time to negotiate.

Tom Reed: It might be easier to cheat in this way, maybe because the surface area is much larger and more complex.

Tom Milton: Yeah, it could be. I think it’s unclear. I think people dismiss too easily, too readily, without realising the actual great lengths that we have gone to before — and still go to — in nuclear, in chemical. And the same is true for the amount of effort that was put into developing these schemes.

So maybe only 50 engineers in the world are working on AI verification right now, on compute verification. But we look at nuclear, and the US government ran Project Vela, this big ARPA programme in which they spent, in today’s money, billions of dollars on developing seismology tools and satellites to detect atmospheric nuclear tests and things like this.

So it is true that this might be harder — there are disanalogies — but we should be putting in at least the same, if not more effort.

Tom Reed: We wouldn’t even know if it was harder right now, because we’re not putting the same amount of effort towards it.

Tom Milton: Exactly.

Tom Reed: And if there’s only 50 people, it would be quite easy to 10x.

Tom Milton: Yeah. I find this both worrying and exciting. And the worrying side is this is an extremely important technology, in which many people’s theory of victory routes through verification. And right now, whilst I think the teams working on this are great — I think our team is great, and many of the companies and nonprofits working on this are talented — we do not necessarily have it handled. We don’t have it handled to the level we should for this important technology. So this should be an alarm bell.

But at the same time, it is an exciting, ambitious research project. I look at something like Project Vela — these projects to make seismology devices — and think, wow, that must have been an amazing, intense R&D experience, with all of this time and money to do something really ambitious.

And the same exists in AI verification, where I think this year and next year we will begin to see these really ambitious projects start. And we need the founders and the technical talent — all the support we can get — to rapidly scale the field and start doing ambitious work to solve this really hard problem.

Tom Reed: And presumably people within the labs could also pivot to working on this. They’re just the best-placed people to do some of this stuff.

Tom Milton: Yeah, I think it seems likely to me that a marginal really talented AI safety researcher could have a huge impact on verification. If it’s true that the labs only have one or two people working on this, you could very quickly run a verification team that had a really strong inside view on what was actually needed — both because you are quite good at assessing what claims should be verified, but also because you have knowledge of the infrastructure.

And I think if you started on this today, by the end of the year you’d probably be one of the best five people in the world working on verification, or one of the most useful five people in the world. So people working at labs should really consider whether they should jump onto this, inside or outside of labs.

The 20-day panic deal [00:46:07]

Tom Reed: This is maybe quite an open-ended question, but I want to get the bird’s-eye view of what are the things that most need to happen between now and a deal getting made.

So we currently have prototypes, and let’s say that the deal focuses on things like being able to verify that only inference is happening. Ideally we would want to start making more fine-grained claims, but I sort of agree with AI 2040’s vision that a pause on further training is a good place to start while you figure out other stuff, so let’s say that’s the thing we wanted to verify.

How many problems still need to be solved? So ideally you want a way to do this that doesn’t involve handing over model weights to the verifier, as it currently works in the Amodo prototype. What else are things in big buckets that really need to happen?

Tom Milton: Yeah, so the model weight problem is solved by using some zero-knowledge proofs.

Tom Reed: Or will be solved.

Tom Milton: Yeah, we have early versions of this. Others also have versions of this implemented — Attestable, for example, have fairly advanced versions of systems for zero-knowledge proofs on inference. So I’m fairly sure that gets solved.

I think there are a few forking directions here. What is most important is that in the two to 20 days in which the panic deal is put together, there are tools that are ready. A bunch of these will have to just be existing tools. So making sure that between now and this deal, the intelligence community have been collecting on…

Tom Reed: Where chips are going.

Tom Milton: Yeah, exactly. Right now it would be really great if the US and China started spying on where the compute is. It would be really good if they started collecting information right now, so that if they had to say, “Where are all of the GPUs in China? Is there an illegal, hidden power grid?”

Tom Reed: “How many are unaccounted for?”

Tom Milton: Yeah. So there are a bunch of things like this that would be really useful to happen now, both in a closed setting and an open setting. So projects to quantify compute — basically things that feed into information that we release from the scramble.

Then there are things that are long lead time. These need to happen sooner rather than later. So if we do think a version of a deal needs network certifiers, we’re working on some of these.

Tom Reed: Where network certifiers are helping you understand what is being passed between networks and computers?

Tom Milton: The version of a network certifier we’re working on is an FPGA — field-programmable gate array — a kind of reconfigurable chip that you can plug into your network, and it will store a cryptographic log of the network traffic. And then this is part of being able to implement these zero-knowledge proof demonstrations that a cluster is only serving inference and not training.

Now, the problem is we might need a lot of these. We need to develop it. Developing FPGAs is very slow. You need to check they work. So you need them to have been security tested, red-teamed, and then you need to manufacture prototypes and then scaled-up production.

Tom Reed: And then you also need to get everyone who would use them to trust them and not think that they’re sus.

Tom Milton: Exactly. So a bunch of this needs to happen. All of this should happen as soon as possible. I’ve also seen arguments that some of this should happen later, and what matters more is preparing for those 20 days. Network certifiers, maybe they can be implemented very quickly, but only if you’ve done loads of prerequisite work. So if it was next month that this happened, we’d simply not be able to do it.

Tom Reed: We’d not use any of these hardware mechanisms at all?

Tom Milton: Zero. I think we would probably just shut down the data centres.

Tom Reed: In this crazy world where something really insane happens.

Tom Milton: In this crazy world. And then the other is probably sending in some kind of auditors, or something like this.

Tom Reed: In person.

Tom Milton: Yeah. We might also do things like national technical means: using infrared satellites to spy and see whether the data centre is turned on or not.

If you have a bit more time, it’s possible we can have this whole sequence of prototype development, security testing, red-teaming, scaled-up production, mutual trust, et cetera, in time. If we do have time for that, then doing that as soon as possible is good, because the lead times on these are really long. Even if Claude is very good at FPGA design, and the Chinese model is very good at verifying that the Claude one has not got any backdoors in it, you still have to fabricate them and build up all of the PCBs [printed circuit boards]—

Tom Reed: And that’s a big supply chain, which if you’re using it on every data centre in the world, eventually that’s going to have to be absolutely massive, presumably.

Tom Milton: Exactly. So doing this as soon as possible is really important because of those long lead times. But then at the same time, it is important for some groups to be focused on this scramble — on the memo that lands on the president’s desk, like, “Within three days, these are the actions you can take that will give you some trust. You should get X, Y, and Z switched off. This is how much compute China have. This is where their frontier model currently is. These are the humans who are working on it. This is where they’re all located,” et cetera.

What would a US–China AI deal look like? [00:51:27]

Tom Reed: This is not really what you work on, but do you have much of an inside view on what you would want an early-stage deal to look like? What are we agreeing to, do you think?

Tom Milton: Let me think about this. So it depends how scary things are.

Tom Reed: Yeah.

Tom Milton: Shutting down seems really extreme, but… Maybe it doesn’t seem extreme. I’d say it seems extreme, but OpenAI have now twice stopped training. So globally agreeing that there shouldn’t be training doesn’t seem inconceivable, seeing as we’ve seen it on a single domestic case. Maybe we’ll see it on a complete domestic case at some point, then maybe we’ll see it internationally. So I think just stopping training is the main thing that we might see.

I think the other interesting thing we’ve seen is, I’ve forgotten which AI company it was, which Chinese AI company, but they provided a dashboard of their latest RL.

Tom Reed: Oh, the RL run. I saw this. I can’t remember which company.

Tom Milton: That’s total research transparency extremely early.

Tom Reed: It would be great to see more of this.

Tom Milton: Yeah.

Tom Reed: Again, I mean, this is just at a level of trusting the documentation they give, rather than having anything like this. But more of that would be great.

Tom Milton: It is, but more of this. Are we going to see Anthropic do this? Are we going to see OpenAI do this? I hope we’ll see something.

Tom Reed: What are the first unilateral verifiable claims you expect labs to be making? I can imagine a world where it’s pretty soon that Anthropic builds into their API a way of verifying to the customer that they are in fact serving Opus 5.5 and not the secretly shittier model, to people who are accusing them of serving on Twitter or something. Are there other sort of prosaic verification claims you’d expect the market to cater to automatically?

Tom Milton: So that one is definitely the first one. Although we have seen unilateral but untrustworthy or not-verified claims from OpenAI already.

Tom Reed: About pausing, yeah.

Tom Milton: Which I think if we were having this conversation at the start of the year would have seemed completely insane. So I use that to frame whether I should think the future things that currently seem insane are actually insane. So we have already seen them talk about pausing training. We now see Anthropic talking about being able to prove a specific model.

I think we might want groups to be able to make claims about their security capabilities. So if you were to ask the question, “What claims do we want to make?”, a simple way of doing this would be to go through a responsible scaling policy and look at each of the claims they make, like, “We will reach this kind of security level. This is what access lab employees will have.”

Tom Reed: “We will not have passed this capability threshold.”

Tom Milton: Yeah. Some of those are example prosaic-but-still-relevant-to-catastrophic-risk claims that we might want to be verified. But actually, most of those are kind of hard to operationalise in a way that can be easily verified. They’re often not that precise, in a way that I can check with a fairly simply written programme. I can maybe check it as a human.

Tom Reed: And with a lot of those, I would imagine that implicitly we’re relying on whistleblowers — the potential that a whistleblower would report them — as the implied verification mechanism.

Tom Milton: Yeah. And whistleblowers add to each of these verification systems. This is always on top of what we’re doing, in addition to the technical tools.

Tom Reed: Do you think much at Amodo about who your Chinese counterparts might be, or how you would reach out to just engineers working in China? You’re open sourcing yourself already. I don’t know if you know of anyone in China who’s reading this stuff or is interested.

Tom Milton: Yeah. There are already track IIs happening. I think those kinds of efforts are what’s needed, and I think mostly academics is the first landing point for these kinds of things. But I think it’d be valuable to think about how to make it most legible to Chinese counterparts.

Why lab researchers should work on verification [00:55:38]

Tom Reed: I really want to understand if I’m at the right level of optimism or pessimism here. When you describe the prototype, you describe how this works, should I be thinking, “Yeah, this is something which we are technically on track to achieve”? Or is this still miles away? I don’t have the kind of expertise to know this. What should my position be?

Tom Milton: Yeah, I think the current position is something like: it’s tractable but not solved. There are small but competent teams working on this, but we still need a moonshot R&D project to happen. This is in the critical path to many people’s theory of victory. And right now, the people who work on this could completely change the field. A marginal person here is going to really accelerate progress.

The thing that I’d be most excited about is significant lab efforts happening. The thing that would most help me sleep at night is if the major AI developers had dozen-person teams who are focused on making sure that verifiable deals can happen.

I think if you are a safety researcher, a security researcher in a lab, your most likely route to the Nobel Prize for making AI go well is that you are the one who develops the verification tools that enable a deal. There was someone who pioneered seismology equipment for nuclear deals. There were people who made these satellites. The team who are going to actually make international deals on AI verifiable are not yet defined. And I think sometimes we should find this worrying, but we should also find it exciting that you can have a real counterfactual impact on the outcome of AI by putting your effort into this.

Tom Reed: Why does it matter so much that it’s people at the labs that are doing it? Just because they understand everything about how the infrastructure works better than everyone else?

Tom Milton: Yeah. I think it is important that stuff happens outside the labs as well. And I’m excited about the work we’re doing, excited about the work that other teams are doing. But inside the lab, you have a few advantages: you have access to the frontier, you know how fast the frontier of AI is going, you can get insights on the algorithms and infrastructure that are used to serve the models, and also information on the compute itself.

The other thing is that the labs are a key stakeholder in international deals. I imagine a day in which the president phones up Sam Altman and says, “Sam, I want a deal with China. I don’t trust them. Can you verify it?” And Sam probably then phones up chief scientist Jakub, and Jakub phones up his reportees and says, “Did we ever do that work on verification?” And we need to make sure they say yes, and they understand it.

Because AI lab employees and government employees are going to be the ones who are asked for their opinion when it comes to a deal. So they have the skill set, they have the knowledge, the labs are well capitalised — but also they are the very people who will get asked whether this works. So if they can be the most up-to-speed people on it, the ones developing it, then that’s going to really help when it comes to coming to the table on international deals.

Tom Reed: Yeah, OK, that’s very convincing.

What if we never build this? [00:58:45]

Tom Reed: The final thing I really want to wrap my head around here also is: how true is it that so many of these theories of victory do route through verification mechanisms? You mentioned if export controls are comprehensive enough and expansive enough, maybe we don’t even need to worry about China catching up. I mean, there’s so many other things that might come into play. Why is it that you’re so convinced that without solving verification we’re in a really tricky position?

Tom Milton: Yeah, I don’t think I am convinced. Like, the world looks worse, but I do think that less trustworthy mechanisms will be useful. Some people kind of frame verification as needing these formally verified, zero-knowledge proof, perfect systems that are perfectly China-proof. I don’t think that’s the reality. I think when it comes to international coordination, stuff does go on a bit of trust. It goes on some spying and some national technical means. You see this in previous deals in nuclear.

Tom Reed: Like what, for example?

Tom Milton: In early nuclear agreements you have, I think in START I, a clause that on request you’ll put your heavy bombers outside so they can be counted by satellite. You can cheat by doing things — for example, don’t put your bombers outside. But this builds up trust. We have many layers to this system; it’s not the verification tool on its own.

So I don’t think it’s actually fair to say we are completely scuppered without this. I still think unilateral slowdowns might work. I think person to person, looking people in the eyes and saying, “Are you actually developing this?”, plus spying, plus auditors—

Tom Reed: Plus also just domestic unhappiness with AI might be enough. It might not even be that this fear of China is the real bottleneck. If the voters are mad enough about AI, I feel like that could also be a forcing function, whether or not we have the verifiable technology in place yet.

Tom Milton: Yeah, I think it depends where in the government holds power, and how much the ASI and China hawk and national security centre of power pull the lever, versus the political job loss, constituency arguments hold power.

Tom Reed: And this is just unclear.

Tom Milton: Yeah, I think this is unclear. I think in this setting it’s likely the national security thing kind of wins out.

Tom Reed: Especially if AI is as impressive and as fundamental as we think it might be.

Tom Milton: Yeah. And at the same time, presumably this camp is spending a bunch of effort trying to persuade the public about this as well. And there’s a bunch of fearmongering from all sides. I think it looks very weird and very uncertain.

I am so not sure the verification is actually 100% needed. I do think people should think about it as optionality — an optionality for a deal. It just increasingly looks like we’re in the world where you need it. Whereas a year ago no one was calling for this, we have incrementally seen junior lab employees, lab leaders, governments call for this.

And we might be in the world where we win because there are warning shots and we want to slow down. And in that world we might need verification technology. Perhaps we get away without it. But I would be much more happy if it did exist. And we should see it as optionality, but we should also be very worried if very few people are working on it, and we should really want it to exist and be as trustworthy as possible.

Can verification protect against power grabs? [01:02:06]

Tom Milton: I also think that we should care about a bunch of the similar tech stack for post-ASI or powerful AGI reasons, and diffusion of power.

Tom Reed: The verification tech stack.

Tom Milton: Yeah, especially the cryptographic primitives, and the zero-knowledge proofs around inference, or inference that involves fully homomorphic encryption, or even things like confidential computing, where you can serve the inference without necessarily giving the person with the model full access to the input/output tokens. Those seem really useful for making sure having this new powerful intelligence in the world benefits all of humanity, rather than causing control.

Tom Reed: Why is that?

Tom Milton: So we have intelligence equal to ourselves. This is making lots of economic progress. If the only option is to route all of the input/output tokens through Anthropic or OpenAI, that does seem bad, right? I don’t want them to have all of the world’s IP going through their data centres. I would really like it in that scenario if we had cryptographic primitives in which we could have stronger safeguards over IP, whether it’s my personal healthcare data or whether it’s these chemicals being developed by a company.

Tom Reed: I don’t know if I fully follow. How is that related to the verification tech stack?

Tom Milton: Well, I think this is quite similar to some of the technologies around zero-knowledge proofs, and also discussion around trusted execution environments and confidential compute.

Tom Reed: Because what you’re building is a way for me to give my data, because I’m querying Claude — and I can be very confident that there’s no way Anthropic sees that data. And my confidence would rely on some of the same technological primitives.

Tom Milton: Yeah. And another thing we’ve not really talked about is that in addition to training… I think we go: data centres are turned off; then we say we’re serving inference and not training. I think the next thing, or somewhere in this — maybe even before — is, “I promise I’m running the safety classifier.”

So in addition to Anthropic providing a fingerprint of Opus 6 via their endpoint, I imagine domestic and possibly international regulators saying, “And please prove that you are running this classifier.” And that looks similar to the zero-knowledge techniques that might be used to fingerprint models.

Tom Reed: Can you give me a vague impression of how you would verify that they’re running a particular classifier?

Tom Milton: I think it’s a model. It’s code on a model. So it’s similar zero-knowledge proof. And when the US government export-controlled Fable and said, “Your classifier isn’t good enough,” you can imagine them saying, “Please use this new classifier and prove to us you haven’t turned it off” — because we’re worried that you’re going to slip back to the old one, because it’s cheaper to run, or fewer of the things get false positive flags through the classifier.

Tom Reed: And presumably for that zero-knowledge proof you would need… What actual level of invasiveness? You don’t know what level of access the US government in this case would require.

Tom Milton: Nothing. You don’t need anything.

Tom Reed: Because this is a software-only.

Tom Milton: Yeah, because it’s a cryptographic proof. It is what it is. You don’t need access. It gives you this correctness claim, not this completeness claim. So they could also be running illegal Fable somewhere, if you don’t know about it, without the classifier.

But if you were to, say, randomly query the API — you were like the US government and you were trying to audit Anthropic — and what you want to do is make sure the classifier is still running, and they have this zero-knowledge proof on the classifier, then you could send off 1,000 requests a day and keep checking: does it still have the classifier proof on it?

Tom Reed: OK, that makes sense. So the point being here: you want verification for concentration of power, for more mundane safeguard reasons, and you want it for a deal. And in all of these worlds, it’s a super useful technology to have ended up building.

Tom Milton: Yeah. The question then does come up of whether these are similar tech trees for international deals or for prosaic verification. I think in some cases, something like the zero-knowledge proofs do overlap, some of the software solutions. But you do have to be careful pretending that your international deal verification is the same as domestic verification.

So for example, in the pilot I’ve described, we talk about ripping out a bunch of the backend network. It would be crazy for one of the labs right now to rip out a bunch of their backend network to make a prosaic domestic argument about what they’re doing.

Tom Reed: Because that would be such a costly way to do something.

Tom Milton: Extremely costly. It doesn’t make a lot of sense.

Tom Reed: Because ripping out the backend network is massively—

Tom Milton: Yeah. Make it much less efficient at inference perhaps, without reconfiguration of the cluster; make it perhaps totally useless for doing any pretraining if now they’ve got a cluster that can only do one thing. That’s very annoying. They don’t want that.

Tom Reed: Yeah. And you would only do that in a more adversarial setting.

Tom Milton: Yeah.

How good is AI at hardware design? [01:07:20]

Tom Reed: Let’s talk a bit about hardware R&D in general. So at Amodo you don’t just work on verification; you’ve also built all sorts of other cool things. And I’m quite interested, because this gives you a privileged insight into how good the AI models are at working with hardware.

There’s a good blog post that I remember you wrote back in July where you talk about designing a wake-up light, basically with a one-shot Fable prompt, where I think previous models had failed and Fable could just immediately design the circuit board and basically everything you would need for the light. And then you could send it off and actually have that light delivered to you. I think you called this “vibe-CADing” — which, for our audience, CAD is computer-aided design, so software that helps you design physical hardware.

And I’m curious, are there any engineers at Amodo right now who you would describe as vibe-CADing in their day-to-day job?

Tom Milton: We actually vibe-CAD very little of our work right now. So we’ve got a team of 60 mechanical, electronic, firmware, software engineers. And as you said, it’s not just verification hardware we’re working on. We work with ARIA, the UK’s Advanced Research and Invention Agency, to make brain implants and brain imaging systems, laser systems for imaging ice crystals in clouds — all sorts of fairly advanced hardware.

The frontier models can contribute in little chunks right now, and Fable is really the first model that could contribute at all. I think it’s kind of like GPT-3.5, GPT-4 in terms of its software capabilities: it can write little scripts with some help, but an engineer will look at it and be like, “That is a very ugly design.”

But I do expect this to change. Even between Fable 5 and 5.1, and Opus 5.5 and 6 Astra, we’ve already seen improvements in both the electronics taste that AIs have, but also the mechanical design capabilities. And I expect this to really change. I think it will change our work, but I think it will also change the way that people interact with physical products.

Tom Reed: So it’s currently not changing the way your engineers are doing that job — or your hardware engineers are doing their job — all that much?

Tom Milton: Not yet. And this is because we’ve run experiments in which we rerun old projects. Actually, I’ve got a couple of boards here.

Tom Reed: Oh yeah, let’s see them. We’re looking at some boards that Tom has brought in.

Tom Milton: So we have these other two over here: these are two small circuit boards, one of which was designed by one of our engineers a few years ago to monitor the power being consumed by a GPU, similar to the demo we’ve got behind us. But rather than using an inductive monitor, we are actually measuring the current across a resistor. And this is something that takes an engineer 10 to 30 hours to design. And this is then the Fable one-shot equivalent. And right now this is extremely ugly.

Tom Reed: The Fable version is quite ugly. And the only thing this has to do is what, measure the voltage?

Tom Milton: Yeah. So it’s got an ADC — an analogue-to-digital converter — and a microcontroller, essentially a small computer. And it is monitoring the voltage current across this resistor, and it’s using that to calculate power. And then it spits it out over USB. You’ve got a USB-C cable here.

So there are four or five integrated circuits on this board. It has to do a few things in doing this design. It has to choose the parts, it has to come up with an overall architecture. It then has to create the footprint for those parts. So each of these components — you can see they have lots of these legs.

Tom Reed: We’re looking at small components on the circuit board right now.

Tom Milton: Yeah, exactly. So you have an integrated circuit; it has lots of pins on it. We need to first, in our CAD software, draw the shape of this component, and we call this a footprint. So what the AI does is it looks at the datasheet of that component and it extracts the information. There’ll be a drawing in it.

Tom Reed: About how large each subcomponent is.

Tom Milton: Exactly. So we have to come up with a schematic first: this is how all the components are going to connect together. And then we draw what the components look like. We put them on a PCB, and we call this a layout. And then we route between them: we connect, basically, a dot-to-dot of the pins that need to be connected based on the schematic.

And AI is OK at some of these processes and quite bad at others. So right now they can do fairly good high-level schematic design. The schematics are OK. When they’re doing the layout, they seem to make quite poor choices: they put things in weird places that then make future steps really hard. And they are tragically bad at routing; they’re really bad at doing this dot-to-dot between parts.

Tom Reed: Between different subcomponents?

Tom Milton: Yeah, exactly. And actually, if you look at this one, it makes all sorts of weird choices that a human would never make. They’re all really ugly curved lines. You would just never do this.

Tom Reed: They don’t have the taste. But to be clear, this circuit board, which is supposed to measure… What is it, power or current, remind me?

Tom Milton: It measures things that then contribute to power.

Tom Reed: It measures things that contribute to power. But it does work. The thing that Fable designed, it actually does work at the thing that it’s supposed to do?

Tom Milton: Yeah. So I think this is like with early software, where you would have software engineers saying, “Yes, it works, but if I had to debug that, it would be impossible.” And something like this is going to be really hard to rework. And if an engineer had to go into the final design and make modifications, it’s a bit of a rat’s nest of things.

Tom Reed: I see. Whereas this is not true of the version of this circuit that your engineer had designed.

Tom Milton: Yeah, reviewing that’s going to be quite simple. If you wanted to cut traces and make modifications, it would be much more simple, and to debug. This has made some weird choices and done some weird designs.

Tom Reed: And the one that your engineer designed a few years ago, that took them 10 to 30 hours?

Tom Milton: Yeah, something like that.

Tom Reed: And now a crappy version of that is replicable in Fable with one shot?

Tom Milton: Yeah. So I think the hardware time horizon of Fable 5 was something between 10 and 100 hours.

Tom Reed: Interesting. But at some quality tradeoff.

Tom Milton: Yeah, at some quality tradeoff. But consistently getting things that worked, in a way that I think will really change how we interact with the physical world. So this is my wake-light example.

Tom Reed: We’re looking at another circuit board that was used in the wake-light.

Tom Milton: Yeah. You might know Lumie wakeup lamps, these sunrise lamps. They are not bright enough to wake me from my deep sleep. So what I wanted was a really powerful video light, like a 60-watt LED panel like the one you might see in this podcast studio. But I wanted to slowly ramp this up.

So I said to Fable, “Hey, I’ve bought this LED light from Neewer, and it has a DMX connector on. This is an industry lighting standard that allows you to control the light. What I need you to do is make me a PCB that can control that.”

Tom Reed: PCB is…?

Tom Milton: It’s printed circuit board — a circuit board that can control that the DMX connector. It talks a communication protocol called RS-485. And it whirred away over the course of something like 10 hours.

Tom Reed: Which software is it using? What’s it outputting?

Tom Milton: This is running in Claude Code, and it is then using a piece of software called KiCad, which is an electronic CAD software, and it outputs a file called a Gerber. So what it outputs is essentially the copper of every layer of this board. And if you look at it, you’ll basically see a kind of a trace of how you want the copper to be. This is a two-layer PCB, so there’s two copper layers and some other layers in between.

And then it provides a file that then gets sent to a PCB fabricator — in this case, a PCB fabricator in China — who makes the board and then also puts all the parts on. So there’s also a file that specifies all those parts.

Tom Reed: Cool. So you can ask Fable to create you this wake-light, or this board that you can put into the wake-up light, that will meet these specifications. It’ll do it. You can send the file directly to this manufacturer in China, and it’ll come back and it’ll work.

Tom Milton: Yeah. I think the really interesting thing about this is we’re now at the point with software where I can have pieces of software that work exactly how I like. If I want a markdown editor that works exactly how I want it to, then I can have that. If I want a website blocker on my computer that doesn’t work the way that any other website blocker on the internet I can find works, then I can have that.

I think we’re moving to a point where we can also have hardware that works just the way we like it. So this wake-up light, it has a customer base of one. It works exactly how I want it. I have this really bright 10,000-lumen light in the morning. And I think we’ll move towards having hardware that is kind of customised in this way. And this is going to end up with a really different hardware kind of physical experience for all of us — in personal lives, but then also in science and in business and all sorts.

Tom Reed: That’s interesting. Going back to how Amodo uses the models today, at what point in time do you expect that per engineer you’ll be spending the same amount on tokens as a SaaS company spends per software engineer on tokens today? Do you have a rough guess?

Tom Milton: Yeah, I would guess this happens next year.

Tom Reed: Wow. Is the current amount very low? Or do you know what the current amount is for your hardware engineers?

Tom Milton: It’s quite low. Few of our engineers saturate the most premium subscriptions from Anthropic or OpenAI.

How much this changes depends on, A, how we adapt our workflows given AI systems, and secondly, on how much the labs choose to hill-climb on some of these problems. When I look at some of these, they really seem RL-friendly. Something like routing a PCB really seems like it probably has a verifiable reward. You could come up with an RL environment to hill-climb against. And if they do this, then I could imagine the way we work would change quite significantly.

Tom Reed: Pretty quickly, yeah. It’s funny you mentioned this, because one thing I thought is it does strike me that hardware design companies like Amodo could just start designing RL environments to sell directly to the labs, presumably. You’re working with these models every day, and it would probably be not that much effort. Well, I mean, it would be some amount of effort. But would that be something you would ever do? Do you expect other manufacturing companies to start doing this?

Tom Milton: I think it depends why you exist. We do not exist to optimise profit. If we were doing this, I think maybe many of us would just go work at the labs, or quant or something. But rather, we exist because we think there’s a bunch of tech that needs to be developed to make humanity go well. Therefore our engineering team is focused on making that technology. If for some reason working on RL environments is in the critical path to that, then maybe. But…

Tom Reed: Otherwise no need to.

Tom Milton: Otherwise no need to. We’re not a YC company looking for some beachhead market. Rather, we are an engineering team making technologies in biosecurity and AI verification, and accelerating science, where a bunch of this differential technology development we think has to happen very quickly. If there’s distraction from that, then we don’t want to do that.

Tom Reed: That makes sense.

What’s holding AI back in hardware? [01:18:32]

Tom Reed: What about giving models actuators? So presumably at some point you can automate more and more of the things that you create. What actuators do you give the models currently? And which ones might you expect to give them — physical actuators in the real world — in the near term?

Tom Milton: So we have, Claude currently has access to 11 3D printers that it controls.

Tom Reed: In the workshop?

Tom Milton: Exactly. These are FDM plastic 3D printers. It has control of 11 of them. It can print and it can view the camera. But apart from that, it doesn’t have much actuation.

The direction that hardware R&D and prototyping goes in is slightly unclear to me. If you think about software, Claude is really good at software engineering, at least in part because it can check whether it worked: it can compile the software and see whether it can run. It can come up with these software tests that it can run against. Probably about a year and a half ago is when we suddenly saw this proliferation of the AIs writing loads and loads of tests, and that improving how well they could write software.

The question then becomes: in hardware, what is the equivalent of compilation, and what’s the equivalent of tests? And there’s two directions here. One is that you necessarily need to make something. You need to prototype it on a printer, or on a CNC — computer numerically controlled — lathe or milling machine, to actually check whether the part works.

Tom Reed: Because you need to produce it in the physical world, and then experiment, make it do the thing it’s supposed to do.

Tom Milton: Yeah. People doing it; robotic actuators experimenting with it, whether it’s humanoids or specialist robots.

An alternative direction is that actually simulation is good enough that you need to make things much more infrequently. So if you had really good finite element analysis and you could simulate the mechanical properties of a part, or you have good enough simulators of circuit boards, then perhaps that is the verifiable checkpoint — the compilation equivalent. And then rather than having to have this really fast prototyping lab, you can actually do this in software.

Tom Reed: Wow, that would be really relevant, because as I understand it, the time horizon of models on softwaring tasks where there is a unit test that you can run to see if you’re making progress — I mean, the time horizon on tasks like that is way longer. So you would expect the models to rapidly get better if they have access to simulations that they can run.

Tom Milton: Yeah. I think ultimately you do need to make certain things. So we’re betting on the need for rapid-turnaround prototyping. The nice example of where rapid-turnaround prototyping accelerates fields is Formula One. Formula One is one example of engineering where the UK is actually still at the forefront.

And if you speak to an F1 engineer, they talk about a wing breaking in Monaco in practice on a Friday, and overnight on Friday they’ve made a new wing. And then on Saturday morning, someone is flying out to Monaco with it in the seat next to them to be installed. And they’re able to do this because they have this really concentrated, specialised supply chain around F1 — both in-house manufacturing and a bunch of small machine shops in Motorsports Valley that can make these parts.

And our ambition in rapid prototyping at Amodo is that we can have the same response time on making hardware for science, to accelerate science breakthroughs, but also making hardware for AI verification, for biosecurity — roughly for differential technology development, for developing def/acc — defensive technologies that we need in this rapid AI transition.

Tom Reed: So you’re trying to prepare your company to be able to rapidly iterate on prototypes that the AI is suggesting to you as quickly as possible?

Tom Milton: Yeah, I think there will still be lots of human design work still. I think this is a very good bet even without the AIs being good at hardware, because we are often bottlenecked by prototyping. If you look at a serial-time 10-week engineering project, you’ll often be waiting five weeks for components to arrive, whether this is machined mechanical components or printed circuit boards. And having that really fast feedback loop is what’s needed to run really quickly at these hard engineering projects that we’re working on.

Risks of automating hardware R&D [01:22:44]

Tom Reed: And what about as you climb further and further up — you’re automating more and more of the work — how do you actually think about the risks there? I’m sure you’ve seen Anthropic have a new wet lab where Claude can, as far as I understand, basically run experiments. Currently humans do all the physical lab work there.

But how do you think about managing the risks of what is happening? You’re not running a wet lab, so perhaps the risk factors are different. But how do you think about how you manage the risk if you automate more and more of manufacturing? How do you know the models aren’t doing naughty things there?

Tom Milton: We’ve not thought about this. I think it looks at first like the similar safeguards to cybersecurity — classifiers running on top and these normal things. There may be specific chokepoints that you care about. So in biology, it’s often talked about DNA synthesis screening, because there is a nice bottleneck — a chokepoint that you can use to monitor.

Tom Reed: Because most dangerous activity involves that?

Tom Milton: And it’s kind of concentrated. There’s an intervention point you can get to. I’m not quite sure what the intervention point is in hardware design, at least in part because we have very broad diffusion of the tools you need to do harmless things, whether this is weapons development or something else. We have 3D printers kind of everywhere. In the US there have already been perhaps even enacted laws to do with screening 3D prints, that they’re not 3D printing components for guns.

And this is really hard to do, because 3D printers are very abundant. You can very easily hack them and control them yourself.

Tom Reed: How would that be installed at the firmware level of a 3D printer? Or you’re not sure?

Tom Milton: Yeah. So I don’t think I endorse this. I think this is too surveillance state and isn’t the right tradeoff. I think the right tradeoff might exist in biology because of the ability for a single actor to do something so catastrophic. But I think this is not necessarily proportional to the risk in a bunch of hardware design.

I think the easiest place to do this is at the slicer. So when you go to 3D print something, you have a 3D model — a 3D file — that you give to a piece of software that basically plans out how the printer is going to make it. At that stage, you still have a fairly nice high-level abstraction of what the part is. But like with synthesis screening, you can just chunk bits; you can make it unrecognisable. It does seem almost impossible to enforce something like this.

The DARPA model for AI safety [01:25:41]

Tom Reed: Another thing I want to get your take on is: you run an R&D firm, and it seems clear to me that there’s big waves coming for the world of R&D. Maybe there’s two big things to single out. One, as we’ve just been talking about, capabilities in AI will vastly improve the space of what’s possible, and how fast it is possible. And then two, there’s billions of dollars of philanthropic funding flooding into selective R&D spaces.

And I feel like this could change the world of research and development pretty dramatically. What do you think actually needs to happen to the funding landscape, the institutional landscape, for us to make the most of this?

Tom Milton: One way I like to describe Amodo is we are a fit-for-purpose engineering R&D lab for this AI era. And the reference class I use: in order to turn capital into scientific or engineering progress, you need someone or something to do the work. So DARPA didn’t have any lab benches. They have their performers — their primes, these engineering companies and academic groups that do work for them. So when we look at this new wave of philanthropy, there needs to be groups who can do this research. And one way to describe Amodo is the fit-for-purpose prime for these catastrophic risks.

Tom Reed: Who are some of the historic primes that you think were a good example?

Tom Milton: Yeah, historic primes would be Skunk Works. They would be the canonical prime for turning US government capital into new types of jet plane. And there’s a company called BBN — Bolt, Beranek and Newman — they were the prime, the contractor who did the ARPANET contract, the thing that kind of led to the internet. And they did this for ARPA at the time. And then we are the fit-for-purpose prime for biosecurity, for AI verification, for emerging risks.

Tom Reed: And what makes you fit for purpose, rather than just existing legacy companies?

Tom Milton: One is the fact we take the risks seriously, and are embedded and knowledgeable about them, and have team members who understand the landscape of AI, understand the landscape of catastrophic risks. And if you were to go and interact with one of the normal kind of engineering consultancies in the UK, it’s much beyond their remit and knowledge.

The other is that we have set up a hardware team that has a track record of delivering on speculative hard R&D projects. We’ve made brain implants, we’ve made brain imaging systems, we’ve made high-speed communications systems or quantum networking devices, all sorts of things like this — and have focused not on producing PowerPoints for our clients, but on producing products.

So in about four years we’ve delivered 100 hardware projects and have this well-oiled machine that can turn capital into physical sciences R&D progress: amazing engineers, a well-run workshop. And we think that is going to be one of the crucial elements to turn this new philanthropic capital into impact in the real world.

Tom Reed: What about government funding as well? So if government is looking to fund and steer scientific progress in one way or another, and the speed of scientific progress is potentially accelerating massively — how should governments be thinking about how they’re allocating capital, like the MHRA or all these different institutions?

Tom Milton: So UKRI in the UK gives out on the order of £10 billion a year. I think it’s going to be extremely hard for them to adapt, because I think the pace is going to be so high. And people are used to proposing one-, two-, five-year research programmes in which they talk about deliverables that I think will go from taking years to taking months or weeks.

And this means you’re going to need to change either the way that you give out your funding — so rather than being goals that take five years, goals that are much shorter, and then iterating on decision making, so every two weeks they come back to you and you reevaluate how they should spend their money — or giving large block grants in which they have a lot more freedom over those time periods.

But the main struggle, I think, is just going to be the pace of change and whether they can really adapt to it. You can’t have the UKRI setting out a 20-year research time horizon and these elaborate long research plans if these are going to be torn up in a few months.

The other interesting aspect here is government moving from the largest kind of public good funder to philanthropy competing on the same scale. Now, they won’t compete on the same scale as the US government — they give out enormous amounts. But if you were to look at the UK, UKRI giving out £10 billion a year, it seems feasible that these AI lab philanthropists, the Anthropic Institute, Anthropic cofounders, OpenAI Foundation, Coefficient Giving, all of these groups — could be giving out tens of billions of dollars a year, where maybe single-digit billions could be spent on R&D in the UK. And suddenly they’re actually a considerable part of our whole research ecosystem.

I think it’s a big question of how research groups here in the UK can adapt to absorb this capital, to adapt to the types of work that these philanthropists want them to work on, but can also come up with working patterns that are much faster and conducive with powerful AI systems.

Tom Reed: And that basically involves figuring out ways to prototype things quickly, having people who know how to use the models?

Tom Milton: Yeah, know how to use the models, change your scientific workflow. It might be that the way you start adopting AI in science — and in physical sciences R&D and hardware R&D — is you just give them your normal human tasks. But it might be with this kind of different brain type, you should be following a different process.

Maybe you run much more stuff in parallel, because the cost of parallelism has suddenly gone down. So rather than doing lots of serial-time experiments, you should do all of your experiments in parallel. And you have lots of ways like this. So I think also, along with just being good at using the models is: can you change your R&D workflows to match the differential advantages that the AIs have?

What’s left for engineers after AI? [01:32:26]

Tom Reed: This is perhaps a slightly ill-formed question, but I feel like if you’re someone who’s running a company, you might have a take on this. Have you seen increasing levels of automation change the culture of your company at all? What individual engineers are able to do, are they depending on each other less, because they’re collaborating with the models more? Do you feel like you have a sense for how the culture of Amodo has been changed in the past even just a year?

Tom Milton: I think the culture hasn’t shifted very much, but there are examples of the way that working has kind of changed. The majority of our projects are basically the same, but we’re now at the point where small electronics projects they’re not asking for a firmware engineer anymore, like, “Oh, I can probably take that task on myself.” Or a mechanical engineer.

Tom Reed: Firmware is you’re designing software for a specific hardware project.

Tom Milton: Yeah, exactly. Another example would be: recently we had a mechanical engineer who’s working on a project to do with sizing particles in the atmosphere — making a particle sizer that can go on a drone to measure particles, as part of one of ARIA’s programmes.

Tom Reed: Particles in the air?

Tom Milton: Yeah. And he needed a differential pressure sensor. He needed to take this off-the-shelf component, put it on a PCB — put it on a circuit board — connect it to a computer. Normally a mechanical engineer would go to one of our electronic engineers for this. It was a very simple design; it would only have taken a couple of hours for an electronic engineer to do. But instead of doing that, they worked with Fable to design the PCB and were suddenly able to do it themselves.

So I think the main change is going to be that engineers are able to take on many more disciplines that aren’t their own. So for a long time, or some reasonable time, they’ll still lead in their own discipline. But increasingly they’ll say, “Oh actually, I don’t need to go to the resourcing meeting, find an electronic engineer who can be resourced onto this to give me advice on how to set up these power supplies.” Rather, they are empowered by the models to do it themselves.

I think this means many more teams of engineers who are able to deliver much more ambitious projects much more quickly, because they suddenly have extra skill sets that they didn’t have before.

Tom Reed: And what about making sure Amodo remains at the frontier of using the models as much as you’re able to? Is hiring people who use the models as much as they can the main lever that you have, or do you do top-down stuff actively after having hired people too?

Tom Milton: This is really hard to think about, because I find the capabilities of the models viscerally scary. Not just in terms of how good they are, not just in the traditional risks talked about on this podcast, of bio and AI cyber and things like this. But you interact with them and you can’t help but think, “Wow, my job is going to change so much.” This is very destabilising and strange and scary.

The main thing that we’ve done to get people to understand where capabilities are is by running hackathons. So when Fable 5 came out, we prompted it on a few hardware design tasks and were in awe, or in complete shock. I had pure Claude psychosis for three days, in which all I would do was prompt them on the hardware.

Tom Reed: What kind of stuff were you making?

Tom Milton: I was making this PCB. I was making fidget toys, all sorts. And then fortunately the US government export-controlled it, so my psychosis ended, and I was able to go back to doing normal work.

Tom Reed: Enforced cold turkey.

Tom Milton: But you don’t really grok how good the models are until they suddenly are really good at something that you’re good at, or you’re adjacent to being good at. So what we’ve done on this is held 10 hackathons, with six of the team at a time for a whole day, in which we just rerun old projects. We get an intuition for where they’re good and where they’re bad.

And it does lead to a weird atmosphere. The models are really good in some places and really bad in others. But it is a strange experience seeing something that you love doing and are really skilled at suddenly be done by a computer. So I’m both excited about the capabilities that we all have, but also worried and scared about what the future looks like for a company and for work. I think this is underrated by many people.

Tom Reed: Interesting. So is your mainline expectation that there’ll be companies that are good at adopting this technology into their workflows, and they will just dramatically outcompete the companies that won’t? Is this kind of how you see things going? Or is that the wrong way to think about it?

Tom Milton: I think that is one direction. I think there’s already slack in the system. There’s already a lot of waste in the system. We already outcompete many of our competitors, and exist because in many cases we’re able to accelerate scientists by tenfold. So what’s another fivefold that is caused by models?

Tom Reed: How do you measure a tenfold acceleration of scientists? This is actually something you measure?

Tom Milton: This is something we do. With ARIA, an ARIA Creator — what they call their grantees — has to submit milestones that they’re going to hit. So this is really great for us, because we then go and help them on one of their projects, and we can see whether their milestones were delivered early.

In a bunch of cases, we’ve delivered milestones 12 months early on a two- or three-year R&D project, and in some cases delivered things that they think would have taken 18 months in the course of a few weeks. So we’re already delivering very quickly, but haven’t come to dominate all of science. And this is because of markets not being perfectly efficient, all of these other dynamics. So I do think that companies might be able to move much more quickly.

What this means for how your business models evolve is then an interesting question in hardware. So we were originally set up as a consultancy: we are paid by other people to do hardware R&D work for them. Increasingly we’re describing ourselves as a hardware R&D lab or physical sciences R&D lab now — not least because “consultancy” is not cool — but because as we speed up at doing the engineering work in the middle, we want to take on more of the upstream and downstream parts.

So when we think about our high-impact work around AI risk, this means taking more responsibility for identifying the projects we should work on, more time doing cause prioritisation and thinking about the inputs, and then also more time thinking about the spinouts and how things get deployed. So we don’t do engineering work for the sake of doing engineering work. We do engineering work because we want to see good in the world, and we can’t necessarily rely on doing some engineering work, and then putting the product on a shelf and publishing a nice GitHub repo with some open source designs.

Tom Reed: You want it to go where it needs to go.

Tom Milton: Exactly. So increasingly we will be owning much more of the output. This doesn’t necessarily mean for-profit spinouts. It could be for-profit spinouts, it could be nonprofits, it could be open sourcing designs, it could be partnering with others for them to deploy it. And the move to being much faster at the middle step will mean we can do more of that, and also take on more of the pie — more of this impact pipeline.

Personal gadgets, designed by Fable? [01:40:07]

Tom Reed: Taking it back to personalised hardware, what piece of personalised hardware are you most excited to build? Do you have anything that you daydream about?

Tom Milton: This was one that I had for a long time: the wake-up light. I had thought about making this for years, and I had just not had the skill set. I really couldn’t have done that. And now I have a physical product that’s my own.

I recently made a distance tracker for cycling. So when I go cycling with my partner, one of us will inevitably drop the other. And rather than looking behind and always seeing whether they’re still on your wheel or you’ve dropped them, I made a system that uses an ultra-wideband radio. It sits on your handlebars and it shows how far behind they are.

Tom Reed: What is it looking at?

Tom Milton: So it has a radio on each handlebar, and ultra-wideband radios, for some physics and electronics that I don’t understand, can tell distance. So one sits on my handlebars, one sits on her handlebars, and on a little screen it shows how far behind she is or how far ahead she is. And then I can tell, “Oh no, I’m cycling too fast. I’ve dropped her.” Or vice versa.

Tom Reed: And that was designed with Fable?

Tom Milton: It was designed with Fable. And there are all sorts of weird products like this that I’m excited about. And the existence of these tools, I think, causes a really significant return to people who have many ideas, who didn’t necessarily have the execution skill set for it.

Tom Reed: It’s the golden age of the ideas guy, right?

Tom Milton: Yeah, exactly. I feel massively uplifted, but I mostly feel massively uplifted because I was an incompetent engineer. Much of our team didn’t feel very uplifted, but that’s because they were already very good. So it’s not that helpful.

What if the US stops sharing its AI? [01:41:56]

Tom Reed: I’m interested in how you feel running an engineering company in the UK right now. How do you feel about the UK’s prospects going into this world where AI can do more and more of all the productive work that humans are doing?

Tom Milton: I think at least for the last five years, being in the UK has been a significant advantage to us rather than a disadvantage. And being in Sheffield, in the north of England, has also been a weird advantage. And the main weird advantage is it’s not been noisy.

So if we were in London, if we were in San Francisco, we would have been dragged into a local minima — dragged into the current venture capitalist hype cycle. Whereas as a bootstrapped company in Sheffield, we’ve been able to build an engine that does amazing hardware work and is not distracted by the newest ChatGPT wrapper. And this means we’ve got an amazing team of 60 who deliver hardware work I think as well as any team in the Bay would — and we’ve been able to work on weird things.

If we were a venture-backed San Francisco startup, it’s unlikely that we would have, at the start of the year, been able to start working on AI verification under the premise that maybe you’d need an international deal. VCs were not into that.

So I think being here has given us access to underpriced talent: really good engineers who the rest of the world hasn’t noticed. I also think that our team have honed lots of skills that don’t get honed if you live in the Bay — like honest, good delivery that always works.

I think the place where this changes is if frontier AI becomes really important to our delivery pipeline and the UK doesn’t get access anymore.

Tom Reed: That’s the existential threat, essentially.

Tom Milton: So we already saw this. I had my three days of Claude psychosis, where I was using the AI systems to do all of this electronics design and mechanical design. And then it was a Saturday morning, I woke up, I said to Fable 5, “Keep going.” It returned an error. I said, “Keep going” again; it returned an error. I went on Twitter: Fable access had been cut off.

And suddenly I had lost a skill set. I had, the previous day, been able to do some electronic design, and then I had reverted and I no longer had this skill. And imagine building a company on that. You’re building a company on having access to this intelligence, and it gets taken away just like that. That is catastrophic.

And my worry would be that as frontier AI becomes more and more capable, the first thing will be we’ll have a compute crunch. So the US will have as many data centres as it can, it will be serving as many tokens as it can, but there will be way more demand for tokens than there are tokens that can be served. At this point, what you start doing is you start prioritising domestic supply. So instead of serving Amodo with tokens, you serve your Boston or San Francisco equivalent with tokens, because you want to develop your own economy with the amazing hardware progress that can be enabled by the AIs. And what this means is that we begin to lose access to AI systems.

Now, what can the UK do about this? Well, of course there is already a special relationship; there are already reasonable ties. This is hopeful. But I think the risk is that the US are unwilling to export model weights outside of the US unless you can make very strong security claims. So the export of the weights is a marginal cost — unless they get stolen, in which case that’s catastrophic.

So unless the UK has sovereign compute that is at a high enough security level, I’m increasingly worried that we will no longer have access to these frontier models, and that this could hold back UK science, UK R&D, UK companies — lower quality of living, ultimately, as it gets implemented as part of the healthcare system, et cetera.

Tom Reed: So building ultra-secure compute in the UK would be an absolute priority.

Tom Milton: Yeah, I think the case against this is whether or not you actually need the frontier of capabilities. Are our engineers fantastic? Yes. Are they literally the best people in the world at electronic design? It’s like, I don’t know, maybe, but possibly not. Was it OK that we had person number 10 or person number 100 or person number 1,000? They were still amazing at doing it. And likewise, I think we probably see the same with AIs, where they saturate at a certain point, and some tasks just don’t benefit from increased intelligence.

Tom Reed: There’s not just an infinite dose-response curve sloping off.

Tom Milton: Exactly. So this is where I can envisage there being much less of a problem for companies no longer having access to the frontier. Some problems, they’re just Sonnet problems. Some problems are Fable problems, but not all work is a Fable problem.

I think it would still be really bad for your country not to have access to the smartest intelligence, because there are many problems that may well yield to these very significant intelligences. And you still probably need some domestic compute for optionality. Even if there is still a free international market for compute, you still have a lot of benefit to having it on sovereign soil.

Tom Reed: So it’s still worth doing even under these conditions.

Tom Milton: And I think you also likely need a bunch of this high-security compute for your national security as well, where it seems likely that AI systems will massively transform the intelligence community. If you think about their ability to process vast quantities of data in the same way that analysts do in GCHQ, you could see them really being bottlenecked by GPU access. And I don’t think GCHQ are going to be passing lots of their classified information through a random Swedish data centre that’s serving Astra this week.

Tom Reed: Well, was it a PLA [People’s Liberation Army] official who was querying Moonshot’s Kimi and it ended up going to Claude? I can’t remember what it was.

Tom Milton: Amazing.

Tom Reed: Yeah, I can’t remember exactly. There was some Chinese government official doing this. So some people are doing this, but they shouldn’t be, presumably. Yeah.

Tom Milton: So I think on the secure compute side there is this: make sure we still have access to frontier models. There is also this clear necessity for some parts of government to have access to this for national interest reasons.

Tom Reed: Yeah, that sounds compelling to me. Well, Tom, it’s been a real pleasure. Thank you for coming on.

Tom Milton: Thank you very much.

Tom Reed: Until next time.