#199 – Nathan Calvin on California’s AI bill SB 1047 and its potential to shape US AI policy

Transcript

Cold open [00:00:00]

Nathan Calvin: Everyone has a duty to take reasonable care to prevent harms. And if there’s a situation where a model causes a catastrophe, I think that there is a very real argument that, under just existing tort negligence law, lawsuits could exist.

And I think the role of this law, and even the fact that we’re reusing these same terms from existing tort standards — like “reasonable care” — is partially to remind and put in companies’ awareness the responsibilities that they already have.

I think that somehow companies take the example from Section 230 or some other areas of law where there is a statutory exemption to liability and therefore extrapolate that to think that, “If I am doing work with software, I can’t get sued no matter what happens.” And it’s not like there’s some part of the common law that has, “If the harm is caused by a computer, then you’re off the hook.” That’s not how this works.

Luisa’s intro [00:00:57]

Luisa Rodriguez: Hi listeners, this is Luisa Rodriguez, one of the hosts of The 80,000 Hours Podcast. Today’s episode is a bit different than usual. We had a last-minute opportunity to briefly speak with Nathan Calvin, who’s helped shape the California AI bill that’s working through the state senate: you might know it as SB 1047.

Nathan and I talk about:

• What’s in the bill, concretely.
• The most common objections to the bill — including how it could affect competition, startups, open source models, and US national security — and which of these objections Nathan thinks hold water.
• What Nathan sees as the biggest misunderstandings about the bill that get in the way of good public discourse about it.
• Nathan’s take on how likely the bill is to pass and become law.

Before we launch into the episode, I want to flag that this interview is a bit less polished than our usual episodes, because this is all happening very fast, there’s not a tonne of great information about it out there, and things change quickly — so we wanted to get good information out about it ASAP.

It’s also worth noting that we recorded this interview on August 19, so some things in the episode are already out of date. The big thing is that, since recording the interview, SB 1047 actually passed the California State Assembly, meaning it just has one final hurdle to jump through before becoming state law.

All right, with that pretty major update in mind, I bring you Nathan Calvin.

The interview begins [00:02:23]

Luisa Rodriguez: Today I’m speaking with Nathan Calvin. Nathan is senior policy counsel at the Center for AI Safety Action Fund, which is the advocacy affiliate of the Center for AI Safety, a technical research organisation trying to reduce societal-scale risks from AI through technical research and field building.

As part of that work, he helped shape a proposed AI bill in California — which would mandate safety assessments, third-party auditing, and liability for developers of advanced AI models. Thanks for coming on the podcast, Nathan.

Nathan Calvin: Thank you. Very glad to be here.

What risks from AI does SB 1047 try to address? [00:03:10]

Luisa Rodriguez: I basically want to dive right into SB 1047. Can you start by saying what kinds of risks from AI the bill is trying to address?

Nathan Calvin: I think it’s very much trying to pick up where the Biden executive order left off.

So there are three categories of risks that the EO talks about: In terms of risk from chemical, biological, radiological, and nuclear weapons, ways that AI could kind of exacerbate those risks or allow folks who were previously not able to weaponise those technologies to do so; then another one is very severe cyberattacks on critical infrastructure; and another one is AI systems that are just autonomously causing different types of havoc and evading human control in different ways.

So those are the three categories of risk that the Biden executive order lays out, and I think that this is very similarly trying to take on those risks.

Luisa Rodriguez: What can you say about how the bill came to be, including any involvement you’ve personally had in it?

Nathan Calvin: I think that Senator Wiener got interested in these issues himself just from talking with a variety of folks in SF who were thinking about these risks. For people who have spent time at SF get-togethers, this is a thing that people are just talking about a lot and thinking about a lot, and it’s something that he got interested in and really taken with.

So then he put out the intent bill and was looking for organisations to help make that into a reality, and make it into full, detailed legislation. As part of that process, he got in touch with us, the AI Center for Safety Action Fund, as well as Economic Security California Action and Encode Justice. And we really worked on putting additional technical meat on the bones of some of those high-level intentions that they laid out.

I think there are some authors in the representatives who defer a lot to staff and other folks they’re working with, but I think Senator Wiener was just very deeply in the details and wanted to make sure that he understood what we were doing and agreed with the approach. It’s really been a pleasure to work with him in his office and kind of the amount of involvement and interest he’s taken in the policy.

Luisa Rodriguez: Cool. So in just incredibly simple terms, what does the bill say?

Nathan Calvin: The way that I most straightforwardly describe the bill is that there have been a lot of voluntary commitments that the AI companies have themselves agreed to — things like the White House voluntary commitments. There were also some additional voluntary commitments that were made in Seoul, facilitated by the UK AI Safety Institute, and it’s saying a lot of things around testing for serious risks, taking cybersecurity seriously, thinking about these things.

And what I really view this bill as is taking those voluntary commitments and actually instantiating them into law. And saying that this is not something that you’re just going to decide whether you want to do, but something that there are actually going to be legal consequences if you’re not doing these things that really seem very sensible and good for the public.

Luisa Rodriguez: Hey listeners, a quick interruption. To give ourselves more time to chat through objections to the bill, misunderstandings about it, and so on, Nathan and I didn’t dive any deeper into the details of the bill during our actual interview — so I wanted to jump in to give a few more concrete details about what’s in the bill (as of August 23).

So first, it’s worth emphasising that all of the provisions of the bill only apply to models that require $100 million or more in compute to train, or that take an open sourced model that is that big to start with and fine-tune it with another $10 million worth of additional compute. At the moment, there are no models that meet these requirements, so the bill doesn’t apply to any currently existing models.

For future models that would be covered by the bill, the bill creates a few key requirements:

First, developers are required to create a comprehensive Safety and Security Plan, or SSP, to ensure that their model does not pose an unreasonable risk of causing or significantly enabling “critical harm” — which is defined in the bill as mass casualties or incidents resulting in $500 million or more in damages.

This Safety and Security Plan has to explain how the developer is going to take “reasonable care” to guard against cybersecurity attacks to make sure that the model can’t be stolen, how it would be able to shut down all copies of the model under their control if there were an emergency, and how the developer will test that the model can’t itself cause critical harm — and the developer has to publish the results of those safety tests. And finally, the plan has to commit to building in appropriate guardrails to make sure users can’t use them in harmful ways.

In addition, developers of these advanced models are required to undergo an annual audit.

If a developer violates these rules and their model causes “critical harm” itself or is used by a person to cause “critical harm,” the developer can be held liable for that harm and fined by the Attorney General.

For fine-tuned models that involve $10 million or more in expenditure, the fine-tuner bears responsibility. For those spending less, the original developer holds responsibility.

Finally, the bill creates protections for whistleblowers — in other words, employees of AI companies who report non-compliance will be protected from retaliation.

There are a few other bits and pieces in there, but those were the things that struck me as most important.

OK, back to the interview.

Luisa Rodriguez: Do you have a take on how valuable the bill is, or how big a step it is toward managing AI risks?

Nathan Calvin: I mean, I think in some ways the bill is pretty remarkably modest and deferential to companies in a lot of ways. I think there are many folks in the AI safety community who I think would say that we need stronger things. There’s conversations around some of the proposals that are floating around or things like licencing regimes or strict liability or the government itself doing testing of systems, and lots of things like that.

And this bill doesn’t have any of those things. I think what it does have is kind of putting the onus on the companies to take these risks seriously and explain what measures they’re taking — and if something goes wrong, to have that be something that they have responsibility for.

So I think it is a quite significant step forward. And I do think that there are things like the Biden executive order, but actually having something in statute, even as a state law, is a big step forward. Particularly, I think it’s a similarly sized thing as the EU AI Act is, is maybe one quick way to put it. But specifically having that be in the United States is pretty significant.

Supporters and critics of the bill [00:11:03]

Luisa Rodriguez: So we’ll come back to more about what specifically is in the bill in a little bit. But I actually want to talk about the proponents and the critics of the bill — because it’s become so incredibly controversial over the last few months and even just last week, that I want to kind of look at that right off the bat. So who supports the bill? Who’s in favour?

Nathan Calvin: There’s a really wide variety of supporters. Some of the most high-profile ones have been Geoffrey Hinton and Yoshua Bengio and Stuart Russell and Lawrence Lessig — some of these scientific and academic luminaries of the field.

There’s also just a wide variety of different nonprofit and startups and different organisations that are supportive of it. SEIU, one of the largest unions in the United States, is supportive of the bill. There are also some AI startups, including Imbue and Notion, that are both in support of the legislation. And a wide variety of others, like the Latino Community Foundation. There’s just a lot of different kinds of civil society and nonprofit orgs who have formally supported the bill and say that this is important.

Luisa Rodriguez: I think from memory, the vast majority, or maybe it’s like three-quarters of Californians also in a poll really support the bill, which quite surprised me. I don’t think of basically any legislation ever having that much support, and probably that’s wrong, but it still seems just intuitively high to me.

But yeah, let’s talk about some of the opponents. I guess naively, it’s hard for me to understand why this bill has become so controversial — in particular, because my impression is that nearly all of the big AI companies have already adopted some version of this kind of exact set of policies internally. And you can correct me if I’m wrong there. But yeah, who exactly are the bill’s big opponents?

Nathan Calvin: I think maybe the loudest opponent has been Andreessen Horowitz at a16z, and some of their general partners have come out just really, really strongly against the bill.

Luisa Rodriguez: And just in case anyone’s not familiar, they’re maybe the biggest investor ever, at least in these technologies.

Nathan Calvin: Yeah, I think that in their category of VC firm, and there are probably different ways of defining it, I think they’re the largest. I’m sure you could put it different ways, such that they’re lower on that list or something, but they’re an extremely large venture capital firm.

So I think there’s a mix of different opponents. That’s definitely one really significant one. I think there are also folks like Yann LeCun, who has called a lot of the risk that the bill is considering science fiction and things like that.

There has also just been, kind of more quietly, but just a lot of the normal Big Tech interests of things like Google and TechNet, like the trade associations that really advocate on behalf of companies in legislative bodies have also been quite strongly against the bill.

We’ve also seen some folks in Congress weigh in, most recently and notably Nancy Pelosi — which is a little bit painful to me, as someone who’s a fan of her and has a tonne of respect for her and everything that she’s accomplished. And can talk a little bit about that specifically as well.

But yeah, there’s a mix of different folks who have come out against the bill, and I think they have some overlapping and some different reasons. And I agree that I’m a bit surprised by just how controversial and strong the reactions have been, given how relatively modest the legislation I think actually is, and kind of how much it has been amended over the course of the process. And even as it’s been amended to address different issues, it feels like the intensity of the opposition has kind of increased in volume rather than decreased.

Luisa Rodriguez: Yeah, I actually am curious about the Nancy Pelosi thing. Did she have particular criticisms? What was the opposition she voiced?

Nathan Calvin: I think it’s a mix of things. She talked about the letter that Fei-Fei Li wrote in opposition to the bill and cited that. I do think that that letter has one part that just is false, talking about how the shutdown requirements of the bill apply to open source models when they’re specifically exempted from those requirements.

I think that the other sense of it is just that they’re pointing to some of these existing processes and convenings that are happening federally, and just saying that it’s too early to really instantiate these more specifically in law, and that this is something that the federal government should do rather than having states like California move forward with.

And our response is really that California has done similar things on data privacy and on green energy and lots of other things where Congress has been stalled and they’ve taken action. And I think we do this similarly. Obviously, they have a difference of opinion there, but I do think that if we wait for Congress to act itself, we might be waiting a very long time.

Luisa Rodriguez: To what extent does this feel like a big update to you against kind of how safety oriented and cooperative about regulation these companies are going to be? It seems like they’ve been saying, like, “Please regulate us.” And then they were like, “No, we didn’t actually want that.”

Nathan Calvin: I mean, I think I had a decent degree of cynicism previously, so I don’t think it’s necessarily a massive update. And I think there has been some variety here, where it’s been reported in different ways, but Anthropic coming and engaging in a lot of detail with kind of the types of changes they would like to see —

Luisa Rodriguez: Yeah, can you give more context on that? So Anthropic submitted a letter that basically said they’d support the bill if it was amended in particular ways. Is that right?

Nathan Calvin: Yeah. And one important clarification of that is that I think some people interpreted a “support if amended” to imply that they are currently opposed. That’s not technically what it was. They were currently neutral, and they were saying that, “If you do these things, we will support.”

Luisa Rodriguez: “We will actively support it.” OK, that is reassuring to me. I did interpret it as, “We oppose it at the moment.”

Nathan Calvin: Yeah. And there’s some vagueness, and in this instance, that was not what was happening. And I still think these are large companies who have some of the incentives that large companies do. I think Anthropic is a company that is taking these things really seriously and is pioneering some of these measures, but I also think that they’re still a large company, and are going to deal with some of the incentive issues that large companies have. It’s a little bit unfortunate how some of their engagement was interpreted in terms of opposition, and I think they do deserve some credit in coming to the table here in a way that I think was actually helpful.

But stepping back from Anthropic specifically, and thinking about folks who are opposing this, it’s not like Anthropic is in any way lobbying against the bill — but there are other ones that certainly are. And to some degree, it’s not surprising. It’s a thing that we’ve seen before.

And it’s worth remembering that Mark Zuckerberg, in his testimony in front of Congress, said, “I want to be regulated.” You know, it’s a thing that you hear from lots of folks, where they say, “I want to be regulated” — but then what they really mean is basically, “I want you to mandate for the rest of the industry what I am doing now. And I want to just self-certify that what I am doing now is happening. And that’s it.” That is, I think, often what this really is.

So there’s some way in which it’s easy for them to support regulation in the abstract, but when they look at it… Again, I think there’s some aspect here of, even within these companies of folks who care about safety, I think there’s a reaction that says, “I understand this much better than the government does. I trust my own judgement about how to manage these tradeoffs and what is safe better than some bureaucrat. And really, it’s ultimately good for me to just make that decision.”

There are parts of that view that I guess I can understand how someone comes to it, but I just think that it ends up in a really dysfunctional place.

You know, it’s worth saying that I am quite enthusiastic about AI, and think that it has genuinely a tonne of promise and is super cool. And part of the reason I work in this space is because I find it extremely cool and interesting and amazing, and I just think that some of these things are some of the most remarkable things that humans have created. And it is amazing.

I think there is just a thing here that this is a collective action problem: you have this goal of safety and investing more in making this technology act in our interests, versus trying to make as much money as possible and release things as quickly as possible — and left to their own devices, companies are going to choose the latter. I do think that you need government to actually come in and say that you have to take these things seriously, and that that’s necessary.

If we do wait until a really horrific catastrophe does happen, I think you might be quite likely to get regulation that I think is actually a lot less nuanced and deferential than what this bill is. So I just think that there’s some level where they are being self-interested in a way that was not really a surprise to me.

But maybe the thing that I feel more strongly about is that I think they are not actually doing a good job of evaluating their long-term self-interest. I think they are really focused on, “How do I get this stuff off my back for the near future, and get to do whatever I want?” and are not really thinking about what this is going to mean for them in the longer term. And that has been a little bit disheartening to me.

One last thing I’ll say here is that I do think that there is a really significant sentiment among parts of the opposition that it’s not really just that this bill itself is that bad or extreme — when you really drill into it, it feels like one of those things where you read it and it’s like, “This is the thing that everyone is screaming about?” I think it’s a pretty modest bill in a lot of ways, but I think part of what they are thinking is that this is the first step to shutting down AI development. Or that if California does this, then lots of other states are going to do it, and we need to really slam the door shut on model-level regulation or else they’re just going to keep going.

I think that is like a lot of what the sentiment here is: it’s less about, in some ways, the details of this specific bill, and more about the sense that they want this to stop here, and they’re worried that if they give an inch that there will continue to be other things in the future. And I don’t think that is going to be tolerable to the public in the long run. I think it’s a bad choice, but I think that is the calculus that they are making.

Luisa Rodriguez: Yeah, OK. So that’s the calculus for the majority of the companies. It sounds like Anthropic has actually been kind of negatively portrayed as more opposed to the bill than they actually are. It sounds like they’re something closer to neutral, and hoping for amendments, and then would get on board.

Have those amendments happened? What were the key ones? And if there are amendments that have been requested by other AI companies that are worth talking about as well, I’m curious about all of those.

Nathan Calvin: Yeah. Anthropic requested a variety of amendments, and I think a good fraction of those were made, though in some different forms and not everything that they wanted.

One of the things was their concern over creating an entire new regulator in California, the Frontier Model Division, and I think just concern that California could do that well. Some of the things about whether you’re able to get enough money and technical talent and different things like that at the level of state government, than you might be able to elsewhere, I do think there’s some fair points there.

I also think that California is just in the worst budget deficit, in nominal terms, effectively, in ages. So I think that there were also other reasons, as we were going through the appropriations committee for the senator, to make those amendments to save costs.

I think where we stand is that a mix of the amendments have been made. I hope they come on in support after that. I think they are going to decide whether enough of those amendments were made for them to feel comfortable doing that. It’s possible they might say things positive that are short of full support, or a variety of things could happen there.

Misunderstandings about the bill [00:24:07]

Luisa Rodriguez: Yeah. OK, let’s go ahead and dive into more of the criticisms. My sense is that there are some reasonable worries that critics have about the bill, but that there are also just some completely false claims people make about the bill. So I guess to start, what do you think people are getting most wrong about it?

Nathan Calvin: I think there are a lot of things that some folks are just wrong. One of the main things is that the bill applies to models that cost over $100 million to train. The bill, in its most recently amended version, does not have the requirement that developers submit things under penalty of perjury anymore. But even when it did have that requirement, perjury is something that requires intentionally lying, not just kind of making an innocent mistake.

And despite those two facts, there were a lot of things about how this bill is going to send startup founders to prison — which I think is just a pretty wild and inflammatory claim, given that, again, it’s focusing on tremendously large companies and about intentionally lying. And again, even that thing is now out of the bill.

But I think that gives you some sense for some of the things that I’ve heard people, who I feel like should know better, repeat. But there are lots of other things in addition to that.

Again, I think that the bill genuinely has improved and gotten better since the senator introduced it. I think there have been real improvements that were made and real issues that people spotted, and I’m happy to discuss what some of those things are. But I also think that there has been some stuff that just is really pretty frustrating to me, and it’s just lowered the standard of public discourse in a pretty unhelpful way.

Luisa Rodriguez: Yeah. From afar, it seemed just incredibly disheartening and disappointing to me, though I’ve been following it only more distantly.

So let’s take a few more criticisms or objections one by one. My sense of one big worry that some of the bill’s critics have is that the bill will kind of impose such big costs on AI developers that it’ll totally just stifle innovation and potentially cause AI companies to leave California. Do you want to say any more about that worry? Or if not, is it fair? What is your take?

Nathan Calvin: I think there are a couple different things. I mean, I think these are things that the companies are already saying they are doing, and are saying that they can do consistent with being at the frontier of this technology.

It goes back and forth, but at least arguably Anthropic has maybe the best AI system in the world right now. I think that they are making clear that they believe that taking these issues seriously and also still being at the leading edge of this is quite possible.

And that’s a similar thing that other companies have said. So we’re taking their word for that, saying that they believe that they can do both. And we agree, but we want to actually make that into something that is really not something that is just left to their discretion. Because I think that when there is competitive pressure to just release products as fast as possible and to be out ahead, I think it’s just very easy for these kind of loftier ideas to go out the window pretty quickly.

In terms of companies leaving California, I think that’s not a super serious objection, because it applies not just to companies that are headquartered in California, but also to companies that are doing business and selling their models in the state. So I think that that would be a pretty wild step to take to change. And when combined with the fact that, again, these things are pretty modest stuff that they are already saying that they are doing, I just think that seems like a not super-credible objection to me.

Luisa Rodriguez: Yeah, this is one of those criticisms that just struck me as totally implausible, and so implausible that it seemed in bad faith to me. Clearly, given that it is going to apply to any company who wants their model to be usable at all in the state of California — the fifth largest economy in the world — companies just aren’t going to decide not to make their product available in California.

Is there anything fair, or anything plausible, if you try to be charitable to that perspective?

Nathan Calvin: I mean, I think there are versions of it that are very… Like, I’ve heard people say all the startups are going to leave — and the startups aren’t covered by the bill anyway. And I think that’s partially a result of some folks just saying this is going to send you to prison, even though you’re not covered, and just really crazy stuff.

I do think that there is some level where there are companies who have chosen not to release specific products in Europe related to some of the regulatory decisions that they’ve made. I think those circumstances are a bit different in important ways, where I think that some of the regulations that were causing some of those decisions were more things actually about antitrust or data privacy, and kind of have their own sets of things.

There also is a question of, because [SB 1047] applies if you are doing business or headquartered, it would mean that Meta would not only have to stop selling their models in the state, but would also have to move its headquarters, which is a larger step. So it’s not the same as with Europe, that they could just not release their product; they would also have to move headquarters, which again seems like a pretty wild step given how modest this legislation is.

I don’t want to say it to be the case that there is no amount of regulation that California could do that could cause them… I think you could think about it just like, they get some percent of their revenue from this, and it’s some percent of their compliance cost — and if that latter number exceeds it, then that is not worth it. And I think that this bill is just not really remotely close to that.

Luisa Rodriguez: Do we have a guess at what specifically the cost will be like for companies to meet the requirements?

Nathan Calvin: Yeah, we’ve chatted with a variety of different folks who are familiar with what’s required to do some of these measures — that, again, lots of the companies are already saying that they are doing — and I think single-digit percents of the cost of training the model is a reasonable estimate.

Competition, open source, and liability concerns [00:30:56]

Luisa Rodriguez: A related worry is that the bill could impose unreasonable costs on startups. Which you’ve already kind of mentioned, but to dig more into it, it’s something like startups wouldn’t be able to implement or afford the kinds of safety tests that the bill requires. Some of the bill’s opponents say that this will put such high barriers in place for AI startups, that it’ll stifle competition, because only the AI companies that are already super big and established will be able to afford to work in the space. So in that sense, the bill’s been criticised as a form of regulatory capture. What do you think of that criticism?

Nathan Calvin: So there are two things to say there. One is that, partially, criticisms like that were what led the senator to make an amendment to the bill saying that models that are covered have to be both above a certain amount of training FLOP, and also, in addition to that, have to cost more than $100 million to train.

So just by definition, if you are spending $100 million training a model… There’s debates in the San Francisco Bay Area about what counts as a startup, and there are people who say a company worth $20 billion is a startup — and maybe that’s fair; maybe you are a startup and you’re not a trillion-dollar company — but I think you can afford to do safety testing. I just think this is not like a mom-and-pop corner store; this is a place spending $100 million to train an incredibly capable AI system.

Luisa Rodriguez: Yeah, yeah. Either you’re rich enough to train the model and therefore rich enough to do the safety testing, or you’re not either of those things, but you don’t have to do the safety testing.

Nathan Calvin: I think that’s right. I mean, the more complex debate that we can talk about is just that there are people who make a version of this criticism: their concern is not that the bill will directly apply to startups, but that it will change the behaviour of companies who are kind of upstream of the startups. I think this relates to the conversation around the bill’s approach to open source and how to think about that. So I do think that that is an area that is more complex, and I do think there are good faith objections to. And I think our approach makes sense, but there are more fair things that can be raised.

Luisa Rodriguez: Yeah, do you mind talking a bit about those?

Nathan Calvin: So the bill, in the most recent draft, doesn’t use the word “open source” at all; it just treats large AI developers the same, regardless of whether it is an open model or a closed model. The only extent to which it treats them differently is actually kind of in the favour of open source, in the sense that the shutdown provision specifically is exempted for models outside of the control of the developer.

I think the argument that people make is that some of the provisions in the bill that say that you have to take “reasonable care” to prevent “unreasonable risks” — which are these kind of terms of art in the law — including from third parties making modifications to your product and then using that to cause harm, that there are ways in which that might be more difficult for an open weight model developer to comply with.

Luisa Rodriguez: So just to clarify exactly the thing that is happening that we might think is good that could be hindered by the bill, I think it’s something like: there are some developers who build on open source models, like Meta’s Llama. And we might think that they’re doing useful, good things by building on Llama. But if Meta becomes kind of responsible for what those downstream developers do with their model, they’re less likely to make those models open access.

Nathan Calvin: Yeah, I mean, to be clear, like, this is not applying to any model that exists today, including Meta’s release of Llama 405B, which I think is the most recent one. Those are not models that are covered under this, and people can build on top of those and modify those forever. I think that model, the best estimates I’ve seen are that it costs actually well under $100 million to train. And you also have, with the cost of compute going down and algorithmic gains in efficiency, the most capable open source model that is not at all covered by this bill is still increasing in capability quite a lot each year. I think that there have been folks in the open source community who really look at this, and say there’s quite a tremendous amount that you can do and build on it that is not even remotely touched by this bill.

I do think there is just a difficult question here around, let’s say that there’s Llama 7, and they test it and find out that it could be used to very easily do zero-day exploits on critical infrastructure, and they install in some guardrail to have it reject those requests. Again, this is also a problem for closed source developers, that these guardrails are not that reliable and can be jailbroken in different ways. So I think this is a challenge that also exists for closed source developers.

But if they just release that model and someone, for a very small amount of money, removes that guardrail and causes a catastrophe, for Meta to then say, “We have no responsibility for that, even though our model was used for that and modified” — in what we believe is a very foreseeable way — I think that is an area where I think just people disagree. Where they think that it’s just kind of so important for Meta to release the really capable model, even if it can be used to straightforwardly cause a catastrophe.

And I do think it’s important to say that, again, it is not strict liability; it is not saying that if your model is used to cause harm, that you are liable. It is saying that you have to take reasonable care to prevent unreasonable risks. And exactly what the line of that and what amount of testing, what amount of guardrails, what is sufficient in light of that are the types of things that people are going to interpret over time. And, these are terms that are used in the law in general.

But I do stand pretty strongly for the idea that for these largest models that could just be extremely capable, that saying you’re going to put it out into the world and whatever happens after that is just in no way your responsibility, I do think that that is just something that I don’t think makes sense.

Luisa Rodriguez: Yeah, I’ve heard people say that this is equivalent to suing an engine manufacturer for making the engine that gets used in a car, where that car then is used to accidentally hit someone. Do you have a take on whether that analogy is good or bad?

Nathan Calvin: I think in some ways it’s a useful intuition pump. One of the things to say is that if it was the case that there were multiple different ways you could design an engine — and some of those engines, when the car was used in a certain way, would explode and cause harm to others, even if it was in cases of misuse and there was like a reasonable things that you could do that are alternatives — then I do think that in those circumstances, engine manufacturers have liability.

One of the things that actually has been really interesting about the debate around this bill is that I think that a lot of people in the AI industry kind of think that existing tort law is not at all relevant to what they’re doing, and that they don’t have any liability under existing law if their model caused a catastrophe. A variety of silly arguments are made here. One thing is when you have an open source licence, there’s a disclaimer of liability — but that is only a disclaimer of liability between the original developer and the person using the model, signing the licence. You can’t sign a waiver that disclaims liability for third parties that are harmed by your thing. That’s not how this works.

And also, in general there’s just a thing where everyone has a duty to take reasonable care to prevent harms. And if there’s a situation where a model causes a catastrophe, I think that there is a very real argument that, under just existing tort negligence law, lawsuits could exist.

And I think the role of this law, and even the fact that we’re reusing these same terms from existing tort standards — like “reasonable care” — is partially to remind and put in companies’ awareness the responsibilities that they already have. But I think that putting it explicitly in a statute, versus having to make these arguments from the common law and court cases, just puts it much more in their face that they have this duty to take reasonable care to prevent really severe harms.

I think that somehow companies take the example from Section 230 or some other areas of law where there is a statutory exemption to liability and therefore extrapolate that to think that, “If I am doing work with software, I can’t get sued no matter what happens.” And it’s not like there’s some part of the common law that has, “If the harm is caused by a computer, then you’re off the hook.” That’s not how this works.

Luisa Rodriguez: OK, is there anything more you want to say on that before we move on?

Nathan Calvin: Yeah, one thing that I want to emphasise is that one thing I like about liability as an approach in this area is that if the risks do not manifest, then the companies aren’t liable. It’s not like you are taking some action of shutting down AI development or doing things that are really costly. You are saying, “If these risks aren’t real, and you test for them and they’re not showing up as real, and you’re releasing the model and harms aren’t occurring, you’re good.”

So I do think that there is some aspect of, again, these things are all incredibly uncertain. I think that there are different types of risks that are based on different models and potential futures of AI development. And I think anyone who is saying with extremely high confidence about when and if those things will or won’t happen, I think is not engaging with this seriously enough.

So having an approach around testing and liability and getting the incentives right is really a way that a policy engages with this uncertainty, and I think is something that you can support. Even if you think that it’s a low risk that one of these risks is going to happen in the next generations of models, I think it is really meaningfully robust to that.

I also just think that there’s this question of when you have areas that are changing at exponential — or in the case of AI, I think if you graph the amount of compute used in training runs, it’s actually super-exponential, really fast improvement — if you wait to try to set up the machinery of government until it is incredibly clear, you’re just going to be too late. You know, we’ve seen this bill go through its process in a year. There are going to be things that even in the event the bill is passed will take additional time. You know, maybe it won’t be passed and we need to introduce it in another session.

I just think if you wait until it is incredibly clear that there is a problem, that is not the time at which you want to make policy and in which you’re going to be really satisfied by the outcome. So I just think that policymaking in the light of uncertainty, that just is what AI policy is, and you’ve got to deal with that in one way or another. And I think that this bill does approach that in a pretty sensible way.

Luisa Rodriguez: Yeah, I’m certainly sympathetic to “policy seems to take a long time and AI progress seems to be faster and faster”. So I’m certainly not excited about the idea of starting to think about how to get new bills passed when we start seeing really worrying signs about AI.

On the other hand, it feels kind of dismissive to me to say this bill comes with no costs if the risks aren’t real. It seems clear that the risk does come with costs, both financial and potentially kind of incentive-y ones.

Nathan Calvin: Yeah. I mean, I think it’s a question of, as I was saying before, the costs of doing these types of safety testing and compliance things I think are quite small relative to the incredibly capital-intensive nature of training these models. And again, these are things also that we’ve seen companies do. When you look at things like the GPT-4 System Card and the effort that they put into that, and similar efforts at Anthropic and other companies, these are things that are doable.

There’s also something like, “Oh, I’m not going to buy insurance because there’s not a 100% chance that my house will burn down” or whatever. I think if you have a few percent risk of a really bad outcome, it is worth investing some amount of money to kind of prepare against that.

Luisa Rodriguez: Let’s move to another one. Some critics have raised objections along the lines of: if AI development is going to be important for national security, and the US is worried about a competitor like China achieving some national-security-related AI capability — so maybe some military use, for example — will a bill like SB 1047 be bad for US competitiveness, and I guess national security?

Nathan Calvin: I mean, I do find it interesting that some of the folks who I hear say this argument are also some of the folks who are most vocal about open sourcing the most powerful systems, no matter what — in a form that countries like China can then have. There may be a mix of things, but I do feel like there is tension in those arguments. I also don’t think that we accept this argument in other contexts.

I also think that China itself is having some actually pretty strict domestic regulations of AI and a lot of concern about whether it’s saying things that undermine the regime. There are regulations in China as well. And I think just in general boogeyman that that will prevent us from competing, I just think is not really that…

There’s some level of regulation where I think that that could be the case, and where you really need assurance that other countries are going to follow you. But again, these are things that companies are already saying that they are doing, and that we are seeing companies being able to do while really operating at the frontier of this technology. So again, I don’t think it’s credible in this instance.

Model size thresholds [00:46:24]

Luisa Rodriguez: OK, pushing on: as it stands, the bill would apply to AI models trained with more than 10²⁶ floating point operations, and that cost at least $100 million to train. But my impression is that some people think that this threshold is kind of arbitrary or unjustified. Does that seem fair to you?

Nathan Calvin: I think to some degree it’s arbitrary in the way that lots of laws are arbitrary. You know, people made this argument, like, “I’m driving on the freeway and the speed limit is 65 miles an hour. Why isn’t it 63 miles an hour? Why isn’t it 72? Why doesn’t it depend on whether there’s a lot of cars near me — in which case I should go slower, and if not I should go faster?” Well, because it’s hard to have a determination that can take all those things into consideration, but also be clear.

And I think you could write this statute instead to say that models that, based on testing and the person’s judgement and all things considered, are dangerous — but then it would be unclear about what models are covered, and you’d be back at the conversation we had earlier of whether startups are covered by this bill.

I think there are important false negatives of models that are not covered by this legislation that are, in fact, dangerous. I think that’s just definitely true, and that is an objection that I’ve heard people raise that I think is quite fair. But I just think that at some level…

One of the reasons why I like the $100 million is that if a model is incredibly cheap to train, I’m just not that optimistic about our ability to prevent its proliferation. I just think that if a model can be super dangerously made for $10,000, that regardless of what California state liability law says, it’s just going to be out there. So trying to focus at the level of resources where something like California state law can shape behaviour, I think you want to place a target where you actually think that you can have a real impact. So I think that’s partially a recognition of not only kind of where the risk is, but also where policies like this can have an impact.

Luisa Rodriguez: Yeah, I buy that. And I also completely agree that it makes much more sense to have a uniform standard, and a line has to be drawn somewhere. So any line could seem arbitrary, but is still valuable. Can you say more, though, about the justification for the specific values chosen?

Nathan Calvin: Yeah, happy to do that. The 10²⁶ FLOP is the same level that the Biden executive order picked. And for folks who aren’t familiar, it basically is saying the next generation of AI systems. The largest systems that we know the public amount of how much that they were trained on today is 10²⁵ FLOP — of things like GPT-4 and estimates for Google Gemini and things like that. I think the difference between the GPT series is two orders of magnitude in FLOP per different generation. So this is talking about the next generation of models, I think models that we expect will come out sometime in 2025.

I think to some degree the intuition for this is that we have pretty good evidence that models trained on the current amount of FLOP are not incredibly dangerous, and as we have models that are trained on lots more than that, we should be keeping our minds open to the potential they could be dangerous. That doesn’t mean that they will be. You know, I think that there were people when GPT-3 came out who were worried about GPT-4, and then it turned out that GPT-4 does not actually present these concerns and does not provide a big step up in terms of bad actors’ ability to do really dangerous things.

But we just are training models that cost a tonne of money and are 100 times bigger, and the people making them are saying they don’t know what they’re going to be capable of or what they’re going to be able to do. So that seems like a fine point of just saying that this is the level at which we’re uncertain, and which we should be kind of keeping our minds open to these possibilities.

Luisa Rodriguez: Yeah, I find that compelling. Does it feel at all plausible to you that there will be algorithmic breakthroughs that mean that we could get similarly capable models with much less in the next year, and therefore this bill is just going to super quickly be out of date and not very useful?

Nathan Calvin: I think this is a point of like, why couldn’t the bill be stronger? And again, given the amount of controversy the bill has courted, I’m not sure that doing that is necessarily something that could happen. But I think it’s a fair point. I do think it relates a bit to the question I was saying before: if it’s the case that you have really dramatic algorithmic improvement such that you can train capabilities really cheaply, I just don’t think any state law is really going to do the trick.

Part of this is like making a bet on worlds where it really is about very large, expensive systems — and where you’re going to have some algorithmic improvements, but it’s going to be on the range of it improves efficiency by, I forget the Epoch estimates, but a single-digit multiple of efficiency per year or something, rather than you suddenly get a 10,000x or 100,000x improvement just based on algorithmic improvements. I agree that in worlds where this happens, this bill is not going to do that much. I think that’s fair to say.

Luisa Rodriguez: Cool. I buy that. I like the idea of making bets for different worlds, given how uncertain we are about how AI is going to play out.

That kind of relates to another objection, which is just that it’s too early to regulate AI at all in this way. We just don’t know yet what the really worrying versions of this are going to look like. Does that sound plausible to you?

On the one hand, I can imagine thinking, why not do both? Planning for both worlds seems better than planning for one or none. On the other hand, it does seem totally plausible to me that political will and all of these resources going into passing this bill is finite, and you’re kind of using it up or spending it on a version of a bill that in six to 12 months we might realise should have just looked really different. Does that feel fair to you? Do you worry about that?

Nathan Calvin: I think there are a few things to say. I do think that this bill is pretty robust to a lot of different scenarios in ways that I think are important — where it is not saying in a tonne of detail exactly what precautions folks should take; it’s saying things like, “If NIST puts out standards, take those into effect; take reasonable care; think about cybersecurity.” I don’t think the idea of cybersecurity is going to be invalidated in two years or something. So I think that part of the language in the bill is deliberately trying to have some flexibility and uncertainty.

To your point about whether we’re using up finite political capital at the wrong time, I’m uncertain. I do tend to think that’s the wrong model of it, where I feel like it’s more like a muscle that you build. And there are things like SEIU supported this bill, and there are parts of this coalition who have come out for this bill that we didn’t know would be in support — and now they exist, and they will exist in the future.

We also brought this bill through the process, and kind of tested it and improved the language and showed that there is this appetite. At the same time, our opposition has also strengthened their own muscles and the coalitions that exist on their end in a way that will also have additional effects.

But, I think that, like, you know, there’s a recent example of Senator Wiener recently finally passed a bill, where it used to be that when there were car break-ins in San Francisco, the police had to show, in order to prosecute someone, that the door of the car was locked at the time. That’s very hard to prove. How do I prove my car door was locked when they broke in? And anyway, it seems like a fairly common-sense thing that you shouldn’t have to prove that your door was locked when your car window gets smashed. But it took, at least introducing the bill, I think three times at least, maybe more than that. And it finally passed this year.

And it’s just that I think there are issues that are far more straightforward than this that we’ve got to try a lot of times, and got to put it through the process and do this. I just think that in general, the idea that you wait for the right moment and then you kind of pull a proposal out of the ether and it becomes law is just not how this process works; it is much more about trying to move things into the relevant set of decision-makers. And I do hope this bill, for all of its backlash and things like that, is helping with that.

Luisa Rodriguez: That feels compelling to me.

How is SB 1047 different from the executive order? [00:55:36]

Luisa Rodriguez: Turning to another kind of criticism, my impression is that most of what’s already covered in the bill is also covered in Biden’s executive order on AI. So some people are like, what’s the point of the California bill if these regulations already exist?

Nathan Calvin: Yeah. I do think they are just pretty different in a lot of important respects. One thing is just that the Biden executive order is not a statute. And I think we’ve seen from some of the actions that the Supreme Court has taken that they are very wary about the idea of the executive branch kind of exceeding their authority when there is not specific legislation passed by Congress — or some other legislative authority, in the case of the states here.

I also think that just even in terms of what it says on the tin, what it is is companies are reporting the results of tests that they’re doing and talking about whether they’re training certain models. And with those, there is some overlap. But then there are also things like: this bill has whistleblower protections; there’s actually the question of liability in the event that harm does occur; there’s also that the bill has mandatory third party auditing — but again, it’s not something that’s in the AI executive order.

So I think there’s some overlap and some continuity with it. I think part of it is that, whether there is a change in administration or a court at some point in the future decides that they don’t have authority to do parts of this, I think that you’re just better off putting things in statute: they’re just going to have more longevity than relying exclusively on executive orders in order to accomplish some of these policy goals.

Luisa Rodriguez: Yeah. Is there a good case, though, that these kinds of regulations should be done at the federal level as opposed to the state level?

Nathan Calvin: I think that would be much better. You know, I worked in the US Senate for a year, and I would very much love for these regulations to happen at the federal level. And it’s worth saying that if, at some point in the future, Congress does want to act in this area, they can preempt state law and they can invalidate 1047 and say that we’re going to have a uniform standard at the federal level. I think that would be great.

You know, Congress has still not passed a data privacy bill. They’ve been saying they’re going to do it for quite a long time. It has not happened. And there are lots and lots of other domains where Congress has struggled to act in a timely fashion, and that states like California have moved forward on. So again, I think it’d be great if it happened at the federal level, but I just don’t see it happening in the near future.

Luisa Rodriguez: OK, so it’s a case of better something than nothing.

Nathan Calvin: Yeah.

Objections Nathan is sympathetic to [00:58:31]

Luisa Rodriguez: Are there any other objections to the bill that we haven’t talked about yet? I’m also just curious if there’s anyone whose judgement you really respect who has serious problems with the bill? And if so, the most respectable, serious, careful thinkers that you can think of, what do they think is wrong with it?

Nathan Calvin: I respect Jeremy Howard a good deal. Some of the folks who really feel strongly there should be zero regulation at all in open source development, they will say things like, “I just know for a fact there will never be any extreme risks, and that’s why I think that’s fine.” That’s not a very reasonable position to me, and seems like just a level of certainty that seems pretty misplaced given where this technology is at.

I don’t think Jeremy Howard says that. He’s written more eloquently about it than I’ll be able to easily describe, but I think he just thinks that the way to improve the world is for everyone to have access to this technology, and be able to improve it and understand it. I think he just also has a very strong philosophical belief that releasing an open weight model is effectively publishing a very long list of numbers on the internet, like the weights. And I think he has effectively a philosophical belief that government policy should not be able to stop someone from publishing a long list of numbers on the internet: period, full stop.

I used to work at the ACLU for a summer, and I definitely respect some of these intuitions of these hard lines. And I think recognising that, if you’re wanting to do something that is restricting something that’s useful and important, of there being a really high bar for that. I just think that when you have these really long lists of numbers on the internet, which can hack into the electric grid and shut it down or do crazy stuff, I just don’t think that people are necessarily… I feel like it’s going to be, you know, “Plutonium is just a combination of molecules” or whatever.

But again, I do respect him, and think that he is honest, and I think that he is not saying that it’s impossible that this could cause risks. I think there are other folks similar to that who are really saying that yes, these risks could be real, but that just any restrictions at any point on open source development is just not the way to best deal with those risks.

I disagree with that, but I think that these questions are complicated. And this bill, one of the reasons that I like the fact that it now has this $100 million threshold is that this would only actually cause damage to open sourcing instances where this model is above $100 million, and they do testing and find things that could cause a catastrophe and don’t have ways to open source it in a way that prevents people from using it — which is a pretty narrow circumstance of cases where open source releases are implicated.

If you just think about what the most powerful open source model is that is not covered by this bill, of a model that costs less than $100 million to train, it’s going to be a really powerful model each year, and it’s going to get a lot better each year. There are going to be extremely powerful open weight models that are not touched by the bill whatsoever.

I think that another example of this is Vitalik [Buterin], who is also a big proponent of open weight development. He likes the bill, and says that he thinks it’s reasonable and good. He thinks that you should be making bets that keep open the possibilities and the goodness of open source, but you also shouldn’t be completely exempting it from everything.

And again, I think open source is really good, and I’m really excited about it. And I honestly think it’s not how it’s landed or what the reaction has been, but I genuinely think that this bill is taking a quite nuanced approach to open source development that recognises that it has a tonne of benefits. There are folks who are thinking about that who are fans of it. And these are complicated issues and there still is some room for disagreement, but I really think that there is a lot of nuance here that I feel like is being lost.

Current status of the bill [01:02:57]

Luisa Rodriguez: OK, let’s leave that there. What is the status of the bill right now? I guess we should flag that we are recording on August 19, so things might have changed by the time this comes out. But as of August 19, what’s the status?

Nathan Calvin: We’re in the final stretch. The way the California legislative session works is that all legislation needs to be passed out of both houses of the legislature by August 31, and then needs to be sent to the governor’s desk for him to decide whether to sign it or veto it, and he makes that decision by September 31. So the bill is going to be up for a vote probably when this podcast comes out. Maybe it’ll already have happened or be extremely imminent. We’ve made it through all six of the policy committees, so all of those have been cleared. It’s passed through the Senate.

Now the question is: can it pass the Assembly, be back to be reconciled in the Senate, and then sent to the governor’s desk for a signature? So we’re really right at the critical juncture.

Luisa Rodriguez: Right at the finish line. How is it looking for the next steps? Assembly and then back to the Senate?

Nathan Calvin: We’re really just not taking anything for granted. I think that the bill has actually, despite all of the backlash online, proceeded in a very effective way through the legislature — where it’s passed with pretty commanding margins and been received quite well by the policy committees who’ve engaged with the substance and details of the bill. I feel cautiously optimistic about us making that out of the Assembly.

We’ll see. I mean, obviously, there are ways in which it is becoming increasingly evident that this is not a normal piece of state law in terms of what the reaction has been from opposition. We’re just not taking anything for granted. But at the same time, we feel as good as we can about our position, given all the craziness. We’re trying to focus on what is in our control and just taking each of these steps one at a time.

How can listeners get involved in work like this? [01:05:00]

Luisa Rodriguez: If anyone wants to get involved in this kind of work, is there anything they can do?

Nathan Calvin: Yeah. Some of the things for folks who do support the legislation: if you’re in California, and you want to call your representative and say that you support the legislation and the reasons why, I think that really actually does matter. Also, I think the discussions online do also matter and are affecting people’s perceptions of this. And insofar as you want to weigh in and say that you think some of these criticisms are unfair, or that this is like a pretty reasonable policy that is engaging with the real uncertainties that exist here, I think that is also super helpful.

And then I also just think that doing more work in the AI policy space, particularly at the state level, is something that I would love to see more people get involved with. I think there’s a lot of attention paid to what the president is doing, what Congress is doing, but that states have a really huge role to play here. It doesn’t even have to just be states like California: in general, states have a pretty broad ability to regulate what products are sold within them and to establish things like liability rules.

If you’re someone who wants to get really involved with politics in some state that doesn’t have a big AI industry, I think there still are things that are pretty relevant in terms of conveying this expectation to companies: that if they want to do business in a jurisdiction, that they have to be taking measures to protect the public and to be following some of the word of what they’ve laid out in these voluntary commitments.

So I think that’s an underrated area that I would love to see more people get involved with. And in some ways, it’s almost been like too much, but my experience working at the state level versus at the federal level is that there just is a lot more opportunity to move policy. And I really think that it’s an exciting area that more people should think about seriously.

Luisa Rodriguez: Yeah. I’m curious about this underrated thing. Is there more you want to say about what makes state policy particularly underrated that our listeners might benefit from hearing when thinking about policy careers?

Nathan Calvin: One interesting thing that also kind of relates with another thing that I know some 80,000 Hours listeners care about is California passed this ballot initiative, Prop 12, which says that pigs sold in the state can’t be held in these really awful inhumane cages. And it included pigs raised out of state, and there were lots of agricultural and pork lobbyists who sued California. It went up to the Supreme Court, and the Supreme Court decided that California does have a legitimate interest in regulating the products that can be sold in its state, including activities out of state.

So I do think that it’s just a thing where there is often this reaction that the industry actually needs to be within that state itself in order for state regulation to have any effect. But there are questions about how it needs to be proportional and it can’t be protectionist; there are different things, and it was a close decision.

But I think in general, states have more power than they themselves realise. And I think there’s some reaction of like, “I’m a legislator in Montana or something; what am I going to do that’s going to be relevant to AI policy?” Companies want to offer a uniform product that’s the same across states. There’s some question of, when it’s a smaller state, there is some level at which if you push too far, then maybe it’s easier for a company to say they’re not going to offer their product in Montana than it is in California. And so you need to [act] based on your size.

But I do think it’s an interesting thing that states like New York or Texas or Florida, very large markets… I think the reason why California is doing this is actually in some ways more about the size of the market than actually about the fact that we have the AI developers physically located in the state, is I think where the power of this is coming from. And I think that’s something that has not really been understood by a lot of the observers of this.

Luisa Rodriguez: Right, right. You might be just as happy with New York doing it because a bunch of the impact comes from affecting the kinds of products that can be made and sold everywhere. As long as the AI companies aren’t willing to lose business, all business in New York, they’d have to change their process kind of globally.

Nathan Calvin: Yeah, that’s right. One thing just briefly to add is that there are parts of this that are relevant to the state itself — like the whistleblower protections, and things about the labour code for folks working in California: that is something where the employees have to be based in California for California law to apply. And there’s some complicated things where there are certain court actions where if it’s before a harm has occurred and you’re taking action against a company, which jurisdiction it is can matter.

But we’re talking about percent changes, not like orders of magnitude. And I think in general, if a state is a large market with lots of consumers that a company wants to access, they have leverage here.

Luisa Rodriguez: That’s super cool. I find that really motivating. I think something like that idea — that when a particular jurisdiction regulates something, depending on how big the changes are that are required for that company to meet those new regulations, that it just makes the most sense for that company to change their whole supply chain for all around the world, even though it’s just changed in that one jurisdiction — I think that came up with Markus Anderljung in our interview maybe a year ago. I think maybe it’s called the Brussels effect.

Nathan Calvin: Yeah, that’s right.

Luisa Rodriguez: Nice. So if you’re curious, it might be worth listening to that episode as well. But yeah, I do feel like this is just an incredibly underrated fact about the world, that when you change policies in some places, even small ones, they can have much wider ramifications than you might guess.

Nathan Calvin: Yeah, absolutely. I think it’s definitely right.

Luisa Rodriguez: Cool. OK, that is all the time we have. My guest today has been Nathan Calvin. Thank you so much for coming on.

Nathan Calvin: My pleasure.

Luisa’s outro [01:11:52]

Luisa Rodriguez: All right, The 80,000 Hours Podcast is produced and edited by Keiran Harris.

Audio engineering by Ben Cordell, Milo McGuire, Simon Monsour, and Dominic Armstrong.

Full transcripts and an extensive collection of links to learn more are available on our site, and put together as always by Katy Moore.

Thanks for joining, talk to you again soon.