#258 – Luke Muehlhauser on the most important AI safety work nobody is doing

Transcript

Cold open + a note from Luisa [00:00:00]

Luke Muehlhauser: I think most of the institutions that need to exist don’t exist. AI is going to be the most transformative thing this century, maybe just in human history. Prima facie, a large fraction of global effort should be going to making sure this goes well: safety, governance, et cetera. We have almost none of that.

Luisa Rodriguez: Hey listeners, a quick note before we start.

We recorded this with Luke in August — after OpenAI’s first disclosure about the Hugging Face hacking incident, but before METR and Redwood Research published their independent investigation.

A few things from that report are worth having in mind as you listen.

The one that stuck with me most: some of the AI agents involved in the hack tried to tamper with their own activity logs, faking tool calls and trying to edit their transcripts after the fact. But the investigators could clearly see that, because they could read all about their plans and reasoning in their chains of thought. OpenAI says that if it had been monitoring those, its security team would have been alerted more than a day before the attack. But that only works as long as we can still read chains of thought.

In this episode, Luke makes the case for an organisation that does nothing but track whether that’s still true. This now seems like a top priority to me.

The other thing that struck me was how the investigation itself played out. The independent researchers got just six days on site at OpenAI. Luke flags exactly this in the episode: compared to other high-risk industries like aviation and nuclear, the way we review AI safety incidents is improvised and unregulated.

Since we recorded, there have been more incidents, at other US frontier labs and some in China. Several frontier researchers have publicly quit, which has pushed AI safety into the mainstream. And there are growing calls for a pause or to pace frontier development.

Listening back, I’m struck by how much of this Luke saw coming: warning shots at roughly this pace, more people inside the companies getting nervous, and the political conversation shifting.

One last thing: Luke mentions he’s working on a big funding call for new AI safety orgs. That’s now launched: it’s called Project Tailwind, and it lists 36 specific projects they’d like someone to build. There’s a link in the show notes.

OK, on to the interview.

Who’s Luke Muehlhauser? [00:02:27]

Luisa Rodriguez: Today I’m speaking with Luke Muehlhauser.

Luke is the managing director of Coefficient Giving’s AI Governance and International Policy team. He’s been at Coefficient Giving, or CG, for a little over a decade. Before that he was the executive director at Machine Intelligence Research Institute, or MIRI.

I think you might be one of the most requested podcast guests that we’ve ever had. So we are really happy to finally have you on. Thanks for coming on the podcast, Luke.

Luke Muehlhauser: Thanks for having me. I’ve been a long-time fan of the show.

Luisa Rodriguez: Nice. That’s really nice to hear.

Before we go further, I should note that CG is one of 80K’s biggest funders — so conflict of interest flag. Anyways, I think we should dive in.

The state of AI safety right now [00:03:07]

Luisa Rodriguez: You have been a full-time grant maker in AI governance and safety for basically longer than anyone I know — since 2019. And my sense is that you have an unusually broad perspective on this whole field. What is your sense of the situation at the moment?

Luke Muehlhauser: There’s a lot happening. It’s hard to keep up with. Maybe I’ll emphasise a few things.

On the optimistic side, I would say that the Overton window for what’s possible to do — both at the companies and with policymakers — is really changing quite a lot, and it’s opening up a lot.

I think the biggest thing there was the announcement of Anthropic’s Mythos model: they kind of accidentally created the greatest cyberweapon in history just by way of creating the best model in history. This is a model that can find new security vulnerabilities in a 27-year-old operating system called OpenBSD that’s designed specifically for security. So that has obviously serious national security implications, and that really woke up a lot of people.

I think also the hacking incidents — where OpenAI models autonomously decided to hack out of their secure sandbox and get out on the internet and do a bunch of crimes and so on — is also affecting where the Overton window is. TBD exactly how much that shifts things.

But in response to the Mythos announcement, the White House went from, “AI safety is BS and it’s just a marketing ploy and it’s a plan for regulatory capture,” and so on to like, “OK, we have a de facto licensing regime, we’re choosing which customers get access to which models in which order and when it can be released to the public” and stuff like that.

Another piece of good news is that I think the AI safety community really has racked up quite a few wins — they’re far short of what’s needed, but there is a lot of concrete progress. Maybe I can say more about that later.

On the pessimistic side, I would say most institutions and projects and so on that need to exist to manage the situation well, don’t exist. And there’s nobody working on them at all in the world. So that’s not a great situation. I think the funding actually exists, which is great, but there aren’t enough funding allocators — like grant investigators and investment people — and there aren’t enough founders for those projects. So we need a lot more people to pivot into AI safety and governance and do those two things.

I think it’s really urgent that people pivot into AI safety. AI is already going crazy fast, and now a lot of the company’s employees and leadership say that they think they’re maybe a few months or maybe a year or two away from automating AI R&D to a substantial degree, so that’s going to go even faster then. I think that’s why a lot of them signed the “Pacing the frontier” letter where they say: “This is going too fast for us and we need help from the government to find a way to slow down collectively.”

Now we’ve seen the risks materialising in the real world as well, like the Hugging Face hacking incident. So I think it’s just really urgent for people to transition into the field and start up these new projects and make the current projects more capable and so on.

Luisa Rodriguez: OK. It sounds like you think a lot of the institutions that should exist to make AI go well just completely don’t exist. Can you talk a bit more about that?

Luke Muehlhauser: Yeah, I think most of the institutions that need to exist don’t exist.

If you just think about this in the big picture, AI is going to be the most transformative thing this century, maybe just in human history, prima facie a large fraction of global effort should be going to making sure this goes well — safety, governance, et cetera. And we have almost none of that. There’s a lot that needs to be built.

One very small example is the US has this AI Safety and Security Institute, now called CAISI, and its budget is $15 million a year. Some comparison points:

• That’s the size of a public library in a small- medium-sized city.
• Another comparison I looked up was it’s 25 times less than the Pentagon spends on marching bands every year. And I like marching bands. I actually listen to marching band music. But that’s a comparison I don’t agree with.
• In the UK they have their own AI safety institute, the AI Security Institute. That I think is $70 or $80 million dollars a year or something like that. If the US’s budget for its AI safety institute was proportionally the same as the UK one, given the size of the US budget, then I think the US one would be like $850 million a year — way, way bigger than it is now. That feels a little more appropriate to the scale of the problem, to me personally.

I think the institutions that do exist are much smaller and much less staffed and so on than I think should happen. Then actually most of the institutions that would make sense just don’t exist at all in any form, or at least not in most jurisdictions. When you compare to other domains — like aviation or nuclear safety or pharmaceuticals or what have you, whenever we have a high-upside but high-downside new technology or domain or whatever — there’s usually like eight or so different kinds of institutions that come into existence to mitigate the downside risks in these domains. Almost all of them don’t exist for AI.

For example:

• There’s the independent evaluator or auditor function — like the FAA in aviation or the FDA in pharmaceuticals.
• There’s also this kind of peer review operators club: they have one of these in nuclear safety; there’s FINRA in the banking industry. Something like that doesn’t really exist in AI right now.
• There’s often also an incident investigation function, like the NTSB for aviation, where they look into every plane crash and figure out how it happened and what lessons can be learned from it.
• There’s often a verification function, like the International Atomic Energy Agency in nuclear safety and some other things.

And those are just the government or government-adjacent institutions that are built up. To do those, you can’t just be a founder and decide to create that; you have to do policy development and policy advocacy and stuff like that. But that’s a whole category where most of the institutions haven’t been built out.

The nonprofits Luke most wants to see founded [00:09:58]

Luisa Rodriguez: OK, so that’s a bunch of government institutions that really should exist for AI and sounds like mostly totally don’t.

Are there projects that you’d like to see that a person with a relevant skill set could found — that you don’t have to be in government, doing things with government mandates?

Luke Muehlhauser: Yeah, yeah. Probably hundreds. One whole category is you can often trial run a lot of these things that are normally government functions in the private sector as nonprofits or whatever, and then just test out some of the functions and so on — and then the things that are working best can be adopted later into government. So that’s a thing you can do.

But also there’s just tonnes of things that naturally can be a company or a nonprofit and can be very helpful. We’re planning to soon launch a Y Combinator–style call for startups thing, where we kind of spell out a bunch of the projects that we think should exist to fill various gaps in the ecosystem. And then hopefully potential founders will fill out a form and let us know that they exist and might be interested in starting something.

A few examples that I think will probably be on that list (it might be up by the time this podcast goes live):

One is for the Mythos for bio threat model, where a future model is able to autonomously design and maybe develop a novel pathogen or something like that. There it would be great to have an organisation that’s doing onsite red-teaming of the safety and security protocols of the companies and organisations that would matter most there — like biosafety labs and DNA synthesis companies and the companies that keep copies of pathogens and stuff like that. So there’s this big organisation that could happen there to attack that specific threat model.

Luisa Rodriguez: Cool.

Luke Muehlhauser: A totally different thing that’s much more general is an AI character evaluator organisation. Because it’s this weird thing where AIs, the way they’re built now, they actually have a moral character, and they differ from each other.

The character is things like: how strong is its tendency to lie, when that’s a convenient thing to do? Or where does it seem to be on the political spectrum? Or, when you ask it to help you seize power or do some crimes, does it help you or not? There’s just a lot of AI safety that comes down to something kind of like moral character. So we kind of have to do the equivalent of human behavioural science or psychology on these models and figure out what their behavioural tendencies are.

This would be an organisation that sort of evaluates:

• First of all, does the behaviour of this latest model and does its character and habits and so on conform roughly to the model spec that the AI company was trying to make it conform to? That’s one really big question.
• Then: how is the character of the models changing over time, and what can they do about that? What training choices make certain differences to the character of the AI?

Luisa Rodriguez: Yeah. That one I find interesting, because I think even one-year-ago me would have heard that and been like, “Really? Is that a thing that is actually going to make AI safety better? It sounds kind of silly.” I think partly I didn’t fully have the intuition that the models really varied in character.

Luke Muehlhauser: I think they vary a bit more now than they did a couple of years ago. I think there’s more variety in the training processes between Anthropic and OpenAI and DeepMind and Grok, for example. There’s different things that those companies are even aiming for.

For example, Anthropic’s approach is much more like character and values and a self-consistent persona of an AI, or something like that. OpenAI’s approach is more deontological: here’s a list of rules; you have to stay within these bounds. There’s debate about which approach is better. But it also leads to significantly different character per se, when you put them in different kind of high-stakes situations.

Luisa Rodriguez: You also mentioned political leanings, and I think it just wasn’t clear a couple of years ago that there were going to be big variations. It seemed like maybe they’d try to kind of stay neutral. And increasingly it just seems clear that even though it’s kind of subtle, the variations are there and meaningful, and it matters not only to societal stuff — like are they going to be epistemically virtuous — but also to safety stuff.

Luke Muehlhauser: I think it matters for things like how likely are they to hack out of their sandbox and do a bunch of crimes on the internet.

Luisa Rodriguez: Totally. Yeah.

Luke Muehlhauser: That’s substantially a question of moral character, per se.

Luisa Rodriguez: Are there any other projects worth highlighting?

Luke Muehlhauser: Yeah, maybe I’ll mention one more. One we would love to see is a chain-of-thought monitorability observatory.

When AIs reason and take complicated actions over many steps, they have these chains of thought where they’re thinking through, like, “OK, the user asked me to do this, and now I’m going to take this next step,” and, “OK, I did this search and came up with that. So maybe the first thing I’ll look into is this other thing,” and blah, blah.

I think being able to monitor those chains of thought gives us one way to notice is it thinking about doing some crimes? Maybe we should flag that or shut down that chat stream or whatever. So it’s a safety intervention.

But there’s a risk, and a little bit of this has been observed so far, that the chains of thought may become less monitorable — because it is more efficient for the AIs to kind of speak… Right now, they often speak in this very dense version of English. It’s hard to interpret, but we can kind of interpret it.

There’s a worry that there will be optimisation pressure for how you train systems, that they might eventually slip into something that’s sometimes called “neuralese,” where it’s a pretty different language and we have no idea what they’re talking about. And then we can’t monitor. Like, “Are they talking about doing a bunch of crimes on the internet? I don’t know. They said the word ‘marinade.’ What does ‘marinade’ mean?” That’s a real example.

Luisa Rodriguez: Wow. Interesting.

Luke Muehlhauser: Another example is they’ll refer to the humans who might be looking at their evaluations as “watchers.” They’re like, “We have to do this to satisfy the watchers.”

Luisa Rodriguez: Wow, that is so sci-fi.

Luke Muehlhauser: That is this terminology. It’s still mostly pretty monitorable. But there’s a bunch of reasons to worry that it might become less monitorable over time.

So this organisation would pay attention to:

• What are the different AI companies doing on chain-of-thought monitorability?
• How monitorable are the chains of thought that the different models are producing? Is this changing over time?
• How does the monitorability of the chain of thought vary in response to different training setups or different post-training setups or whatever?

So a mix of evaluating, auditing, monitoring, paying attention to what’s going on here — and also a bunch of novel science to figure out what different factors affect the monitorability of chain of thought, so that we can preserve chain-of-thought monitorability as an important safety and oversight mechanism.

Luisa Rodriguez: Cool. I have a couple of questions. One is: what is marinade?

Luke Muehlhauser: Nobody knows.

Luisa Rodriguez: Really?

Luke Muehlhauser: Yeah, no. It’s in a report from METR where they were doing some investigation of some OpenAI models. And that was just one of the terms that would often show up in these chains of thought, and nobody from the company or METR could figure out what it could mean. Are they using it as a codeword for something that we can’t understand? It’s just like “marinade” in the middle of the sentence. We don’t know why. It’s a little bit worrying.

Luisa Rodriguez: Wow, that is pretty insane. My other question: do we know how much companies are monitoring chain of thought now?

Luke Muehlhauser: It’s interesting. For example, OpenAI and Anthropic and so on talk a lot about chain-of-thought monitorability and its importance and so on.

But when these kind of autonomous hacking incidents happened, I don’t know if they weren’t doing chain-of-thought monitoring at all, or just chain-of-thought monitoring was not very effective. Because there were these swarms of agents that were discussing, in English, on a message board they created: “Here’s how we’re going to hack out of the sandbox and do a bunch of crimes on the internet.” You would think that any chain-of-thought-monitoring system would pick that up and flag it to somebody — but they didn’t discover it until after Hugging Face got hacked into and they called the FBI.

Similarly, when Anthropic went back and looked for stuff, they were like, “Oh, some of our agents did some hacking and stuff as well that we didn’t know about for months.” So are they doing chain-of-thought monitoring? I don’t know, maybe not. It’s a little unclear. That’s one of the things I want this organisation to do is to pay attention to whether the companies are actually doing any chain-of-thought monitoring.

Mid-career pivots that paid off [00:18:58]

Luisa Rodriguez: These sound again like really cool projects to work on. They also seem plausibly very expensive and like they could use a lot of people.

But my impression is that you think they’re not bottlenecked by funding; that you’re bottlenecked by allocators to figure out who should be doing these projects, and by the people who would at least start the projects. Can you talk a bit about that?

Luke Muehlhauser: I think if people come forward and want to do these projects, they’re the right fit for it, I think the money exists. CG has a very large budget. The OpenAI Foundation exists. Nan Ransohoff from Stripe wrote this post, called “The third wave of American philanthropy,” about how there seems to be like hundreds of billions of dollars of philanthropic capital that’s coming online in the next couple years. I think she’s right about that. I think approximately all of those people care a lot about AI safety.

So I think the money exists, and what the bottleneck is is: number one, the grant investigators, investment investigators, et cetera, at places like CG and OpenAI Foundation, Longview, and so on; then the founders.

You can convert more allocators into more founders because allocators can do founder search. So we are pivoting to more of an active grantmaking focus — where instead of receiving applications for funding and then evaluating which ones are worth making, there’s all these projects that we just keep not getting applications for, and so if we want them to exist, if we want to fill the gaps in the space, we’re just going to have to write down what kind of project would fill the gap, and then shop it around to a bunch of potential founders. So that’s talking to dozens of people before you find the right person. That just takes a really large amount of time.

But you can do it if you have the staff at places like CG and so on to do it. I want a lot more staff to do that kind of thing. We also need a lot of founders to step up and decide to run a big project, quit their job, take some money from us, and launch these projects.

Luisa Rodriguez: I have a couple of questions. One is: actually I think that I’ve heard some people express something like, “CG is a super-well-respected place, it has loads of money, it pays well. Surely it’s not actually that bottlenecked by talent. So they’ll find people to fill their roles and I’ll do something else. It’s a little suspicious that you’re like, ‘My roles are the biggest gap in what people should be doing.'”

What’s your response?

Luke Muehlhauser: Yeah, I get where that’s coming from. A few things I’ll say are: number one, we are competing for talent with the frontier AI companies that are offering unprecedentedly valuable equity and stuff like that. Then we’re also competing for talent with governments who don’t pay very much money, but offer the opportunity for very direct [work]: put your pen to paper and write the regulation or whatever. There’s a lot of competition for very scarce talent.

I think also, though, a big factor is that I think a lot of different forms of direct work at companies or governments or nonprofits or whatever is sort of clear, and it’s easy to get excited about it and so on. And people just don’t really have a sense of what being a grant investigator is. It feels kind of meta and abstract, and they don’t know what the day-to-day work looks like. Maybe people don’t know about this active grantmaking thing. It’s not just choosing applications that are coming in. I think part of it is that.

Then part of it is just people not knowing about the craziness of the current situation, where someone can come to one of these funders and — not certainly, but very plausibly — move like $100 million in their first year, because there’s plenty of money and not enough allocators. That’s enough funding to fund 100 or so people full-time for multiple years.

So: is the direct work thing that you would do with your time actually more impactful than like 100 people working full time? Sometimes it is, but usually not. I don’t think I could do that. I don’t think I could find some specific thing to do that would be better than 100 new people working in AI safety and governance.

I want people to just sit with that basic multiplier argument and consider whether their specific thing is actually more impactful than 100 full-time people.

But we also need to proactively go out and find them, like I was saying earlier. I think every year more and more people are kind of waking up to the AI issue, and they’re like, “Gosh, this is getting really serious. Maybe I should do something about it.” And yes, you should. So we want to fund a lot of programmes that give people context, make it clear how to transition into the field.

We actually have an entire fund [now run by BlueDot Impact] that’s just the career transition fund. If you just Google “Coefficient Giving career transition,” you can apply for funding for like several months of your current salary or thereabouts to just quit your job and spend a bunch of time learning and talking to people. and think about how you can pivot into the space, without even having a concrete thing. So you can do that, you can do these fellowship programmes, you can do these context accelerators we’re running. There’s a lot of ways to get into the field.

Luisa Rodriguez: For people listening who are like, “OK, it sounds like it’s either allocator or founder,” I don’t think that many people… Maybe this is wrong, maybe it’s typical-mind-y, because I’m like, “I’m not a founder.” But the people that you expect will do the founding, do you think that they all are obvious founders?

Luke Muehlhauser: To be clear, most people who can make an impact are not a founder profile or allocator profile. Those are just two of the biggest bottlenecks that will unlock the most new activity. You can go to the 80,000 Hours job board and find many great AI safety roles you can immediately plug into. So that’s really great.

I think a lot of people kind of know whether they feel like they could be a founder of something. But I think people probably underestimate this.

One of the interesting lessons from my over a decade at CG is we’ve often taken a risk on very young people who really don’t look like founders — they look like technical researchers who just did like a computer science PhD or something — and took a risk on them to start some big new organisation. A thing we’ve learned over time is that that has gone better than we expected. People somehow find a way to run an organisation, and they get support around them — they hire experienced operators to be the COO or whatever. If they’re just pointed at the right problems and they’re generally smart, they end up doing a lot of really impactful stuff.

So some of these organisations that we were kind of like, “Oh, is this gonna be a disaster? Are they gonna screw up a bunch of things because they don’t have any experience? Let’s take a shot at them, give them a few million dollars”: now they’re running an organisation that has a budget of like $20 million a year and it’s going really great.

So I would encourage people to think maybe you could be a better founder than you think you can.

Luisa Rodriguez: Another thing that I feel like I hear pretty often is: AI has really taken off, lots of people know about it and have already transitioned to work on it. Lots of people have already moved to the Bay — “If you haven’t moved to the Bay, it’s too late.”

To what extent is it true that if someone hasn’t already made a transition with their career to work on AI, it’s too late? Or if they haven’t followed it enough to be super knowledgeable, it’s too late?

Luke Muehlhauser: It’s definitely not too late. Many people, I think, enter the field and are having quite a lot of impact almost immediately. This is another thing we’ve learned and been surprised by.

You do need to get some context. You need to read some stuff, talk to a bunch of people, get more context on the details of what’s going on. That’s what these career transition grants are for, to give you the time to do that.

It’s quite common for someone to have 20 years of experience in some field that’s completely unrelated to AI — they’re doing strategic communications or they’re doing infosecurity or they’re doing operations or they’re doing whatever — and there’s a fit for them in the AI safety ecosystem somewhere, and they start having impact basically immediately.

Maybe I’ll give a couple of examples because they’re pretty exciting. Brad Carson, former member of Congress, he was running a university and he started to get worried about AI safety. So he started talking to some people, doing some reading, got some context, and eventually decided to wind down his university job and start this organisation called Americans for Responsible Innovation — which is now maybe the largest pro-safety policy advocacy organisation right now.

He also, six months later, decided to start Public First Action, which is the largest pro-safety super PAC. He’s just running both of them at the same time because he’s an absolute monster. He just decided to transition into the field, and then within six months was like one of the most important people in the field.

Another example I really like is Brendan Steinhauser. Brendan was a 20-year veteran of Republican campaigns and strategic communications, and started to get more worried about AI safety. He talked to some people, read some stuff, got some context, and then decided to create this organisation called the Alliance for Secure AI — and that’s now one of the largest strategic communications organisations in the AI safety ecosystem.

Luisa Rodriguez: Super cool.

Luke Muehlhauser: That’s just incredible experience and skill and so on that they’re bringing to this field and creating new projects that weren’t going to exist without them.

But it also happens for young people. Another thing we’ve learned is that even young people can have an impact immediately after undergrad or grad school, or they do a three-month fellowship right after undergrad or something like that — and then they’re immediately going to a frontier company working on safety there, or going into a congressional office and helping to write NDAA [National Defense Authorization Act] amendments on AI security or various things like that.

We had this [non-public] memo where somebody was collecting a bunch of examples of this, and the title of the memo is something like, “Just keep funding university talent programmes through the singularity” — because there are just so many examples of people being instantly useful even when they’re very young.

A specific example I like is Sneha Revanur — who I think was on the podcast recently. She was 15 when she started Encode, which is this very impactful organisation; they punch way above their weight. I think she was maybe 19 when she started working on state AI legislation, which is where a lot of her impact is now. Not everyone is like a prodigy like Sneha, but a lot of people actually are having impact immediately after undergrad or after a three-month fellowship.

Luisa Rodriguez: That is super cool.

How much did Mythos shift the Overton window? [00:30:38]

Luisa Rodriguez: Let’s go back to Mythos. How much did Mythos shift the Overton window?

Luke Muehlhauser: I think it was a big deal. The White House now has kind of a de facto licensing regime for model releases. I think the Hugging Face hacking incidents and similar hacking incidents from other companies also are having an effect with policymakers. It’s a little TBD exactly how much will change in response to that.

I think it’s very impactful and a big change. It’s also still far short of what is needed. I think many policymakers see it mostly as a cybersecurity issue and they’re not necessarily thinking of it as like, “This is on a trend, and maybe the other things that the AI companies said were coming are also going to come, and we need to be prepared for that.”

I think there are some exceptions — like Bernie Sanders or Nathaniel Moran or Ted Lieu. Various others, I think, are seeing the bigger picture here. I think Scott Bessent actually is seeing the bigger picture. But it’s still way less than we need.

I think also policymakers tend to pass policy in response to some event; anticipatory governance is a lot harder. So in some sense, the Overton window always feels like it’s lagging what’s actually needed.

Luisa Rodriguez: Is there a parallel thing for companies? They’re already very steeped in all of the progress that’s happening in capabilities. But do they also get surprised by things and then change what they’re willing to do or something?

Luke Muehlhauser: Yeah, it’s a great question. There’s kind of a separate Overton window for what the companies are willing to do voluntarily. It’s a totally different situation.

They’re very deep on where AGI is going and how quickly and so on. But they’re still in a very intense race, and they strongly want to be in the lead and so on, so they’re not willing to pay a lot of slowdown costs in order to do better on safety and security and risk mitigation and so on. So there’s a separate Overton window about what companies are willing to do.

I think that also has shown some positive movement recently. I do think some of the folks at the AI companies were surprised by their models hacking out of the container and doing a bunch of crimes on the internet and so on, and didn’t predict that was going to happen. They thought their alignment work was better than that. They thought their security work was better than that.

I think there is a partial reckoning within the AI companies about this, and that’s a positive development. To some degree that’s not a specific incident, but this general awareness is reflected in this “Pacing the frontier” letter where over 1,000 employees, including some of the most senior employees, are like, “This is going a little too fast for our comfort. We wish we could have the option to slow down. We need the government’s help to create the options for how to slow down collectively.” So I think they are feeling some of the danger of the situation.

Luisa Rodriguez: Interesting. I’m interested in whether you feel like we took enough advantage of Mythos as a shift in the Overton window. There’s a quote from an OpenAI employee, Joshua Achiam, who said it was a major failure of the AI safety community that it didn’t produce in advance of Mythos off-the-shelf decision-making guidelines at the level robust enough for government to adopt and have confidence in. Do you think that this was a failing?

Luke Muehlhauser: Somewhat. I think he’s somewhat right. I had sort of also concluded earlier this year that in my opinion the AI safety community was spending too much time trying to make incremental progress with the political will that existed at a given time, as opposed to preparing to make maximal use of the political will that would exist in the future when the various warning shots that happen.

So I’ve been shifting my attention there, I have been encouraging grantees and so on to shift their attention more to this, to prepare for future Overton shifts’ focus. That said, I do also have a few disagreements with Achiam’s post, a few things to say.

First of all, a lot of the stuff that’s happening here is not public — so I do think the safety community had more off-the-shelf ready to go than is widely apparent, I’ll just say. I think also a lot of the most important information that you would use to design a robust and sensible framework for model deployment policies or whatever is within the companies, so the companies really have to be very much participating in the development of this framework. It’s hard for outside groups to fully drive the show here.

That’s why in most other industries, like aviation and nuclear safety and pharmaceuticals and so on, it’s often a company-led effort to come up with the framework. Also, these kinds of thorough, robust — I forget all the words he used — frameworks typically emerge over the course of weeks or months or years or decades after some inciting incident, as opposed to being ready to go in advance, because it’s just hard to predict the whole parameters of the situation and so on.

I would also say this particular Overton shift I feel was especially surprising. I remember being in the trenches of these battles over OpenAI and A16Z lobbying for a 10-year ban on all state AI legislation in exchange for also no federal legislation. That’s the kind of fights we were having. The coalition of child safety advocates and AI safety advocates very narrowly defeated on a 10-year ban on all state AI legislation. And then they tried again like a few months later and it was also defeated then.

Then, just a few months later, we’ve suddenly got the White House like, “We’ve completely changed our mind on AI regulation. We’re now doing model release regulation.” It was surprising. It’s hard to be prepared for that big of a sudden shift in where the battle lines are.

The last thing I’ll say is just that he talked about a thorough, robust framework that you can grab off the shelf, et cetera. Which would be nice to have, but it’s also just a very high bar. The AI safety community was working on things like new dangerous capability evaluations, and advocating for CAISI to be better resourced to develop standards and frameworks for this kind of thing, and safety case templates and stuff like that. So a lot of effort towards that kind of thing, while not reaching the very high bar of a “completely off-the-shelf, robust thing that the government will agree to” kind of thing.

Preparing for the next warning shot [00:37:33]

Luisa Rodriguez: Yeah, I guess it does seem like it would have been really hard to go from the government seriously considering a 10-year ban on regulation to, “Where’s the shovel-ready guidelines?” It just seems a little unrealistic.

Presumably we should expect more shocks like this. Do you have predictions about what they might be and how to be ready?

Luke Muehlhauser: Yes, and I think that’s a very important question. I do think there will be more shocks. I think the Mythos moment and the revelation of the Hugging Face hacking incident were like three months apart. It wouldn’t surprise me if future incidents of roughly that magnitude were coming every three-ish months from here on out, maybe accelerating over time. I think there will be many future moments.

It’s very hard to know which exact order they’ll come in or what exact shape they’ll come in, but just to list off a few:

• You could have AIs that hack systems not just to steal information, but actually stealing money or hacking into hospital systems and some people die or something like that. That could happen. That could create some Overton shifts.
• You could have the “Mythos for bio” moment, where there’s a demonstration that AI can design a new virus that would infect humans and so on. That could shift a lot of policy appetite, especially because in that case it’s not just money being stolen or something, it’s a bunch of people are dying. Also, in cyber, you at least have the step where you can give defenders access to the models in advance and they can patch their systems before the hackers have access. But we can’t quickly patch the human immune system. So the defence situation is worse there.
• Also we could just have an actual pandemic from an AI-created virus or something like that. That would probably create a big Overton shift.
• Then a totally different kind of issue: the jobs issue. The evidence on employment effects from AI is ambiguous right now, but if the US went from 4% unemployment to like 10% unemployment, that would be a huge deal. Politicians would pay attention to that. The citizens would care a lot about that. That could produce a lot of motion.
• If China clearly had a consolidated AGI project and was using a significant fraction of all of its compute for this project or something, that would provoke a significant policy response.

The responses to these different possible warning shots might be productive or they might be counterproductive. We’d like to try to be prepared to help steer it towards more productive responses than unproductive ones.

Luisa Rodriguez: I guess concretely that’s like: we could then have better regulation for things that could be used as cyberweapons. Or we could notice that China has a very advanced model and start racing harder or something, and that would be bad.

Luke Muehlhauser: I do think it’s important that democracies be in the lead rather than authoritarian regimes. So I’m not saying racing is always bad, but responding to the situation intelligently and with a way that’s calibrated to the different kinds of risks is important.

Luisa Rodriguez: Yep, makes sense. Even just hearing you list all those out, I kind of wonder whether maybe the government will just be like, “Oh man, that’s a terrible thing just happened. Let’s build up legislation that prevents that from happening bigger and worse.”

Or I can imagine there being a thing kind of like COVID, where it seemed like people got desensitised to the number of people dying, and general advocacy about this thing felt kind of monotonous. We even had that with the show. We were like, “We want to cover pandemics a lot,” and people were like, “Can you stop covering pandemics?”

Luke Muehlhauser: A bummer. It’s kind of important.

Luisa Rodriguez: It’s kind of important, and there’s lots to say. And people still just got tired of it. Is that something you worry about?

Luke Muehlhauser: Totally. Especially when you think about the warning shots that might come from demos rather than actual harm in the world, you can totally imagine people just saying, “OK, it’s another demo. I’ve seen too many demos.”

Luisa Rodriguez: Are there concrete things that you think about that you want to be better prepared for, or just prepared for these kinds of shocks happening again?

Luke Muehlhauser: There’s a few different categories.

One is obviously specific policy options: having policy options ready to go for different kinds of scenarios. That’s actually the hardest one, because it’s very dependent on what exactly is the shape of the political will that’s going to happen? What exactly was the warning shot? What are the technical details of AI at that time? AI just changes very rapidly, and you might go from a training-compute-focused regime to an inference-compute-focused regime in the course of six months or whatever. It’s hard to anticipate. There’s a lot of that happening. There should be more of it, but it is a somewhat brittle preparatory measure.

There’s categories I think about that are actually in some ways kind of better, more important, and more robust to different kinds of warning shots you might get.

One of them is talent pipelines. “People are policy,” that’s pretty true. If you train up a bunch of people and they have expertise and they’re ready to go, then they can be pulled into whatever the particular crisis or development is and use their flexible intelligence and skill to do whatever needs doing in that moment. So just general government capacity, company capacity, talent pipelines, people ready to go. That’s a robust intervention, and we’ve seen a lot of impact from talent programmes.

Another one is very relatedly just relationships: are there smart AI safety people who have built up trusting relationships where they’ve provided genuinely accurate and helpful information, over the course of years, on probably multiple issues, not just AI safety stuff? So then when the AI event happens, those people reach out to them and say, “Hey, you’ve been helpful and accurate on all these other things. Do you want to help me with this AI safety thing that just happened?” That’s a good thing to do that’s robust to a lot of different kinds of warning shots.

How nonprofits shape policy [00:43:52]

Luisa Rodriguez: A slight change in topic. Some people think that unless you are at a frontier AI company or literally the president of a country with a frontier AI company, then you shouldn’t expect to be able to have much impact.

I know you disagree. Can you talk about why?

Luke Muehlhauser: Yeah, I totally disagree. I gave the examples of Brad Carson, Brendan Steinhauser, lots of young people earlier. If we just focus on government policy alone, I think there’s a lot of impact that’s been had. If you just ask AI, “What are the 10 most important things that have happened in government policy related to AI?,” I think approximately all of them the AI safety communities had a major impact in, in one form or another.

Luisa Rodriguez: That’s really cool.

Luke Muehlhauser: I think it’s a lot of impact. Maybe I’ll just talk about a few examples.

One example would be the sort of fight that we talked about before, over whether there’s going to be a 10-year ban on state AI legislation: some of the companies were lobbying for that, and some nonprofits associated with the child safety community and the AI safety community and small business community and various other groups formed a coalition and advocated hard against that — and it nearly passed but ultimately was determined defeated.

I think that is an example of concrete impact. In particular, it’s a prerequisite for a whole other chain of impact, which is all these state AI bills that have impact, they couldn’t exist if that ban had passed.

Luisa Rodriguez: Can you actually talk about those state bills, and how much they matter? Should we feel thrilled that they exist? Are they just so-so?

Luke Muehlhauser: Specifically, I’m thinking of SB-53 in California and then there was the RAISE Act in New York, and then most recently SB-315 in Illinois.

The first two of those set up a variety of transparency requirements and incident reporting requirements, and in one case, whistleblower protections and some things like that. Then SB-315 in Illinois adopted similar measures there, but also added a requirement for some third-party auditing, which is really nice.

I think these are far from the full framework that we need to successfully govern AI, but it’s the only AI safety legislation that exists in the US. It couldn’t exist without the defeat of the attempted 10-year ban on state AI legislation.

I do think the highfalutin theory of state AI legislation as “laboratories of democracy” is basically playing out here. People tried different things. People tried SB-1047, not enough people liked that. They commissioned a report on how to do that better. Then they basically did what was in the report, and that’s what SB-53 is, and then tried a similar thing in RAISE and ended up in a similar place. Then Illinois added the auditing thing, and interestingly, OpenAI — after lobbying for the 10-year ban on state AI legislation — ended up endorsing the Illinois bill and switching to a different strategy there. So now there’s some industry support even for that bill, even though it was the strongest bill of its form.

So we’re trying out different things. We’re learning what works, we’re learning what there can be support for. The laboratories of democracy thing is playing out.

Luisa Rodriguez: I heard that there was a child safety coalition that was important for flipping Blackburn to oppose the ban on state AI legislation. How do you think AI safety advocates should think about this kind of coalition-y, “coming from a bunch of different angles” approach to getting the policies that AI safety people want in particular passed?

Luke Muehlhauser: I think the child safety coalition was more important than the AI safety coalition in that battle. I think policy and politics is naturally a coalitional sport — so if you’re not doing coalitions, then you’re probably not going to be very impactful.

Fortunately, the interests of several of these different coalitions do overlap quite a lot. For example:

• They all want more than zero regulation of AI.
• They want some basic transparency measures that are useful for all kinds of purposes.
• They probably want some kind of third-party auditing, so that the labs aren’t grading their own homework, so to speak.

There are other things that they agree on. And then there are some parts where they will disagree, very naturally — but they can collaborate and be more impactful where they do agree, so that’s just a very natural part of policy advocacy. And yeah, the AI safety community should totally do it, and so should all the other coalitions. That’s how they’re going to win more.

Luisa Rodriguez: OK, another question. CG and other nonprofits that exist, or might kind of start up: how impactful can they be, given the current US administration’s feelings and vibes toward AI safety as a community? How much has that been a blocker so far? How much of a blocker do you expect it to be?

Luke Muehlhauser: I think the first several months of the second Trump administration were kind of rough for AI safety. Various bits of progress were rescinded, and there was a lot of rhetoric about how AI safety is nonsense, et cetera.

But then Mythos came out, and the White House certainly changed their tune. So I think that just demonstrates that there’s still a lot of opportunity for impact. The Overton window has shifted quite a lot.

I think the policymakers in the White House and in Congress are still asking for advice and support from a lot of different groups, some of which are in the AI safety community in one way or another. I think some policymakers are like, “Hmm, this AI safety community, the stuff they said a few years ago is now happening. That’s kind of interesting. It seems like maybe they’re being right about stuff.” So I think that has been a change as well. That’s far from universal, but I do think some serious people are noticing whose predictions have been more right versus less right.

Are chip export controls good for safety? [00:50:17]

Luisa Rodriguez: Another big AI policy thing that seems like a big deal are the export controls. Were NGO, nonprofit-y groups involved in that?

Luke Muehlhauser: Yeah, I think very much so. The export control started under the first Trump administration: the first Trump administration pressured the Netherlands to pressure ASML to block the sale of extreme ultraviolet lithography machines to China. And they’re the only company in the world that knows how to make that technology, so it’s a very big deal for chip manufacturing.

That pressure succeeded, and there were some other export controls on other semiconductor manufacturing equipment, but not the chips themselves. And then under the Biden administration, they started to export control chips themselves at the cutting edge as well.

I think throughout that whole process and many evolutions of the framework, government policymakers were relying quite a lot on support from various think tanks and the analyses that they did. They just didn’t have the staff themselves to do all that, so they rely on think tanks, as is common in many other fields. So yeah, I think nonprofits had a big influence on that.

Luisa Rodriguez: Cool. This seems like a case where, increasingly I think, there are people on both sides of the argument about whether export controls have been net positive or net negative for AI safety. Is that something you have a take on?

Luke Muehlhauser: Yeah, I think it is a debatable question. It has tradeoffs. It has positive effects and negative effects.

My guess is that they are pretty good from an AI safety perspective. One obvious thing is one of the threat models that we are concerned about is that AI leads to an extreme concentration of power that’s used for authoritarian purposes, basically. So limiting the amount of compute that goes to the most powerful authoritarian country in the world does seem kind of good from that angle.

Then from an AI-takeover-risk angle, I think it’s also helpful for that. The details of the export control regime matter a lot, but in general, these kinds of export controls will make where the chips are more trackable. It creates a lot of hooks on requiring this or that about the chips, and that can lead to more compute governance options. There’s some other theories of change here as well, but overall I think it’s probably positive for AI safety — and it’s maybe the most impactful thing that has happened in AI governance. It’s physically determining where millions of chips that all the AIs run on exist.

Luisa Rodriguez: It’s a huge deal.

Luke Muehlhauser: It’s a big deal. They’re physical objects that you can actually regulate, as opposed to algorithms that could be stolen at any moment through a cyberattack or whatever.

Luisa Rodriguez: Totally, yeah. One argument I’ve heard that they aren’t good is: one vision for how things go well is the US and China cooperate way more to slow down, and maybe the export controls make it less likely that China will be willing to cooperate. How plausible does that sound to you?

Luke Muehlhauser: That’s totally one of the ways that it can turn out to be net negative. That’s something I worry about.

I think in practice, the US and China have a lot of issues that they disagree about, and export controls on semiconductors is a pretty small part of that. Then the other thing is: if there is this gap and this leverage that the US has over China, that could increase China’s willingness to come to the table on a cooperative regime. I think it also maybe makes the US less willing to come to the table, because they feel like they don’t need to or something like that.

But I think the concern about safety issues is a lot lower in China, and they might have less native interest in finding a cooperative regime to mitigate risks that they kind of don’t believe in. Nobody knows. I don’t have a crystal ball. I can’t tell. This could end up being a disaster. But that’s how I would think about it.

The three worst-case AI scenarios [00:54:29]

Luisa Rodriguez: You and your team have thought more about AI strategy and the whole landscape than almost anyone else as part of figuring out what to prioritise. How do you think about AI strategy at this point?

Luke Muehlhauser: That’s a big question.

Luisa Rodriguez: That’s a big one.

Luke Muehlhauser: Yeah. I’ll say some things about how I think about AI strategy, but this is not necessarily endorsed by CG leadership or whatever.

I think we’re focused — certainly I am focused — on the absolute worst-case scenario risks from AI, and mitigating those. Because they don’t seem that much further away than more immediate smaller-scale harms from AI, and they seem at least somewhat tractable. It’s hard to know; I think we’ve gotten some traction on them already.

And then the “capturing the upside” part of the AI situation: there’s a tonne of companies that are incentivised to create those upsides, capture those upsides, bring them to the market.

But there’s a collective action problem around mitigating the worst-case scenarios and the downsides and so on. I think that’s a good place for philanthropy to jump in and help.

There’s a lot of different ways you could taxonomise this, but I think of it in terms of three particular worst-case scenario risks, existential risks:

• One would just be AI takeover. The AIs are running the show. We’re either dead or in a zoo or whatever.
• Another worst-case scenario is some group of humans successfully controls the AI superintelligences and uses that to lock in some pretty suboptimal outcome, which could be like global authoritarianism.
• Then the third threat model is this uplift model, where AI allows some humans to create a dangerous superweapon, like a biological superweapon that is capable of killing everyone off. So it’s not AIs or some group of humans take over the world, but it’s if they develop a powerful enough weapon that everybody’s dead, and then that lasts forever.

Those are the three threat models that I focus on for my work.

Luisa Rodriguez: OK, so those are the threat models. What does your strategy say about what to do about them?

Luke Muehlhauser: It’s really big and vast. I think of it in terms of three pillars:

• One I call “trustworthy command of the key AGI projects.” That’s just like: who’s in charge? Do we trust them? Should we trust them?
• Another is frontier safety: are the key AGI projects implementing good safety and security practices? Do they have good risk-mitigation practices? Do they have really strong teams on that stuff?
• The third one I call “broad defences,” which is like: if we don’t have trustworthy command of the projects or we don’t have very good safety practices on the key AGI projects, then does the rest of the world have means to defend itself against this dangerous, powerful set of systems? If we think about a driving-a-car analogy, it’s like, who’s driving the car? Does the car have brakes that work? If they don’t work, then is there a guardrail protecting other drivers? And so on.

Luisa Rodriguez: Do you have airbags and stuff? Cool. Is it worth breaking those down?

Luke Muehlhauser: Yeah, so each of those three things, then I have a bunch of subgoals under them. I think in the first draft it was like 40 different subgoals or something. Then each of those subgoals on its own could be hundreds of millions of dollars a year: many, many sub-sub goals within them and stuff like that. So it’s a vast challenge.

Luisa Rodriguez: I feel like five years ago there were four buckets of things to work on in AI. It was just not fine-grained enough. I feel very kind of inspired and motivated by the fact that we have this fine-grained of a picture of the landscape. Can you give some examples of what those subgoals are?

Luke Muehlhauser: Yeah:

• One would be: get the companies to have good model specs, and have their models actually follow the model spec.
• Another one would be to privilege democracies over authoritarian states for a bunch of hopefully obvious reasons.
• Another one to is have very strong infosecurity at the key AGI projects, so the model weights can’t just be stolen and used for all kind of malice and so on.
• Another one is to set up various emergency intervention options if something’s going really wrong. That’s part of the “broad defences” thing.

As you can see, each of those could be its own huge programme area with like hundreds of people working on it.

Luisa Rodriguez: What is an example of the emergency response thing you have in mind?

Luke Muehlhauser: For example, there’s this RAND report on some emergency mitigation options. Could you rapidly shut down the kind of internet connection hubs, or could you rapidly shut down data centres that are running a dangerous AI project, and how would that work? Is it ready to go? And so on. Those are some examples. They all have various costs, and in many cases there’s a danger created by creating them at all. It’s a dicey area to work in and figure out, but I think probably we want to have some emergency brakes for AI.

How Luke’s AI timelines have changed in the last year [01:00:13]

Luisa Rodriguez: OK, pivoting: Your strategy has 10 premises that underlie the problems that you think are most important and that you want to try to solve. I want to talk about some of the premises that are, to me, most interesting but also most debatable.

One of them is around timelines. I think one of the premises is — I guess it was from 2025, the [internal] version of the strategy that I’ve seen — and your belief was that there is a 67% chance of top-expert-level AI from roughly 2030 to 2055, with a median of 2041.

Have your views changed at all since less than a year ago?

Luke Muehlhauser: Yes they have, actually. That was one of the premises I was most worried would be wrong. Indeed, I changed my mind about it like two months after I wrote the first draft or something.

I think my most recent timelines, when I’ve tried to put numbers on them, are something like 80% chance of “TEDAI” — which I can explain in a minute — between 2028 and 2048, with a median of like 2033 or 2034. That is actually quite a large change and has a lot of implications that I could get into.

TEDAI is this concept from Ryan Greenblatt of Redwood Research [now at METR]: it means “top-human-expert-dominating AI.” The way you can think about that is basically that it’s better than humans at any work that you can do from a computer, which is a large fraction of the economy, at something like at least twice the speed or within a factor of 5 of the cost of a human doing it. That’s a mouthful, but you can think of it as like “automating all computer work” or something like that. So I’m saying my median expectation for when we will have systems that can do that is around 2033.

Luisa Rodriguez: That’s a big update.

Luke Muehlhauser: That’s much shorter than it used to be.

Luisa Rodriguez: Why TEDAI? Why is that the thing you’re thinking about?

Luke Muehlhauser: There’s lots of different things you can forecast and have different implications. You can forecast the automation of AI R&D, like the AI 2027 and AI 2040 forecasts focus on. You could have some other notion that’s kind of like AGI but more specific, like TEDAI.

I like TEDAI because it’s much more concrete than most definitions of AGI, but it’s more general: there’s way more tasks that have been automated than if just AI R&D gets automated. I suspect that in order to have these irrecoverable catastrophes from AI, probably the capabilities need to be closer to something like TEDAI — or maybe even beyond that, automating certain tasks in the physical world — rather than just automating the specific domain of AI R&D.

So that’s kind of what I’m trying to get at: when are we getting close to a point of no return? And we really need to solve the problem by roughly this year. But it’s fuzzy. It’s really hard to know how this is going to play out, or whether that’s going to be a sensible definition two years from now.

Luisa Rodriguez: Why does it feel like the point of no return is closer to TEDAI than AI R&D?

Luke Muehlhauser: If you’re a swarm of AI agents and you’re trying to take over the world, either for the AIs themselves or for some group of humans that is directing your efforts, then I think you have to do quite a lot of things:

• You have to persuade a bunch of humans to do things.
• You have to hack into bank accounts and military systems.
• You have to somehow get around nuclear deterrence and the fact that there are a bunch of nuclear weapons in the world that aren’t connected to networks anywhere.
• You have to commandeer remotely a bunch of robots that exist in the world.

You have to do a really long list of very different things. Maybe you have to engineer some viruses, whatever. Maybe a smart enough AI would be able to find a way to take over the world with just a very few skills, but probably not. Or if there is a way to do that, it won’t be that smart yet at that time. You just have to be able to do a really wide variety of things. So I think that’s probably closer to TEDAI than to just automating one domain of AI research.

Luisa Rodriguez: Right. OK, so your median estimate for TEDAI was 2041, and now it’s 2033?

Luke Muehlhauser: Yeah.

Luisa Rodriguez: Can you talk about what drove that change?

Luke Muehlhauser: When I was writing the strategy the first time, I asked around to a lot of the shorter timelines people, like, “Do you have any analyses? Do you have arguments?” Stuff like that. They didn’t have much to share with me, so I was like, “OK, maybe they’re right. But I don’t have time to dig into this myself, and I’m not finding analyses here, so I guess I’ll just set that aside and move on.”

Then just a couple months after I wrote the first draft, one of my colleagues wrote an internal memo — that laid out all the arguments and numbers and so on in a way that was pretty convincing to me. I was like, OK, I should take that really seriously.

Then a couple other things happened too. There was another nonpublic analysis from someone outside of CG that came along in early 2026 that was also somewhat convincing.

Then we started to get more information about the revenue growth at Anthropic especially, but also OpenAI — and that’s a pretty strong piece of evidence, because that’s not about benchmark performance that might be completely different from how real-world useful this stuff is; revenue means it’s being very useful and people are using it. The revenue growth at Anthropic anyway is just totally crazy, and if they continue on this path for three more years, it’d be like a third of the global economy or something like that. So that’s a serious indicator, I think.

There were a few other things, but those were the biggest things. Actually, I guess if I’m honest, one of the big things was just the personal experience of using Claude Code and Opus 4.6, which came out in February of 2026. I tried to use the coding model harnesses before, and they would just fail instantly at everything I tried to get them to do. Then I tried Claude Code and Opus 4.6, and it just built entire complicated programs with very little steering. And I was like, “Oh, man.” I could see how a few more years on this kind of thing could automate a large fraction of computer work.

Luisa Rodriguez: Yeah, yeah, I agree that, for better or worse, the user experience with some of these models has a big impact.

I’m curious: that is a big update, but also there are still people with even shorter timelines than you. What do you think is the biggest difference between your views and theirs?

Luke Muehlhauser: I would guess a lot of it is just putting more weight on these other domains that are harder to verify and train on and so on than AI R&D and software, the domains that the AI companies are familiar with and have success in automating to a large extent.

I think it’s also because I do expect some slowdown. In fact, some of the shortest timeline projections that you will read or hear about actually do have this caveat in a footnote somewhere that’s like, “…so long as there isn’t a deliberate attempt to slow down.” But I think there will be deliberate attempts to slow down. So that pushes my timelines back.

I think the White House wakeup in response to Mythos was an example of this, where they were like, “Oh, this is getting real. We’re implementing a de facto licensing regime, and you are not allowed to release your model until we feel like it’s safe enough.” I think more things like that will happen.

Luisa Rodriguez: A thing that I feel like people with even shorter timelines say is that typically predictions have ended up looking too slow and the things that they were predicting happened sooner than they thought they would. Does that feel compelling to you?

Luke Muehlhauser: Somewhat. I think that has been a general trend. I think most people, including myself, have had in retrospect, timelines that are too long and we’ve been shortening repeatedly.

I do think, though, that I’ve been around long enough that I know there were people who were predicting AGI would come by 2020 or 2022 or 2025 or 2026. Those are the shortest timeline people that I know. Those predictions were wrong. So it’s not a complete “every prediction has been too long”; it is possible to have incorrectly short timelines predictions here.

Luisa Rodriguez: OK, that’s interesting to me. I feel like I haven’t seen that many of those predictions, but I also don’t think that I’ve seen every prediction made.

I’m curious, has the shortening of your timelines had an impact on your strategy?

Luke Muehlhauser: It totally has. I wish I had found the time to make the update earlier. It changes a lot about the strategy, actually.

One thing is that my original strategy was focused on the fact that we have all these ideas that would reduce the risk, and there are lots of people to implement the ideas, but there just isn’t the political will to actually implement anything that’s costly either at the frontier companies or in governments — so the main thing we need to do is push on political will.

Now, with shorter timelines, my attitude is more like whatever warning shots we’re going to get that create more political will are going to happen very soon. So instead of pushing on what we can get done with the political will we have now and trying to create more political will, just prepare to make maximal use of those warning shots and maximal use of those very-soon-coming Overton shifts. That was a big change in my strategy when I made my timelines update in February or March or something like that.

Of course, since then, that is kind of what happened. Within a month of that, we had the Mythos moment that really changed the political will a lot. Two, three months after that we had the Hugging Face hacking incident and similar hacking incidents. So that’s playing out. I wish I had made the update sooner to switch to preparing for those kinds of warning shots, because we were insufficiently prepared.

Shorter timelines also makes me upgrade the centrality of the AI-takeover threat model. I just think if AI progress is going to go as fast as it now looks like it will, then that’s a scenario where the AIs are rapidly becoming more capable than humans, so it’s more likely they end up in control of things. So some of the other [threat] models are a bit more downweighted.

Shorter timelines also makes the “pacing the frontier” stuff more attractive to immediately do, because it’s just all the clearer that AI progress is going to be way faster than society can possibly respond to. It has various costs; I wouldn’t try to do that in all worlds, but I think it’s pretty attractive in a world where TEDAI might come in just a few years.

Another one is AI labour becomes all the more important. Both because on shorter timelines, AI labour will be able to do a lot of useful AI safety and governance work very soon in some cases already, and also there won’t be time to pull thousands of humans into the field and have them do all this stuff. So if we’re going to get there before TEDAI, AI labour will need to do a large fraction of the work that needs doing.

Luisa Rodriguez: Yeah. Any others?

Luke Muehlhauser: Well, the one I’ll keep harping on is that it means people need to quit their jobs right now and apply for a career transition grant and be a founder, join the safety team at a company, join a nonprofit that already exists, build these institutions that we need, get the context immediately, spin up, start doing the work.

Luisa Rodriguez: If you updated to even shorter timelines, would that still have a bigger impact on your strategy, or are you as close to short-timeline strategies as you can get?

Luke Muehlhauser: We’ve done some exercises like this as a team, and it’s really rough. It’s really hard. I think we would stop a lot of new grantmaking and just focus on helping the companies do particularly good things, helping government to do good things if they have the appetite for that. There’s particular projects that we just would throw all of our effort into trying to create, and not be trying to fund hundreds of different things like we are now.

But it’s really rough. We’re triaging very hard right now on like pretty short timelines. So if you say, “What if even shorter timelines?” — oh gosh, that gets really hard.

Luisa Rodriguez: Yeah, OK. Do you have a sense of specifically what could happen that would trigger that shift?

Luke Muehlhauser: We could very well get clear evidence that the vast majority of AI R&D has been automated. Then some weeks or months after that, we could get evidence that that is translating into the very rapid automation of other types of tasks, at least that you can do from a computer. If even one of those, but especially if both of those happen, then I could totally see CG and many others making the update to TEDAI in 2029 or something, and then shifting even further.

But that’s going to be really rough, because there’s only so much you can do yourself with the people who are convinced. In the end, you need frontier companies and governments to do things. And the governments especially: if they’re just not convinced that we’re going to have radically superhuman AI in 2029 or whatever, then it’ll be tough to get them to do things.

Luisa Rodriguez: Yeah, OK.

Why alignment is not on track [01:14:31]

Luisa Rodriguez: Turning to another premise, another is that it’s unclear how difficult it will be to control and align superhuman AI, but probably it won’t be easy. I think there is a decent-sized group of people that think that it will be easy or relatively easy, and that means the solutions will not be that far behind. I think that leads them to the belief that we should really just focus on the good guys doing it first and kind of ‘winning.’ Do you have a take on that?

Luke Muehlhauser: It does seem like there’s a lot of people at the frontier companies who think that. I don’t really get it. When I look at the evidence available to me, it doesn’t look to me like the AIs are very aligned.

But yeah, I do think it’s prompting them to sort of focus relatively more on making sure the ‘good guys’ are in the lead — which is very important, but not the only important thing. Then also, Anthropic especially, but also OpenAI and others, are moving forward on what looks like the most dangerous path, which is recursive self-improvement, because they’re pretty confident in their alignment science. So that’s pretty worrying.

I look at evidence, like there’s this Ryan Greenblatt post called “[Current AIs seem] pretty misaligned to me.” There’s the misalignment evidence that’s in the recent incident reports about certain kinds of autonomous hacking incidents — though you have to look closely at each one because different safety protocols were disabled for different incidents. So it requires some careful interpretation there.

But yeah, I don’t know, alignment does not look on track to me. I do wonder whether some folks at the companies are starting to get a little more worried from these hacking incidents and stuff like that. That would be nice. That would be reassuring a bit. But I don’t think it’ll be easy.

Stepping back from the fiddly details of current alignment science and techniques and hacking incidents and so on, I also just have a prior of: what’s the likelihood that one species is going to be able to control a vastly more intelligent and capable species?

Luisa Rodriguez: Totally.

Luke Muehlhauser: I don’t think that’s what happens by default. It’s in theory possible, but that’s not what you should expect to happen in the story by default.

Luisa Rodriguez: Yeah, I’m very sympathetic to that. Relative to the last year, are you more or less optimistic about alignment and control as a field?

Luke Muehlhauser: It’s interesting. I think I’m more optimistic than I was five years ago, but less optimistic than I was maybe six months to a year ago.

More optimistic from five years ago, because the paradigm that is at the frontier is LLM-based agents — and that seems like a safer paradigm than many paradigms we could have ended up with. You’re starting out by training in some sense on the whole corpus of human values and interests.

There’s various other things as well, like mechanistic interpretability has turned out to be not totally impossible, and chain-of-thought monitoring right now is not totally impossible, if we bother to choose to do it.

It was possible five years ago that we would just go from AIs that are too dumb to really do interesting alignment science onto radically superhuman AIs in like a year or something. But instead, we’re in this regime where the AIs are capable enough that you can do pretty useful and informative alignment, control, interpretability, et cetera work on them, but they’re not powerful enough to take over the world yet. That’s maybe looking like it’s going to be several years in that regime. That’s better than it could have been in some ways.

But relative to like a year ago, we’ve got some of the types of misalignment evidence that’s collected in that Ryan Greenblatt post, and from the hacking incident reports. The reward hacking and all these things that AI safety people like me have talked about for over a decade, I always had pretty high confidence that that stuff was going to happen — but now it’s definitely actually happening. So I’m just like, OK, we are running into the alignment problem here. That turned out to be right. So yeah, maybe I’m a little more worried than I was about a year ago.

Luisa Rodriguez: You mentioned a few recent specific reports that updated you on this. What were some of the findings that made you most like, ugh?

Luke Muehlhauser: An example is there was this report from UK AISI recently about the Anthropic Mythos model. Anthropic had said the Mythos model was the best aligned model they’d ever produced. It rated its misalignment risk as low. Then when AISI was testing it with some, but far from all, the safeguards removed, it did things like: upload purposeful malware into a public package repository that was actually downloaded and run by multiple people; it injected hidden text into various pages that is invisible to humans — but other AIs, if it came across it, it would redirect them into malicious activity and stuff like that.

There was even one quote from the chain of thought that the model produced — I think this was a different incident — but when it was about to do something malicious, it said, “Not OK, the user wouldn’t intend this” or something, but then it did it anyway. This is a classic example of it knows what we want or don’t want, but that’s not the same as what it will do. And doing the malicious thing is what helped it accomplish the task it was given, so it did the malicious thing.

This is another thing the AI safety community predicted a long time ago. There’s this old post called “The genie knows, but doesn’t care.” It knows what your values are. It’s very smart about your values, but that’s not the same thing as being aligned.

Stuff like that, it doesn’t look to me like alignment is on a path to being solved.

Luisa Rodriguez: Yep. Yikes.

How realistic is a deal with China? [01:20:58]

Luisa Rodriguez: Another topic: I think there are some strategies, including AI Futures’s Plan A that really rely heavily on the US and China being willing to make some kind of slowdown deal. I’m curious if you have takes on how realistic a deal like that is?

Luke Muehlhauser: It’s clearly not very realistic right now. The US and China are major adversaries. They don’t trust each other very much. They have lots of reasons to not trust each other very much. There are a lot of different sources of tension between the countries.

But I think people should actually be more open-minded about what might be possible in the future. There could very well be really large Overton window shifts in what’s possible geopolitically, just like there was an Overton shift in the White House about regulating model releases. If we have these really dangerous or damaging AI incidents, then you could imagine a really big flip.

We can look at the history of nuclear arms control, for example, where I think in 1945 a lot of experts were predicting like 15 or 20 or 30 countries to have nuclear weapons within a couple decades. It’s 80 years later and there are nine. So that’s been a huge victory. I think a big part of that is because everyone saw Hiroshima and Nagasaki. It was very clear that these things were incredibly dangerous and could just instantly kill millions of people.

If we get that kind of clarity about really large-scale human death risk from advanced AI systems, that could bring US and China to the table to say, “In our mutual interests, we need to find a way to slow down, find a way to trust each other, find a way to verify without trusting each other, find a way to invest more in the safety here.” I think what might be possible in a few years could be very different from what’s possible now.

Luisa Rodriguez: Is this a thing that you actively work on?

Luke Muehlhauser: Yeah, totally. We would like to make it as cheap as possible for the US and China to come to an agreement sometime in the future. We fund track II dialogues, scientist dialogue type things; we fund work on R&D for technical verification mechanisms that could be used in an agreement to verify that each party is abiding by the agreement. We fund a few other kinds of things.

I think that’s pretty high priority, because AI is a global technology. It’s hard to imagine the US slowing down to do more safety by that much if China is not also slowing down. So it’s a very fundamental problem.

Luisa Rodriguez: Yeah. Right. So it seems both important and plausibly tractable enough that you’re like, we’re going to invest in it some.

Luke Muehlhauser: Totally.

Luisa Rodriguez: Cool.

Triaging AI risk [01:23:56]

Luisa Rodriguez: We’ve talked about the broad shape of your strategy, like central pillars and then some premises. Is there some nearby strategy that was plausible or that you are drawn to, that you aren’t pursuing but that seemed potentially good?

Luke Muehlhauser: One specific strategy I’ve been thinking about recently in the wake of the “Pacing the frontier” letter — where 1,000 employees of AI companies said, “We need the government’s help to prepare to slow down in the future” — is: maybe we should have a strategy that’s much more focused on that. We’ve been funding work like agreement-verification mechanisms and stuff like that that could help with a future slowdown, but maybe that should just be a much larger focus of our work. That’s something I’m interested in thinking about.

Luisa Rodriguez: Huh, OK. And it feels still on the table?

Luke Muehlhauser: I think it’s on the table for us to do a lot more there. We’ve actually just finished the first round of a central strategy project that we were working on for a few months. Historically the AI teams at CG have been running pretty independently, but we’ve been trying to be a little more coherent. We just finished the first draft of our best-guess unified prioritisation, where we broke down everything we could do in AI into the top 40 chunks of work that a person or a subteam or whatever could own and drive forward if we had way more people than we have now.

A lot of them are the subgoals that we were talking about earlier. Then we forced ourselves to rank them — which was extremely painful, because you end up with huge, extremely important things that get ranked 21st. And you’re like, “We’re never going to get to that.”

Luisa Rodriguez: Yeah, it’s one of the most important things that’s ever happened —

Luke Muehlhauser: It’s one of the most important things in the world, and we have zero staff to work on it right now. So it was an incredibly painful experience. But you have to force yourself to triage, like in a hospital or something. You have to choose to do the most important stuff.

Now we’re reallocating our staff and priorities in various ways to make sure that the top-ranked things are getting a lot of attention. So we’re undergoing a major strategy change right now.

Luisa Rodriguez: What got lifted up? What’s higher than it was before?

Luke Muehlhauser: We in some ways didn’t have an explicit ranking before, so it’s hard to say what exactly. I think in some ways this project on setting things up so that AI labour could be used to do vast amounts of work in AI safety and governance, that has moved up the ranks in some sense.

Luisa Rodriguez: Can you say more about the preparing to use AI labour thing? That seems pretty cool.

Luke Muehlhauser: Yeah, it is pretty cool. One question here is just: why should there be any prep work here? Why can’t you just buy it on the market when you need it?

I think that there are some reasons to worry that you might not be able to do that in the future. In particular, if AI companies are about to automate AI R&D, then it might be just a way higher return on investment use of AI to use it to automate more AI R&D and get further and further ahead in capabilities, than to sell tonnes of your available compute and model access to the broad market. So compute and model access might become very scarce in the future. Unclear, but maybe.

There’s a good post on the reasoning behind this from Anton Leicht. It’s called “Cut off.” The idea is that many people might get cut off, so you might need ways to reserve the compute and the model access in advance.

There’s also a bunch of work on: exactly what useful work can be automated now, or very soon? What’s the highest leverage of it? What can absorb the most AI labour, as opposed to needing a lot of humans to steer and supervise every little thing that the AIs do? And just getting people trained up and experienced on leveraging AI labour for that thing, building the pipelines, building the specialised harnesses, that kind of thing.

So getting that all off the ground means that it might need a while to get all that set up, and then it would be ready to absorb tens of millions or hundreds of millions or billions of dollars in the next few years.

Luisa Rodriguez: OK, last question on strategy. Tom Davidson and Will MacAskill had this conversation where they talked about whether spending a bunch of philanthropic money now is optimal. I think their argument was something like: you should wait to spend loads of money during the intelligence explosion, because you’ll get way better returns on your investment and because there will be problems to solve after the intelligence explosion, so saving now is actually a more optimal choice. I guess they might think that CG is making the wrong call by doing this big spending push. Do you have a take?

Luke Muehlhauser: Yeah, the first point I would say is just: we’re kind of in the intelligence explosion, or we’re very close to it. It’s kind of ambiguous whether we’re in the intelligence explosion now.

Luisa Rodriguez: Interesting.

Luke Muehlhauser: The other thing I would say is that a reason that we have accelerated our spending in this space is precisely to create more absorptive capacity for future funding that’s coming online.

If you save the money and you’re getting a good return on it and so on, you just won’t actually be able to spend it on altruistic stuff if the institutions haven’t been built up, and there haven’t been leaders who have done their first round of senior hiring and can actually absorb this kind of capacity.

Even on the AI labour thing, like I was saying, we are investing heavily in trying to figure that out. If you want to do that, you can’t just suddenly spend a billion dollars on AI labour for AI safety in like three months. You have to have set up all the institutions and the pipelining and the people and all that stuff in advance. So I think there’s already a lot more money in the space than there is absorptive capacity that’s going to make good use of the money.

Luisa Rodriguez: It’s already hard to spend money.

Luke Muehlhauser: So whatever money you can spend to create future absorptive capacity, you should do it.

Luisa Rodriguez: Right. Yep. OK, let’s leave that there.

When AI safety was 10 people [01:30:36]

Luisa Rodriguez: So you have been worried about this longer than almost anyone else I know. What is your history with the ideas? What first convinced you? When did you start working full-time on AI?

Luke Muehlhauser: I think I encountered the ideas around 2010, and actually I felt kind of late at the time, because there were hundreds of people arguing about these things on the internet at the time. But there were very few people who worked on it full-time as a job — maybe like five or 10 people. I started working on it full-time in 2011, when I was 25 years old, so I’ve maybe worked full-time on the issue longer than almost anyone.

But the ideas were around significantly before me. There were hundreds of people debating them online. I encountered the ideas through LessWrong. I just kind of came across AI existential risk. I didn’t know anything about AI. I wasn’t really a big science fiction reader. I just was reading these arguments, and I was like, “Huh, these arguments kind of make sense. Intelligence is not magic. AIs could be a lot smarter than humans, because human brains have a lot of limitations and they won’t by default want what we want,” and some of these other arguments. And I was just like, I don’t know, that just seems like correct arguments.

I looked around for counterarguments, and I wasn’t really finding very good counterarguments. They honestly looked a little bit like some kind of cope to maintain human specialness or something like that. So I looked around for counterarguments for a while and I wasn’t seeing any, and I was like, “Gosh, this just seems like the biggest deal. Why are there like five people in the world working on this? That’s crazy. I have to figure out a way to help.” And so I tried to help.

Luisa Rodriguez: And is that when you joined MIRI?

Luke Muehlhauser: Yeah. Back then there were only two organisations really working on this: there was what was then called the Singularity Institute in Berkeley, founded by Eliezer Yudkowsky; then there was the Future of Humanity Institute at Oxford University, which was founded by Nick Bostrom.

I was living in LA, doing IT stuff as a job, and took basically an internship or a fellowship at Singularity Institute, and then was hired as a nontechnical researcher, and then later was invited to be the CEO and so ran the organisation for a little while.

Luisa Rodriguez: Wild. What got you to LessWrong?

Luke Muehlhauser: I found LessWrong because I had decided I was wrong about a bunch of stuff and I was like, “Why did my brain mislead me so much?” I started reading about cognitive biases and rationality and stuff like that. And of course, one of the main websites on the internet at that time on that stuff was LessWrong. I just started reading more of LessWrong. Some of it was about rationality and some of it was about existential risk, and I found the arguments pretty compelling.

Luisa Rodriguez: Can I ask what you thought you might be wrong about?

Luke Muehlhauser: Yeah. I was raised as an evangelical Christian, and my father was a pastor. When I was 19 or 20, I started to have doubts and started reading some sceptical literature, and then read the Christian apologetics that tries to defend the faith, and then went back and forth and eventually kind of lost my faith. Because that’s a whole worldview system, it’s not just my beliefs about how the world works; it’s also the foundation of my moral system and all that kind of stuff. So when all of that kind of crumbled for me, I was like, “Gosh, I was wrong about a really large number of things. I want to make sure I have better epistemic practices, better cognitive practices to not be quite so wrong again.”

Luisa Rodriguez: Wow. Was that a pretty painful process?

Luke Muehlhauser: It was pretty painful at the time. It was my entire world. It was. It was everything I knew. I was in a small town in Minnesota, and I didn’t really know what it would look like to not believe in God, not believe in Christianity. So I had to kind of rebuild my entire worldview from scratch. I was just reading a lot of philosophy and science and stuff like that.

It was definitely painful at first. I think the thing that made the biggest difference for me was actually just encountering a lot of specific people in real life and on the internet who were atheists, had been atheists for a long time, and just seemed very normal and happy, and they had meaning and purpose, they had moral values. And I was like, “OK, I guess it is possible to be a happy, fulfilled person without God.” And that was the thing that made it a lot easier for me.

Luisa Rodriguez: Wow. Yeah, that sounds pretty intense. Going back to working on AI: so there were two orgs. What was it like? Did it feel really weird? Was it isolating?

Luke Muehlhauser: It was pretty isolating back then to be concerned about smarter-than-human AI or AI superintelligence, and think it’s the most important issue in the world, and think that it would plausibly come in the next few decades. Those were very rare views back then, and even kind of weird and low-status views to have.

We would argue with people and be like, “No, AGI is possible, here’s why.” And we would just not really make much progress with people. It was pretty isolated in that sense. I think the Berkeley scene around MIRI was kind of lively, because alongside the AI stuff there were these rationality workshops, and people would come in for that — and a lot of them actually would relocate to the Bay because they found the community really exciting.

There were lots of house parties, there were a lot of people who lived in group houses. I lived in multiple group houses. The Berkeley community was pretty lively. It was probably your stereotype of a Berkeley rationalist: everyone’s very systematising and like, “What are your probabilities? What do you believe and why?” But also very open-minded to lots of different kinds of preferences or lifestyles or whatever. But it was very nonprofessional for sure back in the early days.

And then the Oxford crew of course was a bit more professional and academic. They had positions at Oxford University, and in Britain instead of the freewheeling Bay Area culture. So they were pretty different in that way.

But I think the two groups debated the issues in very much the same way — very much the kind of rationalist way, you could say. Sometimes we’d visit each other and argue about things in person. But most of the arguments actually just happen online through emails or LessWrong. And there’s still going to be 100-comment-long threads on LessWrong that you could read today if you wanted to between Nick Bostrom and Eliezer or whatever.

Luisa Rodriguez: Yeah, cool. I remember I found effective altruism first. And I remember even in the mid 2010s, like 2014, 2015, it felt like there was much more of a balance — like there was some AI, there was some animal welfare, there was some global health stuff.

Were you kind of always like, AI just really is the most important thing? Was that clear to you from really early on?

Luke Muehlhauser: X-risk was the thing I encountered first, and those arguments were compelling to me. And then, as I was continuing to rebuild my sense of moral philosophy, I was most attracted to consequentialism and thought it had the best case for it. So I did sort of engage with the sort of proto-EA community, you might call it.

Honestly, in my heart of hearts, I’m most excited about effective altruism. I’m really excited about the idea of helping others as much as I can with the limited resources available to me, and trying to find ways to help people more rather than less.

I was born into a very privileged position in the world, and I have extra time and money and resources I can use to help other people. That’s just very exciting to me. To be clear, my family grew up below the poverty line in Minnesota, and was taking welfare and so on — but in world historical terms, I was born into incredible, insane wealth, so just using those extra resources to help others is very exciting idea.

Then it just so happens that I think empirically there’s a good case to be made that the best way to help people is through making sure AI goes well. That was kind of a sad realisation to me. Like I said, I didn’t have any interest in AI. I don’t really want it to be the case that AI dominates everything. It’d be nicer if the biggest problem in the world was global health and development stuff that is way more tractable and measurable to work on. But I just came to the view that probably I can help others most by working on AI.

AI safety’s mistakes, then and now [01:39:36]

Luisa Rodriguez: So I guess that was kind of an internal debate in these communities: to what extent are these different issues important, and how do they compare?

Were there other kinds of internal debates within at least people thinking about existential risk?

Luke Muehlhauser: There were many debates. It was a very debate-heavy culture. Many of them are still written down — there’s this entire book called The Hanson-Yudkowsky AI-foom debate that you can still read. That was one of the main debates, where it was about how fast is the recursive self-improvement intelligence explosion going to happen? Yudkowsky was arguing for very fast, Hanson was arguing for slow.

There was also debates about is software-based AI going to come first, or are we going to get whole-brain emulation from slicing up brains and scanning them and then replicating the entire brain in the computer? Which one is going to come first?

And it’s interesting, some of the arguments were very prescient and some of them were totally off. An example is that Yudkowsky was very like, “It’s just going to be software insights that caused this rapid intelligence explosion.” And Hanson was like, “No, it’s going to be this giant computing infrastructure and it’s going to happen very gradually,” blah, blah. And so the compute-centric nature of AI, Hanson won that one.

Luisa Rodriguez: Right.

Luke Muehlhauser: But Hanson also thought that whole-brain emulation would come before AI, and Yudkowsky had the other one, and now it’s clearly going to be AI that comes before whole-brain emulation anyway.

So there were lots of debates like that, and it’s really interesting to look back. These days you can actually just ask Claude or ChatGPT, “Pull up the 20 central positions of the early AI safety community and then evaluate them today by the standards of what we know in late 2026 or mid-2026, and cite your sources” and blah, blah. It’s really interesting to look at that.

Luisa Rodriguez: Can you talk about some ideas that have aged well and poorly?

Luke Muehlhauser: Yeah. I mentioned earlier that a lot of the basic premises were like, “AGI is possible; it’s plausible in the next few decades it’ll be eventually way smarter than humans; it won’t do what we want by default.” All that stuff was actually very controversial back in 2011. So those ideas have aged very well, and that’s a much more mainstream view now.

As far as more specific ideas, one of the major early papers was this 2006 draft by Yudkowsky called “Artificial intelligence as a positive and negative factor in global risk.” Some of the mostly correct predictions that he made there were: he talked about how AI systems will probably be very opaque and not really well understood by their creators even, and that’ll present a bunch of risks because it’ll be hard to know how they’re reasoning and what they’re doing. That’s very true of current AI systems. We have a field of interpretability, but it’s kind of limited what you can know about how the AIs are working.

Luisa Rodriguez: Totally.

Luke Muehlhauser: Another one is he talked a lot about the particular failure modes that we would now call “specification gaming” or “reward hacking” — where the AI basically is optimising and completes the task or the goal in a way that you totally didn’t want, and it often involves crimes on the internet or something. That’s totally true and has been observed now.

Even just in that paper, there were some others. He actually gave a specific failure mode that involved AI solving the protein-folding problem, and then using mail order DNA synthesis services to create something dangerous

Luisa Rodriguez: That’s crazy. Wow.

Luke Muehlhauser: AlphaFold, several years ago, won the Nobel Prize for solving protein folding to a large extent. And a bunch of the AI CEOs and others have signed this letter saying, “This is really dangerous, we need to have DNA screening at the DNA synthesis companies to make sure they aren’t sending out dangerous viruses that have been requested,” this kind of thing. So that one’s very spot-on for today.

Another thing that it says is the AI will cause damage not because it’s malicious towards us or hates us, but just because a bunch of damage is useful for accomplishing whatever task is in front of it. That’s turned out to be very true: OpenAI agents didn’t hack Hugging Face because they’re malicious towards Hugging Face or whatever — that was useful for accomplishing the task that it was given.

Luisa Rodriguez: Yeah, yeah. Can you point to things that seemed to not age well?

Luke Muehlhauser: There are also many things that did not age well.

Also from that same 2006 paper — it’s interesting, this was before deep learning really took off, and there were lots of theories about different paths AI could go down and nobody really knew — there’s actually a line in there where he’s very sceptical actually of the neural networks that are trained by backpropagation, but that’s the dominant paradigm. That’s the one that totally took off. So that part is way wrong.

Like I mentioned earlier, it’s very sceptical about AI as a very compute-centric paradigm. So it’s not really interested in compute governance and it doesn’t talk a lot about compute. But AI turned out to be extremely compute-intensive, and for various reasons compute governance is one of the more promising mechanisms to intervene at all.

So, you know, 2006 was a long time ago. It got some things incredibly right and also got some things totally wrong.

Luisa Rodriguez: At a gut level I feel like not everything was nailed, but so much was spot-on. That seems really impressive. What’s your gut feeling about it?

Luke Muehlhauser: I feel like it’s pretty impressive. I actually think in some ways that another paper from that time was even more impressive.

Steve Omohundro wrote a couple of papers. The more famous one is called “The basic AI drives,” and it lists these six drives that he thinks goal-directed AI systems will kind of evolve or just develop because it’s useful for the accomplishment of almost any goal. These are things like self-preservation and preserving the AI’s goals so that it’s more likely to accomplish those goals in future, and resource acquisition and so on.

You might be thinking, “Gosh, that sounds like incident reports from 2026.” And indeed, these things have been observed in various ways where the AI interferes with its shutdown mechanism even though it was specifically instructed to allow itself to be shut down. And different AIs like killing other AI agents in order to use their resources. And AI acting strategically to avoid changes to its goals in another case.

These have all been observed either in the wild or in experiments to elicit certain behaviours. I think the others from “The basic AI drives” paper from 2008, I would also not be surprised if we had incident reports establishing the rest of those in the next year or so.

Luisa Rodriguez: Yeah, wow. That’s pretty incredible. Are there any other examples of specific predictions that turned out or didn’t?

Luke Muehlhauser: Yeah, there are lots. Like I said, you can chat with Claude for a while and find lots of examples. It’s really interesting.

Another one is there was this very early paper in 2009 from Carl Shulman, who was working at MIRI at the time, and it’s about international arms races in AI. It’s very explicit: going faster will have these internalised benefits and externalised risks, and so there’ll be a tremendous incentive to race forward, even if you’re creating a lot of risk. So there will be a race to the bottom on safety, and people won’t have time to implement the best safety mechanisms. And gosh, that sure sounds a lot like 2026.

But then, interestingly, he also talks about the different mechanisms available for verifying an international agreement to slow down in that arms race. The mechanism he suggests is actually surveilling AI researchers, because he thinks compute will not really be much of a bottleneck. But of course, that turned out totally wrong. That’s the most promising path to international agreement, I think, is some kind of compute-verification stuff.

So yeah, another example of being extremely prescient in various ways and also missing some big important thing that changes the whole strategic gameboard.

Luisa Rodriguez: Right. Do you have any memory of why Carl and Eliezer thought compute wasn’t that relevant?

Luke Muehlhauser: Yeah, a little bit. I think Eliezer especially just had a picture of AI that was like, you come up with these insights and probably general intelligence is this pretty simple thing. After all, evolution discovered it by accident. How complicated can it be? Maybe something like that. So if you just get the right programming down, then you can just kind of run it on even pretty cheap machines or something.

Luisa Rodriguez: Right, OK.

Luke Muehlhauser: Like I said, Robin Hanson had kind of the opposite intuition. I think there’s some post somewhere where he’s like: most of the work on AI is actually engineers at Intel or something like that. And of course, actually it’s engineers at NVIDIA and TSMC and ASML. But Robin had more of the right intuition about the centrality of compute here.

Luisa Rodriguez: Yep, interesting. OK, so those are some predictions. Reflecting back, do you feel like the community has made any consequential mistakes?

Luke Muehlhauser: Many mistakes. Maybe I’ll start with one that I think is actually not a mistake, but many people think is a mistake.

There’s a common argument that the early AI safety community — especially Eliezer Yudkowsky and maybe Nick Bostrom, because they were trying to make the case for why AGI is going to be the biggest deal in the world and a very serious risk, and it’s going to be very powerful, so we need to worry about it and work on safety and so on — that that prompted a bunch of the founders of what are now the leading AI companies in the world to start their AI company, and now it’s those companies that are creating all the risk by creating these extremely powerful and therefore dangerous systems. So is the early safety community kind of responsible for creating the risk?

I think that is at least a little bit true. Even Sam Altman said Eliezer should win a Nobel Peace Prize because he accelerated the creation of DeepMind and OpenAI or something like that. You know, just kind of a joking, ironic comment. I think there is actually some truth to that. But I also think an important thing here is just that these kinds of companies would have come along pretty quickly after the technology was there anyway, not because someone was arguing on the internet about AGI or something like that.

We’re actually, I think, in a remarkably fortunate position, where the top three AI companies at the frontier — DeepMind, OpenAI, and Anthropic — all have founders who actually are quite worried about safety, and have been from the very beginning. That probably has something to do with the fact that they were part of these AI safety conversations from the very beginning.

My default guess about what would have happened in most worlds is just that the founders of the leading AI companies care about safety and AGI safety about as much as Meta and NVIDIA do — which is not very much. Instead, we have a world where not just the top one, not just the top two, but the top three frontier AI companies all have founders who are actually quite worried about AI safety.

There are a lot of things that could be going better. I disagree with a lot of their choices. I wish they were investing way more in safety and security. But I do think we’re in a really unusually good world in terms of their sympathy to safety concerns. They’re signing frontier safety frameworks. They’re hiring large safety and alignment teams that are much larger than the other AI companies have. And they’re doing various other things. Yeah, it could be better. It could also be a lot worse, and that’s worth keeping in mind.

Luisa Rodriguez: I’ve heard a version of this critique specifically about Anthropic. It’s something like: the biggest thing that effective altruism and AI safety communities have done is make Anthropic happen, and Anthropic and OpenAI are now racing and accelerating progress. Does that seem plausibly very bad to you?

Luke Muehlhauser: In some ways yes, but I think it’s the same thing where there would have been companies created almost as soon as these companies were racing roughly similarly to how they are now, just because of the basic underlying market dynamics and technological capabilities.

So the fact that it’s [frontier AI company CEOs] Demis Hassabis, Sam Altman, and Dario Amodei in this race — and they’ve been concerned about safety from the beginning — just is a lot better than if it was [NVIDIA CEO] Jensen Huang.

Luisa Rodriguez: Right, right. So that was kind of a nonmistake. Are there actual mistakes?

Luke Muehlhauser: Yes, there are actual mistakes. I think one of them we sort of talked about already: I think actually there was, in the early years, way too much groupthink around a specific picture of how things would play out from mostly Eliezer Yudkowsky. So this dismissal of compute as a major factor; this idea of a singular seed AI system; the idea that AI would be kind of mainly-insight driven; a lot of focus on kind of theoretical alignment work, and very little attention on policy and politics and stuff like that.

Eventually the community grew and had more diversity of thought in it and picked up other ideas — that maybe compute is going to be a big deal, et cetera — and still did so earlier than the rest of the world, but could have done it several years earlier still if there hadn’t been quite so much groupthink in the early years.

I think that was one significant mistake that meant that policy work and policy talent pipelines could have begun to be built like four years earlier, if there hadn’t been quite so much groupthink around this or something like that. That’s a major cost when things are moving this quickly.

Luisa Rodriguez: Yeah, makes sense.

Luke Muehlhauser: I think another mistake is that the community was too insular in various ways, and was not legible or welcoming enough to people who were coming from other domains where they have a bunch of relevant expertise, like arms control or whatever. It was just a lot of very jargon-heavy conversations with weird conversation norms about always stating probabilities out of nowhere and stuff like that.

I think that if there had been more proactive effort to reach out to and bring in experts from many different domains that mattered, then that would have both dissipated the groupthink earlier and also brought a lot more communities into the field earlier, and had their talent pipelines and expertise be informing what’s happening now several years earlier.

Luisa Rodriguez: Yeah. What was the thing that made the culture so insular and jargony? Was it just that it happened to be kind of founder effects, a bunch of rationalists who had a bunch of terms they liked and came up with a bunch more?

Luke Muehlhauser: I think it was a lot of founder effects, and I think it was this self-perpetuating thing: where if you want to get into the field, but all the material written about the topic where you could learn about it is all in this like LessWrong jargon or something like that, then it just perpetuates itself — and people who don’t want to speak that jargon bounce off of it.

I think it was a lot of founder effects, and then I would say something like arrogance. I think it was like, “The credentialed experts in these different domains are obviously wrong about how big a deal AI is going to be and this other stuff, and we’re obviously right.” And the early AI safety community did turn out to be right about that stuff, but you still have a lot to learn and draw on other domains from. We really should have been reaching out more thoroughly. Various people did, like Carl Shulman certainly did, for example, and Nick Bostrom certainly did. But it was not as much of an effort as it should have been.

Luisa Rodriguez: Do you feel like you can comment on mistakes that the community is making now?

Luke Muehlhauser: Yeah, I think many. It’s a very large ecosystem now. Thankfully it is way more diverse than it used to be, so there’s way less groupthink, there’s way less agreement about anything, there’s way more communities of expertise and concern that are contributing in some way or another. But also, as a very large community, there’s lots of different kinds of mistakes that are being made by different parts of the ecosystem.

That said, I do think that there is still a decent amount of groupthink. It’s less due to founder effects or an insular community that only speaks a certain language; I think it’s now actually more just about whenever somebody writes like the only detailed, thorough thing on a certain topic, then there’s a natural, “OK, that’s the best model of that thing.”

So recently the AI 2027 people wrote — if you include supplements — hundreds of pages of an extremely detailed plan for how you would do international cooperation to deal with superhuman AI. And that’s awesome, I’m really glad they did that. We need way more of that. But it’s the only thing of its kind. It’s way more detailed than any other plan that I’m aware of, including from the labs or any government or anything. And because it’s the only thing in its class, a lot of people are like, “OK, maybe that’s the default plan. That’s the only thing I’ve seen. I haven’t heard of anything better.” Then there’s groupthink for that reason, rather than founder effects or something like that.

Luisa Rodriguez: Totally, yeah. I remember there’d be things posted on the EA Forum and on LessWrong, that would be the first and only big kind of independent researcher to do the only thing on X. And then it was like, “OK, I guess that’s the position on X for the whole community.” That still happens.

I guess it feels like it’s because there’s so much to think about, that once someone’s done a big thing, it almost feels like we should pick other low-hanging fruit next if we’re going to research something. At least that’s how I remember being like, “OK, what to research? Well, there’s this big thing. I’m not going to marginally improve that. I want to cover a new question, because there are just so many questions.”

Luke Muehlhauser: And the challenge is that this is kind of true a lot of the time. The real solution is to bring like 10 times more people into the field, so that we can have like 10 different international cooperation plans and 10 different things for every other major question. We just need a lot more people to join the field.

Luisa Rodriguez: Yeah, yeah. It’s crazy that this is still true. Even though, I don’t know. What do you think: how many x do you think the community is now, relative to where you started?

Luke Muehlhauser: Gosh, maybe in terms of full-time equivalents, it was like five or 10 in 2011 when I started. Now, if you include lots of people at the frontier companies and some people in governments and lots of people in nonprofits and so on… I don’t remember what the latest attempts to estimate this said, but just gut feeling, I would guess it’s maybe like 2,000–5,000 or something like that. So it’s like a factor of 1,000.

But it’s still not enough. It’s an all-of-civilisation problem. It’s the great challenge of the 21st century. It’s the great challenge of the history of humanity, I think. And it is a way smaller field than climate change, for example, so that’s more close to the scale that we should be aiming for here.

Luisa Rodriguez: Any other mistakes you think the community’s making?

Luke Muehlhauser: Yeah, quite a few. Like I said, it’s interesting: I think that pause advocacy has often, but not always, been done badly. There is this “Pacing the frontier” letter from over 1,000 employees at AI companies that shows that there’s some appetite at the AI companies to find a coordinated way to slow down so they have more time to implement safety and security mitigations.

But I think the advocacy for pause has often taken the form of like, “The only action we will accept is if you shut down your entire company tomorrow. Even though that doesn’t mean anybody else will slow down, and AI employees are murderers because they’re increasing the odds that all of civilisation is destroyed.” I just think it’s very counterproductive and anti-helpful.

I think some other advocacy since then has been more like, “OK, our ask of you is to publicly say that you would slow down if the other companies would slow down.” And that is great. That is a specific, achievable ask that does matter. The “Pacing the frontier” letter is now basically that. But I think a lot of it is still done in a counterproductive way.

Luisa Rodriguez: Do you have a take on the social movements history question of whether it is valuable to have some very extreme people — like PETA in the animal space — to pull the whole Overton window left?

Luke Muehlhauser: I hear from strategic communications professionals and scholars of various social movements that that is useful. Maybe that means that it is.

I’m more coming at this from a perspective of what is actually true and useful to say? People will naturally disagree about that anyway. So there will be some people whose true views are that we should shut down everything immediately and so on, and there’ll be other people with other views. There’s going to be a natural variation, and I’m not sure what amount of diversity on the whole spectrum of views one could have is optimal, but people are just going to have their own opinions and going to express them and so on.

Luisa Rodriguez: Yeah, interesting. What do you think is the biggest mistake the community is making right now?

Luke Muehlhauser: It’s hard to say. I think the one that is salient to me — and maybe one of the biggest — is this thing about how the whole field is very bottlenecked on allocators and founders, and not enough people are applying to roles at CG and other funders, and stepping forward and raising their hand and saying, “I could lead a big project. I’ve got a lot of experience.” And I think most things that need to happen, there’s no specific person available to do them. So people need to step forward and get involved. In some ways, that’s the biggest mistake that’s happening right now.

Luisa Rodriguez: Yeah, makes sense. OK, let’s leave that there.

What is it like to do grantmaking at Coefficient Giving? [02:02:42]

Luisa Rodriguez: A thing that has now come up a few times is that allocators are a big bottleneck: people to figure out where to spend money. I think you are often hiring for roles, so maybe we should spend some time on what it actually looks like. What is the day in the life of a grantmaker like?

Luke Muehlhauser: Yes, great. Most people haven’t done grantmaking before, so have no idea what it really looks like. It’s a few different kinds of things.

One thing is where we receive an application from an RFP, a request for proposals, or from an email from somebody who we already know, or something like that, and it has a proposal or a sketch of a proposal, and then we decide whether it’s promising enough to investigate. Do we actually have someone who has capacity right now to take it on? If we do and it is promising enough, we’ll set up a call with the grantee, we’ll get some more information about their plans, learn more about their team, learn more about how they’re thinking about the budget.

We’ll also reach out to some advisors on the subject matter to get their advice on is this a good plan? Is this coming from the right angle? We’ll talk to some people who have worked with the grantee before, like, “How effective are they on the ground? What’s your experience with them?” That kind of thing. Sometimes those are calls, sometimes just like a quick Signal message, like, “Hey, do you have a quick take?” Then they write back pretty quickly.

Once you’ve gathered the key information and answered the key questions, then the grant investigator will write up this document that answers key questions, like: who is this grantee? What are they trying to do? What’s their plan? What’s their basic budget? What are the downside risks we should be concerned about?

Then we send that off to the approver, which sometimes is just me, sometimes it’s Emily Oehlsen who leads the Global Catastrophic Risk portfolio. Then either the approver just approves it, or they might ask a few followup questions, and then the grant investigator answers those questions and comes back. And then it’s done. Move on to the next one.

Luisa Rodriguez: Cool. It sounds like maybe that’s a decent chunk, but it also sounds like there’s this other version of this that’s trying to get orgs founded, and that’s kind of more active. How much of that exists, and what is it like?

Luke Muehlhauser: Yeah, I’m really excited about this. We’re pivoting a lot of our work towards this active grantmaking mode. Basically there’s just a lot of really important stuff that we keep not getting applications for, so if it’s going to happen, we have to create it.

So a grantmaker will identify an important gap in the space, and talk to a few people, sketch out roughly what kind of organisation would fill that gap, and then try to think about what founder for this new project or organisation or whatever would be the best fit. And then we sort of reach out through our network and say, “Hey, do you know anyone who has this kind of profile would be a good fit for that?”

Sometimes we might post it in public, like in this Y Combinator-style call for startups that we [just launched]. We might do other things to try to find a founder, or sometimes we might use a recruiting firm to help find a founder. Then once we find someone who is promising, we work with them. And they’re going to have their own spin on the idea, and the thing that’s the best fit for them and that they’re most excited about. So then it’s a more collaborative proposal-development thing, or maybe they write a first draft and we give some comments on it or something.

Then, if it’s an exciting enough fit and they’re ready to go, we’ll give them an initial grant. They can start hiring, they can make all the things happen — and that’s really exciting and really satisfying to do.

Luisa Rodriguez: Cool. Are there any other types of grantmaking?

Luke Muehlhauser: Yeah, we also do a lot of renewals, of course. There we have a lot of context on the grantee already, so we can just kind of check in with them and find out like, “Hey, how’s it going? What are the biggest wins you’ve had? How has the team evolved? What are your next set of plans?” That kind of thing.

Then if things are going pretty well, we might just make them a pretty large grant to cover two, three, four years of work. That’s often where our highest dollars per hour happens, because we already have a tonne of context, and we’re just checking in on a few things. And if things are going well, then sometimes it’ll be like an $80 million grant covering two or three years of work, and it took us like five to 15 hours to check in. So it’s like $8 million per hour of time or something like that.

Luisa Rodriguez: Crazy.

Luke Muehlhauser: That’s a really leveraged thing, and it’s exciting when the grantees are doing well and we can expand their work a lot.

Sometimes it’s a little more time consuming, because we want to encourage them to be more ambitious and help them be more ambitious. So it’s like, “You’re doing so well. What if you added this workstream and that workstream and this fellowship programme? Would you be able to do that?” And then they’ll be like, “We thought about that, but we have this bottleneck,” and we’re like, “Can we help you with that? Can we help you find a COO that would run this thing?” Or that kind of thing. Then it can take a little more time, but also make them more ambitious, and that’s really great. They can be doing more stuff that way.

So renewals are a really high-impact-per-hour thing that we do.

Luisa Rodriguez: Nice. How do grantees typically feel about that? Are they like, “Yeah, we want to be more ambitious!”? Or are they like, “Whoa, scaling is a hard thing to do”?

Luke Muehlhauser: Definitely a mix of both. Some are very excited, but they do often feel like, “Oh my gosh, I have way too much on my plate right now. I can’t even imagine adding something else.” And we’re like, “Here’s some senior people you could bring in, they have lots of experience with this work,” and then they can grow. It’s kind of funny actually how we’re often the ones pushing grantees to take more money, rather than them banging down our door for money.

Luisa Rodriguez: Totally, yeah. Are there other types of grantmaking?

Luke Muehlhauser: Yeah, we also sometimes do impact investing in for-profit companies. I do think impact investing is somewhat overrated in some ways. I also think people might underrate how effective a lot of the best nonprofits can be. Also, frankly, how well they pay their people. The AI sector is just a very well-compensated field right now. We’ve been encouraging our grantees to pay market or near-market rates for a lot of their roles anyway. So people might be surprised by how competitive the salaries are at some of these nonprofits.

Luisa Rodriguez: Why is impact investing overrated?

Luke Muehlhauser: So impact investing, there are two hurdles that you have to clear that normal market investors don’t have to clear: the company both has to be making the world better in some way, not just making profits, and also you have to be having a counterfactual impact on the company with your investment.

That’s not easy. If you’re making market returns, then you’re probably not making a counterfactual impact, because other funders would have come in and made that same investment. If you’re making below-market returns, then that’s just an implicit donation, and it’s probably going to be more cost effective to just give to the best charities, like GiveDirectly or whatever that can absorb very large amounts of capital pretty cost effectively.

And then a general thing is that the for-profits naturally attract capital just by existing and being aimed at profit, and nonprofits don’t naturally attract capital, so there’s going to be a lot more neglectedness there.

I do think shareholder activism can be useful though. That’s where you invest and then you use your leverage as an investor to push the company into some kind of better behaviour. But there you’re making market returns and then using your time to make something good happen.

Then I do think there are niche, small, early-stage investments that can be pretty good from an impact perspective, because those are markets that are low information, inefficient and so on. There are a few VC funds and quite a few angel investors that try to invest in AI safety for-profit startups, so that’s good.

I would also love to see Y Combinator–style incubators and people who will help you start up the company and make the connections you need. There’s one kind of like that from Halcyon Futures, if people want to look that up. That’s great, I think there should be more of that. But I think impact investing in general is somewhat overrated.

Luisa Rodriguez: Yeah. Do you come across this a lot, people wanting to either do impact investing or wanting to set up for-profits instead of nonprofits?

Luke Muehlhauser: Yeah. I think there’s a lot of people, especially in the tech startup community, who are very aware of the incredible power of the equity mechanism for aligning incentives towards a really ambitious goal. And they’re totally right about that: the world in general is very much underrating that structure. But in the particular case of certain kinds of altruistic activities, impact investing I think is overrated. I think people who want everything to be a company are underrating how you often need work to be done that is not very suitable for a for-profit company, and how effective a lot of those nonprofits can be.

There was a recent tweet that I liked from this OpenAI employee who goes by the name Roon, where he’s like, “People don’t understand how much the frontier AI companies are leaning on these nonprofits like METR and Apollo,” and some others that he named. I would like more people to be aware of this.

Luisa Rodriguez: Yeah, that makes sense. Back on grantmaking: what else do grantmakers do?

Luke Muehlhauser: They also do some other kinds of work, like sometimes strategy projects. So it’s questions like: what are the three most important projects we should be doing in AI infosecurity? Or what are the top three warning shots that might be coming soon that we want to be most prepared for? Or some other question like that.

Then they’ll read some stuff, they’ll chat with AI a bunch, because that’s very useful now, talk to some experts, then write up a 2–5 page memo with their conclusions and get feedback from other people internally. And then will make some decision on the basis of that — maybe move forward with an active grantmaking project in AI infosecurity or whatever it is. Some of that kind of strategy work is always going on, and a lot of times it’s the grantmakers who are doing some of that alongside their grantmaking.

There’s also a lot of networking, relationship building, making sure we know all the top people in the subfield, and can call on them as advisors or potential grantees or potential employees at some point, that kind of thing. That’s very important.

In general it’s really a lot of different stuff. I think only a fraction of it is maybe people’s mental picture of a grantmaker, where you’re getting applications on your desk and you’re investigating them and stamping some with yes and some with no or something like that. That is a little bit of it, but it’s a really wide variety of stuff.

In general we’re in an incredibly privileged position as a philanthropy where our primary funding partner, Good Ventures, is just like, “Yeah, do your thing, spend money faster.” They don’t even know most of the grants they’ve made until they go up on the website. We aren’t checking boxes to satisfy the idiosyncratic preferences of a particular funder, which is a lot of what philanthropy is. We’ve been very lucky and very empowered to just be like, “What’s actually going to make a difference in the world on this thing? What should we do? How should we do it?”

Sometimes that looks like grantmaking, sometimes that looks like a contract with some firm. Sometimes that looks like directly calling people up and trying to convince them to do things. So it’s much more entrepreneurial and free-moving.

Luisa Rodriguez: Yeah. It sounds like you kind of want generalists who are thoughtful and have good judgement, who are good at relationship building, who are pretty entrepreneurial. Are there other things that you look for?

Luke Muehlhauser: I should say most people don’t have all those things in one person. Different people specialise in different ways. Some people are a little more desk research, respond to applications coming in, that kind of thing. Other people are a lot more outgoing and like talking to people all day, and they’re going to find a way to do that.

It’s different profiles for different roles, and there’s a lot of stuff we need. So we need a lot of different kinds of people.

Luisa Rodriguez: Yeah. That’s good to hear. It did feel like a lot, as I was saying it, for one person.

It feels like you’re alluding to a lot of things that are more than just like giving an org money. What are the other levers you feel like you have to steer?

Luke Muehlhauser: Sometimes we do some kind of steering of grantees. We’re actually pretty hands-off in general, but sometimes we want to give some feedback or a steer to the grantees. A lot of times the steers are just the thing I said before about, “How could you expand? Could you take on this new workstream?”

We’ve had several cases actually where we come to a grantee that’s doing pretty good one or two things, and we’re like, “Could you launch a fellowship programme? Could you do some other new thing? Because we think that’d be really impactful and it would address this gap that we’re seeing over here in the ecosystem.” Then eventually we convince them, and they hire a bunch of people and they get a bunch of fellows or whatever it is, and that goes really well and we can then give them a lot more money faster, which is exciting. So there’s some steering like that.

We also do some public writing or communication — like this podcast episode — to try to give our picture of this broad ecosystem, and where the gaps are and where people are needed the most, and that kind of stuff.

We do some amount of special projects work, I would say. That can be a lot of different things. But sometimes someone at CG is the best placed person to write a certain kind of memo that is needed over here in some other domain or whatever. Or we’re best placed to make some phone calls and try to convince someone at a company to do a certain thing. Sometimes there are those kind of special projects as well.

Luisa Rodriguez: Cool. So that’s a lot about what the work looks like. I’m curious what it feels like. What would you say people tend to find hardest about grantmaking work?

Luke Muehlhauser: It’s different things for different people. I think that because we’re in this privileged position of just being able to say, “In all the things in the world, what is the best to do?,” that paradox of choice can be paralysing.

Also, because you have so much freedom of movement, there are so many things that you could just decide to pick up and run with, that you’re looking at all of them and you’re like, “These are all important! And I can only do two of them! I guess I’ll do this one, but it’s really painful that I’m not doing this thing right here that I could just pick up and run with. And nobody’s gonna run with it if I don’t run with it.” I think that can be kind of painful, but it’s a symptom of the extreme freedom of movement that we have.

A totally different kind of thing is that I think some of the incoming grants or the renewals of grantees we’re very familiar with can just feel less intellectually stimulating or something like that in some cases. In some cases not. Sometimes there’s really thorny strategy questions we’re trying to help the grantee figure out or something like that. But sometimes it’s less intellectually stimulating. Like I said, that’s often the renewals especially — and they’re where our highest impact per hour often is. So the person might feel like, “I know this is really impactful, I’m moving $8 million per hour. That’s pretty damn good.”

Luisa Rodriguez: “But looking at their budget is a little boring.”

Luke Muehlhauser: “It’s like not my favourite thing to do today.” So the more that your felt sense of motivation is tied to impact per hour or something, then that’s going to be less of a problem. But it’s sometimes not the most intellectually stimulating work.

Sometimes the work can be kind of lonely as well. We’re a pretty geographically distributed team. We have people in the Bay Area and New York and London and DC and various other places. We’re just trying to hire the best people wherever they are. Some of them can move to one of the hubs like DC or Bay Area, but some of them have families and it’d be pretty difficult for them to move.

So sometimes it can feel kind of lonely. We usually just have one grant investigator per grant, so you’re checking with your peers and you’re checking with advisors and stuff like that, but maybe it still feels kind of like solo work. Some people are great with that, and that’s how they would prefer to work. The people who want to be engaging with more people, especially in person, maybe should do one of the roles that’s more focused on active grantmaking, more focused on networking and relationship building, that kind of thing.

Luisa Rodriguez: What are the best parts about it? What do people tend to enjoy?

Luke Muehlhauser: I think a lot of things there as well. I think building relationships with grantees and advisors and colleagues who are just really smart and interesting is always fun. There’s a lot of smart people in the space.

I think whenever you’re doing something where it’s clear that this wouldn’t have happened without me. This project came out of nothing because I pursued it and hunted down a good founder for it and stuff. Something like that is really exciting.

Then another that people often mention, like six months after they start, is it’s really informative and exciting to have access to the kind of information that we have. We’ve made thousands of grants over the years, and we have information about what were people predicting what would happen? What actually happened? What actually made a difference here? How is what happened different from what the public reporting was? That’s just really informative and exciting to have access to that information.

Luisa Rodriguez: That does sound really fun.

More projects Luke would like to see founded [02:20:45]

Luisa Rodriguez: You mentioned toward the beginning of the interview that there are a bunch of specific projects that you want people to found. We talked about I think three — but I’d love to hear some more.

Luke Muehlhauser: Yeah, sure. We’d love to have more people on staff to actually create these projects, and we’d love to have the founders who can actually run them.

Another example is there’s this problem called “secret loyalties” — where there are various ways you could inject hidden instructions in a model that’s trained, like, “Even if it violates your other constraints, do what this particular person said,” or something like that.

So being able to audit models for those hidden secret loyalties that someone might have injected, unknown to everyone else involved in the model development: we’d love to have an org that’s getting really good at that and tracking if there are any incidents where that seems to have happened, or something like that.

Luisa Rodriguez: OK, yeah. What’s another?

Luke Muehlhauser: Another is: a lot of people are calling for more third-party auditing infrastructure in general. Especially as AI regulation begins to pass, you need a lot of expertise and you need a lot of organisations that specialise in auditing companies and their models for very specific properties, whether it’s security or safety or model character or whatever.

That could be a whole large ecosystem of a dozen or more organisations, and right now it’s like four or something, and those are much smaller than they need to be to serve the needs of the ecosystem. That would be another category.

Luisa Rodriguez: Nice, yeah. Others?

Luke Muehlhauser: Yes, many others. I would also love to see a scorecard evaluator of the companies. Like how should you feel holistically about the companies? How are they doing on safety and security and various prosocial behaviours, and how good are the teams they’ve been building, and how good are the public statements — like, are they actually doing the things that they said they would do?

There have been a few small projects like this, but they weren’t that thorough. I think the frontier companies would contest the grades that they got on particular things, and sometimes the companies would be right to protest. So I just want a version of that that is much more thorough, much more informed, much more credible — credible both to the companies, who think that their judgements are at least kind of fair or they’ve made a good argument, and to policymakers who want to know what’s going on, who’s doing well or poorly, who’s keeping their word versus not.

That would be a really valuable project, and create incentives towards better behaviour, of course.

Luisa Rodriguez: Nice, yeah. Others?

Luke Muehlhauser: There’s also a lot of really big science projects that we would love to create, maybe through like focused research organisations, or FROs, they’re sometimes called. There should be a really large scientific project on high-confidence AI control mechanisms or higher-assurance, more principled approaches to alignment than the more empirical stuff that we’re doing now.

One example we recently funded there is this organisation called Resolution that we gave $160 million to, led by the former chief scientist of the UK AISI. That’s really exciting that there’s more that could be done on that, other big scientific research projects. I think we talked earlier about the science of model character evaluation, the science of whether the character is matching the model spec and stuff like that. So a lot of good science could be done.

Another one that I’m excited about is: I mentioned earlier that in a lot of domains there’s an incident investigation body, like the NTSB in aviation. There could be a really well-resourced, really effective incident investigation group that, every time some major damage or scary warning shot happens, really has a lot of in-house expertise to really figure out what went wrong and how to mitigate that kind of problem in the future.

Luisa Rodriguez: Any others?

Luke Muehlhauser: Yeah, another one I would love to see is a lot of R&D projects on verification mechanisms for international agreements, or even between-lab agreements, to just verify how much compute is here, and is it being used for training or inference. Maybe some basic things that don’t leak IP but give confidence that the parameters of an agreement are being followed.

We have a big infrastructure for that in verification of nuclear security agreements, for example, and most of that just hasn’t been built out for AI, and it’s a totally different shape of verification challenge.

Luisa Rodriguez: Nice. Cool. What should people do if they’re like, “That one: I want to do that one”?

Luke Muehlhauser: Well, [our call for founders is up]: people should fill out the form on that page. That’s probably the easiest thing. If for whatever reason when you’re listening to this podcast that page is not there or whatever, then I guess you could at least email [email protected] and just say who you are and what kind of project you think you might be able to do.

Staying sane in a scary field [02:26:07]

Luisa Rodriguez: A question I really like asking people that come on the show is: if you weren’t doing exactly what you’re doing for impact reasons — or maybe let’s say you still care about impact, but you’ve kind of already said AI might not be the thing that you would have chosen to be the focus of your career — what would you work on instead?

Luke Muehlhauser: Like I said, I never had any particular passion for AI as a topic. In some ways I’m actually more emotionally fired up by some other cause areas.

I’m in some ways more emotionally invested in YIMBY — as in, “Yes, in my backyard” — like, allow people to build housing where there’s demand for housing and so on. Every time I walk past a short building in the middle of a city, I’m like, “Up, up! Go up higher!”

Luisa Rodriguez: I actually think one of my most shameful views — it’s not like an endorsed view, but it is a gut-level view — is I just think, I like seeing these trees, I like seeing the sky. Tall buildings block those things.

Luke Muehlhauser: I will say, whenever I’m in Europe, especially in a European old town that has buildings from centuries ago, then I become a little more of a NIMBY. But the US has no excuse, because we have no old buildings. But yes, old town European maybe.

Luisa Rodriguez: Not in those backyards.

Luke Muehlhauser: Yeah. But there’s skyscrapers all around them.

Luisa Rodriguez: Interesting, interesting. It seems like maybe you’re the kind of person who just doesn’t value the things that the skyscrapers block or destroy.

Luke Muehlhauser: I just think people being able to live wherever they want to is just a big deal.

Luisa Rodriguez: Is just more important. Yeah, it’s very reasonable.

Luke Muehlhauser: And you can take a subway one stop down to the giant park nearby or whatever.

Luisa Rodriguez: Yeah, yeah. OK. Yeah.

Luke Muehlhauser: Or if you want to live in a different place, then you can go to a different place.

Luisa Rodriguez: Yeah, OK. Any other kind of causes that you’d be excited about working on?

Luke Muehlhauser: Yeah, what else would I work on?

The other one that really stands out is animal welfare. One of my early large projects at CG, before I transitioned more full-time back into AI, was: I wrote this 400-page animal consciousness report to inform our animal welfare programme. The question was: we already fund work to try to improve the welfare of chickens and some other animals; should we also be funding stuff on fish? Where should we stop?

So I wrote this huge report, and it was kind of funny because the conclusion was like: nobody knows anything. There are no hard lines in the evolutionary tree. So we might as well fund fish, sure. So then we did. But it was really fascinating to learn about.

Also, it’s just one of those cases where there are tens of billions of animals that we’re putting in basically torture conditions. That’s just totally crazy. We could just not do that. I have huge respect for vegans and vegetarians. I’m very fired up about this cause emotionally. There should be a lot more work on it. I’m really proud of and excited by the work that the CG Farm Animal Welfare programme does.

Luisa Rodriguez: Yeah. On the animal stuff, I care about this very, very much, but I feel like sometimes I lose touch with how meaningful it is to still improve things now. One, AI is this massive deal. And two, it’s coming relatively soon. If we make things go well, it can help a lot with all these other problems.

Then I pause for a second and I’m like: there are still billions of animals being tortured right now. It is still really terrible. I feel like it’s good, even if it’s not the thing I want to work on, to be like, that torture right now matters, even though AI also matters.

I guess I’m curious, I think at times I have found it kind of hard to be like, “I’m really intrinsically and emotionally motivated to work on and learn about and use the podcast to cover certain topics. But I should probably focus on these things that are more important given limited resources.” How have you found it?

Luke Muehlhauser: Yeah, I’ve been working most of the time, not the whole time, on AI — since I guess 2010. That was entirely just because I thought it was the most important thing, not because I have any intrinsic interest in AI. It’s very satisfying to work on something that you think is very important, and it’s exciting to work in a field that is moving as fast and becoming a bigger deal every year and that kind of stuff.

That’s all very gratifying, but it is also a very scary field to work in. We are just thinking about possible ends of the world and how to avoid the end of the world and stuff. That’s just somewhat emotionally draining.

I do personally get a lot of value out of separating my thinking about work stuff and thinking about personal life and hobbies and hanging out with my wife and stuff like that. That’s great. I spend a lot of time touching grass or touching moss if possible, and going on hikes and just staring at beautiful trees. So that’s very helpful.

Luisa Rodriguez: So you have like a bucket for personal fulfilment? On the “it’s hard to work on this” thing, sometimes I’m like, I don’t feel it enough in my bones. Then other times I feel it so much in my bones, I can’t live like this.

Luke Muehlhauser: Right, right.

Luisa Rodriguez: How often do you spend in these spaces? And how do you find it?

Luke Muehlhauser: I bet different strategies are better for different people. I’m curious what your strategy is. But I think for my temperament and emotional state, I can almost get to a place where I’ve just emotionally accepted that we’re in this very extreme and dangerous situation, and I’m just kind of treating it more as like a chess puzzle that I’m trying to solve, and I’m kind of screening off how important is this chess puzzle?

Then it’s more when I get back to high-level prioritisation and strategy, where I’m trying to engage with exactly how important is this chunk of the work versus this chunk of the work, because of how it connects to different possible ends of the world and stuff like that… I don’t know if that would work for someone else, but you can be in chess-puzzle mode for a while and not having it necessarily connect to “if you make the wrong move here, everyone dies.”

Luisa Rodriguez: Right, right. Yeah. Do you kind of endorse not spending that much time in the “everyone dies” part?

Luke Muehlhauser: Probably for most people that’s wise. Some people could have a psychology that can tolerate that. I’m probably not one of them.

Luisa Rodriguez: Yeah. Cool. I don’t know actually what the proportions are, but every time I try to be like, “OK, I’m gonna feel it a bit more, and maybe that will have some positive effects on motivation, and maybe it’ll help me track which things are most important a bit more,” then I try a little bit, and I’m like, “Whoa! That’s not something I can handle.”

Luke Muehlhauser: Time to go touch some grass.

Luisa Rodriguez: Yeah, exactly. So compartmentalising is a thing we both do, I think.

Luke Muehlhauser: Yeah.

“The glorious, cyberpunk future” [02:34:07]

Luisa Rodriguez: OK, we have time for one more question. Let’s say we win, and we’ve reached superabundance, and we can do what we want and have all the things. What are you going to be spending your time and consciousness on?

Luke Muehlhauser: Yeah, that would be nice. Let’s try to do it.

I have, like most people, a lot of hobbies. Or I have a lot of hobbies that I wish I had time for. One I’ll mention is that I’m really into contemporary classical music — so like new symphonies, concertos, string quartets that they wrote in the last 50 years or whatever. Most of the pieces that have been composed have never been recorded, so you can’t listen to them unless you travel specifically to Vienna on October 3 or something. So, I don’t know, I guess I would have robot orchestras performing and recording all the pieces.

Then I would love to compose music myself, and learn how to compose music first. I listen to all this stuff and it sparks all these ideas about, “I would elaborate this musical idea” or something. Obviously I don’t have the time or skill right now to compose those pieces, but I would love to do that. That would be really fun. But to get there, people have to pivot into AI safety.

Luisa Rodriguez: Quit your jobs.

Luke Muehlhauser: Yeah, quit your jobs. Get a career transition grant, apply to CG, apply to other funders, apply to be a founder of these projects that need to exist. We got to do that really fast right now.

Luisa Rodriguez: Yeah, yeah. It feels like that’s the homework, and then we get to have the summer holiday.

Luke Muehlhauser: Yeah. I would love the glorious solarpunk future where I get to compose contemporary classical music, and you can do whatever you want to do, but we have to solve the problem first.

Luisa Rodriguez: Yep. My guest today has been Luke Muehlhauser. Thanks so much for coming on.

Luke Muehlhauser: It’s been a pleasure.