Transcript
A request for AI company staff [00:00:00]
Rob Wiblin: I have a request of all AI company staff: reflect today on whether the project you’re working on is the best thing you can plausibly be doing to either delay recursive self-improvement (RSI) or make it go well. And if it’s not, push to be moved to an alternative project you think is more valuable.
Obviously many staff are already trying hard to do exactly that. But the strategic situation is changing incredibly quickly, so if you aren’t asking yourself the hardest questions regularly and being honest about it, there’s a growing risk you’ll accidentally be wasting your time.
On the technical side there are a few lines of work that stand out as especially likely to remain relevant up to RSI:
- Work that solves fundamental problems in alignment, obviously
- Work that gets current models to better pursue alignment research today
- And work that ensures we can catch, understand, and announce model misbehaviour or misuse as early as possible
By contrast, some work that looks suspicious to me includes:
- Any alignment methods that throw band-aids on existing issues, but which can’t scale to the models that will be engaging in recursive self-improvement
- Anything that has the practical effect of making underlying alignment problems less visible to the public and policymakers — without simultaneously fixing the drives causing the behaviour
- Anything that primarily improves model capabilities — especially the ability of models to autonomously do AI R&D
- And anything that’s primarily about attracting customers, or achieving commercial advantage
I appreciate that these prioritisation decisions are extremely difficult and complex. Already in these two short lists there’s a clear tension:
- On the one hand I think it’s valuable to get today’s models to more vigorously and sincerely help with technical alignment research.
- On the other hand, I’m wary of alignment methods that work today but won’t work tomorrow, and of anything that makes AIs better at AI R&D — even though many types of AI alignment research are extremely close to AI R&D.
That’s why neither I nor anybody else can do this prioritisation for you. You have access to experimental results we don’t. And you can sensibly ask yourself questions like: “Is my work doing more to assist with getting useful alignment research out of today’s models, or doing more to shorten timelines to RSI?” in a way nobody far away from your particular situation can do.
I also know that for almost all of you, this is not going to be the first time someone has prompted you to make these tradeoffs better. But with things as chaotic as they are, it can be easy to slip into working on something that’s not especially impactful, or even harmful.
For instance:
- Maybe you’re working on a project you started a year ago, when you thought RSI might first become possible in 2029. Now you think recursive self-improvement will become possible in 2027 most likely, and your current work doesn’t seem likely to be relevant by then. But that shift in timelines hasn’t yet flowed through to a change in your work.
- Maybe you’ve gotten more safety-pilled this year, by Mythos and Hugging Face and all the rest. But you haven’t yet made the hard decision that requires you to work on something substantially different.
- Maybe you’ve been so rushed fighting fires and shipping models the last three months that you haven’t had a single afternoon to reflect on what your high-level priorities ought to be.
- Maybe you think there are more important projects, but what you’re currently doing is what your manager wants, and pushing for something else feels awkward, difficult.
- Maybe you’ve invested a year in a promising project and gotten real results from it. But you’re increasingly unsure whether it deserves a second year, and giving up partway through would be unpleasant.
These are all incredibly human and relatable things we’ve all done. And in a situation that weren’t so incredibly dangerous and urgent, we’d leave people to reassess when they’re ready to do so. But as it is, there’s no time for that.
Not being at a frontier company myself, there’s a lot of confidential information I lack. But to give you some ideas, here’s how some priorities appear to have shifted to me.
For this purpose, I’m going to take it as given that the internal models inside Anthropic and OpenAI are much more powerful again than Opus 5.5 and Astra, and that OpenAI has seen alignment on its unreleased models go backwards. And in keeping with the dominant attitude within frontier companies, I’m taking it as given that fully automating AI R&D may become possible in 2027, 2028, or 2029.
Chain of thought [00:03:34]
As I discussed last time, OpenAI has reported that the effectiveness of chain-of-thought (CoT) monitoring on its models is dropping precipitously. Google has now reported the same with its latest model, Argon.
This trend has various implications:
- Chain-of-thought monitoring and analysis today is absolutely still useful inasmuch as it helps us to identify, understand, publicise, and respond to misbehaviour sooner. Only finding out about rogue AIs months after they happen robs us of time to react.
- But chain-of-thought monitoring will only scale to the beginning of recursive self-improvement if we can find ways to reduce what models can accomplish without a chain of thought, or keep models honest and forthcoming on that chain of thought.
If you’re working on chain-of-thought monitorability as a strategy, you have to ask yourself if you think those things are going to happen. Opus 5.5 doesn’t have the same ability to control what it writes in its chain of thought, so maybe it can be done. If you believe that will work, keep going and maybe pitch in to make it work. If not, it’s time to reprioritise.
The increasing problems with monitorability are also an update in favour of white-box monitoring techniques like mech interp, over black-box ones like CoT monitoring.
Separately, I think they’re also an update in favour of making recursive depth or no-CoT capabilities a regulatory target. That’s a story for another day.
Don’t paint over the cracks [00:04:44]
Imagine that you’re responsible for building maintenance on a large apartment block. People keep complaining about cracks appearing in an outer wall, and paint and plaster falling down on top of them. You do a walk around the building and realise that this isn’t just a superficial problem with the outer wall. The building has a problem with its foundation that’s leading half of it to subside, and the cracks are just a sign of that deeper issue. Over time the subsidence will likely get worse and eventually become actually dangerous.
Like anyone, you’ve only got limited time and money. There’s four things you might do:
- Do nothing about either problem.
- Repaint the wall but do nothing about the foundation.
- Alternatively, let the visible cracks stay, and prioritise fixing the underlying issue.
- Or finally, both repaint the wall and fix the foundation.
Now, there’s nothing wrong per se with painting over cracks and repairing plaster. High-quality building maintenance is going to do both.
But the underlying problem with the foundation is obviously far more important. If you only have resources to do one of the two, you fix the foundation that could lead the building to collapse, not the surface issue that’s merely aesthetic, and the thing that allowed you to notice the problem in the first place.
And if you know the foundation is not on track to be repaired, painting over the cracks in the wall so externally things look fine — that’s a sort of deception on the residents.
Don’t be cajoled or peer-pressured into participating in an effort that only makes a risk harder to see, rather than actually reduces it.
It may not be your most aligned model ever [00:06:00]
Every time a model card comes out now, labs effectively say, “Our safety evals look better than ever” — but then also say in slightly more technical language, “We trust our safety evals less than ever.”
That’s for many reasons, including eval awareness, ability to scheme off the chain of thought, and the risk of having effectively trained on the test set.
I’m glad that reality isn’t being covered up. But if you’re working on safety evals you suspect now misinform people as much as they inform, ask what it would take to make them truly credible again:
- Scenarios the model actually can’t tell apart from real use
- Methods of directly suppressing eval awareness
- Or maybe something new I’ve never heard of
If your work is helping restore credibility so you really believe your safety evals, great. If not, in my view it’s time to reassess and probably find something more useful to do.
Capabilities advances look worse [00:06:44]
However good or bad it was in the past, as we get closer to recursive self-improvement and to superintelligence, work that improves AI capabilities looks less attractive than it used to.
That’s for a number of reasons:
- Today’s models are closer to the models you plan to hand over AI R&D to, and so which pose the gravest risks. That means we have more relevant model organisms to study and improve, without needing to improve capabilities further.
- The models we have now are actually able to make really useful contributions to all kinds of technical alignment work, and even policy and governance work sometimes. The more we can draw things out, the more time we have to exploit that assistance.
- The later things get, the less time there is remaining for any career capital you accumulate in the course of improving AI capabilities to pay off and more than compensate for any damage done.
- Existing internal models are smart enough for most business applications now, so one can potentially have a flourishing AI business by making them better and cheaper and more aligned products for enterprise customers, rather than pushing them closer to superintelligence.
- Capability advances beyond what internal models can already do now pose actual catastrophic risks primarily now via bio misuse, or models silently compromising AI company systems.
Doing capabilities research at an unusually reckless lab is just extremely difficult to defend at this point. If you work at a company you think has the world’s best safety practices, though, I know the steelman: if the most careful lab isn’t at the frontier, someone else more reckless will be.
And it’s a reasonable point. But two things are gradually eroding its persuasiveness over time:
- Firstly, breakthroughs are leaking to less careful actors very quickly now via distillation, espionage, and poaching of staff. So a lead is harder than ever to maintain.
- And second, the frontier we’re approaching is now automated AI research itself. Getting there first only helps if the careful lab can tell whether it’s possible to do it safely, and if so, how. And today, nobody can.
That’s why we have to pace the frontier.
And it’s also why, if you’ve been striking one tradeoff between ‘increasing capabilities’ against ‘other impacts,’ it’s necessary to recalculate and double-check if what you’re doing still really makes sense.
Ever less need for altruists to do commercially focused work [00:08:50]
This shift has been going on for years, but all the frontier AI companies are huge commercial enterprises now. Many people working at all of them are thinking of them primarily as businesses, and their job is to figure out how to attract customers and build better products. The commercial incentives to do that kind of work are enormous, because it brings in a lot more revenue than it costs.
If you’re motivated primarily by reducing catastrophic risks, and you’re primarily working on improving products or the customer experience, you should think hard about whether that’s actually the greatest contribution you could be making. I think typically, in your absence, the company would get similar results from somebody else, while you could be doing much more important and neglected safety and alignment and governance work.
Pacing the frontier [00:09:29]
We’ve gotten to the point where AI capabilities are truly dangerous without solving alignment. We tried to avoid that for many years by waking up the world sooner, and sadly we’ve failed.
If we just continue to barrel forward, improving capabilities faster than alignment — or, heaven help us, proceed to fully automate AI R&D with safety cases as weak as the ones we possess today — the risk of human extinction is terrifyingly high.
That means most good futures now run through scenarios where we somehow succeed at pacing the frontier, to buy ourselves a couple of years for automated alignment research and data centre security and the law and so on to catch up.
Many actors can contribute to that effort:
- There’s only a few relevant companies at the moment, so we only need a handful of companies to voluntarily limit the types of RSI they engage in to begin buying time. OpenAI’s Safety and Security Committee could potentially step in and object to dangerous research practices at OpenAI, for example.
- California might contribute some useful laws in a special session that they might hold in November and December.
- And if the problem gets even more obvious, a new Congress might act decisively as soon as next year — whether or not the president tries to veto it.
- If Congress doesn’t act, public outrage might be necessary to force their hand. But if the problems are obvious enough I think that’s actually pretty realistic, given how the public feels about AI even now.
- And such pacing arrangements would soon have to extend to China, which will require credible technical demonstrations of why they’re necessary, and all the rest of it.
As you know, the Overton window here has shifted enormously in the last two months. So if you’ve been working on something other than pacing the frontier, but you think one of the steps above is something you could make a big difference to, you should reassess and decide if that might now be your greatest opportunity for impact.
Organising staff [00:11:07]
Changing projects is something you can do on your own. But you may be able to have even more leverage acting as a group. If you’re on the technical staff, you have extraordinary bargaining power. You can slow things down simply by not taking part. You can leave for other impactful, well-paid work. And when you speak up, the public takes you far more seriously than outsiders like me.
But your influence within the company only lasts so long as your employer needs you, and you’re building the machine researchers designed to make you obsolete. Once that happens, you’ll lose much of your ability to direct events. So if this power is going to be used, it has to be used ASAP.
If staff at OpenAI and Anthropic simply refused to keep walking us closer to the cliff edge, that could be the first of several mechanisms that let us pace the frontier. In practice, that might mean pushing for limits on how far and how fast AI R&D gets automated; or insisting on credible, independent safety assessments before dangerous thresholds are crossed; or demanding a way to take serious concerns all the way to the board without any risk of retaliation.
This approach actually sidesteps a problem the companies face, and helps them out in one respect: an agreement between lab leaders to slow down runs into antitrust law. But staff pressing their own employer to slow down as a matter of personal conscience don’t have that problem.
The hard part is usually that nobody wants to be the one to go first, which is why agreements that only kick in once some critical threshold has been reached might help out.
You will know far better than me how much any of this is realistic, but maybe somebody listening is in a position to help get the ball rolling on this. One group already working on this is the Coalition of Concerned AI Staff, which launched earlier this year to help staff act on their beliefs as a group.
Consider working somewhere else [00:12:38]
Alternatively, your best opportunity might not be at the organisation you currently work for.
We recently interviewed Geoffrey Irving about the new organisation he’s cofounding: Resolution.
They’re working to two- to three-year timelines to superintelligence, scaling up incredibly fast, and aiming to tackle neglected bets in alignment that can plausibly continue working through to recursive self-improvement, and perhaps beyond.
The primary mechanism will be automating the research — that is to say, guiding AIs through doing it for us. And their approaches are more formal and mathematical than what AI companies are doing, so now that AI is basically in the process of solving mathematics for us, it’s easier to see those lines of work paying off. It’s extremely compute intensive, but if they make early progress I think they’re very likely to get access to the billions in compute that they’ll need to make the full project work.
Listen to our interview with Geoffrey if you want to learn more. And if you don’t feel amazing about what you’re doing now, consider applying there or to another group doing outstanding work outside the leading companies.
Reprioritise [00:13:27]
The above was just my current best guess based on what I see not being at a frontier company. I’m sure some of it is wrong, and if you do think some of it is misguided or misinformed in some specific way, I’d sincerely like to hear — drop me an email at [email protected].
I’m sure many of you have reprioritised and been bold about changing what you’re working on in recent months. But if you haven’t gotten to that yet, there’s no time like tonight.
Take 20 minutes. Write down in a few sentences how your current project makes RSI go better or come later. Is it focused on a sensible target? Will it pay off in time? If you walked away, would the company find someone else to do a similar job to what you’re already doing? What else could you do that you suspect might be even more impactful? Then show your answers to someone you respect and ask them for a candid opinion.
We all wish this weren’t happening in our time. At least I definitely do. But sadly that’s not for us to decide. All we have to decide is what to do with the unfortunately brief time that is given us.
And on that note, I’ll speak with you again soon.
