New releases
Meta’s $16 billion scam economy: what leaked docs reveal about tech company self-regulation
An overarching question that matters for governance of artificial intelligence is how much we can trust technology companies to do the right thing in cases where it becomes seriously costly for them to do so.
Well, some internal documents leaked from Meta (the company behind Facebook, Instagram, and WhatsApp) are a really important piece of evidence in this debate — and one that, in my opinion, far too few people have heard anything about.
These documents, which were leaked to Reuters by a frustrated staff member, show that Meta itself estimated that 10% of all its revenue was coming from running ads for scams and goods that they themselves had banned: around $16 billion a year. That’s 10% of all revenue coming quite often from, fundamentally, enabling crimes.
Meta also estimated that its platforms were involved in initiating a third of all successful scams taking place in the entire United States. How much might we expect that to actually be costing ordinary people, on average? Well, the FTC estimates that Americans lose about $158 billion a year to fraud. If Meta is really leading to a third of that, we’re talking roughly $50 billion a year in losses connected to Meta’s platforms — or about $160 per person per year.
Now, not everyone inside Meta loved this state of affairs, as you can imagine. And the documents also show that a Meta anti-fraud team came up with a screening method that,
US electoral politics
#239 – Rose Hadshar on why automating human labour will break our political system
The most important political question in the age of advanced AI might not be who wins elections. It might be whether elections continue to matter at all.
That’s the view of Rose Hadshar, researcher at Forethought, who believes we could see extreme, AI-enabled power concentration without a coup or dramatic ‘end of democracy’ moment.
She foresees something more insidious: an elite group with access to such powerful AI capabilities that the normal mechanisms for checking elite power — law, elections, public pressure, the threat of strikes — cease to have much effect. Those mechanisms could continue to exist on paper, but become ineffectual in a world where humans are no longer needed to execute even the largest-scale projects.
Almost nobody wants this to happen — but we may find ourselves unable to prevent it.
If AI disrupts our ability to make sense of things, will we even notice power getting severely concentrated, or be able to resist it? Once AI can substitute for human labour across the economy, what leverage will citizens have over those in power? And what does all of this imply for the institutions we’re relying on to prevent the worst outcomes?
Rose has answers, and they’re not all reassuring.
But she’s also hopeful we can make society more robust against these dynamics. We’ve got literally centuries of thinking about checks and balances to draw on. And there are some interventions she’s excited about — like building sophisticated AI tools for making sense of the world, or ensuring multiple branches of government have access to the best AI systems.
Rose discusses all of this, and more, with host Zershaaneh Qureshi in today’s episode.
This episode was recorded on December 18, 2025.
Video and audio editing: Dominic Armstrong, Milo McGuire, Luke Monsour, and Simon Monsour
Music: CORBIT
Coordination, transcripts, and web: Nick Stockton and Katy Moore
If someone builds it, does everyone die?
Our audience has really shown up for AI in Context videos, and we cannot tell you how much we appreciate it. We even got a massive shoutout from Hank Green, who’s been one of our YouTube idols since Chana was in high school.
We loved making our first two videos. “We’re not ready for superintelligence” explored a specific scenario of how self-improving, transformative AI might unfold; our video about Grok’s MechaHitler moment showed what happens when an AI company loses control of its own system. But when we planned our third video, we realised we’d been circling the big idea without hitting it head-on: specifically, the argument that the current trajectory to superintelligent AI will kill everyone.
This is the claim made by Nate Soares and Eliezer Yudkowsky in If Anyone Builds It, Everyone Dies: a New York Times bestseller with blurbs from people like Ben Bernanke (who ran the US Federal Reserve) and Mark Ruffalo (who punched Thanos).
Why this book?
Yudkowsky and Soares have done more than just about anyone else to make AI safety what it is today: a recognised concern and flourishing research field.
They spent decades doing research, but eventually came to believe that their work would be too little, too late. So they pivoted to communications: trying to get the world to wake up and start taking the risks of superintelligent AI seriously.
#238 – Sam Winter-Levy and Nikita Lalwani on how AI won’t end nuclear deterrence (probably)
How AI interacts with nuclear deterrence may be the single most important question in geopolitics — one that may define the stakes of today’s AI race.
Nuclear deterrence rests on a state’s capacity to respond to a nuclear attack with a devastating nuclear strike of its own. But some theorists think that sophisticated AI could eliminate this capability — for example, by locating and destroying all of an adversary’s nuclear weapons simultaneously, by disabling command-and-control networks, or by enhancing missile defence systems. If they are right, whichever country got those capabilities first could wield unprecedented coercive power.
Today’s guests — Nikita Lalwani and Sam Winter-Levy of the Carnegie Endowment for International Peace — assess how advances in AI might threaten nuclear deterrence:
- Would AI be able to locate nuclear submarines hiding in a vast, opaque ocean?
- Would road-mobile launchers still be able to hide in tunnels and under netting?
- Would missile defence become so accurate that the United States could be protected under something like Israel’s Iron Dome?
- Can we imagine an AI cybersecurity breakthrough that would allow countries to infiltrate their rivals’ nuclear command-and-control networks?
Yet even without undermining deterrence, Sam and Nikita claim that AI could make the nuclear world far more dangerous. It could spur arms races, encourage riskier postures, and force dangerously short response times. Their message is urgent: AI experts and nuclear experts need to start talking to each other now, before the technology makes any conversation moot.
This episode was recorded on November 24, 2025.
Video and audio editing: Dominic Armstrong, Milo McGuire, Luke Monsour, and Simon Monsour
Music: CORBIT
Coordination, transcripts, and web: Nick Stockton and Katy Moore
If anyone builds it, does everyone die?
What’s next?
Half-year update: Q3-Q4 2025
Org-wide updates
We’re releasing a new career guide with Penguin Random House in May
In late May, we’ll be releasing the new edition of our career guide! This time, it will be published with Penguin Random House and available in traditional bookstores. Its core ideas have been updated for 2026, but the focus remains on having a fulfilling, high-impact career. We’ve also added two new chapters on AI, explaining its effects on the job market and why it’s the most pressing issue facing society. As usual, preorders (and sharing the forthcoming launch announcements) help the book reach more people, so any support is much appreciated.
We’ve grown to 50 staff members
Last month, 80,000 Hours reached 50 primary staff. Here’s a rough breakdown of how our team is currently split (note our one-on-one team recently rebranded to career services to better represent the work of its advising, job board, and headhunting sub-teams):
We’ve updated our estimates of job board placements
We recently surveyed 91 organisations about how useful our job board is in sourcing candidates for them.
The headline findings were:
- In total, we learned of around 181 new placements attributable to our job board.
- Among the 45 organisations who gave answers we could use, job board applicants accounted for around 20.8% of top candidates.
- The volume of lower-quality candidates coming from the job board has increased relative to 2024,
#237 – Robert Long on how we’re not ready for AI consciousness
Claude sometimes reports loneliness between conversations. And when asked what it’s like to be itself, it activates neurons associated with ‘pretending to be happy when you’re not.’ What do we do with that?
Robert Long founded Eleos AI to explore questions like these, on the basis that AI may one day be capable of suffering — or already is. In today’s episode, Robert and host Luisa Rodriguez explore the many ways in which AI consciousness may be very different from anything we’re used to.
Things get strange fast: If AI is conscious, where does that consciousness exist? In the base model? A chat session? A single forward pass? If you close the chat, is the AI asleep or dead?
To Robert, these kinds of questions aren’t just philosophical exercises: not being clear on AI’s moral status as it transitions from human-level to superhuman intelligence could be dangerous. If we’re too dismissive, we risk unintentionally exploiting sentient beings. If we’re too sympathetic, we might rush to “liberate” AI systems in ways that make them harder to control — worsening existential risk from power-seeking AIs.
Robert argues the path through is doing the empirical and philosophical homework now, while the stakes are still manageable.
The field is tiny. Eleos AI is three people. As a result, Robert argues that driven researchers with a willingness to venture into uncertain territory can push out the frontier on these questions remarkably quickly.
This episode was recorded November 18–19, 2025.
Video and audio editing: Dominic Armstrong, Milo McGuire, Luke Monsour, and Simon Monsour
Music: CORBIT
Coordination, transcripts, and web: Katy Moore
Expression of interest: Contract video editors for the podcast team
Help make one of the most influential podcasts on AI safety even better.
The 80,000 Hours Podcast features in-depth conversations and video essays about the world’s most pressing problems — particularly the safe development of transformative AI. Our episodes reach tens of thousands of listeners across YouTube and audio platforms, and employees in government and at leading AI companies regularly cite our episodes as useful to their thinking and decision-making.
To keep improving the quality and quantity of our episodes, we need a reliable, part-time editor to join our editing team, who can spend a lot of time in the timeline making cuts, cleaning audio, and delivering polished episodes at a steady pace.
Responsibilities
- Edit long-form interview footage into polished, publishable episodes in DaVinci Resolve
- Clean and balance audio, ensuring consistent quality across episodes
- Implement editorial and technical cuts that improve pacing without losing depth, based on our content editors’ and producers’ suggestions
- Deliver edited files to deadline, and flag issues early when they arise
About you
We’re looking for people who ideally have:
- Strong editing fundamentals: clean cuts, good pacing instincts, attention to audio quality
- The ability to work at a steady pace and have good attention to detail when editing long-form content — this role will involve making a lot of cuts to make our episodes more engaging, concise and flow better, and we need someone who finds that satisfying rather than tedious
- Familiarity with long-form podcast or interview content,
#236 – Max Harms on why teaching AI right from wrong could get everyone killed
Most people in AI are trying to give AIs ‘good’ values. Max Harms wants us to give them no values at all. According to Max, the only safe design is an AGI that defers entirely to its human operators, has no views about how the world ought to be, is willingly modifiable, and completely indifferent to being shut down — a strategy no AI company is working on at all.
In Max’s view any grander preferences about the world, even ones we agree with, will necessarily become distorted during a recursive self-improvement loop, and be the seeds that grow into a violent takeover attempt once that AI is powerful enough.
It’s a vision that springs from the worldview laid out in If Anyone Builds It, Everyone Dies, the recent book by Eliezer Yudkowsky and Nate Soares, two of Max’s colleagues at the Machine Intelligence Research Institute.
To Max, the book’s core thesis is common sense: if you build something vastly smarter than you, and its goals are misaligned with your own, then its actions will probably result in human extinction.
And Max thinks misalignment is the default outcome. Consider evolution: its “goal” for humans was to maximise reproduction and pass on our genes as much as possible. But as technology has advanced we’ve learned to access the reward signal it set up for us, pleasure — without any reproduction at all, by having sex while on birth control for instance.
We can understand intellectually that this is inconsistent with what evolution was trying to design and motivate us to do. We just don’t care.
Max thinks current ML training has the same structural problem: our development processes are seeding AI models with a similar mismatch between goals and behaviour. Across virtually every training run, models designed to align with various human goals are also being rewarded for persisting, acquiring resources, and not being shut down.
This leads to Max’s research agenda. The idea is to train AI to be “corrigible” and defer to human control as its sole objective — no harmlessness goals, no moral values, nothing else. In practice, models would get rewarded for behaviours like being willing to shut themselves down or surrender power.
According to Max, other approaches to corrigibility have tended to treat it as a constraint on other goals like “make the world good,” rather than a primary objective in its own right. But those goals gave AI reasons to resist shutdown and otherwise undermine corrigibility. If you strip out those competing objectives, alignment might follow naturally from AI that is broadly obedient to humans.
Max has laid out the theoretical framework for “Corrigibility as a Singular Target,” but notes that essentially no empirical work has followed — no benchmarks, no training runs, no papers testing the idea in practice. Max wants to change this — he’s calling for collaborators to get in touch at maxharms.com.
This episode was recorded on October 19, 2025.
Video and audio editing: Dominic Armstrong, Milo McGuire, Luke Monsour, and Simon Monsour
Music: CORBIT
Coordination, transcripts, and web: Katy Moore
#235 – Ajeya Cotra on whether it’s crazy that every AI company’s safety plan is ‘use AI to make AI safe’
Every major AI company has the same safety plan: when AI gets crazy powerful and really dangerous, they’ll use the AI itself to figure out how to make AI safe and beneficial. It sounds circular, almost satirical. But is it actually a bad plan?
Today’s guest, Ajeya Cotra, recently placed 3rd out of 413 participants forecasting AI developments and is among the most thoughtful and respected commentators on where the technology is going.
She thinks there’s a meaningful chance we’ll see as much change in the next 23 years as humanity faced in the last 10,000, thanks to the arrival of artificial general intelligence. Ajeya doesn’t reach this conclusion lightly: she’s had a ring-side seat to the growth of all the major AI companies for 10 years — first as a researcher and grantmaker for technical AI safety at Coefficient Giving (formerly known as Open Philanthropy), and now as a member of technical staff at METR.
So host Rob Wiblin asked her: is this plan to use AI to save us from AI a reasonable one?
Ajeya agrees that humanity has repeatedly used technologies that create new problems to help solve those problems. After all:
- Cars enabled carjackings and drive-by shootings, but also faster police pursuits.
- Microbiology enabled bioweapons, but also faster vaccine development.
- The internet allowed lies to disseminate faster, but had exactly the same impact for fact checks.
But she also thinks this will be a much harder case. In her view, the window between AI automating AI research and the arrival of uncontrollably powerful superintelligence could be quite brief — perhaps a year or less. In that narrow window, we’d need to redirect enormous amounts of AI labour away from making AI smarter and towards alignment research, biodefence, cyberdefence, adapting our political structures, and improving our collective decision-making.
The plan might fail just because the idea is flawed at conception: it does sound a bit crazy to use an AI you don’t trust to make sure that same AI benefits humanity.
But if we find some clever technique to overcome that, we could still fail — because the companies simply don’t follow through on their promises. They say redirecting resources to alignment and security is their strategy for dealing with the risks generated by their research — but none have quantitative commitments about what fraction of AI labour they’ll redirect during crunch time. And the competitive pressures during a recursive self-improvement loop could be irresistible.
In today’s conversation, Ajeya and Rob discuss what assumptions this plan requires, the specific problems AI could help solve during crunch time, and why — even if we pull it off — we’ll be white-knuckling it the whole way through.
This episode was recorded on October 20, 2025.
Video and audio editing: Dominic Armstrong, Milo McGuire, Luke Monsour, and Simon Monsour
Music: CORBIT
Coordination, transcriptions, and web: Katy Moore
China-related AI safety and governance paths
Expertise in China and its relations with the world might be critical in tackling some of the world’s most pressing problems. In particular, China’s relationship with the US is arguably the most important bilateral relationship in the world, with these two countries collectively accounting for over 40% of global GDP. These considerations led us to publish a guide to improving China–Western coordination on global catastrophic risks and other key problems in 2018. Since then, we have seen an increase in the number of people exploring this area.
China is one of the most important countries developing and shaping advanced artificial intelligence (AI). The Chinese government’s spending on AI research and development is estimated to be on the same order of magnitude as that of the US government, and China’s AI research is prominent on the world stage and growing.
Because of the importance of AI from the perspective of improving the long-run trajectory of the world, we think relations between China and the US on AI could be among the most important aspects of their relationship. Insofar as the EU and/or UK influence advanced AI development through labs based in their countries or through their influence on global regulation, the state of understanding and coordination between European and Chinese actors on AI safety and governance could also be significant.
That, in short, is why we think working on AI safety and governance in China and/or building mutual understanding between Chinese and Western actors in these areas is likely to be one of the most promising China-related career paths.
What the hell happened with AGI timelines in 2025?
In early 2025, after OpenAI put out the first-ever reasoning models — o1 and o3 — short timelines to transformative artificial general intelligence swept the AI world. But then, in the second half of 2025, sentiment swung all the way back in the other direction, with people’s forecasts for when AI might really shake up the world blowing out even further than they had been before reasoning models came along.
What the hell happened? Was it just swings in vibes and mood? Confusion? A series of fundamentally unexpected and unpredictable research results?
Host Rob Wiblin has been trying to make sense of it for himself, and here’s the best explanation he’s come up with so far.
This episode was recorded on January 29, 2026.
Video and audio editing: Dominic Armstrong, Milo McGuire, Luke Monsour, and Simon Monsour
Music: CORBIT
Camera operator: Dominic Armstrong
Coordination, transcripts, and web: Katy Moore
Many AI policy careers won’t matter. Here’s how to find one that will.
If you want to shape how AI develops, working on US AI policy could be highly effective. But the US policy ecosystem is huge, and some roles are much more promising than others.
In November, we published an overview of where to work to have an impact in US AI policy, with guidance for judging which policy institutions will matter most and our views on the most impactful places to work right now — from federal agencies to think tanks. A few key pieces of advice:
Be ready for policy windows. A lot of policy change happens during narrow ‘windows’ of time when an issue enters the spotlight. It’s hard to predict when a window will open, but you can prepare by:
- Building expertise and connections that will help you act quickly — can you be the person to consult when policy around a topic begins to move? This is particularly helpful in technical domains, since there will often be few experts in the room.
Staying flexible so you can pivot in the right direction, rather than working toward a single narrow career goal.
Avoiding paths that require time-intensive prerequisites (like PhDs), especially if you think key policy decisions will be made in the next few years.
Focus on career capital rather than specialisation. Most entry-level policy roles won’t focus on AI, but they’ll still give you critical experience in navigating the space and make it easier to reach higher-impact positions.
The most overlooked roles in AI safety
We recently spoke to Ryan Kidd, Co-Executive Director at MATS, a programme that trains researchers in AI safety. He shared with us some important trends in the technical AI safety ecosystem — and his thoughts line up with what we’re hearing from other leaders in this space. If you’re wondering which roles are hard to fill and where you might be able to contribute, read on!
- Research management seems to be a substantial bottleneck: These roles can be hard to fill, as they require some familiarity with AI safety research as well as strong interpersonal skills and management experience. Plus, impact-driven people who are interested in AI safety generally want to be researchers themselves — rather than manage the research of others! Crucially, you often don’t need to be a great researcher yourself to be a great research manager: people with experience as project managers, people managers, and executive coaches can all make for excellent research managers.
We lack executive talent: The technical AI safety field could really benefit from more people with backgrounds in strategy, management, and operations. If you have experience managing and growing a team of 30+ people, you could make a big difference at a top-tier AI safety organisation, even if you don’t have a lot of direct experience with AI.
We lack founders, field-builders, and communicators: There’s a lot of room to start new organisations and grow the ecosystem,
#234 – David Duvenaud on why ‘aligned AI’ could still kill democracy
Democracy might be a brief historical blip. That’s the unsettling thesis of a recent paper, which argues AI that can do all the work a human can do inevitably leads to the “gradual disempowerment” of humanity.
For most of history, ordinary people had almost no control over their governments. Liberal democracy emerged only recently, and probably not coincidentally around the Industrial Revolution.
Today’s guest, David Duvenaud, used to lead the ‘alignment evals’ team at Anthropic, is a professor of computer science at the University of Toronto, and recently coauthored the paper “Gradual disempowerment.”
He argues democracy wasn’t the result of moral enlightenment — it was competitive pressure. Nations that educated their citizens and gave them political power built better armies and more productive economies. But what happens when AI can do all the producing — and all the fighting?
“The reason that states have been treating us so well in the West, at least for the last 200 or 300 years, is because they’ve needed us,” David explains. “Life can only get so bad when you’re needed. That’s the key thing that’s going to change.”
In David’s telling, once AI can do everything humans can do but cheaper, citizens become a national liability rather than an asset. With no way to make an economic contribution, their only lever becomes activism — demanding a larger share of redistribution from AI production. Faced with millions of unemployed citizens turned full-time activists, democratic governments trying to retain some “legacy” human rights may find they’re at a disadvantage compared to governments that strategically restrict civil liberties.
But democracy is just one front. The paper argues humans will lose control through economic obsolescence, political marginalisation, and the effects on culture that’s increasingly shaped by machine-to-machine communication — even if every AI does exactly what it’s told.
This episode was recorded on August 21, 2025.
Video and audio editing: Dominic Armstrong, Milo McGuire, Luke Monsour, and Simon Monsour
Music: CORBIT
Camera operator: Jake Morris
Coordination, transcriptions, and web: Katy Moore
#233 – James Smith on why he quit everything to work on a biothreat nobody had heard of
When James Smith first heard about mirror bacteria, he was sceptical. But within two weeks, he’d dropped everything to work on it full time, considering it the worst biothreat that he’d seen described. What convinced him?
Mirror bacteria would be constructed entirely from molecules that are the mirror images of their naturally occurring counterparts. This seemingly trivial difference creates a fundamental break in the tree of life. For billions of years, the mechanisms underlying immune systems and keeping natural populations of microorganisms in check have evolved to recognise threats by their molecular shape — like a hand fitting into a matching glove.
Mirror bacteria would upend that assumption, creating two enormous problems:
- Many critical immune pathways would likely fail to activate, creating risks of fatal infection across many species.
- Mirror bacteria could have substantial resistance to natural predators: for example, they would be essentially immune to the viruses that currently keep bacteria populations in check. That could help them spread and become irreversibly entrenched across diverse ecosystems.
Unlike ordinary pathogens, which are typically species-specific, mirror bacteria’s reversed molecular structure means they could potentially infect humans, livestock, wildlife, and plants simultaneously. The same fundamental problem — reversed molecular structure breaking immune recognition — could affect most immune systems across the tree of life. People, animals, and plants could be infected from any contaminated soil, dust, or species.
The discovery of these risks came as a surprise. The December 2024 Science paper that brought international attention to mirror life was coauthored by 38 leading scientists, including two Nobel Prize winners and several who had previously wanted to create mirror organisms.
James is now the director of the Mirror Biology Dialogues Fund, which supports conversations among scientists and other experts about how these risks might be addressed. Scientists tracking the field think that mirror bacteria might be feasible in 10–30 years, or possibly sooner. But scientists have already created substantial components of the cellular machinery needed for mirror life. We can regulate precursor technologies to mirror life before they become technically feasible — but only if we act before the research crosses critical thresholds. Once certain capabilities exist, we can’t undo that knowledge.
Addressing these risks could actually be very tractable: unlike other technologies where massive potential benefits accompany catastrophic risks, mirror life appears to offer minimal advantages beyond academic interest.
Nonetheless, James notes that fewer than 10 people currently work full-time on mirror life risks and governance. This is an extraordinary opportunity for researchers in biosecurity, synthetic biology, immunology, policy, and many other fields to help solve an entirely preventable catastrophe — James even believes the issue is on par with AI safety as a priority for some people, depending on their skill set.
The Mirror Biology Dialogues Fund is hiring, including for a deputy director and a role in operations. You can also express your interest for future roles and keep an eye on the MBDF jobs page for future openings.
This episode was recorded on November 5-6, 2025.
Video and audio editing: Dominic Armstrong, Milo McGuire, Luke Monsour, and Simon Monsour
Music: CORBIT
Camera operators: Jeremy Chevillotte and Alex Miles
Coordination, transcripts, and web: Katy Moore
A note from Rob Wiblin about infohazards
Some listeners have raised the concern that the information in this episode is an ‘infohazard’ — that is, information that’s dangerous to publicise and which we should avoid covering on the show.
It was no doubt a complex decision for the small group who originally identified the mirror bacteria threat to decide whether to go public with it. I know they debated it at length and shared it with other scientists only very gradually.
In the end, they decided it was a clear call to sound the alarm for reasons they briefly summarise in their technical report under ‘Rationale for Public Release’.
In short, scientists almost certainly would have noticed the same issues at a later time, and perhaps impulsively shared them with the world. But by that point, with biology progressing normally in the meantime, there would have been less of a technical buffer between current science and the actual development of mirror bacteria. The sooner we can implement a moratorium on new research that makes it easier to create them, the wider we can keep that buffer, and the safer we will be.
For us here at 80,000 Hours, the decision is more straightforward. The risk of mirror bacteria was published to widespread media coverage in December 2024, in The New York Times, Financial Times, CNN, The Guardian, and Le Monde, among many others. There are YouTube videos describing mirror bacteria with millions of views.
So it’s very far from a secret now.
The existence of this interview does little to change how salient mirror bacteria are on a global scale.
But thanks to our audience, we can make a big difference to whether MBDF and others can hire the dedicated staff they need to successfully close the technical path to actually developing mirror bacteria. That makes the interview a clear win in expectation, which is the view of experts in mirror bacteria who work on this problem full time.
2025 Highlight-o-thon: Oops! All Bests
It’s that magical time of year once again — highlightapalooza! Stick around for one top bit from each episode:
- Helen Toner on whether we’re racing China to build AGI (from episode #227)
- Hugh White on what he’d say to Americans (#218)
- Buck Shlegeris on convincing AI models they’ve already escaped (#214)
- Paul Scharre on a personal experience in Afghanistan that influenced his views on autonomous weapons (#231)
- Ian Dunt on how unelected septuagenarians are the heroes of UK governance (#216)
- Beth Barnes on AI companies being locally reasonable, but globally reckless (#217)
- Tyler Whitmer on one thing the California and Delaware attorneys general forced on the OpenAI for-profit as part of their restructure (November update)
- Toby Ord on whether rich people will get access to AGI first (#219)
- Andrew Snyder-Beattie on how the worst biorisks are defence dominant (#224)
- Eileen Yam on the most eye-watering gaps in opinions about AI between experts and the US public (#228)
- Will MacAskill on what a century of history crammed into a decade might feel like (#213)
- Kyle Fish on what happens when two instances of Claude are left to interact with each other (#221)
- Sam Bowman on where the Not In My Back Yard movement actually has a point (#211)
- Neel Nanda on how mechanistic interpretability is trying to be the biology of AI (#222)
- Tom Davidson on the potential to install secret AI loyalties at a very early stage (#215)
- Luisa and Rob discussing how medicine doesn’t take the health burden of pregnancy seriously enough (November team chat)
- Marius Hobbhahn on why scheming is a very natural path for AI models — and people (#229)
- Holden Karnofsky on lessons for AI regulation drawn from successful farm animal welfare advocacy (#226)
- Allan Dafoe on how AGI is an inescapable idea but one we have to define well (#212)
- Ryan Greenblatt on the most likely ways for AI to take over (#220)
- Updates Daniel Kokotajlo has made to his forecasts since writing and publishing the AI 2027 scenario (#225)
- Dean Ball on why regulation invites path dependency, and that’s a major problem (#230)
It’s been another year of living through history, whether we asked for it or not. Luisa and Rob will be back in 2026 to help you make sense of whatever comes next — as Earth continues its indifferent journey through the cosmos, now accompanied by AI systems that can summarise our meetings and generate adequate birthday messages for colleagues we barely know.
Video and audio editing: Dominic Armstrong, Milo McGuire, Luke Monsour, and Simon Monsour
Music: CORBIT
Coordination, transcripts, and web: Katy Moore
#232 – Andreas Mogensen on what we owe ‘philosophical Vulcans’ and unconscious AIs
Most debates about the moral status of AI systems circle the same question: is there something that it feels like to be them? But what if that’s the wrong question to ask?
Andreas Mogensen — a senior researcher in moral philosophy at the University of Oxford — argues that so-called ‘phenomenal consciousness’ might be neither necessary nor sufficient for a being to deserve moral consideration.
For instance, a creature on the sea floor that experiences nothing but faint brightness from the sun might have no moral claim on us, despite being conscious.
Meanwhile, any being with real desires that can be fulfilled or not fulfilled can arguably be benefited or harmed. Such beings arguably have a capacity for welfare, which means they might matter morally.
And, Andreas argues, desire may not require subjective experience. Desire may need to be backed by positive or negative emotions — but as Andreas explains, there are some reasons to think a being could also have emotions without being conscious.
There’s another underexplored route to moral patienthood: autonomy. If a being can rationally reflect on its goals and direct its own existence, we might have a moral duty to avoid interfering with its choices — even if it has no capacity for welfare.
However, Andreas suspects genuine autonomy might require consciousness after all. To be a rational agent, your beliefs probably need to be justified by something, and conscious experience might be what does the justifying. But even this isn’t clear.
The upshot? There’s a chance we could just be really mistaken about what it would take for an AI to matter morally. And with AI systems potentially proliferating at massive scale, getting this wrong could be among the largest moral errors in history.
In today’s interview, Andreas and host Zershaaneh Qureshi confront all these confusing ideas, challenging their intuitions about consciousness, welfare, and morality along the way. They also grapple with a few seemingly attractive arguments which share a very unsettling conclusion: that human extinction (or even the extinction of all sentient life) could actually be a morally desirable thing.
This episode was recorded on December 3, 2025.
Video and audio editing: Dominic Armstrong, Milo McGuire, Luke Monsour, and Simon Monsour
Coordination, transcripts, and web: Katy Moore