#248 – Jasmine Sun on what the people building AI really believe

Transcript

Cold open [00:00:00]

Jasmine Sun: Most people in the world do not have the opportunity to get themselves out of the permanent underclass by getting a job at OpenAI.

Sure, you could call it like a normal amount of “people like money,” but I think the scale of how much talent is flowing into a few specific labs really has to do with the fact that the AI community itself is promoting these widespread beliefs that if you do not have a “stake” — if you do not have literal equity in frontier AI — you are not going to have economic leverage in the future, and you and your kids are going to be screwed.

Who’s Jasmine Sun? [00:00:30]

Zershaaneh Qureshi: Today I’m speaking with Jasmine Sun. Jasmine’s a writer who’s focused on the AI world in Silicon Valley, and she lives in San Francisco, spends lots of time with the AI community that’s based there. She describes her work as an anthropology of disruption.

Basically she’s documenting the cultures and the beliefs of the people who are building frontier AI. That means basically she’s going to all the parties, seeking out people from all the different factions in this world, interviewing them, learning what makes them tick.

Now, if that sounds interesting to you, Jasmine’s got a Substack that I highly recommend checking out. She’s also a guest contributor for The Atlantic. Jasmine, thank you so much for joining us.

Jasmine Sun: Thanks for having me.

Escaping the permanent underclass [00:01:22]

Zershaaneh Qureshi: You wrote this opinion piece recently for The New York Times, which opened with this really striking observation that most people you know in AI think that the median person is screwed.

Now, I’m assuming what you mean by that is — on the more moderate end of things — people are anticipating serious short-term disruption, things like job displacement and so on. But for some people, in some cases, people are talking about more extreme things like existential-scale threats that affect all of humanity, and the idea of a permanent underclass.

The people that you’re talking to in these circles, they talk about these things, but they don’t really have a solution. Yet they’re still going ahead and building the technology. Can you help us get into their heads a bit here? Do they feel guilty or worried about the costs that they might impose on society if they mess things up?

Jasmine Sun: I’ve been super interested in these questions.

Part of the reason I wrote that New York Times piece is because I was doing this little personal project, where I was doing almost these long-form AI ethnographies where I’d sit down with a random researcher at one of the frontier labs — sometimes a friend, sometimes an acquaintance, or a stranger.

And I just asked them for an hour about their very broad views about AI, its impact on the world, and how optimistic and pessimistic they were. All off the record.

One thing that I noticed is: one of the questions I ask is, “What advice would you give to a normal 17-year-old? Not like a super cracked, brilliant coder — but just pick a random state in America, normal 17-year-old, they’re getting Bs in school, they’re not thinking about AI that much. How do you think
that they can prepare for the future?”

What was really shocking to me was almost everyone I spoke to, regardless of their personal political beliefs — some of these are more libertarian types, some of these are more leftist types, some of these don’t care about politics at all, some of these have a wide range of beliefs about even AI itself — most people just told me: “Honestly, I have no idea what I’d tell that 17-year-old. It’s a really scary time. I don’t think there’s going to be a lot of jobs for them left. I think they’re caught in this painful transition.”

On the economic front, it did become clear that there was this quiet consensus that, regardless of your future beliefs about whether there will be a permanent underclass or more jobs than ever, the vast majority of people I know in the AI industry believe that in the near term, in the next few years, we are going to see mass job displacement. It’s going to be especially hard for a lot of young people, and a lot of current knowledge-work jobs are going to be disrupted and displaced completely.

Then of course it goes to the other thing you said, which is for me personally, I just feel that if I thought that the technology I was building had a very high chance of displacing millions of jobs and maybe also a 10% chance of killing everybody or something like that — which is also not an uncommon belief from researchers at the labs — I probably wouldn’t build it.

So then I go and ask: “Why are you doing this? Does that feel dissonant at all?” And you hear a range of answers to that.

I think that, for some people, I would say the probably more common belief among researchers at the labs is, “Yes there are harms, yes there are risks. But if we really get this AI thing right, we are going to enter a world that is better than ever. We are going to get the end to all disease because AGI [artificial general intelligence] is going to help us find the cures to diseases, and humans will be immortal. We’re going to have all consumer goods and services be cheaper than ever because the space factories, or whatever it is, mean that we’re not going to have to pay for a house or a car. Maybe it’ll be better when the AIs can do all the jobs and we’re able to live off UBI [universal basic income], and we can engage in these lives of infinite leisure and focus on our relationships and our communities instead of having to engage in demeaning wage labour” or something like that.

This vision, I think, is genuinely inspiring to a lot of people who work in AI. They do believe that technological progress, for all of its bumps and disruptions, ultimately leads us to a world where we live longer and happier lives. So I think one is that a lot of folks are willing to take the risk or weather this near-term disruption in order to get that future.

The other thing that I noticed though, which is in some ways a little bit more disturbing to me or challenging for me, is a pretty broad sense of techno-determinism. I think a lot of people in the AI industry, and in tech more broadly, do think that the path of technology is pre-set — that somebody is going to build AGI, somebody is going to build superintelligence.

Especially at this point with how fierce the race has become between the frontier labs, there is a sense of, “If I don’t do it, they’re going to do it.” Maybe I’m at Anthropic and I actually am really worried that the thing I’m building might have all of these negative consequences. But if Anthropic exits the race, OpenAI, xAI, Google, they’re going to stay in the race and they’re going to keep competing.

Then you might think, “At least if I was involved, I could help make it go better. I could build a better product, a safer product — technology that cares more about human welfare. And in addition to that, I might as well get my bag and make sure that I have a stake in the future.”

I do think a lot of people, if they believe that this huge disruption is coming, do care somewhat also about securing their own spot, through financial upside in the future. That one’s a little bit more worrying to me because sure, maybe I get it and I’m happy for them, but most people in the world do not have the opportunity to get themselves out of the permanent underclass by getting a job at OpenAI. So that’s what I worry about.

Zershaaneh Qureshi: You’ve also described coming across — and this is probably from a smaller subset of people — some kind of nihilism about AI: this idea that it actually doesn’t matter if technology serves humans at all, and I guess a related idea that actually maybe we should be handing over the reins of the future to AI and away from humans. In that sense, it’s not just that it doesn’t matter if technology doesn’t serve humans, it might actually be a good thing if it’s not serving humanity, but serving AI as our successor.

Can you tell us a bit more about those views, and how you see them influencing the AI scene these days?

Jasmine Sun: For sure, yeah. I’ll share your caveat that I would say that these, what I call ‘successionist-lite’ views, are more niche and definitely less common than the previous set that I described.

But I have been worried in hearing, in just casual conversation, fairly prominent and influential people within the AI community express an indifference or even a preference to a world where AIs are making most of the decisions in society rather than humans.

I won’t disclose people’s identities because they were Chatham House or off-the-record contacts. But I’d be having a conversation with somebody at a conference workshop and they would say, “Everyone at this conference is so worried about what’s going to happen to humans in the post-AGI world. But frankly, I’m more worried about what if the monkeys are still running the show?” And in this case the monkeys refers to the humans. He was more concerned that humans would still be in charge of governments, of corporations, of society because he had this view that AIs in their superintelligent existence were more deserving and more important to preserve a future world for.

Or I was talking to another very influential person in the AI scene and we were just on a walk and musing about, if all labour gets automated, what are people going to do? And he was like, “Maybe you’ll be like Nozick’s experience machine, and we’re all going to be wearing these VR headsets and watching perfectly personalised AI slop. So we don’t have any leverage in society, but in a way it’s very pleasurable and people are very happy.”

And I was like, “I don’t know. Do you think that’s actually the world we want?” We’re going back and forth on it a bit. And he’s like, “Honestly, Jasmine, I’m a lot more interested in what the AIs are going to do with each other 1,000 years from now than I am in what’s going to happen to humans next year.”

So, in a way, I think that there are people — influential and prominent people in the AI community — who, whether for intellectual reasons (as in a belief that the more intelligent species is more deserving, or simply because it’s a more interesting thought experiment) have arrived at an indifference towards whether humans stay in control. I find that quite worrying.

I don’t think it’s the majority belief or common belief within the AI labs or something like that, but I do think that it makes it very hard to move forward policy action, to move forward the safety conversation — if we’re not able to agree on even very foundational concepts such as: humans should stay in control of our governments, and it is possible to decide what technologies we build and don’t build, and our conversations should centre what happens to people. So I find that a little bit troubling.

Zershaaneh Qureshi: I think maybe I’m just a jaded person, but when I hear you talking about people expressing these nihilistic beliefs or these deterministic beliefs, in my head, I’m just like, “Surely these people don’t genuinely believe what they’re saying. It’s just a bunch of tech bros who are doing some kind of mixture of posturing, edgy humour, irony, that kind of thing.”

I wonder if through this conversation we are paying too much attention to what’s essentially just an aesthetic choice that they’re making, rather than a real set of beliefs. What do you make of that?

Jasmine Sun: I think it’s fair. I think that when I report, I try to have an eye for some of the hyperbole. In the permanent underclass piece, I try to make it pretty clear that in a lot of ways it is an edgy joke. Most people do not literally believe in a permanent underclass, even if they believe in mass job displacement and increasing inequality, which I also think is somewhat concerning and worth talking about, even if it’s expressed in more extreme ways.

But in general, I like paying attention to these cultural quirks and these memes because I think they are a release valve for things that you can’t say, or things that feel almost sometimes taboo or inappropriate.

I would not have written that permanent underclass piece if I have not had lots and lots of conversations with young workers in Silicon Valley who have decided to try to get a job at an AI lab rather than doing a variety of other things, specifically because they were specifically worried about being trapped in a permanent underclass.

Maybe you could say they were actually just worried about being poor or having less money when AGI comes and returns to labour stop working. But it’s true that these memes, silly as they may sound, are shaping people’s individual behaviour. I’ve been thinking a lot specifically about this question of where people choose to work, and brain drain in particular, because, for example, one of my close friends is a PhD student who studied in computer science in AI at Berkeley, and she was telling me how like half of her cohort has decided to graduate early, within just a few years of beginning their PhDs, in order to get jobs at AI labs. And they’ve actually curated their PhD dissertations for research that would be most likely to get them a job at an AI lab.

I’ve seen in my own community tonnes of people — who were independent writers or independent policy thinkers and policy advocates — decide to leave those roles behind in order to get jobs at AI labs, and these views. Sure, you could call it like a normal amount of ‘people like money,’ but I think the scale of how much talent is flowing into a few specific labs really has to do with the fact that the AI community itself is promoting these widespread beliefs that if you do not have a “stake” — if you do not have literal equity in frontier AI — you are not going to have economic leverage in the future and your kids are going to be screwed.

I’ve just heard in enough private conversations that these are motivating people to make such decisions, and it’s clearly draining talent from the non-corporate independent ecosystem, that I do start to take those memes somewhat seriously.

Jasmine’s “anthropology of disruption” [00:14:02]

Zershaaneh Qureshi: If you’re trying to understand what the future of AI is going to look like and how AI is going to shape the future, the most natural thing — at least from my perspective — to do would be to focus directly on thinking about the technology itself, or maybe on surrounding policy issues.

But you do something different. You take this “anthropological” approach, where you’re focusing on the cultures and beliefs of the people who are building these technologies. What does that lens capture that other lenses might miss?

Jasmine Sun: I certainly think it’s important that other people focus on just the policy or just the technology itself. But one thing that I think a lot of the community, or even outside observers, underestimate is that there are really very few people who are building and shaping the trajectory of frontier AI.

There are not that many researchers at these labs. The reason they’re getting paid these crazy $100-million packages to get poached from here to there and back again is because there are only so many people who have this much leverage.

So what that means is that personal-idiosyncratic belief systems can be quite load bearing for really big decisions around what technology is built, who it gets sold to, how it gets diffused.

In many ways I think the existence of Anthropic is a bet on this thesis. It’s the fact that — if there’s only one lab that is building the best enterprise agents or coding models or whatever it is — then they can actually make principled decisions based on, say, Dario Amodei’s personal beliefs that we don’t want Mythos to be used for mass surveillance or for autonomous robots right now. We actually don’t want to sell to Chinese companies and the Chinese government, even though market incentives would dictate that we ought to, because we have these principled beliefs around whether authoritarian governments ought to have access to frontier AI.

So I think that, in a world where there are just not that many actors and not that many players, understanding the particular philosophies and motivations of the few people who can decide actually is pretty predictive of some of these decisions that are made.

Zershaaneh Qureshi: For people who are still skeptical at this point, it would be helpful I think, if you could give some historical examples or indeed recent examples where the values or culture or idiosyncratic beliefs of people building a technology have actually tangibly impacted how that technology has, as you say, how it’s been built, how it’s been deployed, how it’s affected the world.

Jasmine Sun: I think one example is just looking at the history of the early internet, which was not actually built — AI was primarily in these private megacorporations — but by a guy at DARPA and a guy at Stanford working on these TCP/IP protocols, building out ARPANET, building these versions of the internet where it was decided from the beginning that it should run on open protocols, that it should be this open ecosystem — that different networks between countries, between institutions, ought to be able to communicate with each other.

If you read the interviews or the writing of these folks like Vint Cerf, they were very clear that they believed that the internet ought to be open. The whole point was this sort of interconnectedness that would transcend borders. And to this day, we are using the same internet based on these open architectures, right?

Similarly, I’ve been reading Sebastian Mallaby’s biography of Demis and Google DeepMind, and one of the things that I find so fascinating is how — in the early days of DeepMind, which is very arguably the first serious AGI lab — they could not get any funding at the beginning because no venture capitalist was willing to back a bunch of these weirdo scientists who insisted that you could build superintelligence in 2010 or whatever, who had no business plan, refused to commercialise, had no product, et cetera.

They were running out of money until Luke Nosek from Founders Fund — simply out of his personal belief that AGI ought to be possible, that the invention to end all inventions would just be in some ways the coolest thing ever — thought that it was worth a multimillion dollar bet, even while Peter Thiel and many other venture capitalists were like, “You’re freaking crazy for even considering funding these guys without a business plan.”

I think when we look at the history of these frontier technologies, when there are only a few people who have, frankly, the technical abilities to make them possible, those people’s idiosyncratic beliefs in this thing or another — believing that artificial general intelligence, for example, might be possible at all, believing that the internet ought to be an open thing that transcends borders — they set these precedents, and they make these fundamental architectural decisions or funding decisions that lay the groundwork for everything else that comes after.

Vice signalling in Silicon Valley [00:18:46]

Zershaaneh Qureshi: Something I was really interested to read on your Substack is how the word ‘doomer’ has become the lowest status label that you can wear in Silicon Valley. That really surprised me just because AI company leaders are very much acknowledging the plausibility of doom scenarios. In fact, that sense of ‘doominess’ is so pervasive that people on the outside think it’s a form of marketing. They’re like, “They’re not serious, it’s just hype.”

I’m wondering how you think this idea of ‘doomer’ an insult meshes into that picture?

Jasmine Sun: In this case I’m talking a lot more, I think, about classic Silicon Valley and the VC startup corporate ecosystem around tech, which is still the majority of the tech industry. It’s somewhat true, I would say. It’s less true at the AI labs just because there are a lot of people who are, as you mentioned, ‘doomers’ who are working at frontier labs.

But I think when you ask what is Silicon Valley’s political philosophy or what is its ideology in general, you can identify a bunch of little tribes: you can identify the effective altruists here, and the progress studies people over there, and the e/accs [effective accelerationists] over here.

But one of the things that a lot of them share is a very firm belief that technological progress is necessarily a good thing. So I think that when some of the more accelerationist types have turned ‘doomer’ into this slur almost against people who want to slow down or to regulate AI or other technologies, they’re presenting a group of people as being anti-progress — or as being in opposition to this core belief in Silicon Valley that technological advancement is the only thing that has ever brought us growth, prosperity, equality, et cetera.

If in all of history — in spite of again, the Industrial Revolution, automation, every wave of technology has brought on its own wave of disruptions — but if one were to believe so firmly in these apocalypse scenarios that you would be willing to slow down the only thing that might again bring us this future utopia that would keep the progress engine going, then you must be this kind of pessimistic, low agency, tech-sceptical kind of person, I suppose.

So again, I would say that this belief is more of the VC startup crowd. But I think the idea that we as humans are first of all not powerful enough to be able to control our own technologies, the idea that we should stop progress just because there might be some marginal side chance of harm, I think people are seeing AI in the context of all the previous technological revolutions and thinking: “Yeah, these doomers would still be on the farms being peasants. They would never have allowed the Industrial Revolution to happen.”

Zershaaneh Qureshi: I think what I want to understand more is what the practical consequences actually are of having safety concerns being associated — at least in these kind of VC startup-y circles, which is a big part of culture in the area — being seen by these people as embarrassing or low status. How does that actually affect what’s going on in the labs?

Jasmine Sun: I would say again, to me, a huge part of this is the talent-flows question. Just because talent is so scarce, as we see from the salaries that people are willing to pay, talent is incredibly scarce. So when you have these cultural beliefs, the main thing that they influence is actually what young people decide to do with their lives — and how many people decide to, say, work on safety vs working on capabilities.

One thing that I’ve noticed rise over the past few years — even outside of AI, but in Silicon Valley broadly — is the rise of vice signaling: this idea that actually the edgy and cool thing to do is to build a startup like Mechanize, where the tagline is that, ‘We’re going to automate and replace all human labour,’ or to build something like Cluely, where the tagline is, ‘Cheat on everything.’

I think that the idea that trying to be careful — trying to be constrained about the way that you build technology — by labeling that as ‘doomery’ or low status, or as not being a pioneer and a serious builder, you’re basically preventing a bunch of very impressionable young people from going into a field that they otherwise would.

That’s to say nothing of obviously the policy conversation in DC, where ‘doomer’ has been employed as an extremely effective insult. I think one important thing actually also about the ‘doomer’ insult is that it’s coming mostly from the tech right that has pretty effectively tied ‘doomerism’ and AI safety to ‘woke’ identity politics and previous attempts to regulate technology by alleging discrimination, civil rights concerns, labour concerns, et cetera.

The folks like Marc Andreessen and David Sacks who are using the ‘doomer’ label most loosely, they are building this perception — on the American right in particular — that to be interested in AI safety is the same as being a ‘woke regulator’ who wants Facebook to censor all non-liberal content.

Zershaaneh Qureshi: I think what I’m interested in here is that at 80,000 Hours we see a lot of people who are very attracted to careers in the Bay Area at AI companies and the general AI ecosystem, specifically because they really genuinely want to positively influence the trajectory of AI development.

I wonder, in this environment, how optimistic you are that safety-focused people can actually succeed at pushing forward what they want to push forward?

Jasmine Sun: I think that folks who are working on technical safety work, there are still a lot of opportunities and a lot of support — both socially, financially, et cetera — professional ecosystems towards doing that kind of work.

I think that there’s a well-developed enough ecosystem of AI safety organisations. There’s the fact that actually most people who work on capabilities at the labs really care that their models are aligned. It’s also very hard to sell unaligned models. It coheres with the business incentives of these companies so that you can get made fun of a little bit by Marc Andreessen in order to do work that you think is really important, and really well respected by a lot of other technical people in your community.

One thing that I sometimes tell folks who are not in AI is what’s really notable is actually a lot of the researchers who have made the biggest breakthroughs in AI, people like Geoff Hinton, are some of the ones who are raising the alarm the most on AI safety being an important thing to work on. I think that affords working on technical safety quite a lot of status, at least within those technical research communities.

I think where I start to worry a little bit more is on types of work outside of that core safety work — which is the technical safety work, which is more well funded. Maybe it’s just because I’m a journalist, but I think a lot about how much incentive there is for people to go into these fields where one, I don’t think that you can do them inside a lab.

I don’t think that you can do especially credible either journalistic work or policy advocacy work if you are employed by a lab: you are going to be seen as a shill. You are correctly seen as a shill. You are basically doing marketing and lobbying for that company.

So if you completely 100% align with that company’s overall position, then OK, great, go do that work because you’ll help your firm win. But for the most part, I really wonder and worry about how many positions there are for very AGI-pilled and very safety concerned people to work in these sort of independent roles.

Even the journalism community has appropriated these words to make fun of people who believe in things like existential risk. Like in the journalism and media world, people will make fun of you if you are seen as an effective altruist, if you are seen as someone who believes in existential risk, if you are seen as buying into the ‘doomer hype’ like you mentioned of these AI companies.

So it’s actually kind of low status in journalism to believe in these things. It’s low status in the policy community. I mean, Politico, I think last year or the year before was running these stories, attempting to weed out and witch hunt anyone who has any connection to effective altruist fellowships within the AI policy community so that people would then go and discredit them.

I’ve had, again, senior media people tell me very similar things where they’re going after all the Tarbell fellows and saying these people are just Anthropic shills, or these people are shills for some crazy Silicon Valley billionaires.

So especially because these are the roles that I already think are less supported — they are not as well remunerated as roles at the labs, you don’t have this social and professional ecosystem — I tend to worry more so about people who are in academia, in research and policy, and in journalism who take AI really seriously and whether they are supported to do their work.

AI populism will shape 2028 [00:28:11]

Zershaaneh Qureshi: Let’s move on now to talk about another group of people that you’ve been observing.

You’ve spent a bunch of time with, I guess, a group of people that you would describe as AI populists. Can you tell us more about who these people are, what their views and motivations are and so on?

Jasmine Sun: I’ve been tracking the general anti-AI backlash and the political backlash in the United States. I’ve been curious about that. So I made two trips to Washington, DC in February and March of this year. One of my core goals during these trips was to talk to some of these advocates and interest groups who were basically anti-AI advocates in various forms.

I specifically really wanted to talk to folks who are not part of either the industry or the sort of classic AI safety community, but people who are actually coming from other factions within American politics who have only recently begun engaging on AI.

This includes maybe labour groups who are labour-union advocates, who are worried about the job stuff. This includes maybe social conservatives who are very concerned about AI replacing human relationships with AI companions and AI relationships. It might include environmentalists who are worried about data centres energy-use impacts, or states-rights advocates who are worried about things like federal preemption.

One of the big trends that I’ve noticed in the US over the last few months is that AI is no longer the remit of these few technocratic, very AI-focused policy wonks who are very concerned with the specific characteristics of AI as a technology. Rather you have all these other interest groups that have long histories in Washington lobbying for this and that, who are now realising, “Wait a second, AI intersects with my issue” — usually in a way that makes us not like AI very much.

Now they’re actually banding together and forming these, in some cases, bipartisan or cross-issue coalitions to support things, like whether that’s a data centre moratorium or fighting preemption or things like that.

That was just a really interesting trend to see people — who again probably two years ago had no clue what an LLM was — to now be focusing so much of their time and political energy on organising against AI. Some people have asked why I call these folks the “AI populists”? Where does that term come from?

I think it comes from me trying to understand the difference between these political factions that are anti AI vs say the AI safety types of researchers who we talked about earlier. The main thing that I realised was that these new AI populists are not primarily concerned with the technical characteristics of AI that might make it uniquely risky, uniquely dangerous.

Rather, they see AI as an extension of a broader concern that they have: that a small number of wealthy and powerful corporate elites are concentrating control in society at the cost of everybody else.

So when the AI populists are trying to do things like impose data centre moratoriums or regulate AI, it’s often less again about the specific nature of the technology and where it’s going, and much more about a broad backlash, a preexisting populist backlash against billionaires, against tech companies, against corporate concentrations of power that hurt workers and hurt ordinary Americans.

Zershaaneh Qureshi: So these people, their hatred of AI is very deeply entrenched in their distrust of the people who are building it — to the extent where it doesn’t seem to matter to them how useful ChatGPT gets as a tool. There’s something fundamental about what’s going on here that they feel inherently skeptical of.

I think that’s interesting because, based on what we’ve heard so far about the attitudes of people who are building the technologies, we’ve seen they’re quite varied. But ultimately we’ve got a group of people who are aware of the risks, aware that the things that they’re building could cause a lot of at least short-term disruption — if not worse, longer-term harms — and they’re building it anyway.

They maybe have an ‘automate or be automated’ mindset and some group of them, maybe a small group even, has a belief like technology doesn’t even need to serve humans.

With all of this stuff in mind, I guess it’s not wholly surprising to me that some large part of the public feels pretty distrustful of this group of people. I’m wondering how much sympathy you have for these views overall?

Jasmine Sun: Frankly, quite a lot. I think mainly for the reasons that you just described, I think that people are correctly assessing that there are some folks in Silicon Valley who are taking a gamble on their futures.

Like you said, maybe the gamble is their jobs, maybe the gamble is their literal survival. Maybe the gamble is if they believe that AI is a bubble, it’s like a giant economic bubble. It’s true that AI is propping up American GDP growth, and it has been for the last year or so. I think St. Louis Fed said that 39% of growth in 2025 came from data centres and AI.

So when people understand that they have no capacity for democratic control or having a voice over the direction of the technology, when the leaders of those technologies are being very open about the risk that they think it’s going to pose and saying, “Hey, we’re going to do it anyway.”

For normal people, they actually have no leverage, right? They don’t know anyone who works in an AI lab. They can’t just tell them, “Hey, you should worry about this.” Most policy action has been extremely limited. Most people don’t know anyone in AI policy anyway.

I think that when I see things like data centre protests or booing at college graduations or whatever, I don’t necessarily think that they’re the smartest levers to make AI safer or whatever, but I think that they are the only levers that people feel that they have access to.

And like, yeah, maybe Waymo is a little bit safer. Maybe ChatGPT helps me write my emails faster. I think people are willing to cop to those particular things. But when ‘write my emails faster’ is positioned against ‘this might take my job and kill us all,’ it just doesn’t feel like that good of a trade.

And then the AI people will say, but it also might cure cancer. And they’ll say, “Where is the cancer cure?” I hear this all the time. The AI CEOs keep talking about curing all diseases. Where are the cures to the diseases? At a certain point, you’re just not a credible actor anymore.

So I understand why the populist backlash is happening. I would like to see it funneled towards maybe more what I see as productive policy aims than protesting data centres and booing people at graduations and whatever. But it kind of makes sense to me.

Zershaaneh Qureshi: Yeah. Something else that we’re seeing, we’re already beginning to see these kinds of AI populist type attitudes beginning to escalate.

Recently there have been some attacks on Sam Altman’s house and also on the house of, I think, a councilman who had just approved a data centre project. Obviously these things are extremely horrific.

I think what I want to know from you is what you think the natural trajectory is for these kinds of attitudes and behaviours. Are you anticipating that AI populism is going to start influencing political campaigns a lot more? Are you sort of envisioning that there’d be some kind of, I don’t know, Luddite-style revolt in which rather than smashing looms, people are attacking data centres? How do you see this playing out?

Jasmine Sun: All of those things seem pretty possible to me. In the US we have the midterms this year, and we have a big presidential campaign in 2028 where we’re expecting both a very crowded Democratic and Republican primary.

One is: I think that there’s going to be a lot of political opportunism, some of it justified and some of which centres AI probably even more than it needs to be. Because I think AI is a perfect political bogeyman also. One interesting thing I notice is — while I do believe in a lot of the AI risks being very serious — you also notice that politicians will use AI as a justification for the thing that they already wanted to do.

For Elizabeth Warren it’s, “I want to tax the companies and we should do it because of AI.” Or for people who wanted to regulate how much kids had access to technology for social conservative reasons, they’re like, “We should do it because of AI.”

I’m already seeing some of the candidates who we expect to be running in 2028, whether it’s Mark Kelly or Ro Khanna on the Democratic side, maybe Josh Hawley on the Republican side. They’re already announcing these big, bold AI action plans because they see AI as a new issue that Americans are extremely scared about, and that creates an opportunity for them to define their campaign.

So because they’re all going to be running these ads and talking about these AI companies — and because AI companies are also, frankly, not very sympathetic to most Americans right now — I actually expect populist backlash and public salience of AI to increase quite a lot as these campaigns kick into motion, because politicians have an incentive to talk about AI more in order to justify their policy proposals.

The other thing that I think about is these more diffuse and volatile forms of backlash, such as political violence, as we’ve seen against these actors, such as even things like these witch hunts. In creative communities, among artists and among writers, there are informal cancellation witch hunts against any writer or artist who is suspected of using AI, who partners with an AI company, et cetera.

One thing that I think about is in the 20th century, we had a big wave of factory automation where a lot of the factories went from primarily humans on factory lines to humans managing machines, more so. Part of the way that this was negotiated, such that there was not a massive backlash and there weren’t waves of violence by workers who were being automated, was that these factories were mostly unionised.

When they were mechanised, the people coming with the machines would meet with the union leaders, sit down at the bargaining table and say, “OK, first of all, we’re going to pitch this to you as it actually makes your job safer. You guys are really worried about workplace accidents, and this is going to reduce that. It gets at this issue that you guys are really worried about, we’re going to actually tie wage increases for workers to the increased productivity of these factories.”

Also you had just had this new social contract that FDR introduced to introduce better working legislation in general.

But more importantly, I think that there was a bargaining table and there were these mediating institutions of the unions between the people who were impacted by automation and the people introducing the automation. One really interesting thing I think about today is that there are no mediating institutions of that form. If you are a white collar worker in most industries, you are not part of any sort of organisation that is any sort of union or whatever.

So when your boss decides to maybe introduce AI, consider eliminating your job, there’s not actually anything that you can do about it. There’s not like a particular democratic or collective channel to express how you want AI to be incorporated into your life or your workplace.

I think that means that when people are really pissed off, they start going for social media cancellation and stochastic violence, and all these more diffuse forms of political backlash. I think I just have a view that public sentiment will find an outlet. And right now we’re just going to see quite a lot of it from very different directions, at least until people feel credibly reassured that they will have the downsides cap, that they will share in the growth that AI gives, et cetera.

Does AI populism distract from safety? [00:40:20]

Zershaaneh Qureshi: I guess a reflection that I have — besides AI populism contributing to scary, violent things in the near term — it also seems like potentially longer term, it might be bad overall for AI safety, in a couple of senses:

• One effect that you’ve mentioned is that because AI has kind of become the new ‘because China,’ politicians are using it to push whatever thing they were already trying to push for.
• Another thing is just this misguided focus on things like banning data centres, which I can understand the fact that that’s coming from feeling powerless about what else you can do.
• Then you also have this very deep lack of belief and trust in the AI company leaders who are saying that the technology is going to be a really big deal and potentially pose life-altering transformative effects.

I feel like all of these things seem like bad news for getting good safety policies in place and making sure things go well. What do you make of that?

Jasmine Sun: It seems to me that the safety community is somewhat divided on this question. I’ve talked to folks on both sides. I’m not quite as much of a policy expert here compared to a lot of folks.

But there are clearly some folks from Future of Life, et cetera, who have actually really leaned into the idea that you can build coalitions and build bridges with some of these AI populists because they have existing networks, they have existing relationships with policymakers, they have bases who they can activate around the issue of AI — where a lot of people were probably never going to quite grok a lot of frontier safety issues, but maybe they could be brought on board through some of these other issues, like kids safety or the environment that they otherwise cared about.

There are clearly some folks who think it’s a good idea to forge these alliances. However, I think what you’re saying is very compelling. I do think that by making AI a much more mainstream political conversation — and because AI is such a broad-reaching technology with all these different axes of potential policymaking — it’s a real question as to whether the people who are now the most activated around the AI backlash are focused on the set of issues that the experts would prefer.

Like you said, I think that the fact that the AI water use issue is still as salient in conversations as it is — that it’s something that’s an extremely widespread belief among, say, college students — is pretty crazy because I think it would be a total misallocation of legislative energy for us to focus on legislating data centres’ water use.

It is true that I think a lot of folks are going to be distracted, in an ecosystem where there’s limited political capital and limited attention, by marginal issues — if one believed that some of these frontier safety risks are actually the ones that need to be worked on most.

I will say that it’s possible that the politics of the issue and the broad politics of the issue can coexist with more precise and technocratic policymaking at the legislative level. Especially now that Mythos and the Pentagon fiasco have sort of gotten a lot of the policy and expert community much more worried about the existential risk side of things.

Americans don’t want Silicon Valley’s utopia [00:44:06]

Zershaaneh Qureshi: I think another dynamic that we’re seeing with AI populism that seems worth unpacking a little bit is that AI companies are just spending lots and lots of money on things like PR.

Yet somehow public hostility about AI populist ideas and so forth are still very much rising. What do you think is going wrong here? Is the AI community misunderstanding something fundamentally about its audience?

Jasmine Sun: I had a conversation about this recently at a roundtable with some of the AI executives, and it seems to me that we are past the point where marketing and nice ads and AI-for-good stuff can get the public back on board.

A huge part of it is just that so much of the messaging coming out of the AI industry for the past five years has been focused on the risks. So even if we turn around now and say Sam Altman says, “Actually, I don’t want AI to replace you, I want AI to be a tool.” There are still videos and on-the-record statements from all of these executives about the risks that AI poses. So people see it as flip-flopping.

I think the other issue is just that most people are not seeing tremendous benefits from AI in their personal lives. Yes, again, they may be able to write emails faster, they may get something that they consider 50% better than having access to Google in terms of looking up little things here and there.

But the way that AI shows up in most people’s day-to-day lives is also equivalently slop. Finding it hard to apply for jobs, finding it hard to review job applications, their kid cheating on all of their homework, or maybe having an AI companion that they talk to 24/7. The way that AI shows up in people’s day-to-day lives in the median experience is not necessarily that good.

Again most people aren’t yet getting the tremendous benefits from something like Claude Code for software engineers, where they’re actually seeing in their own productivity, their job get easier, faster, being able to output more.

I think in a world where AI actually has diffused quite a lot — most people encounter AI quite regularly in their lives — then there’s only so much that your marketing can do. Because again, if your marketing’s like, “We’re going to cure cancer eventually” or “We’re going to go to the moon eventually,” and people are just looking at their lives and the way that they use AI in the present, I just don’t think that the marketing is that persuasive yet.

Zershaaneh Qureshi: Yeah, this does seem like one way in which the insularity of the AI community is having an effect, right? This is not true across the board, but in lots of cases we’ve got people who are living together, reading the same books, going to all the same parties, using all the same jargon.

I wonder how much you think that’s kind of creating a divide with the public, or how much that’s sort of reinforcing this gap that you’re observing?

Jasmine Sun: I think it’s a big part of it. Even when you ask people what do they look forward to when AI goes well, what do you look forward to in the post-AGI utopia?

A lot of folks who I know in San Francisco, in the AI research community, they’re really excited about UBI and they’re really excited about this infinite leisure point of view. When you test UBI with the public, it’s extremely unpopular because most Americans really want to work. They really want to have a job.

Maybe they don’t want the job to be 40 hours a week, maybe they don’t want the job to be this or that. But fundamentally, people really want to work. Or when you ask people, “Do you want to live forever?” I think it’s actually quite a controversial poll.

A lot of people don’t want to live forever. The utopia that Silicon Valley and the AI industry is outlining is not actually a very compelling utopia to a lot of the other people in society, and they don’t realise that because they are in these very insular communities — then even your attempts at positive storytelling don’t really land.

The other thing that I think about sometimes is that — especially as the companies are going to go public soon — for the first five years of these AI labs’ existence, all the messaging is really focused on recruiting investors. You want to get the best technical talent to join your lab, and you want investors to give you a lot of money and take you really seriously.

So you’re actually marketing to the in-group. Even somebody who may not actually care very much about AI safety: an executive might pretend to really care a lot about AI safety because they know that a lot of the researchers do. They’re not going to get the best researchers if they don’t pretend to care about safety.

We saw this with the acquisition of DeepMind into Google. Google made a bunch of promises about an ethics board and AI use in the military. I think arguably a lot of Sam Altman’s community communications about safety have this flavor where he knows a lot of the people who are very important to him care about it, and so he’s happy to say the words.

But when you have people saying words in order to recruit more in-group people, and these words are on the record and they’re on Twitter and they’re on video and then they get shared out to the broad public and it’s like, “Oh man, Sam Altman says that most people are probably going to be worse off.” Or like, “He says that there might be like an X percentage chance that we all die.”

Then they’re like, “Whoa, what the hell are these guys doing? We’re not into that.” So I think that the insularity is also reinforced by some of the market incentives of the first five years of their existence. So now that the companies are trying to pivot into messaging to the public, it’s a much harder task.

Zershaaneh Qureshi: I’m curious what you think needs to happen to get these groups — that is the AI community and the public too — I don’t want to say reconcile because they may never get on, may never sort of agree, but at least start sort of speaking the same language.

Jasmine Sun: I frankly think that at this point, in order to get the public back on board, you’re going to need to start giving people real material benefits. Or maybe it does look like the long-awaited cancer cure. I think if people see medical breakthroughs that are actually happening as a result of AI, then they are going to have a much more positive view on things.

When you look at David Shor’s Blue Rose polling, health advances and scientific advances as a result of AI are some of the highest-polling things among the public. So if it also looks like more investment is shifting towards domains that people are really excited about AI being a part of — and those are visible, those are well marketed — then I think that could also really work.

But again, when we talk about scientific and health advances, it can’t be these advertisements about plausible future things. I think we actually have to show that we are able to access cheaper or better, or whatever it is, whatever cures, treatments, et cetera, as a direct result of the AI companies and their wealth.

Zershaaneh Qureshi: I’m wondering how that view squares up with the phenomenon that you’ve noted where people — at least in the sort of AI populist category that you’ve sketched out — people are so fundamentally mistrustful of the elite groups who are building these things that it kind of doesn’t matter to them matter to them how useful the tools are.

What needs to happen? How do getting these material benefits change their psychology around the mistrust of the elite?

Jasmine Sun: In this case, I think that’s where the economic redistribution stuff is actually really important.

I’m actually not sure that, say, the health advances alone would do something here because I think the mistrust of the elite is fundamentally based in rising levels of wealth inequality, and concerns over affordability and corporate concentration of power in the US — in the sense that wage growth hasn’t kept up with corporate growth, and labour share of income is declining and all of that kind of stuff.

I think because so many of the populist concerns are wrapped up in economic inequality, unless there is a clear economic benefit — the sharing of the pie in a highly legible way — we’re not going to smooth over that set of concerns; which is why sometimes I’m half joking that you need the big gold Trump cheque in the mail that says ‘Trump x OpenAI.’

But I actually think something like that would do quite a lot to persuade people that there was some deal made to share OpenAI wealth or OpenAI equity or OpenAI cash or whatever it is with the public.

But I’m actually cautiously optimistic about things like the OpenAI Foundation: if they can be independent from the corporate side, and they are able to fund an incredible amount of public goods, community programs, research, et cetera, maybe it’ll help. I think they’re good people, they’re doing their best.

Why the Chinese public embraces AI [00:52:52]

Zershaaneh Qureshi: I want to move on here because you’ve also spent a lot of time in China, where I think public attitudes towards AI at least seem on the surface to be very different to what they’re like in the US. You’ve got this situation where people’s grandparents in nursing homes are just exuberant users of DeepSeek.

Also when you have, I think, some survey results, you can see that in China most people see more benefit than drawback to AI, which is a very different picture from in the US.

How do you explain this contrast? Is what’s going on here just some sort of culture of techno-optimism in China at play?

Jasmine Sun: I was there for a couple weeks, meeting with both people in the Chinese AI industry as well as just other random people around. I went for two weeks a little bit ago and I spent a lot of that time asking people about whether there was an AI backlash, whether people were worried about risks, worried about jobs, all of this kind of thing. I think my view of it is it’s less a techno-optimism than it is a kind of techno-determinist pragmatism.

I actually see a lot of the same techno-determinism in China as I do in Silicon Valley, where there’s a sense that progress and modernity has a particular direction — like we are always going to see the economy and technology march forward.

The CCP [Chinese Communist Party] has been very clear about the need for technological advancement in order to advance China itself, as a country and as an economy, to its next stages of growth. Given that, people sort of see it as the only choice you can make: to get on board. There’s not a culture of protest or a culture of resistance in China. I think that’s just part and parcel of being in an authoritarian society, where there’s a lot of government repression of any contrary speech or any attempt to organise in some form or another.

What that means is just ordinary people, like when I talk to my family members, the idea of resisting AI or protesting is almost unfathomable to them because you’re going to get crushed because, you know, the government is very supportive of AI diffusion. You’re not going to win.

Second of all, with things like jobs, China already has a lot of white-collar unemployment. It’s already a very competitive job market. It’s sort of: “If I don’t use OpenClaw, I’m going to fall off and somebody else who is using OpenClaw is going to take my spot.” So a lot of the attitudes around AI are just much more practical. There’s no choice economically and politically except to get on board with AI.

So people are more open to it, and I think more open to adopting AI, and focus more on the economic value it provides. But I would say that a lot of this does have to do with almost this resignation that there is no choice.

Zershaaneh Qureshi: Interesting. I wonder how much what you’re seeing here tracks the reactions to sort of previous waves for automation in China. I don’t know how much you know about this.

One example that slightly goes against the grain of what you’re saying is that I know there has been some backlash to technology in China. There was definitely some public scrutiny and reaction to, I believe, delivery driver algorithms some time ago. I know this is just one example.

Do you have a sense of whether this is the classic trend you see in previous waves of automation in China, or if it’s a little bit different?

Jasmine Sun: Yeah, I wouldn’t say that there’s no dissent or no backlash at all. Like you mentioned, I think delivery drivers have engaged in some forms of organising. There are actually protests around robo-taxis that were introduced in Wuhan. So Baidu has a robotaxi outfit. They started rolling them out and there are like some like ‘protest-lite’ type things.

I will still say that these are very marginal compared to the level of public dissent and protest in the US. There are many, many fewer Chinese people who are willing to engage in these high-risk actions because it is just extremely politically high-risk to do so.

At the same time, the China scholar Matt Sheehan, who’s been tracking some of the AI regulatory conversation in China really well, he did note that he heard in his conversations with the Chinese policy community that the robo-taxi protests were part of what inspired Chinese policymakers to be more active on the labour legislation front in particular.

The other thing that I’ll say is that the Chinese government, while overall still being a promoter of AI and a promoter of technology very broadly, they are more willing to introduce lots of little pieces of iterative legislation around the potential societal harms of AI.

There’s a recent court case in Beijing that ruled it wouldn’t count as just cause to fire a worker because of AI, or China has also legislated companion chatbots quite aggressively.

They obviously have a lot of speech regulations. When I was talking to my aunt who works at a state-owned enterprise in China, she mentioned like, “Yeah, I use Doubao,” which is their chatbot, “and it can probably do the job of two employees. But they’re not going to lay us off because there’s been a promise to at least the state-owned enterprise workers that we will not lay you off because of AI. That’s not a good enough reason.”

And so for some people there’s also a slight sense of security that the government, by being much more active than the US in attempting to regulate AI uses, is going to protect people from the worst downsides.

But overall, I think the main thing I would say is just that China has seen a lot of both tumult in the last 50 years, but also a lot of progress — where if you ask a middle-aged or an older Chinese person, they’ve seen revolutions in their lifetime and they’ve also seen their standard of living just increase dramatically year on year on year.

So yes, they may not like this or that about AI, but fundamentally there’s a sense that this is the way that progress moves and you better get on board or you’re going to fall behind.

AI hype and the journalist’s dilemma [00:59:04]

Zershaaneh Qureshi: I want to talk now about media coverage of AI, because you are someone who works in media and I know that you’ve observed some effect where — in an attempt not to play too much into hype — other media professionals and critics are maybe just not engaging as they should be with the things that advanced AI could do in the future, the more speculative things that haven’t happened yet.

I’m wondering what your perspective is on how to report about these things without falling into this trap of generating too much hype.

Jasmine Sun: It’s really interesting, I think that journalists are still trying to figure out how to cover AI because there is this competing scepticism of any large institution, and particularly one where they have marketing goals, to talk about how powerful and capable their technologies are while also being serious about potential future possibilities.

One is: I do think there’s a pretty prominent bias I’ve observed in the mainstream media to avoid almost ever taking the AI companies at their word, or taking any claim even about the technology itself at their word.

I think a lot of this comes from almost PTSD from the crypto era, where a lot of folks made a lot of promises about how crypto was going to change the world and replace the global financial system.
And none of this really materialised, and there was a lot of scam and fraud.

So journalists felt burned and frustrated by that, such that now, even when I’m writing about AI capabilities, I almost always get a question from my editors that says, “But isn’t it just marketing? Can they really do that? I’m pretty sure it’s just marketing. You should include a caveat about that.”

I understand the default sceptical role that journalists have to play in society, particularly when the entity on the other side is a really powerful corporation.

But for me, one is like: you should never just talk to the companies. You should always — if you’re reporting on the company — sure, ask them what they think about AI progress, but make sure that you also spend a lot of time asking independent experts about the same exact questions. When I report on the economics of AI, I interview people at the companies. I also spent a lot of time talking to actual economists with a wide range of views and other policy professionals.

If there’s an issue that actually it looks like a lot of these entities agree on, for example, I feel like cyber risk and biorisk are things that you can find a lot of people both inside and outside of the companies to say, “We’re all pretty worried about this, and this is a real serious thing that might happen.” Then it’s probably not hype, because a bunch of people who don’t have financial incentives to say so are saying it too.

This is all very simplistic, but rather than this attitude that, “If the companies say it, it must be wrong,” my request is just go talk to some independent experts and see if they also think it’s wrong. If they don’t, maybe there’s something to it. But I don’t know.

I think I basically try to spend quite a lot of time trying to talk to every party or every tribe on an issue, when possible. I try to talk to both Republicans and Democrats. I try to talk to academia, as well as people in the industry, as well as people in policy.

Anytime I report on an issue, I’ll try to make sure I’m hitting every stakeholder bucket to avoid a sort of tribalism. Also I even have a Claude prompt that I’ll use sometimes, where I feed it a draft of something I’ve written and I say, “Do you think that every stakeholder group, or every party I’ve represented and quoted, would consider this a fair interpretation?” — “fair” being different than flattering; I think flattering is super different.

But I try to basically make sure that I’m having conversations with and attempting to see my story from every possible side, rather than just neglecting entirely some group of people. Again, there are journalists who will not talk to people at a company. There are others who will not talk to some AI sceptic.

There’s never been a better time to work in AI safety [01:03:07]

Zershaaneh Qureshi: Through this conversation and from reading your Substack, it does feel like we have unearthed a lot of uncomfortable truths about the AI world. Are there any reasons to be hopeful that you want to highlight?

Jasmine Sun: I actually think that it’s a really great time to be working in and around AI even, and maybe especially if you are concerned with safety, concerned with societal impacts, concerned with some of these broad public issues.

That’s partly because, as I’ve started to engage a little bit more with the policy and the political conversation, it’s clear to me that the Overton window in, one, policy and, two, actually philanthropy, both seem to be expanding extremely quickly.

Literally between my trip to DC in early February and late March, I noticed — in a month and a half — how some of the same people who were saying, “I don’t think policymakers even believe that these things are more than stochastic parrots,” were starting to go, “Oh, it looks like a lot of people in my constituencies and my voter base are really worried about AI, guess I’ve got to start taking this seriously.”

I think that Mythos was a big update. I think that Claude Code was a big update, and Claude Cowork. I think that for people, for a lot of these more legacy institutions in the policy world or in the philanthropic world or in civil society who have spent years shutting their eyes to the possibility of advanced AI progress, that is starting to fall away.

People really are starting to take AI seriously. What that means is we’re actually in this kind of funny scenario where there are a lot more politicians looking for ideas of what to do about AI than we actually have compelling ideas and solutions.

I think for anyone who wants to work, for example, in AI policy who is willing to actually branch out and, say, build relationships with labour groups or environmentalists, or whether it’s on the right or the left or whoever, and spend some of the time to get some of these people who are brand new to AI but who actually really want to do something on it, because they believe it’s important and they believe it could be politically helpful to actually get some good, serious ideas in their hands.

Just the number of times that I’ve had conversations with congressional staffers — very influential people in Congress — who are saying, “Yeah, my representative really wants to do something on this. They’re just still in the process of learning what to do.” So I think the potential for impact in policy is really high.

Then the second one I mentioned was philanthropy, of course, which I think a lot of folks are aware of. There’s tons of money, both individual donations from people who are about to get really wealthy off these IPOs [initial public offerings].

There’s things like the OpenAI Foundation, but also the traditional philanthropic actors, whether Gates or Rockefeller or whatever — all of them are starting to build out their own AI plans. There are billions of dollars in the next couple years that could go towards tackling AI-related issues. Once again, I think that there is more desire to spend the money than there actually are organisations and solutions to fund.

So I think there’s a tonne of opportunity, if you’re a young person looking to make your career right now, to be working on AI issues.

Zershaaneh Qureshi: Awesome. Thank you so much, Jasmine. I think that’s all we’ve got time for, but you’ve been a pleasure to have on the show. Thank you for coming.

Jasmine Sun: Thanks so much for having me. This was fun.