Transcript
Introduction [00:00:00]
Tom Reed: AI systems are starting to build themselves. Because each generation of models will be better at building its successor than the last, it seems plausible that the full automation of AI R&D could rapidly lead to an exponential growth in overall AI capabilities. A natural inference is that domain-general superintelligence arrives shortly after AI research is automated.
This natural inference is, indeed, almost the default story of AI progress, as well as the strategy of those pursuing it. Where the AI industry once distinguished itself by the search for general intelligence, it has in recent years converged on a narrower path: the targeted pursuit of coding and AI research. For Sam Altman, for instance: “perhaps nothing is quite as significant as the fact that we can now use [AI] to do faster AI research.”
To many, the (ostensibly imminent) automation of AI research looks sufficient for the rapid subsequent development of other world-changing capabilities. It would constitute, in Jack Clark’s words, the crossing of “a Rubicon into a nearly-impossible-to-forecast future.” The writers of AI 2027 think the full automation of coding will produce systems “vastly better than the best humans at every cognitive task” within just one year.
On the extreme version of this view, as Ajeya Cotra puts it, “it’ll take only months or weeks or days to create sci-fi technologies… that would have taken centuries at human rates.” Perhaps most importantly, this progress could unfold almost entirely within the confines of data centres, unconstrained by the rate of AI’s wider diffusion into the economy, and take the rest of the world by surprise.
I don’t think this will happen.
I don’t think that automating AI R&D will rapidly produce domain-general superintelligence. I don’t think that a data centre of geniuses, largely separated from the rest of the economy, will figure out how to cure Alzheimer’s, design mosquito-sized killer drones, or manage a Fortune 500 company.
Of course, I am far from alone in thinking this. Others have explored the possibility that even full R&D automation will be bottlenecked on experimental compute, or rapidly hit diminishing returns, or that the main barrier to transformative change is the size of the economy itself. But I have a sneaking suspicion that there is a further, as-yet-unarticulated reason not to expect automated R&D to rapidly produce a domain-general superintelligence from within the confines of a data centre.
Put simply, I think it’s impossible to get good at solving problems without access to a source of those problems, and that arduous interaction with the real world in serial time is the only such source. I therefore believe that the development of domain-general superintelligence will depend on the laborious, painful and expensive deployment of AI into the real world. This would in my opinion, of course, be great news.
Versions of this view have been articulated by Herbie Bradley (in a scattershot series of excellent tweets) and Dwarkesh Patel (who writes of the necessity of a “broadly deployed intelligence explosion“). But I have yet to see this view laid out in terms clear enough for me to internalise — so what follows is my attempt to do so.
I believe the automation of AI R&D will not rapidly lead to domain-general superintelligence because:
- It’s impossible to get good at most things without practice.
- AI companies lack the data their models would need to practice most things.
- I don’t think this can be fixed with “sample efficiency.” In most cases, the relevant data doesn’t exist at all.
- This also can’t be fixed with simulations or synthetic data. In part, this is because the automated AI researchers won’t be able to evaluate success on tasks they have yet to understand. But it is also because the relevant data can only be generated by interactions with a market whose actors have preferences that are unknown and therefore unsimulable.
- In sum, this means that the relevant data for superintelligence in most non-coding domains will only become available through deployment of AI models throughout the economy.
The singularity, therefore, will be bottlenecked on signal. The output of the R&D produced by an isolated data centre of geniuses would be a mere “Goodhart Singularity.” An isolated AI improving itself against benchmarks would only appear to be approaching superintelligence, while actually optimising for eval performance that fails to generalise beyond the lab.
This suggests to me that the automation of AI research will not rapidly produce superintelligent capabilities in other domains — their arrival will largely be a function of deployment and data collection in the real world. AI models need real-world deployment for the same reason the body needs pain and corporations needs profit: signal is sovereign.
Let’s take these points in turn.
Practice makes perfect [00:05:05]
First, I think that domain-specific intelligence requires practice. This is because the current trajectory of AI progress belies the notion that AIs acquire capabilities by virtue of a singular g they are accumulating. Instead, it looks to me that AIs become good at the things they are given the chance to practice.
I think this is one reason AI capabilities have turned out to be far, far more jagged than we had any right to expect. Few in 2016 would have predicted an intelligence that could outscore Stanford Law graduates on the Bar without coming even remotely close to practicing law, let alone consistently lose at tic-tac-toe.
Stories like AI 2027 sometimes read as if “automating AI R&D” were a phase-shifting inflection point after which jaggedness inexplicably ends. But I don’t know why that would be the case, when jaggedness seems primarily an artefact of the distribution of training data. We should expect jaggedness to last longer than we expect, even after we adjust for jaggedness lasting longer than we expect. Call this Toner’s law.
To illustrate this point, let’s turn our attention to the current toolkit of capabilities progress. From what I can tell, the elaborate construction of bespoke RL environments is a major engine of such progress. How do these environments get made?
Well, if you read an account by one of the providers, you may be surprised to find that the work of building superintelligence is proceeding one patch at a time. As Mechanize describe it, each environment is the product of a single engineer spending a week patching a single observed failure in a frontier model. “We used to get bundles of capabilities with each scale,” observes Zhengdong Wang, but “now we put in the work to unlock each one.” Are we expecting to solve week-long problems all the way into the singularity?
The belief that “automating AI research” is the way to rapidly get pretty much “everything else” looks like an undercooked bet to me. Companies are aspiring to unlock the singularity through a war of attrition on AI coding and research, after which the magic of “algorithmic progress” and “sample efficiency” imply that further capabilities can be summoned without requiring a week of bespoke schlep from a blackpilled Mechanizer. Dwarkesh I think has clearly internalised this: the labs’ own actions “hint at a worldview where these models will continue to fare poorly at generalizing… making it necessary to build in the skills that they hope will be economically valuable.”
Now of course it’s possible I’m mischaracterising RL. Maybe RL does generalise better than the above Mechanize description would imply. But if this were the case, I’d have expected AI companies to do a much better job of demonstrating this by now! We’ve been pumping the RL paradigm for well over a year, and I have yet to see any compelling evidence or narrative that RL will introduce anything other than a piecemeal singularity. When Dwarkesh puts this challenge to Dario — repeatedly! — in their second interview together, every single one of Dario’s counters ends up weakly analogising RL to GPT-2-era generalisation from pretraining.
Good data is hard to find [00:08:22]
My second point is that AI companies don’t have the data to practice the things that matter, even if sample efficiency is improved.
I don’t think AI companies currently have access to the right kinds of data to enable their models to practice things they want their models to do. I also don’t think this data will be cheap or easy to acquire in the future without widespread diffusion of their models throughout the economy.
This assertion is partly a guess. But I feel fairly confident that no AI company has access to datasets that provide much signal on, say, which of several candidate drone designs would be a greater contribution to Ukraine’s war effort.
The natural rebuttal to the claim that we don’t possess enough data for superintelligence is something like: “LLMs are wildly sample inefficient! This can be dramatically improved, to match human-level or beyond!” To make this case, AI bulls can smugly point to the ways in which DeepSeek achieved GPT-4-level capabilities at a fraction of the compute.
I’m not convinced this is correct. To begin with, I’m far from convinced that the LLM industry has a strong track record of wild improvements to sample efficiency. As both Anson Ho and Beren Millidge point out, improvements in intelligence-per-FLOP seem largely attributable to improvements in data quality, not algorithm quality. The record for pure algorithmic innovations — particularly ones which reap rewards at a constant compute stock — looks fairly weak.
But more fundamentally, it strikes me that for most tasks of interest, the data required does not exist at all — at least not in a form that companies can readily feed to their hungry, hungry models. Even orders of magnitude of increase in sample efficiency get you nowhere if the sum of relevant data is literally zero.
A good test for this intuition is to ask how much coding capability progress we would get at current levels of talent and compute if the internet had never existed. My guess is that it’d be a very painful process, requiring tens of billions of hours of human labour. My claim is that we live in this “no internet” world for most kinds of tasks.
We live in the “no internet” world for most tasks because the relevant data in the training corpus is of the wrong kind. For most tasks, what we have amounts to descriptions, commentary, or advice — but no record of the tasks involved in performing the task itself. So even when there’s a lot of text about a task, that text does not provide the model with what it needs to perform the task. My sense is that, for most economically interesting tasks, there’s a large gulf between predicting tokens describing a task and predicting tokens that perform that task successfully.
One way to make this gulf obvious is to ask LLMs for strategy advice on a boardgame. Ideally this boardgame should be popular enough to have plenty of strategy advice online. Dominion — which Claude isn’t great at — makes for a good example. The training pipeline that has taught Claude to write compellingly about Dominion is ill-suited to producing an agent that could actually win a game. This is because the objective it’s optimising for is “produce advice that looks good,” and not “propose a strategy that wins.” I would say LLMs relate to most tasks kind of like McKinsey does to running a company.
The obvious exception to the above is coding, for which the tokens left to us by our ancestors are constitutive of the task itself. LLMs are best at tasks whose natural substrate is text — tasks for which the textual version of a problem is the very problem you want to solve. They are good at tasks whose primitives are naturally represented as tokens in training data. When the task is one which, by default, we have conducted in and through tokens left behind in the training data, then predicting tokens in the pretraining corpus amounts to completing the version of the task that the text describes. But where this is not the case, the relationship between accurately modelling a description of a thing and actually doing that thing breaks down.
For most tasks in the economy, the pretraining corpus contains writing about the task, but not a record of the task itself. This is of course one of many reasons coding has progressed faster than other domains — code is one of the neat cases for which the task itself is almost entirely reducible to its token trace.
This means that, for most tasks, our approach to training LLMs is akin to trying to train AlphaZero on chess commentary rather than on games of chess. My take would be that even a 100% success rate at predicting the tokens of a Robert Caro biography doesn’t generalise to becoming POTUS.
Of course, this suggests two complementary paths forward:
- We find ways to faithfully represent ever-greater portions of the universe in token form, giving everything an API, allowing us to train agents to solve real-world problems in the substrate of tokens.
- We give agents the ability to act more directly on the world (through computer use now, robotics later), instantly unlocking a broader class of non-textual problems currently incompressible into tokens.
Either way, AI companies won’t be able to get their hands on data that make their models good at non-coding problems without deploying these models into the economy (and successfully enticing their customers to share the relevant data with them).
Simulation is shallow [00:13:43]
My third point is that the data required for practice cannot be simulated.
There’s a natural rebuttal to the above: you can just simulate all the data you need, such that diffusion is unnecessary. Maybe this is in fact true. But I’m sceptical.
One source of my scepticism is entirely unoriginal: I think it will be difficult for AIs in the data center to tell whether they are in fact improving at a skill if they lack ground-truth access to that skill.
Consider that almost half of submissions to the eval SWE-bench that were accepted by AI auto-graders would be rejected by the actual human maintainers of the relevant repositories. To me, the fact that you can pump these SWE-bench scores without increasing actual merge rates is suggestive of the situation the data centre genius will find itself in.
Just how is Claude 8 supposed to know whether its latest galaxy-brained algorithm has improved Claude 9’s ability to become POTUS purely by looking only at the results of the Become-POTUS eval? We all know that evals, whilst useful, remain stubbornly incomplete proxies of real-world capabilities. At the end of the day, the only eval that matters is the voter, the customer, the economy, and the bottom line. Without access to any of these, your singularity will be mercilessly Goodharted.
The great Zhengdong makes this point about the progress of AI research itself. Not only are “evals” the only things that models are capable of getting good at, but “the researchers [themselves], they just wanna optimise… they just want an important problem to solve, a clear evaluation that measures progress towards it, and then they just wanna optimise it.” I suggest that AI companies need real-world deployment as a source of problems, or else they will have no good targets for optimisation.
I have another reason that is both more fundamental and more speculative for asserting the implausibility of reaching domain-general superintelligence through simulation. I suspect that even an arbitrarily intelligent simulator would be incapable of generating relevant practice data for several tasks, because the relevant data can only be generated through an interaction with the relevant market. The data simply does not exist anywhere prior to the deployment that would generate it, and that deployment involves modelling the preferences and knowledge of actors in a market that are unknown to the simulator.
From what I understand (and I’m not entirely sure I understand it well), this is basically a Hayekian point. The signal that something is good is generated by millions of actors revealing their preferences through their behaviour. You can’t simulate this from inside a data centre, because the inputs aren’t being withheld through any contingent failure of data collection — they just don’t exist anywhere yet, and they won’t until the market generates them. Unless your AIs have access to this market, they won’t be going anywhere.
Again, board games provide a fruitful analogy here. Twilight Imperium is one of my favourites. Because each game takes around 12 hours, I play only rarely. This gives me weeks and weeks of time to enjoyably theorycraft my strategy ahead of each prospective game. Twilight Imperium is deliberately complex and Byzantine, so the prep is usually quite a lot of fun to me: I sit alone with a Google Doc and tabs of Reddit strategy threads, gaming out my turns of every round, getting a kick out of how absolutely cooked all my opponents are going to be. Unfortunately, the reality never lives up.
Time and again, my best-laid plans bear shockingly little resemblance to how the game actually plays out. As it turns out, I kinda suck at this game, and my model of how the game was supposed to play out approaches zero predictive validity by about minute 5 of the total 12 hours of game time.
I posit that the RSI’d “superintelligence” that breaks free from the data centre after a year of “algorithmic progress” grindset will feel a little like I do by Round 2 of a game of Twilight Imperium: my fleets destroyed, my resources depleted, and the allies I planned to cultivate conquering my home system.
A further obvious objection to the picture I’ve painted above is, of course, continual learning. Intelligent humans thrive in foreign environments because we can learn as we go along. Is it not at least conceivable that our automated AI researchers, toiling away in their secluded data centres, would be capable of crafting the perfect continual learning algorithm? They need not know everything before breaking out and taking over; they need only to have developed superhuman sample efficiency, such that they become rapidly godlike within mere moments of leaving the data centre.
This too, I am sceptical of. How can you craft a perfect continual learning algorithm without a proper diverse set of problems to learn over? That’s how evolution did it. It’s just not clear to me that the data centre of automated AI researchers would be able to judge the relative merits of one continual learning algorithm over another without a real problem set to test it on. As Herbie Bradley puts it, we should not “model AI labs as a black box separate from the world, within which superintelligent AI can improve with no outside dependencies.” Intelligence can’t explode in a vacuum.
What a Goodhart Singularity looks like [00:19:04]
To be absolutely clear, I don’t think any of the above means that superintelligence is impossible, or that the world still has a shot at looking remotely normal 50 years from now. It almost certainly does not. It’s also clear to me that most people warning about AGI don’t think that we should only be worried because the automation of AI R&D will rapidly lead to domain-general superintelligence.
But I think that the conversation as a whole has tended to treat “diffusion” as something of an irrelevance to the progression towards superintelligence. The fact that businesses may be slow in adopting AI is only deemed relevant insofar as it means the world outside San Francisco won’t know what’s going on, or insofar as it might make raising money harder for labs. Diffusion itself is not usually seen as a source of capabilities progress.
Similarly, on the other side, those who have for years harped on about the significance of diffusion have not (at least to my knowledge!) clearly articulated why exactly the future AI superintelligence would have to be preceded by thousands of deployments into mom-and-pop shops, middle schools, and Siemens. The above argument is my attempt to bridge this gap.
If I’m correct, a couple things follow.
The first is that AI policy’s growing focus on “internal deployment” and “automating AI R&D” may well be misplaced. I don’t think superintelligence is impossible — it just won’t arrive in the shape we currently anticipate, or on the timelines currently forecasted. The next phase of capability progress will look less like a data centre compounding in private and a little more like a grind of deployment, customer discovery, and real-world data collection. If that is what the future looks like, the policies we should be pursuing look quite different.
For one, this world would suggest that data holders hold far, far more power than they currently realise, and they should leverage it (as, for example, Ukraine is doing). The asymmetry between “company that can build a frontier model” and “entity that holds proprietary, deployment-grade data on a real economic activity” might shift, and we should think about what a preferable shift could look like. This is probably of particular significance to Europe, a continent with an unfortunately ever-vanishing shot at building superintelligence, but a glorious legacy of doing science and industry (and thus a rich stockpile of data).
Another implication is that AI labs may well just set out to conquer verticals one by one. If the relevant signal can only be acquired through deployment, it may well be the case that the winning move is to just become the deployer. David Oks makes the point that labour displacement is more likely to come from something like Dwarkesh’s fully automated firm than from incumbent AI adoption. It strikes me as plausible that Anthropic or OpenAI’s best bet is to be clever about which verticals to serially conquer and own end to end. We should all start thinking about the conditions under which that pathway is or is not desirable.
