The most interesting startup idea I’ve seen recently: AI for epistemics

By · Published May 19th, 2024

This was originally posted on benjamintodd.substack.com.

If transformative AI might come soon and you want to help that go well, one strategy you might adopt is building something useful that will improve as AI gets more capable.

That way if AI accelerates, your ability to help accelerates too.

Here’s an example: organisations that use AI to improve epistemics — our ability to know what’s true — and make better decisions on that basis.

This was the most interesting impact-oriented entrepreneurial idea I came across when I visited the San Francisco Bay area in February. (Thank you to Carl Shulman who first suggested it.)

Navigating the deployment of AI is going to involve successfully making many crazy hard judgement calls, such as “what’s the probability this system isn’t aligned” and “what might the economic effects of deployment be?”

Some of these judgement calls will need to be made under a lot of time pressure — especially if we’re seeing 100 years of technological progress in under 5.

Being able to make these kinds of decisions a little bit better could therefore be worth a huge amount. And that’s true given almost any future scenario.

Better decision-making can also potentially help with all other cause areas, which is why 80,000 Hours recommends it as a cause area independent from AI.

So the idea is to set up organisations that use AI to improve forecasting and decision-making in ways that can be eventually applied to these kinds of questions.

In the short term, you can apply these systems to conventional problems, potentially in the for-profit sector, like finance. We seem to be just approaching the point where AI systems might be able to help (e.g. a recent paper found GPT-4 was pretty good at forecasting if fine-tuned). Starting here allows you to gain scale, credibility and resources.

A forecasting bot made by the AI company FutureSearch is making profit on the forecasting platform Manifold. The y-axis shows profit. This suggests it’s better even than the collective predictions of existing human forecasters.

But unlike what a purely profit-motivated entrepreneur would do, you can also try to design your tools so that in an AI crunch moment, they’re able to help.

For example, you could develop a free-to-use version for political leaders, so that if a huge decision about AI regulation suddenly needs to be made, they’re already using the tool for other questions.

There are already a handful of projects in this space, but it could eventually be a huge area, so it still seems like very early days.

These projects could have many forms:

  • One example of a concrete proposal is using AI to make better forecasts or otherwise improve truthfinding in important domains. On the more qualitative side, we could imagine an AI “decision coach” or consultant that aims to augment human decision-making. Any techniques to make it easier to extract the truth from AI systems could also count, such as relevant kinds of interpretability research and the AI debate or weak-to-strong generalisation approaches to AI alignment.
  • I could imagine projects in this area starting in many ways, including at a research service within a hedge fund, in a research group within an AI company (e.g. focused on optimising systems for truth-telling and accuracy), at an AI-enabled consultancy (trying to undercut the Big 3), or as a non-profit focused on policy-making.
  • Most likely, you’d try to fine tune and build scaffolding around existing leading LLMs, though there are also proposals to build LLMs from the bottom-up for forecasting. For example, you could create an LLM that only has data up to 2023, and then train it to predict what happens in 2024.
  • There’s a trade-off to be managed between maintaining independence and trustworthiness, vs. having access to leading models and decision-makers in AI company and making money.
  • Some ideas could also advance frontier capabilities, so you’d want to think carefully about how to avoid that. You might stick to ideas that differentially boost more safety-enhancing aspects of the technology. Otherwise, you should be confident any contribution a project makes to general capabilities is outweighed by other benefits. (This is a controversial topic with a lot of disagreement, so be careful to seek out and consider the best counterarguments to your conclusion.) To be a bit more concrete: finding ways to tell when existing frontier models are telling the truth seems less risky than developing new kinds of frontier models that are optimised for forecasting.
  • You’ll need to try to develop an approach that won’t be made obsolete by the next generation of leading models but can instead benefit from further progress at the cutting edge.

I don’t have a fleshed out proposal, this post is more an invitation to explore the space.

The ideal founding team would cover the bases of: (i) forecasting / decision-making expertise (ii) AI expertise (iii) product and entrepreneurial skills and (iv) knowledge of an initial user-type. Though bear in mind that if you have a gap in one of these areas now, you could probably fill it within a year.

If you already see an angle on this idea, it could be best just to try it on a small scale and then iterate from there.

If not, then my normal advice would be to get started by joining an existing project in the same or an adjacent area (e.g. a forecasting organisation, an AI applications company) that will expose you to ideas and people with relevant skills. Then keep your eyes out for a more concrete problem you could solve. The best startup ideas usually emerge organically over time in promising areas.

Existing projects:

Learn more:

If you’re interested in this idea, I suggest talking to the 80,000 Hours team:

APPLY TO SPEAK WITH OUR TEAM

I’m writing about AI, doing good and using research to have a nicer life. Subscribe to my Substack to get all my posts.