High impact interview 1: Existential risk research at SIAI
The plan: to conduct a series of interviews with successful workers in various
key candidates for high impact careers.
The first person to agree to an interview is Luke
Muehlhauser (aka lukeprog of Less
Wrong), the executive director of the Singularity
Institute for Artificial Intelligence, whose mission is
to influence the development of greater-than-human intelligence to try and
ensure that it’s a force for human flourishing rather than
extinction.
Each interview will have one or two goals. Firstly, to probe the experience of
the job itself, to give readers a sense of what sort of life they’d be letting
themselves in for if they followed a similar path I’ve divided these question
types under headers.
Secondly, where the interviewee’s organisation is a candidate for philanthropic
funding, to seek their insiders’ perspective on why donors should consider
picking them over the other options.
On with the interview:
Working in SIAI (and similar X-risk-related careers)
ZR: Can you describe a typical working week for you? How
many hours would you put in, what proportion of them would be spent on work you
find engaging and what on admin/other chores? More importantly, what,
physically do you do in what proportion from day to day? Can you give a sense
of the highs and lows of the job?
LM: My work log says I’ve worked an average of 61 hours per
week since the beginning of September, when I was hired. This period covers a
transition from Research Fellow to Executive Director, so my “typical work
week” and the ratio between engaging/boring hours has changed over time. At the
moment, a typical work week consists of (1) managing the 80+ projects in my
Singularity Institute project tracker, along with all the staff members and
volunteers working on pieces of those projects, (2) taking calls and meetings
with advisors, supporters, volunteers, and external researchers, (3) writing
things like So You Want to Save the
World and Facing the
Singularity, and also internal
documents like a list of potential AI risk papers, and (4) working directly on
10-30 of our ongoing projects: everything from our forthcoming website redesign
to our monthly progress reports to strategy
meetings to improving our internal systems for communication and collaboration.
Physically, it’s mostly working at a computer or sitting at a coffee or lunch meeting.
The highs come from doing things that appear to be reducing existential risk.
The lows come when I smack into the extreme complexity of the strategic
considerations concerning how to get from the world in its current state to a
world where people are basically happy and we didn’t destroy ourselves
with our ever-increasing technological powers. How do we get from here to
there? Even things you might think are “obviously” good might actually be bad
for complicated sociological and technological reasons. So, I often have a
nagging worry that what I’m working on only seems like it’s reducing
existential risk after the best analysis I can do right now, but actually
it’s increasing existential risk. That’s not a pleasant feeling, but
it’s the kind of uncertainty you have to live with when working on these
kinds of problems. All you can do is try really hard, and then try
harder.
Another low is being reminded every day that humans are quite capable of
“believing” that AI risk reduction is humanity’s most important task without
actually doing much about it. A few months after I first
read
about intelligence explosion I said, “Well, damn, I guess I need to change my
whole life and help save the world,” then quit my job and moved to Berkeley.
But humans rarely do things like that.
ZR: What about a slightly atypical week? What sort of
events of note happen rarely, but reliably?
LM: Once a year we put on our Singularity Summit, which for
about a month consumes most of our staff’s time, including mine. When I need to
give a speech or finish a research paper, I will sometimes need to cut myself
away from everything else for a few days in order to zoom through the writing
and editing.
ZR: SIAI seems to employ a number of philosophers, as its
more scientific researchers. Many readers of this blog will likely be midway
through philosophy or similar degrees, which might not lead easily to high
income careers. Suppose that they consider X-risk research as another option –
how plausible an option is it? Ie what proportion of postgraduate philosophy
students who take a specific interest in X-risk issues do you think will be
able to find work at SIAI or in similar organisations?
LM: The number of x-risk organizations is growing, but they
are all quite funding-limited, so jobs are not easy to find. In most cases, a
skilled person can purchase more x-risk reduction by going into finance or
software or something else and donating to x-risk organizations,
rather than by working directly for x-risk organizations. This is true for
multiple reasons.
Also, the kind of philosophy you’re trained in matters. If you’re
trained in literary analysis and postmodern philosophy, that training won’t
help you contribute to x-risk research. Somewhat less useless is training in
standard intuitionist analytic philosophy. The most relevant kind of philosophy
is naturalistic “formal philosophy” or the kind of philosophy that is almost
indistinguishable from the “hard” cognitive sciences.
Mathematicians and computer scientists are especially important for work on AI
risk. And physicists, of course, because physicists can do anything.
Naturalistic philosophers, mathematicians, computer scientists, and physicists
who want to work on x-risk should all contact me at
[email protected].
Especially if you can write and explain things. Genuine writing ability is
extremely rare.
ZR: Can you describe roughly the breakdown of different
types of specialist who SIAI employs, so that anyone wanting to dedicate
themselves to working for you can see what the most plausible routes in are?
Are there any other relevant factors they should be aware of? (eg you employ
more people from field x than y, but so many people from x apply that it makes
y a better prospect)
LM: More important than our current staff composition is who
we want to hire. The most important and difficult hires we need to
make are people who can solve the fundamental
problems in decision theory,
mathematics, and AI architectures that must be solved for Friendly AI to be
possible. These are people with extremely high mathematical ability:
gold-medalists in the International Math Olympiad, or people who ranked top 10
on the Putnam, for example. For short, we refer to this ideal team as “9 young
John Conways” + Eliezer Yudkowsky, but that’s not to
say we know that 9 is the best number nor that they all need exactly the same
characteristics of a young John
Conway. We also need a
math-proficient
Oppenheimer to manage the
team.
Most young elite mathematicians do not realize that “save the world” is a
career option for them. I want to get that message out there.
ZR: Overall, what would you say is the primary limiting
factor in attaining SIAI’s goals? In other words, would someone keen to help
you by any means necessary do better by going into professional philanthropy
and donating to you, or by training themselves in the skills you look for,
assuming they could expect to be about equivalently successful in either route?
(or some third option I haven’t thought of?)
LM: Funding is our main limiting factor. The possible
non-existence of 9 young John Conways is another limiting factor. But if we
could find the right people, it’s actually not that expensive to take a shot at
saving the world once and for all with a tightly-focused team of elite
mathematicians. The cost could be as low as $5 million per year, which is far
less than is spent on cosmetics research every year. Unfortunately, humanity’s
funding priorities are self-destructive.
Considerations for philanthropists
ZR: Givewell & Giving What We Can’s research has
tended to focus on health interventions, since QALYs and similar offer simple
metrics for judging how well each organisation does at achieving the underlying
goal. It’s understandable given the data available, but makes it tough for
people considering other types of cause. So I’ll throw the challenge over to
you – treating your goal simply as reducing X-risk, can you think of any
measure by which someone could evaluate how effective you guys are? – ie the
effect of a dollar spent on SIAI vs FHI vs one spent on something more
conventional with a similar goal? (CND or asteroid defence, for example)
LM: Alas, there is no QALY-like unit for measuring existential
risk reduction! Measuring how much x-risk reduction is purchased per marginal
dollar invested in the Singularity Institute vs. the Future of Humanity
Institute vs. other organizations is difficult to measure. Rather than get into
a long and fuzzy analysis about that, I can approach this topic from another
angle. The short story is:
In the next two centuries we will have a multitude of chances to destroy
ourselves with powerful new technologies. AI looks like it may be one of the
first existential threats to be created. But unlike the others, doing AI
right can actually prevent the other existential risks from
happening. There is no more important thing humanity can do. So, prioritize
support for the organizations that look like understand the problem better than
everyone else and can make progress on it. Right now, the two most plausible
candidates for this are the Future of Humanity Institute and the Singularity
Institute.
ZR: In a recent interview, Nick Bostrom was asked who he
would recommend giving to of SIAI and FHI given their similarities. His reply,
slightly paraphrased (Thanks to George McGowan for transcribing (and asking!) this.):
The two organizations have a lot of overlap in terms of their missions. They
are pretty synergistic – and therefore if one were about to go under you should
probably donate to that one. There is also a lot of collaboration between the
two organizations – in papers we write and so on. However there are notable
differences. SI doesn’t have to deal with bureaucracy and try to get grants (as
we do). They can also more easily hire people from non-academic backgrounds to
do useful work. On the other hand – we have more influence in academia and turn
out a greater number of papers. Our sights are on all x-risks, whereas SI
focuses just on AI. So it’s really a question of which set of characteristics
you think are the most important.
ZR: Do you agree with his answer? Is there anything you’d add to it?
LM: Yes, actually. I made roughly those same
comments
about the synergy between the two organizations a few months before that Q&A
with Nick was posted online.
I think focusing on AI risk is most important, and Nick might actually agree.
He has spent much of the last year writing a book analyzing the AI risk
situation, and myself and others at the Singularity Institute have given him
comments on early drafts.
Lastly, Luke has kindly agreed to field some follow-up questions. If there’s an
important question I didn’t ask, or you’d like clarification on any of hi
replies, post the question in the replies by Sunday 8th February. Luke
won’t have time to check back on this thread, but has kindly agreed to field
some follow-up questions, so I’ve agreed to be pass on the questions to him in
a single email by that date.