Transcript

Robert Wiblin: Hi listeners, this is the 80,000 Hours Podcast, the show about the world’s most pressing problems and how you can use your career to solve them. I’m Rob Wiblin, Director of Research at 80,000 Hours.

If you’re very familiar with GiveWell you may want to skip through the first 5-15 minutes of this episode as it may cover things you already know.

Alright, here’s James!

Robert Wiblin: Today I’m speaking with James Snowden. James is a research consultant at GiveWell and he graduated from Oxford University with a degree in philosophy, politics and economics in 2011. He then went on to do an MSc in philosophy and economics from The London School of Economics. After that, he spent several years working as a strategy consultant and then as a researcher for The Center for Effective Altruism. So thanks for coming on the podcast, James.

James Snowden: It’s great to be here, Robert.

Robert Wiblin: GiveWell is actually hiring at the moment, so later on in this interview, we’ll get to discussing what it’s like working there and what kind of people they’re looking for, in case our listeners might be able to fill some of the talent gaps they have. But first off, I imagine most listeners will have heard of GiveWell, but for those who don’t know a lot about it, what does it actually do and what’s its history?

James Snowden: Sure, so we’re an organization based in San Francisco that tries to identify evidence-backed, cost-effective charities and publish all of our research and reasoning online so that anyone can check our rationale. We focus on international giving opportunities, because we think donors can generally have a greater impact in poorer countries. We currently recommend nine top charities, to which we direct about 100 million dollars a year. We also run an incubation grants program to support the development of future top charities.

In terms of our history, we were founded about 10 years ago by Holden Karnofsky and Elie Hassenfeld, who were two people who just came out of the hedge fund industry and were trying to find the most effective way to donate their money. Once they started looking at this online, they found that nobody was really providing this advice, so they started to do it themselves. Yeah, there’s a lot more details and nuance I’m sure we’ll get into, but that’s the broad picture of GiveWell.

Robert Wiblin: Yeah, I’ve already done an interview with Holden, who was one of the founders, where he talks about this a little bit. I hope to do an interview with Elie, the other founder of GiveWell at some point and perhaps go more into the history, because he’s been around from the beginning.

But to begin with, it was just interesting finding the best giving opportunities anywhere in the world, but over time it’s narrowed down to focusing on the developing world and, I guess, very often health charities. Is that right?

James Snowden: Yeah, I think that’s fair. We also recommend charities that are focused on increasing consumption or income for individuals, but a lot of our work is in health.

Robert Wiblin: And that’s just because that’s where you found the best giving opportunities, or the ones that met your criteria?

James Snowden: Yeah, so that’s right. I think … I’m not entirely clear on the history here, because I’ve only been working at GiveWell for one year, but it started off as kind of, let’s find outstanding giving opportunities in a wide variety of different causes.

Over time, they kind of narrowed their focus. I think it was in 2011, though I’d have to check that, and kind of realized that most of the giving options they were most excited about and where they thought they could have the most impact was in development. I think the focus on health … One particular reason for that, other than it’s just a really important thing, is the evidence base in health just tends to be much better developed than for other kinds of things. Because we kind of generally focus on identifying top interventions or types of program, there tends to be more homogeneity between different health programs than you might expect in other areas. So that makes it a lot easier for us to find priority programs.

Robert Wiblin: A lot of people say that GiveWell is looking for the best charities, or top charities, which is kind of true, but, in fact, you’re looking for charities that meet particular criteria, right?

James Snowden: Yeah, so our three main criteria are cost-effectiveness, evidence of effectiveness and transparency, commitment to monitoring evaluation. It traditionally, in the past, we have focused largely on charities that are providing quite direct services to people, so things like anti-malarial bed net distribution or deworming and the reason for that is that it’s generally a lot easier, at least relative to some kinds of giving opportunities to really give the full rationale for that online. I’d say insofar as GiveWell is restricted in the kinds of things we’re looking for, it does tend to be … That restriction tends to be we don’t want our giving opportunities to rely on maybe idiosyncratic world views, that not a lot of people share, or things that are very difficult to justify online. That’s the main constraint we face. We’re still working out how far we want to push outside that.

Robert Wiblin: So I guess you could say you’re looking for the most cost-effective charities where you’re able to demonstrate that they’re actually cost-effective, with a high degree of confidence?

James Snowden: Yeah. I think that’s 90% right. I think there’s a bit of nuance here about what it is to demonstrate with a high degree of confidence. So, for example, deworming is an intervention where we’re not particularly confident about the evidence base, but we are confident at the higher level about our rationale for it. So we’re willing to accept some degree of risk, but I think it’s … Rather than thinking of it as a risk aversion constraint, it’s more about can we explain this based on reasons that we think that other reasonable, well-informed people would come to the same conclusions.

Robert Wiblin: It can’t just be an idiosyncratic opinion that James has about which policies are going to be used, which you can’t then convince other people of.

James Snowden: Yeah, as much as I’d love for it to be that, that doesn’t tend to work. I guess, this isn’t to say you can eliminate all judgment calls, because there are … It’s almost impossible, well, it is impossible, to do that, but it’s about making judgment calls that we think that other people, other reasonable, well-informed people would probably make similar calls.

Robert Wiblin: Yeah. So how many people work there these days? It’s gotten quite large recently.

James Snowden: Yeah, so we’ve got, certainly, more than 20 people working for us right now. So maybe not as large as mega-foundations, but maybe large for an effective altruism organization. That’s roughly split between research and operations. It’s slightly more than 10 in each. Some of the researchers also spend their time working outreach and speaking to donors, which we find for this outreach work, it’s kind of important that our front-facing staff are really familiar with our research, because that’s what a lot of the outreach involves.

Robert Wiblin: How much money is GiveWell moving these days?

James Snowden: Yeah, we move about 100 million dollars a year and we don’t have our full figures in for last year yet, but the last two years, that’s remained relatively stable. We did decline slightly, I think it was last year, but that was based on a decline in large donors, so those giving a reasonably large amount. Our best guess is that that’s likely to be noise, but we still saw a decent growth amongst smaller donors. In terms of smaller donors, we’ve been growing quite steadily.

Robert Wiblin: So you’ve been a research consultant there for about a year. What sort of projects have you been working on since you joined?

James Snowden: Well, when I first joined, and that’s reasonably easy for me to remember, when you first join GiveWell, it’s all about finding the right fit. So you’re just trying to work out what you’re good at and where you can add the most value. So I worked in a bunch of different things early on. One is quick evidence assessments, which are these four hour initial scans we do in a particular intervention where we don’t read the papers in detail, but we grab out the abstracts and try and get a sense of what the evidence base looks like for this program. So that’s QEAs, that’s a very common work you would do as a researcher at GiveWell.

Another is writing interim intervention reports. This is a stage before our full intervention reports, but beyond our quick-evidence assessment, where we spend somewhere between 15 and 40 hours kind of summarizing that evidence and putting it online. I did work on that for eyeglass distribution and anti-retroviral therapy. I’ve done some deeper reviews of the evidence base for some of our top charities, such as Seasonal Malaria Chemoprevention, which is one of our top programs. Also, quite early on, I spent quite a bit of time working on a grant investigation that I brought over with me from my previous role at the Center for Effective Altruism.

Another piece of work was working on more conceptual questions around a cost-effective analysis and writing a few blog posts to make our thinking about those questions explicit. What I’m spending time on at the moment, it seems like the best fit for me and what GiveWell would best be served by me doing at the moment, is these big open-ended investigations into giving opportunities, which are quite different from what we’re currently doing. I’m spending most of my time looking at different policy-based opportunities, but also doing some more process-style work about what our research strategy should be and how we’re performing against the expectations that we set last year.

Robert Wiblin: That’s quite a wide range of projects. Do people typically jump around between different topics that much in GiveWell?

James Snowden: Yeah, I think certainly early on. When we first join, we’re optimizing for just really exploring and understanding what we’re best at. As I’ve been there longer, my work has tended to be more focused. Over the last few months, I’ve really been focused mainly on these policy opportunities. But people really don’t have very fixed roles and we do jump around quite a bit.

Robert Wiblin: Recently you published a blog post on GiveWell’s site talking about your approach to cost-effectiveness analysis and especially the uncertainty and sensitivity analysis that you have to do to figure out how confident you are in the figures that you’re citing on how effective these charities are. Broadly, what was the case that you were making in that blog post?

James Snowden: Yeah, I think what we’re really trying to do with that blog post is just present some of the sensitivities in our cost-effectiveness analysis and show how uncertain we are about these things. I think there can be a tendency, when people see that we put a number on our best guess of how cost-effective a charity is. I think when you put numbers on things, people tend to interpret that as this is it, this is a result, this is science. We try and be as scientific as possible, but I think what we really want to be is as explicit as we can about … not just how uncertain we are, but why we’re uncertain.

The basics of that blog post is I’m trying to explain how we account for uncertainty and also what the most uncertain parameters in our cost-effective analysis were. So in terms of how we account for uncertainty, we try and get inputs from all our research staff for some of the most difficult things in our cost-effectiveness analysis, so usually one or two who have the best context on that particular thing can discuss their findings to the team and then we use a wisdom of the crowd, evidence-based guess to come up with our median estimate. So we have a bunch of empirical parameters and also this is how we do our moral weights, which you might even say the moral weights, there may be no right answer to this. But we need some number to ensure we’re making consistent decisions, so we use the median of our staff’s best evidence-based guesses.

Robert Wiblin: So just to make this more concrete, say if you’re looking into deworming, there are various studies showing what is the potential health gain from receiving deworming tablets if they have worms, but you’re not sure how much to trust these, you might think some of the studies are better than others or you might think that they’re all overstated, so you have to reach some kind of overall judgment about how large the effect is, relative to the literature on this. So a few people look into it and then they talk to everyone and then you do a survey across all of the researchers to see what is their holistic judgment of this question and then you take the median?

James Snowden: Yeah, that’s basically right. So just to clarify, on deworming, again, we actually a very, very small proportion of the benefits that we estimate for our deworming charities are from improved health. The vast majority of the impacts, I think over 90% is from long-run increases in consumption. So what we’ve done there is we’ve taken the headline results from the single long-run RCT, which looked at the line income. Then we thought about how we would discount that and that’s what we call an internal validity adjustment. Do we think that this study would have replicated if it was done perfectly in that context?

The other part of that is external validity adjustments, so if deworming had this large effect in Kenya, doesn’t mean that it will have it elsewhere at different times and different places. So these are the two adjustments that we apply to various headline results. I think this is somewhere we differ from published, academic cost-effectiveness analysis, so the things that are published in journals, which normally take the results of a study at face value. Because we’re trying to make decisions, we just need to make these judgment calls and they’re pretty difficult to make and they’re pretty difficult to justify in a really objective way.

So, for example, we’ve roughly divided the headline results of the Baird randomized control trial, so that’s the study on the effect of deworming on long-run income. We roughly divided that by five, because we think the wider evidence base doesn’t really support deworming. Well, we’ve only taken about 5% off the headline result of bed net distribution, because there’s a lot of randomized control trials on that and they generally find similar results.

So a lot of our staff disagree about these. I don’t think the world has worked out exactly what the best way to interpret evidence is, how much to trust a particular kind of trial. So they are pretty subjective but where possible, we really do try and ground this into thinking about what our prior would be and how much each trial might abate that and try and be as explicit about that as possible. So if you look in our … For donors that are really interested, you can see we’ve all put little notes on the cell tabs in our cost-effectiveness analysis for each of the parameters. You’ve got to trace back the rationale for why it is that we’ve chosen that number and not others.

Robert Wiblin: So just to explain, in that blog post, you linked to this spreadsheet where you’ve got all of the different parameters that all of the different staff members estimated. I think it’s 40-something different figures that I think ten different staff members are trying to estimate and you’ve each explained why you’ve chosen that number.

James Snowden: Yeah, that’s right. I think one thing we are trying to do is reduce the number of things on that sheet. It turns out some of them we did all agree on. It’s also true that some of them are kind of correlated. So the various deworming charities have some of the same inputs going in. But, yes, I think something we have found is that’s very time consuming and I think we do think it is worth doing for some things and we think for other parameters it might make sense to move some of them off the parameter sheet and just take our best guess. Or the people who’ve been working on this for the longest take their central best guess.

Robert Wiblin: So what are some examples of the key uncertainties that you face that are listed in that spreadsheet?

James Snowden: Yeah, so one example would be the effect of malaria on long-term income. I think that this is just a difficult evidence base to assess and we haven’t completed a full review of it yet, but we do still need to make decisions in the short term before we’ve done that. The evidence broadly, there are four natural experiments which look at this, but they’re mostly in P. vivax endemic countries, so that’s a different type of malaria to what’s prevalent in sub-Saharan Africa where our charities work. So people can have really different interpretations about what that effect might be and how to apply that to our charities. I think that’s an example of the kind of input that once we’ve spent more time looking at it, we might well want to take away from the parameter sheet when we’ve done more work.

But until we’ve done that, we think the most robust decision rule we can use, in the short-term is just taking the median of the best guesses of the various staff members.

Robert Wiblin: Okay, so once something becomes certain enough, then you just put in a figure that the researcher has determined, but inasmuch as something is very unclear, then you use this kind of survey method?

James Snowden: Yeah, I think that would be the direction it would move in. I think there are a lot of things we’re unlikely to ever take off the parameter sheet, so the parameter sheet being the one where we use this wisdom of the crowd estimate. I think, in particular, I doubt we’ll ever have a certain moral weights that we all agree on. I think that’s probably going to be something we always use are kind of median estimates from our staff. But yeah, I can certainly imagine other things being taken off that sheet, either because we’ve just got a bit more certain about it or because we just realize that even though we’re really uncertain, most people agree about what their best expected guess is.

Robert Wiblin: So can you just describe the process by which you come up with these numbers? Every year you produce these charity recommendations and I guess you have to produce these updated cost effectiveness numbers for each of the charities that you’re recommending. Each time in December when you update the charity recommendations, so is there some kind of crunch time in November where everybody has to come up with their estimates for these different numbers?

James Snowden: Yeah, that’s right. There is a crunch time, it’s in November and not a lot of us get a lot of sleep in that time. So yes, we basically we have an owner of the cost-effectiveness analysis, so that’s Chris Smith. He spends a lot of his time working on the spreadsheet. He’s generally in charge of just making sure everything’s linked up properly and making sure that we’re being consistent in our calculations and thinking about ways to improve that.

So around about November we all get together and discuss various different inputs. This is kind of an iterative process, so we’ll get together, discuss it and then we’ll try and identify which of the things we really disagree about and what would change our minds. Then have these kind of conversations with each other where we kind of try and focus on the thing … Where two people disagree, we try and focus on what would change each other’s mind. It’s not always possible to drive consensus on this, but that’s our broad approach. We often still end up with large differences and that’s just a result of different people having different, maybe epistemic about what kinds of studies they trust or what their general prior for how much they should discount a particular intervention might be.

What we do find, often, is that the people who have the greatest context and the best knowledge of that particular intervention are often the ones who were driving what the decision was there. If you look at the cell notes that we put in our latest cost-effective analysis, a lot of it will be like, I am deferring to this other staff member because I am convinced by their argument, rather than everybody doing everything completely independently.

Robert Wiblin: Not everyone has the time to look into all of these different numbers so they what, pick a few where they think that they have an edge at producing a better estimate and look into those a lot and try to convince others?

James Snowden: Yeah, so I think that’s something that we’re trying to move towards. I think at the moment we probably do spread ourselves too thin. That’s my view, that’s not the GiveWell consensus view. I think it would be pretty helpful if we had owners of each parameter, people who are in charge of that. I still think it’s worth everybody putting in their best guesses, but having a few people who are really responsible for making the case.

One advantage to weighing the subject expert more is they just generally have the best context. You want to have the person who knows the most about something making decisions. I think a disadvantage to getting the subject expert weighing their opinion more is it will often penalize an intervention if that subject expert is generally just somebody with a skeptical prior. I think it’s important that they’re making the arguments and kind of laying out what the most important considerations are, but I also think it’s important that everybody has input onto what the value would be.

Robert Wiblin: So the problem is if you have different researchers looking into different numbers and one of them is just temperamentally very pessimistic, then that can kind of drive down the cost-effectiveness for the intervention that they happen to be studying. That’s just a random happenstance, really?

James Snowden: Yeah, so that would be something we’d really want to avoid and something that could happen if we just gave ownership and parlay to one person.

Robert Wiblin: So which of these numbers did you find hardest to estimate last November? Was there one that you were tearing your hair out about or having sleepless nights?

James Snowden: Right, yeah, I think we covered one already, which was the long-term effects of malaria on income. That was one. I’m trying to think of what another good example would be … I think the discount rate is a pretty interesting one. So the discount rate, that’s basically the percentage discount that we apply when benefits are realized in the future. So, for example, a 3% discount rate would mean that we’d value the benefits in one year’s time at about 97% as much as the benefits now.

I think there’s lots of different considerations that can go into what you think the discount rate would be. One would be if people get income earlier in life, then they can presumably invest that and gain some return on it. Another might be, some people might value helping people sooner, other people would just think that’s just a present bias and we shouldn’t take that into account.

So everybody’s taking slightly different things into account. We try to make that explicit when we’ve done that, but we still ended up at quite a wide range, so 3% to 6%, which doesn’t sound like a lot, but when you compound that over many years, it makes quite a big difference to how effective you think deworming might be. That’s something that I think we haven’t really reached consensus about.

When we’ve reviewed how other people think about discount rates, we’ve realized that nobody else has really got a good consensus on this either. I think that’s something that we’d like to look into more.

Robert Wiblin: Yeah, that discount rate is super tricky. I think we’ll have to have an episode about that at some point, because there’s so many different considerations that point in all kinds of different directions and I don’t really have a great view on what exactly it should be either. I think that episode could end up going for two or three hours.

James Snowden: Yeah, I think that would be. I’d listen to it. I think one thing I would highlight is that we generally … The discount rate, as it’s used in our model, is generally applied to increasing consumption, rather than to health benefits and that’s not actually an explicit decision we’ve made, it just happens to be that the charities we, I think, have this long-run impact, it tends to be income that we’re bundling in the long term, whereas with malaria, that tends to be something that happens if you hang a net this year, you’re probably going to prevent a case of malaria this year.

As it happens, yeah, the discount rate should be thought of as applying to raises in income, rather than health.

Robert Wiblin: So all of this was a bit of a preamble to discussing the sensitivity analysis that you did, where you took a range of the different estimates that the different staff members had and then saw how much that affected the outcome. Do you want to talk about that?

James Snowden: Yeah, so there are two broad ways you can do a sensitivity analysis. One is an all things considered sensitivity analysis and this might be a Monte Carlo simulation, so you put probability distributions over every parameter and then you randomly sample from those parameters somewhere along the probability distribution, like hundreds and hundreds of times. It kind of spits out a probability distribution in the end and this gives you some sense of how uncertain you are, all things considered, in your model. Although I would caveat that it doesn’t really account for, maybe, the most important kind of uncertainty, which is the structure of the model right? Are you even putting the right parameters in here?

I think that’s actually a big part of the uncertainty we have. So we were trying to achieve a slightly different thing with our sensitivity analysis, which was trying to work out which parameters are our models most sensitive to? That allows us to do two things, first it’s just communicate to donors why we’re uncertain, which I’ve always found is a far more compelling thing than just saying, no, we’re really uncertain.

It’s like, well, tell me why. What are the things that you’re not really sure about?

The second is allowing us to prioritize what are the things that we need to spend more time thinking about or doing research on. I think it’s most useful from that perspective, kind of thinking about what should we be spending time on. It turns out, there are a few things that our model is highly sensitive to. Two of them we covered already, the long-run income gains from malaria prevention, the discount rate. I think another one is fairly obvious and fairly controversial, which is the correct replicability adjustment to apply to a deworming study.

That’s probably the thing that GiveWell as a whole has spent more time thinking about over the course of its ten years, I would guess. We’re constantly trying to think of ways that we could get more information on this, although a lot of avenues that we’ve looked at just haven’t ended up being that promising. That’s another one. I think the obvious one is just it turns out which charities you think are most cost-effective, it’s just extremely sensitive to what your moral weights are on different kinds of good.

Our staff disagree on that, to a large extent and you can see, if you go to that blog post, we’ve got a chart where we look at if you flexed the value of doubling consumption for one person for one year, compared to preventing the death of an under five child and if you took that value and you flexed it between the person who thinks that’s highest, who’d be most pro-doubling consumption for the people versus the people who would be most pro-preventing the death of an under five child. You just get these huge bounds on how cost-effective our charities are, relative to each other.

Robert Wiblin: Yeah, let’s talk about some of these moral weights, as opposed to empirical weights. How do you go about estimating the value of these different benefits and harms. Do you look at surveys of what the recipients would value and how that would weigh it off, or do you just kind of have to decide yourself?

James Snowden: Yeah, it’s a good question. So actually, that’s something that we’re currently partnering with IDInsight to run exactly this survey. This is something we’re quite excited about. I think there’s a question, there’s a really high-level question about whose values should we be using. Make our best guesses. One, you might think the beneficiaries are the values we should care about. Secondly, maybe our staff are the best people to be making these decisions. Thirdly, our donors. Fourthly, maybe some idealized consensus of all people or what do other major prioritization organizations, how did the World Health Organization make this trade-off? That kind of thing.

At the moment we do it for our own staff, but trying to take into account these other various data points. We have asked our donors, we’ve surveyed a few of our donors and generally got the impression that most people just don’t have numbers in mind, which is just consistent with my understanding of how people think. Not many people are walking around with this figure in their mind of how they would value preventing a death compared to increasing somebody’s consumption.

This is not true for some of donors. Some of our really engaged donors, I really encourage them to go to our spreadsheet and just input their own values and see how that changes the answer. I think it’s quite unlikely that we’ll end up investing a lot in trying to elicit our donors’ weights, because we just don’t think most people have a good answer there. We have invested a lot in trying to understand how other organizations make these decisions. So there’s a few different ways you can think about this. The World Health Organization has these thresholds for what they consider a highly cost-effective intervention, in terms of dollars per DALY, so anything which is more than three times GDP per capita, per disability adjusted life year is not a cost-effective use of money in that country and anything which is between one and two would be considered pretty cost-effective.

There’s another question about what does that actually mean, pretty cost-effective versus not cost-effective? I’m not entirely sure how well, whether those thresholds are really used in practice because it just depends on your budget, I guess.

That’s one input. You can also look at stated preference surveys or revealed preference surveys. There’s a very famous study, I forget the name now, but it looked at different occupations and tried to come to an estimate of how much people were willing to pay to avoid a micromort, a very small chance of death. You could use that to think about how to trade off consumption.

Robert Wiblin: Against survival.

James Snowden: Against survival, yeah. One big problem we’ve seen is a lot of this literature is based on developed countries, and so that’s why we’re doing this work with IDInsight to go into an area of Kenya where we think the people there are quite likely to be, they’re relatively similar to our typical beneficiaries, although that’s obviously still a huge generalization. IDInsight are going in there and asking them questions about how they would make these trade offs, and we hope that that might be useful in thinking about what the people we’re trying to help actually value. I think one problem with this, because it’s pretty hard to ask most people these questions and particularly to try and get people thinking in a pretty consequentialist or utilitarian mindset. I think that’s something that’s likely to be quite challenging. We’ll see how that goes.

Robert Wiblin: Just to make it more concrete, you’re weighing up things like increasing someone’s income, versus the risks of them dying, versus the risk of maybe their child dying, versus perhaps direct suffering from disease. Those are the kind of things that you’re weighing up and trying to say, well, ten of this is worth two of that.

James Snowden: Yeah, that’s right. One of the most controversial ones would be, what’s the value of preventing a death at a particular age relative to preventing the death at another age? This is quite interesting, because I think the global health community generally has a way of thinking about this, which is a disability adjusted life year or the quality adjusted life year. We actually don’t use that anymore as a baseline for our own moral weights, and the reasoning for that is that it turned out that a lot of our staff actually value preventing the death of an adult more than they value preventing the death of a very young child. We really want our staff to say what they really believe, rather than just going along with whatever the normal position is. You can’t account for that within a disability adjusted life year framework, that preventing the death of an adult is more valuable than preventing the death of a child. You can discount each marginal year, but you can’t say that these extra years are essentially negative.

We stopped using that, and now we all have our own ethical systems that we use to prioritize lives at different ages. I can speak to my own one, which is very briefly based on the time-relative interest account, which is an ethical theory by Jeff McMahan. I think this allowed me, so Andreas Morgenson has a really nice paper on his website where he formalizes this, and the basic idea is that there are two things that are bad about dying. There are lots of things that are bad about dying, but one is how many years of life you lose and the other is how much interest you have in those future years of life. This helps account for my intuition that I don’t feel particularly strongly that preventing the death of a very, very newborn child might actually not be so much more valuable than preventing the death of an adult.

You take the number of expected years of life left and you multiply it by this pretty subjective factor, which accounts for, does this person have cognitive function? Can they make plans? Are they a functioning agent in the world? You multiply those two things together to get something that looks roughly, you end up with a roughly log normal distribution over age. The death of a very young child is something I would value relatively less, I think my peak value is the death of an eight year old, is where I end up. Other people have really different-

Robert Wiblin: That’s when it’s the worst for them to die.

James Snowden: That’s when it’s the worst on my values, yes.

Robert Wiblin: I actually don’t think that that explains people’s intuitions. I’d be more inclined to look at evolutionary psychology. I’ve seen a paper that says that if you survey people, coincidentally the age that they think it’s worst for someone to die is also the age at which it’s worst from the parent’s reproductive fitness point of view. If someone dies when they’re 16, then you’ve invested all these resources in raising this child to reproductive age that you can never get back, and then they die as soon as they might start being independent and start having children themselves. If you think about it from an evolutionary point of view, in terms of maximizing the propagation of your genes, at what age is it worst for our children to die, it happens to line up extremely well with when people morally think it’s worst for them to die, which makes me suspicious that it’s actually a moral philosophical issue that’s going on here rather than maybe more biological issues, actually.

James Snowden: I think that makes a lot of sense. I think there’s another thing you might point to which is back in caveman times, very, very young children actually had a lower remaining life expectancy than medium aged children. Again from a reproductive fitness point of view, once you nurture a child to the age of ten, say, they then matter to you more than a very, very young child. I do think it’s a good point, once you start to realize how much your intuition is influenced by not just this coincidence of evolutionary psychology but also this coincidence that doesn’t even apply anymore. That does make you question your intuition. This is something I’m struggling with at the moment. There’s another perspective on this that I find quite persuasive, which is say that you were to take the value of preventing a death as purely linearly associated with the expected years of life remaining. I think you do have this problem of, okay, when do you start counting, or when does life start?

Robert Wiblin: You get a sudden flip from 0 to 80 expected life at that point.

James Snowden: Well, exactly. That does seem weird, right? This discontinuity doesn’t seem like something that we’d want to have, in an ethical system. I don’t know if I’ve fully justified that in my own mind, but one advantage I think of Andreas’s formulation is you do get this kind of gentle, well, this quite steep slope up but still a slope. You might think a fetus starts to gain, at some point they start to gain some cognitive function, so they’re a little bit worth it, gain some value. You don’t get this big jump, which seems unlikely to be at birth and we haven’t really pinned down exactly, my understanding is we haven’t really pinned down exactly when that is.

Robert Wiblin: I guess on the total consequentialist view that I’m inclined towards, it starts even before birth. It could be valuable to create someone even before they exist, potentially, if their life would be good. That gets around the discontinuity but is intuitive in a different way.

James Snowden: I think that makes sense. It’s interesting, I think another thing that I’ve been struggling with, and this is all my own personal moral reflections, certainly not the GiveWell consensus view, is I find myself quite persuaded by a lot of the total utilitarian arguments. I think there’s a question here about, and I’m not sure if this even makes sense, but maybe using the right tool for the job. It feels like once we start making our moral weights on total utilitarian grounds, then everything gets a bit wacky.

Robert Wiblin: Yeah. Well, you’d have to change other parts of your model as well.

James Snowden: Yeah. It might just be that GiveWell’s thing isn’t … it’s just not based, our product isn’t based on a total utilitarian perspective. Even if my own personal views place quite a lot of credence on that, I wonder if it’s maybe not … I set that aside when I was thinking about what I should do from the perspective of GiveWell.

Robert Wiblin: Another controversial moral issue that you guys encounter is suicide versus involuntary death. Have you discussed that a whole lot? You were supporting the anti pesticide suicide group.

James Snowden: This is probably the most controversial of any of our moral weights.

Robert Wiblin: I know this issue is controversial with my friends whenever it comes up.

James Snowden: It’s really something that sparks people, and I think fairly, it’s a very difficult thing to talk about neutrally. Maybe if I give a bit of background about the organization that we ended up making a grant to, and that’ll maybe give a bit of perspective on which we can think about this moral question. We made a grant of about $1.3 million to the Center for Pesticide Suicide Prevention. I won’t go through the full evidence base, but basically the idea is they’re going to go into India and Nepal and work out which pesticides people are using to attempt suicide, and what the case fatality of those different pesticides are. For context as well, we think that about 800,000 people a year die from suicide, and somewhere between 100,000 and 300,000 of them are through pesticide. The data’s that bad, it’s quite hard to narrow down closer than that.

This is a very common method of suicide, particularly in agricultural communities in developing countries, and particularly in South Asia and Southeast Asia. The mechanism here, there’s two possible mechanisms. One is that you’re making a common and lethal form of suicide more difficult to access, and that might reduce the amount that people have this very easily available method of suicide in front of them. And then the other method is, the other possible mechanism, we’re not really sure which of these is more likely, the other possible mechanism is people are still attempting suicide but they’re doing it with less toxic pesticides, or pesticides which are less likely to kill them. They generally have a higher chance of getting through that period and go on to make a full recovery.

I think there’s a big thorny argument here about how valuable it is to prevent a suicide. You’re not treating these people, you’re not treating mental health disorders, you’re essentially just preventing them dying. I think this is an interesting one, because two of our staff place zero weight on preventing a suicide through means restriction. I think there is a decent argument there. A lot of it depends on what your definition is of a life worth living, and I’m not sure anyone has a really good answer for that. There is also a lot of empirical information which is relevant, so I think this highlights the nature of these conversations. A lot of times it’s going to be dominated by your ethical view, but other times it really does seem like, there’s actually evidence we can go out into the world and find out which should change our minds.

Robert Wiblin: Something I’m very interesting to know is when you have these conversations about these moral issues, can you find dialectically persuasive arguments? Are there things that you can say that move other people, or is everyone entrenched in their own moral intuitions?

James Snowden: It’s a good question. I moved around a bit on this, and a few other staff did as well. My recollection is that most staff stayed pretty entrenched. Having said that, I think that is, we only ask for people to put down their views after we’ve had that initial discussion where we present the evidence. It’s not necessarily that people aren’t responding at all to the evidence, it might just be that they’re not changing their minds about new evidence. There’s a reason we do that, which is that I think once people write a number down it’s just human nature to feel quite defensive about that number. Things generally seem to go better if you present the evidence first and then think about it.

Robert Wiblin: Do people think about the effects of a suicide on the children or the family of the person who dies?

James Snowden: Some of our staff did take that into account, it actually wasn’t a key consideration for me. Maybe that’s wrong. I guess my perspective is generally that I tend to value these things because of the effect on the person, and I wonder if the sense of grief that family members might feel, a lot of that grief comes from a place of, I am sad because a bad thing has happened to somebody I care about.

Robert Wiblin: I was thinking more from the perspective of if a parent dies, then the children might grow up in more dire poverty.

James Snowden: Yeah, right, of course. That’s relevant. I wonder. That’s not something that played a key role in my decision, but I guess that consideration should lead me to retroactively move it up a little bit.

What we were doing was comparing it to preventing the death of an adult from malaria, because they have quite similar age bounds. The odds we got was about 70%, so it feels like these considerations might apply to both.

Robert Wiblin: Right. Perhaps it cancels out.

James Snowden: Yeah, but then you might say, well okay, if this consideration applies equally to both, then that would lead you to raise the percentage that you’re assigning to the value of preventing a suicide relative to preventing a death from malaria, because they apply equally to both so you’re not discounting that. And then I guess maybe there’s an argument about, well, you might think that people who attempt suicide and then go on to survive because they haven’t taken the right pesticide might not be great providers for their family. I don’t know this, actually. This is speculation. But they might generally be people who are likely to earn less later in life, say. And so then you might say, well, actually, that consideration should also be discounted, and I’m not sure whether that should be discounted more or less than other things that you would think about in making that comparison.

Robert Wiblin: One thing I noticed looking at that spreadsheet is that most of the numbers that you’re trying to estimate are about estimating the effect of the intervention on the recipients who directly receive it, like the people who get the deworming tablets or get the bed nets to protect them from malaria. There’s perhaps less attempt in the model to estimate what effect these different interventions might have on the development of a country over 50, 100, 200 years. Is that something that you’d like to estimate more, or would you challenge my perception of it?

James Snowden: I think that’s right. I think there are two ways that you can think about these long run flow through effects. One is in terms of how far they go into the future, and the other is how easy are they to estimate. We do tend to, we try to estimate into the future but still relatively direct effects. For example, our deworming charities, these kids are getting dewormed when they’re young, but it’s not till later that we expect that they might see a rise in income. Those effects are over multiple years.

But you are right that we aren’t thinking maybe as explicitly as we should, although that’s controversial as well, about what this effect might have on the long term development of the country or something. I think the simple reason for that is, these things are really hard, and we don’t think we have strong evidence about what that might be. There’s a huge literature on this work, and maybe we should prioritize it into that, but it doesn’t feel like … at least in the short term, it’s likely to change our views hugely.

I think an assumption that’s slightly built in to our model is … Well, I’m not sure if it’s an assumption, because I’m not sure every staff member would agree with this, but I think something that would make, would push you in the direction of GiveWell charities over maybe other options would be if you think, there’s this kind of trade off where we think that the short term effects of an intervention, which we might be able to make a pretty well evidenced guess about, are a pretty decent proxy for the overall effectiveness of an intervention. I think that certainly doesn’t apply in all cases, but when you’re thinking about these very difficult to estimate effects, you’re making this trade off here between having to make judgment calls based on very limited evidence, versus potentially the kind of long run effects being the most important thing. And so we see our role largely as using these short term effects as a proxy for what we think might be best in the long run.

Robert Wiblin: I’ll stick up a link to that blog post, and of course the enormous spreadsheet where you all offer your views and you show the mechanics of how you estimate the cost effectiveness of all these different organizations, so people can take a view and potentially have a play with the numbers themselves.

What would you say are the biggest research priorities for GiveWell, as a whole, at the moment?

James Snowden: Sure, yeah, I guess there’s four main categories of goals that we have for our research. At a high level, so the first is maintaining our current research project, so stay on top of our current top charities and incubation grantees and making prioritization decisions between them for our end-of-year giving season recommendations. That’s quite a big part of what we spend time on, so most of us … That’s actually the part that I’m less involved in, but a lot of researchers at GiveWell work on that. As we’ve added new top charities, that becomes a bigger and bigger task.

The second is prioritizing interventions we haven’t looked at yet to find new priority programs. The third, and this slightly overlaps with the second, is maybe building for future years. As things aren’t likely to lead to top charities in the next couple years, but could have a really big impact on GiveWell’s recommendations over longer time scales. That’s a lot of our incubation grant work.

The fourth is kind of cross-cutting, so improving the tools we use, improving the methodology. I’ve done a bit of work on this, so kind of conceptual questions related to our cost-effectiveness analysis and improving the way that we think about how we might weigh, for example, particular core list of factors into our decisions and trying to do that in a consistent way.

Robert Wiblin: So GiveWell doesn’t really do global cause prioritization, weighing up poverty reduction versus other causes. I think that’s basically gone over to Open Philanthropy, which has broken off, but you still do prioritization between different sub-causes, even though you’re not prioritizing between all the different causes, there’s still going to be a lot of prioritization work between different subareas within poverty reduction and global health. So you’re thinking, do we want to lean on improving life expectancy or quality of life or improving income or improving education. There’s still some sub-cause prioritization to do, right?

James Snowden: Yeah, I think that’s right. So that’s our main area of expertise and what we do. We’re not working on the very high level different types of causes, like animal welfare or global catastrophic risks, because we think there are other really good organizations working on that, so Open Phil is one and some of the folks at Center for Effective Altruism, as well. Our work tends to be about prioritizing at a lower level, so within global health and development, what are the best giving opportunities we can find.

We do, periodically, reconsider whether we should be working outside global health and development. We don’t really have any plans to do that in the near term. I think in terms of different … The way that we prioritize within global health and development is we tend to focus on interventions rather than particular kinds of causes. So what specific things can we do? How cost effective is a particular kind of program or a solution, rather than focusing on the problems.

I think there are two reasons that’s a good way of splitting up the case. One is that we think that the different kinds of interventions, there’s a lot of variance between them in terms of their cost-effectiveness. The second is it’s relatively easy to evaluate the differences between those types of interventions. Whereas, when you’re focusing on a particular disease, there might be very different kinds of interventions there and it really does bottom down to what’s the most effective intervention you can do to prevent that problem.

For our direct work, that’s the way we tend to prioritize at the intervention level. So between bed net distribution, deworming and anti-retroviral therapy, rather than at the problem level. So between malaria, worms and HIV. It’s just a lot of information out there that lets us get a handle on which interventions are likely to be cost-effective and you can see these big differences between them.

I think prioritizing at the problem level is a more difficult question. It really bottoms out at how cost-effective the best interventions are. That’s a good place to start, by identifying priority interventions for us and then finding charities doing those things that meet our standards of transparency and monitoring evaluation. And we’re also working in the right geographic locations, because anti-malarial bed net distribution wouldn’t be very cost-effective in the UK, say.

So I think, yeah, there are lots of different ways of subdividing the space. You might go higher than the intervention level. You might say, well, one thing we could think about is policy interventions versus direct interventions.

You could have a heuristic that said that one of those is much better and so you should focus on those to have the most impact. But I think that question is really hard to answer in abstract level. Our strategy here, really, is to look at some really promising interventions you could do in the policy space that we think might be very cost-effective. So we’re thinking about tobacco control or lead regulation and then do our best … This is very difficult, but do our best to compare them with our top charities. You’re comparing things of very different kinds, but I do think it’s possible to do that.

Robert Wiblin: One of the most common questions I hear about GiveWell is how you think about the long-term effects of these different interventions. So you just mentioned lead reduction and smoking cessation. A lot of people would think, well, the lead one might be much more important than it seems, because it kind of improves people’s education in the long run or makes them more intelligent, because they don’t have lead poisoning, basically.

Is there any way that you can take into account these kind of long-term knock-on effects from the interventions that you’re funding?

James Snowden: Yeah, it’s a good question. So first I’ll distinguish between two kinds of knock-on effects, so those that happen in the long-term, which you might expect lead’s effect on IQ and therefore income would be quite a long-term thing, or the effect of deworming on income is relatively long-term. That’s one thing that people could call a knock-on effect. Then there’s this other dimension, which is correlated with it, but which is just, how difficult is it to assess these effects? Obviously, it’s harder to predict things in the long run, but these are slightly different questions, I think.

An example of the first would be, as I said, the effect of deworming on later in life income, whereas an example of the second might be something like the effects of increased voter empowerment on long-term democratic institutions and how that might affect economic institutions and the long-run growth in countries. Those second questions are really hard to answer well, which is not to say they’re not worth answering well, but that’s generally not our focus.

Robert Wiblin: So what have been the biggest changes in GiveWell’s advice over the last couple of years? As a casual observer, it feels like things have become fairly stable, like your top few charities have typically been the same for a while now.

James Snowden: Yeah, that’s a good question. I think the first thing to note is that we’ve added quite a few top charities to our list in the last two years. A couple of years ago we added Malaria Consortium’s Seasonal Malaria Chemoprevention Program and then SightSavers and The END Fund are relatively new additions, they both work on deworming and then this year, well, the end of last year, we added No Lean Season and then Helen Keller International vitamin A program. So we are adding charities to our list.

I think to the casual or even quite attendant observer, that’s been slightly obscured by our recommendations to donors being pretty similar each year. I’d say to a large extent, that’s kind of a coincidence. Our recommendation has usually been along the lines of, donate to AMF.

In fact, we ranked Malaria Consortium, No Lean Season and Helen Keller International’s funding gaps above AMF at the end of last year. The reason we don’t recommend that to donors is because we get a large grant for disbursement from Good Ventures each year and we use that to fill the highest priority funding gaps first. Last year we filled those gaps and then recommend the next best funding gaps on our website, which means a lot of donors are probably thinking nothing’s changed, which isn’t quite right.

Robert Wiblin: So you found things that were better, but then you funded them from the grant from Good Ventures, so what’s left is the large pool of money that Against Malaria Foundation needs?

James Snowden: Yeah. That’s basically right, yeah.

Robert Wiblin: And that’s where you ask the broader audience to fund?

James Snowden: Yes. That wasn’t a decision on thinking … It’s largely coincidence it’s turned out that way.

Robert Wiblin: So I guess that’s really good news then, in the sense that you found several charities that were better than what you were recommending before.

James Snowden: Yeah, I think so. I think the charities we’ve added are really exciting. I think there’s a sense in which you can still say that we’re answering these narrow questions. So with the exception of No Lean Season, they’re all organizations doing fairly direct health interventions with established evidence bases. I do think there’s space for us to grow, in terms of looking wider than what we’ve done before and thinking about very different opportunities. So one good reason for this is how we allocate our research capacity, given that we’re funding constrained.

I think just to simplify, there are two ways we could allocate our research capacities. The first is making sure all organizations that fit our current criteria are recommended and we might expect them to be similarly cost-effective to our current top charities. Where the second would be maybe doing more speculative research into areas that don’t necessarily fit perfectly with our current criteria, but might be just substantially more cost-effective than our current charities.

I think when you’re funding constrained, that really pushes towards the second, because the first would just be displacing funds from similar opportunities. I think there’s this question that we’re asking ourselves. There’s two roles for GiveWell, one is as an objective recommender of really great charities. I think it’s really important there that your recommendations aren’t cause dependent. But the second is as a kind of crowdsource foundation, so trying to do as much good as possible with the money that we get. That influences how you spend your research capacity, because from the second perspective, there’s just not a lot of value in identifying more funding opportunities that look like the Against Malaria Foundation on the margin.

Robert Wiblin: So just to explain that, you’re saying the Against Malaria Foundation, hypothetically, could absorb hundreds of millions of dollars of donations a year and you’re only moving a hundred million dollars a year, then there’s not really much point in finding even more capacity in another organization that’s equivalently as good, because you’re just not going … You can’t even fill the current bucket that you have of that kind. So instead you want to find something that’s more cost-effective than the Against Malaria Foundation, maybe even if it’s a bit more of a risky bet?

James Snowden: Yeah. I think that’s right. So just to be clear, I don’t think the Against Malaria Foundation could absorb hundreds of millions of dollars a year, but I think, yeah, that’s the principal I’m gesturing at.

Robert Wiblin: So if GiveWell’s advice were going to end up being very different in a couple years’ time, how could you see that happening and what organizations might end up on that list?

James Snowden: Yeah, that’s a great question, I guess that’s my job. I think there’s a number of ways that could happen at a high level, so first is just identifying new charities that look similar to our current top charities, either if we’re finding new organizations that we haven’t been able to get to apply before or maybe some of our incubation grants resulting in top charities.

But maybe that’s not what you mean by very different. At some point, we’re going to have to end up taking top charities off our list, if we continue finding really good opportunities. Either because we think they don’t have a lot for funding or because we’ve just found other better things. Or maybe because we’ve changed our minds about a key part of the open space. I think in terms of charities that are very qualitatively different to our current top charities, I think the policy space … My instinct is that the policy space is the most likely place that that’s going to come from.

Robert Wiblin: Okay, so potentially advocacy to change aid budgets or something like that?

James Snowden: Yeah, it could be something like that. There’s lots of different areas that you could think about in policy. If you put it under the broad bucket of influencing what domestic governments or international organizations do, so one way you could divide up the space is … On the one hand, you have quite a lot of health regulation, things we take for granted in developed countries, say things like laws to prevent the use of lead in paint or regulation of particular methods of suicide, tobacco control. Those are the kinds of things that only governments can really do. That’s one area that we think could be very promising.

Another is this, more evidence-based policy work I’ve seen. The kind of work that a lot of academics are doing, so your JPALs of the world, your IPAs. Going out and trying to improve the way that governments use evidence. I think there’s tons of other things that you could think about in this space. Those are the two that we’re starting with and seeing where we can get with that.

Robert Wiblin: So the new charities that you added in the last few years were No Lean Season, SightSavers, Helen Keller International and Malaria Consortium. Is that right?

James Snowden: Yes, that’s right and also The END Fund. Yeah, in the last few years.

Robert Wiblin: Do you want to just briefly describe what those do and why they ended up being recommended and, perhaps, even regarded as more cost-effective than AMF?

James Snowden: Sure, so Malaria Consortium’s Seasonal Malaria Chemoprevention program. So basically what this is, is in some areas of sub-Saharan Africa, a high portion of malaria occurs during the high-transmission season and in that period it’s a pretty good idea to, basically, give children intermittent preventative treatment for malaria. You see these very large reductions in the incidence of malaria from doing this. That’s a program that’s relatively new. I think it was only a few years ago that the global fund started spending money on this, so we picked it up quite early on. There were a number of randomized control trials and a Cochrane Review published quite early, so we’ve recommended Malaria Consortium’s Seasonal Malaria Chemoprevention program now for two years.

We think it’s roughly at similar levels of cost-effectiveness to Against Malaria Foundation, which is interesting because I think a lot of that is driven by, or at least partially driven by SMC is a program which has received relatively little funding compared to insecticide-treated nets. So the countries that Malaria Consortium is working in tend to be higher prevalence countries than Against Malaria Foundation. So even though, in the abstract, in most countries, it depends on the various transmission mechanisms, in most countries you might prioritize nets above SMC. In the countries where there are funding gaps on the margin that they look pretty similar.

Robert Wiblin: Interesting, okay. So SMC is kind of, maybe, less cost-effective all else equal, but it’s more neglected, so there’s more opportunities to do it in places with high malaria prevalence?

James Snowden: Yeah, that’s our best guess. There is nuance around this, so where a particularly high proportion of malaria occurs during the high transmission season, that would be a time that those places would be places where SMC could have a particularly large impact.

Robert Wiblin: Yeah, what are some of the others?

James Snowden: Yep, so No Lean Season is an organization that was an incubation grantee and basically they provide seasonal migration incentives to people living in rural Bangladesh. So during the lean season, it’s not a great time for crops and people who migrate to the city tend to earn a lot more, so this is a program that’s run by Evidence Action, who also run Deworm the World, which is one of our top charities. They have run a bunch of randomized controlled trials showing that people who were given this incentive to migrate are more likely to migrate and therefore have higher consumption, and that outweighs the cost of providing this incentive.

Robert Wiblin: So No Lean Season buys people a return bus ticket, is that right, to their city to make money there and finding a job in the city while they can’t do farming, and then they come back on the bus and do farming during the season when agriculture is more productive?

James Snowden: Yep. That’s right. I think there are a number of big questions related to this grant and it’s something that I should say again, I’m not the researcher’s best place to talk about this, but I can kind of give you the overview. So one of the big questions is why is it that people aren’t doing this anyway.

Robert Wiblin: Right, yeah, it’s quite an odd intervention, in a way.

James Snowden: Right, so I think there’s … The evidence about the mechanism isn’t completely clear, but if we were to take the most skeptical view, it would be, well, say that we’re … everybody is homo economicus and we’re modeling the world as an economist, a classical economist would. There’s a very obvious reason why people wouldn’t migrate to the city, it’s just their revealed preference. There’s such disutility from living in these horrible cities and it’s just not worth it for them.

If that was the case, then No Lean Season wouldn’t be a great intervention, because you’re incentivizing people to do something that they don’t really want to do, but they’re doing it for that extra money. That’s possible, but I think that’s quite unlikely. So one of the adjustments we include in our cost-effectiveness model now is kind of subjective adjustment for how much worse you think life would be in a city and different staff have very different views about that.

I think there’s another thing there, which is, in my experience at least, people don’t tend to be homo economicus and providing this relatively simple behavioral intervention can just change the way that people think about things. There are various data sources. I’d encourage staff, encourage donors, to take a look at our intervention report there, because we examine it in more detail there than I’d be able to give on this podcast.

Robert Wiblin: Okay. Sure. Do you feel able to speak to any of the other new charities?

James Snowden: Yeah, so I think SightSavers and END Fund. The rationale there was basically we wanted to identify more room for more funding in deworming, which is one of our priority programs, specifically combination deworming for schistosomiasis and soil-transmitted helminthiasis. Those are organizations which were doing that. So they were recommended very much on that basis and the cases very similar to the case for SCI and Deworm the World.

Helen Keller International is a vitamin A program. Helen Keller International are an organization that we’ve been trying to get to apply for a while and we were trying to find a good fit and a program that we could evaluate there. They provide vitamin A supplementation to kids in areas where it’s quite likely that people suffer from vitamin A deficiency. We expect that that has quite a high impact on cause mortality and might also have an effect on the developmental effects of very young children.

Robert Wiblin: So there’s actually been quite a bit of turnover, quite a bit of discovering of new organizations that seem really, really cost-effective.

James Snowden: Yeah. I think that’s fair, yeah. We’ve been quite excited because I think for a long time, GiveWell hadn’t found a new top charity and now we seem to be identifying them with a bit more regularity.

Robert Wiblin: I was worried that perhaps as a researcher at GiveWell, it felt now like you were just tweaking small aspects of the model for the charities that you already understood, but it sounds like there’s reasonably good opportunities to look into new areas and have big new insights.

James Snowden: Yeah, I think so. We do feel we are at a point where it is worth expanding what we’re looking at, but at the same time, there are constantly new organizations coming up and these aren’t actually new organizations. I think a lot of the value here has been in as we’ve grown bigger and we’re able to direct more funds, there’s more incentive for established, big organizations to go through our process. So we’ve had quite a lot of success there with Helen Keller International being an example.

Robert Wiblin: Are there any charities that you, personally, think are really cost-effective that GiveWell hasn’t funded yet?

James Snowden: Yeah, it’s a good question. First of all, if I think that there’s a charity that has potential to be really, really cost-effective, that’s likely to be the thing that I’m working on at the moment. These are mostly just things that we’ve touched on already, during this conversation, which I’m particularly excited by. They’re all in this influencing policy space.

The first is an incubation grant we’ve already made to the Centre for Pesticide Suicide Prevention, so we don’t recommend them as a top charity yet, because it’s a very new organization. So this is the first significant funding that they’ve received, which was quite exciting, because this is us starting a new organization from scratch. So we made an incubation grant there and I’m happy to go into more details on that later, it’s a long story.

The second area that I’m excited about is tobacco control. Lots and lots of people die from smoking every year. The big question for us is if those require funding. So Bloomberg and Gates are already doing this in 20 of the highest priority countries. So it really depends on what that looks like on the margin.

The third is lead paint regulation. There’s pretty compelling evidence that lead paint is still one of the primary sources of lead exposure in the US. It seems likely that it could be a big problem in developing countries as well, though the evidence there is thinner. There’s also pretty compelling evidence that lead exposure leads to lower IQs, so I think that could be a really cost-effective thing to do, to fund, if it’s done right. So basically going in, surveying paint manufacturers or surveying paint that’s sitting on shelves in countries and working out how much lead they have in there and then advocating for either new regulations or enforcement of current regulations.

The fourth is this kind of evidence-based policy, so various groups like J-PAL, Innovations for Poverty Action, IDInsight, CEGA, Building State Capabilities and Evidence for Policy Design at Harvard and many more, which are trying to improve the use of evidence in policy decisions in developing countries. I’m pretty uncertain about that one, because it’s hard to evaluate the causal stories behind their influence, but it is an area that I’m excited about and I want to spend more time on.

So beyond those, and those are all things I’m working on at the moment, I’d want to think about things that are really hard to evaluate but might have a huge impact, so thinking about improving macro-economic policy or democratic institutions could be really good bets in the long term. So obviously, they’re really hard to assess and I think reasonable people disagree hugely about what the best approaches are there. I think our current strategy is to really start from where we are, which is these relatively direct interventions, and then move outwards and see how far our approach can take us. But I do certainly believe there’s a decent argument to be made for some of those areas at a very high level being even better than our top charities.

Robert Wiblin: Do you mean macro-economic policy in the developing world?

James Snowden: Yeah, there’s a bunch of different ways you could think about this, so I’m not even sure macro-economic is quite the right term. So one might be tax reform in developing countries, which leads to sustainable financing of programs in the long run. Another one might be reducing agricultural subsidies in developed countries. All of these areas that you read about and have been written about from a big picture development perspective for a long time, but we haven’t spent that much time evaluating ourselves.

Robert Wiblin: Let’s move on now to talking about another blog post that you published just a few days ago, actually, about leverage and funging in the charities that you recommend. Could you describe what those two things are, and why they matter to your recommendations?

James Snowden: Sure. Leverage and funging are two jargon-y words we use to describe how a charity causes other actors to spend their money in a way differently than they would have otherwise. Leverage is when a charity gets other actors to spend more money on the particular intervention than they otherwise would have, and funging is the opposite of that, it’s the flip side, it’s when you think you might be crowding out other actors from funding that intervention. They’re two sides of the same coin.

I think these are important because there’s a lot of potential additional value there. If you’re causing other actors to spend their funds on something more valuable than they otherwise would have, that should be a good thing, and we think that’s something that charities should be rewarded for. On the flip side, if we thought that without the Against Malaria Foundation these anti-malarial bed nets would be fully distributed by the country governments, that seems like something we’d want to penalize AMF for.

Robert Wiblin: Okay, so in one case if you spend money delivering part of an intervention then you might get the government to match it and do the other half. In the opposite case it’s possible that if you go in and do something, then that means that the government just decides, well, they don’t have to do this so they can spend the money elsewhere, or maybe just give people tax cuts, and so the money you’re spending could be better offset by someone else withdrawing their funding.

James Snowden: Yeah, that’s exactly right.

Robert Wiblin: How did you deal with that issue in the past, and how has that shifted recently?

James Snowden: In the past we counted all funds the same, so we included all the costs of the intervention on the cost side, no matter who pays them. A concrete example is our deworming program. For example, for schistosomiasis control initiative, SCI, one of our top charities, we estimate about one third of the costs are paid by SCI and the rest of those are borne by either drugs donated by pharmaceutical companies or the government costs, which are basically comprised of the time teachers spend distributing the drugs. What we used to do is just count all these costs as the same, and try to work out the cost-effectiveness of that entire group of funders. Which I think is a reasonable simplifying assumption, but it’s maybe not the more accurate thing we’re trying to do, because really what we’re trying to get at is how cost-effective is it to donate to this particular charity, rather than that group as a whole.

Robert Wiblin: I guess one reason to not really consider the leverage and funging is that it would give people the ability to trick you by, say, taking a bunch of donations and then committing them to do half of it, and saying well you’re just going to fund the second half that will allow the first half to function, and they can make your fraction of it look twice as cost-effective.

James Snowden: Yeah, this is a really big question I think, which is broadly something that’s very difficult to think about conceptually. You can think about this for multiple things, how we’re going to draw the circles about what the intervention is, and so at an extreme example you might say well, the intervention isn’t distributing these nets. The intervention is the whole world, running all these randomized controlled trials, developing the nets, and then you should include all of these costs on the cost side. But I feel like there’s some kind of happy medium you want to get to, which is say, well, this seems a bit absurd, we know these RCTs exist in the world and it seems unlikely, we’re not going to go back in time and those costs would have been spent differently. At some point you do have to think about, well, how is the world as it currently is? And I think it’s this difficult balancing act that you’ve got to get, and I don’t think there’s necessarily one strict correct answer. But it is at least something you want to consider.

Robert Wiblin: Now you’re thinking about the extra money that’s being brought in, and I guess you then also have to think about how that money would have been spent otherwise. That sounds pretty challenging to estimate both of those things. How do you actually do that?

James Snowden: It’s hard. This is one of the reasons we were reluctant to include this in our cost-effectiveness model, because it is something that requires a lot more guesswork than maybe some of the more objective things that we estimate. On the other hand, we do think it is really useful to include these things in our model, because we don’t just want to know whether it matters, which would be what we’d be doing if we were taking it into account qualitatively. Ideally we’d want to know exactly how much that is, or at least get a decent proxy for that.

I think it is really challenging. There are two parts to this, so one is what’s the percentage chance that these distributions, say in the case of bed nets, wouldn’t have been funded otherwise and the other question is what would the money have been spent on.

With nets, we actually have a pretty decent answer to what’s the chance that these distributions wouldn’t have been funded otherwise, and so we’ve looked at the distributions that Against Malaria Foundation have considered and then didn’t end up funding for various reasons. We looked back at those distributions and were like, okay, well did they end up happening? And so what we found is about 40% of them did. We’re guessing that there’s about a 40% chance that the Against Malaria Foundation is funging with other actors.

Robert Wiblin: Although if that happens, isn’t it the case that those other actors, in the case where AMF funded the distribution, they then have more money to fund some other distribution elsewhere that you’re not seeing?

James Snowden: I think that’s taken into account in our estimate of what the counterfactual value of those funds is.

Robert Wiblin: Right, okay, yeah.

James Snowden: We aren’t answering this perfectly, because ultimately what we’d want to be doing is estimating our general equilibrium effects. Every time we shift funds to another thing, those funds also shift other funds and so on, and so on, and so on. That’s pretty difficult to do. We think we’re getting a decent proxy of this, one level down the causal chain.

I think on a specific AMF question, if we fund the distribution and that means the global fund doesn’t, does that mean they fund another distribution? Maybe. We did take that into account when we were trying to estimate the counterfactual value of global fund money. But at the least in principle, if everyone’s perfect prioritizers which is not the case, it’d mean they funded a distribution lower down in the lower prevalence area instead. We still think it would be marginally less cost effective.

And I think in practice, global fund tends to set budgets at a country level for malaria, so it’s quite likely that if the bed net funding gap was filled they’d shift down to lower priority interventions in that country rather than provide bed net funding in a different country. And of course you can say in the long term all these things shift around in quite a nebulous way. So it is very difficult to come up with precise estimates.

Robert Wiblin: When you started taking into account leverage and funging in this way, how did it shift around the relative cost effectiveness of the different charities that you’re looking at?

James Snowden: Good question. I don’t have the precise numbers right in front of me, but it did make our deworming charities look quite a bit better.

Robert Wiblin: They’re getting so much matching funding effectively from the pharmaceutical companies and the government to do the distribution.

James Snowden: Exactly, yeah. That’s right. They have a high proportion of funding from other funders. I think another high level thing was, this was part of the thinking for why I started to think about why people at GiveWell start to think that maybe these policy interventions might be a good option. If you were to, again it’s a question of how you draw the circles around the intervention. Say that you were getting … Somebody could advocate that it cost a million dollars for a charity to advocate for a billion dollars on something twice as effective. What you could do is you could put the million dollars on the cost side, and then put the billion dollars on the cost side, and then your thing is looking 2X good, relative to what that money was being spent on otherwise. But if you think, what we’re putting on the cost side is just the one million dollars and then we’re getting that billion dollars to be spent twice as effectively, then that looks like one of the best things you could possibly do.

That influences how we think about those things. I think it didn’t have, we saw a small penalization to Against Malaria Foundation because we think it has a pretty high chance of funging, and then most of our other charities remained pretty similar. Helen Keller International, GiveDirectly and Seasonal Malaria Chemo Prevention, they’re not leveraging huge amounts of government funding and we also think it’s quite unlikely that they’re funging, they’re crowding out government spending.

Robert Wiblin: One thing that surprised me in that post was that it seemed like you thought that counterfactual government spending in the countries where your charities operate would be, it’s 75% as cost effective as GiveDirectly, which gives cash transfers to extremely poor people. That seems like a fairly optimistic estimate to me of how effectively government spends money in countries like Kenya or India. Was there much controversy about that?

James Snowden: Yes, it was interesting. I don’t think we’ve discussed this enough internally, actually. We’re pretty uncertain about this, so I’d appreciate your thoughts. I’d also appreciate other people trying to do this kind of analysis and come up with their own figure. I can describe how we came up with that estimate, and then maybe give a couple of pushbacks about why that might not be overly optimistic.

We came up with our estimate by looking through domestic government spending on different social, education and health programs, and then putting down our best guesses at how cost-effective we thought each of those things were. This is obviously extremely difficult because we haven’t looked in depth at a lot of these programs. But we do have, we’ve built up quite a bit of knowledge of various big parts of particularly health spending, and so we can come up with estimates which are better than made up, but aren’t precise.

We’re taking that as a working guess, and I would love to spend more time on that for other people to come up with their valuations.

Robert Wiblin: The way that it’s surprising, perhaps, is that in the past GiveWell have said that GiveDirectly is a really outstanding charity that’s spending money extremely effectively relative to many other charities. But then it seems according to this estimate that it’s 33% more cost effective than marginal general extra government spending. In that context it seems a little bit surprising. It would suggest that GiveDirectly is not that much of a standout charity relative to other things that people do in the developing world.

James Snowden: It’s a good point. I think maybe my first pushback would be that Against Malaria Foundation, according to our cost-effectiveness estimates, we estimate it’s about four times as good as GiveDirectly. So that’d be about five times better than marginal government funds, for example. We use GiveDirectly as this kind of baseline. And actually a lot of our rationale for GiveDirectly and recommending them as an organization is they score so highly on our other criteria, which aren’t built in to our cost-effectiveness analysis. Their monitoring and transparency is superb, and the relative simplicity of their intervention also means that from a model uncertainty perspective we think there are just less things that could go wrong.

We don’t have a precise estimate for how much better this makes, than the cost-effectiveness analysis, makes them look, but we do think they are … I have to be careful here because I think different people at GiveWell might interpret this actually differently. But I think most of us would say, we would prefer that four dollars went to GiveDirectly than one dollar went to AMF. This is a sense in which we’re not taking our cost-effectiveness analysis completely literally.

I think that’s one point. And then the other point is, we’re not just talking about general government spending. We’re talking about government spending that otherwise would have gone to bed net distribution or deworming or whatever they were doing. And so it seems reasonably likely that money earmarked for that kind of activity would otherwise have been spent on quite good things, so I think it would be better than the average government spending. And that’s one reason we’ve looked at social, health and education programs rather than for example military spending.

Robert Wiblin: Yeah, that makes a lot more sense now.

James Snowden: Yeah. I’m not, I wouldn’t want to defend this too strongly. It’s not something I feel confident in, and it’s something I think we could do a lot more work on. But then even if we do do more work on it, are we really going to feel that much more confident? It’s pretty uncertain.

Robert Wiblin: It seems like honing this leverage and funging issue hasn’t changed the ranking all that much, so maybe it’s not something that deserves a ton more attention.

James Snowden: Yeah. I think that’s a question we’re always thinking, right? We prioritize decision relevant information over mathematical elegance, and so it seems unlikely to me that us spending a ton more time on this is going to change things hugely. I think where it could change things quite a lot is when you’re thinking about these policy interventions. They are a relatively high proportion of the funding, so say you’re advocating for regulating certain pesticides. You’re spending relatively little on the advocacy there, but it’s going to cost the government some money to test particular pesticides in labs, to roll out this regulation, to enforce it. There a relatively higher proportion of the spending is government spending, and so this would be a more important question.

It’s one of those things where we end up in a difficult position, where we think this is kind of important but we don’t really see a path to getting a better sense of it. I think the main takeaway is the world is incredibly complicated and we do the best we can, but we should really be very modest in our beliefs.

Robert Wiblin: Let’s move on to the policy oriented interventions that you’ve been looking at. What kinds of policy change are you guys investigating, and what are the main challenges there?

James Snowden: As I mentioned earlier, there’s two broad buckets that we’re starting with. One is this advocacy and technical assistance to impose particular health regulations, which can only really be done by governments, so we’re thinking about tobacco control, lead paint regulation, pesticide regulation, we’re going to be looking at road traffic safety at some point. This is something that government’s uniquely well placed to do.

And then the other is this evidence based policy bucket, which is saying, what can we do to improve the use of evidence in governments, and the question there is can we look at case studies of when we think one of these organizations has gone into a government and changed a decision they made? That’s the approach we’re taking there. Say they’re the two broad buckets that we have.

Robert Wiblin: How do you figure out whether this advocacy is likely to succeed?

James Snowden: It’s a good question. There’s a bunch of different ways. Thinking about, I don’t want to get too much into the specifics of grants that we’re still thinking about, but maybe if I talk in broad terms about what kind of research this involves. So we might think, well, okay, here is this change that was made in government policy in India, say, a couple of years ago, and we’ll ask the charity, do you think that was a lot of your value was from influencing this change? Then we have to piece together, it’s interesting, it’s a bit more like investigative journalism than maybe the traditional work we’re doing. We’re just trying to piece together from different sources, okay, what are the other causal strands that could have played a role here, how important was this actor relative to other actors, and sometimes it’s very difficult to tease these apart.

But at least some narratives we can kind of, they seem a lot more plausible to us. And then we can test those narratives by talking to other people who were involved, and think, well, is there a consistent story here that does seem to draw a plausible causal pathway between the actions of this charity and this particular policy change coming into effect? And then the other part of that of course is, how good was this policy? And that looks a lot more like our traditional work.

Robert Wiblin: Yeah. Are there any particular policy changes that you’d like to dive into more, and discuss the evidence base for?

James Snowden: I think we did touch on it a bit already, but because we’ve already made the grant, it’s something that I’ve spent relatively large amounts of time on. I think pesticide regulation would probably be the best thing to focus on. The other things, we’re very much in the middle of our investigation, I wouldn’t want to state things publicly before I got a better degree of confidence over what I feel about those.

Robert Wiblin: What were the main uncertainties or main doubts that you had about that intervention?

James Snowden: It’s interesting. One difficult thing about these policy changes is you often don’t, it’s quite hard to run a randomized controlled trial on say, a regulation which comes … well, it’s impossible to run a randomized controlled trial on a regulation which comes into effect in a country all at once. There are some times that regulations might be imposed, they might happen in a sequential way in which case you might be able to tease out a well identified quasi-experimental study there, but often that’s just not the case.

For pesticide regulation, there appears to be some evidence that pesticide regulation reduced total suicides in Korea, Sri Lanka, Jordan, just to name three, and a few other countries as well. We talked about potential mechanisms before, but one potential mechanism is that suicide attempts remain relatively stable but because people are using pesticides of a lower case fatality, and they can vary a lot, that’s less likely to result in death. And then another potential mechanism is you make the means with a very low barrier difficult to acquire, and that reduces spontaneous suicide.

I think there are really serious issues with the direct evidence base, so we don’t have a well identified study and we’re basically relying on time trends, which can very easily be confounded. For example, there was a civil war in Sri Lanka.

Robert Wiblin: This is before and after the policy change, basically, and then you hope that most of the shift there is because of the policy rather than something else.

James Snowden: Yeah, so I think it’s partly hope. I think there are things you can do … you’re never going to be as certain, I think, as if you have a well identified experiment. But there are things you can do to investigate different hypotheses, so there was a civil war in Sri Lanka around the time that suicides started to drop, but if you look at when the civil war fighting intensified and when it started and when it finished, it doesn’t really line up with the dates that suicides started to drop. It seems reasonably unlikely that this would have had a big effect, although it’s really hard to rule out various things that happen within that.

Robert Wiblin: Does a civil war generally reduce suicide?

James Snowden: We don’t know, right?

Robert Wiblin: Right, yeah. But it’s a big change, so it could have had a lot of effects.

James Snowden: Yeah. One plausible mechanism is, so people who commit suicide particularly through pesticide are often people who are working in agricultural jobs, and if they’re out fighting in a war you might expect that they’re not at home. That feels like a bit of a stretch, like a maybe. You could also make the other argument that civil wars tend to make people less happy, and so you might increase suicide.

Robert Wiblin: Possibly increase the sense of group solidarity, and gives people more reason to live, because they’re part of some broader mission of winning this war.

James Snowden: Yep, that’s definitely possible. That’s definitely possible. It’s really difficult, right, with these confounders. It could go in either direction, it’s hard to know what to make of that. You can also look at other things that you know are associated with suicide, like unemployment. What happened to unemployment in Sri Lanka, what happened to economic growth, and there again the dates just don’t really line up. You can’t rule these things out. And of course there might just be things you haven’t even thought of, right? That’s the trouble with these studies.

I think something I did find reasonably compelling is, what we do have is medical reports from hospitals in Sri Lanka which look at how many people came in for pesticide poisoning, and which pesticides were being used. There, what’s cool is we can see what was the mechanism that was going on here, and there does seem to be some suggestion that less toxic pesticides became more common after the bans, and this explains some of the reduction. Those broadly but not perfectly support the story that people are switching to less toxic options.

This is interesting, because it’s a question about how you do research, right? The right question here isn’t what characteristics did this study have, was it well identified, it’s often, what’s the most plausible explanation given the broad sweep of evidence? And when you look at that broad sweep, it did seem likely to me that pesticide regulation had a fairly big impact on suicide.

Robert Wiblin: This is a fairly unusual policy change. It’s not something that I’d ever thought about before speaking with you. How did you find this one, and do you do a lot of exploration to try to find these unusual interventions that other people haven’t thought about yet?

James Snowden: This is actually a slightly odd one. I think this maybe highlights the importance of, assuming this ended up being a good idea which is not at all obvious, highlights the importance of reacting to things that you come across rather than, as well as having this very systematic way of looking through things. The story was, when I used to work at the Center for Effective Altruism, we were doing a report on mental health so a cause specific report. We were like, okay, say mental health is the thing you care about. What would be a good organization to do there?

And this involved looking through various sources like Disease Control Priorities 2, and so I came across this idea of pesticide regulation there. At the time I didn’t prioritize it, there were two reasons for that. The actual quote in DCP2 doesn’t make it look cost-effective, and it also appeals to different moral values. It’s a different worldview that would lead you to think that this was important versus mental health is important. It’s almost the opposite worldview.

And so that’s not something I dug into deeper at the time, but then a while later we posted the mental health report on the effective altruism forum and a guy called Austin Forrester who was pretty interested in this issue, he offered to introduce me to Michael Eddleston who is the professor who worked in Sri Lanka, helping regulate these pesticides at the time. And so I had a conversation with him, and that conversation made me look back at the cost-effectiveness analysis in DCP2 and realize it was just using a very different methodology from how I would have done it. In particular, it was only looking at the impact over one year, which is going to make it look a lot less good than we would think looking from a longer term perspective, which is an interesting data point in itself. It’s how much can you trust the headline cost-effectiveness results without understanding exactly how it’s done?

That was interesting because it’s very much like, it was a bit of a fluke that this came up. There happened to be this guy in the effective altruism community who had thought about this. When I moved to GiveWell, I took that work with me and we dug more into the research and the more we dug, it really did start to look like quite an exciting grant and quite an exciting opportunity.

The other thing here of course is there wasn’t an organization working on it, and so it’s kind of a happy circumstance, I guess it’s not purely circumstance because there was desire to create an organization from Michael Eddleston who we ended up funding. But I do think there’s a lesson here about first being responsive to opportunities when they start to arise, and recognizing the value when you see it, and also the benefits of the effective altruism community and lots of people thinking about this stuff, for identifying new interventions we just might not have considered.

Robert Wiblin: It seems like in the past GiveWell has been a little bit reluctant to look into these policy advocacy issues, because there was a sense that the evidence base was never going to be really sufficient to meet GiveWell’s standards. Why are you looking into it now? Is it just that the reward to risk ratio looks different now?

James Snowden: I don’t think I have a great sense of what GiveWell’s attitude was to these things before. Maybe there’s a few different things at play. I can’t say exactly which of these were the real reason, but I think all of these might lead us to think more about this. One is that we think we have covered quite a lot of the space in our more traditional work, but there’s always more to be done. There are other interventions which look a lot more like our general top charities that we still haven’t reviewed, as well as other sectors that we think we’re still in the pretty early stages of, such as education and agriculture, two in particular.

I think one push was this funding constraint issue. When we’re pretty funding constrained the value of finding something, if we have a 10% chance of finding something 20 times as good as cash, we’d want to take that over 100% chance of finding something 4 times as good as cash, because at that point we’re just displacing funds from our marginal charities and not making a huge difference.

I think those are two reasons, so one is it might not be as relatively valuable to identify other similar charities, and this seems like an area that might be more … if we can feel confident in it, or even if we can’t but we make our best guess, that this could be an area where there’s potential to find things which are better than our top charities. And the other being that we think we’ve covered quite a lot of the space.

Robert Wiblin: Are there any other mental health interventions that you’re looking at other than preventing pesticide suicide?

James Snowden: Yes, so it’s on my list of priorities this year, to go back and look again at mental health. I’ve done some work on it, but I think it’s such a large area that I really haven’t done the space justice. I can talk about what my thoughts are so far, and I’ll focus on depression because that’s something that I’ve spent the most time on, partly because it’s the largest burden of disease.

I think there are a few different ways that you can treat depression, so one that I’ve been mostly focused on is psychotherapy to treat depression. My current best guess is that psychotherapy to treat depression generally isn’t as cost-effective as our top charities.

Robert Wiblin: Wouldn’t it be quite expensive to deliver? You’ve got to have someone there for many hours, potentially, treating someone.

James Snowden: Yeah. I think this is the problem, right? You’ve hit the nail on the head. I think there are ways around this but maybe not ways that make it perfectly scalable. If I was to recommend an organization based on my current feelings, it’d be an organization called StrongMinds that works in Uganda. That’s a charity I’d recommend to someone who wanted to donate in that area, and they use a task shifting model where they use mental health facilitators rather than psychiatrists to deliver interpersonal therapy. They also treat people in groups.

Those two things bring the cost down, but it still costs … the last time I spoke to them, this was over a year ago, it cost about $200 per person. I expect that that’s come down now as they’ve reached a greater scale, but it still is relatively intense. Even though it’s so much cheaper than therapy in developed countries, delivered by psychiatrists, and I think what they’ve done there is really great, it’s still compared to other things … a deworming pill costs 50 cents. Of course, you have to scale that against the scale of the benefit, but it is a relatively resource intense intervention.

Robert Wiblin: What about delivering antidepressant drugs? I’ve heard that they can be extremely cheap once they’re off patent.

James Snowden: Yeah, it’s a good question. This is something I feel like I haven’t got to the bottom of. On antidepressants, it’s a pretty controversial evidence base but my view on the evidence is that I find it pretty compelling. One of the key skeptical claims is the possibility of publication bias, particularly with pharmaceutical companies not publishing their negative results, although I think the evidence for that is pretty mixed, that publication bias is a strong thing there.

I think there was a new meta-analysis that came out, and I haven’t looked at it yet but it looked at unpublished data from pharmaceutical companies and it claims … I haven’t vetted this in depth, but it at least claims not to have found strong evidence of a lot of publication bias.

Say we think that antidepressants are effective. It’s interesting to me that it was not a lot of charities I could find, when I did my initial search, using this strategy. My guess is it’s something that generally needs to be integrated into the healthcare system, because it needs to be prescribed through a doctor and so these would be delivered through clinics. This would have two effects, so rather than, one is it would increase the cost because you need to have a doctor rather than a community health worker, although it might still be cost-effective. The other is that it makes it quite a poor fit for a vertical program which would be delivered by a charity. But these are just my initial guesses, and it’s something I want to look into more, because it certainly seems strange to me that this isn’t something which is more widely used.

Robert Wiblin: Let’s talk now about what it’s like working at GiveWell. I guess the conversation so far has given people a reasonably good sense of what kind of questions the researchers at GiveWell are looking into, but how did you end up working there? What’s been your career progression so far, and why did you choose to move to GiveWell?

James Snowden: Maybe if I start from the beginning, since I left college, and give you a very quick overview. I finished a degree at Oxford in philosophy, politics and economics in 2011 and then didn’t really know what I wanted to do with my life. Like anyone who didn’t know what they wanted to do with their life, I became a consultant.

Robert Wiblin: Classic.

James Snowden: Yeah, I know. How are you feeling about that on 80,000 Hours at the moment, where you landed on it?

Robert Wiblin: Yeah. It’s been something that we’ve recommended for people who didn’t know at all what they wanted to do in the past, but I think we now think it may be one of the biggest misunderstandings that people have taken away from our advice, is there’s just a lot of people going into consulting and gaining skills that aren’t directly relevant to what they’re going to do later. I think we’re actually going to do an episode where we talk about misconceptions about our advice, and this one will come up. We think more people are going into consulting maybe than really should, but that’s another conversation.

James Snowden: Yeah, that’d be a great episode. I guess maybe this is a good stage for me to reflect on that a little bit. It’s really difficult to know whether you did the right thing, because you never observe the counterfactual. I think there’s definitely, I don’t know how much I’m fooling myself here but I think there are definitely skills that I picked up in consulting which have been pretty useful to GiveWell work. I don’t know how that would compare to doing an economics PhD in the same amount of time. I suspect it would be better to do an economics PhD, but at the same time this isn’t something that I knew I wanted to do, so there was value there.

I do think in terms of prioritizing work fast and make a decision, think about things in a decision relevant way rather than necessarily in a more academic way, that’s a pretty good place to learn those skills, in consulting. Just being really competent, hopefully quite competent with Excel and just being able to model things quickly, and not being afraid to do back of the envelopes. That was something that was quite useful as well.

I think there are a lot of problems as well, consulting doesn’t always have the epistemic culture that really teaches you to do good research, I guess. That’s something I’ve had to unlearn quite quickly as I’ve started to do jobs which … we’re trying to get the right answer.

Robert Wiblin: Where the right answer matters more than a compelling answer.

James Snowden: Well, exactly. I wouldn’t want to overblow, we weren’t just lying, but it wasn’t perfect in that regard, I would say, consulting. They are my quick thoughts about whether that would be a good thing to do.

So I did that for three years, and then decided that that … you do a wide range of things in consulting, so it did let me rule out a lot of things quite quickly. At that point I was thinking, okay. My initial plan was go and learn these skills in consulting and then go out and do something good in the world.

By the way, I kind of skipped over a chapter, which is that I joined Giving What We Can in 2011, so this is right at the end of my degree, as a member. At the time, I think this was before effective altruism became a thing, and I also wasn’t a deeply engaged member of the community, just a friend told me about this website, I looked at it and was like yeah, they’re totally right. I signed up for it that way without becoming deeply embedded in the community, but it was something that was already happening. I think I’m quite rare in that. I think a lot of people who got into effective altruism very much got into the community first.

So I did that for three years, then went away and did a masters in philosophy and economics, because I still couldn’t decide which of those subjects was better, and during that time, started getting really interested in the philosophy of effective altruism and that way of thinking about the world, and thinking, what is the best thing you can do?

And then on leaving, on finishing my masters, it was a one year masters, I ended up, I saw Giving What We Can was applying for a job and this was an organization I’m a member of, it’s an organization I think their mission is incredible, it’s the thing that I want to work on, and so joined them, initially at Giving What We Can and then Giving What We Can kind of got absorbed into the Center for Effective Altruism, moving across there, doing relatively similar kinds of work. I worked as a researcher at Giving What We Can and also spent a bit of time on outreach. The research was what I really wanted to do, and I think it was the thing that I was better at, as well.

Then I ended up moving to GiveWell after … It turned out, and I think this is something that we’ve been thinking about for a while, which was that we didn’t feel at CEA that our research was adding a lot of value over what GiveWell was doing. It just made sense that I could … One thing that happened at CEA was I was investigating this grant and I had come to an answer and a recommendation, but there wasn’t really a good way to raise funds for that. We didn’t have that structure in place at Giving What We Can. We think we’ve done this great organization, what do we do now?

Robert Wiblin: You can recommend it to the members but they won’t necessarily follow the recommendation.

James Snowden: Exactly. I think by this point GiveWell, well I think this is partly fair and partly reputation. GiveWell’s reputation and also I think its ability to do research was I think at a much more developed stage than Giving What We Can or Center for Effective Altruism.

It was two things, it was having trouble raising the funds for a thing and also having this metric open to you, and often thinking, you know what I’d really like, I’d like Elie to spend a load of time vetting my work. It made sense to make that shift, and CEA were, we were shutting down our research arm because we’d come to this realization. I think it was really, I only spent a year at CEA and because it was such a similar job to what I’m doing at GiveWell, it was also very self directed, because there weren’t many of us. It was me and Hauke Hillebrandt and Marinella Capriati, and so there was a lot of, okay, we were working this out as we went along. Which I think can be a really good learning experience.

I ended up at GiveWell, and I guess the rest is very recent history.

Robert Wiblin: If someone was listening and they were considering working at GiveWell, what are the most satisfying things about working at GiveWell and what are some things that you wish you could change if you could?

James Snowden: I think personally what I’ve found very satisfying, just from a purely what makes me feel good perspective. The feeling of making a grant which you know wouldn’t otherwise be made, and then following that, is just one of the most satisfying experiences I’ve ever had. It really feels like there’s a concrete way in which you’ve made a difference, and you probably have made possibly a very large positive difference in the world, assuming it was a good grant.

Personally I found that more emotionally fulfilling than when I’ve done various direct volunteering work, although maybe that might be more of a reflection on my personality than a general truth. I think that’s, I guess that’s true of a lot of funders, right? That’s something you can experience there.

I think the thing that really struck me working with GiveWell is you are working with incredibly smart and incredibly curious, incredibly kind people. That’s something I’ve really enjoyed. You had that at Giving What We Can as well, but I think at GiveWell there’s a unique epistemic culture there where it really does feel like everybody’s trying to get the right answer. Nobody’s trying to convince other people of things necessarily, they’re trying to find out the right answer in a cooperative way.

That’s something I really enjoyed. And people are really honest and open with each other. One thing is, if you’re thinking about working at GiveWell, it is really important you’re able to take critical feedback well. People are quite good at delivering this feedback. People aren’t generally just, you’re terrible, you suck. It’s pretty constructive but some people are better than others at taking that well, and that’s a good trait to have if you want to be at GiveWell.

And then in terms of things I would change. I think some of this is there are good reasons for these things, but say, from a personal development perspective, I think compared to most other funders we make relatively few grants. That might not be optimal from trying to learn how to do this well, and getting quick feedback loops. Because we’re directing large amounts of money to relatively few charities, we spend a lot of time per grantee, which is good in a way, because it really lets you get to the bottom of things, but I think you can probably … my guess is you can develop your intuitive grasp for complex problems quicker if you spend less time on each grant and then just get stronger feedback by making lots of smaller grants.

That’s not GiveWell’s model, so I think that’s something that, I don’t think this is something that GiveWell should change because it’s a core part of our model, but it is something that you might want to think about if you’re optimizing for really learning a lot about the field of philanthropy in a very short amount of time.

Robert Wiblin: What’s the management style like at GiveWell? Is each staff member free to set their own agenda, or do you have to fit into a very specific project that is an important part of the broader research agenda for that year?

James Snowden: I wouldn’t say we’re free to set our own agenda. For this year we planned out, we went through quite a long process where we worked out what our research priorities were. We’ve got this pretty amazing spreadsheet, which Elie put together, which tries to estimate the ROI on every kind of research we could do. Obviously this is highly subjective, but it’s at least a way of thinking about the problem.

I think once that research agenda’s set, if there are particularly high priority things that come up, then yes you could do that but you’d have to make a case to get it on that list. We think it’s really important to be focused and make sure that we’re doing the highest priority things first.

I think where you do have a lot of freedom is at the point where that list is being put together, or afterwards, if you make a strong argument for this is what I think is the most valuable use of my time, or this is something I’m really excited about, I want to spend a bit of time looking into it, that’s generally going to be listened to. Although I think it’s often, if someone’s like, oh, I want to spend the next three weeks looking at this crazy thing … I got quite obsessed with snakebite relatively recently and was looking at that, but it’s very much a, okay, let’s see what the 5 hour version looks like, let’s see what the 10 hour version looks like. Then let’s think, is this worth prioritizing?

You’re not free to set your own agenda, but you have an input into what the overall agenda is.

Robert Wiblin: How much time do you spend talking to charities or traveling to meet charities that you might recommend?

James Snowden: We don’t spend that much time traveling to meet charities. Personally I’d like to do more field visits, partially because I think they are really valuable for personal development, and partially because I think there might be a lot of unmeasured benefits we’re getting from them in terms of getting a more intuitive understanding of how the implementation of a lot of these programs work.

We’ve done field visits for most of our top charities but not all of them, and that’s something, I think we’ve probably spent, not having a great understanding of exactly how other funders work but my sense is that we spend less time on that than most other funders.

In terms of talking to people, yeah, I think that’s really important. As you know we publish all our conversation notes from conversations we have, and in a typical week I might have three or four conversations with external experts on a variety of topics. We think that’s really important, particularly when … we’re an organization with quite a large focus area, it’s quite a broad scope of stuff we do, relative to a lot of funders. Because we go into things in quite a lot of depth, most of our researchers are generalists and so it’s quite important to be, okay, we can talk to the foremost expert on drug resistance to anti-malarial drugs. I personally, generally find those conversations incredibly valuable.

Robert Wiblin: What’s the office culture like within GiveWell? Does it still feel like a startup or is it a fairly mature organization now?

James Snowden: I’d say somewhere in between. I think there’s a sense in which I’m uniquely badly placed to comment on the physical office culture at GiveWell, because I work remotely from London. I have spent about three months in our office in San Francisco. I think the impression that most people get is that it’s pretty serious, and pretty quiet.

Robert Wiblin: That’s my impression, visiting.

James Snowden: I know you spent a bit of time there, Robert. It’s this deafening silence which greets you.

Robert Wiblin: People are very focused on the spreadsheets on their screen, was my impression.

James Snowden: I think that’s right. I think we’re, this is an obvious point but we’re doing a serious job so I think it’s right that we take it seriously. We also attract quite a lot of nerdy people, which makes sense, given the work we do.

I do think maybe this is overplayed a bit, in terms of the general perception. I remember when I first started I was a bit worried about it, so I remember going on a walk with Elie and mentioning that I worried I wasn’t that good a fit, because everyone is so serious and quiet. Now I’m not worried about that at all, and I think that was something to do with false impressions. I think a few things, one is that we actually have a pretty vibrant Slack culture, so that’s the instant messenger we use, it’s our main method of communication. I don’t know if that’s a very good defense but we’re hilarious on Slack, Rob.

It’s not like people aren’t communicating, but we prioritize being respectful of other people’s focus.

Maybe like two other things I’d point to. Because we write in such a formal way on our website, it was interesting, something that surprised me was how much less formal internal communication was. Actually much less formal than, say, when I was in a consulting shop. The argument there is it takes so much time to edit your language, when you’re sending emails or writing internal reports, and I think there’s a lot of time wasted in a lot of jobs where you’re not really trying to make yourself clearer, you’re trying to make yourself sound more impressive. It’s very obvious when you read that kind of writing.

Robert Wiblin: Or, I guess, do the right social grooming, be polite to the right people.

James Snowden: Yeah, exactly. I don’t think there’s much of that in GiveWell, and I think that’s pretty valuable.

You know, seriousness isn’t the same as formality, and often something written informally can actually be a lot clearer than something written formally. Maybe another data point is in my experience, I went through a minor depressive episode when I was working at GiveWell, and my managers were really excellent there, better than anywhere else I’d been. I think people can be emotionally intelligent without being extroverted, obviously. That’s something, and I don’t know, that’s just me who’s found that, just with my managers. I wouldn’t be surprised if that was a broad rule at people at GiveWell.

Robert Wiblin: Maybe let’s talk about the jobs that are available at GiveWell at the moment, if someone’s been enjoying the conversation so far and it sounds like the kind of office that they’d like to work in. Do you know what roles are available right now?

James Snowden: Yeah, so we’re hiring for a number of positions and we’d encourage people with the right skill sets to apply for them. They’re all advertised on our website, and maybe if you could put up a link in the description, they’re up on our jobs page.

If what you want is a part-time position, we’re always looking for new conversation note writers so these are the people who do really invaluable work listening to the recordings of all our conversations and then writing them up and publishing them. That’s a part-time role but in the past, that has led to people who will start as a conversation note writer and then they’ll become a research analyst.

Robert Wiblin: What’s the natural path to getting a job at GiveWell? What kind of undergraduate degrees can people study or perhaps postgraduate degrees, or other places they can work to make sure that they’re gaining the skills they need to thrive at GiveWell, and actually get a job there?

James Snowden: We actually have quite a variety of career paths that have ended up at GiveWell. Some things that do stand out is it’s good to have a quantitative background, whether that’s in academia or in a career as a researcher or maybe doing some kind of consulting which is more focused on analysis. I’d recommend going and doing a subject like economics or statistics at university is a pretty good preparation for the work we do at GiveWell.

I’d also mention that we’re not … if what you’re really interested in is going very deep on the academic research, we do have some people who do that, so David Roodman is one, but generally the skill set of our research is more generalist and it’s not necessarily, you don’t need to be an expert statistician to work at GiveWell.

I think in terms of the character traits we’re looking for, ideally we’d be looking for people who are really interested in the question of how to do the most good with your giving. This is a really interesting question, and if you’re not interested, this wouldn’t be the place for you, quite obviously. Often that means that someone who’s already somewhat familiar with our work, which is why I expect a lot of people who are involved in effective altruism might be a good fit at GiveWell because often we find a lot of our most engaged donors are from the EA community.

Robert Wiblin: To wrap up, do you want to give people a call to arms to actually apply for these jobs at GiveWell and not just shut off the podcast and not think about this again?

James Snowden: Yeah. I’d really encourage people to work at GiveWell. I’ve been there a year and it’s the best job I’ve ever had. I think there are three really important reasons that somebody would want to work at GiveWell. The first is a combination of working with smart, interesting people who really do care about the world. Secondly is just working on some of the most fascinating questions I’ve come across, and really thinking through a problem and trying to work out what’s true. Third is I think there’s a really high potential for impact by working at GiveWell.

Robert Wiblin: Fantastic. Thanks so much. My guest today has been James Snowden. Thanks for coming on the 80,000 Hours podcast, James.

James Snowden: Thanks Rob.

Robert Wiblin: The 80,000 Hours Podcast is produced by Keiran Harris. Thanks for joining, talk to you next week!