Enjoyed the episode? Want to listen later? Subscribe here, or anywhere you get podcasts:

A lot of the problems that we seek to solve are going to require long-term sustained investment. So if a donor doesn’t want to or is not able to provide that support for a long period of time, that’s fine.

But then the conversation we should have is, “How are we going to shift the coverage of this to some other donor or some other source of funds?” and not, “How are we going to put in a little money, and then it’s going to create an engine that’s going to fund itself?”

Karen Levy

If someone said a global health and development programme was sustainable, participatory, and holistic, you’d have to guess that they were saying something positive. But according to today’s guest Karen Levy — deworming pioneer and veteran of Innovations for Poverty Action, Evidence Action, and Y Combinator — each of those three concepts has become so fashionable that they’re at risk of being seriously overrated and applied where they don’t belong.

Such concepts might even cause harm — trying to make a project embody all three is as likely to ruin it as help it flourish.

First, what do people mean by ‘sustainability’? Usually they mean something like the programme will eventually be able to continue without needing further financial support from the donor. But how is that possible? Governments, nonprofits, and aid agencies aim to provide health services, education, infrastructure, financial services, and so on — and all of these require ongoing funding to pay for materials and staff to keep them running.

I buy my groceries from a supermarket, and I’m not under the illusion that one day I’ll be able to stop paying and still get everything I need for free. And there’s nothing wrong with this way of getting life’s necessities being ‘unsustainable’ — so long as I want groceries, I’ll keep paying for them.

Given that someone needs to keep paying, Karen tells us that in practice, ‘sustainability’ is usually a euphemism for the programme at some point being passed on to someone else to fund — usually the national government. And while that can be fine, the national government of Kenya only spends $400 per person to provide each and every government service — just 2% of what the US spends on each resident. Incredibly tight budgets like that are typical of low-income countries. While the concept of ‘sustainability’ sounds great, to say “We’re going to pass the cost of this programme on to a government funded by very poor people’s taxes” sounds at best ambiguous.

‘Participatory’ also sounds nice, and inasmuch as it means leaders are accountable to the people they’re trying to help, it probably is. But Karen tells us that in the field, ‘participatory’ usually means that recipients are expected to be involved in planning and delivering services themselves.

While that might be suitable in some situations, it’s hardly something people in rich countries always want for themselves. Ideally we want government healthcare and education to be high quality without us having to attend meetings to keep it on track — and people in poor countries have as many or more pressures on their time. While accountability is desirable, an expectation of participation can be as much a burden as a blessing.

Finally, making a programme ‘holistic’ could be smart, but as Karen lays out, it also has some major downsides. For one, it means you’re doing lots of things at once, which makes it hard to tell which parts of the project are making the biggest difference relative to their cost. For another, when you have a lot of goals at once, it’s hard to tell whether you’re making progress, or really put your mind to focusing on making one thing go extremely well. And finally, holistic programmes can be impractically expensive — Karen tells the story of a wonderful ‘holistic school health’ programme that, if continued, was going to cost 3.5 times the entire school’s budget.

Smallpox elimination was one of humanity’s greatest health achievements and its focus on one thing to the exclusion of all else made it the complete opposite of a holistic program.

In today’s in-depth conversation, Karen Levy and I chat about the above, as well as:

  • Why it pays to figure out how you’ll interpret the results of an experiment ahead of time
  • The trouble with misaligned incentives within the development industry
  • Projects that don’t deliver value for money and should be scaled down
  • Whether governments typically pay for a project once outside funding is withdrawn
  • How Karen accidentally became a leading figure in the push to deworm tens of millions of schoolchildren
  • Logistical challenges in reaching huge numbers of people with essential services
  • How Karen has enjoyed living in Kenya for several decades
  • Lessons from Karen’s many-decades career
  • The goals of Karen’s new project: Fit for Purpose

Get this episode by subscribing to our podcast on the world’s most pressing problems and how to solve them: type “80,000 Hours” into your podcasting app. Or read the transcript below.

Producer: Keiran Harris
Audio mastering: Ben Cordell and Ryan Kessler
Transcriptions: Katy Moore

Highlights

Pre-policy plans

Karen Levy: I think [one] thing that’s really useful and important about a pre-policy planning process is that it’s a wonderful tool for aligning the research and policymaker perspectives. Very often there’s not enough communication upfront, and what this can lead to is evidence not being useful because it’s not available on time, for example, or the results are not applicable to a population that policymakers are actually needing to make policy for.

Karen Levy: And so with a pre-policy plan, the idea is to get people to sit down in advance and really line up: “What are the questions we’re asking? What are the outcomes we’re measuring? When are we going to have this evidence?” alongside “What decisions are being made as policymakers? What choices do they actually have? When and how are resources getting allocated?” So researchers can make their work relevant and responsive to the actual choices policymakers are making and vice versa.

Rob Wiblin: Pre-policy plans are, I guess, figuring out under what conditions would you want to scale a program up, under what conditions would you hold steady, under what conditions would you cancel it — but ahead of time, before you get biased by what the results you’re getting actually are. It seems like in [the case of No Sugar], the results were very clear cut and quite devastating. It’s interesting to me that even in this case, people kind of wanted to push on. It’s so hard to let go of a program, even when the most likely outcome seems to be that you’re causing harm. It’s possible that having this conversation ahead of time might have made the difference between continuing the program and not.

Karen Levy: Yeah. I think the truth is that most people want to do something. And I can relate to that: I am action-biased, so all else equal, I’d rather do something than nothing. So I understand where it comes from. And in this case, as you say, I think it was very clear cut. But often it’s not, right? So the other thing that a pre-policy plan can do is for some tough conversations about “How much impact is enough to make this worth doing?” or “What if it’s effective for some subpopulation, but not for another?”

Karen Levy: These are all questions that are much easier to really think through and interrogate when you don’t know the answer. And so, just in the very same way that a pre-analysis plan forces a clarity of thought ex ante, I think there’s a lot more that we can do in the global health and development space to be thinking about how we would use new information before we get it.

'Sustainability'

Karen Levy: So one of these concepts that I think we should interrogate a little bit is that of sustainability. And it’s not that sustainability is bad, but I think that insisting on sustainability — making it a prerequisite of support for a particular program or policy — can really distract us from what matters or what should matter.

Karen Levy: In some ways, I think that over-privileging sustainability in assessing the effectiveness or cost effectiveness or the extent to which we should prioritize a particular intervention can be almost like the moral equivalent of saying — to draw on Peter Singer’s famous analogy — “Well, a lot of kids drown in this pond, and since we don’t have a plan to save them indefinitely into the future, we shouldn’t save this one now.” And similarly, if I told you that for some reason, we were going to have to say, stop manufacturing a particular vaccine in five years, would we just stop giving that vaccine now? If that vaccine is truly effective and cost effective, well, then we should grab all of the impact that we can while we can.

Karen Levy: So what people usually mean by sustainability is, “How is this program going to continue after whatever initial funding is given has ceased?” But unfortunately, I think what people often really are asking is, “How are you going to create a perpetual motion machine?” And in high school physics, I was taught that there’s no such thing as perpetual motion, so I feel like we’re kind of collectively trying to defy the laws of nature here.

Karen Levy: A lot of the problems that we seek to solve are going to require long-term sustained investment. So if a donor doesn’t want to or is not able to provide that support for a long period of time, that’s fine. But then I think really the conversation we should have is, “How are we going to shift the coverage of this to some other donor or some other source of funds?” and not, “How are we going to put in a little money, and then it’s going to create an engine that’s going to then fund itself?”

Karen Levy: I think we should be thinking about how long we think a particular program or a particular problem is going to require resources. Is this a permanent need? Is this something like basic education, which needs to be funded essentially indefinitely? Or is this something like deworming? I don’t want the Kenyan government to have to sustain a deworming program 20 years from now — I want them to not have a worm problem anymore, right?

Karen Levy: Is this something we think with sufficient investment now we could really move the needle on over the next five to 10 years? We may find that identifying a highly cost-effective solution — and funding it reliably for 10 or 15 years — is way more cost effective than constantly starting and stopping and reprioritizing and dabbling in various pilot programs that never get continued, because we have a little bit of magical thinking that they’re going to somehow just perpetuate themselves.

Karen Levy: But there are some examples where that can be done — there are some programs that can ultimately rely on market mechanisms. That’s terrific, and that can be a highly leveraged investment for philanthropic funds. But a lot of the work that we do, a lot of the problems that we seek to solve, require the provision of public goods and services that will not have market-based solutions — that are just going to require ongoing investment.

Misaligned incentives within the development industry

Karen Levy: I have been thinking a lot recently about how the structure of the business model of NGOs that work in development is really counterproductive in many ways to optimizing impact and cost effectiveness. We’ve come a long way as an industry in understanding that the percentage of overhead is not what matters, right? I think that is an issue that’s been extensively debated and most people who think in forward-thinking ways about this get it: it doesn’t matter if it’s 7% or 27% — what matters is, are you having impact?

Karen Levy: At the same time, as long as the business model is essentially that you’re funding an organization based on what’s pretty much a flat tax on your throughput, think about the perverse incentives that that creates in terms of efficiency, for example. A big threat to an NGO’s budget is underspending. Well, I would love NGOs to underspend, as long as they’re holding their impact constant, right?

Rob Wiblin: What do you mean by they’re kind of fundraising based on a tax? Do you mean that the organization overseeing it can only justify taking some cut of the total amount being spent on the program as a whole? So the more they find a way to actually spend on something — whether it’s good value for money or not — then that kind of justifies having the people at the head office, basically?

Karen Levy: An overhead rate is a tax, right? It’s a flat rate. So it doesn’t matter if that rate is 7% or 17% or 27%. What it means is that as the leader of an NGO, the way that you fund your core costs is by being bigger — not in terms of your impact, but in terms of your financial throughput.

Karen Levy: I think this really creates some suboptimal decision making on the part of NGO leaders — they have to keep the lights on, right? If you are funding your core operations by taking a percentage of the money that you spend on charitable aims, then your goal is to just be spending more and more on charitable aims. And that’s not always consistent with what’s best in terms of impact.

Rob Wiblin: Do you have an example of a perverse incentive that donors face themselves?

Karen Levy: I definitely think that there is an inherent incentive when you are working for a donor organization to want to come up with new ideas, new strategies, new programs. If you have a “set it and forget it” portfolio of programs that are working and they’re delivering impact, then the decision is really, “Should we keep funding this — yes or no?” Well, what’s the job for your program officer then?

Karen Levy: I remember in the early days when we were starting Evidence Action, at one point we made a trip and visited five or six donors in the UK. And I think every single one of them was in the middle of a strategy refresh. I remember feeling like, “All right, well, when you guys all figure out what you want to support, let me know.” The donors are in many ways the client: they pay for the work, and NGOs can only do work that they can get funding for. So I think donors sometimes underestimate the extent to which changes in strategy can really create incredible inefficiencies for organizations as they scramble to follow whatever it is that donors are funding. I think that we could probably all do a lot more good collectively if we made much longer-term bets that were a little bit less sexy.

Lessons Karen learned from coordinating with governments

Karen Levy: Well, there were certainly times where I was out of my depth, let me tell you. I think that there is a perception of civil servants or working with governments that it’s slow, nobody cares. I actually have not found that to be the case. I find that by and large, the people that I have met and have worked with in the Kenyan government and in other governments are public servants: they are incredibly dedicated and they are working within really tough constraints.

Karen Levy: Sure, there’s gaps in capacity that we all have to face and work on together. But I really think that very often those perceptions come from misunderstandings about how decisions are made — who has authority to make certain decisions and why. So we can very easily say, “This person’s not being helpful” or “They’re being obstructionist” — but they’re working within a set of rules that is not of their making. So I think that there are way more opportunities to work constructively and collaboratively with government partners than we allow ourselves, perhaps because of some of those assumptions.

Karen Levy: I think it’s really important for people to learn more about how governments spend money, how they make decisions, and how they deliver goods and services. I often like to use the university as an example when I talk to people about how you should be thinking about working with the government. So let’s say you wanted to do a project with a university, and you said, “Well, I’m going to go and I’m going to talk to people. I want to talk to a broad selection of people at the university to find out about this project. And I’m going to do that in July.” Well, anyone who’s been to a university, certainly in the US, would say, “But you can’t; nobody’s there in July.”

Karen Levy: Similarly in governments, there are fiscal years. Talking to somebody right at the end of a fiscal year is not the right time to be thinking about how they’re spending money. The plans for the next year have already been made. The money from the previous year has already been spent. So being cognizant of the cadence of the way that a government works is really important.

Karen Levy: Another analogy that I think works well with universities is, let’s say you wanted to say, “What does the university think about this?” People ask me that all the time: “What does the Kenyan government think about this?” Well, who do you mean? Shall we ask the president? Shall we convene the cabinet? Governments are these incredibly complex institutions, within which there’s a lot of disagreement. And at the same time there is policy that flows from the top. There are structures, there are annual operating plans, et cetera, that you need to understand how to work with.

Karen Levy: So I think that understanding the periodicity and cadence of the way that governments work is really important: budgets, election cycles, the time horizon for policy change. And meanwhile in the development sector, everything works to the timeframe of the donor project cycle: you need to put in your reports to the donor; they’re marching to their cadence. And so working with the government, it’s a big ship: it does not turn on a dime, and you wouldn’t want it to. So really recognizing that and embracing it as a pathway to massive scale that you have to work within and alongside is absolutely essential in terms of getting things done.

Karen Levy: I also think one of the most important functions of a government is a coordinating function — and it’s all too rare to find development partners that are willing to be coordinated. I remember sitting in a meeting of a bunch of partners in the school health space. It was representatives from a bunch of different NGOs, all of whom had grants to coordinate the school health sector. Nobody was there to be coordinated. That’s what we should be letting governments do, and let them coordinate us. And that’s hard.

Why the Kenyan government isn't paying for these programmes

Rob Wiblin: To some degree we’re already talked about this, but a lot of people might have the reaction that the Kenyan government should be funding this, that it’s weird that a nonprofit is involved in coordinating this kind of thing. It seems like there was a test where that disappeared for a year and the program stopped — it wasn’t as if the government stepped up and paid for it. I guess there’s probably quite complicated, interesting, decision-making institutional reasons why it is the case that it’s not easy to grab a few million dollars and a bunch of staff to take over the role that you were doing. If you walk away, then the program freezes.

Karen Levy: Kenya is a resource-constrained place. So again, I mean, this may sound glib, but it’s kind of like saying, “Well, if poor people just spent more money on stuff, they would be less poor.” They’re poor because they’re poor. And so there are enormous pressures: there are lots of needs that resource allocators within the Kenyan government need to fulfill. Countries like Kenya borrow a lot of money. So I think it’s an overly simplistic mental model to say, “Well, there’s this big pile of money. The government, they’re so rich, why don’t they just spend money on this?” If you think about those funds as taxpayer funds: we’re convinced that the people receiving the deworming drugs shouldn’t pay for them, but that’s their government that we’re talking about.

Rob Wiblin: They would be paying for them through the taxes, effectively.

Karen Levy: Exactly. So, look, this can be taken to extremes. Of course there are things that governments can and should pay for. But in some ways, I almost feel like the amount of effort that it would take to ensure that these funds were allocated and dispersed and delivered and used well, year after year, is frankly —

Rob Wiblin: It’s easier just to pay for it.

Karen Levy: — we should do that for Kenya. We should just do that for Kenya. And there are other countries, like South Africa, that pay for it themselves. India is paying for most of it themselves. It’s leveraged by a much smaller amount of money that pays for the technical assistance and support around those basic costs.

Karen Levy: It’s a similar thing, like about blaming poor people for being poor. Kenya is resource constrained. There’s a lot of things they need to spread their resources over, and if this is something that can be easily taken care of by philanthropic funds, I would much rather see that happen. And hopefully there will be a time, not that far from now, where Kenya won’t need to have a deworming program.

The importance of a theory of change

Karen Levy: The most important lesson that I’ve learned in taking a program from evidence to scale is the importance of starting with, and always coming back to, a theory of change. A theory of change is essentially a description of a journey — from where we are now to where we want to be — and a set of hypotheses about what needs to happen along the way to get us there. It’s a causal pathway from a set of inputs to a series of outputs, which then lead to a set of outcomes and then ultimately to impact.

Karen Levy: So for example, inputs and activities are like the program components. For a school feeding program, that would be like the food and the training and the funds. And that then leads to a series of outputs, which is, say, a certain number of children receiving meals at school.

Karen Levy: That then leads to outcomes, which are intermediate- or short-term changes that result from those outputs. In the case of a school feeding program, that might be more children are attending school and are able to concentrate better in class. And then that finally leads to your impacts or your goals, which are the long-term changes that result from this causal pathway. In our example: improved learning, better socioeconomic outcomes, et cetera.

Karen Levy: When you have a piece of evidence or a set of evidence that kind of maps that out, the key to scaling is to figure out how to automate and simplify and deliver at scale those initial inputs, such that that causal chain remains. And what is it that you need to test along the way in order to make sure that that’s still happening?

Rob Wiblin: So you want to make your theory of change quite explicit, and then see what are the parts that might not work as it gets larger. What are the things that we need to be paying close attention to, basically?

Karen Levy: Exactly. For different types of interventions, you’re going to end up having different weaknesses along that chain, right? So for example, with deworming, once you get tablets in mouths, you can be reasonably confident that those are going to kill the worms. That’s not something you need to relitigate. So coverage in that case — the number of children receiving deworming drugs — as long as you know that they’re not fake, that they are actual drugs and they are actually being taken, you don’t need to necessarily relitigate that all the time.

Karen Levy: However, in many other programs, particularly ones that have behavioral components, you have to double-check that. So for me, the biggest lesson is unpacking the evidence into a really robust theory of change, having a very clear strategy for delivering those inputs and those activities in such a way that you’re going to set off that causal pathway, and so that you’re going to know along the way if each successive step is happening.

Academic work vs policy work

Karen Levy: In academia, you’re trying to find the optimal answer or the optimal solution. But in a policy setting, you’re trying to optimize within constraints — you don’t have an unlimited set of options from which to choose. So the way you would construct a study or the way you would ask a certain set of questions is really very different. The timeframes are really different. What success looks like and how one is rewarded for a study happening well or not — they’re just very, very different.

Karen Levy: But even though these differences exist, many of the examples that we have of programs that have scaled successfully have been ones that have been driven by academics who really care about the policy influence of their work. I think the reason that matters is because evidence isn’t just a set of answers that you pull off the shelf. In taking something from a study to scale, all kinds of questions come up about, “What are the underlying mechanisms? Why did we do it that way? If we change that, are we now pulling apart the whole theory of change, or is that still going to work?” So I think some of the most exciting examples of times when studies have moved from evidence to scale have been when you have a really committed academic, who’s willing to kind of take that journey with policymakers.

Karen Levy: There is sometimes some tension between what a program needs from a data and evidence perspective, and what is attractive to an academic. Sometimes you really just want to replicate something, or you need a bunch of descriptive statistics — and this is not necessarily what’s going to attract the attention of a world-class development economist. At the same time, many of those world-class development economists, we’re lucky to have them involved and caring about the impact of their research.

Karen Levy: So I find it’s really useful and important to have an open conversation with academic partners about what would make it interesting for them. I have found that almost always people are absolutely willing to do the less sexy stuff, if you also allow them to ask some really interesting, innovative questions that will drive their research agenda — and nine times out of 10, that part of it is going to end up leading to new insights that affect your work anyway. So the marginal cost of adding another module to a survey, or trying a slightly different version of an intervention — doing that can create the space for intellectual engagement from top thinkers, and then brings those people to the table for all of the other things that you need. And so I think we should have those conversations.

Articles, books, and other media discussed in the show

Karen’s projects:

Exciting developments and organisations in the aid industry:

Other 80,000 Hours Podcast episodes:

Everything else:

Related episodes

About the show

The 80,000 Hours Podcast features unusually in-depth conversations about the world's most pressing problems and how you can use your career to solve them. We invite guests pursuing a wide range of career paths — from academics and activists to entrepreneurs and policymakers — to analyse the case for and against working on different issues and which approaches are best for solving them.

The 80,000 Hours Podcast is produced and edited by Keiran Harris. Get in touch with feedback or guest suggestions by emailing [email protected].

What should I listen to first?

We've carefully selected 10 episodes we think it could make sense to listen to first, on a separate podcast feed:

Check out 'Effective Altruism: An Introduction'

Subscribe here, or anywhere you get podcasts:

If you're new, see the podcast homepage for ideas on where to start, or browse our full episode archive.