Thumbs up or thumbs down? Did the Millennium Villages Project work?

Guest post by Chris Barnett, Director of Technical Excellence, Itad

Back in the mid-2000s, one project stood out as a bold attempt to achieve the Millennium Development Goals (MDGs) at a local level: The Millennium Villages Project (MVP). Spearheaded by Professor Jeffrey Sachs, the project sought to demonstrate how the MDGs could be achieved through an integrated approach to rural development. For the past six years, I’ve been managing an evaluation of the MVP in northern Ghana, funded by DFID, to provide rigorous evidence about whether this approach to development really works.

Intuitively, the MVP made sense in many ways: do everything at once (health, education, agriculture, etc) so the effect will be far greater than a piecemeal approach of single, small-scale interventions. Not only that, but we do actually know what works in lots of areas: applying artificial fertiliser to maize rapidly increases yields, bed nets protect people from malaria-carrying mosquitos; vaccinations prevent disease.

As the former UN Secretary-General, Ban Ki-Moon was quoted as saying said back in 2010: “Millennium Villages throughout Africa are showcasing… how effective an integrated strategy for health care, education, agriculture, and small business can be. We are seeing how to make the most of new technologies. And we are seeing how empowering women can empower whole societies…”.

So, lots of optimism… but, it’s been controversial.

Controversy seemed to peak around 2011 to 2013, which just happened to be when we started out in northern Ghana. Putting aside the more heated ‘war of words’, Michael Clemens and others were writing papers and blogging on the need f`or rigorous evidence (such as this Guardian blog, or putting the case forward that ‘rigorous impact evidence is not a luxury…’).

So, here we are – more than five years later – bringing together evidence of the impact of the Millennium Villages Project (for northern Ghana at least):

Did the project have an impact? Well, yes and no.

No, in terms of achieving the MDGs at a local level (see MVP Evaluation Briefing 1). Certainly, for the key indicators for poverty (MDG 1) there does not seem to have been a reduction. There were income improvements that could be attributed to the Millennium Villages project, but there was no accompanying increase in consumption, and the improvements in income were insufficient to break the poverty trap. The population remains poor.

But yes, there were impacts. We found statistically significant impact against 7 of the 28 MDG indicators. For example, primary school attendance improved by 7.7%, intermediate health indicators improved for births attended by skilled professionals, children sleeping under bed nets, and so on.

Of course, assessing the project just against the MDGs is somewhat ridiculous. Rarely can a project achieve such high-level goals in a short period (although that is what the project set out to do), and the MDGs themselves are far from a definitive list of indicators. Nor indeed do they reflect the realities of people living in the area. So, there is lots of exploratory analysis that we cover elsewhere…

Does the project offer value for money? This is THE controversial one.

Caution needs to be applied when interpreting the analysis, as no direct comparison with similar projects was possible (see MVP Evaluation Briefing 3). Our analysis shows that the returns to investment in education appear to be highest, although both the returns in education and health could be achieved at a much lower cost.

Given that this was always intended as a demonstration project, we also undertook a sensitivity analysis to see if transferring such a project to local ownership could improve the value for money. Even with a 50% reduction in overheads however, there is questionable cost-effectiveness.

So, what can we conclude?

Certainly, achieving lasting change is difficult. The project’s over-ambitious aims (and often claims) made it the target for much disagreement. There were some good ideas, and things that worked well (for example, the tractor hire services, vaccinations, investments in health centres and school infrastructure). But it was starting from a low base of investment – as anyone who’s travelled to the very northern areas of Ghana knows – and it takes time.

Finally, a few things stand out for me about the Millennium Villages approach in northern Ghana:

Firstly, although many ‘quick wins’ had been scientifically tested (with rigorous evidence), the project seemed to underestimate the need for considerable experimentation in each context – working with the grain of local government and communities, as well as working with how people respond differently to interventions and adapt solutions. For example, people were not always using bed nets, finding hard at certain times of the year to reconcile the use of bed nets with the night heat. (As an aside, it is also surprising how many creative uses for bed nets there were, whether as screens for toilets or bathing areas, to protect young fruit trees, to cover store rooms, as pillows or bedding to lie on, as fishing nets, door curtains and sun blinds).

And secondly, sustainability is a huge challenge (see MVP Evaluation Briefing 9): Approaches like top-up allowances for education and health workers, new ‘super’ health facilities, and free ambulance services, etc., all had a measure of success, but were beyond what could be sustained locally. Spending by the project tended to focus on building infrastructure, providing supplies and extra staffing (‘things’) with insufficient attention to the behaviour change needed to create long-lasting impact. For example, the statistical analysis shows a sharp increase in the access to toilets (part of MDG 7), with toilets rapidly built during the last two years of the project – yet, the qualitative research highlights how people still preferred to go ‘free range’, under a ‘nice baobab tree’ or among rocks away from the village.

I am sure that for many readers of ‘From Poverty to Power’ this evidence only serves to reconfirm their perspective of development – the need to understand power, work with complexity, disrupt the system. But, I leave you with one challenge: would your chosen project or policy have fared any better if subject to robust evaluation? And, how do we know?

Subscribe to our Newsletter

You can unsubscribe at any time by clicking the link in the footer of our emails. For information about our privacy practices, please see our Privacy Policy.

We use MailChimp as our marketing platform. By subscribing, you acknowledge that your information will be transferred to MailChimp for processing. Learn more about MailChimp's privacy practices here.

Comments

4 Responses to “Thumbs up or thumbs down? Did the Millennium Villages Project work?”
  1. Catherine Dom

    Somebody should do research on creative uses of bednets… This seems a fixture of the thing all over Africa (and perhaps SE Asia I don’t know). I knew these uses from here Ethiopia. Here some people also use it as an undersheet cover to protect from bed bugs. Anything else anywhere else?

  2. milford bateman

    Going solely on the extreme measures taken by the MVP and Jeffrey Sachs to make sure no–one could independently evaluate the project (see the article by Japhy Wilson “Sabotage of Development: Subverting the Censorship of Renegade Research “ 
Journal of Extreme Anthropology, Vol. 2, No. 1:1-27, ISSN: 2535-3241), I would tend to think they must have been hiding a complete disaster of a project!

  3. I don’t see much discussion of context here. By ‘context’, I mean governance or enabling environments (for individual and private sector actors AND the effectiveness and capacity of government systems for service delivery in all of these sectors. I guess, in terms of the classic Easterly vs Sachs debate, I lean towards Professor Easterly.

    In the context of a nonexistent, unsupportive, or otherwise insufficient enabling environment, I imagine that it would take more money than any donor could or would contemplate spending to fix some of these ills. Then, the question would be how long money would have to be poured in for any of this to sustain gains/improvements. (Even with that recent cash aid report that everyone is talking about, you’ve got to wonder. For how many years would donors have to give $500+ to each household?)

    Long term solutions are likely economic growth and improved democracies (accountable governments), but that does not help anyone in the short or medium terms.

  4. The two findings (the need for considerable experimentation and local sustainability) are key issues we find all over Africa. However, I am impressed by the example of the digitalization of Ghana as country.
    In the named fields of education (edtech), health (medtech) and financial access (fintech) there are many positive signals arising from the young population and its many young startups.
    Ghana is really worth to follow up closely.

Leave a Reply

Your e-mail address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.