African techno-euphoria and the origins of Kenyan mobile exceptionalism

August 31, 2012

The best cat video ever (hey, it's Sunday – I'm allowed…..)

August 31, 2012

What have the MDGs achieved? We don't really know… Heretical thoughts from Matthew Lockwood

August 31, 2012
empty image
empty image

A second instalment in Matthew Lockwood’s series of valedictory boat-rocking blogs (his first was on fossil fuel subsidies) as he leaves the IDS Matthew_lockwood125Climate Change team for a new role in the UK energy sector. This time, he asks why the results agenda often stops short of being applied to the big picture stuff like the MDGs.

One of the interesting things about having come back to the international development field after some years away is the greatly increased emphasis on results, across all areas of activity, including not only projects and programmes, but also policy making, research, and advocacy.

Many people and organisations are interested in the results agenda, including the big foundations such as Gates, influential bloggers like Owen Barder, my boss Lawrence Haddad, and DFID’s Secretary of State, Andrew Mitchell. In his first big speech in office, in Washington in June 2010, Mitchell said “we’re also fundamentally redesigning our aid programmes so that they build in rigorous evaluation processes from day one.”

Like many others, I think aspects of the results agenda are important, reasonable and politically wise, although there are also some interesting critiques of the approach. But I also think that, if you really take it seriously, it throws up some challenges and dilemmas.

For me, this is clearest in the case of development’s big frameworks and policy directions. One prime example is the UN Millennium Development Goals (MDGs) and their proposed replacement with more development goals after 2015. As most readers will know, the MDGs are a set of human development goals, with subsidiary targets and indicators, formally adopted by the UN in 2000.

There is pretty broad agreement that progress towards meeting the MDGs is partial and uneven – some of the goals have been met or look very likely to be met, in some countries, while other goals (such as the target reduction in maternal mortality) may not. Asia, especially East Asia, has done better than Sub-Saharan Africa.

However, applying the results agenda to the MDGs is not simply a matter of asking whether the goals will be met. Rather, it is about asking whether the goals have been met as the result of the MDGs having been adopted. The purpose of having high level goals, including any that come after the current MDGs, is to create political will, the mobilisation of resources, policy change and delivery, all of which should bring about a positive change relative to what would have happened in their absence.

Many in the aid world would say that, of course, the MDGs have had a major impact, and that it is absurd to even raise the question. However, a rigorous assessment of the evidence suggests that it is actually quite hard to make a strong case.

mdg-iconsFirst, the evidence that the MDGs may have made a difference is, at best, mixed. The most comprehensive and rigorous independent assessment is by Andy Sumner and Charles Kenny for the Center for Global Development. They look for significant differences in outcomes and impacts before and after 2000, when the MDGs were adopted.

The clearest effects were on aid levels (which are not an ultimate impact but an intermediate outcome). Compared with the previous decade, official aid increased in the post-2000 period, but not as a proportion of rich country GDP. More aid went to the poorest countries, including to Africa. There was a small shift in the share of aid going to the social sectors, on which the MDGs tend to focus, and this happened soon after 2000.

There is plenty of evidence of the influence of the MDGs on policy discourse, if this is measured by mention of the goals or their presence in donor policy documents, PRSPs and developing country government goals. However, the effects on actual policy change are less clear. Sumner and Kenny find it “hard to detect a trend” in low income country government spending on health and education. They also find no trend in the quality of developing country policy making, as measured by the World Bank’s Country Policy and Institutional Assessment ratings.

On the actual impact indicators themselves – such as income poverty, malnutrition and mortality rates, educational enrolment etc – Sumner and Kenny’s most relevant assessment is whether progress was faster pre- or post-MDGs, and whether progress post-MDGs has been faster than what would have been expected based on past trends. Again, results are inconclusive. The data “suggest that in no case is there an obvious sign of a significant break towards faster progress since 2000. Nonetheless there has been somewhat faster global progress on income, primary completion rates, child and maternal mortality over the post-Declaration period”. A study by Fukuda-Parr and Greenstein of country level data gives a similarly mixed picture. The comparison with predicted rates of progress based on historical analysis implies slightly better than expected outcomes post-MDGs on primary education and gender equality in education, but worse on maternal mortality.

Second, there is the problem of attribution. As Sumner and Kenny put it, “even ignoring the very limited evidence of faster progress since 2000 in the average (unweighted) developing country, it is a considerable step from ‘more rapid progress’ to ‘the MDGs caused more rapid progress’”. In other words, bilateral aid may have increased somewhat, some indicators have improved, but how do we know that these changes are due to the MDGs, and not to some other factor?

It is not possible to know what would have happened in their absence. This is not a case of running randomised controlled trials across a number of interventions. And as Richard Manning points out, it is hard to separate out the potential effects of the MDGs from the environment that produced them.

In some areas, such as vaccination or primary education enrolment in sub-Saharan Africa, the links between the MDGs, the mobilisation

      Remind me, who's 'we' again?

and focusing of additional aid, and subsequent impacts seem convincingly close. But in others, the links seem less plausible, especially where there are also good alternative candidates that may explain changes in indicators better than the effect of MDGs. Poverty reduction in Asia, for example, is more likely to have been driven by the extraordinary period of sustained economic growth in China, than by a set of UN targets. It is also plausible that China’s growth will have pulled along a number of countries in its wake, including commodity exporters in Africa. The rapid reduction of poverty in Brazil is due in part to the development of social safety nets such as the Bolsa Familia. When I recently asked Romulo Paes de Sousa, Brazil’s former Deputy Minister for Social Development, and closely involved in the design of the Bolsa, whether it was the result of the MDGs, he dismissed this immediately, saying it was the outcome of a domestic debate that emerged from the minimum wage.

Yet despite the lack of clear, strong evidence of the impact of the Goals, and the difficulties of attribution, the MDGs are routinely hailed as a success. Most importantly, this success is asserted in the context of discussion about a new set of post-2015 development goals. When it was announced that David Cameron would be co-chair of the UN High Level Panel on post-2015 goals, Andrew Mitchell hailed the “huge progress that has been made through the Millennium Development Goals” and “the successes of the current goals”.

When challenged with the point that attribution is often difficult in cases such as these, and that you can’t compare counterfactuals, many proponents of the results agenda recognise the problem. However, their argument is that, in such circumstances, it is the duty of those proposing any particular approach to be explicit about their “theory of change” – that is, be explicit about the full chain of causal linkages you think is going to run from your intervention (here adopting international goals) and the impacts you hope for. Identify your assumptions. Assess the evidence for and against those assumptions, and weigh up the risks.

If done properly, this wouldn’t be just about ticking a box. The point of such an analysis should be to help understand how to make such goals more effective. It should look at why some goals were easier to meet than others (gender equity on education as opposed to access to clean water or reductions in maternal mortality) and in some countries than in others. It should look in a systematic and rigorous way in how the goals were used (or not used) and where there is evidence that they failed to lead to a result, explore alternative, potentially more effective “pathways to impact”.

The point here is not that the MDGs are somehow a bad thing, or that there should not be a new set of goals. In any case, it is not seriously in question that there will be further goals post-2015, of some form. Too much political capital has been invested in them for this to be the case, regardless of the ambiguity of the evidence base. The results revolution will not change the reality that some policies and initiatives are often inevitably driven by more than evidence, and that politics plays a major role.

Nor am I advocating a view that we should not try to measure impact or wrestle with the problem of attribution. What I am saying is that I think the example shows that really, really applying the agenda of results and evidence-based policy consistently and rigorously can be more difficult than the current discourse acknowledges.

Matthew Lockwood is a Research Fellow at the Institute of Development Studies at the University of Sussex. From October 2012 he starts work on a four year project on innovation and governance in the UK energy sector. 

11 comments

  1. For me, part of what makes the MDGs interesting is that they allow us a specific framework against which to measure progress (or a lack thereof). So rather than them needing (or aiming) to increase the success of development efforts, they are useful for other reasons. They say that measuring children doesn’t make them taller, but it does tell us how tall they are.

  2. Sure, evidently sluggish progress to meet MDG5 in Africa suggests that such global commitments have limited impact. But my own qualitative research on the politics of maternal health in Zambia suggests otherwise. Certainly, the birth of the MDGs did not trigger an amplification of efforts to address maternal mortality. Safe motherhood has long been overshadowed by an earlier preponderance of donor-driven attention to HIV/AIDs and malaria. Focus only sharpened more recently, with the realisation that Zambia was lagging behind the rest of the world, unlikely to meet agreed international targets. The following selected extracts from interviews with senior managers in the Ministry of Health indicate their keeness to avoid the embarassment of trailing behind other countries making rapid progress:

    ‘In last year’s budget, there was a big increase. We looked at MDGs and felt it necessary to support MCH. We recognised we may not achieve MDGs’.

    ‘We are getting towards the MDGs deadline and our graphs have not been very good. We haven’t done much in mother and child health [hence recent increase in funding for reproductive health]’.

    ‘MDGs – we have to be part of the world… We found that we are not on track. The commitment has been there but it was enhanced by the MDGs’.

    ‘We are a stable country; to be put in that place [sharing rankings with conflict-afflicted states] is a shame’.

    ‘If you look at our performance towards the MDGs, the only indicators that are quite a challenge are MDGs 4&5, so it prompted us to say, ‘What can we do?’. To meet the targets we need to do some extraordinary things’.

    International benchmarking and consequent awareness of comparatively poor national performance appears to galvanised a shift in national health priorities. Interviewed district health managers and health workers universally maintained that there is increasingly strong top-down pressure to improve their maternal health indicators.

    The MDGs do seem to have played a major role in focusing attention on safe motherhood, albeit tempered by local concerns and constraints. I think the above extracts indicate the importance of domestic perceptions of global commitments; particularly the desire to be ‘developing’ (i.e. achieving shared socio-economic targets) at par with other countries, not lagging behind. The dawning prospect of failing to successfully partake in a global development agenda seems to have fostered greater support for maternal health, within the Ministry of Health – an institution already working towards this objective.

    These narratives seem to highlight the importance of ensuring that the MDG 2.0 deliberation process ensures that future goals reflect shared priorities of member states, for it is only with such commitment that we are likely to see implementation.

  3. P.s. so my short answer to the question of whether it’s possible to apply the results agenda to big global agendas is YES, so long as it’s not just a numbers game but instead allows for qualitative methods, so that we ask various policy-makers and service providers why a certain development issues come to be prioritised.

  4. Alice

    The extracts from the interviews in Zambia are very interesting. Most important is the attribution of a budget increase [on maternal health services?] to the MDGs. Sounds like the MDGs had an impact here, but only quite late in the day – any thoughts on why that should be?
    Looking at such cases in a systematic and rigorous way is precisely what I think should be done. But also remember that the MDGs had the biggest resonance in the aid world, and much of the change in poverty, health outcomes, etc. came in non-aid dependent countries, includign India and China. We need to understand the role (or lack of role) of the MDGs in these cases too.

  5. Matthew,

    Yes, budget increases are partly attributed to the MDGs. So too is support for new policies, like performance-related pay.

    The looming 2015 deadline may have galvanised a degree of urgency of late. International benchmarking of MMR trends also seems important.

    But there are also other factors enabling a shift in attention:

    (1) Global health priorities. There seems to be a shift in internationl attention to maternal mortality reduction. Flagging progress towards this target was explicitly mentioned at the London Family Planning Summit, as well as in DfID’s increased funding for such activities in a number of countries. The external environment has thus become more conducive. Previously, even if civil servants sought to promote maternal health, they found it very very difficult to secure funding.

    (2) A critical mass of high-level health advocates in the Zambian executive, following last year’s election. (The President is a former minister of health, the first lady is an Obs-Gy doctor, a number of ministers are doctors). Owing to long-standing personal experience in the health sector, these now powerful individuals have particular concern for these issues. The health sector has been identified as a priority and budget allocation has increased by 45%.

    But even if MDG 5 has recently played some role in galvanising attention, I still find it difficult to work out what this means for developing a post-2015 agenda.

  6. Important points and indeed a relevant post. Thanks

    Just a lunch time quibble, isn’t “.., it is hard to separate out the potential effects of the MDGs from the environment that produced them.” quite the understatement?
    Many great thinkers, at least since the old Greeks, have tried and haven’t yet really, really solved the structure/agency problem.

    You write that: “applying the agenda of results and evidence-based policy consistently and rigorously can be more difficult than the current discourse acknowledges.” Does that imply that it’s merely ‘difficult’ or might the needed acknowledgement be more fundamental in nature?

  7. In reply to Alice, I would say that one lesson might be to put more effort into thinking about how different post-2015 goals get promoted and what that means for design – e.g. fewer might be better as easier to pay attention to a smaller number simultaneously. Also invest right from the start in better data (in the early 2000s DFID was saying it couldn’t report on progress in maternal mortality because the data was so patchy and so out of date).
    Again, my point is that if we had a lot more evidence on how the use of the MDGs worked, where that use was succesful and where not, we’d be in a much better position to design new goals better. If we can’t learn from past experience on something this big, then the development industry is in a sorry state.
    In reply to Søren, I think it’s all about having a reasonable amount of evidence for how things work and the impact they might have. You obviously can’t asssess the impact of the MDGs in the same way that you can assess the impact of a new drug for cancer, but that doesn’t mean you should just throw your hands up in despair. In the case of the MDGs, I’d like to see more systematic investigation of evidence for precisely the links that Alice’s interviews related to – between references to the MDGs as clearly distinguishable drivers of policy change, policy outcomes and then final impacts on the ground for poor people.

  8. Do we not also need to think about what issues the focus on MDGs has taken our attention away from? Rights, for example, and commitments such as CEDAW, the Beijing platform for Action, or the Convention on the Rights of the Child…

  9. Matthew, I think that it is important to raise these issues about the impact of MDGs especially given that people are now considering postMDGs. I think that one problem in assessing the impact of the MDGs is that they did not replace a period of zero commitments. The MDGs grew out of the goals of a number of international summits and then were aggregated and re-dated by the DAC as the International Development Goals. In many ways we really just changed the label of the commitments already existing in 2000. It is likely that post MDGs we will also have some form of commitments (even if they are not integrated as one set of goals) and development advocates, short of influence and advocacy weapons, will seek to use these to promote development action just as they have with the MDGs. Given that perhaps it doesn’t matter that much what the commitments are, or whether they are integrated like the MDGs.

  10. Hi Duncan,

    Love your blog!

    Don’t love the new red background though- it is so tricky to read! Is it supposed to be like this or are there further changes coming?

    Thanks!

    Victoria

  11. Hey Matthew thanks for your article i never knew much about the mdgs but i now have an idea since its the year 2015 coz wev bn givn an assignment on the impacts of mds in the education sector in Zambia.thanx once more

Leave a comment