Empowerment and Accountability in Messy Places. Need your advice on Nigeria, Pakistan, Myanmar and Mozambique.

June 8, 2017

How will we know if the SDGs are having any impact?

June 8, 2017
empty image
empty image

As long time readers of the blog will know, I’ve been a Sustainable Development Goal (SDG) sceptic since long before they were evensdgs agreed. However, I’ve been hearing a fair amount about them recently – people telling me that governments North and South, companies and city administrations are using them to frame public commitments and planning and reporting against them. So maybe it’s time to take a second look.

My problem with the SDGs is not with the subject matter – all very creditable – or the number of targets, which bugs some people, but not me. It’s the design, and in particular, the lack of analysis on how they could have political impact. For me, the key question for any international instrument ought to be traction – will this or that convention, undertaking etc (of which there are hundreds, if not thousands) influence the day-to-day behaviour of governments, sub national bodies, private sector or others and how? And yet weirdly, this was never raised during the design of the SDGs, which instead were dominated by adding more topics to the Christmas Tree of SDG issues, and long discussions about indicators, data and metrics.

Even odder, this question has barely been asked of their successor, the Millennium Development Goals. In a classic fudge of causation

Causation v Correlation 101

Causation v Correlation 101

and correlation, the MDGistas said  ‘look, extreme poverty has halved, the MDGs are a success!’ Yet much of that reduction was down to China, and no-one can credibly claim that it was the MDGs that got the Chinese Communist Party out of bed every morning to transform the Chinese economy.

There are a few exceptions I’ve managed to collect in response to repeated rants: Alice Evans showed how MDG5 on Maternal Health prompted the Zambian government to take action for fear of reputational damage;  May Miller Dawkins wrote a nice paper on what the SDGs could learn from international environmental agreements in terms of design/traction. Moizza Binat Sarwar studied MDG implementation in five low and middle income countries (Indonesia, Liberia, Mexico, Nigeria and Turkey), largely by interviewing staff of relevant ministries. When Columbia University’s Elham Seyedsayamdost surveyed 50 countries’ implemention of the MDGs, she found that whether the goals were reflected in plans or not, they did not have any apparent influence on how governments spent their money. All great stuff, but hardly a body of research that matches the importance of the issue – a very odd lacuna indeed.

Now the same thing looks to be happening again. Lots of talk of monitoring and reporting against the SDG indicators, and as far as I can tell, no attempt to establish whether the SDGs (rather than other factors) are responsible for changes to those indicators.

So what would an ‘SDG politics watch’ look like?

First the data: we need to know which bodies are using the SDGs in their planning, budgeting, reporting etc. Maybe we could crowdsource this – governments, NGOs, academics etc can all upload evidence to some kind of WikiSDG so we can start to build up a picture of formal commitments

But that is nowhere near enough – how to distinguish between lip service and genuine traction? And why do the SDGs get traction in

Traction of Lip Service?

Traction or Lip Service?

some places and not others? That’s where the academics could get stuck in. Alice Evans at Cambridge is issuing an open invitation to supervise a good PhD student on this topic. In her words ‘It’s not just about whether governments use the SDGs in their policy documents, but their attitudes towards these goals: do they dwell on issues they previously didn’t because they are concerned about regional benchmarking? Do civil servants in meetings spend time looking at SDG 9.ii (whatever that is), and put pressure on provinces to improve this indicator, or just whizz through it. Do parliamentarians raise these issues?’

Why does this matter? Because understanding when/how an international instrument has traction ought to be the starting point for tweaking SDGs, and designing future instruments that work better. If for example, we were to identify regional rivalry as a key driver in state action, we would put much more emphasis on reporting in regional league tables. If the key to impact is civil society picking up and using the instruments, then (as in the SDGs) it becomes even more pressing to involve CSOs in monitoring and reporting, as well as initial design. And so on.

Thoughts?

Previous rants on this topic include one paper and lots of blogs:

Paper: How Can a Post-2015 Agreement Drive Real Change? The political economy of global commitments

Posts:

Hello SDGs, What’s Your Theory of Change?

How will the SDGs differ from the MDGs?

The SDGs are just getting interesting – what needs to happen next to make them have impact?

SDGs in Zambia

14 comments

  1. What might be interesting is to see the extent to which the SDG agenda can effectively be pushed in developing nations by bilateral donor agencies, if at all…!

  2. Thanks, Duncan.
    I would love to see a piece on data quality and the MDGs/SDGs. How reliable/ accurate are measurements of MDG achievements? Which SDG indicators can realistically be measured with any meaningful degree of accuracy? And how does reliability vary geographically?

    Any takers?

    1. Lots I would expect – there’s a big constituency for gathering data on health, water, education etc, just as there was for the MDGs. What is missing is any systematic research on the politics of SDG traction – that’s the real gap, IMO

  3. Thanks for keeping the SDG discussion alive. Indeed, it’s rather weird in the first place that we now have “Sustainable” Development Goals, which suggest that the first MDG’s where not supposed to be sustainable? Or is this the first lesson to be learned from the MDG lobby failure?

    So we make some progress at last in the way we think about development; it should be sustainable? Which brings us to the key question:”What exactly do we mean by “Sustainable”? For instance, is a community handpump in Africa that is maintained with the donations of good willing people in the USA sustainable, because people will always give to Water Charity NGOs? So it does not matter if the community gets a durable pump that they could maintain themselves, or a fragile one that is more expensive to maintain, because the USA donors will provide anyway. One could argue both sides.

    Next question is “What is development?”. It’s not just about “Poverty Reduction”, because poverty is rather relative. For instance, wages in China are lower than in the USA, but does that mean that China is less developed? We tend to look at these issues from our own perspective, and we judge things based on our own perception.

    Fact is, that the MDGs and now the SDGs are used as an “excuse” to start projects and to spend a lot of money for “Poverty Reduction”, because it is said that we need to go “From Poverty to Power”.

    But i think it remains to be seen if this “Fighting Poverty” attitude is the end goal. It seems indeed more an excuse to justify all kind of initiatives to spend (free…)donor money in a black box, without any realistic monitoring of the impact or “Value” that is created for these poor. Most of the time, it’s the implementing organisation and it’s employees that profit the most.

    So one could argue, that the money is not well spend, however, to me the key question (or problem) is not so much what they actually do with the money, but where the money comes from. So it’s not so much about FTM (Follow the Money), but more WITS (Where is the Source) and what the Source wants to achieve.

    The sources, the public donates in good faith, and (Democratic) Governments that represent tax payers also donate because their voters want them to do so. By the way, do you know of any non-democratic government that has a donates as well?? Companies donate as well, but most of them mainly to boost their “public profile” to sell more and make more profit, and not so much from an empathic attitude to lift people out of poverty.

    But at the end of the day, if their donations (and the MDGs and SDGs are basically all about donations…) comes in the hands of people that love to spend the money of others without taking any personal responsibility for it… , you end up with a weird situation, because indeed, how can we know if these donations make any difference, and how sustainable is this?

    Just a simple example, how this works in the water sector MDGs, for instance in Benin, in the framework of the MDGs, the Dutch Government donated some 7 years ago 1.200 handpumps to the public with a 10 year functionality warrantee, but most of them don’t work anymore, because they break down all the time, so after a few years, maintenance became very expensive and spare parts are not available. Now they blame it to the people, that they don’t want to pay for repairs. In fact, it is seen as a national disaster, but nobody takes responsibility for it, all stakeholders point at each other, but the people are left without water again.

    This is a typical example of thousands similar cases. What it indicates is even more worrying: It means that even in case monitoring of the impact is rather simple ( a pump works or does not work…) this failure to make an impact seems not important to change the policy or to give less funding. On the contrary, in the recent High Summit WASH meeting in New York, the NGOs just stated that “More funding is needed”, instead of asking themselves questions about their impact and efficiency.

    Since we have Donald Trump as president of the USA, it seems to be normal to simply deny the truth and get away with it?

    Conclusion: To have an impact, it is more efficient to work with your own money and not with a “free gift” that you can spend on whatever, just because of your “Good intentions”… And secondly, if it’s your money, like an investor, you want to follow up, before you give for the second time.

    Albert Einstein said: “The definition of insanity is to do the same thing over and over again, and expect different results” …. I’m afraid this also counts for the MDGs and SDGs projects; as long as they spend other’s donation, there will be no “other” (sustainable) results…

  4. For anyone interested in this there is a network called EVALSDGs, which aims to “inform, support, measure and assess development efforts around the SDGs”. It was set up in 2015 by the UN Evaluation Group and the International Organization for Cooperation in Evaluation. More info here: https://evalpartners.org/evalsdgs/about.

  5. I absolutely agree with this. There’s a massive gap when it comes to thinking about the issue of additionality: what is happening because of the SDGs that wouldn’t have happened anyway?

    An additional complication is that even if a government changes its priorities due to a particular goal or target, how do we know its not taking attention and funding away from another goal? You don’t have to think about that so much when you’re pushing a single-issue campaign.

    I think it would be more fruitful, at least initially, to look at changes in language and norms. For instance, can we see if the shift in education from thinking about enrollment to learning been accelerated by the language used in the SDGs?

  6. Kudos to Alice Evans for paying attention to issues of on-the-ground implementation–it’s generally overlooked by those interested in development, but it’s crucial if systemic change is to be successful.

  7. Hi Duncan,
    In 2011, CAFOD commissioned a study amongst civil society leaders in the global south to ask whether they found MDGs useful. (100 Voices http://cafod.org.uk/content/download/553/5570/version/2/file/Policy_Beyond-MDGs_100-voices-report_English-full-report.pdf ). Despite their limitations, the overall answer was that the MDGs provided a language, a hook, an entry point for the civil society to engage with government and coordinate at different levels, and most people felt that it helped achieve important objectives. This does not proof causation but at that time, CAFOD felt it was enough to mobilise and ask for another global development framework – this time designed in a more participatory way. The UN was not very proactive and thus a civil society network BEYOND2015 was founded to push for a new agenda. From 2012, in the process of getting to the SDGs I witnessed many important conversations on issues of development between different actors that weren’t happening before. These conversations are in itself an achievement. As countries defines national implementation plans, hopefully more interesting conversations will take place at different scales.
    Recently, Felipe Castro, Director of Monitoring and Evaluation of Public Policies at the National Planning Department of Colombia, gave an interesting presentation (The SDGs: a tool for peacebuilding in Colombia) at UCL explaining how Colombia is aligning national and regional strategies with SDGs in a very interesting way, focusing on different SDGs according to local priorities.
    (the talk is available here: https://www.mixcloud.com/dpuucl/the-sdgs-a-tool-for-peacebuilding-in-colombia they used good visual which are not available).
    We can’t easily measure the impact/added value of SDGs as a framework, attribution is difficult. Even without the framework countries would have some sort of national development strategies.
    SDGs in themselves will not be responsible for progress, national and local policies and their implementation deliver change. However, SDGs are an important common language which frames the path towards change and enables a number of actors to participate and converge in the process of defining local priorities.

    1. Thanks Andrea, I remember the study. What might have been more interesting is to ask civil society leaders ‘what international instruments do you find most useful?’ and then seeing whether they mentioned the UN, the ILO, regional treaties or whatever. Asking ‘do you find the MDGs useful?’ is a very leading question!Then a follow up question could have been, and what is it about those instruments that make them relevant – then we could have tried to feed that into the design of the SDGs. But no-one ever asked…..
      The Colombia case is interesting – definitely a candidate for WikiSDG!

      1. Hi Duncan,
        Agree the 100 voices was leading.
        With lots of limitations what we tried to do with the Participate initiative and the COMPASS studies ( http://cafod.org.uk/content/download/11319/89078/version/10/file/CF-Compass_report.pdf ) was very much along the lines of what you suggest. The participatory research with women and men living in poverty was asking how change happened in their lives in the past 15 years, roughly the MDG period but the questions was open. These studies contributed to the design of the SDGs.
        I know this is slightly different but Participate also worked on a review of how other policy framework embedded the views and perspectives of those living in poverty and what worked. (http://participate2015.org/publications/what-do-we-know-about-how-to-bring-the-perspectives-of-people-living-in-poverty-into-global-policy-making/ )
        This is to agree completely with you on the need for more research and data on this, but wanted to point out that some initial steps in that direction were made.

  8. Hi Duncan
    I’m not an M&E specialist and I haven’t seen any systematic examinations of the impact of the MDGs (and I would love to do so!).

    What I can draw on is my experience as an auditor in developing countries (especially Vietnam and Laos) for the last 20 years. In the last 5-6 years, discussions about development have tended to use the MDGs as a framework: as a result, all ideas about development (whether from international aid agencies, governments or local CSOs) were created and assessed in relationship to the MDGs. This was particularly noticeable in Laos, where the reduction of the impact of unexploded ordnance (UXOs) was phrased as a 9th MDG, specific to the Lao PDR. Part of this was driven by the UN and other aid agencies, but I have also been to meetings where the Lao Minister of Education was using the results against MDG targets to identify the specific priorities of his department.

    I realise that the plural of anecdote is not data, and would therefore be very interested in seeing a formal study of the impact of MDGs.

    Best wishes
    Mark

Leave a comment

Translate »