Links I Liked

May 12, 2015

Can aid agencies help systems fix themselves? The implications of complexity for development cooperation

May 12, 2015

Which bits of advice do developing country decision makers actually listen to? Great new research

May 12, 2015
empty image
empty image

Another interesting feedback loop in the aid system: a new report, The Marketplace of Ideas for Policy Change summarizes a

The marketplace

The marketplace

survey of 6,750 policymakers and practitioners in 126 low- and middle-income countries to find out which of the innumerable bits of advice and analysis churned out by aid agencies, international organizations and NGOs actually influence their work.

What’s most alarming is how original this is – I am still looking for a similar exercise on the MDGs, which might have made the whole post-2015 process less of a donor-driven gabfest. Right now the SDG wallahs should be reading this paper and asking – what kind of reporting structure might actually influence government behaviour?

A lot of the conclusions had me scribbling ‘NSS’ (No Shit, Sherlock, since you ask) in the margin: influence is to some extent a function of how much the organization spend on marketing? Wow, who would have thought it! But some are worth noting, and a few are genuinely surprising. Here’s some that jumped out for me:

Advocating on family/gender policy has the best chances of success, anti-corruption the worst: ‘The family and gender policy domain is one characterized by relatively high levels of external assessment influence, relatively low levels of (net) domestic opposition to reform, and reasonably good odds of success in reform implementation. This finding suggests that external efforts to encourage and support family and gender reforms may be particularly fruitful. Anti-corruption stands apart as the policy domain with highest level of (net) domestic opposition to reform and the worst track record of reform implementation.’

light-bulbSimilarly, ‘The democracy and decentralization policy domains appear to be least susceptible to external influence at the agenda-setting stage.’ Yup, that’ll be power and politics, then (see NSS).

‘External assessment influence is strongest at the agenda-setting stage of the policymaking process.’ – i.e. get in early in the policy funnel, help define problems etc. Compatible with the whole PDIA approach.

‘Paying attention to “nuts and bolts of government” may result in greater assessment influence.’ Please note, all campaigners – think about advocating on boring but important stuff like data collection, staff training, info management.

‘Country-specific diagnostics generally exert greater influence than cross-country benchmarking exercises.’ OK that’s interesting – maybe publishing regional league tables, naming and shaming etc is not the best way to influence government (but perhaps tables comparing regions or cities is?)

‘External assessments that rely on host government data are more influential.’ Use government data, rather than your own, or some international body’s, and you are half way to getting buy in.

‘The longer an assessment’s track record of publication, the more influential it becomes vis-à-vis others.’ International organizations tend to have much more staying power than INGOs, producing annual reports on this or that, which slowly accumulate brand awareness and impact. By hopping from issue to issue, INGOs may keep the media interested, but they sacrifice impact.

‘Neither incentives nor penalties seem to easily explain assessment influence.’ Oh good, we don’t have to pay people to read our reports.

‘Prescriptive assessments appear to be slightly more influential than descriptive assessments, and decision-makers in the developing world seem to want more, not less, specific policy guidance.’ OK, that’s definitely a challenge to all the complexity wallahs and Doing Development Differently crowd who say that outsiders should focus on highlighting problems, not suggesting solutions (which need to be designed by local actors).

‘Assessments were influential because they promoted reforms that aligned with the priorities of national leadership.’ A ‘working with the grain’ argument that it’s best to try and influence ongoing processes rather than start new ones.

‘Senior government leaders and their deputies engage with external assessments in different ways. One potential interpretation of this finding is that leaders, mindful of their domestic audiences, project strength in the face of external pressure, while their deputies work behind the scenes to secure material rewards from donor agencies and international organizations.’ Love it!

Some important findings on the limits of external influence:

‘The picture that emerges is not one of governments being cajoled or coerced into pursuing reforms that align with donor priorities, but rather that governments pick and choose assessments based on whether they advance domestic priorities.’

‘External sources of analysis and advice rarely help to neutralize opposition to reform or build coalitions in support of policy change.’

And on the range of country types:

‘Some of the most successful reformers “go-it-alone” and shield the domestic policy formulation and execution from external pressure (e.g., Rwanda and Ethiopia), while others rely more heavily on external sources of analysis and advice (e.g., Liberia and Georgia).’

Some weaknesses:

It’s just a survey – so no in depth interviews, focus groups etc to dig deeper, which I am sure would have produced further insight.

Massive blind spot on critical junctures – policy makers everywhere ignore advice until they need it, which is often after a scandal, crisis or obvious failure in previous policy. Building detailed timelines with decision makers would have revealed much more about how they take up policy advice and analysis at such moments.

Also nothing on the role of people and institutions outside government in persuading the state to adopt particular pieces of analysis – coalitions, insider-outsider alliances etc. This is a world where civil servants and pols read or don’t read/listen to reports – in real life, things are a bit more complicated than that.

The good news is that the AidData lab that conducted the survey plans to repeat it (and so accumulate influence, presumably). Now can someone apply this approach to the SDGs, please?


  1. Great post – thanks Duncan. I agree that more in-depth research could be of great value.

    A good issue for focus groups might be to explore the apparent contradiction between the statements below, on the (in)effectiveness of policy advice and the desire for more of it.

    – ‘the picture that emerges is not one of governments being cajoled or coerced into pursuing reforms that align with donor priorities, but rather that governments pick and choose assessments based on whether they advance domestic priorities’
    – ‘external sources of analysis and advice rarely help to neutralize opposition to reform or build coalitions in support of policy change’


    – ‘decision-makers in the developing world seem to want more, not less, specific policy guidance’

    1. This doesn’t seem to be a contradiction to me – I read it as implying that in considering whether an assessment is useful, decision-makers filter it through a criteria of whether it helps advance domestic priorities as that decision-maker sees them. Therefore, more specific guidance is better, because that specific guidance advances those domestic priorities further (probably by agreeing to an important extent with what the decision-maker already thought was relevant or important).

      In other words, because the analysis and advice doesn’t go into a blank slate, it is not purely applied to understand or think about a problem, but to advance existing political preferences. A political actor can make more hay out of a prescriptive piece of advice because it is an outside authority backing a particular action they wanted to take; a descriptive analysis might help to frame an issue but do less of the lifting of moving change forward. Their preference for specific policy guidance could be because of how they will use it to advance an agenda (not just how helpful they found it in thinking about an issue they hadn’t considered before).

      I’d suggest that this is an important reason NOT to only provide prescriptive advice – it can play into an existing conversation over what to do about an issue in ways that are less fully locally-owned, precisely because it privileges one side of an existing local debate. Part of the value of what is laid out in PDIA/DDD approaches is the importance of the process, of forging consensus locally going into a policy-experiment reform effort, and that process and consensus have independent value from getting actors to move on a reform. Presumably outside advisory groups care both about getting decision-makers to make certain specific decisions and to forge their own local consensus with other local actors around what is important; the balance of prescriptive/descriptive analysis might then depend on whether which of those outcomes is accorded greater priority in a given instance.

  2. Thanks of this. Great to see such a sector-wide piece of policy M&E.
    Having had a quick look through it, the report doesn’t seem to draw a distinction between the International organisations and national civil society which are generating ‘external assessments’. As it highlights elsewhere, some international sources – such as the World Bank- are much more influential than others. The same may be true of internal constituencies since all governments (however repressive) have to address domestic interests. An example of this is church advocacy in Uganda and South Sudan where church sourced reports and policy statements on government policy often address uncomfortable issues but are far harder to ignore or dismiss as foreign hectoring, especially in these post-colonial states.

  3. Thanks for sharing. A very interesting piece of work that should help to spend funds on diagnosis in a more efficient way in order to really produce policy changes. It could definitely be complemented with the studies/analysis on how decision making takes place, since it is normally not based on the availability of objective information that could guide a rational analysis. I recall a previous post where you were talking about influence and this equation should definitely be strengthenned: policy changes= information + advocacy + capacity building + …

  4. Thanks Duncan!

    Reminds me of this excellent paper by ODI saying that

    Policy-makers face difficulties using research-based evidence because of the ‘Five Ss':

    Speed – they have to make decisions fast;
    Superficiality – they cover a wide brief;
    Spin – they have to stick to a decision (at least for a reasonable period of time);
    Secrecy – many policy discussions are held in secret; and finally,
    Scientific ignorance – few policy-makers are scientists, and they may not appreciate fully the scientific concept of testing a hypothesis.


  5. This is worth amplifying:
    “‘External assessments that rely on host government data are more influential.’ Use government data, rather than your own, or some international body’s, and you are half way to getting buy in.”

    For the Open Government/ Open Data crowd, this suggests a very practical and beneficial focus: work with governments to gather better quality data, and to make the often-incomprehensible data they *do* have easier to access and work with. Bonus: resulting recommendations will have higher sticking power!

  6. The assumption here is that all donor advice is good and countries govt. should play passive and accept it.

    As a senior policymaker in Pakistan, I saw a lot of poor quality reports that we rejected to the dismay of donors. Their programs when reviewed were totally incoherent. Thru consultants mostly if poor quality. Should we listen to them with no quality check.

    Moreover, there is now serious capacity in-country. What roe do they play in donor government dialog. The survey here ignites this aspect. Do the donors listen to these domestic thinkers. In my experience donors see them as competition and try to marginal use them.

    See my posts in the issue of aid.

    Should these views be taken into account?

  7. its a great post Duncan. It has helped me frame the opening lines of some remarks I need to make to senior government leaders on Wednesday morning. The cartoon is wonderful too. It will be my backdrop, but I need to add, verbally, a 3rd option…pleasant truths. I am sure that governments do respond to praise for things done right, no? Even if this praise is a devise to reduce resistance to subsequent tough messaging.

Leave a comment

Translate »