If you think knowledge is expensive, try ignorance. Smart new job in Oxfam’s research team

March 23, 2016

What did trade campaigns achieve? Plus reinventing Robert Chambers & changing aid narratives: some Berlin conversations

March 23, 2016

Payment by Results hasn’t produced much in the way of results, but aid donors are doing it anyway. Why?

March 23, 2016
empty image
empty image

I recently attended (yet another) seminar on the future of aid, where we were all sworn to secrecy to allow everyone

Simples. Or is it?

Simples. Or is it?

(academics, officials etc) to bare their bosoms with confidence. So I can’t quote anyone (even unattributed – this was ‘Chatham House plus’).

But that’s OK, because I want to talk about Payment by Results, which was the subject for my 10 minutes of fame.

When reading up I was struck by the contrast between how quickly PbR has spread through the aid world and how little evidence there is that it actually works. In a way, this is unavoidable with a new idea – you make the case for it based on theory, then you implement, then you test and improve/abandon. In this case the theory, ably argued by CGD and others, was that PbR aligns incentives in developing country governments with development outcomes, and encourages innovation, since it does not specify how to, for example, reduce maternal mortality, merely rewards governments when they achieve it.

Centre for Social Justice, October 2012

Centre for Social Justice, October 2012

Those arguments have certainly persuaded a bunch of donors. The UK government website says that this ‘new form of financing that makes payments contingent on the independent verification of results is a cross government reform priority’. DFID called its 2014 PbR strategy ‘sharpening incentives to perform’ and promised to make it ‘a major part of the way DFID works in future’. British Prime Minister David Cameron waxes lyrical on the topic (left).

Which made the prevailing scepticism of much of my reading all the more striking – see for example Paul Clist and Stefan Dercon: 12 Principles for PbR in International Development, or BOND, Payment by Results: What it Means for UK NGOs, or 3 studies by NORAD (the Norwegian aid agency) .

Clist and Dercon‘s principles set out a series of situations in which PbR is either unsuitable or likely to backfire. For example if results cannot be unambiguously measured, lawyers are going to have a field day when a donor tries to refuse payment by arguing they haven’t been achieved. They also make the point that PbR makes no sense if the recipient government already wants to achieve a certain goal – then you should just give them the money up front and let them get on with it. There’s also an interesting sleight of hand in the argument, as the kind of incentive argument that might work for individuals is applied to institutions, even though it is not at all obvious how eventual PbR payments to governments will translate into improved performance by individual officials.

NORAD points out that if PbR is to be used when you are trying to persuade a government into doing something it

But what if it's a department that gets the bonus, not an individual?

But what if it’s a department that gets the bonus, not an individual?

doesn’t want to do, we already know how unlikely that is to succeed from the whole aid conditionality experience.

BOND finds that PbR contracts with NGOs are plagued by micromanagement and often amount to little more than transferring risk from donor to recipient (no results, no dosh).

Even its originators, CGD seemed pretty underwhelmed at that earlier discussion on PbR, so why has it got such momentum among donors?

The PbR hype cycle seems to follow a well-established pattern in the aid biz, which I call the ‘microfinance syndrome’ (policy entrepreneur comes up with whizzy new idea → massive overselling to donors → disillusion when it fails to produce predicted miraculous results → reduced to niche product as we learn when the new snake-oil might actually be worth applying). At best, it’s a painful, inefficient way to innovate and improve the impact on poor people’s lives. Why not try positive deviance  or venture capitalist style multiple parallel experiments instead?

Then what exactly are these results and who are we measuring them for? PbR pushes project implementation even further towards ‘upwards accountability’- mainly developing country governments collecting and processing results (which can be an expensive business) in order to satisfy aid donors and their political backers and tax payers. To what extent are those results any use for a) learning and improving or b) increasing accountability where it is really lacking – downwards to poor people and communities? My fear is that this will merely create a parallel system of results alongside the kinds of information practitioners need to learn and improve, diverting effort and money and doing nothing for downwards accountability.

Gartner_Hype_Cycle.svgTime horizons are a real problem: If (a big if) you could solve the problem of attribution becoming more attenuated with time, a 30 year PbR contract might indeed encourage innovation, certainly far more than a 3 year one, where there is too little time to experiment and take risks and find a better path to results. In fact, short-term PbR contracts appear to discourage innovation – there isn’t time to learn by failing, so stick with the tried and tested, even if it’s not that good. But there is precious little appetite for longer project cycles – political and management timelines seem to be shortening instead.

Trawling through the interwebs for this piece it looks like a lot of the PbR discussion comes from the health sector – could this be another problematic attempt to import medical thinking into development, like randomized control trials?

Anyway, even though the evidence seems pretty thin that introducing PbR achieves improved results (ironic, eh?), donors are jumping into PbR contracts with gusto. Why is that? Conspiracy theorists/political economists please form an orderly queue.

Update: Excellent comments thread below, with some useful updates on latest experience with PbR


  1. Brilliant, fascinating post. As to why cash for delivery is popular with donors, two ideas strike me.

    (1) Maybe there is evidence of effectiveness? E.g. maternal health care improvements in Rwanda and Zambia.

    (I appreciate there are problems with both. And doubtless you discussed all this in depth at the meeting. My point is just that there is *some* evidence of effectiveness)

    (2) Perhaps donors have put all their faith in rewarding outcomes because nothing else seems to work? They previously tried telling countries what ‘processes’ to improve but that didn’t improve outcomes (just led to bums on seats but nobody learning etc). Donors want to stay in the game, to maintain the aid project, to believe external interventions can ‘fix’ developing countries, so this is their latest effort…?

    We all want to believe in a ‘fix’. That’s why superhero films are so popular.

  2. Interesting thoughts- and thanks for citing Bond’s paper on PbR. In fairness, in the 18 months since we wrote that, I think we’ve seen some much smarter thinking, practice and willingness to learn – albeit still within the overall cycle from overselling to identifying niches that you talk about. There are still huge risks and it can be done horribly badly: we’ve seen many of the risks we warned about being realised. But it’s also worth noting some of the positives that we now hear from organisations implementing PbR. The focus on results is helping improve M&E and performance, but within the narrow scope of whatever results are defined as triggering payments: get those right and PbR helps; get it wrong and you go off on a tangent (no surprises there). However, those benefits may not spillover into wider M&E systems and learning beyond the project. Tantalisingly, at a meeting hosted by WorldVision last week, we heard of communities who were involved in verification of results from PbR projects feeling that NGOs were much more accountable to them in that PbR project than in other work. There is scope to do this in a clever way to promote agendas like adaptive management, downwards accountability – but it takes a lot of thought.

    1. That’s fairly encouraging Michael, esp on downwards accountability – looks like the transition from oversell to useful tool in some situations is well under way.

  3. Slight tangent. Another example of the ‘microfinance syndrome': ‘women’s football in Burundi offers hope to a battered nation’. http://www.theguardian.com/global-development/2016/mar/23/womens-football-in-burundi-offers-hope-to-a-shattered-nationOften lots of donor enthusiasm for this. As an athlete, I absolutely get that sport can enhance self-esteem as well as being great in multiple ways. But surely misses the much much bigger issue of job creation for the masses. Instead focuses on little project interventions – perhaps because that’s what they’re used to delivering?

  4. If Cameron is so gung-ho on PbR, why doesn’t he fund his own government that way? Surely the same logic applies. I would venture that PbR has an alluring conceptual appeal but when it comes to practice – especially in complex, fragile or low-income contexts – it starts to unravel. Too many unpredictable variables, risk allocation is hugely inequitable (yet name one organisation that has turned down a PbR contact on principle – would love to know that actually), verification of ‘results’ is problematic for all but the simplest interventions. If this incentive based contract modality is as great as it’s cracked up to be let’s test it out in stable, advanced economies and then roll it out elsewhere. How about senior government salaries are tied to PbR for election manifestos? As to why the donors are still lining up to use PbR I would look to the shiny object syndrome, the appeal of anything that has a market/efficiency dimension to it, and importantly, the inability or unwillingness of most in this sector to bite the hand that feeds us and push back against an unsuitable and ineffective delivery mechanism for development assistance.

    1. Risk there is that aid systems end up succeeding or failing collectively. Even with strong communications and accountability efforts, disaster/conflict-affected populations are not necessarily likely to make the distinction between well-behaved Agency A and miscreant Agency D when giving their assessments (having other things on their plate to worry about). Or is this an unfair assumption?

  5. Are there any examples of payments based on customer satisfaction type ratings? So, in humanitarian aid, you could have a bonus for agencies where disaster affected people rate the aid provided highly on measures such as whether it was fairly targeted, delivered with respect to dignity, consulted fully with local people, timely, adequate and appropriate.

  6. I love the juxtaposition of a Chatham House Plus rules meeting discussing the accountability of Payment by Results. I can imagine Duncan’s discussions with his manager:

    Manager: How did the meeting go? Can you give me examples of why it was worthwhile?
    Duncan: I couldn’t possibly say.
    Manager: Did people like your presentation?
    Duncan: I couldn’t possibly say.
    Manager: Why is there a receipt for the matinee at the National Theatre in your expenses?
    Duncan: I couldn’t possibly say.

  7. I’ve just written a (currently being reviewed so not published) paper on results-based payments, which looks at a lot of these issues. One of the things we observe, and which is a bit of a problem with this post, is the assumption that RBP/PBR is THE answer. It’s not, it’s perhaps AN answer, but properly approached is just one of the suite of mechanisms that should be considered in the complex web of interactions and influence between countries. And as much as there are plenty of examples of it being badly done and not working (as with much else), there are also plenty of examples of it working very well, and not just in the apocryphal health type examples.

    The incentives part is really interesting. We found that money is probably necessary for getting a seat at the table, and the PBR process can help to facilitate a better conversation on performance, but has little bearing itself on behaviour change. As one commenter suggested, what about customer satisfaction: well, a very interesting example which needs more follow up is how the process of publicising comparative scores of govts or civil servants can have a marked effect on performacne, even to the extent that no additional funding is required at all to see change happen.

    But, as one person who we spoke to in the course of the paper said, ‘this is political’, and any attempt to see this as a purely technocratic exercise, or see it outside of ‘the diplomacy of development’ is pretty much doomed to see it fail.

  8. Dear Duncan,

    A very good post, and necessary debate. I agree with your points above, aso I will not repeat them.

    Payment for Results touches different issues, all important for the political economy. I will try to build up one by one, so the discussion stays linear instead of complex.

    The first element that is important to understand donors is the concept of staffing. Nearly all donors are under pressure to cut their staff. Having less civil servants is, for some reason, seen as “good” in the developed countries, although it is unclear how rising budgets and cutting staff at the same time can be good for a professional service.

    The results points straight to larger projects, totally unfit for a complex systems approach (fail early, fail often), and enthusiasm, on flimsy evidence, for magical bullets that will let you disburse without actually doing any work: budget aid, Payment for Results.

    Meanwhile we know budget aid works very well to keep governments in power, and not really well for long term governance (Daeton).

    The second element I want to bring is the concept of “transaction costs” this is OECD/World Bank speak for donor staffing. In some circles transaction costs are the difference between what the principal wants and what the agent gives him. In development aid this definition would be not useful as very often the people in power (like most politicians) want to stay in power, while donors want other things like poverty alleviation, etc.
    In OECD language however transaction cost is the hassle in “delivery”. Delivery is getting the money spend. Transaction costs are missions per project, number of projects signed by the minister on both sides, , etc.

    The transaction cost ideology supports large projects, preferably budget aid, because they can be arranged in one big donor meeting, a policy dialogue lead by the World Bank. The WB is in favour of this approach.

    The approach has a lot of problems however:
    – it is clear that a policy dialogue on everything is more colonial than collaborating down the line in projects. If donors don’t have the discipline to fund only projects fitting in the national strategy, perhaps you don’t want them in a policy dialogue neither?
    – It is against all evidence how change happens. It is a very top down approach. Fail early, fail often does not apply. It is democratic centralism, as promoted by Lenin, Mao and the World Bank.
    – Dit I explain already somewhere that the relationship between size of an intervention and the impact is probably not positive, nor linear?

    In this narrative of transaction costs, Payment for Results is King.

    A third element is moral hazard.

    With Payment for Results the donor removes himself from the governance debate. He pays for the results, no questions asked.

    He makes an offer the recipient cannot refuse.

    An example: all over Africa there are refugees, most of the time for generations. Most of the time in countries that are rather under than overpopulated. These camps could be thriving poles for development in the region. If only the host country let them do their thing (work, study) in brief, just naturalised them with a stroke of the pen.

    This would be very good for the host country, but also very good for the donors who pay now for decades to feed the refugees.

    Imagine a scheme payment for results: the donor just offers the President on his account in Switzerland the amount of money they would have to spend to feed the refugees for 5 years. How fast could this pesky administrative problem be solved?

    However, except for the mention of the account in Switserland (money is still fungible, so the location is not consequential) I don’t see a difference with the statement we should pay for results the government would otherwise not want.

    Extra: audits instead of anticorruption. The more I work in the sector the more I feel we just use audits as an excuse for not to tackle corruption. It is clear to everyone, even in a modestly corrupt country that good crooks can find the necessary papers to pass the audit. A balanced approach, where the donor would actively probe the sector for corruption is rare. Rather a stand-alone anti-corruption program.

    Payment for results is another anticorruption program for donors, where the donor risk for corruption is eliminated without eliminating corruption. Nicely done. One donor in Vietnam answered the press after the bridge funded by them collapsed: “this was budget aid, so ask the government”. I am myself not sure whether anti-corruption should be central. However if we don’t put it central, we should have a lot of explaining to do.


    Yes, I am in favour of payment for results. Indeed sometimes, we need the result now. With other questions being less important.


  9. Hi Duncan

    Disclosure – I’m a DFID PbR policy person. I’d like to chip in a few thoughts. Firstly, I love your graph. To help us struggle up that slope of enlightenment, I have an ask – for more care on definitions. If we are to learn and share experiences more effectively, I think we need to move beyond seeing PbR as one homogenous thing which is either ‘good’ or ‘bad’, and start talking as a community about the detail of which theories of change, and which programming approaches, work more or less effectively in which contexts. And most importantly – why.

    There is a real richness of experience being generated. CGD’s ‘Cash on Delivery Aid’ theory of change is well thought out – but it’s only one of a very rich field of innovative PbR approaches. You have namechecked the health literature – but also check out DFID’s Results Based Finance for Low Carbon Energy Access (bit.ly/1T6nRuG) – which is using carefully-designed PbR pull-mechanisms to shift market equilibria for pro-poor pico-solar and other potentially disruptive tech. No kerosene lighting in rural Tanzania by 2020 has moved from unimaginable to feasible – deeply exciting. The degree of PbR attribution TBD. AgResults (bit.ly/1UGDkCS) and Ideas to Impact (bit.ly/1o2Hq6Y) are each using that same PbR pull-mechanism thinking in very different arenas. Alternatively DFID’s WASH Results programme (bit.ly/1RyhOtg) used a PBR approach to drive rapid output delivery at scale – to help hit important MDG15 targets by 2015 – but also to simultaneously incentivise longer-term sustainable maintenance and community use of our WASH investments – crucial for delivering the underlying outcome benefits. In turn the Girls Education Challenge (bit.ly/1PrG8M0) is using a PbR approach to shine a very bright light on the degree to which our work is truly improving girls’ literacy – and generating a wealth of evidence in the process, as it tracks the learning of 70,000 girls and households across 18 countries.

    DFID’s Strategy for PbR (bit.ly/1maTimC) is a learning strategy – it commits us to expand the evidence base on what works best, and build capabilities for doing PbR in the right way. Our PBR Evaluation Framework (bit.ly/1MPLKAk) focuses down on the priority questions we want to answer. Your help in answering these – through your hype-free, careful thinking and analysis – will be very welcome!

    Final point – I agree with Paul Harvey – I really like the idea of using direct beneficiary feedback as a form of independent verification of results, within a PbR framework. Suggestions for candidate projects to test this very welcome!

    1. Thanks Donald, very useful – when you’re ready, wd love to have a post from you on what you’re learning about the do’s and don’ts of PbR, perhaps with a reflection on which of Clist and Dercon’s 12 principles need to be tweaked/improved on

  10. Great insight, giving us a lot to think about. I would be very interested to learn what you think about a performance bonus instead of RbP. E.g. if an NGO achieves a certain milestone, they get an unrestricted bonus of e.g. 30% of the initial budget. Interesting concept, but what would be the consequences? I hope for a post on this soon :)

    1. Strikes me as essentially the same thing Monika – NGOs or governments would get a bonus as unrestricted income for achieving the desired results.

      1. That would indeed be nice! However the one instance I’ve seen up close and personal of PbR for NGOs involved the NGOs losing money out of a budget that has been 100% allocated if they miss their targets. This strikes me as a pretty egregious arrangement that only encourages NGOs to hide their actual costs so as to cope with the likelihood of some losses. (Or low ball their targets to the point of ridiculousness if the donor will let them get away with it.)

  11. Hi Duncan,

    As one of the consortia implementing DFID’s WASH Results Programme, the SWIFT Consortium (http://swiftconsortium.org) is doing its bit to expand the evidence base by sharing our experiences of providing sustainable WASH in fragile contexts through a PbR contract.

    Two of my colleagues have blogged about the programme (here – http://policy-practice.oxfam.org.uk/blog/2015/07/fit-for-purpose-or-fit-for-payment – and here – http://policy-practice.oxfam.org.uk/blog/2016/03/850000-success-stories-for-world-water-day), and last year we produced a learning brief (see http://swiftconsortium.org/resources/) which provided a kind of checklist for NGOs considering implementing a WASH programme on a PbR basis. As we near the end of the first, ‘output’ phase of the programme, we are conducting learning reviews in DRC and Kenya which will look at many of the issues raised here. Watch this space!

  12. Excellent post, Duncan,

    one comment on the graph: it is incomplete. The “plateau of productivity” is no plateau at all, but a slippery downward slope, as the unintended consequences emerge. these may take time to materialize, but they are inevitable, in particular when the model is theory- rather than experience driven.

  13. Thank you for the thoughtful article. To me, PbR is symptomatic of a more fundamental issue, namely development thinking on “how change happens”. If we believe that change happens because “we make it happen”, then PbR makes sense. Big Push thinking (Paul-Rosenstein Rodan, Jeffrey Sachs) builds on the hypothesis that, with enough of the right efforts, change can be managed. Planners and experts can rationalize contextual dynamics, predict what will happen through defined and researched change pathways, pull levers, insert inputs, produce outputs and deliver change to order. It is straightforward to those so initiated, with failure being down to poor plan implementation.

    If on the other hand, we regard change as something that is happening anyway, that we cannot make happen, then we have to engage differently. This is the heart of complexity thinking that was the subject of one of your blogs last week. Here, we say that complex systems are dynamic and unpredictable. By definition therefore, we engage with intention, but not certainty. Here, iterative approaches that learn and navigate uncertainty become important, in the same way that an entrepreneur figures out a changing market place by messing about with products and ideas.

    PbR is built around the certainty of assumption. On short chain relationships (say input to output) it has utility. I order a car. You deliver a car. I pay you. In complex relationships, such assumption becomes invalid. Digging wells does not assure access to drinking water. Building schools does not assure access to education. It therefore is an inappropriate mechanism to invest in change processes in complex systems, which is invariably the characteristic of systems marked by poverty issues.

    Why donors persist in this I think relates to the way they see change happening, changing frameworks and underpinning values. Development frameworks change regularly – technology driven development (Green Revolution); good governance and democratization; rights-based approaches; food security; climate change and environment; global security; all have variously framed international development efforts in the past 40 years. As donors change their focus, new ideologies require new drivers, and new Big Push plans are put in place to drive new change processes. Governments are expected to justify their expenditures to parliament and must offer certainty. This raises the value of accountability to a status higher than that of effectiveness, or even results. It is the promise of certainty that is enough to secure the investment. By the time proof of results are required, the framework has once again changed, obviating the need to demonstrate what actually worked.

    It is time therefore to question not the nature of PbR, but the nature of how we engage and invest in change.

  14. Hi all,

    I agree with much that’s been said so far – the devil really lies in the detail. There are many examples where PbR has failed or even created perverse incentives, but there are equally as many examples where it has been effective, and often a very small difference in design can make the difference.

    What often gets lost in the PbR debate is the success story of performance-based grants from central government to local governments. Many countries across the world have implemented such grants systems, and while there are challenges, it has been a relatively positive experience overall.

    This brings us back to the discussion on incentives: A key reason for the effectiveness of performance-based grants appears to be the creation of a healthy competition between peers, i.e. the fact that local governments often care about their standing vis-à-vis their peers and do not want to rank lower than their neighbours. Second, performance-based grant systems also work well because they are often able to increase downward accountability, through creating structures for citizens to hold their local governments to account for their performance.

    Both of these mechanisms illustrate how PbR can align incentives in a way that isn’t actually about the payment/money, especially considering how small the sums are that are usually transferred via performance-based grant systems. It’s about other incentives too, not just financial ones, and PbR can in some cases play a helpful role in unlocking them.

  15. Call me a cynic but it always struck me as an idea that was all about the spin, with little practical application. Too many practical problems to actually work in the first place. And, above all, a rather patronising approach to development finance..”you do all the things that we would like to see, and at the end we will reward”. it seems some ‘deep’ development thinkers have a lot of time on their hands to pontificate on these things, but ultimately it hasn’t proved very helpful, has it? The next big idea, please come in!

  16. Some good points in both your article and the comments. Something that I think is missing, is the potential for PbR to get us to the ‘money question’ (literally) a lot quicker. If a developing country government with a large fungible budget and a stated interest in achieving a laudable goal which a donor wants to support, then my question is why would they not want PbR??? Too many aid programmes waste money funding stuff the government doesn’t really want to to do. E.g. if the whole education ministry budget is mostly one gigantic patronage scheme for useless teachers then the government will shy away from anything results-based in education, and that response should tell you a lot.

    But here’s the rub: for the most part I don’t think donors want to hear that message! Despite all the evidence to the contrary, they would prefer to carry on believing that they are making a difference and more money will help them make even more difference. Most donor staff (and indeed their political masters & mistresses – as prodded by the likes of Oxfam) are under pressure to disburse more money not less. In this context big changes that lead to them not disbursing are not terribly helpful. So though many donors may be dipping their toes into the PbR waters, I also hear of occasions where they retreat pretty quickly in the face of recipient government opposition.

    So, despite all the problems with PbR, and the fact that it is clearly no silver bullet / panacea, I like the idea that PbR might push a greater proportion of aid funds towards those organisations and governments with a higher interest and greater capacity in actually achieving worthy outcomes with it. And despite the pain that may cause us – including myself and my organisation, of course, in that category of superior performers – I think the rigour of PbR could therefore be worthwhile.

  17. Perhaps one of the most interesting features of the payment by results phenomenon across a number of different sectors (international aid, worklessness, reoffending, substance misuse) is that PbR is the subject of sustained criticism (quite reasonably) and yet has become increasingly common.
    I’m currently involved in a piece of work for the Oak Foundation to develop an interactive tool to help commissioners (and investors as the Social Impact Bond model takes off) and providers to think through whether PbR might be effective for a particular initiative.
    The project involved a literature review which is free to download: http://drugscrime.uk/1nAcXjK
    The tool itself is at a prototype stage and I would very much welcome readers’ feedback: http://payment-by-results.site-previews.net/

  18. Thanks for kicking off this important conversation, Duncan, and for stimulating such great comments.

    I’ve added my two cents here, arguing that taking context and complexity seriously, and making the case for doing development differently, need not imply dismissing Payment by Results, but must involve – as your question suggests – shifting the focus to results that are useful for learning and downward accountability too.


    1. Thanks Alan, I’ve been trying to read your blog, but my computer says no because ‘This site can’t provide a secure connection’ – any idea what the problem is?

Leave a comment

Translate »