‘Stuff happens’: the risks of a results agenda. Guest post from Rosalind Eyben

March 16, 2011

So where have we got to on Value for Money, Results etc?

March 16, 2011

If not results, then what? The risks of not having a results agenda

March 16, 2011
empty image
empty image

The ODI’s Claire Melamed replies to yesterday’s guest post from Ros Eyben: 673-claire-melamed

“Ros Eyben suggests that instead of a results agenda, we should rely on good relationships to deliver good aid.  And indeed, if all relationships were good, and all the people involved in making decisions about aid were thoroughly well-informed, open to new ideas, flexible in their approach, lacking in ego, adept at dealing with cultural and religious differences and aligned with the needs and priorities of poor people, that might just work. 

But just supposing, for a moment, that aid bureaucracies aren’t all like that, let’s think about the risks of not having a results agenda.

If you don’t define in advance what the objectives of an aid programme are, you leave it up to the managers who make the decisions and the politicians who guide them to impose their own values and prejudices onto the aid programme.  Of course if they could all be trusted to make the right decision, there’s no problem.  But evidence suggests that might be over-optimistic.  Exhibit A: attempts to fund the building of a dam in Pergau that had nothing to do with poverty and everything to do with arms sales.  Exhibit B: the ideological pursuit of structural adjustment programmes in the face of substantial evidence of the harm they were causing.

A focus on results can help to rebalance inequalities of power. When the Labour Government created the National Institute for Health and Clinical Excellence (NICE) in the UK in 1999, to ensure that evidence about value for money and effectiveness was used in deciding what drugs to prescribe in the National Health Service, pharmaceutical companies were among the most hostile to the idea. Naturally, from their point of view, they preferred their own marketing ‘evidence’ to help doctors make prescribing decisions. 

Actually, of course, leaving all decisions about prescribing up to doctors – informed by partial evidence – led to inequalities (the dreaded ‘postcode lottery’), and to millions of pounds wasted on ineffective treatments. NICE’s role in bringing together evidence from clinical trials (which included patients’ own assessments and valuations of changes in their health) with the costs of treatment, has started to improve value for money in the NHS and also to take more account of health benefits (or lack of them) from patients’ own point of view. 

A results agenda, as long as the right results are being pursued, can help to rebalance inequalities of power and make the actions and decisions of the powerful more transparent. It helps people to know what the objectives of decision makers are – and so to argue that they should be different, if that’s the case; and also to hold people to account for their success or failure to meet those objectives. Without measurement, there can be no accountability. 

The real question is what results we are looking for, and how to measure them. Of course if donors want to do the wrong things, and measure the wrong things, they won’t get good results. But pointing to examples of the wrong way of using results and saying, ‘so let’s not measure results’, seems to me as big a folly as the, sadly all too popular, pastime of pointing to the latest example of unsuccessful aid and saying ‘so let’s give up on aid altogether’. 

So if the numbers of polio vaccines isn’t the right result to ask for, then let’s look for something that is a better measure of the strength of health systems. And instead of counting the length of roads, let’s measure the strength of solidarity in communities – that’s doable. 

The results agenda is actually a huge opportunity for people who care about relationships, trust, empowerment, rights and complexity to find ways of getting these things firmly integrated into how we measure development.  Then they’d be part of the mainstream.

These things can be counted. There are approaches developed in the UK’s National Health Service, for example, which allow patients to say how much they value different health outcomes, like the absence of pain or the ability to move about normally. Research shows that the values that ordinary people attach to different outcomes are different to those of even the most well-meaning professionals – which should be a warning to us all not to make assumptions about what people want. This information is turned into numbers and used to allocate funding and to measure results.  Imagine if we actually knew what poor people wanted and if they were getting it?  Everyone who works in development should surely admit that we don’t know as much as we should about if we are actually delivering ‘value’ as the recipients of our efforts would define it. 

We should be welcoming the focus on the results, because a world where we don’t know the results of our actions is not one that any of us would want to live in. This agenda should be used too, to  encourage a focus on what results poor people themselves (or, more likely, poor women, poor men, poor people in cities, in rural areas and so on, who would all have different priorities) most want to see, and how they’d define ‘value’ or ‘effectiveness’. 

Information is power. I say, don’t fear it. Use it. ”

Claire Melamed is the Head of the Growth and Equity Programme at the Overseas Development Institute

Update from Duncan: In a desperate attempt to stem the tide of consensus and mutual respect sweeping over the comments, I’ve put up a poll to the right of this post that allows only a yes/no answer to the question of whether the current focus on Value for Money is a Good Thing. ‘Sometimes’ ‘Maybe’ ‘It depends’ type answers all forbidden!


  1. I totally agree with this as well!! Am I just fickle and indecisive, or is there perhaps a middle ground? Or more likely, is development just incredibly complex. I can imagine situations where a narrow focus in results is going to undermine exactly the kind of trust relationships that Ros Eyben emphasised and that are so crucial to building a movement for sustained change. But I can also imagine plenty of occasions where such a focus will brush away some cobwebs and force development practitioners to really up their game.

    Perhaps it just comes down to committed and intelligent professionals making the right decisions.

    Duncan: Thanks Jonathan, clearly if we’re not confused, we’re not thinking hard enough……..

  2. Really great debate. Thanks for both of the very helpful and well written pieces. Is it not possible to enable a results and relation approach or does it have to be either or? Effective work and progress always has a ‘result’ and needs measurement but its often rooted in and successful because of a realtionship and a good relational agenda.

  3. Good debate indeed. But, yes it must be a both and approach. Because while it would be easy to measure quality of health services (NHS), it would not be easy measuring changes in power relations. Yet it is not impossible. Perhaps the question that also needs to be asked is who is setting the measurement standards and indicators?

  4. The problem as i pointed out yesterday in my comment is that this spoilts the idea of what development really is. I’m not saying we should not look to the future to see millions cured and lifted out of poverty but that is saying that development is about numbers/figures. Results for development should be about transformation as well.

  5. Really interesting debate. Thanks very much for getting it going.

    I completely agree about the risks that come from imposing inappropriate targets (e.g. numbers of nurses trained) and inappropriate ways of managing (e.g. logframes). The evidence is compelling – programmes need to be responsive to ‘stuff happening’. And I completely agree about the importance of relationships in tackling politics at every level.

    But I’m with Claire on not throwing the baby out with the bath water. Senior decision makers need a sensible & responsible way of allocating resources. They also need to review and understand performance. The pressure to measure results is growing and isn’t going away.

    So surely our challenge is to work out approaches to measurement that create incentives for responsive and high quality aid? For me, this is one of the most important challenges in the sector.

    I’ve called the most encouraging approaches ‘the demand-side revolution': approaches to measurement that (a) focus on the outputs & outcomes that are important to local people and partners, (b) are primarily informed by how well local people & partners feel they have been met, and (c) generate quantitative performance data.

    See http://ngoperformance.org/2011/03/02/measuring-results-the-demand-side-revolution/ for more, including examples.

  6. Great debate and I agree with most of both too Duncan! Its fair enough that the public are challenging ‘why increase aid budgets when my local services are being cut?’ We do have to show what they are getting for their money, but we need to do so in ways that makes us MORE accountable to poor people, not less….so I am more in the ‘How can we use the push for results to support our learning on what we know is critical to good quality development….like relationships? I would add to your comment…..and if we value relationships – why not have a go at using measurement to get some feedback via email/text message to check on the quality of the relationships we hold so dear? perhaps ‘x% of our clients/partners felt better off/worse off/no difference had been made as a result of our intervention’ or ‘x% of our partners feel they have an open trusting relationship with us’ …and heres some qualitative examples given from the ‘open comments section’…
    In terms of what results we measure this is also an opportunity to ensure poor people themselves have a say in what results they would hope for and would value at the outset of aprogramme…… an opportunity to bolster the need for honest open conversations that would in turn reinforce the importance of relationships?

    Duncan: Oh well, consensus and an outbreak of classic NGO ‘both-and’ thinking. And there I was hoping for some serious hostilities to boost the traffic…….

  7. This is little to disagree with in either piece, really. Where one upholds the importance of trust in building relationships and engaging with systems, the other calls for a focus on identifying that right results and the right measures. Claire’s response is built on the assumption that pursuing the results agenda will yield a better understanding of what the poor actually want and their feedback to projects that attempt to improve their quality of life.

    From what I understand, Ros is not dismissing the importance of results either. She is opposed to the manifestation of the so-called results agenda. The manifestation, according to her is that complex problems risk being ignored, donors/implementers will tend to focus on projects, rather than systems and a general break-down of trust between the different actors. The response to this cannot be an argument highlighting the importance of results. It should be by demonstrating ways in which the results agenda can positively influence development work and focusing on the mechanisms by which the multiple actors in development can come together in making this happen. Importantly, this would have to start with better education of donors and other policymakers in development on how to handle the results agenda, as Claire seems to suggest in her piece as well. Is that possible? I don’t know!

    Such debates mirror what development itself is usually about. We can’t always win by proving someone right and the other wrong. Confrontation is easy, cooperation is not. The answer definitely lies somewhere in between and demands that the two sides work together to figure out the way ahead. Just as we sometimes build social capital in communities by accommodating opposing view-points and evolving a consensus, so too in this world of opinions.

  8. Yes, Claire, from years of experience, I do believe that it is the quality of relationships in the aid system that is the biggest influence on our collective capacity for aid to make a positive difference in the lives of people in poverty.

    You quite rightly point out that bureaucracies are not that good at managing relationships and that a results framework keeps them on the right road. I agree that it is fundamentally important for any aid agency to have a sharp vision of what it is trying to achieve, a vision that will shape its relationships and expenditure decisions. For example for every project concept note, we could ask: How will our proposed support contribute to changing power relations in favour of poor people?

    You could frame this question as a ‘result’ i.e. changed power relations. Funny thing is that I don’t see this kind of result being suggested very often. And this is the crux of my argument. Of course aid agencies need to define the change that they wish to see and identify how they can contribute to that change. But that is not happening. Instead, increasingly they want to identify outcomes that can be both easily counted and specifically attributed to their intervention. This means that aid activities in relationship with many others in supporting systemic change processes are being sidelined – or dressed up as something else to secure financing. Yet it this kind of support that offers the greater potential for helping make a difference.

    Aid agencies cannot deliver sustainable development outcomes. They can only contribute to these through politically sensitive and contextually specific relationship building with organisations – government and non-government – who are trying to make a positive difference to the wellbeing of their citizens. And, yes there is increasing knowledge about methods for assessing such contributions. But at the end of the day it boils down to how we understand change happens. Do we agree about that? I am not sure.

  9. I think we’re talking about two different things here. There’s a theoretical discussion about how measurement might be used to improve aid – what one might need to measure, and how, and then how to use that information. And then there’s a more immediate discussion about the current ‘results agenda’ in DFID and elsewhere. I agree that what is currently measured, and the indicators set to evaluate aid, are a long way from perfect.

    So for me, it boils down to what you do faced with that mismatch between what the results agenda is and what it could be. Here my sense is that your view, Ros, is that we weren’t doing so badly before the ‘results agenda’ came along, so we should go back to where we were and make incremental changes to practice if needed (apologies if I am putting words into your mouth – that’s what I’ve taken from your blog and comment).

    If that is a correct summary of your view, then that’s maybe where we disagree. I’m far from an aid sceptic, but I do think there are things we have systematically ignored, and finding out what poor people value is the biggest gap of all.

    Yes we must build relationships with organisations as a strategy of change, I think we agree about that. But for me it’s not those organisations in the end that are the point – it’s the people they work with and support. And the more information we have about them and what they value, the more we’ll be in a position to make decisions about which organisations might be driving change in ways that people themselves would want.

    Think about the amazement of many donors at the news that micro-credit might be used as much for consumption as for investment. If they had spent more time talking to people who take out loans and less to micro-credit organisations whose funding depended on defending the investment story, this might have come as less of a surprise, and micro-credit might not be facing the backlash that it is today.

    All decisions that donors make involve choices about what to do with their resources. There will almost always be more credible organisations in a country that deserve funding that the resources available to any one donor. So information is needed to help them make the right decision.

    So I’m not sure the argument here is about how change happens, for me it’s about the role that information can play in helping donors to participate in that process in a helpful way. In the end, I’m an empirical optimist – I think that more data has got to be better than less, and that data about the people that are in the end the whole point of the exercise has go to be the best of all.

  10. I could not agree more. We are spending money of the public on what? We claim to do good, and believe the ” normal” dynamics happening when money changes hands don’t apply. If you organise a tender procedure only on quality, you will pay way too much. If you never check on what was actually delivered, in the end you end up not getting the goods any more, and still paying.

    It is amazing this discussion can exist even.

    Doing good without any results accountability is just too naive for this world. Something you can accept to exist when an eccentric spends his own money, but not when spending from the public.

  11. It would seem beneficial to me, if we did ourselves the favour of thinking hard about the distinction between accountability and countability. It shouldn’t be so hard but quite clearly is, which I’ll get back to.

    I believe there is all reason to pursue a focused, accountable, results-based agenda for all the justifications in Claire Melamed’s post.

    But for all the reasons laid out in Rosalyn Eyben’s, if that equates to nothing more than that which can be counted in binary numbers, little will those results accomplish.

    My own perception of why it has proven so difficult to combine the two assertions -and in practice separate accountability from countability- is that our models of change are flawed. As long as we remain stranded on shoulders of social sciences that brought us far but never beyond linear cause-effect thinking, I don’t see us really getting our heads around it.

    More research is advised :)


  12. Thanks to Ros, Claire and Duncan.

    I’m with Claire on this one.

    The average size of Oxfam’s country programmes is about a million pounds a year or so (I may be out of touch now, with the consolidation across the different Oxfams that is currently taking place, but indulge me). These are then divided among a number of programmes and within them, projects. Small amounts of money in the grand scheme of things.

    Working on evaluation for Oxfam, the biggest challenge was figuring out exactly what was being done in these projects, as they were all described in terms of the impact they were going to have (e.g. “Women’s empowerment”; “Improving the resilience of livelihoods”) rather than the necessarily much more modest means they had at their disposal to actually achieve these (e.g giving money and training to three local NGOs to represent themselves better in national policy circles).

    I think that by focusing on results we helped to bring this aspiration gap to light. I got good feedback from staff for the training on logic models that we organised, where everyone was encouraged to draw diagrams describing the change they wished to see, and to think about what is necessary and sufficient to achieve it. This was based on expected cause-effect relationships. IF we do this AND the government does this THEN we think this OR this will happen. Debates about killer assumptions ensued. It was, whisper it, fun.

    The things that were then actually measured (as determined by the staff) were just the points of information that gave an insight into whether our diagrammatic story about the future was playing out in reality. The main benefit of the whole results push was this debate about strategy and the need for realism and clarity. Utopianism is for the campaigns teams – achieving development is difficult but worth the effort. Any sensible donor will recognise this.

  13. Thanks to all for this very timely debate. For my money, I do think that it is in part about how we think change happens and the role that we think donors play in bringing about change.

    I say this carefully, as I have not quite taken off my DFID hat!Donors pretend to have more control than they do, because that is what they think their political/taxpayer masters want to hear. Maybe it is or maybe we should give taxpayers more credit for their ability to acknowledge that the world is messy?

    The reality of course is that in many spheres of development, donors have very little control over how change happens. The challenge for DFID and others is to navigate this tension.

    For what it’s worth, I wobbled off the fence to vote yes in the poll 😉

  14. I’m not sure that the example given is meaningful — corruption is corruption, and neither approach guarantees that the moneys won’t be simply stolen.

  15. very interesting back and forth, but I agree with Claire that we sometimes debate at cross purposes. Many have tended to equate results with numbers/measurement. The idea behind RBM in the first place was actually something more systemic: to get aid agencies to think strategically about the results they want to see (demand-driven), rather than what they can do with the money they have (supply side). Instead of focusing on the delivery of scattered training activities and goods and services which may or may not be needed or have any useful impact (a huge problem previously in large bureaucracies – one that has far from disappeared but has at least got better, I would argue, in the UN system), the plan was to prioritse and get people thinking about the real-world development changes they want to see (whether this is do with rights, power, relationships, capacity etc.) The challenge was to define these results and find out the best way of getting there, bearing in mind, as Rosalind has argued, that no one agency has the power to enact sustainable change alone.

    I worry that if we reduce the debate on results to just value-for-money and counting, and thus move aid away from results, we’ll lose this key focus. In my mind, the debate should be less about whether we have a results-based agenda (I’m not quite sure how we can justify going back to the old input-based-planning model) and instead be more about how these results can be defined, how they can be measured and what models we use. Rigid logframes and quantitative indicators clearly have huge and acknowledged limitations, but the fact that they are widely used to this day speaks volumes about the lack of an alternative.

    As Serverino and Ray point out in the excellent ‘End of ODA’, “in the absence of credible indicators of aid impact, development agencies are assessed according to an extremely weak proxy: their capacity to generate visible projects and disburse high volumes of funding quickly”.

    So results are now seen as something akin to number of children in school and relative cost to the taxpayer in donor country X. But this wasn’t what development results were meant to look like, and it seems the whole idea of results has been hijacked, simplified and re-packaged in the guise of value-for-money.

    To me, the debate should be not whether results but whither results.
    Can outcome mapping be useful? What do the results we want to contribute towards look like? Can we find a way of setting out a path towards these results (and agreeing how to measure them) which takes into account the in-country reality? How does this take into account broader structural considerations and the interplay of aid, trade, taxation, finance, industry? What role can impact assessments play?

  16. Just to say that for some of us the challenge is not particularly limited to “counting”, it is also about what has termed as the “change we want to see”. A lot of times the change we want to see is probably not the change poor/vulnerable communities “want to see”.

    Yes, it is absolutely necessary to show results (although I feel more comfortable with terms such as outcome and impact), otherwise we are at risk of being accused of what we accuse governments of on a daily basis. But my fear is that the pattern the current VFM is taking is to have results/change determined through the lenses of donors. The recent review BAR and MAR processes and outcomes appear to support that

  17. To an extent, I think Claire and Ros go different routes to the same thing – both accept that the bog standard results agenda is rubbish – Ros’s solution is to support domestic change processes, Claire’s solution is to have as the key result ‘delivering what poor people really want’ so to my mind, if we can accept that domestic change processes are in the direction of ‘what people really want’ then these two results might not be that different.

  18. great point, Abi, I think this is the crux of the debate. Who defines the “change we want to see” (organisational priorities)? Does it flow from the needs of poor/vulnerable communities? Or is it being vaguely aligned to some development framework over which beneficiaries have had little voice anyway. As Matt said, perhaps the destination is the same in both arguments, but we just have to find middle ground on the path there. Either way, this debate is wholly useful in making sure that the focus on value for money (as currently defined) doesn’t serve to alienate the whole idea of development in the first place and replace it with a short-termism designed solely to satisfy the visibility and accountability demands of rich nations.

  19. I know and agree that good relationship is important but at the same time we also need the results for accountability purposes and to substantiate what was achieved due tom the investment of the amount. Relationship and working towards results should go hand in hand so that you are achieving the twin objectives simultaneously. One of the objective of developing and focusing on result statements and depending on performance evaluation is to build good relationship as well. So both would go hand in hand as the target for both is to deliver good programs for the communities and the poor people in development.

  20. What other agenda can we have if we don’t have a results agenda? What is a results agenda?I think we need to appreciate that the term results is a packed term, referring to three levels of data – outputs, outcome and impacts.

    So i think what we need to be discussing is how we balance our results agenda and move away from focusing on one set of results (outputs- the numbers game) at the expense of the others (outcome and impacts – hard to measure ^_^).

    I think we have always had a “results” agenda in development and all three levels have their role – for politicians, numbers may be all they want to work with but that does not mean that development managers then neglect the other results (outcomes and impacts) that make our work enjoyable… I always say “outputs just measure how busy one is”, where as “outcome and impacts demonstrate how effective in delivering and sustaining change one is”.

    Lets not only welcome the focus on results, lets support a more balanced approach to results with the knowledge that different levels of results will be useful in different ways to different people. As development managers we need results at all three levels and our M&E systems should allow us to have produce and communicate at these levels so as to meet the demands of our diverse stakeholders.

Leave a comment

Translate »