More: http://ngoperformance.org/2013/04/23/wonks-make-peace-not-war/

One point of concern, though: When Roche and Eyben argue that permissible evidence should include ‘other ways of doing and knowing’ and ‘multiple perspectives and understandings of what’s at stake’ it sounds a lot like they are saying that anything goes or at least that it should. Of course cultural and political sensitivity is good, but a suggestion that everyone has their own culturally relative set of definitions and standards is not helpful.

In the experience of the International Budget Partnership, southern civil society has a deep and nuanced understanding of their own impact.We should find ways to bring that understanding into the evidence debate, not yield to the temptation of scientific relativism that will struggle to get beyond a discussion of ideology and power analysis.

Very much agree with Marcus Jenal’s summary – I think the set up as a ‘wonk-war’ may have contributed to both sides talking past each other. In fact, the crux of the two arguments were less contradictory, and more coherent, than the authors suggest. I just wanted to add two points to the debate, drawing on previous writings:

1. In one previous post (http://aidontheedge.info/2011/06/30/results-2-0-towards-a-portfolio-based-approach/) I suggest that there is more that unites the different philosophies around results than divides them. Specifically:

– Data availability, coverage and quality are perennial problems
– Participation and ownership; as Robert Chambers might ask: ’whose results count?’
– Incentives and disincentives to use information and results, especially when they run counter to individual and institutional interests
– Bureaucratic inertia: all too often results-related work is placed on top of and increases the already considerable bureaucratic and administrative burden on aid agencies, rather than simplifying and reducing it
– Risks and fear of failure: How can we manage and be transparent about the different kinds of risk failures inherent to development projects & programmes?
– Many conflicting imperatives: learning vs accountability, policy vs operations, domestic vs international

I think it is clear that these issues apply equally to both sides of the results ‘tug of war’. By bridging the divide, we can deal with these collectively rather than in entrenched intellectual silos.

2. In a follow-up post (http://blogs.cgdev.org/globaldevelopment/2012/09/complexity-and-results.php) Owen Barder and I argued for what we call results 2.0, which explores what an complexity-informed approach might look like. I don’t know how to post visuals into blog comments, but this is the framework we suggested could be utilised: http://blogs.cgdev.org/globaldevelopment/files/2012/09/resultscube.png

We suggest that the key starting point for any discussion of results approaches is to think through the problem, the context and the intervention at hand. We conclude that “there is no contradiction between an iterative, experimental approach and a central place for results in decision-making: on the contrary, a rigorous and energetic focus on results is at the heart of effective adaptation.”

I’d suggest that a good follow up to this series of exchanges would be to identify the things that people actually agree on, and to establish a shared agenda for moving this area of work forward? A kind of wonk’s reconciliation commission, perhaps?

This bit made me think: “Chris Whitty and Stefan Dercon should give some examples of gold standard anthropological or sociological evidence to allay the doubts over their true commitment to multi-disciplinarity, and take the complex systems question more seriously.”

Perhaps they should indeed give some examples but I think the purpose of this type of research is fundamentally different from the purpose of an RCT. One of the attractions of RCTs is that by randomising they can deal with the problem of bias that cannot otherwise be measured/adjusted for; and that therefore the conclusions are fairly generalisable. In my view, qualitative and sociological evidence is much more focused on identifying differences and biases and subjectivities within contexts. The methods might be applicable, but while the findings of this sort of research might tell us what we should be watching out for when implementing a given intervention, I don’t think they can ever be generalised. So while an RCT can tell us what effect, all else being equal, we can expect an intervention to have, qual research is essential to figuring out how the intervention may play out in a specific situation.

On who should be making choices:

I should clarify that my intention was to suggest that while in developed countries decisions are public and political (or at least we demand them to be) in developing countries where donors still have a say in how resources are allocated decisions are often made privately and presented as ‘technocratic’. At least this is what the industry tends to say when confronted with claims of ideology: ‘no, no,’ the seem to say: ‘we are doing this because its evidence based (or informed)’.

e.g. Cash transfers are a new-classical (don’t give services, instead, give money and let individuals maximise their utility by buying the services they want) solution that came out of Latin America at a time of economic liberalisation. In between decades of left wing socialist idealism, governments, and even dictatorships. They served very Latin American purposes: addressed the social pressures associated with budget cuts, were controlling (had conditions) but not too much (market freedom was at the centre of politics at the time), and fitted nicely into a very Latin American political practice: populism.

When donors and NGOs push them left right and centre as ‘evidence based’ solutions they forget their rich (and interesting and important) political origins and history.

So whether the idea emerged from research or experience or ideology is, I think, not the most important thing. What is important is: WHO makes the choice because this can define whether it incorporates politics and power or not.

There is an excellent speech from Woodrow Wilson in which he warns of the threat of technocrats and experts to democracy. If all policy discussions are controlled by experts the public will soon be crowded out from all spaces where decisions about them are being made.

There is a great study about Argentinean economic policy that shows this sudden change from ‘public experts’ (in political parties, NGOs, universities) or ‘private experts’ (from consultancies, foreign banks, int dev organisations) when it came to advising economic policy in the decade before the last crash there.

Now, what appears to be taking place is that it is not enough to be an expert on a subject or country or culture, etc. but its more important to be an expert on a particular type of research method.

In the UK the Institute of Ideas organises a fantastic festival of debates in which the panel is not the expert and has no more time to voice its views than the audiences. Everyone is allowed an opinion.

This inclusiveness happens because the debates ask important and fundamental questions: what is the purpose of education? and not problem solving ones: how to get more young people from poor backgrounds into university? The former lets everyone in (from all backgrounds and level of specific knowledge and expertise) while the latter makes it easier for the ‘experts’ to control the conversation.

In Aid (and RCT’s in a way need this) we seem to have answered the fundamental questions (even if we are still asking them in our own ‘donor’ countries) and appear to focus on the technical ones: how to.

But RCT (if and when carried out properly) should and can also be employed to address these fundamental concerns.

….

On studying politics and power:

I particularly liked the characterisation of this often repeated statement:

‘Understanding power and politics and how to assist in social change also require careful and rigorous evidence.’

What many researchers and decision makers in donor or intervening countries do not seem to get is that the people that live in these countries about which more careful and rigours research is needed do not need more careful and rigorous research.

The idea that they are able to survive without understanding power and politics and how to navigate the complex social, economic, political and intellectual spaces that they live and work in is not just patronising but also ridiculous.

The problem is not that ‘they’ (in Malawi, Indonesia, Peru) do not know; the problem is that ‘we’ (in London, New York, Canberra) do not know. And this is only a problem because we want to intervene and make choices for them.

But if we cannot stop interfering in other people’s business we should at least acknowledge that to be better informed we do not need to reinvent the wheel. We are lucky to have the tools to understand (or get as close to understanding) other cultures, societies and political systems: how about some good old fashion anthropology, sociology, demographics, economic and political history, language studies, etc?

Why not? Because it takes too long and it is hard work. And because, I think, few Aid workers today have the patience nor the interest they claim to have on the countries, peoples and societies that they work with.

They are not interested to know them; just to change them.

……

On the debate:

It would be unfair to dismiss the value of RCTs. I am not one to say that they are NEVER useful. They can be and are quite a fantastically powerful source of argumentative power; as well as fun to design and implement.

But we must not give them more power than they deserve -certainly not at the expense of methods that, as Ed Carr says, help to explain to explain the findings of RCTs. After all, it would be impossible to design an RCT intervention without anthropology, sociology, history, politics, etc.

And since they are powerful tools we should not forget they are also political ones. Conducting an RCT on programme X is a political statement that programme X is liked, preferred to others that did not even get an RCT, or supported.

These are not cheap interventions and since they have been designed on theories and theories are inherently ideological (they are after all based on assumptions what how they world is and is not and this is based on our assumptions of human nature and our understandings of fundamental concepts such as justice and fairness) sanctioning one or the other can be seen as a proxy to sanctioning the ideological assumptions that underpin them.

…..

And here lies the beauty of the debate: by choosing a particular method (and expertise) as the most desired method to inform the manner in which we go about solving society’s problems, its advocates are being just as ideological as those who claim that markets or the State should solve social problems.

‘Have enjoyed the posts on results in the last week or so. Just wanted to suggest that one way around the impasse / wonk war is to try and apply a horses-for-courses / portfolio approach.

See my June 2011 post avoiding civil war in results here: http://aidontheedge.info/2011/06/30/results-2-0-towards-a-portfolio-based-approach/ and the Sept 2012 follow-up post, co-authored with Owen, here: http://blogs.cgdev.org/globaldevelopment/2012/09/complexity-and-results.php‘

I think Enrique Mendizabal’s point that you flag up here is central.

We absolutely need evidence. This can be RCT, survey data, process tracing, focus groups, ethnographies, and so forth. Agreed.

But, what is done with this evidence base matters. It is rarely clear cut and there are always, always trade offs between goals and resources. The decision to fund or champion one policy initiative always comes at the expense of another. And the benefits of the two are not always comparable – is reducing poverty, inequality, violence, or environmental damage better?

If what we’re really interested in is success (which I think is the case) we need to talk about what success is (not just evidence) and how this is achieved. For my money – and I know this is close to your heart too, Duncan – it’s about political change. Political change that is locally-owned, locally-appropriate, legitimate and sustainable. See the Developmental Leadership Program for more along these lines (lots of case studies) http://bit.ly/XYMRm1

Like in the UK, other donor countries, and (say it quietly) in DFID and the World Bank even, we need to accept that evidence is part of (and a force for good in!) a complicated process of consensus building, compromise, political and personal competition. To assume differently is Utopian and naive.

Three cheers for rigorous evidence, but let’s get real about the politics too.

I have spent 25 years working for various INGO’s and have jumped through many M&E hoops. My sense has been that many staff sadly do see M&E as something they do for others and as a manager you are often looking for PR rather than real learning. We are not really into celebrating learning from multi million pound failures.

Furthermore it is often a struggle getting budgets for M&E teams and there is often a feeling in many INGO’s that staff in M&E and Policy teams are on another planet. So M&E can often become a contractual obligation that is hastily delivered to keep funders happy.

The answer? perhaps we need to look outside the world of INGO’s and ask why Govt’s at a time when so much modern ideology is unravelling seem intent on defining development in stark number crunching terms. It may sound a bit soft and liberal but surely development agencies, donors and Govt’s need to acknowledge that often we do not have the “big” answer but we can try and learn by using a range of methodologies that must include listening to the people we are seeking to benefit. It may not be so “sexy” to sell a learning approach as against guaranteed “magic bullets” but in the end I would argue that really is the best we can do.