Which governments are doing best/worst in the fight against hunger and undernutrition?

July 4, 2014

Links I liked

July 4, 2014

Is ‘thinking and working politically’ compatible with results? Should advocacy ever be done in secret? Big questions at the LSE this week.

July 4, 2014
empty image
empty image

This week I found myself on a fun panel at LSE discussing ‘can politics and evidence work together?’  with Mary Kaldor (LSE), Ros Eyben (IDS) and Steven Rood (The Asiaevidence pyramid Foundation – TAF has a really interesting partnership with LSEto study its use of theories of change). Early last year, I promised to revisit the topic after this blog hosted an epic debate on the politics of evidence between some top DFID people and (in the sceptic corner), Ros Eyben (again) and Chris Roche. Has anything changed since then?

My 15 minutes of fame are summarized in this page of speaking notes: Notes for Thinking and Working Politically LSE panel July 2014

I largely summarized previous pieces on this blog, but added some alliterative categories for some of the obstacles aid agencies face in thinking and working politically, namely:
· The Toolkit Temptation: managing large bureaucracies creates a demand for standardized and simplified procedures (indeed it’s often the first thing hard-pressed staff ask for). Difficult to shift from that to deep engagement with the local context, thinking on your feet and responding to events etc, trying multiple experiments and failing faster etc
· The Planning Preference: Similarly, big aid organizations maintaing internal coherence and external direction by agreeing (at great length) and implementing plans – strategic, operational etc. After years of thrashing out a plan, they are often understandably reluctant to begin all over again because some major event has presented a new window of opportunity
· The Data Delusion: non-scientists in particular are dazzled by numbers, even if they don’t mean much. I still remember my alarm in a meeting of Oxfam’s big cheeses when, after a discussion based on the experience and judgements of senior people with decades working in the development business, someone said ‘right, so much for the opinions – what do the data tell us?’

The other presentations and ensuing discussion (lots of sharp questions, as you’d expect at LSE) drew out some further lessons.

Ros sees actual state of discussion on evidence in DFID and elsewhere as very unreconstructed, with my preferred TWP-compatible approaches still languishing at the bottom of the ‘quality of evidence pyramid’ (see graphic from her slides)

keys and street lightsThis leads to a major ‘drunk under the streetlamp problem’ of going where the data is, rather than looking at what’s important (see cartoon, not that I’m suggesting MEL people are usually drunk)

Fear of failure is deeply rooted at field level – you can forget all that stuff about celebrating/learning from it, at least when there’s a funder in the room.

The real challenges to TWP are often ‘managerial’ – processes like staff rotation, and the kinds of people you employ.

I’ve been mulling over one issue in particular since then: Mary Kaldor was very opposed to any suggestion that TWP might involve some level of acceptance of the ‘dark arts’ – i.e. working in the shadows, persuading people to do things or talk to others in ways they would not want to become public etc. In contrast, Mary argued for transparency and openness throughout, as the only way to ensure accountability.

Which raises some pretty massive dilemmas: TWP and politics in general often moves forward by opposing forces doing deals that would rapidly collapse if exposed to the light too soon – think of just about any peace negotiation in history, or even just the Northern Ireland one. In NGO world, there’s a reason why advocacy is separate from public campaigning, with different staff, language and tactics. One is based on clear, simplified messages, the other on arm twisting and compromise.

But it must be right to be sceptical of advocacy and policy insiders singing the virtues of secrecy (it’s so easy to be coopted – there’s nothing more intoxicating for an advocacy type than to be the only NGO in the room, get the draft document before all your colleagues etc). Fundamentally, secrecy disempowers anyone who is not in the room, and that usually includes the people we are trying to help.

LSE panelI wondered if the ‘policy funnel’ – a way to understand how new policies evolve from outlandish ideas to agreed policy – might need to be adapted. We need maximum openness in the early stages, eg trying to get issues onto the official agenda, but have to accept a degree of non-transparency if you get as far as negotiating deals, policies, or cash. But then whatever is agreed needs to be open to public scrutiny (so the funnel becomes more like an egg timer on its side). Mary was entirely unconvinced.

As to the main question, it didn’t feel like much has changed since last year’s FP2P wonkwar on evidence. Maybe there’s been some progress on learning how to measure what matters, rather than just what’s easy to count, and the randomista craze seems to have passed its peak and hopefully RCTs will soon settle down to become one tool among many, rather than the ‘gold standard’ for everything (remember what Keynes said on the real Gold Standard – a ‘barbarous relic‘).

So over to you – do you see progress on reconciling TWP and evidence? And how do you balance TWP and commitment to openness and transparency? Any good examples?

And here’s the rather nice slide that ended Ros’s presentation (don’t suppose Mary will thank me for the panel pic – click to expand and see why)

evidence emperor's new clothes

9 comments

  1. Hi Duncan,
    Interesting issues but I am slightly vexed by the continued concern over whether ‘more political approaches’ (for lack of a better definition) are compatible with results.

    Surely if the results agenda is about securing results then more political approaches are key to that, so it becomes ABOUT results, as opposed to results presenting an incompatible challenge.

    If the ‘results agenda’ is about demonstrating results to donors then that’s a slightly different matter but if we know that political approaches are probably more likely to deliver results then you can hopefully worry about demonstrating them once you’ve helped reduce poverty and other good things. If we need to measure things better that is a legitimate challenge but do we wait ten years to nail that before we start getting on with it? Perhaps we should be going more iteratively, to use a popular phrase… the Einstein quote ‘Not everything that can be counted counts, and not everything that counts can be counted.’ springs to mind…

    There’s a risk of a technical question about measurement for a handful of donors trumping the need to do more of this – perhaps a better framing would be. ‘If thinking and working politically gets results, why is there not more donor support for it’?

    Jonathan

    1. You’re right of course, Jonathan. The issue is not results, but their measurement. But this poses a genuine quandary – sure, if we spend enough time and money, we can ‘count what counts’, do rigorous qual etc, but there are at least two obstacles to that: firstly mindsets (the rigour discussion), second budgets – anyone measuring results is usually doing so on a shoestring, and has neither time nor money to do lots of innovative measurement, so the institutional pressure pushes you towards what is easily counted (bednets, school books, vaccinations), not subtle issues of empowerment and wellbeing, even when those are worth pursuing.

  2. Hi Duncan and Jonathan,

    A brief answer to your final question Jonathan: I think its because no one (and especially not donors) wants to be seen to actually practice the ‘dark arts’ (a phrase that is already irritating, although does conjure up ninja images in my mind).

    Thinking and working politically apparently means a lot of different things to different people, which was one of my take aways from the conference. But I think for many it seems to mean engaging with ‘political’ actors at a number of levels to get stuff done (nothing new there then).

    The problem is, this can mean engaging with those aren’t the cleanest of characters: say, for example, a chief of police in a conflict-affected country responsible for human right abuses. As we all know, if that becomes public, its Daily Mail bait. Perhaps everyone needs to get better at telling and justifying these relationships. But this isn’t just about the donors or NGOs: those in the countries donors and NGOs work in (such as the chief of police) may not want the relationship public. Building relationships in the messy world of politics just won’t work sometimes if you have to tell everyone you’re doing it.

    That links to the tension with transparency and evidence: can/will donors and NGOs write down these kinds of interactions? Is it even helpful if they do? What do we lose, and what do we gain through writing it all down? I’m still not sure.

    As an aside, it seems to me that what everyone is talking about now is what diplomats (and probably most NGOs and donors) have been doing for years. Maybe if we want lessons on thinking and working politically we just need to analyse the Wikileaks cables!?

    1. On your last point, couldn’t agree more Craig. Once attended a UK Foreign Office internal seminar on its climate change campaign – using ex Greenpeace consultants they had identified 100 key opinion formers in a large developing country, and then used trad diplomacy skills to find out how best to influence them – media coverage, evidence, peer pressure etc. NGO meets MI6. Fine when it’s for a good cause like climate change, but a bit less so on other stuff…….

      1. Yes exactly Duncan. Lets be wary of the darker side of the dark arts – not all of this stuff is a force for ‘good’ – and people shouldn’t pretend that ‘working politically’ means everyone has the same political goals.

  3. Dear Duncan,

    You must have seen this 2007 paper already. Just in case you haven’t, there is a whole section that starts on page 486 that describes how Professor Gary King (a University Professor in the Department of Government at Harvard University and Quantitative methods expert), achieved a “politically acceptable randomization”.

    Deborah

  4. Apologies Duncan. I thought I had pasted a link: http://gking.harvard.edu/files/abs/spd-abs.shtml

    Full Citation:

    King, Gary, Emmanuela Gakidou, Nirmala Ravishankar, Ryan T Moore, Jason Lakin, Manett Vargas, Martha María Téllez-Rojo, Juan Eugenio Hernández Ávila, Mauricio Hernández Ávila, and Héctor Hernández Llamas. 2007. A “Politically Robust” Experimental Design for Public Policy Evaluation, with Application to the Mexican Universal Health Insurance Program. Journal of Policy Analysis and Management 26: 479-506. Copy at http://j.mp/lXnta2

    Best,
    Deborah

  5. More and more, I’m coming to the conclusion that the primary challenge is best analysed using (not so) public choice in donor countries. The current state of things isn’t so bad for the odd cocktail, which is the very ‘realist’ agenda surrounding the corporate leg of aid and the highly idealist agenda of the rights and empowerment leg of aid.
    Sadly, I’m not too positive about the prospects for unsettling it.

Leave a comment