Why building ‘resilience’ matters, and needs to confront injustice and inequality

May 22, 2013

‘Squeezed': how are poor people adjusting to life in a time of food price volatility?

May 22, 2013

Aid and complex systems cont’d: timelines, incubation periods and results

May 22, 2013
empty image
empty image

I’m at one of those moments where all conversations seem to link to each other, I see complex systems everywhere, and I’m wondering whether I’mtyranny is the absence of complexitystarting to lose my marbles. Happily, lots of other people seem to be suffering from the same condition, and a bunch of us met up earlier this week with Matt Andrews, who was in the UK to promote his fab new book Limits to Institutional Reform in Development (I  rave reviewed it here). The conversation was held under Chatham House rules, so no names, no institutions etc.

Whether you work on complex systems or governance reform or fragile states, the emerging common ground seems to be around what not to do and to a lesser extent, the ‘so whats’. What can outsiders do to contribute to change in complex, unpredictable situations where, whether due to domestic opposition or sheer irrelevance to actual context, imported blueprints and ‘best practice guidelines’ are unlikely to get anywhere?

In his book Matt boils down his considerable experience at the World Bank and Harvard into a proposal for ‘PDIA’ – Problem Driven iterative adaptation, which I described pretty fully in my review. The conversation this week fleshed out that approach and added some interesting new angles.

PDIA needs funding, but not big million dollar cheques that come with all the paraphernalia of targets, milestones, logframes etc that are more likely to kill thought than promote experimentation and learning. Instead, it needs a trust fund approach – lots of small grants that allow incubation of local solutions to a given problem while ‘avoiding a premature results agenda’.

But does that mean that institutional reform should avoid the big aid dollars altogether? Matt thought not – he portrayed PDIA as a new and extended incubation phase, which can then take the homegrown solutions that emerge and move into the more traditional aid world of large scale, large budget programming. So the challenge for aid agencies is how to create, fund and protect a space within their institutions for small budget experimentation and incubation, sitting in parallel with the big stuff.

timelineTimelines emerged as a useful, but undervalued tool. But these are timelines of what has actually happened in the past, not the imaginary future timelines of funding applications. Matt reckons any project seeking funding should start by building a 20 year timeline of what has happened on that issue/in that locality. If done properly, the exercise of reconstructing the timeline using documents and interviews will reveal overlapping interpretations of what actually happened and recover the kinds of knowledge and experiences that all too often go missing in Aid World as staff leave and projects are wound up. We need a decent timeline methodology – Matt uses the work of Peter Hall at Harvard but it also sounds a lot like process tracing, something our MEL team uses.

The issue of narratives is central – it lies at the heart of the response to a reductionist results agenda that privileges pseudo medical trial data over real experience. Claire Melamed likes to say ‘the plural of anecdote is not data’. True, but I think that a well researched anecdote rapidly becomes a ‘narrative’, and the plural of narrative can definitely be evidence, if not data. Matt, ODI and Oxfam are all separately thinking about the need to build a collection of rigorous, nuanced narratives on stories of power and change – we’ll be swapping notes and hopefully coming up with some ideas for working together on this. What would people recommend in terms of references on rigorous narrative methodologies?

There was a good discussion on what constitutes ‘results’. Good PDIA-type work in developing countries requires a rapid feedback loop of results, but of a different kind to those typically demanded by the aid business. Developing country politicians want to know what’s happening with their money, what has been learned, what has worked and what hasn’t, and how the project has responded. They don’t need the (often bogus) certainty and data demanded by aid planners.

I do find this all slightly baffling – politicians intrinsically know how to navigate in complex environments, respond to shocks and opportunities, using trial and error, instinct and rules of thumb. They make decision on partial information and change direction if things don’t work. That’s what politics is about. But then they become aid ministers in donor countries, and suddenly buy into a paraphernalia of logframes and a particular understanding ofcomplexity signresults that in some other part of their brains they must know has huge limitations in the real world. How to get ministers to think more like pols and less like aid bureaucrats?

All fascinating and thanks to Matt for kicking off and CGD Europe for organizing the discussion (am I allowed to say that under Chatham House rules? If not, please ignore). I’m thinking of writing a paper on the ‘so whats’ of complex systems, but will first wade through the draft of Ben Ramalingam’s forthcoming book before deciding whether it’s necessary.

Update: more thoughts from Matt Andrews on his blog


  1. Interesting read, though it adds to the challenge for CSOs of what to take up as a valid accountability challenge from the results agenda and what to push back on as excessive or unhelpful. As for a steer on rigorous narrative methodologies, the Learning History methodology is one you could consider – although I imagine its process for building a common and nuanced narrative would be incredibly challenging if you took a 20-year timeframe.

  2. Lee wrote an article in MISQ, “Generalising generalisability” which showed how to use qualitative research in a field dominated by quantitative techiques. Quantitative measures (be it from experiment or epidemiology) generalises to populations. Qualitative results generalise to theory.

    So you can create a possible theory explaining lots of observations of experiences. Then, when you know the right questions to ask, and the contexts in which they apply, you can design quantitative techniques to find answers to those questions and validate your theory.

  3. Timeline past 20 years about the topic/the locality… Yes, but it should mean what happened about and also around the topic in the locality. To understand change in one “area” (or sector) you need to look at past events in relation to this area (sector) but also more generally, what may have contributed (external factors, factors from other sectors/areas, internal dynamics etc.) to change things in that area… I.e. and to respond to Eamonn, yes, building a robust narrative is challenging and I don’t know whether the aim should be a common narrative…

  4. For a good example of useful historically based narrative(s) upon which to build current approaches you may want to look at Alison Mathie & Gordon Cunningham, eds,From Clients to Citizens, Communities Changing the course of their own Development (Intermediate Technology Publications, 2008).

  5. One of the previous comments reminded me that this reasoning of qualitative results generalise to theory also is developed in “Case Study Research” by Robert K. Yin. About narratives, a good book for social sciences is “Narrative Methods for the Human Sciences” by Catherine Riessman. And you may also know the work around stories that GlobalGiving has been doing: http://www.globalgiving.org/stories/

  6. Re Politicians not acting as “normal” when it comes to Aid wonder if this is because (especially) in times of austerity ministers want assurances that monies will be well spent and improve lives? Listen to any UK radio phone in about cuts and Aid and you can see that Politicians do have a big job on their hands persuading public of the “worth” of AID at time of cuts.

    From UK perspective New Labour did bring a lot of Logframe, Key indicator type thinking into most areas of public services. So many public sector managers feel overwhelmed by performance targets and having to constantly prove success.So not just an aid issue but more a culture of “managerialism” based on neo liberal principles.

    It is an old cliche that we get the politicains we deserve and as we increasingly live in the world of instant feedback,X factor, sack the manager if the team loses more than 3 games perhaps it is no surprise that we elect Ministers who want “success now”.

    The challenge for the development sector is how do we sell the “fuzzy Logic”, lets learn from failures etc to a target drive, success orientated media/public/political class?

    A reality TV show perhaps!

  7. Wot no Complex Systems vocabulary?

    Assuming I can out myself as having been at the meeting …

    … one of the things that struck me was that we had a very fruitful exchange without a mention of “complex systems”.

    It just wasn’t necessary. Everything that needs to be said about social systems can, I reckon, be said without reaching for that vocabulary.

    Yes, that language can be a useful corrective to the overly simplistic worldviews that have tended to dominate the development sphere (mainstream economics, I’m looking at you!), but there are other ways to talk about that messy reality.

    Ask a geographer. Or a sociologist. Or an anthropologist. Maybe not a mainstream economist.

    I didn’t say this at the CGD event, so I’m not breaking any rule? :)

    1. I agree up to a point Alan. Complex systems vocabulary is off-putting, and the people using it often seem to take a perverse pleasure in being incomprehensible (look how smart I am, I make no sense whatever!). But call it systems thinking, or complexity or whatever, there are useful generic lessons from thinking about the whole as different from the sum of its parts, and the language of complexity keeps you focussed on that.

  8. Have a look at Making Social Science Matter, by Bent Flyvbjerg. He is very strong on the power of case studies (as narratives) which can act as a mirror to society and engage the “reader” in deciding on meaning. He also has a good chapter on common misunderstandings about case studies, misunderstandings I see in development circles, particularly among some M&E types.

  9. You really should read that book by Prigogine I recommended, Duncan. Ilya Prigogine, The End of Certainty

    (Er, I probably shouldn’t have written this in a hurry but here goes..)

    It’s my personal conviction that the most important lesson to draw from Complexity Theories is not the mundane insight that the whole is different (not more, it could be less as well) from the aggregate. Even Classical Economic theory recognised that. (It’s just contemporary Econ that has gone on a binge)

    Rather, it’s the insight that only a very few things in this world simply ‘are’. Complexity is not something that happens between point A and B. B in itself is continuously sustained by a dynamic and complex process. For example, taking an institution like the ministry of agriculture in a given country and wanting to reform water allocation for irrigation (either through lobbying or policy consultation). One thing is to understand that the current system is determined by the complex interaction between policies during colonial times, former patronage, elite interests, popular politics etc.
    Another, and more important if you want change, is to recognise that these together are continuously upholding the system, strategically testing the boundaries, unconsciously shaping behaviour. The institution is sustained by the social transactions that are shaped by formal rules, strategic beliefs, behavioural norms, organisations and the physical (e.g. precipitation and canals).

    My worry is that the way in which the complexity debate in Development is taking place at the moment, the second doesn’t get enough attention. That’s a problem because it matters that what you’re trying to change is ‘alive’. You cannot pause the existing while you sort out and reshuffle all those elements into a new institution.

    I do believe that the PDIA or the Convening and Brokering idea manages to deal with this but I see two challenges. First, I wouldn’t be surprised to see that it is lost in the travel from the Kennedy School to practice (which is why I hope Andrews will stop talking about icebergs:)). Second, I’m not sure there’s being paid enough attention to the fact that while you may find an effective way to solve a specific policy problem by working with the grain, you may at the same time –because institutions are dynamical and therefore share elements with other institutions- end up reinforcing other institutions that are counterproductive.

  10. Great blog entry, and discussion.
    I would add that the proposed approach to aid projects (designing over time the “intervention” to be supported at a larger scale later) requires a big change in how evaluation is done. Currently the vast majority of evaluation resources go to ex post quantitative experimental or quasi-experimental evaluations aiming to prove an “intervention” works or did not work. These evaluation findings come too late to be useful, and, in any case, rarely give much insight into useful adaptations to the “intervention” or the implementation strategy, or useful changes in donor support. The iterations (and adaptations) you and Matt are talking about would require that evaluations be designed to inform real time learning, the findings would need to be examined on a continuous basis or at punctuated intervals. And, they’d used mixed quantitative and qualitative methods (including, but not only, case studies). In the education field they call these “formative evaluations”, in health services research, they call them “action evaluations”. Whatever you call them, aid agencies would need to recalibrate the evaluations they support. Probably development/ evaluation researchers would need to retool; relatively few, I believe, have experience with these evaluation methods.

Leave a comment

Translate »