What does the evidence tell us about ‘thinking and working politically’ in development assistance?

We’re having an ‘Adaptive Management week’ on FP2P, because so much good material has been coming through recently.

First up is a new paper by Niheer Dasandi, Edward Laws, Heather Marquette, and Mark Robinson that I read on the way to the TWP conference in Washington that I wrote about recently. It really got me thinking.

The paper is pretty damning: ‘much evidence is anecdotal, does not meet high standards of robustness, is not comparative, and draws on self-selected successes reported by programme insiders.’ Ouch.

To arrive at this, the authors analysed and compared 44 case studies – a really useful contribution, which generates some important insights:

The dangers of a single, very influential piece of research: ‘If TWP is at its heart about illuminating contextual differences in order to move away from ‘cookie cutter’ best practice approaches, then we would expect to see variations in programme design, implementation and outcomes. However, while many different case studies have been published since Booth and Unsworth’s comparative study, there is very little, if any, variation between them along these factors. Indeed, given the similarities highlighted below, it is difficult, if not impossible, to discern if the patterns that are beginning to emerge from comparing the various cases genuinely reflect an emerging consensus, or if, in fact, they reflect growing ‘group think’ among TWP insiders about the necessary programme design characteristics.’

It’s all their fault

Blind spot on Fragility and Conflict: Of the 44 programmes that we identified as being the subject of TWP research, only seven are based exclusively in countries that are featured on the World Bank’s most recent Harmonised List of Fragile Situations. Given the growing concentration of aid from major donors, including DFID and the World Bank, in fragile and conflict-affected states, a greater emphasis of TWP research efforts in violent and unstable political contexts would seem to be important given the untested nature of these ways of working. [nb this is what we are trying to do in the A4EA research, but for some reason our Myanmar paper did not make it into the selection of case studies].

Unintended consequences of hiring connected lobbyists: Given the argument found in many of the case studies that effective programmes require politically well-connected staff, there has been surprisingly little analysis about how these staff are recruited, how their activities are assessed or what this may mean in practice in politically divided societies. I sent this post in draft to the authors, and Heather Marquette said in reply

‘’I’m increasingly worried about potential unintended consequences from untested approaches that carry their own

risks: what will happen if aid programmes start routinely embedding politically well-connected insiders with the ability to direct resources? Will ‘working with the grain’ help to speed up or slow down the closing of political space? What does any of this mean when programmes are being delivered by non-aid actors with close links to intelligence operating in conflict-affected areas? Basically, these programmes are experiments, but they’re not treated like experiments, with the sort of safeties you’d build into an experiment, both at the case study level and at the body of evidence level.’

All the case studies were reform programmes, with governance, justice and security and infrastructure being the most common: TWP might look similar, in terms of programme design, for reform programmes, regardless of sector; whether or not that is useful for someone trying to design an infrastructure programme, or a service delivery one, is not clear.

Donor dominated: The literature on TWP programmes is focused primarily on the role of bilateral and multilateral donors. For the most part, the agencies in question are DFID, DFAT and the World Bank.

But why should we trust the peers?

Weak Evidence base: The literature continues to be almost entirely made up of single-programme case studies, with few attempts at comparison and written for the most part by programme insiders. Even [more rigorous efforts] rely largely on interviews and documentary analysis, or a form of action research, rather than methods more appropriate for establishing causal explanations, and approaches to triangulation are often unclear or entirely absent. As a result, in the case studies reviewed it is often hard to discern a direct causal relationship between TWP and the outcomes that were said to have been achieved. Only one study in our sample discusses counterfactuals, and very few discuss challenges faced in the programmes or areas that were unsuccessful….. Studies rarely focus on outcomes, instead focusing on the reform and/or programming process instead.

This is a really good challenge, but I have some serious questions about the paper too:

Much of what is TWP is not actually called TWP – for example there are no case studies deriving from the work of the Building State Capability team, and its work on ‘problem driven iterative adaptation’, or any of the work of the Action for Empowerment and Accountability programme, including papers I have written on donor theories of change, and Pyoe Pin in Myanmar.

But more interestingly, what would constitute evidence in the eyes of the authors? There are in depth, warts-and-all case studies of the Coalitions for Change programme in the Philippines by an independent academic John Sidel (also not in the reviews list) – are they evidence? Would process tracing help establish the degree of attribution a TWP programme can claim for a given ‘win’? In another version of the paper the authors call for ‘triangulation’ (not sure what they mean by that) and greater discussion of failures. In reply to my email, Niheer Dasandi expanded on this point:

‘The papers we reviewed imply causality – they clearly suggest that the use of TWP has led to improved outcomes. In fact, that’s at the centre of most of these papers. However, very few of them directly engage with the issue of causality. I think that’s the main problem rather than them really trying and us simply dismissing their efforts as ‘anecdotes’. If you were to seek to justify the causal claims then there a whole range of things that might arise that will help you evidence the changes that have occurred as a result of incorporating TWP components into a programme. This might be spending a bit more time discussing similar programmes that were tried in the same context that did not take a TWP approach, and didn’t have the desired results. It might mean comparing different programmes. It might be presenting different forms of data/evidence that all point to a TWP aspect of a programme being critical for achieving the positive results – this is what we mean by ‘triangulation’. Using different forms of data – interviews (with a range of different actors – something often not done), focus groups, reports, media coverage, any available statistics (that show improved results), etc. Many of the studies we look at provide a few quotes from people involved in the programme, but don’t seek to really justify their claims with different forms of data.’

I actually think what they dismiss as ‘action research’ is probably the best way to provide convincing evidence that pays due attention to context, chance, and the interaction between political economy and human agency and leadership. How else can you ‘establish causal explanations’? But anyone who is predisposed to be sceptical will say ‘that’s just an anecdote’. Anyone got a better idea?

Tomorrow: Heather Marquette explores the differences between all the acronyms (TWP, DDD, PEA etc) and why it matters

Subscribe to our Newsletter

You can unsubscribe at any time by clicking the link in the footer of our emails. For information about our privacy practices, please see our Privacy Policy.

We use MailChimp as our marketing platform. By subscribing, you acknowledge that your information will be transferred to MailChimp for processing. Learn more about MailChimp's privacy practices here.

Comments

10 Responses to “What does the evidence tell us about ‘thinking and working politically’ in development assistance?”
  1. John

    Glad to see strong pushback on the latest alphabet soup in development- TWP. For starters, isn’t it axiomatic that an NGO – which by their “corporate” status – NG – haven’t a bloody clue with G is all about? So .orgs, think tanks, University think tanks, research institutes… by definition are incapable of TWP. On the other hand donors and .coms work in this space every day! So, how do people with a smug disrespect for politics and government suddenly see TWP as a winning strategy? Could this be explained by self interest? Hahaha.. of course! Therefore, let people who see politics as a « plus » do what they know how to do without attacking to their efforts a meaningless label that tries to provide legitimacy to amateurs. If you have never worked in or for Government or sought public office there is no way in the world you can TWP.

    • Duncan Green

      Not sure it’s that simple or stark, John. Virtually everyone involved in TWP (even me!) seems to have worked in both government and non government arenas, so does that mean your objection falls?

      • John

        Well of course if you were working for DFID, you were expected to TWP… that was the essence of your job. However, if you were working for an NGO without any prior exposure to public policy, financial management, service delivery, etc. I doubt that you have the experience to TWP…. Ohh, you may include it in your proposal and score some points but, actually, I doubt it. Many of the TWP claimants couldn’t get an appointment with a senior official to discuss policy options or suggest priority investments. Those who can are to be admired; those who just wear the TWP badge are not.

        • Duncan Green

          Thanks John, Afraid I don’t share your optimism about DFID (often dominated by technocratic/economistic approaches) or your pessimism about NGOs (increasingly sophisticated in TWP in their influencing work and elsewhere.

  2. Nigel Thornton

    We need this challenge. However, let’s be careful about what TWP is and is not and while doing it watch out for babies being thrown out with bathwater.
    My take is that PEA and then TWP was intended to be a corrective to ahistorical, decontextualised technocratic development solutions. Chambers and other long-toothed gurus have always taught the practitioner community to embed programming in reality. Helping practitioners think about politics and history (as Sue Unsworth was and Booth is passionate about) is A GOOD THING.
    TWP gives a framework for doing that, but as Jonathan Fisher and Heather Marquette argued about PEA in 2014, there are dangers when practitioners confuse the production of reports with ways of being and thinking (see here https://www.dlprog.org/publications/research-papers/donors-doing-political-economy-analysis-from-process-to-product-and-back-again).
    I think that TWP isn’t really a technology of its own. It’s a thought paradigm, not a magic bullet. And as a result, I don’t think that badging a programme TWP is particularly helpful at all.
    In ICAI’s 2018 Review of Governance in two DFID offices (https://icai.independent.gov.uk/report/governance/) we saw how DFID had yet to fully internalise TWP. There were lots of analytical reports being produced and TWP (and even more PEA) type language being banded about. The bureaucratic way of being, even for right on development types, tends towards reports and producing cogent analysis rather than creating space for thought, discussion and engaging with sticky issues on a continuous basis.

    I’d strongly support the finding in this paper that there is insufficient diversity of views or voices in the TWP debate; there is a default to those who commission think pieces know (again, we found that in ICAI’s Governance work in Uganda and Nepal). We are asking the same people to write reports and finding ourselves trapped in group think too often.

    I’d argue there are lots examples of good programming, where TWP principles have been used, but they’ve not been badged as such – we should be looking elsewhere for our evidence, I’d suggest. One of the source examples that Unsworth and Booth quote in their paper is the review for ICAI of DFID’s work on livelihoods in ODISHA (https://icai.independent.gov.uk/wp-content/uploads/ICAI-Report-DFIDs-Livelihoods-Work-in-Western-Odisha.pdf). That wasn’t at the time termed TWP. It was just really good development practice, drawing on long tried participant approaches, local implementation and field work nous.
    We need this challenge. However, let’s be careful about what TWP is and is not and while doing it watch out for babies being thrown out with bathwater.
    My take is that PEA and then TWP was intended to be a corrective to ahistorical, decontextualised technocratic development solutions. Chambers and other long-toothed gurus have always taught the practitioner community to embed programming in reality. Helping practitioners think about politics and history (as Sue Unsworth was and Booth is passionate about) is A GOOD THING.
    TWP gives a framework for doing that, but as Jonathan Fisher and Heather Marquette argued about PEA in 2014, there are dangers when practitioners confuse the production of reports with ways of being and thinking (see here https://www.dlprog.org/publications/research-papers/donors-doing-political-economy-analysis-from-process-to-product-and-back-again).
    I think that TWP isn’t really a technology of its own. It’s a thought paradigm, not a magic bullet. And as a result, I don’t think that badging a programme TWP is particularly helpful at all.
    In ICAI’s 2018 Review of Governance in two DFID offices (https://icai.independent.gov.uk/report/governance/) we saw how DFID had yet to fully internalise TWP. There were lots of analytical reports being produced and TWP (and even more PEA) type language being banded about. The bureaucratic way of being, even for right on development types, tends towards reports and producing cogent analysis rather than creating space for thought, discussion and engaging with sticky issues on a continuous basis.

    I’d strongly support the finding in this paper that there is insufficient diversity of views or voices in the TWP debate; there is a default to those who commission think pieces know (again, we found that in ICAI’s Governance work in Uganda and Nepal). We are asking the same people to write reports and finding ourselves trapped in group think too often.

    I’d argue there are lots examples of good programming, where TWP principles have been used, but they’ve not been badged as such – we should be looking elsewhere for our evidence, I’d suggest. One of the source examples that Unsworth and Booth quote in their paper is the review for ICAI of DFID’s work on livelihoods in ODISHA (https://icai.independent.gov.uk/wp-content/uploads/ICAI-Report-DFIDs-Livelihoods-Work-in-Western-Odisha.pdf). That wasn’t at the time termed TWP. It was just really good development practice, drawing on long tried participant approaches, local implementation and field work nous.

  3. Sandra Thompson

    I agree with Nigel Thornton that TWP is a thought paradigm, and for me an essential thought paradigm for working in systems thinking and development. When we know that the political is personal understanding the how twp will influence our approach and possible outcomes is essential.

Leave a Reply

Your e-mail address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.