Is effective global governance now impossible? If so, what comes next?

August 3, 2012

Dead presidents; academic sentences; Garifunas v charter cities; white journos on Africa; evolutionary economics; a wonk nightmare: links I liked

August 3, 2012

Can theories of change help researchers (or their funders) have more impact?

August 3, 2012
empty image
empty image

Got dragged into DFID this week for yet another session on theories of change. This one was organized by the DFID-funded Research for r4dtaglineDevelopment (R4D) project (sorry, ‘portal’). A lot of my previous comments on such sessions apply – in DFID the theories of change agenda seems rather dominated by evaluation and planning (‘logframes on steroids’), whereas in Oxfam, it is mainly used to sharpen our work in programmes and campaigns. But the conversation that jumped out at me was around ‘how do we influence the researchers that we fund to use theories of change (ToCs) to improve the impact of their research?’

It’s risky to generalize about ‘academics’, but I’m going to do it anyway. Let’s apply some ToCs thinking to academia as a target. Applying ToCs to try and understand why academics don’t use ToCs may feel a bit weird (like the bit in Being John Malkovich where Malkovich enters his own brain), but bear with me.

Let’s start with the 3i model – processes and decisions are influenced by institutions, interests and ideas. Because academia is largely non-profit making, institutions and interests are pretty much the same thing, and come down to incentive and career structures. Here I think DFID has a problem in getting researchers to be more concerned with impact – whatever favourable ideas are around in terms of academics wanting to change the world are likely to be neutralised by the institutional culture:

Career progression takes place largely through peer approval rather than through any ability to influence the world outside (in fact, being dubbed a ‘media don’ can damage your promotion prospects).

One of the big risks for an academic is being rubbished in public for being wrong, naive or insufficiently nuanced – academics love snark and gossip (not like NGOs then…) and that kind of kicking can damage your reputation for years. So there are strong disincentives to set out clearly your assumptions about how change happens (especially if they’re really naff, like ‘all you need is robust research to convince grateful-but-dim policy makers to change their misguided ways’, which I suspect is actually the theory of change behind a lot of research).

That fear of clarity may explain why when I worked as a publisher, I watched how perfectly good, clear writers started a PhD and were lost to me, entering into several decades of inaccessible post-modern gibberish before emerging blinking into the light as self confident, respected professors once again able to communicate in normal English (e.g. talking to a potential young author on Mexico. Me: ‘so who has the guns then?’ Author – light dawns after baffled look – ‘Oh, you mean the repressive apparatus of the state!’)

then a miracle happensWhat other ideas might ToCs suggest? That you need to reward and build alliances among the drivers of change (eg encouraging young Blattmanesque bloggers who ‘get’ communications and influencing, while doing your best to neutralise ‘blockers’ – custodians of the peer-reviewed flame, perhaps?).

Or that you need to spot and capitalise on windows of opportunity, since change is seldom smooth and continuous. In the UK, one such window of opportunity is the new version of the Research Assessment Exercise (RAE), the enormously influential scheme by which UK universities are assessed for state funding. The next round of the RAE, now renamed the ‘Research Excellence Framework’ concludes in 2014. Importantly, it will allocate 20% to impact, defined as ‘reach and significance’. Could DFID and other funders pick that up and use it in their own assessments?

How else could DFID help turn this around? It has a lot of clout, largely coming from its sizeable research budget (about £200m a year last time I looked). Here’s a few ideas, in no particular order and mixing up sticks and carrots.

How to get researchers to understand the minds and lives of the non-researchers they hope to influence? How about insisting that any recipient of a DFID research grant not only identifies the non-academic targets of their research, but gets credit if they manage to arrange to shadow these targets for a few days to find out how they absorb and use information (I learned more about advocacy from shadowing a UK Development Minister for a day than from dozens of workshops).

Publish (and require recipients to publish) stats on blogging, citations in the media (not just journals) and any other indicators of communications and/or impact, by named academics, in order to generate some positive competition. Let the league tables commence….

Start ‘a window of opportunity fund’ that specifically excludes new research in favour of funding previous or actual research recipients to rapidly repackage existing research in response to major new opportunities in terms of demands for new thinking – e.g. change of leadership in target institution, scandal, external shock etc.

In funding applications, insist on a proper power analysis/theory of change, including which target institutions are to be influenced, what the opportunity timetable looks like (eg new legislation or drawing up manifestos). If anyone limits their ToC to ‘changing the discourse’, they should probably be taken out and shot (unless they can plausibly suggest how they aim to achieve that).

Ask researchers to explain how they will involve both influencing targets and communications people in the governance of their research from the outset (rather than completing the research and then saying, ‘oh blimey, how do we communicate this to keep DFID happy, we’d better organize a seminar and send a copy to the Minister’).

There are also risks here – people are sometimes scarily ready to blur/erase the boundaries between advocacy and impartial academic research – more on that to follow.

I’m sure there are lots of other ideas – please send them in

Previous thoughts on getting research into policy here and here.

Other thoughts from the workshop here.


  1. Duncan I think the most insidious bit of all of this is, as David Booth has pointed out in his ‘Working with the grain and swimming against the tide’ paper(, that it can lead to creating perverse incentives for researchers to ‘bend’ their findings in order to make them more palatable to those they seek to influence. This is particularly the case if the ‘Theory of Change’ developed suggests that the research needs to lead to some sort of policy shift amongst donors in order to be judged succesful.

  2. In general my view is that researchers should firs worry about doing good research and then about communicating it. The fact of the matter is that there is not enough good research to go around on most issues and on most countries where the Aid industry is working.

    I like the idea that higher density (more information better disseminated) will inevitably lead to a more informed decision making community:

    This density includes organisations like Oxfam and others that can use good research to advocate. Sure they will be some researchers who keen to do so too but we still need a critical mass of researchers who are just good researchers (being a good researcher, by the way includes being out of the office and finding out about the real world and engaging with others).

    The problem with the TOC approach that DFID and its advocates use is that it is not interested in the complexities you describe, Duncan (and that David Booth and Emma Broadbent illustrate:, but on getting THEIR research to affect policy.

    Not research in general to inform policy but THEIR research in particular. This is the problem. This is why they come up with A theory of change that is nicely fitted into the LogFrame format and the then the Business Case. See my latest post on frameworks:

    A tiny final comment on ToCs. I remember using this term 3 or 4 years ago on a short note on M&E for DFID. I used it to refer to theories of change (elites, social movements, tipping points, etc.). I was asked to remove the word ‘theory’ because is sounded too academic and this would put people off. Now the word Theory has been shorted to T put next to o and C and it has lost all meaning.

  3. As a researcher turned ‘grateful-but-dim policy-maker’, I have experienced both sides under discussion here. I can only agree that the side I am currently on needs more input into the design of research if it is to be used and we may even be able to articulate what issues we are wrestling with. Researchers certainly need to be able to answer the ‘so what?’ question much more effectively than most of them currently do – in my experience that is much more of an issue than palatability problems. As Duncan says, a few days shadowing a target would be an investment well spent, not least to understand our political economy. Oh, and by the way, it might help if some researchers stopped patronising ‘the policy-maker’ in what they write and say. It might be a surprise, but many of us can understand what researchers do and say, even when they use long words (or numbers).

  4. Duncan – great post.
    One additional area to think of is the important role of knowledge brokers or intermediaries who help connect the academic and policy spheres. A few thoughts on this role from a previous blog post of mine:

    We need to get away from thinking of the policy maker as an audience for knowledge, and rather see them as actors in their own right. In this sense knowledge intermediation is about building relationships between policy makers and researchers, and also those who implement policies and those affected by them. This is important to be able to bring in the different perspectives on an issue so that researchers can also understand what kind of knowledge is useful to policy makers and how the policy making process works, and also to bring together the different types of knowledge that come both from scientific research, and from experience of implementing and being affected by policy decisions (or to put a human face on scientific information).

    This is also important to build trust between researchers and policy makers. While intangible, trust and mutual understanding is an important factor in whether a particular voice is listened to (especially since policy makers don’t have the time or means to verify the technical merits of the research they encounter). Building and nurturing these relationships over time (i.e. not only focusing on the content of the exchange) is a key yet under-emphasized role of the knowledge intermediary.

    Here’s the full post:

    @enrique not sure I agree that a researcher’s focus should be on good research first then communication afterwards. I think this depends on why you want to do research. If you want to research because you are seeking the truth (only) then that’s fine, but if you want to do research to improve practical implementation of policy or programmes I think you need to think of it from the outset – and if I were in an aid agency funding research I think I’d also want to know from the outset what the practical questions the research might at least potentially help me with).

    Purely academic research is great – but its not much use to aid workers apart from as a nice intellectual distraction from our day to day work.

  5. Interesting post Duncan, thanks. Sorry for coming to this late.

    The Justice and Security Research Programme at the London School of Economics and Political Science and The Asia Foundation are currently collaborating in a two year research project exploring ToC approaches to international development practice. The research and outputs from the collaboration aim to provide a basis for development organisations to produce more empirically-grounded theories of how change happens, leading to improved development programming. The collaboration between a research consortium and an NGO should also allow for better integration of wider social science research findings into tangible meanings for field practitioners.

    We are only really just getting under way, but our first publication ‘Understanding Theory of Change in International Development: A review of existing knowledge’ is now available online here:

    Hopefully this and forthcoming work will be of interest to yourself and others thinking about ToCs.

  6. @Ian. I think there is a myth that researchers are researching esoteric or abstract issues. Of no value to real life. Good research first does not mean literally studying useless stuff. It means doing good research. That is it. Good research design, pick the right methods, carry it professionally, etc.

    At a recent meeting in South Africa someone mentioned the concept of ‘Use inspired research’. I like that. Its better the narrow ‘demand led research’ that assumes the policymaker knows best (and has all the right questions -we know this is not true).

    But not all research needs to be immediately practical. Theories need to be developed, and tested. Some databases need to be played with. New research methods explored. Etc. Without these all the ‘practical research’ that funders want is not possible. Or, if it done, will be based on secondary (or worse) research.

    A good old ethnographic study; a historical account of a country, city or culture; a sociological reflection on the social changes a society endures over decades; etc are invaluable inputs for any serious ‘practical research’.

    That not more of this done is quite simply a shame.

  7. I would encourage wariness about efforts to develop sophisticated theories of change about how research will affect policy. There are usually too many unknowns about the various communication channels that may might connect providers and users, and many conflicting incentives for participants to report on their responses.
    I would focus on a more immediate and observable results and make some testable assumptions about consequences of variations in these results. I have outlined an argument in favour of one measure – defined in some detail,here

Leave a comment