All hail FP2P-reading nerds! Completing the round up of top posts from last year, the most read from 2017 is on research impact. Here’s the original for a lot of comments, many of them heaping scorn on me for being so out of touch – always a treat.
As someone who works for both Oxfam and the LSE, I often get roped in to discuss how research can have more impact on ‘practitioners’ and policy. This is a big deal in academia – the UK government runs a periodic ‘research excellence framework’ (REF) exercise, which allocates funds for university research on the basis both of their academic quality and their impact. Impact accounted for 20% in the last REF, which reported in 2014, and I’m told could get an even greater weighting in the next one, in 2021. That means hundreds of millions of quid are at stake, so universities are trying to get better at achieving and measuring impact (or at least looking like they are!) as they start to prepare for the next round of submissions.
And they have a way to go, based on my recent experience of listening to pitches from a range of researchers. Some interesting patterns emerged, both in terms of weaknesses and ways to address them.
First up, many academics fall into the classic mistake of equating frenetic activity with impact. Long lists of meetings attended, papers published, speeches given etc do not impact make. I think they should start from a different place – how would you set about convincing an intelligent, fair minded sceptic of the impact of your research?
That might be a challenge because the way a lot of researchers think about their impact seems pretty rudimentary. Finish the research, publish the papers, run a few seminars, produce a policy brief and (at the less stuffy end) bash out some social media and voila! Unfortunately, ‘blog it and they will come’ is often not a very convincing influencing strategy.
A popular alternative in REF submissions is the adviser/consultancy model – ‘the UN/Government asked me to present/be on a panel/draft their guidelines – now that’s impact’. A bit more convincing, but both approaches would benefit from a more explicit theory of change, running through some of the following issues:
Stakeholder and Power mapping
- For the change you are advocating, who are the likely champions, drivers and undecided who you can seek to win over
- Who has the power over the decisions you are trying to influence? They are probably going to be your main targets.
- Who/what in turn influences those targets? Is it evidence (if so, what kind?), or something else entirely, like the identity of the messenger?
- How do you get targets to be aware of and interested in your research in the first place (since both decision makers and practitioners live in the land of TL;DR)? Have you tried to involve them in it, for example by asking them to be on a reference group, comment on drafts or be interviewed for it? Much better than just adding your paper to their reading pile.
Is the change you seek likely to be smooth or spikey – some changes are largely crisis driven, in which case your influencing strategy will have to try to get ready in advance, identify and make the most of crises or other ‘windows of opportunity’. As Pasteur said, ‘fortune favours the prepared mind’. This is antithetical to the normal rhythm of research – a steady, high pressure grind of papers and presentations, largely oblivious to events out there in the world.
Insider v Outsider strategy
Insiders prize being ‘in the room’, and see themselves as deeply engaged in policy formulation. Outsiders are more interested in constructing alliances, working with (and providing ammunition for) activists and civil society organizations, and talking to the media. But both need to think through their influencing strategies. On the insider route, are the people in this particular room really the people making the decisions? Which invitations do you say no to (if only to avoid death by consultation)? For outsiders, what are the alliances you need in order to reach and influence the decision makers on this particular issue? In both cases, what kind of research (both in content and presentation) is most likely to get people’s attention?
Both approaches struggle with attribution. According to one insider ‘the ministry is a black box – you’re invited to speak. They say a polite thankyou. That’s it.’ How do you know if you changed anything? The challenges with outsider strategies are different – in an alliance of a dozen CSOs, thinktanks and universities, even if something changes, how do you apportion credit?
The efforts at attribution are currently pretty crude: count the citations and if you think your research has helped someone, you beg them for an email saying just how brilliant you are and how much impact you have had. Better than nothing, but pretty dubious as a sole source of evidence (right up there with job references in terms of rigour….).
There’s a lot that academics could learn from the MEL (monitoring, evaluation and learning) teams in the aid agencies here – they have some incredibly smart people working on this. I’ve seen Oxfam’s Effective Reviews really invest in trying to measure impact by constructing a plausible control group for comparison, through propensity score matching, or comparing and ranking different causal factors through process tracing. The academics I’ve talked to seemed entirely unaware of these kinds of methods. Given the multi-million pound benefits of getting it right, there may be a case for universities to outsource (and pay for) this either to ‘impact partners’ such as NGOs with more experience in the area, or to third party specialist organizations that could accompany researchers in design for impact, and then measure the results more rigorously.
Then there’s comms. The default tone is neutral and cautious – goals were to ‘shift’ this or ‘support’ that without ever specifying why or towards what. That feels more like avoiding mistakes than communicating a message. One alternative would be to come up with a clear mission statement – ‘we use evidence to change public opinion so that X happens’. If that’s too close to a credibility-damaging move into overt campaigning, academics should at least be able to clearly identify their USP (unique selling point) – eg ‘I’m the only person in the UK who knows what is going on in these particular far-off places’, and clearly state what their research is about in such a way that normal people and REF panels can understand.
That sounds like an extended moan, but perhaps most striking in the conversations I’ve had is how much impact appears to be taking place anyway – I came away with the impression that a bit more analysis and explicit discussion of assumptions and influencing strategies could be an incredibly productive exercise for Universities seeking to turn research into real impact.
And such was the heated discussion that ensued, that I wrote another post summarizing the points for/against.