How did the Randomistas get so good at influencing Policy?

I’m a critic of the degree of overselling of randomized control trials (RCTs), but there’s no denying that the randomistas have been phenomenally successful snake oil salesmen and women, persuading large chunks of Big Aid to adopt their approach to what constitutes evidence and truthiness. If you want to learn how they did it, try reading their 3 part blog post from Innovations for Policy Action (IPA), one of the main RCT thinktanks, introducing their 2025 strategy, with a particular focus on ‘co-creating’ research with the institutions and people that you are seeking to influence and doing that right from the start.

Two things caught my eye: their Theory of Action, and their guide to deciding how to prioritize among all the potential avenues for influence. The Theory of Action pays more attention than previously to the demand side – helping policy makers use the evidence you are churning out, summarized in the lower rows of this table:

Choosing what to prioritize means identifying issues that tick all 4 blobs in the diagram.

  1. A body of evidence to build on:One single study doesn’t often present the best policy opportunities. This is a generalization, of course, and there are exceptions, but typically our policy teams pay the most attention to bodies of evidence that are coming to a consensus. These are the opportunities for which we feel most able to recommend next steps related to policy and practice—there is a clearer message to communicate and research conclusions we can state with greater confidence.
  2. Relationships to open doors: Our long-term in-country presence and deep involvement with partners through research projects means that we have many relationships and doors open to us. Yet some of these relationships are stronger than others, and some partners are more influential in the processes we want to impact. We use stakeholder mapping tools to clarify who is invested and who has influence. We also track our stakeholder outreach to make sure our relationships stay strong and mutually beneficial.
  3. A concrete decision or process that we can influence: This is the typical understanding of a “policy opening,” and it’s an important one. What are the partner’s priorities, felt needs, and open questions? Where do those create opportunities for our influence? If the evidence would indicate one course of action, but that course isn’t even an option our partner would consider or be able to consider (for cost or other practical reasons), we have to give the opportunity a pass.
  4. Implementation funding: In the countries where we work, even when we have strong relationships, strong evidence, and the partner is open to influence, there is still one crucial ingredient missing: implementation funding. Addressing this constraint means getting evidence-based programming onto the agenda of major donors.

Thoughts?

 

Subscribe to our Newsletter

You can unsubscribe at any time by clicking the link in the footer of our emails. For information about our privacy practices, please see our Privacy Policy.

We use MailChimp as our marketing platform. By subscribing, you acknowledge that your information will be transferred to MailChimp for processing. Learn more about MailChimp's privacy practices here.

Comments

3 Responses to “How did the Randomistas get so good at influencing Policy?”
  1. Thanks for this, Duncan. It would be good to say that, before pointing any fingers, we all try to influence someone. OXFAM does, ODI (where I used to work) does, OTT does, etc. So, this is excellent advice for anyone who wants to influence policy (or inform it).

    But the IPA story (and its relationship with J-Pal – because they are never too far from each other) is an interesting case (certainly NOT black and white) which offers a lot of food for thought. It looks, from their strategy that there is an effort to address some of the critique in my comments below, buy I fear it is not enough. And I fear it will never be enough because their business model makes it impossible.

    First, they would accept that they are NOT arguing that RCTs are the only way to “know” and that RCTs are not for everything. And although I have heard them sell RCTs as a solution (for many things – even thought the new economic theory they are based on would have to accept that all an RCT can do is tell you about a cause and effect relationship that has happened in the past) I think the bigger problem is in not correcting those who choose to believe that RCTs ARE the only way to really know something and CAN solve every problem.

    Second, the business model that underpins this Theory of Action (we love jargon, eh?) is worth paying attention to much more closely. Take a ministry in a developing country. It buys into the idea that it must invest in RCTs as a tool for evidence informed policy. It sets up a “policy lab” or team to identify and commission them. If this is an IPA led project (sometimes paid for by a foreign funder or multilateral – or even by public funds) then you’d likely find that high level advisory support is likely to be provided by US/Europe based academics. This is something that IPA, through its partnership with J-Pal, can offer. Who’d say no? The new policy lab then chooses the interventions to test (helped by this international expertise) using RCTs. But who pays for them? In Peru, the Labs that have been set up have a “policy” of not paying for them. Local researchers, with the capacity to undertake RCTs, do not have public or private funds available for them to volunteer their services. Only foreign based researchers, with access to J-Pal (managed) funds or public private funding that is far more readily available in the US or Europe, can do this. And they do. Sometimes hiring local researchers for support but only as the junior partners in the relationship.

    The irony is that some of the funders of J-Pal and IPA globally are also funding local capacity to undertake RCTs. But without an effective demand…. what is the use of this new skill?

    Can we blame them (IPA, J-Pal, the academics in the US/UK, northern think tanks, etc.)? They are doing their job and they are likely producing good research. (Although I would question how relevant it is to the local context.) Is it their responsibility to care for the local research community capacity? They suggest it is….. but do they do enough? or, is their business model incompatible with a real and meaningful support of local capacity?

    Let’s say that there is local public or private money for RCTs and other related work. In those cases, should a foreign/global programme or organisation bid for this kind of work if there is growing local capacity -which needs to win a few if it is ever going to gain the necessary experience? Or should they recuse themselves if there are capable local options? Or offer their services as reviewers or to help disseminate their work more broadly?

    What should be, if any, the responsibility of “northern” academia and think tanks vis a vis their peers in the “south”?

    I have heard pitched by these institutions to governments. I do not suggest these reflect what theses institutions actually do or intent to do. But this is the impression they leave. In both pitches, they have emphasised that they offer:

    -Access to high level academics from the world’s top universities
    -Free RCTs
    -Global visibility

    This is almost total in contrast to what a local think tank or university may offer. They cannot work for free, they are not part of the world’s top universities, and they cannot offer global visibility.

    In a middle income/sized country this approach may undermine the local academic community and keep it from developing skills in the field. But in a low income or smaller country it will simply keep local academia from developing at all.

    And in the long run, those countries will have to deal with the costs of depending on foreign expertise.

    Again, is this IPA’s fault?

    At the very least, I would argue that, where it exists, local academia and think tanks need to be bolder and speak up about the manner in which RCTs have been oversold and how the “business” of foreign agencies (including funders) affects the often delicate balance of power (and opportunity) that enable local research communities to develop.

    First and foremost they have to fight their ground and spend as much or more time and resources as these foreign players spend making a stronger case for public and private support for them (and their institutions).

    The state has to step up too. It should consider the effect that bypassing local academia has to its long term future -and its impact on society: who will train future civil servants, entrepreneurs, politicians, researchers, etc.?

    I share your concern, Duncan. But one way forward that I would like to see is for IPA and J-Pal to play a more public role at the local level. Whatever the means, they are producing research. I rather they stay and contribute that stop altogether. This should contribute to the public debate on policy at the local level. Not simply pursuing private influencing through direct engagement with policymakers but holding their work up for scrutiny by their local peers. And scrutiny in policy research involves politics. This is what local research has to deal with all the time. This is what makes the IPA (and most foreign agencies in the “development sector”) business model a block for a significant and meaningful change.

  2. Priyanthi Fernando

    RCTs like Systematic Literature Reviews privilege a certain kind of knowledge and research over others, and are therefore highly political. Enrique has pointed out how RCTs privilege researchers in the global north, as does Systematic Literature Reviews, given that much of the globally acknowledged published research is often in journals that are inaccessible to researchers in independent think tanks, or even universities, in the global south. But it is also about privileging quantitative over qualitative or mixed method research, about valuing research published in journals , mainly English language journals etc etc – all ways that support and validate knowledge produced by the privileged, and maintain existing knowledge hierarchies.. Just saying….

Leave a Reply

Your e-mail address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.