One thing that could accelerate the evolution of RCTs is a better communication between researchers. Hopefully now it will be changing, as there is a new community forum dedicated to RCT http://rctrials.org

As stated in Duncan’s posting, the vast majority of social and economic development interventions are probably not randomisable. It’s been estimated that only 5% are, because of ethical reasons, since the control group population needs to be isolated from the intervention, at least during the study period, or for political reasons, such as the desire to reach certain populations or the neediest. RCTs may also not be feasible for certain types of programme, such as a national policy change. But there are other approaches to estimating impact rigorously which don’t require randomised assignment of an intervention to people or communities – so-called quasi-experiments.

Generalisability of findings is indeed incredibly important. What works in post-conflict Liberia may not be readily transferable to Rwanda, and even more so in Latin America. This is one argument why more impact evaluations in different contexts are needed. But, while the RCTs that were done in the US in the 1960s only provided a number for impact – what Melvin Mark calls “bare bone RCTs” – the emphasis in international development programme evaluation, especially with the push from 3ie and impact evaluation network NONIE, is for theory-based impact evaluation which aims to answer not just what works, or not, but why and in which contexts.

A good theory-based impact evaluation “opens the black box” by collecting information on the reasons for effectiveness, or lack of it, which can then be used to inform programme planning elsewhere. It builds on a theory of change and assesses information along the causal chain, as set out for example in a log-frame, to assess the reasons why interventions succeed or fail – what Chris Blattman has called Impact Evaluation 2.0. This information is just as likely to be opinions of participants and practitioners collected using beneficiary interviews and focus groups, as it is to be ‘hard’ numbers collected using surveys or administrative data. Use of mixed methods to triangulate data is a key tool in theory-based impact evaluation.

Despite earlier doubts, the range of interventions which are amenable to rigorous impact evaluation is large. For example, 3ie is funding RCTs of:
– Community driven development in Sierra Leone
– Monitoring patient compliance with tuberculosis treatment in Pakistan
– Expanding secondary education in Ghana
– Mexico’s Payments for Ecosystem Services Program
– Scaling up male circumcision service provision in Malawi
– BRAC’s Graduation Model in Ghana
– Micro-entrepreneurship support in Chile
– Diffusion of health knowledge through social networks in Burkina Faso
– Property taxation in Pakistan

Innovations are being piloted so that impact evidence collected and disseminated is relevant to all stakeholders, not just researchers in ivory towers. For example, 3ie’s Policy Window enables policy makers in developing country governments and NGOs to request programmes that they want to be evaluated. Systematic reviews aim to provide an unbiased assessment and synthesis of all the available evidence on an intervention; this includes the unpublished studies which may remain hard to access because they do not provide evidence supporting positive effects. 3ie, DFID and AusAID will be launching a joint call for systematic review proposals in late September.

We sincerely look forward to seeing the growth of theory-based impact studies and systematic reviews, and their dissemination and use by policy makers and practitioners in international development.

Duncan: thanks Hugh, really interesting comment. For Chris Blattman on Impact Evaluation 2.0, go to http://chrisblattman.com/2008/02/16/impact-evaluation-2-0/

The RCT bonanza seems to me to be a reaction to the data-less (or at least data poor) development of the past and a desire to be associated with the success of medical science over the past decades (explicitly stated by Duflo). RCTs are the gold standard of medicine, but the assumption of external validity is much more reasonable. Human beings, more or less, have the same biochemistry; if a drug works in an RCT, it probably will work for everyone (yet personalized medicine is challenging even this assumption). Human behavior is much more variable than human biochemistry, and so I agree with Duncan and others who criticize development RCTs on these grounds. I think both for personalized medicine and for development, new techniques will need to emerge to monitor and characterize local and individual differences.

It’s important to consider not just the factual truth of a claim, but also its local validity. To an NGO, “low external validity” doesn’t mean that we shouldn’t give out bed nets or deworm kids. It means that those interventions are probably a good place to start, but progress should be monitored and the intervention changed if it’s not effective. I’m very glad that I can search the Literature and find a hundred studies on the cost effectiveness of every possible intervention under the sun. But that only gives me a ranked list of options, not a rigid prescription. Neither medicine nor development can ever be done completely by the book; I am no more in favor of cookbook medicine than I am of cookbook development.

Like good doctors, development workers need to monitor the progress and impact of their work. There have been some pretty exciting developments in technology that I’ve been trying to stitch together for my NGO, Nuru International. I’ve been able to set up a mobile data collection system for our seed project in Kenya that has been extremely helpful. We use cell phones and the web browser on the phone to collect data with Google Forms at very low cost; the total set up was about $5000 for 75 users (every member of our field staff), and has been costing around $1/user/month in data costs. For more info: bit.ly/cOYOyg .

http://www.princeton.edu/~deaton/downloads/Instruments_of_Development.pdf

Good to see this post. Panacea or mirage? Neither, but a very useful development in a sector that is hardly the most transparent about success and failure.

I would second everything that Harry has posted about pitfalls, especially assumptions about external validity. Most methods, RCTs included, have low external validity – i.e. it is difficult to extrapolate research findings to other places and times. This is especially so for interventions where the causal mechanism plays out in a socially contingent, open system – contrast with a medical intervention where human bodies are more similar to each other than they are different.

RCTs have the highest internal validity of any method – they are excellent at isolating specific causal factors. The problem comes in conflating high internal validity with “rigour”. We usually want to be able to make inferences for how things will play out for populations, not samples, and so external validity is also important. I like Nancy Cartwright’s writing on this topic (the philosopher, not the woman who does the voice for Bart Simpson…):

http://personal.lse.ac.uk/cartwrig/PapersOnEvidence/Are%20RCTs%20the%20gold%20standard.pdf

That said, I think one of the best things about the randomistas is their sense of humility and inventiveness. These are not ivory tower academics. I really appreciate the spirit in which they work, and the method allows us them to be very transparent in testing out new ideas. For example, Oxfam America are running some on a savings group intervention (I think) in Africa. This really helps them understand whether this kind of intervention does work in that context, and if they do it well, they’ll be able to also understand why.

The reason RCTs get so much attention is because policy-makers / NGO directors (I hope so) can understand the methods and the results much more easily than a complex bit of econometrics, and so have a greater degree of trust in the findings.

See also Chris Blattmans’ excellent note on Impact Evaluation 2.0:
http://www.chrisblattman.com/documents/policy/2008.ImpactEvaluation2.DFID_talk.pdf

All power to the elbow of those with an interest in promoting more transparent and credible evaluation methodology in development agencies of all stripes – it’s a hard row to hoe.
Cheers,
James

You’ve made some good points – many of which are corroborated by last year’s ODI study of the production and use of impact evaluations (see link at the bottom).

Aside from it being unwise to engage in methodological argument with people who are so hot on that sort of thing, randomistas should, to a certain extent, be left to carry out their work which is essentially a scientific enterprise, producing ‘global public good’ knowledge. There are a few problems with this however:
– RCTs do tend to be carried out where they are methodologically convenient rather than where the new knowledge is really needed
– there is massive publication bias, by our count over 95% of published RCTs show ‘positive’ impact, which severely limits the ability to really learn

The bigger problem is about the relationships between RCTs and policy. They were hugely fashionable in the US in the ’60s until it was clear that they couldn’t deliver all that was hoped, and now it seems that development policy makers are in love with the idea of having clear numbers on the impact of all of their work. The problems:
– policy makers often assume a much higher level of external validity than is actually appropriate, and tend to ignore the careful caveats which come along with RCTs
– As even people like Duflo would have to admit, the RCT model is only suitable for measuring impact in a subset, nay a minority of the kinds of intervention required for development. However they are being given a disproportionate amount of attention
– this is an issue of cost-effectiveness as you say, about where we choose to spend evaluation budgets and with what coverage
– but this is also about the danger of demanding ‘rigorous impact evaluation’ (which =RCTs for some) incentivising those programmes which fit easily into that model. These are generally output-driven programmes like distributing bed-nets, and things which will struggle are governance, capacity building, budget support, policy influencing.

I wrote up my worries about these issues in this ODI opinion piece: http://www.odi.org.uk/resources/download/2811.pdf
And here is the ODI working paper on the production and use of IEs http://www.odi.org.uk/resources/download/3177.pdf

Cheers

Harry