I need your help: Theories of Change for promoting Empowerment/Accountability in Fragile States

July 28, 2016

Getting carbon inequality onto the political agenda: the lessons of Brexit

July 28, 2016

Deworming Delusions and the flimsiness of ‘evidence-based policy’

July 28, 2016
empty image
empty image

This post is co-authored with Mohga Kamal-Yanni (right)MohgaKamalYanni

Should I blog about things that are way over my head? Well it’s never stopped me in the past…… My LSE colleague Tim Allen, along with Melissa Parker and Katja Polman have edited an issue of the Journal of Biosocial Science on ‘Biosocial Approaches to the Control of Neglected Tropical Diseases’. It’s open access and worth a skim, because even though it’s health-techie, it contains some pretty explosive stuff.

In particular Tim and Melissa lob another can of petrol on the ‘Worm Wars’ fire with ‘Deworming Delusions’ a damning account of the way routine deworming has become an iconic development intervention in East Africa on the basis of highly questionable evidence and dodgy ethics.

Nice graph, but is it true?

Nice graph, but is it true?

To recap, deworming school age children is claimed to improve attendance rates, and so educational attainment. A famous 2004 study by Michael Kremer and Edward Miguel using data from Kenya in the late 1990s found a reduction in absenteeism of 25%, following deworming with albendazole at 6-month intervals. On the basis of this study, a vast industry was founded – ‘According to the Gates Foundation, this is the largest public health programme ever attempted. The relevant tablets are being donated in huge quantities by leading pharmaceutical companies, and free preventive chemotherapy is being rolled out on a massive scale, with countries in Africa setting the pace.’

But that edifice is pretty shaky. First Allen and Parker review the many conflicting studies and conclude:

‘Assertions about the effects of school-based deworming are over-optimistic. The results of a much-cited study on deworming Kenyan school children, which has been used to promote the intervention, are flawed, and a systematic review of randomized controlled trials demonstrates that deworming is unlikely to improve overall public health.’

Tim’s an anthropologist and so spends a lot of time investigating how people on the ground understand what is going on. His approach is the exact opposite of trying to distil an impersonal, decontextualized ‘scientific truth’ about the efficacy or otherwise of drug programmes – he highlights the human dimension and concludes:

‘There are social problems arising from mass drug administration that have generally been ignored. Notably, there are serious ethical and practical issues arising from the widespread practice of giving tablets to children without actively consulting parents.’

It was this second aspect that got my attention (I really can’t follow the endless to and fro over methodology between

Anyone ask their parents?

Anyone ask their parents?

protagonists in the Worm Wars). The authors point to a WHO recommendation in ‘School Deworming at a Glance’: ‘Don’t waste time and resources trying to examine each school or child. Deworming drugs are safe and can be given to uninfected children. No individual diagnosis, or assessment of each school is needed.’ The trouble is that:

‘Research carried out in Uganda and Tanzania has shown that deworming tablets can have side-effects, which are viewed as serious in local terms. Many children complained of stomach upsets after being treated in schools, some adults were incapacitated for days, and there were uncorroborated rumours that a few individuals had died. Such experiences and stories led to widespread anxiety and fear, and fostered rumours about the real purpose of the treatments. They also prompted a different kind of resistance in the form of a refusal to distribute or take the tablets, and occasionally confrontations with angry parents. …. If schools in a high-income setting were instructed to treat all children with a drug regime that would only benefit a minority, it would raise concerns. Indeed, it would probably be viewed as completely unacceptable to roll out such treatment without securing the consent of parents beforehand.’

But for the Worm War battlers, the authors conclude ‘the issue at hand is the divergent interpretations of data published in an old paper, rather than the actual effects of deworming policies on African children and their relatives’.

My impressions from all this? That the solidity of ‘evidence-based policy’ is often much more tenuous than is claimed – what is presented as ‘hard evidence’ is often contested just as fiercely as any historical or social science claim. The search for scientific certainty can easily lead researchers into ignoring the ‘biosocial’ – the interplay of norms, practices, opinion, politics and power that shapes how any apparently neutral intervention plays out. And maybe that we need more anthropologists as well as medics and economists.

Other essays in the issue address this wider point. In the introduction on the wider topic of “neglected tropical diseases”, the editors say:

Words on worms

Words on worms

‘Social priorities, social relations and social behaviour profoundly influence the design, implementation and evaluation of control programmes. Yet, these dimensions of neglect are, themselves, neglected. Instead, emphasis is being placed on preventive chemotherapy – a technical, context-free approach which relies almost entirely on the mass distribution of drugs, at regular intervals, to populations living in endemic areas.’

I asked one of Oxfam’s health gurus, Mohga Kamal-Yanni, for her views, and got this in response:

I share a lot of their concern on programmes that ignore the local (and medical actually) realities and needs.  The world faced the same issue in responding to Ebola.  Donors wanted to rush workers in space suits to tell communities what to do and tell patients to come to treatment centres (where people knew they would die).  It took few valuable months for donors and governments to realise that they had to first understand the local reality and to engage people in planning and performing activities.

Worms eat away kids’ nutrients and therefore negatively affect their health and their cognitive abilities.  Deworming programmes could be beneficial to kids’ health and education if they were implemented with local knowledge and participation.  Like many development programmes, it is not a question of what, but how.

I loved that somebody critiques the perception of “community” as a homogenous group that loves and cares for each other.  Such beliefs are also widely accepted even within NGOs, as if people in the South are different from people here. I remember in the 80s an Oxfam engineer was shot at because of disputes over the site of the water projects in a small isolated village.  For me “community” equals power relations whether that grouping of people is a remote village in Sierra Leone or a neighbourhood in London.

However, I disagree with the authors on their analyses of the concept of “neglected diseases”.  I think the concept has been very useful in raising attention and investment in research for treatment of diseases such as TB and sleeping sickness, where the only options available for doctors have been very old complicated regimes or even toxic medicines. In reality these diseases have been way off the radar screen of pharmaceutical companies because they will not make profits from selling a medicine to a poor African patient. It is only now, after much global effort, including by civil society, that governments, the pharmaceutical industry and philanthropist have engaged in researching new diagnostics and medicines for these diseases.  It is worth noting that some of the medicines used for deworming, like albendazole, were actually developed for veterinary use (where profit is possible) and are very cheap to produce.  “Neglected diseases” are the stark example of the failure of the current global R&D system to produce health technologies needed for public health.  The system is driven by commercial profits based on pharmaceutical companies’ monopoly on medicines.

I wish I had time to blog about socio-medical interventions!’

Well now she has – note to colleagues, careful when sending me interesting emails, you never know where they’ll end up……

And here’s Tim presenting his paper (30m), and then being interviewed (20m)

7 comments

  1. Blimey – desertification demolished one week, deworming the next. Is nothing sacred? In both cases I wonder whether the evidence will change the industries that have been built on, and protect, these sacred tenets?

  2. I don’t think deworming has been demolished. Even the critics referred to don’t seem to say so:
    Chris Blattman, in the same in the same “10 things i learned” blog post that Tim refers to, has this in point 1: “My conclusion is that the Kenya deworming results are relatively robust” [http://chrisblattman.com/2015/07/24/the-10-things-i-learned-in-the-trenches-of-the-worm-wars/] (in another post Blattman writes this: “There are clearly serious problems with the Miguel-Kremer study. But, to be quite frank, you have throw so much crazy sh*t at Miguel-Kremer to make the result go away that I believe the result even more than when I started” [http://chrisblattman.com/2015/07/23/dear-journalists-and-policymakers-what-you-need-to-know-about-the-worm-wars/ ].)
    Macartan Humphreys, also cited as someone who took down the study and showed it to be wrong, is more critical, but not entirely dismissive either: “My conclusion is that the replication has raised (or in some cases, highlighted) important questions both over the strength of evidence for spillovers and for the strength of the direct effects of deworming on school attendance – at least insofar as these pass through a worms mechanism” [http://www.columbia.edu/~mh2245/w/worms.html].
    The Cochrane Review is perhaps most critical, but still reserved: “There is good evidence that regular treatment probably has no effect on … formal tests of cognition (moderate quality evidence), or exam performance (moderate quality evidence). We do not know if there is an effect on school attendance (very low quality evidence)” [http://www.cochrane.org/CD000371/INFECTN_deworming-school-children-developing-countries].

    None of this suggests to me that the original Kremer/Miguel study has been entirely debunked or dismissed, that we now know it’s all wrong and so on. There are a lot of important lessons to be learned from this, lesson’s that Allen and Parker’s wonderful work in this area point to, but “it was all wrong, after all” is probably not the main lesson to be learned.

    1. Exactly – I don’t think the case for deworming has been “debunked” or dismissed (though I don’t think the blog post was claiming that). In fact a new meta-analysis appeared a couple of weeks ago, criticising the Cochrane review from last year: http://www.nber.org/papers/w22382 Chris Blattman’s comment is here: http://chrisblattman.com/2016/07/11/14340/

      I also think that the review of the evidence by Givewell is very good: http://www.givewell.org/international/technical/programs/deworming But it doesn’t address the consent issue.

  3. I feel like the title of this blog was a bit misleading. The evidence for giving preventative worm treatment is quite strong and the blog shows both sides.
    In criticising the school worming, the many risks of nematode carrying are not in questioned. A routine antelmintic program is likely to reduce the number of worm eggs in the environment and thus reduce worm burden and further malnutrition, larva migrans, intestinal bleeding, bronchitis and so on. This is an important aspect. To scrap a deworming program because of spurious interpratation of school attendance studies (which could have been influenced By many factors) sounds reckless.
    The main issue for me is the lack of consent from parents – or at least prior information to allow them to opt out. I can see some comparison in Finland in primary school, where all kids in my son’s class were given xylitol tablets after lunch. Parents could opt out, but I don’t think anyone did, as it was considered a no-brainer. In the case of deworming the benefits extend beyond the kids themselves, By decreasing environmental contamination.

    1. My bad Pamela, I guess the flimsiness lies more in how such policies are implemented, rather than the policies themselves.

  4. I appreciate Tim bringing up the ethical issue of widespread deworming of children without parental consent and Duncan for highlighting this. The effectiveness of community deworming has been challenged for some years in an independent assessment by Cochrane, but there hasn’t been much traction for these findings with the economists, parasitologists and development specialists who promote community deworming in poor countries.
    Obviously a child with worms should be treated, this is not the question. The question is whether governments should treat all the children in their country for fear they have worms that are harming them, and this produces a societal and community benefit. For example, the national deworming day in India targeted 140 million children. This is not a “cheap” exercise for anyone.

    It seems decisions about policy in this area has often been based on this single study by Miguel and Kremer to which you refer. The mistake that has been made is that this is not the only trial. In fact, there are more than 45 trials evaluating deworming, as outlined in the most recent version of the Cochrane review. Decisions in policy should be made from a body of evidence, not one individual trial. Please do look at this review on: http://onlinelibrary.wiley.com/doi/10.1002/14651858.CD000371.pub6/full

    I have coordinated a team preparing and updating this Cochrane review in the light of new trials, new methods, and comments, since 2000. What is extraordinary is that over time evidence has accumulated to now show really quite good evidence of NO effect. So this leads to an additional ethical problem: should governments and philanthropists be exerting effort and spending money on a policy where there is quite good contemporary evidence that it has no important effects?

    Paul Garner, Centre for Evidence Synthesis in Global Health, Liverpool School of Tropical Medicine

  5. First a big thank you to Duncan Green for taking the initiative to blog on these important issues and to all those who have added their own remarks. Here are a few responses to comments by Mohga, Ken, Rob and Paul.
     
    Mohga Kamal Yanni says that she disagrees with our analysis of the concept of neglected diseases, but we don’t think she really does! We certainly agree that the introduction of the term ‘Neglected Tropical Diseases’ around 15 years ago has concentrated attention on largely ignored infections. Indeed, it has resulted in what has been hailed as the largest public health programme ever undertaken. However, the success of the term has created problems, which we have analysed in detail. One issue is that the term makes no biological sense. It refers to a wide range of diseases that are very different to one another, and which often require very different kinds of treatment.  To use the term as some sort of homogeneous category can thus be misleading. Also, the fact that ‘neglect’ arises from social priorities, social relations and social behaviour is commonly set aside. It is neglected people who are typically infected with these diseases; and that kind of neglect has political and economic connotations that cannot be wished away.
     
    Another point discussed in our paper ‘Deworming Delusions?’ is the confusion that arises between the terms ‘mass drug administration for neglected tropic diseases’ and ‘deworming’ (http://journals.cambridge.org/download.php?file=%2F1858_8A648315C426F852070D1357F46A2694_journals__JBS_JBS48_S1_S0021932016000171a.pdf&cover=Y&code=01089c670d2a9f8f6cb278dbfe6e39fd). Both terms involve treatment for particular diseases with tablets and they can, in practice, be the same; but, the former often refers to treatment for multiple and diverse diseases in a population, while the latter often refers specifically to the treatment of children for a range of parasitic worms called ‘soil transmitted helminths’. This conflation can result in considerable confusion in the academic literature and public debates. Arguments about one are applied to the other, without a careful unpacking of what is being addressed.
     
    It is one of several problems with the seminal article by Ted Miguel and Michael Kremer on ‘deworming’ in Kenyan schools. Debates about that work are central to the fierce debates that have been described as ‘Worm Wars’. However, a point which is almost always overlooked is that their 2004 article actually deals with treatment for both soil-transmitted helminths and a very different kind of disease: bilharzia (schistoma mansoni), and the authors do not separate the possible effects of treatments with two different drugs in their analysis. Ken Shadlen’s comments relate to this 2004 paper on deworming by Miguel and Kremer.

    Ken observes that deworming has not been demolished, and refers to the work of Macartan Humphries and Chris Blattman. To some extent we agree. No-one – as far as we know – would argue that heavily infected people suffering from parasitic infections should not be treated. However, the 2004 paper by Miguel and Kremer on deworming in schools in western Kenya has been given huge status, and is often the main or only cited source of evidence that school-based deworming is effective. A re-study of their data set, which was actually collected as long ago as the late 1990s, has shown that the findings presented in the 2004 paper were flawed or wrong. It was this restudy by Alexander Aiken, Calum Davey, James Hargreaves and Richard Hayes that initially triggered the ‘Worm Wars’ debate.
    http://ije.oxfordjournals.org/content/early/2015/07/21/ije.dyv127.full
    http://ije.oxfordjournals.org/content/early/2015/07/21/ije.dyv128
     
    The contribution to that debate by Macartan Humphries that Ken Shadlen highlights is one of the most thoughtful and careful contributions to the discussion. He is restrained in the wording of his criticisms of the 2004 Kenyan deworming paper, but a close reading of his assessment ends up being devastating. He re-enforces the points made by Aiken et al, and agrees with them that the results reported in 2004 by Miguel and Kremer were unsound.
     
    Chris Blattman, who openly acknowledges that he worked with Miguel and Kremer on some of their Kenyan deworming work, does his best to be more supportive of their 2004 paper in the blog post that Ken also cites, while acknowledging some of the problems with their econometrics. But, he does not engage with the findings of other studies, or the wider debates about whether or not school based deworming in practice is likely to be effective. Chris ends the blog post that Ken cites with the following statement: ‘To me, the real tragedy is that, 18 years after the Kenya deworming experiment (which was not even a real experiment) we do not have large-scale, randomized, multi-country, long-term evidence on the health, education, and labour market impacts of deworming medicine. This is not some schmuck cause. This is touted as one of the most promising development interventions in human history.’

    As Paul Garner makes clear in his comment, Chris is unaware that there are more than 45 other trials evaluating deworming. Some of these have been on a large scale. The results have been collated in a Cochrane Review (Cochrane Reviews are generally regarded as the ‘Gold Standard’ of medical evidence), and tend to show that deworming programmes have minimal or no positive effects. When the latest version of the Cochrane Review on deworming was published in 2015, the World Health Organisation’s Advisory Group on Neglected Tropical Diseases took the extraordinary step of setting aside the conclusion. This must have been an unprecedented move for a WHO committee. Those associated with the mass deworming agenda went so far as to argue that the Cochrane Systematic Review methodology is not appropriate for assessing its impact, and that it is of little value in guiding global deworming policy.

    While there may well be problems with overemphasising evidence from random control studies, the WHO dismissal of the Cochrane Review process is ironic. An important reason why Miguel and Kremer’s 2004 article has been given so much prominence is that it has been viewed as replicating the rigour of the random control design. Unsurprisingly, therefore, Miguel and Kremer have taken an alternative approach to the Cochrane Review. They have tried to show that its conclusions are incorrect. They do this in the new paper to which Rob draws attention in his post (K.Croke, J.H.Hicks, E.Hsu, M.Kemer and E.Miguel, ‘Does Mass Deworming Affect Child Nutrition? Meta-analysis, Cost-Effectiveness, and Statistical Power’ July 2016 http://www.nber.org/papers/w22382). This is a draft article that has been circulated recently, and which engages with the findings that Paul Garner summarizes in his post. It is basically a critique of the whole ‘Cochrane Review’ of random control trials dealing with deworming.

    Croke et al make econometric arguments which Paul and his colleagues will need to address at some point, but it seems appropriate to make some brief non-technical comments here. In a nutshell, Croke et al argue that the Cochrane Review of deworming Random Control Trials is ‘underpowered’, because deworming treatment is so cheap that even if it has a very small overall effect, it is still worth doing. It is suggested that the key question facing policymakers is whether the expected benefits of mass drug administration of tablets to potentially infected populations without any individual testing exceed the estimated $0.30 per treatment cost.
     
    This strikes us as an inherently problematic argument. To begin with, it is not clear where the figure of $0.30 per treatment comes from. The source is given as GiveWell, an organization which endorses the work of deworming charities, but does not do field research. Yet even the figure used by parts of the World Health Organisation promoting deworming is $0.50, and other estimates are well above $0.60. It depends, in part, on which drugs we are talking about. Mass drug administration for lymphatic filariasis in Haiti, which targeted 105,750 people, reportedly cost $2.23 per person.
     
    So where is the evidence that mass deworming is being effectively delivered at $0.30 per person? If there is such evidence, can it be replicated and up-scaled? Importantly, what are the consequences of chasing this $0.30 target?

    To do deworming on the cheap implies that funds are not available for some rather important activities. It requires most or all of the following: (1) the drugs are indefinitely donated free of charge by pharmaceutical companies; (2) most/all of the costs of transport are excluded; (3) those distributing the tablets are volunteers and given only the most rudimentary training; (4) the salaries of national staff in Ministries of Health, staff in district offices, staff in medical schools, and staff in international organizations are all excluded from the calculation; (5) there is no significant expenditure on health education, or ‘community’ mobilization, or the seeking of parental support; (6) there is no significant expenditure on effective monitoring of whether or not those targeted have actually swallowed the tablets, and (7) there is no significant expenditure on monitoring of the social and biological effects of handing out so many medicines; (8) there is no significant expenditure on vector control or other measures to reduce the rates of reinfection.
     
    The problem is that if these things apply and the target for low cost deworming is achieved, the possibility of the programme sustainably controlling diseases is unlikely. Interestingly, Miguel and Kremer have themselves recognized this in an excellent paper they published in 2007 (http://www.nber.org/papers/w10324). Also, with respect to point (8), it worth noting that recent research on school-based deworming for soil transmitted helminths in the same locations of Kenya where Miguel and Kremer’s old data set was gathered in the late 1990s, has demonstrated that deworming can be effective when there is a corresponding improvement in other things, including improved sanitation. In schools were there have been no such improvements, treated children are almost immediately re-infected (Nikolay et al, 2015, PLoS Negl Trop Dis 9(9): e0004108. doi:10.1371/journal.pntd.0004108).

    If ‘deworming’ is to be done well and to really change lives for the better, it requires a biosocial approach, drawing on evidence from diverse disciplines (http://journals.cambridge.org/download.php?file=%2F1879_8A648315C426F852070D1357F46A2694_journals__JBS_JBS48_S1_S0021932016000389a.pdf&cover=Y&code=2b2e9727133113aa83c2e95bce65491e). It also requires serious engagement with the impoverished people who are at the receiving end of deworming programmes. No matter how cheap deworming can be made to seem according to certain modes of accounting and econometric modelling, if it does not work, then it is still expensive.

    There are, moreover, ethical concerns that urgently need addressing. A policy that is, in practice, based on teachers and volunteers rounding up thousands of school-age children and making them take tablets without parental approval, adequate explanation or any proper monitoring should be as unacceptable in Africa as it is in other parts of the world.
     
    Tim Allen (London School of Economics) and Melissa Parker (London School of Hygiene and Tropical Medicine)

     
     

Leave a comment

Translate »