More details can be found here: http://r4d.dfid.gov.uk/Output/194107/Default.aspx

But you’re right Duncan, we should have got better at this by now. I remember going to this ODI debate on the “long term effects of project aid” years and years ago and wondering then why so little of this work happens. http://www.odi.org/events/117-long-term-impacts-project-aid-evidence-china

This is a really interesting post, and something that is very close to my heart.

I’ve been involved in emergencies work where the focus is often on the next big disaster and not what happened or was learnt after the last one. KP (formerly NWFP) that Masood refers to was one of the areas that was badly affected by the Kashmir earthquake in 2005, so I wonder if that’s part of the reason why most of the people involved in the design and implementation both in the Government and donors have long gone…? Any ideas Masood?

That said, the comments in this post have given me hope that things are changing!

I’m familiar with the journey that VSO is on as Janet and I have been in touch to swap ideas about post-project evaluations.

In case anyone is interested, EveryChild is currently in the process of closing all of our international programmes through a carefully managed process, and transfering our income and assets to a new global alliance (Family for Every Child), in order to order to increase the long-term impact on the lives of children. More background to this here: http://www.theguardian.com/global-development-professionals-network/2014/feb/07/power-international-ngos-southern-partners

We decided we wanted to evaluate the impact of these exits by looking at the situation over time. The report from the first Phase of the evaluation which was carried out last year by INTRAC is available here http://www.everychild.org.uk/intrac-responsible-exit-report

Phase 2 of the research is due to take place in 2016, to see how things are going several years later.

We debated how long to leave it before doing the follow-up evaluation, and decided that attribution would be become difficult after a couple of years, that ethics came into play because there would be minimal ability to respond to negative findings if we discovered that things had gone really badly later down the line, and we needed to factor in the deadline of asset handover to a new organisation. Therefore it made sense to complete the evaluation in 2016 rather than leaving the 2nd part of the evaluation any later than this. However, it would be absolutely fascinating to return to see how things are going 10 years down the line…

I am writing about refugee education projects and their effect in the home country after the return (usually poor), schools founded in emergency which have survived well many years later (surprisingly common).

Emergency interventions which were basically fulfilling the organisation’s agenda not that of the parents and children (many peace programmes that ‘helped’ the victims but never touched the perpetrators); unnecessary psychosocial interventions when the parents said they just wanted to get back to normal, do the official exams and let the children get on with their lives.

The key in education is that it is always longer than any NGO or UN intervention – we can find the children years later. I will publish a short paper on this shortly.

At CDA, we advocate for use of systems thinking in evaluations of peacebuilding programs along with examining both ‘collective impact’ and ‘cumulative impact’ of socio-political and development efforts by multiple local and international actors. Later this year, we will release a report on cumulative impacts of peacebuilding based on analysis of 16 case studies (as wide ranging as N. Ireland, Solomon Islands and Guatemala). Similarly, we took the cumulative impact angle while examining aid recipient’s perspectives on international assistance during the Listening Project exercises in 20 countries (some had seen 5+ decades of international assistance.) These learning processes were by all means purely qualitative, focused on joint learning, analysis and reflection and not evaluative.

In 2012, Oxfam invited CDA to do a longitudinal listening exercise in Tamil Nadu, with a focus on “How did Oxfam assistance make a difference in people’s lives during and after the Asian tsunami of 2004?” I was accompanied by staff from the Secretariat and Oxfam India. Indeed, we engaged in retroactive inquiry about what was indeed “done” and “how things were decided” 8 years earlier by 6 different Oxfam affiliates who responded in Tamil Nadu. A fascinating listening exercise and insights into long-term effects of Oxfam’s decision to register newly rebuilt houses to women, or to distribute boats to non-fishing castes. A complicated story of long-term impacts– social (i.e. gender roles, women empowered through property titling which was reinforced by changes in Indian govt policies), intercommunal (i.e. effects on caste relations due to changes in labor equation after too many boats were distributed), environmental (overfishing), etc. More here: https://www.oxfam.org/en/research/listening-exercise-report-tamil-nadu-southern-india Again, not a research study but I would have wished someone could explore these topics further and deeper!

On attribution analysis within longitudinal evaluation… use caution? There is an important distinction between examining the cause of a known effect versus looking at the effect of a known cause (e.g. change in govt policy on registering property to both men and women, or to women only). In this case, the cause is the policy, and we can study the effects. Oxfam’s contribution was important on that front – people spoke about it 8 years later – about Oxfam modeling and advocating for this. But the “sociopolitical change” river is wide and many other streams flow into it! Tracing from “women perceive a shift in household power dynamics” (effect) to identify the “cause” of this and attribute “credit”, is not just harder but perhaps also futile…

a) Our programmes are not the only element acting on the field. External forces (global/national economy winds, natural disasters, political and demographic changes…) could be stronger. When measuring, we may find noise is bigger than signal.

b) Aid Agencies and the different kind of donors may have a limited interest in long-term evaluation. NGO should warn donors: “Thank you for your support. Please kindly wait for ten years to be told about your donation’s effect”. What would happen?

c) Even companies are not very eager to long-term anything. They plan and operate for the next quarter or for the current fiscal year. This is especially true for bonuses (not only for the top management). Optimizing the current period of reporting can very well kill you for the next period, but this is widely ignored by business people.

d) Did anyone mention Climate Change?

Long-term thinking is very good. And someone must be the first one to do it, I guess…