The link between Income Inequality and Public Services is stronger than I realized (thanks to Emma Seery for putting me straight)

April 4, 2014

Now that’s what I call transformation: Latin America then and now, and Tony Benn RIP

April 4, 2014

‘How DFID Learns’. Or doesn’t. UK aid watchdog gives it a ‘poor’ (but the rest of us would probably do worse)

April 4, 2014
empty image
empty image

The UK Department for International Development’s independent watchdog, the  Independent Commission for Aid Impact (ICAI), has a report out today on ‘how DFID learns’. Or doesn’t. Because the report is critical and gives DFID an overall ‘amber-red’ assessment, defined as ‘programme performs ICAI logorelatively poorly overall against ICAI’s criteria for effectiveness and value for money. Significant improvements should be made’.

I’m not gloating here – in my brief time working at DFID, I was struck by its investment in staff training and the general level of curiosity and intellectual enquiry. I suspect ICAI would be even more critical of NGOs and other aid organizations, so anyone working in research and development should probably spend a few minutes skimming the report.

Here’s the overall assessment:

‘DFID has allocated at least £1.2 billion for research, evaluation and personnel development (2011-15) [see graph]. It generates considerable volumes of information, much of which, such as funded research, is publicly available. DFID itself is less good at using it and building on experience so as to turn learning into action. DFID does not clearly identify how its investment in learning links to its performance and delivering better impact. DFID has the potential to be excellent at organisational learning if its best practices become common. DFID staff learn well as individuals. They are highly motivated and DFID provides opportunities and resources for them to learn. DFID is not yet, however, managing all the DFID research spendelements that contribute to how it learns as a single, integrated system. DFID does not review the costs, benefits and impact of learning. Insufficient priority is placed on learning during implementation. The emphasis on results can lead to a bias to the positive. Learning from both success and failure should be systematically encouraged.’

Recognize any of that? Thought so. And here are the report’s recommendations, some of which probably also sound pretty familiar:

1: DFID needs to focus on consistent and continuous organisational learning based on the experience of DFID, its partners and contractors and the measurement of its impact, in particular during the implementation phase of its activities.

2: All DFID managers should be held accountable for conducting continuous reviews from which lessons are drawn about what works and where impact is actually being achieved for intended beneficiaries.

3: All information commissioned and collected (such as annual reviews and evaluations) should be synthesised so that the relevant lessons are accessible and readily useable across the organisation. The focus must be on practical and easy-to-use information. Know-how should be valued as much as knowledge.

4: Staff need to be given more time to acquire experience in the field and share lessons about what works and does not work on the ground.

5: DFID needs to continue to encourage a culture of free and full communication about what does and does not work. Staff should be encouraged always to base their decisions on evidence, without any bias to the positive.’

Some other interesting extracts from the main 40 page report:

Staff turnover: ‘Staff are continuously leaving and joining DFID (sometimes referred to as ‘churn’). Fragile states are particularly vulnerable to high Dilbert on quantification of research impactstaff turnover by UK-based staff. For instance, in Afghanistan, DFID informed us that staff turnover is at a rate of 50% per year. We are aware of one project in the Democratic Republic of Congo having had five managers in five years.

This process represents both a constant gain and loss of knowledge to DFID. When staff depart DFID, their knowledge should be retained in the organisation. Similarly, when staff join DFID, their prior knowledge should be made available to others.’

Learning by failing: ‘During 2013, DFID began to discuss failure in a more open and constructive way than it had previously done. This began substantially with the February blog of the Director General for Country Programmes. Following this, a short video was produced by DFID staff in the Democratic Republic of Congo that discussed failures in a water supply improvement project. This internal video has been catalytic in stimulating discussion about how DFID should be more honest about failure. It has resulted in the introduction of ideas, such as the need to fail fast. During 2013, DFID’s Research and Evidence Division has piloted approaches to discussing failure in ‘fail faires’, where staff come together to identify what can be improved. It is too early to say whether these will support a change of culture in DFID in its attitude to learning from failure, albeit they appear to be a positive innovation.’

(not) Listening to staff and partners: ‘Junior and (even senior) locally employed DFID staff generally ‘only give our opinion if asked’. Generalist, administrative and locally employed staff are not being listened to sufficiently by DFID’s specialists. They often have much experience of how aid is delivered: know-how….. DFID staff do not appear to prioritise how they listen to others. This applies to learning internally and from external sources…. Staff believe that DFID remains too much in a mode of trying to manage or change others rather than listen to and support them.’

All good stuff, but what is lacking is any discussion of the institutional constraints to DFID being able to implement these recommendations – after all, there must be reasons why they haven’t done so already – see Neil McCulloch’s recent piece on the political economy of donors.

Finally, props to the UK government for setting up ICAI in the first place. Really impressive example of rigour, transparency and accountability (more on that topic here next week, when some T&A gurus discuss its limitations).

9 comments

  1. Interesting!
    But in my experience all institutions suffer from this problem – as a researcher I’m appalled by how little notice or action is taken of the recommendations of work I have been involved in. Or indeed any feedback about it, or follow up questions.

    I’m just wondering what would be a good example to bench-mark from, a quick scan of the report does not enlighten.

    As ICAI are so smart, they must have their own mechanisms for doing this? What are they? Is someone studying ICAI?

  2. Thanks a bunch for sharing.
    Re your second last paragraph. It really is key, isn’t it. You could say it’s time for a further step back and ask first why it isn’t brought up in the report ” – after all, there must be reasons why they haven’t done so already”.

  3. I have been amazed by how much DFID has invested in learning looking at the large number of think-tanks that have acknowledged its support over the past decade. Perhaps it is time the think-tanks returned the favour and helped DFID to translate the information into learning. The idea that the information that DFID has helped to produce is valuable as a ‘public good’ is clearly insufficient as a rationale to spend the billions of pounds on research.
    To be fair to DFID, it is not alone and probably should rank better given its efforts to support rigorous learning. In my view, the ICAI report challenges how development agencies learn from development work. Right now, the emphasis on experimental learning is overshadowing the learning that happens during the implementation process. My experience as a development practitioner tells me that in reality, organizational learning is an uphill political battle. The ‘top-down’ nature of development agency operations often dictates that the ‘latest and greatest’ from home offices prevails over local learning. For good reason, Roslyn Eyben (2013: 3 – Uncovering the Politics of ‘Evidence’ and ‘Results’) “encourage[s] development practitioners to strategize in expanding the politico-bureaucratic space to make room for flexible and creative support of locally-generated and transformative change.” It is not easy, however, because when policy dictates a particular direction, it is difficult (not impossible) to change course locally. Moreover, the daily grind and requirements of agency work, leaves little room for reflection.
    Perhaps DIFD should commit some of its research funds towards learning from the implementation process. There is little interest in the systematic collection of information on the nuances of implementation (despite ‘black box’ concerns). The fact that each project, activity, or intervention is of necessity a socio-political experience (plus experiment) requires that organizations are learning constantly to achieve transformation. Each act to stimulate or cause social action is an odyssey into the unknown. Timing; changes in the cast of characters; the reality that most social projects are operationally large, take time and are complex (in equally complex settings); the influence of past and competing interventions; the self-agency of participants and non-participants; changes in policy environments; among many factors, challenge us to have open minds in our efforts to understand project impact. The results we seek offer us direction but not necessarily how to get there successfully. A compass will point us in the direction we want to go but will not tell us about the twists and turns or the adversities we might encounter as we traverse the terrain to our destination.
    The things I point out have been well articulated by advocates of realist evaluation (Pawson and Tiley 1997; Pawson 2004). What is baffling, at least for me, is why these voices are unheard as the push for experimental learning matches on. My sense is that we need exploratory approaches (John Gerring’s 2004 ‘What Is a Case Study and What Is It Good for?’ is helpful here) to continually challenge our work and how we learn.

  4. Really interesting post, Duncan. Thanks! I’m printing up the report for my train ride home reading. DLP is working on a methodology paper on action research for learning, coming out of experience of working in this way with DFAT (ex-AusAID) in the Pacific, that may be of interest when it’s done.

    In a recent government consultation on DFID’s funding of, and use of, research, I said that I think part of the issue is the 5 year RPC model. It has produced (and is producing) some absolutely fantastic, paradigm-shifting research, but it requires ongoing support and leadership within DFID, and that doesn’t always happen with staff turnover. Sometimes the world moves on in a 5 year period, and it could be that the research isn’t as relevant to practice as it may have been when it was first commissioned. Better support for the big ‘flagship’ RPCs throughout their life, with more flexibility for ‘responsive research’ within the overall budget package could also be helpful.

    1. Thanks Heather, but I’m even wondering whether with that kind of staff turnover, experiential learning of the kind Pablo Suarez has developed (see blog on serious games earlier this week) might be a better way for incoming staff to absorb wisdom

  5. The real reasons why DFID appears not to ‘learn’ are quite simply sometimes due to politics. DFID and the FCO often know precisely that they are probably doing the wrong thing, or trying to do some too quickly or in the wrong place (think Afghanistan), but it is politically expedient to do so. It may not be about the development project at all but something else that is being pursued. We are being naive if we are constantly looking for reasons why organisational learning is not happening. Maybe that is one of the reasons staff leave when they know that often they cannot ‘do the right thing?’ Who knows

  6. ‘Not listening to staff or partners’. Sadly DFID’s behaviour in this respect has deteriorated over the last decade. The institutional drivers make staff ‘perform’ rather than ‘learn’ for example in international discussions where accusations of arrogance have become increasingly common. Back home in a political environment judged hostile to aid when invited to participate in an outside event such as a seminar or meeting DFID staff arrive just in time for their slot and then rapidly depart, having delivered their message. All this not only stops DFID from learning about other perspectives but also has a negative impact on those DFID works with.

    I agree with you, Duncan, ICAI is (for me) unexpectedly good news. Not such a bunch of bean counters as I had initially feared. I wonder if they did this particular study because they themselves were becoming frustrated that DFID wasn’t even learning from ICAI reports?

  7. Thanks Duncan for your review of the ICAI Report on DFID and thanks also to earlier contributors for their comments on Duncan’s review. My view is a bit different – it urges Duncan, Contributors and Readers to consider shifting focus away from problems and passing blame and shift focus towards solutions.

    It is clear that without new thinking, new ideas and new ways of doing things; there cannot be new rules, new partnerships and new coalitions necessary to cure the disease underlining the problems thrown up by the ICAI Report on DFID, Duncan’s review and follow up comments. Let us start this way:-

    In the first 50 years of International Development Cooperation, 1960 – 2009; several Study Reports indicate the following Scorecard:-
    1. 1/3 Good Evaluations; 1/3 Flawed Evaluations and 1/3 Failed Evaluations.
    2. 1/3 Good Programs; 1/3 Flawed Programs and 1/3 Failed Programs.

    5 years into the second 50 years of International Development Cooperation, 2010 – 2059; Study Reports indicate the same Scorecard as in the first 50 years.

    If International Development Cooperation Goals and Targets such as FAO End Hunger in Africa by 2025 and World Leaders End Poverty in the World by 2030 are to be met and impressive Scorecard is to be obtained by 2060 NOW is the time for a TURNING POINT in International Development Cooperation, with Universal Evaluation – Evaluation Advocacy; Evaluation Innovation; Evaluation Promotion and Evaluation Protection as KEY Lever.

    IDEAS Global Assembly 2013 focused on Poverty. The outcome of this conference organized by the only World Association of Individual Development Evaluation Practitioners has had ZERO IMPACT on Country and Global M & E Systems. This Underline the point that, the Turning Point Time, in International Development Cooperation is NOW.

    In Nigeria, the World Bank and DFID started Joint Country Partnership Strategy, JCPS, in 2005. Fundamental Issues raised by at least One stakeholder in 2005 are still relevant today, such that Nigeria’s Investment Climate is worse in 2014 despite Investment Climate Reform Component of JCPS that started in 2005. Each JCPS up to the current JCPS 2014 – 2017 raise Country M & E issues, yet Society for Monitoring and Evaluation, SMEAN remain comatose and M & E Systems at Federal Government, 36 States Governments and FCT and 774 Local Governments remain very weak.

    We understand the current USAID Director seek to pass Aid through Communities because past effort at passing Aid through Government Agencies and CSOs’ have not worked. If Evaluation could not identify why these past efforts failed and recommend practical solutions, it is inevitable that the current effort will also fail.

    USAID and AfDB have joined WB and DFID as Partners in the JCPS. We know that AfDB has Aligned and Harmonized its Country Programming Framework, CFP with the JCPS. Assuming WB, DFID and USAID have also Aligned and Harmonized their respective CPF with the JCPS, as long as Major International Institutions – UNICEF, UNDP, FAO, IFAD, EC etc do not Align and Harmonize their respective CPF with the JCPS on one hand and the Budgets and Development Plans of Federal Government, 36 State Governments, FCT and 774Local Governments are not Aligned and Harmonized with the JCPS and respective CPFs’ of all International Institutions / Donor Agencies / International Foundations, the fundamental issues of Aid Effectiveness cannot be tackled resulting in Nigeria’s problems remaining or getting worse with consequences for National and International Stakeholders in Nigeria’s Economy.

    The recent death of 19 Youth at Nigeria Immigration Service Interview has generated ANGER across Nigeria which properly channelled to focus on solutions to Nigeria’s problems, can help End Hunger and Poverty in Nigeria, Africa and World. Please help us to ENSURE the 19 Youth do not die in VAIN through supporting Campaigns that help lay foundation and build edifice for New Institution Order fit for the 21st Century in Nigeria, Africa and World within a New End Hunger and Poverty, NEHAP Initiative.

    It is true that in every discipline Nigeria has World Class Professionals. Nigeria Professionals are helping to build other Nations – Developed and Developing. For example, should Nigeria Professionals be withdrawn today, the UK Health System will collapse. Inspite of this, it appears that Nigeria is yet to break the vicious circle of being too Rich to be Poor and too Poor to be Rich. Notwithstanding the differences between Consulting Services for Commercial Gain Approach and Consulting Services for Pro Bono and Commercial Gain Approach, the Task to break this vicious circle is one that Universal Evaluation – Evaluation Advocacy; Evaluation Innovation; Evaluation Promotion and Evaluation Protection needs to effectively and efficiently address within the NEHAP Initiative.

    The point we are making is that:-
    1. Universal Evaluation – Evaluation Advocacy, Evaluation Innovation, Evaluation Promotion and Evaluation Protection can help solve Unempolyment, Terrorism, Hunger and Poverty problems facing our world today, if there is genuine determination on the part of Program Commissioners and Evaluation Commissioners in International Institutions; Donor Agencies; International Foundations and Governments sides to help achieve impressive International Development Scorecard by 2060 through supporting NEHAP Initiative Activities in Nigeria, Africa and World. For copy of the Document on NEHAP Campaigns, please send email to nehap.initiative@yahoo.co.uk.
    2. The solutions to the problems highlighted in the ICAI Report on DFID; Duncan’s Review and follow up comments is probably best found within NEHAP Initiative Pilot Program in select Countries where DFID Operate and that should DFID support the Campaign to ensure 19 Immigration Interview Youth do not die in vain and Campaigns, Revolutions, Coalition and Platform in NEHAP Initiative in Nigeria, that include a Bridge Building between Lessons Learning and Lessons Forgetting Instrument.
    3. There is a need for ICAI to find a way to nudge DFID; without running foul of its mandate; to implement Pilot Programs such as that suggested in 2 above; if ICAI is genuinely interesting in practical solutions to real problems on the ground highlighted in its report on DFID:

    Should ICAI and DFID accept above points the UK will be helping to make Nigeria, Africa UK; Europe; US and Global Society Safe, Secure, Peaceful and Prosperous for all Citizens and Immigrants

    God Bless our World.

Leave a comment