What kind of evidence might persuade people to change their minds on refugees?

November 8, 2017

Is Recognition the missing piece of politics? A conversation with Francis Fukuyama

November 8, 2017

Should the Gates Foundation Do Data Differently?

November 8, 2017
empty image
empty image

Spent a fascinating day last week talking to staff at the Gates Foundation at its HQ in a cold, grey and sleety Seattle (felt quite at bmgflhome). I presented the book in one of those ‘brownbag lunches’ that Americans love (although these days ‘clear plastic box lunches’ would be more accurate), and we then got on to discussing the implications for aid agencies in general and foundations in particular (here’s the slides from my presentation – foundation-relevant stuff at the end).

One big issue for them is data – how can they get the data that they and their partners collect to have more of an impact on policy discussions? I drew a comparison with other areas like governance, where people have realized that the kind of focus on supply side that we now see on data has failed to deliver the anticipated results – all those ‘capacity building’ workshops on good governance don’t amount to a revolution, apparently.

The governance people then moved on to work more on the demand side – let’s strengthen civil society to demand better governance (cue more workshops). More recently, they have gone into ‘convening and brokering’ mode (bringing together different, even mutually hostile actors to look for common solutions), along with ‘working with the grain’ of existing institutions, rather than just trying to implant alien institutions from elsewhere.

What might be the equivalent process on data, which in some ways is really just another institution?

Big DataOn the demand side, we know that decision makers are influenced by ‘critical junctures’ – those moments when windows of opportunity open up, often linked to shocks and failures, and those in charge suddenly become open to new ideas to sort out the mess. But the supply side of data is remorselessly steady state and not geared up to respond to such moments. So how about setting up a rapid reaction unit that spots such opportunities, then mobilizes the data from Gates and elsewhere and gets it into the hands of the right people?

And what’s the equivalent of ‘working with the grain’ when it comes to data – are there different kinds of locally data that are more deeply embedded in different societies than the standard universal metrics used by the international aid system? Would using that kind of data make it easier to influence policies and (maybe even more) social norms?

In passing, I also think the Foundation is slightly deluding itself when it swears allegiance to the unadulterated elixir of ‘evidence based policy making’, where evidence alone determines what governments do, and the nature of that evidence is heavily skewed towards statistical data (rather than, say, experience and judgement). Why? Because the Foundation’s own success illustrates that influence is often more about the messenger than the message – the fact that it is Bill and Melinda lobbying governments often matters as much or more than whether their message is data driven. So perhaps the Foundation should be more explicit about the importance of champions (and expand them beyond the Gateses – BMGF fellows dotted around the world? Evidence Elders?)

As for the equivalent of ‘convening and brokering’, how about hosting data surgeries, where the Foundation uses its convening power to persuade decision makers to present their thorniest problems, and a team of evidence geeks then help them identify what data could be of use in addressing them, and process it into a useable form. Is that already happening somewhere?

A couple of other ideas: select a dozen of the Foundation’s greatest hits – the iconic stories of success that every Too-much-data-300x150institution tells itself – and go back and do a rigorous ‘how change happens’ study on each. That would reconstruct a timeline, identify the accidents and blind alleys, champions, blockers etc and come to a more nuanced view of the role of data in influencing policy, compared to other factors like messengers, chance and so on. See previous rant about the risks of airbrushing out all the messiness and politics in the stories we tell ourselves and others about social change.

One other exchange at the Brownbag was interesting – I was asked what to suggest when there was ‘no neat conceptual model’ for the kind of systems-driven, adaptive management approach I was advocating. My response: ‘well, actually there is a very good model – entrepreneurs’. Can’t believe I ended up telling the Gates Foundation to learn from entrepreneurs! What is interesting is that the way they behave often bears little relation to the kinds of procedures even the Gates Foundation demands: I suggested they ask Bill to submit Microsoft’s strategy circa 1980 to see if it would have qualified for a Gates grant…..

Over to any of the Gates people who were present to set me straight or add their bit.

 

5 comments

  1. “As for the equivalent of ‘convening and brokering’, how about hosting data surgeries, where the Foundation uses its convening power to persuade decision makers to present their thorniest problems, and a team of evidence geeks then help them identify what data could be of use in addressing them, and process it into a useable form. Is that already happening somewhere?”

    Hi Duncan,

    It is! For the best part of the last two years a number of organisations, including InterAction and Publish What You Fund, have been working on Gates-funded a project called the Initiative for Open Agricultural Funding. Briefly, the project is about improving the availability of quality open aid data on where major donors are disbursing funds, how and to what effect.

    As part of this, we’ve been working with a number of donors to identify their data publication problems and subsequently worked to identify solutions, either through direct technical advice or by producing open source tools to help donors extract structured data from already existing project documents — therefore creating better data without forcing a change in how policy staff and in-country implementers report. Lastly, we’ve been working on a data visualisation portal to help better display aid data in a way users can usefully understand. If you’d like more details, feel free to drop me a line or check out: https://www.interaction.org/project/open-ag-funding/overview

  2. Fascinating stuff, Duncan. I’ve been managing a Gates Foundation-funded project on data related to agricultural development over the past several years. A few points…

    The journey from “data” to “evidence” is long, fraught and complex. It also varies significantly across sectors and contexts. Standards of evidence in public health programs (i.e. randomize or go home) are relatively well-established. These evidence standards were developed and refined in medical research trials, but as Robert Chambers has noted, people’s bodies are fairly uniform and standard compared to the complexities of human behavior in societies, or the ecologies into which agricultural interventions fit. This idea that we can “solve” the problem of causal inference for a particular intervention through randomized trials, and then merely monitor the uptake of that “effective” intervention, is, I think, a reasonably fair explanation for the obsession with metrics at the Foundation. The story goes: 1) monitor aggregate progress towards the long-term goals; 2) monitor to make sure that these “proven” interventions are actually taking place; 3) feel confident that we’re all doing a great job.

    An extraordinary example – of where this model for how development works meets with a more complex set of problems – is the recent “Stories behind the data report” put out by the Gates Foundation (link here: http://www.globalgoals.org/goalkeepers/datareport/assets/downloads/Stories_behind_the_data_2017.pdf and hat-tip to your blog, I think). Big picture encouraging trends for the data on child and maternal mortality, HIV deaths, etc are presented, whereas there is a page towards the end simply titled “INSUFFICIENT DATA” for the education, gender and agriculture metrics, with an empty graph put up to ram home the point. Granted, it would seem that these SDG indicators were chosen without any real thought about how they would be monitored and by which institutions. But there’s also a bigger problem than lack of data. Take their chosen agriculture SDG indicator: “Volume of production per labor unit by classes of farming / pastoral / forestry enterprise size” Really? I could make a case for why that would be a useful metric to track, but it is far from certain that it would be unambiguously a good thing for that number to go up.

    To give the Gates Foundation its due, in our case (www.cgiar.org) they have played a key role in pioneering new methodology for addressing bad data. They convened expertise on DNA fingerprinting of samples from farmers’ fields across the world, to understand better how farmers manage and use crop genetic resources, and then have provided funding for some work to scale it up. Previously, we had to rely on “expert opinion” estimates for these data, or highly biased self-reported data from farmers (skilled farmers know the varieties, but many don’t for a whole host of reasons and we end up with skewed picture). These are the data we need to understand whether crop varieties that are more resistant to drought or disease are actually getting out there and being taken up by farmers. This stuff is expensive to do it properly, requires a lot of patience, expertise and coalition-building, and yet is totally worth it. That said, we are still miles away from being able to make comparisons over time for different countries – i.e. the kind of thing that the Foundation ultimately wants to use it for.

  3. Having worked on a few Gates grants my suggestion would be that the foundation rethink reporting requirements. The grant reporting (which may have changed in the 2 years since I last saw it) requires heavy inputs on making tenuous assumptions about how organisations have ‘achieved’ outcomes that are simply out of their control.

    Getting the kind of data they are interested in would require reporting to be more of a learning discussion about context and politics.

  4. Your point that “influence is often more about the messenger than the message” is my big take away from this. I’ve been building a database of youth serving organizations in the Chicago region for over 20 years and have shared that on-line since 1998, with the goal that more volunteers and donors seek out programs and offer consistent, on-going support to help each become great at how it helps youth. In this blog article I include a graphic, with a picture of President Obama, showing how people with high visibility could have a huge impact in getting attention and drawing potential volunteers and donors to this information. http://tutormentor.blogspot.com/2016/08/mapping-activities-of-intentional.html

    If Gates Foundation has maps showing all of the areas of the world where it’s working to help people, and includes a list of NGOs and social entrepreneurs working in those areas, and points to this regularly in its publications, speeches, etc., I would expect more people to visit the web sites of those organizations and look for ways to offer help.

  5. Great to hear that the Gates Foundation is open to thinking (somewhat) differently about the role that data can play in addressing development challenges.

    Your emphasis on the “demand-side” for data is helpful; definitely a step forward for organizations that have focused more on supply. And the “data surgeries” idea could be an interesting way of starting with problems that people care about, and then thinking about how data can help people to address those problems. Starting with problems faced by local stakeholders is a much better approach than starting with the data. And/but this might require funders to take more of a back-seat, letting others determine which problems take priority.

    Here at Global Integrity, we have a long history of generating data about Governance, but – having reflected on the extent to which that data gets used [hint: not a lot] – are increasingly focused on supporting locally-led innovation, learning and adaptation to address specific problems in complex political systems, with data an important input into those politically-engaged learning processes. Here’s our latest 2-pager on what we do and why we do it. http://www.globalintegrity.org/wp-content/uploads/2015/04/GI_2017.10_V4.pdf

    Are the Gates Foundation up for thinking about the role that they play, and might play, in shifting the dynamics of systems that are not only complex but also inherently political? Are they comfortable with the idea that development – and the use of data – is political? And if so, how is that shaping the ways in which they work? Maybe we’ll follow up directly and share what we find!

Leave a comment

Translate »