Earlier this week I spent an afternoon at the Gates Foundation in London, discussing what systems diagnostics can offer to groups like the World Bank, DFID and RISE, a big research programme on improving the quality of education worldwide.
We kicked off with an education systems guru, Luis Crouch, describing the 3 things he thinks characterize a system:
- The objects/actors in it – wolves, deer and tree saplings
- the feedback loops between them – wolves eat deer; deer eat saplings.
- The emergent properties that the system produces. Not intentional on part of any one actor.
In his view, most diagnostics of education performance spend a lot of time enumerating and assessing the individual actors (teachers, principals, training institutions etc), but don’t think much about the feedback loops between them. Also, they mainly look at purposive properties – did the programme achieve the intended outcome – not the emergent properties that no-one intended.
We then moved on to Jaime Saavedra, who now leads the Education Global Practice at the World Bank, but from 2013-16 was Peru’s Education Minister. His reflections on his reform efforts there were fascinating.
Firstly (in passing) a ringing endorsement for league tables. He talked of the ‘PISA shock’ from Peru’s poor performance in the global PISA education ranking. ‘‘PISA influences countries: I don’t want to look bad’.
But what really gets him fired up is data. ‘We had a huge obsession with data, throwing information into the system. There were dashboards for absolutely everything. We tracked the life of the textbook from printshop to student. Ditto with all the inputs.’
On the basis of data analysis and brainstorming, the Ministry identified poor quality teaching, pedagogy and (lack of) school leadership as the main things they needed to tackle.
Lant Pritchett chipped in (very excited that he’s going to be in the UK for a while, after leaving Harvard) and said education reformers need to come up with something like the Growth Diagnostics that he developed with Dani Rodrik, Ricardo Hausman and Andrés Velasco. Education reform is now where growth promotion was, circa 2005: ‘the problem was that every time the World Bank did a country report, it would come up with a checklist of 60 things to fix, and no country could deal with all of them. We need a structured way to identify the binding constraints to learning, and tackle those first’.
Ping! On goes a lightbulb. These two are talking about a complicated system, not a complex one. The distinction is crucial. Complicated is like sending a rocket to the moon – a difficult problem, but one that can be broken down into its component parts, ‘solved’ with data and smarts, and reassembled into a successful solution. That’s what Jaime and Lant were describing.
In contrast, a complex problem is more like raising a child – it’s all about antennae, judgement, guesswork, collaboration, trying stuff out and then realizing quickly when it’s not working. Data is useful, but not as central. ‘Lessons’ from raising one child are likely not to transferable to the next. And if you break a complex system down into its component parts (please don’t try this with your child) you won’t get much insight into how all the different feedback loops produce the ‘emergent properties’ of the whole.
The act of being in charge, of trying to get stuff done, drags you from the complex into the complicated quadrant. The data offers you a handle, you need to set priorities, and before you know it, you are doing the Growth Diagnostics thing and breaking up the system into its parts. That’s probably inevitable, and not such a bad way to proceed compared to, for example, just making stuff up. But it loses touch with those elements that are complex, and likely to mess up your complicated plan.
So what countervailing forces can push decision makers to keep at least one foot in the complex camp? The two obvious ones are politics and MEL: politics – what Jaime knows will fly and won’t – is all about judgement and spotting emergent patterns and opportunities; Monitoring, Evaluation and Learning, done in real time, will tell you when your attempt to disaggregate has backfired, and unintended consequences are springing up like mushrooms in the night.
Cue second light bulb. One of the things I have been failing to do is distinguish clearly between system, problem and solution. On the same topic, they may be in different quadrants – eg health, whether individual or societal, is definitely a complex system, but how to establish healthcare clinics in every corner of the county can be a complicated problem, while vaccinations may be a simple solution.