Links I Liked

December 15, 2015

Is Paris more like Kyoto or Montreal?

December 15, 2015

How on earth can you measure resilience? A wonk Q&A

December 15, 2015
empty image
empty image

Jonathan LainResilience is one of today’s omnipresent development fuzzwords, applied to individuals, Rob Fullercommunities, businesses, countries, ideas and just about everything else. But how can it best be measured? To plug their new paper on the topic, Oxfam’s measurement wonks Jonathan Lain (left) and Rob Fuller (right) argue with their imaginary non-wonk friend……

So they’ve let the beancounters loose on resilience now. Do we really have to measure everything?

Well, look at it this way: Oxfam projects are focusing more and more on helping people to build their resilience to shocks, stresses and uncertainty. If we want to evaluate the success of these projects and learn from them, then we need to know what outcomes we’re looking for. This means we need a way to think about and to measure resilience.

resilienceBut can’t you see how resilient people are simply by looking at how well they’ve been able to deal with crises?

Sometimes, yes. Ideally, if we wanted to see whether Oxfam projects have succeeded in building resilience, we would wait and see what happens after people have experienced some actual crises. But often we are trying to evaluate projects that are still ongoing, or that have recently ended. In some cases a crisis may have happened since the end of the project and we can ask people about how they coped: whether they had enough food, whether they lost any livestock or other assets, and so on. But in most cases there has (thankfully) not been a major crisis since the project was carried out. So we have to figure out whether the project that is being implemented now or in the recent past will help people to deal with crises and stresses in the future.

That sounds tricky.

Sure is. The best way we’ve found to deal with this is to identify some characteristics that we think will enable people to cope better with shocks in the future, and then look to see whether households or communities in the project area have those characteristics. Owning assets or having savings are obvious examples. But we also try to think more widely than that: Can people count on support from their neighbours when they need it? Are people aware of climate change and its effects? Do people participate in community groups? Considering these issues has proved useful in many of our previous evaluations of resilience-building projects.

How do you know what the right indicators are?

In a nutshell, we talk to a lot of people who know the local context well, and ask them what characteristics they think resilience blobsa resilient household has. Usually this means a combination of doing focus groups with local people, spending lots of time discussing with local Oxfam and partner organisation staff, and reading whatever reports or studies already exist on resilience in the area. This guides us on what questions we should include in our surveys in order to measure resilience effectively.

For example, when we were working in a peri-urban area in Bolivia, many people said that resilience depended mainly on whether or not they had a salaried job. Of course in other places, regular employment is much less of a factor: for example, in Somaliland, people were mainly concerned about having basic access to water, grazing land, and veterinary care – all factors that help to sustain pastoralist and agro-pastoralist livelihoods.

But hold on! If you are coming up with different sets of indicators in each place, you can’t compare the results between countries.

Well, yes and no. We are not trying to create a universal standard measure of resilience. That would be much harder (some might say impossible). So, we cannot say whether people in peri-urban Bolivia are more or less resilient, in general, than people in rural Somaliland. However, what we can say is whether our projects have had more or less impact in each country where we do a survey. When we compare across countries, we effectively normalise or adjust our results according to the average levels of the specific indicators that we used in that context. We can, in fact, compare the percentage change in resilience that our projects brought about in peri-urban Bolivia and rural Somaliland. So even if we aren’t aiming for a universal standard measure, we can at least pull together the results from our evaluations of resilience-building projects.

concept-resilienceThis approach is all very well for indicators about how people can cope with crises in the short term. But isn’t the idea of resilience bigger than that?

Agreed. The examples we mentioned before could be indicators of people’s ability to deal with crises when they happen. But just as important is whether people can adapt proactively, so they are less vulnerable in the future. That might depend on factors like access to credit, but also on personal, psychological characteristics, such as whether people feel able to take a risk by changing their livelihood or making an investment. It’s harder to find indicators of these more subjective characteristics, but we’ve been experimenting with some different approaches.

Then there’s the issue that resilience is also about transforming the systems that people live in – the village, the region, the country, or even the world… If our projects can change the way these systems work – for example, by advocating for policy change or protecting the environment – we can limit the stream of shocks and stresses that people face. In many of our previous evaluations, we have tried to measure resilience simply by collecting large-N data through household surveys. But if resilience is also about stuff that happens at the level of the system – the village, the region, and so on – then it also makes sense to collect small-N data at the system-level too. This is exactly what we have been trying to do. For an example of the kind of rigorous qualitative techniques we want to incorporate into framework for measuring resilience, see this previous blog post on ‘Process Tracing’ by Claire Hutchings.

This is all work in progress, and we are constantly feeling our way and trying to improve our methods for thinking about and measuring resilience. If you would like more detail, please read our new paper. And, if you’d like a deeper dive into Oxfam’s resilience Effectiveness Review evaluations, including the newly published set, they can be accessed here.

7 comments

  1. I’m sure it is very complicated to determine the outcomes of work on resilience – when I talk to supporters and donors though I sometimes just use the empirical example of the cyclone that hit Orissa in 1999 (where there was limited preparation or preventive action by the state, communities were under prepared, 10,000 people died and there was limited community or state to deal with the aftermath) with the 2013 Orissa Cyclone (lots of preventive action in the days leading up to the cyclone hitting, fewer than 50 people and lots of community and state capacity to deal with the aftermath and rebuild livelihoods etc).

  2. Have not yet reviewed the paper, but this post takes a practical approach to assessing resilience. Agree fully that indicators to assess resilience will vary significantly from one context to another. The process outlined above seems well within the capacity of staff of local organizations to apply. Although resilience is complex, we need to ensure that the ways of assessing it don’t require highly specialized methods and researchers. But I have two major concerns about what is outlined above. The first is about targeting (i.e.equity). The post refers often to “people”. The reality is that in any region, in any community, in any household, some “people” are much more vulnerable to context related shocks and stresses than others. This applies particularly to gender relations. In an already vulnerable household, women are almost always more vulnerable than men in that household. Resilience as a concept is supposed to be a way to bridge the divide between humanitarian and development action, particularly in chronically vulnerable areas. This means many things, but one key thing is that “development” actors have to do a far better job is adapting some of the tools used by humanitarian actors to identify those who are most at risk, most vulnerable, and provide specialized types of support to them, while also working with the wider community. Particularly in “chronically vulnerable areas” like the Sahel and the drylands of Africa, (where I work), we need to “focus like a laser” on the chronically vulnerable households that, good year or bad, slow or rapid onset shocks, are caught in a hunger/debt trap. The second major issue concerns the highly misleading narrative embedded in Figure 1. The implicit narrative is that the development process is progressively trending upward over time, and is set back only by stresses and shocks. Again, from my experience in the Sahel and the Africa drylands, this is false. The current development model is generating high, even robust rates of economic growth in the Sahel, averaging over 5% /annum since 2000. At the same time, chronic food and nutrition security has increased, not decreased. OCHA is calling for 1.8 billion dollars of humanitarian assistance for the Sahel. Since 2012, the number of destitute people who need humanitarian assistance has never fallen below 20 million. In short, the “development model” itself, is deeply flawed, generates inequality and marginalization, particularly for small scale farmers (men and women), is failing to address adaptation to climate change, or soil degradation. No point in trying to overcome stresses and shocks to get on the development line again, if the development model itself is “business as usual” generating reduced resilience, higher levels of chronic vulnerability and inequality (even within communities as the household economy assessments HEA) show. Well, I will now read the paper, but wanted to share these initial observations for the many who may only read this post, and not get to paper itself. Of course, I am biased by my experiences in the context of the Sahel, but would be surprised if similar concerns were not relevant in Asia or Latin America.

  3. Peter – Those are great observations, thank you. It’s certainly true that resilience can be different for different people in the same household. We talked about “people” in this blog post partly as a way to avoid referring to “households”, with the idea that a household is a homogeneous unit. As you’ll see in the paper, we are still some way from adequately capturing gender differences and other within-household inequalities in resilience in our surveys (partly due to the constraint that we can normally survey only one person in each household), but we are actively looking for ways to do this better.

    We share your concern about Figure 1, which in fact is not a diagram we have used ourselves. Apart from whether it is showing the right development pathway, it also doesn’t reflect that building transformative capacity should change our idea of what counts as a “shock”.

    Barney – The Orissa example is a great illustration of the idea of resilience, thanks! And it reinforces the point that there’s no such thing as a “natural” disaster.

  4. Gentlemen,

    Happy New Year! I am pleased to see that there are many who have not given up on the concept of resilience as just another fad or too nebulous for any real, meaningful impact on poverty reduction. Resilience gives name and face to ideas that we’ve all been thinking about and working on for quite some time. I appreciate the continued conversation on how we measure it and completely agree that its a noble and daunting endeavor. The more I study, as I’m sure you’ve found as well, the more complicated (and sometimes convoluted) it all seems. Particularly, the thought that gives me most cause for concern is that when a community is faced with a shock or stress, the social, economic, and even sometimes environmental contexts shift/transform/morph under the new pressures of the shock or stress. This, often fundamental, rearrangement of these systems sometimes bares very little functional resemblance to their pre-shock state. So, when we discuss measuring resilience pre-shock or pre-stress, we are examining and measuring systems that may look and function one way now, but will likely not function the same post-shock or even during a prolonged stress (drought, famine, conflict, etc.). As you can imagine, my fear is that we’re investing a lot of time, energy and resources in measuring something that may fundamentally reorganize itself when it encounters a shock or stress. How do we really know that the capitals (human, social, financial, etc.) will be available and/or utilized the same way to recover from the shock or stress?

    Am I missing something? Am I overstating it? Have you seen any research or anything else written on this topic? It is, quite frankly, something that discourages me from investing too much into measurement.

    Thoughts?

  5. Agree with Peter: too many “people” and not enough men and women. See FINDING WAYS TOGETHER TO BUILD RESILIENCE
    The Vulnerability and Risk Assessment Methodology by
    DANIEL MORCHAIN FRANCES KELSEY
    Oxfam GB

Leave a comment

Translate »