How to Plan when you don’t know what is going to happen? Redesigning aid for complex systems

May 15, 2013

A crucial step in fighting inequality and discrimination: the law to make India’s private schools admit 25% marginalised kids

May 15, 2013

What do we know about the impact of savings groups on poor African women?

May 15, 2013
empty image
empty image

Savings for Change (SfC) is one of Oxfam America’s flagship programmes, reaching 680,000 members, mostly women, in 13 countries.sophie romana 2 Here Sophie Romana, Oxfam America’s Deputy Director of Community Finance, reports on some findings from an innovative qualitative and quantitative survey of the groups in Mali, published today (click through to summary or full report).

How do you save money and borrow when you live in rural sub-Saharan Africa?  Millions of women do just that every week, through their Savings Group.  Formed and monitored by teams of field agents from local organizations, 20 to 25 women gather every week at the same time and place to put a few cents in a wooden “savings box”. Once there is enough money in the box – i.e. the saving fund – members who need a small, short-term loan come in front of the self-managed group to explain the purpose of the loan (food purchases, life’s emergencies or working capital for an income generating activity).  The loans are paid back to the group with interest, which provides them with a return.  In a nutshell, savings groups provide basic financial services to poor rural women underserved or ignored by commercial banks and microfinance institutions.

But does belonging to a group actually improve the lives of members, their families, and their villages? To answer this, Oxfam America and Freedom from Hunger commissioned Innovations for Poverty Action (IPA) and the Bureau of Applied Research in Anthropology (BARA) at the University of Arizona to conduct a unique piece of joint research on Saving for Change groups in Mali: a randomized controlled trial (RCT) combined with a qualitative longitudinal study, funded by the Bill & Melinda Gates Foundation.  The RCT included 500 villages: in 210 of them we introduced SFC, the other 290 were “controlled” (intentionally left out of the intervention) to try and measure the difference, hence the impacts. The qualitative survey focused on 19 villages included in the RCT and interviewed members, husbands, women non-members, villagers etc… This mixed-methods approach combines the benefits of ‘quant’ and ‘qual’ to try and get under the skin of the impacts of savings groups.

Saving for Change fig 1The findings of the three-year study (see chart) show encouraging results in terms of increased saving (up 31%) and lending (12% more women took a loan from a savings group), increased food security, and an increased investment in livestock (households in SfC villages own on average $120 more in livestock, which buys you four goats, three ewes or one calf).  The findings also demonstrate that savings groups reach the poorest of the poor with 82% of households in study villages living on less than $1.25 a day.

The results from the RCT also show that there was almost no change in income and health and education expenses. We hope that these results will come with longer study, but we are not sure.

Social capital, one of the outcomes most valued by group members, is proving to be a puzzle. The group offers a safe space for women to share family problems and seek advice from each other. Outside the meeting, women have also reported over the years that they tend to greet each other more in the village, and engage with each other more often than before they joined.  But here’s our evidence puzzle: this is what the anthropological findings support, but they were not captured at all by the quantitative-RCT.

Take up rate: how do groups get created in zones where we don’t run the program?

Based on feedback from our partners and staff, Oxfam started to train “volunteer replicators” members who themselves train new groups. They have been responsible for SfC “going viral” In treatment zones the take up rate is 40.5% of women – by comparison in other similar approaches such as microcredit, the take up rate is 15% to 22.5%.

But the replicators have unexpectedly ‘spilled over’ into control villages, far away from a treatment village. This may mess up the control zones by “contaminating” the sample for the RCT, but it’s potentially good news for the women in those villages, and a testament to the attraction of savings schemes like SfC.

Depending on how strict a definition of a Saving for Change group we used (other traditional groups resemble SfC groups), we see a take up rate in control zones varying from 6% to 12% of women.  So how did that happen?  Did a conversation in the market lead to the replicator offering to go and create a new group there?  Did a member get married, move to another village and start a group there? Did a woman decide to help her daughter in another village to set up a group? Traveling to another village to form a group is challenging for many Malian women, yet SfC groups were created with no encouragement or promotion from the project, no visits from paid field agents.

We also found that women who are more socially integrated and already have an income generating activity are more likely to join earlier, but that more marginalized women do indeed join later on. When women want to save money together, they find a way to make it happen.

Are members of SFC more resilient?

Whatever your own personal definition of resilience may be, in the Sahel any sign of resilience is a success. The study took place in the Segou region ofSaving for Change logo Mali, where 40% of the households experienced a ‘shock’ last year (food price increase, drought, or illness) and 40% are food insecure (unable to produce or buy nutritious food). Households in SfC villages experienced an 8% increase in reported food security and were also eating more during the hungry season – spending 39¢ more per adult per week on food during this difficult time of year and eliminating the seasonal dip. In Mali 39¢ buys you a plate of nutritious beans or a few large cassava roots.  We also found that this impact is greatest for one of the most marginalized groups of women, those women married to younger brothers in large households.

From my point of view as a program manager, I see a value in combining an RCT with a qualitative study because I need to know if the program produces the impacts we designed it for and if it does not, what needs to be corrected.  However I do have a lot of questions around the findings, which I regularly debate with my Monitoring, Evaluating and Learning colleagues. That being said, would I run another RCT if a donor asked for (and funded!) one? Why not? Would I look for funding to run another RCT? Not necessarily – there are other less expensive tools to measure program impacts.  But for the time being, I’ll say with the confidence that only statistical evidence can give me: belonging to a savings group does make your life better!

Sophie Romana. with Janina Matuzeski and Clelia Anna Mannino. Today also sees an important Mali donor conference. Oxfam report here.

17 comments

    1. Apologies John, should have spotted this. First two columns are take up rates in control and treatment villages, and fourth column looks like overall (treatment and control villages combined), but no idea what ‘organic’ refers to. Will get Sophie to explain properly (when she wakes up in Washington!).

  1. Thanks for this Sophie. Could you expand on some of your final thoughts? Especially regarding the questions you have around the findings, what other tools you are using to measure impacts, and what changes you have made (or are considering) to implementation? The question about other ways of measuring impacts seems particularly important given that the evaluation covers a 3-yr period, but as the theory of change reasonably suggests (p128 in the full report for anyone wanting another diagram with lots of arrows!), many of the hoped-for impacts are only going to emerge longer-term.

  2. When the data is collected, our numerators cannot say they are there to collect data pertaining to SFC, so we had to use the following definition of groups
    Darker green: refers to a loose definition of a savings group: “accumulating savings groups that women report report as being “applause groups” because of the SFC clap”
    – Lighter green: stricter definition with the additional criteria that the group meets regularly and has received training outside the group (field agent or volunteer replicator)
    – Structured replication is when volunteer replicators receive formal training and a pictorial training guide to use in forming the group
    – Organic replication involved replicators who had received neither training nor formal guide

    I do hope this clairifies the chart!

  3. Hi there,

    I would be interested in knowing a lot more about the learning from these approaches. What did they tell you about why savings groups had positive effects in some cases and not in others? What were the barriers to inclusion if any? What kind of links did you draw between how the savings groups increased resilience?

    What different things did the RCT and the qualitative study tell you, and especially why do you think outcomes around empowerment weren’t captured by the RCT?

  4. Hi John, good question on the chart. We (ie. IPA) also randomly split our treatment group into two groups, and each received a different level of training for the volunteer replicating agents. In the “organic” group of treatment villages, paid field agents implemented our original replication approach in which the agents identified an promising woman member and trained her via informal, ad hoc conversations to create additional groups in her village. The “structured” replication is our new way of doing business and involved these informal conversations PLUS a three-day formal training for the volunteer replicator, a pictographic manual (literacy in Mali is very low for women) and a certificate. Thus the RCT was able to test if this new method, which costs $40 more per replicating agent, is worth it. The answer is a decided YES. The chart shows that the take-up rate in villages with “structured” training of the replicating agents was higher by about 8%. We don’t show it in this post, but there were also stronger impacts in villages with structured replication. This is the way that the Saving for Change program was headed – to use formal training and the other aspects of structured replication. But the RCT is a great confirmation that this approach is worth it. Feel free to post follow-up questions. (Apologies for any typos and if my colleague already replied.)

  5. Sophie and Janina, Its good to see that this study has now come out. It is the most in depth study to be implemented on savings groups to date. Both its methodology and findings are of importance and I look forward to reading the full report. But for now a couple of comments. These will appear to perhaps be pedantic but – given the issues around the politics of evidence – which Duncan’s blog has so much covered – it is important that claims are made with appropriate precision:
    – on poverty outreach, to say that it reaches the poorest of the poor is not necessarily justified by the figure that 82% of members are below the poverty line. This simply demonstrates that the programme is operating in a situation of absolute poverty on international metrics. The programme may in fact be excluding those who are the poorest of the poor in the local context (as some of the points about marginalisation in the rest of the blog imply). It would be necessary to look at poverty rates in the treated and control groups to establish whether as a methodology it necessarily reaches the relatively poorest of the poor whatever the context (see the chapter on Outreach in the book Savings Groups at the Frontier) and would help to know the poverty rate in Mali as a whole.
    – it seems to me the results on lack of impact on income and food security also present a conundrum and need some explanation. How does no increase in income relate to increased food security? [ presumably the value of own food produced has been imputed in the income data.]
    – as for your final conclusion it would be good to see some greater care in expressing findings of this type. There are two issues here:
    First the statement implies external validity – ie that no matter where you belong to a savings group it will make your life better – and you express this in a way that suggests that it is because of the statistics that you feel justified in saying this. Let’s be clear that RCTs do not give any greater validity externally than any other method of impact assessment. The conclusions must be located in time and place.
    Second, RCTs deliver average effects comparing the treated with the untreated samples. The distribution of impacts is very important. It is easily possible for improvements for a minority of participants to create an average effect which is statistically significant for the whole sample even when a majority have experienced little or no change.
    So (assuming it is a majority that have benefitted) and while no doubt it is a very unsexy way to write on a blog, the conclusion would be better phrased as “In the Segou region of Mali over the last three years we can say with some confidence that belonging to a savings group has improved food security for the majority of those participating”.

  6. HI Susan,

    Great comments and questions and it’s wonderful to have your input and perspectives. I’ll attempt to address the issues you raise.

    I agree that we need to find out if we are reaching the poorest of the poor within these poor-by-global-standards villages. This is addressed in the full report and in the 4-page summary – we didn’t get to cover it in the blog. From an outreach perspective, 42% of women from households in the top tercile (by food consumption per capita) join the groups but 33% of women in the lower tercile also join. So better off folks are more likely to join but we are still including the poorest. Likewise, the RCT measured social networks in 40 of the study villages. We learned that women who are more socially connected are more likely to join but more marginalized women still do join (page 41 in the full report). And women who join groups that form later on are more socially marginalized than those who join the program in the first six months that the program is in the village. So the program eventually reaches more socially marginalized women.

    The improvement in food security findings are strongest for women in low-status sub units of large extended families (defined as women married to men who are not also the extended family household head). The anthropology work in this research also illuminated that these women and their children can be the most vulnerable in the lean season so I’m particularly happy that we have an RCT finding that we are helping this population most.

    From other research, we have qualitative and anecdotal evidence that some of the most marginalized women do join (e.g. widows and women with very few livelihood options) but in some cases these women may struggle to keep up with the group obligations. So the jury is still out on how we are doing with, say, the bottom 5% of women in these villages. But we are definitely helping some very poor households and vulnerable folks. The IPA and BARA researchers may have additional insights. (Note to blog readers: I coordinated the Oxfam side of this research project.)

    The impact on food security but not on income is a puzzle. One answer is that research findings don’t always add up. For example, we also see improvements in livestock assets and some marginally statistically significant (10% level) increases in the size of businesses (but not the profits). Yet we see no statistically significant impact on income or total (food plus non-food) consumption. Another possible explanation is that savings groups help with consumption smoothing which has a big impact on food security. But savings groups may not have a bit impact on the total resources within a family or village, which would be reflected in income. We invite comments and suggestions on interpreting this set of findings – we are still digesting the implications.

    I agree about the external validity issue, but let’s have a moment to celebrate that we put the savings group program up to a pretty rigorous test (if not the only rigorous test out there) and we had some positive findings! This is not mentioned in the blog but there are also several other recent RCTs on savings groups supported by other INGOs and local NGO partners, in other countries. So, one way to help with external validity is to consider all these studies together. The SEEP network is commissioning a synthesis study which will be very useful in this regard, although nothing can completely solve the external validity issue no matter the approach.

    Average impact versus impacts on sub-populations is important. This RCT was large enough that it was possible to consider some heterogeneity for the main impact hypotheses. We looked at the data by poor versus better-off terciles (measured by food consumption), household structure (no extended family, high status sub unit within the extended family, low status sub unit within the extended family), and ethnic group. There were some differences but nothing earth-shattering, aside from the food security finding I mentioned just now. Are there other important heterogeneities that you would suggest we explore, given your experience?

  7. Many thanks to all for these great questions. To address the issue of social capital and empowerment – I think our social capital puzzle is a really interesting one. There could be a few reasons why RCT findings didn’t point to impact. Part of the reason may be because questions in the RCT measured whether women had formed new relationships, whereas anthropological findings pointed to strengthening of pre-existing relationships. So, it could be that forming new relationships is an impact of SfC over a longer period of time than the study captured. It could also be that the quantitative questions included in the RCT really didn’t capture those areas of social capital and empowerment that change as a result of membership in SfC. The baseline survey was designed early in the life of SfC and researchers made their best prediction of possible impact areas. We’re curious to hear others’ ideas/experiences on this topic as well.

  8. Good, now we can have a debate on RCTs!

    This is the first RCT I been involved with and to be fair, I was not with Oxfam at the design stage. But I think this one was a heavy piece of “machinery”. The spill over rate in the control zones show that it is very difficult to maintain “perfect” RCT conditions through the entire study. As Susan mentions, we can’t declare anything definitive without caveats, cautions, and being very specific. That frustrate the program manager that I am, because I would love to be able to say that indeed we can generalize the results and capitalize on them to keep on building groups!

    In the course of running Saving for Change, Oxfam America has undertaken 22 evaluations of the program, using every possible evaluation methodology available under the sun. This corpus of research and findings put together across geographies along with an increasing number of members makes me believe that these women indeed find that there lives are better if they belong to a group and that’s all the last sentence of my post meant, but I am personally sticking to it!

    As we say in French: ne soyons pas plus royalistes que le Roi!
    (let us not be more royal than the king)

    Thank you all for comments, ideas and suggestions!

  9. This is interesting, at Theatre for a change in Malawi, we are using the same model to for the sex workers groups sine sex work here is lagely because of poverty. Now it looks like this model is working on them, currently we have done no evaluation and this article gives me a guide on how to evaluate this. Thank you very much for sharing

  10. Thanks all for the responses. By saying it is the most indepth study I was actually referring to the combination of quant RCT and qual ethnographic work done in this case. While RCTs may be happening elsewhere I think few have the combination you had in this case. It is also great the OA has invested significantly in studies. Dare I ask if Oxfam would have done an RCT if Gates wasn’t involved as a funder?

    It is interesting how Janina and your responses further reveal the politics of evidence!! Janina cites that it put SGs to a rigourous test! This seems to underline the fact that the previous 21 studies were perhaps not regarded as rigourous and didn’t allow Sophie to make the upbeat conculsion that she now feels able to!

    However, while I think SGs may be less problematic than microcredit has been in terms of indebting people (largely because they tend to deal with much smaller amounts) I would hope that this won’t result in the creation of another attempt at a mythical silver bullet. The SG “movement” starts to have the feel of the microcredit movement in the 1990s with its Summits etc!

    There are so many issues to address around their dynamics, sustainability etc as with any programme I would want to encourage a careful and balanced approach. But unfortunately you have to shout from the rooftops to get the funding!

  11. Susan, great point that it is actually the combination of RCT and ethnographic elements that make the research particularly powerful. To the SfC teams credit, they have understood, nurtured and fought to make sure that the mixed method approach continue through many obstacles.

    Answering as the Director for Learning, Evaluation and Accountability at OA, we may very well have not done an RCT if Gates was not involved.

    Every agency needs a major donor to fund an exercise as resource intensive as what was done for SfC. But lets also remember that Gates funded the 3 year in depth ethnographic component of the research, and many of the other evaluations using all sorts of other methods. So if Gates was not generous in their funding of a variety of evaluations, Oxfam America would not have done the RCT, or many of the other evaluations. But is that about the politics of evidence or more about the reality of international development funding?

  12. Sophie ton analyse est tres pertinente et aide a mieux comprendre les resultats de la recherche.
    Tres bon travail.

    Courage.

  13. Thanks for this blog as it shows how the impact of SFC is immediate in rural communities in sub-Saharan Africa. In Liberia, We are beginning to see how SFC is helping women build a coherent livelihoods, build their self-esteem and economic viability.

  14. Hi there,

    Can you please elaborate on the qualitative methods used to gather data in this study? (Interviews with open-ended questions, surveys, participant observation, etc.?) And, in more detail please, what were the results? You mention that the qualitative methods showed increases in social capital not captured by quant methods. What quant measures of social capital were being used? And, are the qual measured increases in social capital correlated with any quant. finding?

    Also, out of curiosity, why were only treated groups assessed using the qualitative methods?

    Thanks!

Leave a comment