The emerging answer seems to be similar to the answer to the question: "Does Aid Work?" It depends.
We just finished up the first panel session at Microfinance Impact 2010, moderated by Jonathan Morduch, and with presentations by Dean Karlan (Yale and IPA), Abhijit Banerjee (MIT), Esther Duflo (MIT), Carlos Danel (Compartamos Banco) and Tanguy Bernard (Agence Française de Développement).
Dean Karlan began by setting the scene, describing the remarkable “audacious to humble” transformation of claims about microcredit. From the magic bullet for fighting poverty, to merely lifting millions from poverty, to raising consumption, to just helping the poor cope, to not doing any of that but allowing for greater freedom and empowerment.
Why do we care about measuring impact? For three reasons:
1. To bring in the skeptics
2. To decided how to allocate our scarce aid resources
3. To do better than the "market test." This test is also passed by cigarettes and fatty foods, for which we don't see much social entrepreneurship. Similarly, research in the US tends to be more about getting people out of credit than into it. What is going on?
So how do we measure impact? We randomize. In the context of microfinance this has so far meant finding all of the clients who are on the borderline of being passed or rejected on credit score, and randomizing within this group.
He then described 3 studies that have been done to date:
1. In South Africa – a study looking at consumer payday lending rather than "traditional microcredit" – which found a 10% increase in the likelihood that participants remaining employed.
2. In the Philippines - a study with First Macro Bank – measuring the 1-year impact of lending to individuals with microenterprises, which found no microenterprise growth, but consolidation of activities and a reduction in the number of employees, but and an increase in the ability to manage risk.
3. Again in the Philippines - a another study with 3 different banks looking at what people spend their money on. Now you might expect people to lie to their bank. So the study did 4 different things - asked people directly what they did with the money, did a survey using independent (of the bank) surveyors, used a "list method" for indirect elicitation, and finally compared spending patterns in the aggregate between treatment and control.
It turns out that people are indeed cautious about what they tell their bank, and that significant proportions of people are spending their microenterprise loan for paying down existing debt or buying household items.
Points to remember
1. Credit is not a panacea (but using money for paying debt and buying items can be welfare-enhancing)
2. Impact measurement is not always possible – monitoring is also important – but we need to be cautious when doing social performance measurement to stick to that, and not start trying to evaluate impact.
3. We need more information – especially on savings – lots of research suggests that this what people are really interested in. Also the product design is crucial. Sendhil Mullainathan's plenary tomorrow will discuss using psychology to improve product design.
Next up was Abhijit Banerjee discussing some new analysis of results from South India. He also gets the prize for best quote of the day, noting that "This is the fanciest room in which I’ve ever spoken about the poor." The key insight from his presentation was the important of heterogeneity. In plain english - microfinance can have drastically different impacts for different kinds of people, and average figures can hide much of what is happening.
On average there was no evidence of significant impacts upon profits, revenues, spending or women's empowerment.
But what if you look at existing business-owners separately from those without businesses? Those who already had a business, or were planning on setting one up, did increase their investment, and cut down on "temptation goods" such as cigarettes and tea. For those without businesses the pattern was the opposite, increases in everyday consumption goods and no increase in investment.
Esther Duflo presented some brand new data “fresh out of the oven” from a study in Morocco with Al Amana Bank. The set-up was similar to Spandana in Hyderabad, but in a rural setting. The data is about 3 weeks old so we got some raw results. After 2 years, between 10% and 16% of the targeted population had taken up the loan offer (lower demand than expected). A key feature of the results is again the importance of heterogeneity (differing impacts upon different types of people).
The results (on average):
• A decline in existing loans.
• No effect on consumption.
• Reduction in social expenditure (e.g. festivals)
• No impact upon the starting of new businesses (there were lots of new businesses - around 43% of the population starting one - but no difference between treatment and control).
• A change in activity – diversification – through buying more types of animals
• A change in composition of income – from wage to own-production
• The volume of activity increased – profit, sales, wages, assets, savings
• No impact on education or women’s empowerment
Does microcredit allow people to deal with shocks? If it does then consumption should decrease less in response to negative shocks than in places where there is access to finance. Disappointingly, no results were found that microcredit is a way of coping.
Carlos Danel gave a perspective from a bank working on several impact studies in partnership with IPA. "How do you tell a loan officer that he can serve clients on one side of the street but not the other? For four years?" Thankfully Compartamos are committed to learning about what really works and moving away from operating on pure assumption. Carlos also described a process which is going on within the community of microfinance institutions of coming to terms with the emerging evidence on what the real impact can be. There is also a trend to building research capabilities within the banks.
Tanguy Bernard gave a perspective from a donor agency, looking at some of the limitations of impact evaluations; namely the short time horizons that have been studied so far, and the focus on 'untouched' areas which may not be representative of the wider population.