Lee Crawfurd, who you may know has the best economics blog (previously) in South Sudan, recently highlighted a study that assessed the impact of a job placement program in France. The paper provides an excellent case study in how perceived impact can disappear, highlighting the importance of sound estimation of what would happen without an intervention, i.e., the counterfactual.
The program provided assistance to jobseekers, and depending on how you measured your impact, you would have received a very different story. Read the rest of this entry »
Linking payment to performance on pre-defined outcomes makes a lot of sense. And in the social sector, where payment has historically been linked to simply providing services, it’s easy to understand the excitement for Social Impact Bonds and “Pay for Success” contracts. You get what you pay for, the thinking goes, and a shift from outputs to outcomes could be transformative.
Yet it is worth reflecting on the experience of those who have come this way before us, and just as the social sector has followed the health care industry’s lead on randomized controlled trials, so too the recent enthusiasm for “Pay for Success” follows the health care sector’s embrace of “pay for performance” (P4P) and “value-based purchasing”. Read the rest of this entry »
It’s not clear why Daron Acemoglu and James Robinson (A&R) wrote a short chapter on Venice in their recent book, “Why Nations Fail: The Origins of Power, Prosperity, and Poverty.” The Republic didn’t play a significant role in the research that led to the book, and even their telling of its rise doesn’t align with their emphasis on political inclusion as the first-mover (here, they state, economic institutions led to the political institutions.)
Yet the chapter is short and quickly digested, and spurred a Times article based on their book’s narrative of how broad-based economic opportunity and political participation led to Venice’s growth and inequality and concentration of political and economic power to its decline. The Times article writes itself: in today’s America, we are seeing the same “self-destruction of the 1 percent.”
A&R’s history of Venice therefore not only resonates with a deep-seated moral discomfort, but seems to substantiate it, to affirm its primacy as the determinant of national prosperity. In reality, Venice’s rise and fall speaks more to the importance of geostrategic factors outside the control of even the elite than the pernicious effects of concentrated elite power or “extractive” political and economic institutions. It’s not that political or economic institutions don’t matter at all, but that “good” institutions aren’t always the most externally competitive, and that technological changes and new competitors can lay to waste the greatest institutions and elevate the worst. Read the rest of this entry »
IPA continues to find inventive new ways to use randomized controlled trials to better our understanding of giving and development. While usually the focus is on how well programs help their beneficiaries, a recent study explores how individual giving is affected by a matched donation campaign. With the cooperation of the Gates Foundation and Technoserve, Dean Karlan and John List tested the impact of a matching grant solicitation on individual giving (with a solicitation with no mention of a match as the counterfactual), as well as the impact of naming the Gates Foundation as the matching donor (with an anonymous grant match solicitation as the counterfactual). The former test was directed at prior donors while the latter at prospective donors. The offer was a $3:$1 match.
Besides identifying the income generated by a matching program, this study is an important step in quantifying the brand equity of organizations like the Gates Foundation. That is, what’s the value of the reputation that the Foundation has established in the field. For example, while the average return was $0.57 for the match offer from an anonymous donor, the return was $0.81 for a Gates grant, a 42% increase. It’s worth noting that the populations of the two different tests in the study are not identical: one solicited donors that had previously given and the other solicited individuals who had not previously given. Still the pair of experiments allows for at least an intuitive understanding of the Gates’ value-add. Further by examining how this reputational impact varies by donor type, Karlan and List find that there are significant distinctions in reputational value between different audiences.
The Gates brand is an asset with significant value, and this study clarifies one part of that value. The business world has long observed the value of experiments for market research purposes, perhaps this is a first step for the philanthropic world in similarly leveraging experiments to better understand the value of their less-tangible assets in order to support better-informed management.
Some of the key results:
Gates reputation test: Solicitations targeting non-prior donors with and without Gates named as the 3:1 match donor
The total revenue per solicitation was $0.81 for Gates-named solicitations and $0.57 for anonymous-match solicitations. In the match period, the revenue per solicitation in the $0.29 with anonymous matching, and $0.41 with Gates named (40% increase).
Matching test: Solicitations targeting prior donors with and without Gates 3:1 match offer
Total revenue per solicitation during and after the match period was $0.64 for the Gates match solicitations, versus $0.29 for the no-match solicitations. Average revenue per solicitation during the match period went from $0.15 to $0.28 with the Gates match (81% increase). This was driven largely by an increase in the probability of a gift, with no change in size. It also increased the likelihood of giving again after the matching program ended, as the likelihood of a repeat gift increase from 0.24% to 0.46%. This suggests that the increase in giving was driven by the perception that the organization was of a higher quality due to the Gates endorsement, rather than the matching program itself.
Heterogeneous impact on “poverty-oriented” charities
The study finds that the Gates brand has a larger impact on individuals who have previously given to poverty-oriented charities. Karlan and List speculate that this may be because the effect of the quality signal is greater with individuals who know more about the sector and presumably the importance of the Gates Foundation.
Disclaimer: This is based on a quick read of the article, please let me know if there are any errors or misinterpretations of the results.
While progress has been made with outcome reporting standards, e.g., IRIS, much work remains to establish impact measurement practices that will allow Acumen staff to credibly assess the effectiveness of different products and services. Once developed, these practices will support both greater internal learning and external understanding of Acumen’s social return on investment.
Last February, ANDE’s Randall Kempner wrote a blog post very much related to this subject that likely caused some pulses to quicken:
Organizations join ANDE because they share the common mission of increasing the prosperity of poor people in emerging markets, believing that supporting entrepreneurship and market-based solutions is part of a sustainable approach to poverty reduction. … But that’s a hypothesis; there isn’t sufficient evidence to back the assertion that small and growing businesses are a key part of poverty reduction. That’s why I was scared… I was standing there thinking “I don’t want to be sitting in a room like this 10 years from now, hearing how my industry, my movement, my life’s work had ‘an average impact on poverty of zero.’”
ANDE is collaborating with researchers to improve the robustness of the existing evidence base on the impact of small and growing businesses on poverty. But what does this mean for Acumen? Acumen now finds itself in the same position as impact-driven MFIs in the 90s. The additional social value of the products and services supplied by Acumen portfolio companies accords the organization a competitive advantage, but the less rigorous and robust the impact measurement, the more difficult it is to communicate this added value. Indeed without more robust impact measurement, the impact component of impact investing will be treated as a commodity. The story of microfinance offers a clear example, as institutions with marginal impact were perceived to be the same as those with significant impact. The impact-driven MFIs benefited less than they should have in the short term, while then sharing fully in the bust prompted by their less noble counterparts.
Outside of external communication, Acumen also has much to gain from impact measurement with regards to refining portfolio strategy and service offerings.
As Kempner noted in his piece, impact investing is a nascent field and there is not yet sufficient evidence to validate its theory of change. Given the heterogeneity of the impact investing field, there will never be a study that will show that it does or does not reduce poverty. Every country context, every industry, every combination of capital, assistance, and facilitation will ultimately result in a “new” program with a different set of causal mechanisms. There will likely be some combinations that result in successful enterprises with limited impact, and others that produce less successful enterprises with more impact. Acumen already rightly understands that the point of entry in an enterprise’s lifecycle is one such crucial factor. The Head of Impact will need to help systematize and deepen this learning to better understand and validate the mechanisms by which Acumen is achieving impact. While monitoring and evaluating alone won’t lead to dramatic changes overnight, in time, the work produced by the Head of Impact should undoubtedly shape Acumen’s portfolio strategy.
So what does this mean in practical terms? Acumen’s already worked to help establish a common IRIS lexicon. The next step is increasing the robustness of impact measurement through embracing the counterfactual (more here) and developing strong internal controls. This doesn’t mean turning Acumen into nothing more than a research lab for RCT advocates. It does mean taking concrete, transparent steps toward estimating the incremental impact catalyzed by Acumen’s work, by creatively utilizing the myriad techniques available that require less in additional resources than they do a change in mindset. And yes, it means engaging with the empirical researchers in order to ensure the research agenda of initiatives like IPA’s SME initiative reflect the research interests of Acumen. It means advocating for small steps that will result in stronger learning and understanding of what works and developing processes and tools to help make this vision a reality.
This undertaking will require an academic’s sense of rigorous impact estimation, a cutting-edge NGO’s creativity in devising cost-effective methods, a market researcher’s sense of turning data into useful managerial information, and a controller’s insistence on sound internal controls to maintain the needed integrity and objectivity of the results. The high-impact MFIs of the 90s lacked this combination of skills and saw their stock and reputation vacillate and ultimately fall with that of low-impact MFIs. Acumen is well-positioned to avoid this fate. By developing the infrastructure and processes needed to ensure it is successfully managing to impact, Acumen can at the same time set the bar for other impact investors, supporting the development of a true market for impact and catalyzing the philanthropic support needed for the impact investing field’s Blueprint to Scale.
Dave Algoso recently wrote up an overview of One Acre Fund’s performance dashboard that got me again thinking about how we can create dashboards to both support performance management and communicate value to donors and partners. I have a lot of respect for One Acre Fund’s work, it’s efficiency, impact, and, as Dave notes, emphasis on evidence-based program development. Certainly, from the outside, it appears that the bundled services have been well chosen to meet the needs of farmers while minimizing the subsidy required. Read the rest of this entry »