The Principled Agent

Thoughts on development economics and impact measurement

Separating the Gold from Pyrite: Analytical Rigor and the Potential of Impact Investing

with 4 comments

This post addresses two questions raised in the recent discussion on impact evaluation and investing: to what extent is the burden of proof lower for impact investing than traditional philanthropy, and what type of analysis does the industry itself need to separate social enterprises with goods intentions and significant social impact from those that struggle to translate the former to the latter.

Acumen Fund’s Sasha Dichter recently responded to Dean Karlan’s critique of the lack of rigorous analysis of social investments by differentiating between the social investment and the “purely philanthropic intervention”:

No matter what scale a pure philanthropic intervention reaches, the total marginal cost of delivering the nth “thing” (any intervention) is always positive, so you’re in the business of figuring out how the impact relates to that cost and how the impact relates to other similar interventions.  Not so if you find the “next cellphone” – except it’s not a cellphone, it’s safe drinking water or a bio-mass powered light on a mini-grid or a safe and affordable place for a mother to give birth.

Sasha seems to be differentiating between the two because the philanthropic venture’s product or service always requires some subsidy, while the social enterprise does not. Similar to the cell phone manufacturer, the social enterprise will (at least partly) find its profit and social incentives aligned, with healthy growth and adoption signaling that it’s having a social impact without the need for a rigorous impact evaluation. Continuing with the cell phone example, Sasha notes, “Since we’re not using up grant money to pay for it, it stops being the purview of development economists to fret about this.”

From my perspective, this last point is crucial. If you are providing investors with market-rate returns without direct or indirect subsidy, I completely agree that your burden of proof is fundamentally different than that of a philanthropist.

If you are providing submarket risk-adjusted rates of returns, however, or accepting subsidies of one type or another, then you are leaving the world of cell phones and entering the world of bed nets. I believe that the majority of impact investors fall in this category, with the subsidy coming in a number of different forms.

Now if JP Morgan is correct, impact investing has the potential to be a $1 trillion market, and if just one-percent of that total is a subsidy in one form or another, we’re talking about a potential $10 billion philanthropic market. 

Given this outsized potential, the fact that a large number of purely philanthropic ventures with clear-cut social value propositions and a 100-percent focus on social returns have shown  virtually no impact on the lives of the poor should give pause to impact investors who are now laying the foundation for their own high-potential philanthropic market.

In addition, the microcredit industry demonstrates the damage that can be done to an industry with financial and social ambitions that allows all the players to sell a significant social impact not based in rigorous analysis but solely in the supposed benefit of providing a service “that make[s] a material positive impact on the lives of poor people.”

Microcredit surely has benefitted the poor, but would the poor have been better off if those hundreds of millions of dollars of subsidy had been put to microsavings? Or would it have been better spent subsidizing capital equipment loans for producer associations? Or, perhaps most importantly, could it have been better spent on pro-poor microcredit itself if its allocation and social investment criteria were informed by more rigorous social performance measurement before the IPOs?

Like microcredit, I’m not worried about whether impact investing will have a benefit greater than zero. I am worried that the social performance data collected will not allow impact investors to separate the gold from the pyrite, which will reduce the effectiveness of the budding industry.

Setting aside the methodological question of how to assess impact, I think the most important take away from the RCT movement is that there’s a lot of good looking pyrite.

Within the impact investing sphere, there are a huge variety of industries (e.g., irrigation systems, medical care), business models, and markets, all of which will affect the ultimate impact of a financially-successful social enterprise. That is, impact will likely vary across industry, market type, and business model (among other variables), and this presents a number of questions:

Based on the emerging impact analysis and reporting best practices…

  • Are impact investors able to assess the relative social impact of these different interventions and to shift their investment focus and criteria to ensure that they are maximizing their social impact (within the constraints of their financial goals and requirements)?
  • Can an impact investor who is very adept at separating the impact gold from the impact pyrite distinguish herself from the impact investor who can’t?
  • Can impact investors shift the basis of “competition” for social investment from the investor’s social value proposition (e.g., financial services for the poor, rural business development) to social impact? In other words, are we seeing the birth of an impact investing industry or just the maturation of the socially-responsible investing model into a more proactive “good intentions” investing model?

Written by Chris Prottas

August 18, 2011 at 2:18 am

Posted in measurement

4 Responses

Subscribe to comments with RSS.

  1. Hey Chris – thanks for the post. The impact investing debate is interesting, because – maybe like microcredit – people are still trying to figure out what it is and what direction it should go. What do you think about the importance of financial returns? Kiva has proved that people are willing to sacrifice for delivering some social value. Also, in the Fast Company article, Dean Karlan talks about spending 33% of the investment on impact studies. Why not leave the impact studies to the economists – make them broad-based, and targeted at the types of investee companies that companies – and, instead, spend the money on other forms of diligence or make additional investments? Great post – I’ll be checking out your blog in the future.


    August 20, 2011 at 10:23 am

  2. Thanks Josh! Great job with your blog as well (I especially dig the posts related to value chain work, which I find fascinating and under-addressed by the blogosphere). I’ll respond to your comments/questions in turn:

    Financial returns: I think they are important to the extent that they serve social impact. I think there are some enterprises-with-social-benefits that come to fruition based solely on their financial potential (e.g., cell phones), and that’s terrific. For those that have insufficient potential (or too high startup costs, or information problems — all those challenges common in immature markets), it might make sense to subsidize them, as Kiva does, in order to get those social benefits. Kiva, and impact investors, have adopted the posture of ‘we’ll channel subsidized finance to it if it sounds like a person would be better off with it than without it.’ To me, this is inefficient. It *might* make sense for an organization like Kiva, which derives its power and money from the resonance of the stories with users (so they can raise more money with that broader spectrum of potential recipients), but impact investors are raising large sums of money from relatively nuanced donors and financiers.

    The cost of understanding how your impact varies across different types of social investments (i.e., industries, market, size) is worth it, if it allows you to intelligently allocate resources among a nearly infinite number of enterprises that sound like they help to varying degrees.

    Impact Studies/Diligence: Related to that last point and your comment, I am not advocating that impact investors judge their impact by RCT (I am worried that people read this post as an insistence on RCTs — it’s not — though hopefully my proposal(s) are more clear here: RCTs are not feasible for judging the impact of individual investments, likely don’t make sense for the current pilot stage of most of the impact investing firms, and are certainly not cheap. I completely agree with your proposed agenda for the economist’s study, and I think IPA is doing that now with their new SME initiative.

    That said, I don’t think the choice is RCT or nothing. In the policy debates, the big question is to randomize or not to randomize, but I don’t think that is the biggest distinction between current NGO/impact investing monitoring and evaluation practice and the gold standard. The biggest distinction is that (many) NGOs/impact investors don’t make an attempt to estimate the counterfactual. Sure, an RCT provides the best estimate of the counterfactual, but just because it’s not feasible to spend 1/3 of the budget on an RCT doesn’t mean you have to give up on estimating the counterfactual. There are plenty of (less rigorous) methods that will provide a much more realistic estimate of impact than ascribing any and all positive change to the intervention.

    Frankly, I think the biggest obstacle to rigorous evaluation is not the cost, but the attitudinal change required. It requires defining your organization not by what it does (which may or may not work) but by what it wants to achieve. It requires opening up the possibility that you may need to ‘kill your darlings’ in order to serve your target population most effectively. Sure, some NGOs do this, thanks to great leadership and individual effort, but it’s a rare organizational cultural trait from my limited perspective.

    When I look at impact investing, I see an engine with great potential for social impact, and a lot of ideas for how to do it – ranging from the inspiring to the cringe-inducing. It’s not unlike the internet boom. And just like the internet boom, out of this mix of startups will be some studs and some duds. Unlike the internet boom, however, without more rigorous evaluation I don’t know how we’ll tell the difference between Amazon and WIthout more rigorous evaluation, I’m afraid that reality will never hit and we’ll keep channeling money to that same mix of great and bad ideas (all with good intentions). There will be no indicator(s) we can trust to signal impact that will allow high-impact ideas to grow and low-impact (but still good intentioned) ideas to wither and die.

    Chris Prottas

    August 20, 2011 at 2:45 pm

  3. […] Separating the Gold from Pyrite: Analytical Rigor and the Potential of Impact Investing […]

  4. […] Separating the Gold from Pyrite: Analytical Rigor and the Potential of Impact Investing […]

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: