The Principled Agent

Thoughts on development economics and impact measurement

Impact Investing in Rigorous Analysis

with 7 comments


In a recent interview with Fast Company, Dean Karlan noted:

The social entrepreneurship world is in a weird spot, to be honest with you. It’s a world full of rhetoric about impact investing, yet I have very rarely seen an investor actually take that seriously. When you look at the actual analysis it lacks rigor.

Ouch. Some readers will surely dismiss Karlan as a ‘randomista’, but his critique is fair, and one that I have made before.

Take microcredit. What they do is they track how many borrowers, what their repayment was, and whether the businesses grew over time. That is not telling you your impact at all because you don’t know what would have happened if you didn’t lend.

They push toward collecting data that doesn’t necessarily answer the question. Now you’ve burdened your organization with data collection so you can feel rigorous.

He is exactly right. The emerging industry standard, the Global Impact Investing Network’s “Impact Reporting and Investment Standards,” presents a case study. While the standards represent a dramatic step forward for the standardization of data reporting, these standards require organizations to invest a lot of time answering a lot of questions, e.g. volunteer hours worked by employees, square feet of community facilities financed. The goal is clearly to be comprehensive, with organizations and investors able to evaluate their respective areas of interest. However, while the breadth of the data collected is impressive, “[it] is not telling you your impact at all because you don’t know what would have happened if you didn’t lend.”

Like with Karlan’s microcredit example, these standards only permit a “monitoring exercise.” The standards opt to monitor a little bit of everything, rather than concentrating attention and resources to try to estimate the social impact of the organization’s core social value proposition (e.g., increased income for low-income employees, reduced water pollution).

More broadly, none of the emerging ‘best practices’ for impact reporting in impact investing even attempt to estimate what would have happened without the intervention. Because of this, the use of the term of ‘impact’ is incorrect and misleading. That said, I fear that this is partially due to the failure of the rigorous evaluation camp to sign off on non-randomized methods of estimating the counterfactual, that is, what would happen in that investee firm, community, etc. without the intervention. For sound practical reasons, RCTs will never be the impact investing standard for measuring impact, as Karlan himself accepts. So what now?

The challenge is to find a middle ground that allows for the best possible estimate of social impact given the resource constraints of impact investors. First, impact investors must believe there’s value in investing a bit more time and energy estimating social impact (perhaps reallocating some of that time and treasure spent monitoring volunteer hours worked, for example). After all, if you’re investing in a company with the goal of creating jobs or cleaning the environment, wouldn’t you rather have a good estimate of your investee’s impact on those core goals rather than a lot of monitoring data on things that wouldn’t otherwise justify your investment? Sure, you won’t have an RCT, but that doesn’t mean you can’t estimate the counterfactual in order to greatly improve your understanding of your impact and inform your allocation of resources across competing investment opportunities.

Since this isn’t new ground for this blog, I thought I’d add a few ideas on how an impact investor might estimate the impact of their investment in a firm. The principles are similar for estimating the impact of a single firm on a community, employees, pollution, etc.

a)      Create a “living” forecast of the firm’s performance with best alternative financing at baseline and plug in actual future economic variables (e.g., commodity prices) to update forecast estimates over time

b)      Performance of businesses that operate in the same industry and of a similar size

c)      Performance of similar businesses that apply for financing but do not receive it

d)      Triangulation of counterfactual estimation based on multiple estimation methods

Each option has its own drawbacks, and I only briefly note these ideas as examples of relatively cheap methods of estimating counterfactuals. (Perhaps an RCT could shed light on the relative accuracy of these different estimation techniques?)

Back to the big picture, it’s difficult to create a standard by which investors or firms would estimate the counterfactual, given the different types of impact sought by investors. However, it’s not impossible, and it certainly is possible to create standards for a) employing a counterfactual, b) reporting the process of estimating the counterfactual, c) reporting how potential positive bias was mitigated, d) reporting on the justification for this method of counterfactual estimation, to include the marginal cost of a more rigorous counterfactual estimate.

These are just some ideas, but given impact investing’s great potential for social impact, it would be a shame to ignore this opportunity to better understand this impact and to guide investors to the opportunities for greatest social impact in the future.

Advertisements

Written by Chris Prottas

August 12, 2011 at 3:11 am

Posted in Uncategorized

7 Responses

Subscribe to comments with RSS.

  1. Quick question, Chris. If this criticism of evaluation (it’s not a criticism of impact investing, really) is based on a comparison to the purely for-profit sector, then why is the need for counterfactuals so important? Businesses don’t use the kind of scientific research that Dean Karlan is advocating, and they don’t try to measure what might happen to the market/industry/society if they DIDN’T make the investment. The only impact they are concerned about is their own bottom line.

    While impact investing does have a double reporting responsibility, dismissing it on the basis of imperfect program evaluation (which is really what both you and Dean Karlan are rightly focusing on) seems a little harsh.

    Also, there are multiple evaluation models – it doesn’t necessarily have to be before-and-after research, doing comparisons with a control group can be just as valid depending on the specifics of the intervention.

    So, again, why are counterfactuals SO important?

    Nabeel

    August 12, 2011 at 3:47 pm

  2. Again, evaluation is obviously critical. Social impact measurement has dominated many of the discussions around impact investing, which makes it doubly disappointing that Dean Karlan thinks that impact investing dismisses the need for impact measurement.

    Nabeel

    August 12, 2011 at 3:48 pm

  3. Nabeel,

    Thank you for your comments. You make a good point that “Businesses don’t use the kind of scientific research that Dean Karlan is advocating, and they don’t try to measure what might happen to the market/industry/society if they DIDN’T make the investment.”

    However, investors focused on making profits have a clear way to evaluate their financial returns – profits. If investors assessed their financial performance the way they are currently assessing their social performance, then they would not look at the profits (suppose the accounting was a bit expensive) and instead focus on metrics of how many stores their investees operated, or how many shoes they distributed. Clearly, without the ultimate indicator of financial success (profits) they would be ‘bowling with the lights off’, to borrow from Duflo.

    Further, I doubt we’d trust our money to an investor who invested in companies that didn’t report their profits and have sound accounting in place, instead supplying us with information on what activities their investees were doing. Yet this is what we’re doing now with social performance.

    Unfortunately, it’s easier to assess financial returns than it is social returns, and we’ll never have that same level of certainty (a profit is a profit… usually, whereas an estimate of impact is just that.)

    To me, if an organization is going to ask an investor to accept sub-market returns in exchange for ‘social impact’, then it should at the very least attempt to estimate this impact. As I said, I don’t think this means allocating 20% of your budget to M&E or randomizing your investment decisions, but I do think it means making a concerted effort to construct a reasonable counterfactual and to report on how the counterfactual was constructed.

    Creating a comparison group, as you mentioned in your comment and I mentioned in the post, would be way one way to do so. There are a number of challenges to creating a ‘well-matched’ comparison group, but I would be happy to see this integrated into how ‘impact investors’ and ‘social entrepreneurs’ assessed their impact. However, right now this is neither standard practice nor (to the best of my knowledge) a part of the emerging impact analysis and reporting standards being put forth.

    I see that you work for Social Finance (I enjoy your website!) and if you are aware of some best practices that I have missed I certainly would love to see what’s being worked on!

    Best,

    Chris

    Chris Prottas

    August 12, 2011 at 4:42 pm

  4. Chris,

    Sorry for the late response. You make some good points, but seem to ignore that impact investments have a financial AND a social benefit. Of course we wouldn’t trust money to investors who don’t report profits, but impact investors do report profits and do have accounting in place. Those aren’t necessarily being taken away, but yes, the measurement of the social impact needs to be improved.

    There is a way to go before social impact is properly accounted for, and I appreciate your insight on how it can be improved. I also think we need to define impact. Is it the incremental outcome (which relies upon having a baseline measure or control group as you and I have, respectively, suggested) or is it ANY positive outcome that can be causally linked to (or even correlated with) the intervention (and thus the investment)? I think that you are (as mentioned in your previous post) defining it as the former, while many take it to be the latter.

    I agree, there needs to be improvement, and from conversations with impact investors I can tell you that this is a shared concern. There are people addressing it…let’s see what they come up with.

    Nabeel Ahmed

    August 17, 2011 at 3:10 am

  5. […] Setting aside the methodological question of how to assess impact, I think the most important take away from the RCT movement is that there’s a lot of good looking pyrite. […]

  6. […] Impact Investing in Rigorous Analysis Advertisement GA_googleAddAttr("AdOpt", "1"); GA_googleAddAttr("Origin", "other"); GA_googleAddAttr("theme_bg", "ffffff"); GA_googleAddAttr("theme_border", "f3f3f3"); GA_googleAddAttr("theme_text", "1c1c1c"); GA_googleAddAttr("theme_link", "004276"); GA_googleAddAttr("theme_url", "cc0000"); GA_googleAddAttr("LangId", "1"); GA_googleFillSlot("wpcom_sharethrough"); Share this:StumbleUponDiggRedditLike this:LikeBe the first to like this post. […]

  7. […] Impact Investing in Rigorous Analysis Share this:StumbleUponDiggRedditLike this:LikeBe the first to like this. […]


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: