Impact Investing in Rigorous Analysis
In a recent interview with Fast Company, Dean Karlan noted:
The social entrepreneurship world is in a weird spot, to be honest with you. It’s a world full of rhetoric about impact investing, yet I have very rarely seen an investor actually take that seriously. When you look at the actual analysis it lacks rigor.
Ouch. Some readers will surely dismiss Karlan as a ‘randomista’, but his critique is fair, and one that I have made before.
Take microcredit. What they do is they track how many borrowers, what their repayment was, and whether the businesses grew over time. That is not telling you your impact at all because you don’t know what would have happened if you didn’t lend.
They push toward collecting data that doesn’t necessarily answer the question. Now you’ve burdened your organization with data collection so you can feel rigorous.
He is exactly right. The emerging industry standard, the Global Impact Investing Network’s “Impact Reporting and Investment Standards,” presents a case study. While the standards represent a dramatic step forward for the standardization of data reporting, these standards require organizations to invest a lot of time answering a lot of questions, e.g. volunteer hours worked by employees, square feet of community facilities financed. The goal is clearly to be comprehensive, with organizations and investors able to evaluate their respective areas of interest. However, while the breadth of the data collected is impressive, “[it] is not telling you your impact at all because you don’t know what would have happened if you didn’t lend.”
Like with Karlan’s microcredit example, these standards only permit a “monitoring exercise.” The standards opt to monitor a little bit of everything, rather than concentrating attention and resources to try to estimate the social impact of the organization’s core social value proposition (e.g., increased income for low-income employees, reduced water pollution).
More broadly, none of the emerging ‘best practices’ for impact reporting in impact investing even attempt to estimate what would have happened without the intervention. Because of this, the use of the term of ‘impact’ is incorrect and misleading. That said, I fear that this is partially due to the failure of the rigorous evaluation camp to sign off on non-randomized methods of estimating the counterfactual, that is, what would happen in that investee firm, community, etc. without the intervention. For sound practical reasons, RCTs will never be the impact investing standard for measuring impact, as Karlan himself accepts. So what now?
The challenge is to find a middle ground that allows for the best possible estimate of social impact given the resource constraints of impact investors. First, impact investors must believe there’s value in investing a bit more time and energy estimating social impact (perhaps reallocating some of that time and treasure spent monitoring volunteer hours worked, for example). After all, if you’re investing in a company with the goal of creating jobs or cleaning the environment, wouldn’t you rather have a good estimate of your investee’s impact on those core goals rather than a lot of monitoring data on things that wouldn’t otherwise justify your investment? Sure, you won’t have an RCT, but that doesn’t mean you can’t estimate the counterfactual in order to greatly improve your understanding of your impact and inform your allocation of resources across competing investment opportunities.
Since this isn’t new ground for this blog, I thought I’d add a few ideas on how an impact investor might estimate the impact of their investment in a firm. The principles are similar for estimating the impact of a single firm on a community, employees, pollution, etc.
a) Create a “living” forecast of the firm’s performance with best alternative financing at baseline and plug in actual future economic variables (e.g., commodity prices) to update forecast estimates over time
b) Performance of businesses that operate in the same industry and of a similar size
c) Performance of similar businesses that apply for financing but do not receive it
d) Triangulation of counterfactual estimation based on multiple estimation methods
Each option has its own drawbacks, and I only briefly note these ideas as examples of relatively cheap methods of estimating counterfactuals. (Perhaps an RCT could shed light on the relative accuracy of these different estimation techniques?)
Back to the big picture, it’s difficult to create a standard by which investors or firms would estimate the counterfactual, given the different types of impact sought by investors. However, it’s not impossible, and it certainly is possible to create standards for a) employing a counterfactual, b) reporting the process of estimating the counterfactual, c) reporting how potential positive bias was mitigated, d) reporting on the justification for this method of counterfactual estimation, to include the marginal cost of a more rigorous counterfactual estimate.
These are just some ideas, but given impact investing’s great potential for social impact, it would be a shame to ignore this opportunity to better understand this impact and to guide investors to the opportunities for greatest social impact in the future.