Cost-effective Impact Assessments for the Impact Investor
This long post is available as a PDF.
Impact investors can better estimate their impact on growing businesses and, in turn, the impact of supported businesses on communities without a significant increase in evaluation cost. While impact investors rightly regard randomized controlled trials (RCTs) as inappropriate for day-to-day performance measurement, impact investors can use low-cost alternatives (specifically, control groups and Best Alternative Pro Formas) to construct counterfactuals in order to estimate what would have occured in the absence of their investment. While these methods will not meet the standard of RCTs, they can offer a significant and meaningful improvement in the measurement and management of social performance.
To evaluate impact, the impact investor must make “a comparison between what actually happened and what would have happened in the absence of the intervention1.” RCTs represent the most rigorous method available to approximate the counterfactual, which in any case can only be estimated. Presently, most impact investors do not utilize a counterfactual, and this omission introduces both bias and noise into social performance measurement.
This document will address the role of RCTs, the necessity of a counterfactual estimate, and practical alternatives for the impact investor to construct a realistic counterfactual and thereby understand and maximize impact. These principles apply both to estimating the impact of an organization on a community, and the impact of an investment on an organization, the latter of which is the subject of this analysis.
Randomized Controlled Trials and Impact Investing
In a recent presentation at DFID, Chris Blattman made two points on the proper role of RCTs particularly relevant to this topic: “Don’t evaluate [with RCTs] things that you cannot generalize,” and, “Test ideas not programs2.”
The impact investing community should work with evaluators to establish an RCT agenda that will test industry assumptions and mechanisms (i.e., “ideas”) that underpin the industry at large. However, while RCTs can play a crucial role in validating (or calling into question) the theoretical underpinnings of the impact investing model, they are not appropriate for impact investors to evaluate the performance of individual investments (i.e., “programs”) in their portfolio.
Motivation for a Counterfactual: Why Changes in Outcomes are Not Enough
In order to understand and then maximize the impact – value-added – of impact investing, the investor needs to assess the performance of its investments based on the incremental performance improvement that is due to their support. There are a number of reasons why a change in outcome indicators will not reliably provide an (even roughly) accurate picture of the investor’s impact on the firm or community. Three of these will be addressed below:
a) Maturation: Organizations may demonstrate improvements over time that are not due to any intervention, but due to the passage of time which allows for growth and thereby improved outcomes.
Implication: If you assume that changes in your investment’s outcome indicators are attributable to your investment, your perception of your impact will be greater than the reality.
b) Selection: Impact investors are likely to invest in organizations that show particular promise, some aspects of which are difficult to capture in any metric (such as, management quality, industry connections).
Implications: Compounding the maturation bias, selecting investments based on positive characteristics makes it more likely that your investments would have matured without your investment. Therefore, if you assume that changes in your investment’s outcome indicators are attributable to your investment, you will overestimate your impact.
If you choose as a counterfactual the change in outcomes of another organization with similar observable characteristics (though less impressive in some way that preempted investment) your perception of impact will be greater than the reality, as your investment would likely do better than the comparator with or without your support.
c) History: External events, such as changes in commodity prices, may positively or negatively affect how well your investment does.
Implications: If you assume that changes in your investment’s outcome indicators are attributable to your investment, you may over or underestimate your impact, depending on whether external conditions have become more or less favorable for your investment.
If you choose as a counterfactual the change in outcomes of another organization that was not affected by the same external events, you may again over or underestimate your impact, depending on which organization’s external events were relatively more favorable.
There are a number of qualitative and quantitative methods for constructing counterfactuals, which – while not reaching the RCT’s standard of internal validity – can improve impact investors’ understanding of an investment’s impact. This document will focus on quantitative methods with particular attention paid to addressing the aforementioned threats to the internal validity of the counterfactual estimate.
Estimating the Counterfactual: Alternatives to RCTs
In theory, Control Groups can address all three biases, albeit imperfectly, depending on how well the comparators match your investments.
The impact investor should have strong contacts in the areas and industries in which it operates and have relationships with organizations that it does not assist. The impact investor could “match” other firms to their portfolio companies, and interpret the performance of the comparison organization as the counterfactual. At the same time, if the portfolio companies differ systematically from these comparators (e.g., management team quality) then these firms will not provide appropriate counterfactuals. In addition, depending on the industry, it may prove quite difficult to find an appropriate comparator.
Weaknesses: The greater the impact investor’s focus on finding high-potential individuals or companies, the more difficult it will be to find adequate comparators. In addition, it may be difficult to acquire performance data from privately owned comparator companies, especially as they may be competitors of your investments.
Best Alternative Pro Forma Forecasts are Pro Formas that project future performance of the organization had they not received your investment. For example, if you were proposing an equity investment of $200,000, the best alternative might be financing growth with a series of loans from a local bank or microfinance institution. The Best Alternative Pro Formas would provide an estimate of the counterfactual at the time of the investment.
Pro Formas are a natural component of the due diligence process. With minimal incremental cost, an additional Pro Forma could be constructed for each prospective investment that reflects the enterprise’s forecast without the impact investor’s involvement. In some cases, this might assume additional capital from a local source. In all cases, it would amount to the best estimate of future performance based on the resources available outside the impact investor.
In addition, the impact investor can compare the original Pro Forma (with investment) with the actual performance of the investment. Investments that under- or over-perform their forecasts may have similarly under- or over performed their Best Alternative Pro Formas had they not received an investment. In this case, the impact investor could adjust it’s counterfactual estimate up or down by the amount that the investment under- or over-performed its Pro Forma with investment.
The forecasts can and should incorporate the flow of economic and social benefits to different stakeholders. The Asian Development Bank, among others, has produced a number of guidelines on this subject3.
Weaknesses: Foresight is limited: the enterprises make forecasts with imperfect information at a point in time, after which company and market conditions change. It’s worth differentiating the error introduced through:
a) Unforeseen external conditions with a predictable impact on company growth
b) Unforeseen external conditions with an unpredictable impact on company growth
To the extent the impact on growth is predictable, the impact investor can create a Pro Forma that links key external condition variables to company growth and thereby improve the accuracy of the estimate (by updating the values in the future when the values become known).
There are two additional weaknesses alluded to above. First, forecasts are often optimistic. To some extent, organizations can control for overly optimistic forecasts by accounting for the extent to which the organization under-performed its forecast with investment.
Second, a forecast is more subjective than a control group comparison. It introduces a greater possibility for gaming, and requires internal controls.
- Changes in outcome indicators cannot be attributed solely to your investment due to maturation, selection, and history biases.
- Your investments are special: it will be difficult to find comparator firms because these comparators did not receive your support for a reason –likely a lower growth trajectory or less appealing intangibles.
- Forecasts are difficult to do well: it will take thoughtful financial modeling to construct forecasts with and without investment (prior to investment) that will provide a realistic and objective estimate of the counterfactual.
- Pilot the construction of Best Alternative Pro Formas and discuss utility of incorporating key external condition variables (to be updated in the future) in the financial model and adjusting the Best Alternative Pro Forma based on the firm’s under- or over-performance of its Pro Forma with investment.
- Explore the viability of matching investments with comparators.
- Discuss challenges to ensuring the objectivity of the impact assessments through internal controls or external contracting.
If you have any questions, comments, or feedback, you can reach me at cdp283 at nyu dot edu.
1. White, H. (2006) “Impact Evaluation: The Experience of the Independent Evaluation Group of the World Bank,” World Bank, Washington, D.C.
2. Blattman, C. (2011) “Impact Evaluation 3.0?” DFID Presentation. http://www.chrisblattman.com/documents/policy/2011.ImpactEvaluation3.DFID_talk.pdf
3. ADB. (1993) “Guidelines for the Economic Analysis of Projects.” http://beta.adb.org/documents/guidelines-economic-analysis-projects