The Principled Agent

Thoughts on development economics and impact measurement

Cost-effective Impact Assessments for the Impact Investor

with 5 comments



This long post is available as a PDF.

Impact investors can better estimate their impact on growing businesses and, in turn, the impact of supported businesses on communities without a significant increase in evaluation cost. While impact investors rightly regard randomized controlled trials (RCTs) as inappropriate for day-to-day performance measurement, impact investors can use low-cost alternatives (specifically, control groups and Best Alternative Pro Formas) to construct counterfactuals in order to estimate what would have occured in the absence of their investment. While these methods will not meet the standard of RCTs, they can offer a significant and meaningful improvement in the measurement and management of social performance.

To evaluate impact, the impact investor must make “a comparison between what actually happened and what would have happened in the absence of the intervention1.” RCTs represent the most rigorous method available to approximate the counterfactual, which in any case can only be estimated. Presently, most impact investors do not utilize a counterfactual, and this omission introduces both bias and noise into social performance measurement.

This document will address the role of RCTs, the necessity of a counterfactual estimate, and practical alternatives for the impact investor to construct a realistic counterfactual and thereby understand and maximize impact. These principles apply both to estimating the impact of an organization on a community, and the impact of an investment on an organization, the latter of which is the subject of this analysis.

Randomized Controlled Trials and Impact Investing

In a recent presentation at DFID, Chris Blattman made two points on the proper role of RCTs particularly relevant to this topic: “Don’t evaluate [with RCTs] things that you cannot generalize,” and, “Test ideas not programs2.”

The impact investing community should work with evaluators to establish an RCT agenda that will test industry assumptions and mechanisms (i.e., “ideas”) that underpin the industry at large. However, while RCTs can play a crucial role in validating (or calling into question) the theoretical underpinnings of the impact investing model, they are not appropriate for impact investors to evaluate the performance of individual investments (i.e., “programs”) in their portfolio.

Motivation for a Counterfactual: Why Changes in Outcomes are Not Enough

In order to understand and then maximize the impact – value-added – of impact investing, the investor needs to assess the performance of its investments based on the incremental performance improvement that is due to their support. There are a number of reasons why a change in outcome indicators will not reliably provide an (even roughly) accurate picture of the investor’s impact on the firm or community. Three of these will be addressed below:

a) Maturation: Organizations may demonstrate improvements over time that are not due to any intervention, but due to the passage of time which allows for growth and thereby improved outcomes.

Implication: If you assume that changes in your investment’s outcome indicators are attributable to your investment, your perception of your impact will be greater than the reality.

b) Selection: Impact investors are likely to invest in organizations that show particular promise, some aspects of which are difficult to capture in any metric (such as, management quality, industry connections).

Implications: Compounding the maturation bias, selecting investments based on positive characteristics makes it more likely that your investments would have matured without your investment. Therefore, if you assume that changes in your investment’s outcome indicators are attributable to your investment, you will overestimate your impact.

If you choose as a counterfactual the change in outcomes of another organization with similar observable characteristics (though less impressive in some way that preempted investment) your perception of impact will be greater than the reality, as your investment would likely do better than the comparator with or without your support.

c) History: External events, such as changes in commodity prices, may positively or negatively affect how well your investment does.

Implications: If you assume that changes in your investment’s outcome indicators are attributable to your investment, you may over or underestimate your impact, depending on whether external conditions have become more or less favorable for your investment.

If you choose as a counterfactual the change in outcomes of another organization that was not affected by the same external events, you may again over or underestimate your impact, depending on which organization’s external events were relatively more favorable.

There are a number of qualitative and quantitative methods for constructing counterfactuals, which – while not reaching the RCT’s standard of internal validity – can improve impact investors’ understanding of an investment’s impact. This document will focus on quantitative methods with particular attention paid to addressing the aforementioned threats to the internal validity of the counterfactual estimate.

Estimating the Counterfactual: Alternatives to RCTs

In theory, Control Groups can address all three biases, albeit imperfectly, depending on how well the comparators match your investments.

The impact investor should have strong contacts in the areas and industries in which it operates and have relationships with organizations that it does not assist. The impact investor could “match” other firms to their portfolio companies, and interpret the performance of the comparison organization as the counterfactual. At the same time, if the portfolio companies differ systematically from these comparators (e.g., management team quality) then these firms will not provide appropriate counterfactuals. In addition, depending on the industry, it may prove quite difficult to find an appropriate comparator.

Weaknesses: The greater the impact investor’s focus on finding high-potential individuals or companies, the more difficult it will be to find adequate comparators. In addition, it may be difficult to acquire performance data from privately owned comparator companies, especially as they may be competitors of your investments.

Best Alternative Pro Forma Forecasts are Pro Formas that project future performance of the organization had they not received your investment. For example, if you were proposing an equity investment of $200,000, the best alternative might be financing growth with a series of loans from a local bank or microfinance institution. The Best Alternative Pro Formas would provide an estimate of the counterfactual at the time of the investment.

Pro Formas are a natural component of the due diligence process. With minimal incremental cost, an additional Pro Forma could be constructed for each prospective investment that reflects the enterprise’s forecast without the impact investor’s involvement. In some cases, this might assume additional capital from a local source. In all cases, it would amount to the best estimate of future performance based on the resources available outside the impact investor.

In addition, the impact investor can compare the original Pro Forma (with investment) with the actual performance of the investment. Investments that under- or over-perform their forecasts may have similarly under- or over performed their Best Alternative Pro Formas had they not received an investment. In this case, the impact investor could adjust it’s counterfactual estimate up or down by the amount that the investment under- or over-performed its Pro Forma with investment. 

The forecasts can and should incorporate the flow of economic and social benefits to different stakeholders. The Asian Development Bank, among others, has produced a number of guidelines on this subject3.

Weaknesses: Foresight is limited: the enterprises make forecasts with imperfect information at a point in time, after which company and market conditions change. It’s worth differentiating the error introduced through:

a) Unforeseen external conditions with a predictable impact on company growth
b) Unforeseen external conditions with an unpredictable impact on company growth

To the extent the impact on growth is predictable, the impact investor can create a Pro Forma that links key external condition variables to company growth and thereby improve the accuracy of the estimate (by updating the values in the future when the values become known).

There are two additional weaknesses alluded to above. First, forecasts are often optimistic. To some extent, organizations can control for overly optimistic forecasts by accounting for the extent to which the organization under-performed its forecast with investment.

Second, a forecast is more subjective than a control group comparison. It introduces a greater possibility for gaming, and requires internal controls.

Conclusions

  • Changes in outcome indicators cannot be attributed solely to your investment due to maturation, selection, and history biases.
  • Your investments are special: it will be difficult to find comparator firms because these comparators did not receive your support for a reason –likely a lower growth trajectory or less appealing intangibles.
  • Forecasts are difficult to do well: it will take thoughtful financial modeling to construct forecasts with and without investment (prior to investment) that will provide a realistic and objective estimate of the counterfactual.

Recommendations

  • Pilot the construction of Best Alternative Pro Formas and discuss utility of incorporating key external condition variables (to be updated in the future) in the financial model and adjusting the Best Alternative Pro Forma based on the firm’s under- or over-performance of its Pro Forma with investment.
  • Explore the viability of matching investments with comparators.
  • Discuss challenges to ensuring the objectivity of the impact assessments through internal controls or external contracting.

If you have any questions, comments, or feedback, you can reach me at cdp283 at nyu dot edu.

References

1.  White, H. (2006) “Impact Evaluation: The Experience of the Independent Evaluation Group of the World Bank,”  World Bank, Washington, D.C.
2. Blattman, C. (2011) “Impact Evaluation 3.0?” DFID Presentation. http://www.chrisblattman.com/documents/policy/2011.ImpactEvaluation3.DFID_talk.pdf
3. ADB. (1993) “Guidelines for the Economic Analysis of Projects.” http://beta.adb.org/documents/guidelines-economic-analysis-projects

About these ads

Written by Chris Prottas

September 29, 2011 at 12:03 am

Posted in measurement

5 Responses

Subscribe to comments with RSS.

  1. Chris, thanks for this. Lots of interesting stuff here. I guess the first question I have is what the relevant comparables are for an impact investment.

    For example, say someone makes an impact investment in a hospital chain – the counterfactual that you’re discussing seems to say that one would have either to figure out if there’s another comparable potential impact investment OR figure out what would have happened to that company without your investment capital.

    In reality, though, most of the comparables to the work we’re doing exist in the charitable space and have a long history (e.g. a public hospital). Isn’t that the starting point, and if so maybe we can connect this to Brian Trelstad’s work on best available charitable options (BACO).

    http://www.midot.org.il/Sites/midot/content/File/Trelstad%20-%20Simple%20Measures%20for%20Social%20Enterprise.pdf

    Put another way, and pulling the lens back a bit, the question this post seems to direct us towards is:
    a) with another form of financing from another source, would the enterprise have succeeded?
    b) is there another similar enterprise out there, and how do they do relative to the investee?

    …and I just wonder if that’s the right starting point – versus existing interventions that are not potential “impact investments” and understanding their impact and cost effectiveness.

    Sasha

    October 4, 2011 at 2:35 pm

  2. Thanks for your thoughts, Sasha! You raise an interesting issue in terms of the appopriate starting point, and as you noted, the BACO work is directly connected to this question.

    I think we are talking about two different questions with two different appropriate starting points.

    The first question is whether a for-profit business or charity can best achieve a social goal. In this case, the BACO analysis is quite relevant as is an analysis of the impact and cost-effectiveness “of existing interventions that are not potential “impact investments”.”

    A second question is what was the impact of your investment on that business? (A third question would be what was the impact of that business on the community, but better to leave that aside for clarity for now.)

    If the investment would have otherwise chosen to operate as a nonprofit, then I think a valid counterfactual would be (an estimate of) the organization-as-nonprofit’s performance (estimated based on a nonprofit comparator or by a forecast).

    If the investment would have been a for-profit with limited financing options then I think the a valid counterfactual would be the estimated performance of the for-profit-with-limiting-financing options.

    Acumen is in an interesting position, as a number of the other social investors focus more specifically on creating social gains through employment. For those social investors, it’s more likely that the poorly-financed-for-profit would be the appropriate counterfactual.

    That said, I think that second question has value for Acumen. While the first question speaks to whether a social problem can best be mitigated through charity or a for-profit, the second question speaks to how much incremental impact Acumen is having through its financing and/or technical assistance. For the Maturation/Selection/History reasons mentioned in the post above, without a counterfactual for the investment an investor will have a hard time understanding where it’s $$$ and assistance make the greatest impact, and how it can target industries, geographic areas, or deal terms in a way that will maximize that impact in the future.

    BACO provides insight into the cost-effective good that could be done by a for-profit vs. charity — which carries a lot of value. However, I don’t think it answers the question, how much good has the social investor done with an investment. If a goal is to build a market for social impact, then I think that’s a question that needs to be answered through comparison with a counterfactual.

    Thanks again and look forward to any further thoughts!

    Chris Prottas

    October 11, 2011 at 8:04 pm

  3. [...] Cost-effective Impact Assessments for the Impact Investor [...]

  4. [...] measurement and hidden costs. As I have addressed the former issue in the discussion note, “Cost-effective Impact Investments for the Impact Investor,” this note will explore the cost of impact investing, specifically the subsidy intrinsic in the [...]

  5. [...] next step is increasing the robustness of impact measurement through embracing the counterfactual (more here) and developing strong internal controls. This doesn’t mean turning Acumen into nothing more than [...]


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: