The Principled Agent

Thoughts on development economics and impact measurement

Archive for the ‘measurement’ Category

What can “Pay for Success” learn from health care?

leave a comment »

Linking payment to performance on pre-defined outcomes makes a lot of sense. And in the social sector, where payment has historically been linked to simply providing services, it’s easy to understand the excitement for Social Impact Bonds and “Pay for Success” contracts. You get what you pay for, the thinking goes, and a shift from outputs to outcomes could be transformative.

Yet it is worth reflecting on the experience of those who have come this way before us, and just as the social sector has followed the health care industry’s lead on randomized controlled trials, so too the recent enthusiasm for “Pay for Success” follows the health care sector’s embrace of “pay for performance” (P4P) and “value-based purchasing”. Read the rest of this entry »

Written by Chris Prottas

January 14, 2013 at 9:01 pm

Posted in measurement

Acumen and the impact learning agenda

with one comment

While progress has been made with outcome reporting standards, e.g., IRIS, much work remains to establish impact measurement practices that will allow Acumen staff to credibly assess the effectiveness of different products and services. Once developed, these practices will support both greater internal learning and external understanding of Acumen’s social return on investment.

Last February, ANDE’s Randall Kempner wrote a blog post very much related to this subject that likely caused some pulses to quicken:

Organizations join ANDE because they share the common mission of increasing the prosperity of poor people in emerging markets, believing that supporting entrepreneurship and market-based solutions is part of a sustainable approach to poverty reduction. … But that’s a hypothesis; there isn’t sufficient evidence to back the assertion that small and growing businesses are a key part of poverty reduction. That’s why I was scared… I was standing there thinking “I don’t want to be sitting in a room like this 10 years from now, hearing how my industry, my movement, my life’s work had ‘an average impact on poverty of zero.’”

ANDE is collaborating with researchers to improve the robustness of the existing evidence base on the impact of small and growing businesses on poverty. But what does this mean for Acumen? Acumen now finds itself in the same position as impact-driven MFIs in the 90s. The additional social value of the products and services supplied by Acumen portfolio companies accords the organization a competitive advantage, but the less rigorous and robust the impact measurement, the more difficult it is to communicate this added value. Indeed without more robust impact measurement, the impact component of impact investing will be treated as a commodity. The story of microfinance offers a clear example, as institutions with marginal impact were perceived to be the same as those with significant impact. The impact-driven MFIs benefited less than they should have in the short term, while then sharing fully in the bust prompted by their less noble counterparts.

Outside of external communication, Acumen also has much to gain from impact measurement with regards to refining portfolio strategy and service offerings.

As Kempner noted in his piece, impact investing is a nascent field and there is not yet sufficient evidence to validate its theory of change. Given the heterogeneity of the impact investing field, there will never be a study that will show that it does or does not reduce poverty. Every country context, every industry, every combination of capital, assistance, and facilitation will ultimately result in a “new” program with a different set of causal mechanisms. There will likely be some combinations that result in successful enterprises with limited impact, and others that produce less successful enterprises with more impact. Acumen already rightly understands that the point of entry in an enterprise’s lifecycle is one such crucial factor. The Head of Impact will need to help systematize and deepen this learning to better understand and validate the mechanisms by which Acumen is achieving impact. While monitoring and evaluating alone won’t lead to dramatic changes overnight, in time, the work produced by the Head of Impact should undoubtedly shape Acumen’s portfolio strategy.

So what does this mean in practical terms? Acumen’s already worked to help establish a common IRIS lexicon. The next step is increasing the robustness of impact measurement through embracing the counterfactual (more here) and developing strong internal controls. This doesn’t mean turning Acumen into nothing more than a research lab for RCT advocates. It does mean taking concrete, transparent steps toward estimating the incremental impact catalyzed by Acumen’s work, by creatively utilizing the myriad techniques available that require less in additional resources than they do a change in mindset. And yes, it means engaging with the empirical researchers in order to ensure the research agenda of initiatives like IPA’s SME initiative reflect the research interests of Acumen. It means advocating for small steps that will result in stronger learning and understanding of what works and developing processes and tools to help make this vision a reality.

This undertaking will require an academic’s sense of rigorous impact estimation, a cutting-edge NGO’s creativity in devising cost-effective methods, a market researcher’s sense of turning data into useful managerial information, and a controller’s insistence on sound internal controls to maintain the needed integrity and objectivity of the results. The high-impact MFIs of the 90s lacked this combination of skills and saw their stock and reputation vacillate and ultimately fall with that of low-impact MFIs. Acumen is well-positioned to avoid this fate. By developing the infrastructure and processes needed to ensure it is successfully managing to impact, Acumen can at the same time set the bar for other impact investors, supporting the development of a true market for impact and catalyzing the philanthropic support needed for the impact investing field’s Blueprint to Scale.

Relevant posts:

Written by Chris Prottas

April 1, 2012 at 2:42 am

Posted in measurement

Case study: Assessing the cost-effectiveness of Impact Investing

with one comment

The cost-effectiveness of impact investing is unclear due to both poor impact measurement and hidden costs. As I have addressed the former issue in the discussion note, “Cost-effective Impact Investments for the Impact Investor,” this note will explore the cost of impact investing, specifically the subsidy intrinsic in the investments.

Impact investments are generally a hybrid of equity and/or debt capital and subsidy, and likewise have two types of returns, financial and social. To understand the cost-effectiveness of impact investing it’s necessary to not only better quantify the social returns, but also isolate the subsidy. Only after clarifying the subsidy component of the investments can one assess the financial and social performance of the two components of the investments. Read the rest of this entry »

Written by Chris Prottas

February 7, 2012 at 7:17 pm

Cost-effective Impact Assessments for the Impact Investor

with 5 comments

This long post is available as a PDF.

Impact investors can better estimate their impact on growing businesses and, in turn, the impact of supported businesses on communities without a significant increase in evaluation cost. While impact investors rightly regard randomized controlled trials (RCTs) as inappropriate for day-to-day performance measurement, impact investors can use low-cost alternatives (specifically, control groups and Best Alternative Pro Formas) to construct counterfactuals in order to estimate what would have occured in the absence of their investment. While these methods will not meet the standard of RCTs, they can offer a significant and meaningful improvement in the measurement and management of social performance. Read the rest of this entry »

Written by Chris Prottas

September 29, 2011 at 12:03 am

Posted in measurement

Separating the Gold from Pyrite: Analytical Rigor and the Potential of Impact Investing

with 4 comments

This post addresses two questions raised in the recent discussion on impact evaluation and investing: to what extent is the burden of proof lower for impact investing than traditional philanthropy, and what type of analysis does the industry itself need to separate social enterprises with goods intentions and significant social impact from those that struggle to translate the former to the latter.

Acumen Fund’s Sasha Dichter recently responded to Dean Karlan’s critique of the lack of rigorous analysis of social investments by differentiating between the social investment and the “purely philanthropic intervention”:

No matter what scale a pure philanthropic intervention reaches, the total marginal cost of delivering the nth “thing” (any intervention) is always positive, so you’re in the business of figuring out how the impact relates to that cost and how the impact relates to other similar interventions.  Not so if you find the “next cellphone” – except it’s not a cellphone, it’s safe drinking water or a bio-mass powered light on a mini-grid or a safe and affordable place for a mother to give birth. Read the rest of this entry »

Written by Chris Prottas

August 18, 2011 at 2:18 am

Posted in measurement

Measuring Impact in CSR, Impact Investing

with one comment

The Corporate Social Responsibility (CSR) and Impact Investing movements have each developed reporting guidelines and impact reports to measure the positive social impact of business activities. On the corporate side, there are GRI’s Sustainable Reporting guidelines, and IFC/World Business Council for Sustainable Development’s “Measuring Impact: Understanding the business contribution to society.” Social investors have created their own assessment tools and come together to develop industry standards with the Global Impact Investment Rating System. Read the rest of this entry »

Written by Chris Prottas

April 4, 2011 at 7:25 am

Posted in measurement

Tagged with ,