Accounting and Auditing for Social Impact
The past few years have seen the emergence of a general consensus that social initiatives that claim to improve the lives of others should measure their impact. There has been a successful (if incomplete) push for more randomization by individuals like Esther Duflo and organizations like Innovations for Poverty Action. At the same time, the impact investing community has developed a Global Impact Investment Ratings System and Impact Reporting & Investment Standards to create a common language to speak about social impact.
Despite the increased attention paid to measuring social impact, there are crucial monitoring and evaluation (M&E) issues that are currently unaddressed, and promise to undermine all attempts to promote honest and accurate M&E reporting.
Michael Clemens and Gabriel Demombynes recently critiqued the Millennium Village staff for failing to take relatively cheap and straightforward steps to improve the rigor of their evaluation, as well as taking liberties with the reporting of their results (MV response). The critiques range from the subjective process of site selection (which could introduce bias) to the misrepresentation of descriptive time-series results as “effects” (and, more cringe-inducing, “quick wins” ) of the program.
More generally, there are thousands of decisions made in the processes of site selection, sample size, time horizon selection, output/outcome selection, survey design, data collection, data cleaning, data reporting, etc. It is not reasonable to assume that every constituent or prospective social investor will develop the expertise required to judge the rigor and honesty of every social initiative’s M&E study design, processes, and reporting.
It is quite likely, however, that social enterprises (broadly defined) will find it in their interest to “round up” in each one of these decisions, compromising the integrity of the entire process. Some of the more common problems are addressed in the CONSORT Transparent Report of Trials and CDC’s Transparent Reporting of Evaluations with Nonrandomized Designs (TREND) guidance (the latter referenced by Clemens and Demombynes). Few if any of these issues are addressed in the international development community’s push for more rigorous M&E.
Surely no single M&E strategy can apply to all interventions (just as no financial accounting rules would apply to all firms), but principles of procedural disclosure and assessments of the relative significance of the results certainly can. Industry-wide standards for reporting on study design and implementation, along with peer review and formal auditing, is the first step toward a world where investors and constituents, with limited time and expertise, can choose to fund enterprises that take M&E more or less seriously, and have more or less evidence that their programming works.
Until that time, initiatives like the MV will not choose their control group until after the study has begun, and will release mid-term reports that omit results (e.g., Kenya results are omitted multiple times). Initiatives like the MV will release mid-term reports right before UN summits that present descriptive results as “effects” and “quick wins”, while choosing to release more rigorous data months after the summit — the same UN summit at which Nestle handed over a sum of money to the MV project.
It’s worth mentioning that MV staff responded to criticism of their mid-term report in part by explaining that they are going to release more rigorous analysis in the future. The problem is that there is no distinction between M&E reports and marketing materials, for MV or anyone else. As long as there is no sharp division between M&E and marketing, M&E will be subordinated to the marketing strategy of organizations interested in attracting greater funding.
Take the MV project as an example. When the MV project had just started, Jeffrey Sachs “called for foreign aid to Kenya to increase fifteen-fold in order to provide $1.5 billion per year for Millennium Village interventions in that country alone,” and “before any evaluation had been published, MVP (2008b) applauded plans to expand the MVP in several countries and concluded, “The MVP has thereforecreated a powerful pressure to expand as a result of its notable successes”” (cited in Clemens and Demombynes). Does this sound like an organization interested in evaluating a poverty alleviation strategy or finding facts to support their fundraising?
To conclude, for all the emphasis on increased accountability in the social sector, there has been little of talk of accounting. If the MV staff believe that descriptive time-series data justifies a funding increase at the UN Summit, that’s their prerogative. But do their constituencies know what their M&E reports mean? Did Nestle? Do social investors?
The challenge is to create a social accounting system that accurately and consistently records, classifies, and summarizes (to borrow from the AICPA) M&E processes and results for non-experts so that constituents and social investors can make informed decisions. This requires a) full disclosure of study design, implementation processes, confidence levels, b) some process of auditing and/or certification to ensure compliance, and c) synthesis of major categories of M&E rigor and results for mass consumption.
I’ll finish with a disclaimer: I am in no way stating that funds should only flow to development projects with 15 years of proven impact in 10 randomized trials. My emphasis is on full awareness of the relative robustness and accuracy of M&E processes and results, and creating incentives to do M&E properly.