Managing to Impact: One Acre Fund’s Performance Dashboard
Dave Algoso recently wrote up an overview of One Acre Fund’s performance dashboard that got me again thinking about how we can create dashboards to both support performance management and communicate value to donors and partners. I have a lot of respect for One Acre Fund’s work, it’s efficiency, impact, and, as Dave notes, emphasis on evidence-based program development. Certainly, from the outside, it appears that the bundled services have been well chosen to meet the needs of farmers while minimizing the subsidy required.
In some ways, their performance dashboard reflects their emphasis on efficiency and impact. For example, the sustainability metric reveals that 80% of program costs are covered by program fees. Even more strikingly, One Acre Fund establishes a counterfactual in order to assess the impact of their program services on farmer incomes. The fact that One Acre Fund can establish a counterfactual in order to assess it’s impact and still show such operational efficiency should give pause to the social enterprises and impact investors who claim counterfactuals are not feasible.
Below I’ll explore point-by-point some of the questions that emerged in my review of the dashboard, before tomorrow sharing my own One Acre Fund performance dashboard (based on the information available on their website and financial statements.)
Managing for Scale
The value of monitoring the number of members is clear, but I’m not clear why employees and districts are considered to be performance metrics. If taken seriously as performance targets, they actually create perverse incentives to add staff and to grow geographically. Presumably, the goal here is to reach more farmers; employees and districts are means to an end (individuals), and an end that can be measured directly. This is not to say that this information shouldn’t be tracked somewhere, but the display of employee and district figures as top-line performance metrics unnecessarily clouds the picture and distracts from the truly important metrics.
Managing for Sustainability
I love the clarity of the program fees / costs metric, and the additional emphasis on the “best site” performance. That said, the footnotes reveal that these sustainability metrics ignore a number of the organizational costs. With a large portion of the organizational costs omitted from the sustainability picture, viewers are left with a very incomplete picture of the organization’s sustainability and efficiency.
Managing for Impact
As I mentioned above, the integration of a control group really impressed me. However, it did raise some M&E housekeeping questions that I was unable to answer through the website. First, does the return on member investment take into account the cost paid by the member for the service? That is, if the member pays $50 for the service and increases their crop returns by $100, their net profit is $50. I assume that the cost of the services is included as an additional expense into the calculation, but it’s not clear.
Second, it’s worth noting that there may be bias in the estimates due to the selection process for inclusion in the program. An illustration:
You have a group of 10 farmers in 2010. Five have a good year, and five do not.
In 2011 you advertise your new program, hold a meeting and see who is interested. Three of the farmers who had a good year are not interested and don’t show up. One of the farmers who had a good year is interested, but he finds out he doesn’t qualify for the program because he is not sufficiently poor.
You now have a group of five new program members who had bad years, and one who had a good year. This group will serve as the counterfactual for the farmers you did work with. That is, you assume that only 1/6 of your members would have done well. However, in reality, the counterfactual is that half of the farmers would have done well, rather than the 1/6 in your sample. Your selection process, either through formal exclusion of those less poor or by less interest in the program by successful farmers, can thereby bias your results.
One option One Acre Fund may wish to explore is to integrate monitoring and evaluation with its expansion strategy. For example, if there are four districts that are currently being explored for future expansion, send M&E staff to the sites to survey the farmers in advance, and compare the changes in their income with that of the One Acre Fund farmers in the same period. It’s worth noting that there are also questions of whether the increased farm activity is crowding out non-farm income, for example. It would not be difficult to better clarify the impact of the program on the individuals, but it is not clear on the website.
Managing for Social Return on Investment
One Acre Fund has an intriguing business model, with a portion of its program costs covered by program fees paid by participants. Further, it’s impact and scale are both sizable. However, the organization’s performance in turning charitable dollars into returns for farmers is surprisingly absent from the picture. The pieces are here, outside a few outstanding questions (see below), but the value proposition is fragmented and incoherent. Tomorrow I’ll post a performance dashboard I created for One Acre Fund based on their publicly-available information. Until then, here are some of the key questions and/or reservations I have about the information available:
+ Do the member profits incorporate the cost of the program to the members?
+ Can we have faith in the counterfactual?
+ How can we estimate the sustainability of the impact in order to better capture the long-term effects? For example, what assumptions can we make about additional subsidies required or not required to continue this higher level of production, and how might we capture longer-term data for a subset of members? This would allow us to estimate not just the annual change in income, but any long-term income effects of that year’s program.