The Triviality of Measuring Ultimate Outcomes: Acknowledging the Span of Direct Influence

Abstract

Sustainability standards and certification schemes have been promoted as a market-driven instrument for realising development impacts and receive public funding. As a result, companies, NGOs and supporting donors and governments want to know if these ambitions have been fulfilled. Their tendency is to commission household surveys to assess net effects of certification in areas such as poverty, productivity and food security. This article argues that, rather than trying to measure precise net effects on farmer income, the focus should be on detailed measurement of more immediate outcomes in terms of knowledge and implementation of good agricultural practices. Contribution analysis is proposed as an overall approach to verify the theory of change, combining survey-based net-effect measurement of these immediate and intermediate outcomes with less precise, lean monitoring of indicators to verify the contributory role of these outcomes that are outside the span of direct influence, such as household income and poverty alleviation.

Effects of certification schemes for agricultural production on socio-economic outcomes in low- and middle-income countries: A systematic review

“…Wherever the data allow to, this systematic review reports on both intermediate and endpoint outcomes, since many CS are primarily focused on and interested in these intermediate outcomes, which may often be only one of many contributors to the ultimate or endpoint outcomes (Ton et al, 2014 Fourth, while the ToC of most CS are explicit about the expected positive outcomes, there seems to be a gap in understanding unintended outcomes, whether negative or positive, and the circumstances in which these arise. Fifth, there are distributional dynamics and contextual aspects that can shape the effects of CS, with possibly uneven distribution of benefits and costs among participants.…”

Towards a complexity-aware theory of change for participatory research programs working within agricultural innovation systems

“…While this impact pathway applies to research carried out within existing innovation trajectories (Ekboir, 2003), for example plant breeding and maintaining the yield potential of modern crop varieties, the overwhelming focus on it has obscured other ways in which agricultural research and innovation efforts are producing development impact. Complexityaware programs are particularly disadvantaged by the adoption impact pathway narrative because they are not attempting to manage towards predicable outcomes within existing innovation trajectories, but rather to provoke and then harness beneficial system interactions and dynamics (Douthwaite et al 2003;Arkesteijn et al 2015;Ton et al 2014) in the process of catalyzing and supporting new ones. These programs therefore cannot easily forecast their impacts ex ante and may also produce unexpected impacts which are not included in the adoption impact pathway and which can therefore remain invisible to evaluators, donors, and organizational decision-makers.…”

Contribution Analysis: Theoretical and Practical Challenges and Prospects for Evaluators

“…Bayesian inference relies on the estimation of probabilities based on prior probabilities and new evidence, which is diff erent from traditional methods of statistical inference based on relative frequencies ( Befani & Mayne, 2014 ;International Society for Bayesian Analysis, 2009 ). Th e triangulation of methods and evidence is considered important in CA when trying to verify the theory of change and discount alternative explanations and the role of infl uencing factors in observed results ( Nakrošis, 2014 ;Noltze, Gaisbauer, Schwedersky, & Krapp, 2014 ;Rotem, 2008 ;Ton, Vellema, & Ge, 2014 ). Delahais & Toulemonde (2012) go one step further and suggest that evidence triangulation takes place at each level of judged reasoning, not just for each overarching evaluation question or even the whole theory of change.…”