Bridging to Action Requires Mixed Methods, Not Only Randomised Control Trials

Abstract

Development evaluation refers to evaluating projects and programmes in development contexts. Some evaluations are too narrow. Narrow within-discipline impact evaluations are weaker than multidisciplinary, mixed-methods evaluations. A twostep process leads toward profoundly better arguments in assessing the impact of a development intervention. The first step is setting out the arena for discussion, including what the various entities are in the social, political, cultural and natural environment surrounding the chosen problem. The second step is that, once this arena has been declared, the project and triangulation of data can be brought to bear upon logical arguments with clear, transparent reasoning leading to a set of conclusions. In this second step, we do need scientific methods such as peer review, data and so on, but, crucially, the impact evaluation process must not rest upon a single data type, such as survey data. It is dangerous and undesirable to have the entire validity of the conclusions resting upon randomised control trials, or even a mixture of data types. Different contributions to knowledge exist within the evaluation process, including the interaction of people during action research, ethnography, case-study methods, process tracing and qualitative methods. The cement holding my argument together is that multiple logics are used (retroductive, deductive, and inductive, in particular). Deductive mathematics should not dominate the evaluation of an intervention, as randomised controlled trials on their own lend themselves to worrying fallacies about causality. I show this using Boolean fuzzy set logic. An indicator of high-quality development evaluation is the use of multiple logics in a transparent way. Keywords Evaluation • Randomised control trials • Comparative case-study research • Methodology • Retroduction • Mixed-methods • Impact evaluation Resume L'évaluation du développement se réfère à l'évaluation de projets et programmes dans le contexte du développement. Certaines évaluations sont trop restreintes. Les évaluations étroites, au sein de la même discipline, sont plus faibles que les évalua-Extended author information available on the last page of the article terventions en matière de développement, un procédé a deux étapes nous aide à bâtir des argumentations beaucoup plus fortes. La première étape est d'établir l'espace de la discussion, y compris les organismes sociales, politiques, culturales et naturels qui encerclent le problème choisi. La deuxième étape, une fois cet espace est défini, est d'utiliser le projet et la triangulation des données de forme logique, produisant un raisonnement transparent et clair qui nous amène à des conclusions. Cette deuxième étape nécessite de méthodologies scientifiques tels que la révision par des pairs, l'utilisation des données, etc. C'est crucial que le processus d'évaluation des impacts ne s'appuie pas que sur un seul type de données, comme par exemple les données d'enquête. Que l'ensemble de la validité des conclusions soit basé que sur des e...

A Realist Alternative to Randomised Control Trials: A Bridge Not a Barrier?

“…Olsen's point of departure is simple, why not adopt a methodology that works with (claimed) characteristics of social reality? For realistically posed development research one can look to a (non-foundational, "immanent critique" derived) philosophical realism that is compatible with social construction and, based on its principles, embraces action research (Olsen, 2019a). Situated social subjects produce, reproduce and transform social reality, subject to the potentials and limitations of the materiality their activity is embedded in and subject to norms and practices (beliefs, habits, ethics etc), according to social position (that may be class based, gendered, and affected by multiple other factors such as ethnicity, caste, religion etc), and subject to irreducible personhood (I and we are more than the sum of socialization or mere roles), all of which varies in place and through time.…”

“…This meta-ethic does not make agreement and progress neutral or simple. As such, it has similar strengths and weaknesses to Habermas's ideal speech -one either adopts a good faith approach or one does not; but any version of progress must seemingly begin from a meta-ethic of good faith and to some degree once one starts to think seriously about an objection, then the objection becomes self-cannibalizing, since no development researcher will position themselves as a carrier of bad-faith, and so cannot start from the assumption that good faith is impossible or that learning, in a position of dispute and difference is impossible (it is merely fallible within what Olsen terms structured or strategic pluralism, shaped by objectivity as open-mindedness, see also Olsen, 2019b). In any case, action research would make little sense without a meta-ethic of good faith.…”

“…Thereafter, a realist approach is highly attractive because it makes sense of and works to realise this good faith through the informed use of methods -a methodology. As a prominent and highly competent social statistician, Olsen is well placed to make this case -for example, the issue of "equifinality" as an issue of multiple pathways to a given outcome that RCTs can only express in one reductive way (see Olsen, 2012Olsen, , 2014also Freedman, 2010;Gillies, 2000).…”

“…Wendy Olsen's "Bridging to action" is a timely intervention that seeks to reorient development research (Olsen, 2019a). Olsen's position takes as its point of departure the growing schism between action research and the use of randomised control trials (RCTs) and specifically argues for a realist alternative approach as one way to underpin and operationalize development research.…”

A Commentary to ‘Bridging to Action Requires Mixed Methods, Not Only Randomised Control Trials’

“…Olsen (2019) argues that this transparency which includes registering protocols and pre-analysis plans prevents adaptive research, but this is not the intention nor the case-rather, researchers can update their plans with a good justification but at least there is a logged and transparent record that prevents adapting the research question to the data.…”