EMMIE and engineering: What works as evidence to improve decisions?

Abstract

While written by a proponent of realism, this article argues in favour of a pragmatic approach to evaluation. It argues that multiple sources of evidence collected using diverse research methods can be useful in conducting informative evaluations of programmes, practices and policies. It argues in particular that methods, even if their assumptions appear incommensurable with one another, should be chosen to meet the evidence needs of decision-makers. These evidence needs are captured in the acronym, EMMIE, which refers to Effect size, Mechanism, Moderator (or context), Implementation and Economic impact. Finally the article questions evidence hierarchies that are inspired by clinical trials, and suggests instead that, notwithstanding the clear differences in the physical and social worlds, engineering may provide a superior model for evaluators to try to emulate. And engineering is, above all, a pragmatic field.

Applying Evidential Pluralism to the social sciences

“…) Moreover, one of the What Works centres has begun to recognise the importance of mechanisms. The What Works Centre for Crime Reduction has developed the 'EMMIE' framework for systematic reviews of evidence: Effect size, Mechanism, Moderator, Implementation and Economics are all components of an evaluation (Johnson et al, 2015;Tilley, 2016;Thornton et al, 2019). From the point of view of Evidential Pluralism, considering mechanisms is an important step in the right direction.…”

Development of a New Business Model to Measure Organizational and Project-Level SDG Impact—Case Study of a Water Utility Company

“…www.mdpi.com/journal/sustainability  Context: the conditions in a context of action encompass "material resources and social structures, including the conventions, rules and systems of meaning in terms of which reasons are formulated" [69].  Mechanism: the underlying entities, processes or structures that operate in particular contexts to generate outcomes of interest [70].  Outcome: the practical effects produced by causal mechanisms being triggered in a given context [71].…”

“…Outcome: the practical effects produced by causal mechanisms being triggered in a given context [70].…”

Extending the remit of evidence-based policing

“…being as much about the biological, physical and computer sciences as it is about social science. Rather than a medical model, crime science allies itself to an engineering approach (Tilley, 2016) as being both practical and iteratively working towards solutions.…”