Evaluating unintended consequences: New insights into solving practical, ethical and political challenges of evaluation

Abstract

Evaluating complex interventions and policies is challenging. This is particularly true for the identification of unintended consequences, whether negative or positive. This paper uses data from a workshop with policymakers and evaluators to explore the evaluation of unintended consequences. We identify three main challenges for policymakers and evaluators: being able to identify and evaluate unintended effects, to avoid creating unintended effects, and being able to explain these effects. We discuss practical, political and ethical issues for each of these challenges, and identify recommendations for evaluators who want to consider unintended consequences. Firstly, use a broader range of methods to explore how policies play out. Secondly, use theory to plan evaluations, and thirdly discuss both methods and theory with relevant stakeholders to make these as useful as possible. We offer novel insights into recent debates about theory-led and co-produced interventions and policies. L'évaluation des interventions et politiques complexes est délicat, particulièrement en ce qui concerne l'identification des effets imprévus, qu'ils soient négatifs ou positifs. Cet article utilise des données provenant d'un colloque avec des décideurs politiques et les valuateurs sur l'évaluation des effets imprévus. Nous identifions trois défis principaux pour les décideurs politiques et les évaluateurs: identifier et évaluer les effets imprévus, les éviter, et les expliquer. Nous examinons les questions pratiques, politiques et éthiques pour chaque défi, et proposons des recommandations pour les évaluateurs qui souhaitent considérer les effets imprévus. Premièrement, utiliser une plus grande variété de méthodes pour examiner les effets des politiques; deuxièmement, utiliser la théorie pour planifier l'évaluation; et troisièmement, discuter les méthodes et la théorie avec les parties prenantes pour les rendre aussi utiles que possible. Nous offrons de nouvelles perspectives informant des débats récents sur les interventions et politiques coproduites fondées sur la théorie.

How Times of Crisis Serve as a Catalyst for Creative Action: An Agentic Perspective

“…Regardless of the scale, however, taking creative action is no guarantee that doing so will result in creative or even beneficial outcomes. Indeed, as illustrated in the model and discussed below, even creative actions that lead to creative outcomes may have unintended secondary outcomes or even negative side-effects ( Merton, 1936 ; Baert, 1991 ; Peterson and Skolits, 2019 ; Oliver et al, 2020 ). Another possible pathway, illustrated in the model, asserts that if creative action does not lead to what is judged to be a creative outcome and people are willing to continue to take creative risks of trying new things, then it may eventually result in creative outcomes.…”

“…Moreover, the primary outcomes can, over time and in different contexts, result in secondary outcomes, which also can range from potentially beneficial to potentially problematic. Specifying the difference between action and outcome highlights the need for conceptualizing and evaluating creative outcomes separately from creative actions, including the recognition that different people and groups may evaluate outcomes differently based on their unique contextual, socio-cultural, and historical vantage point ( Baert, 1991 ; Peterson and Skolits, 2019 ; Oliver et al, 2020 ).…”

“…As mentioned, creativity researchers have increasingly stressed the importance of recognizing that creative action and inaction can result in negative outcomes ( Cropley et al, 2010 ) and therefore it is important to engage in creative actions from a more principled and ethical stance ( Moran, 2016 ). This requires an ongoing effort to anticipate, monitor, identify, and address potentially harmful secondary outcomes ( Peterson and Skolits, 2019 ; Oliver et al, 2020 ), which can emerge on an elongated time frame, when taking creative and deferential actions aimed at navigating and resolving the crisis.…”

Transforming evidence for policy and practice: creating space for new conversations

“…How does this compare with, for example, random research funding (Shepherd et al, 2018)? Stakeholder engagement is said to be required for impact, yet we know it is costly and time-consuming (Oliver et al, , 2019a. How can universities and funders support researchers and users to work together long-term, with career progression and performance management untethered from simplistic (or perhaps any) metrics of impact?…”

“…Ultimately, methods must be found to evaluate the impact of evidence on policy and practice change, and on populations-including unintended or unwanted consequences (Lorenc and Oliver, 2013;Oliver et al, , 2019a. Some have argued that the primary role for researchers is to demonstrate the consequences of decisions and to enable debate.…”