Reflections On the Use of Complexity-Appropriate Computational Modeling for Public Policy Evaluation in the UK

Abstract

In the UK, calls for the application of insights from the study of complex adaptive systems to public policy evaluation are beginning to be taken seriously in government. Policymakers and analysts are accepting the fallibility of overly simplistic, definitive, or linear analysis, or are finding traditional forms of analysis and evidence less appropriate or feasible. Through our work in CECAN (the Centre for the Evaluation of Complexity Across the Nexus), we reflect on our experiences and the practical challenges of using complexity-appropriate computational modeling with policy ana

How does the commissioning process hinder the uptake of complexity-appropriate evaluation?

“…believing that being complexity-appropriate is more expensive, or creates room for uncertainty to be used as a defence by evaluators for lack of rigour, precision, or clear recommendations) may be a barrier in some cases. Barbrook-Johnson et al (2019) more fundamentally suggest profound inflexibility, conventionality, and inertia in the research and evaluation commissioning process undermines potential for complexity-appropriate evaluation. None of these studies pay explicit attention to the rationale for the features of the commissioning processes that create the barriers and how these confer benefits to the organisations commissioning the evaluation (e.g.…”

COMPLEX-IT: A Case-Based Modelling and Scenario Simulation Platform for Social Inquiry

“…(3) Reuse potential Case based modelling and scenario simulations embodied in COMPLEX-IT, inquiry keeping in mind its strengths and limitations, hold great potential for several groups looking to engage in social inquiry. Most prominently, applied researchers and analysts in areas such as healthcare, education, public infrastructure, social services, and policy and program evaluation must confront understanding and affecting open systems e.g, see [22], which is supported through identifying different complex systems, trajectories when time-series data is available, as well as the impact of various interventions into the modelled system. Social scientists interested or already involved in the study of complex data/systems likewise can leverage COMPLEX-IT to expand our fundamental understanding of these systems.…”

Cased-based modelling and scenario simulation for ex-post evaluation

“…COMPLEX-IT, in particular, offers a tool that enables evaluators to quickly engage in case-based modelling and scenario simulation regardless of prior experience. This is especially timely in light of the ongoing challenges evaluators face including limited-time to try new methods, work-contexts that are often less supportive of new methods, and evaluators perceptions of required data, that may then constrain their use of new methods (Barbrook-Johnson et al, 2019); COMPLEX-IT’s increased accessibility can help alleviate the first two barriers, and its support for exploration and iterative analysis can help with the third barrier.…”

“…Hence, the purpose of the current paper. We have been working, over a series of papers, to develop a new mixed-methods approach for thinking in complex systems terms about interventions in complex systems (Barbrook-Johnson et al, 2019; Castellani et al, 2019; Schimpf and Castellani, 2020). This approach is based, in part, on our integration of two key methodologies: scenario analysis (Morell, 2005; Schwartz, 1991) and case-based methods (CBMs; Byrne and Ragin, 2009), specifically case-based computational modelling.…”

“…Case-based computational modelling is particularly useful because it leverages advances in computational social science (e.g. machine learning, cluster analysis, data visualization, data forecasting) to engage in a case-based approach to policy evaluation (Barbrook-Johnson et al, 2019; Castellani et al, 2019; Schimpf and Castellani, 2020).…”