A critical analysis of multi-criteria models for the prioritisation of health threats

Abstract

Multi-criteria assessments are increasingly being employed in the prioritisation of health threats, supporting decision processes related to health risk management. The use of multi-criteria analysis in this context is welcome, as it facilitates the consideration of multiple impacts of health threats, it can encompass the use of expert judgment to complement and amalgamate the evidence available, and it permits the modelling of policy makers' priorities. However, these assessments often lack a clear multi-criteria conceptual framework, in terms of both axiomatic rigour and adequate procedures for preference modelling. Such assessments are ad hoc from a multi-criteria decision analysis perspective, despite the strong health expertise used in constructing these models. In this paper we critically examine some key assumptions and modelling choices made in these assessments, comparing them with the best practices of multi-attribute value analysis. Furthermore, we suggest a set of guidelines on how simulation studies might be employed to assess the impact of these modelling choices. We apply these guidelines to two relevant studies available in the health threat prioritisation domain. We identify severe variability in our simulations due to poor modelling choices, which could cause changes in the ranking of threats being assessed and thus lead to alternative policy recommendations than those suggested in their reports. Our results confirm the importance of carefully designing multi-criteria evaluation models for the prioritisation of health threats.

A Risk Analysis Framework for Prioritizing and Managing Biosecurity Threats

“…This method is not without its controversies, but has been suggested by Mangen et al (2010) for the prioritization of biosecurity threats. However, most of the models for the prioritization of biosecurity threats reported in the literature have employed multicriteria decision analysis as the valuation method (Montibeller et al, 2020), given its ability to consider hard-to-monetize impacts and represent the priorities of policymakers (Del Rio Vilas et al, 2011). Irrespective of how impacts are valued, risk analysts also need to identify or develop adequate attributes to assess these impacts (Keeney & Gregory, 2005), aligned with the fundamental objectives of policymakers and stakeholders (Keeney, 1992;McDaniels, 2019).…”

“…The former two approaches are usually focused on the risk analysis of a single threat. The prioritization of multiple threats, on the other hand, has usually been conceptualized as a multi‐impact problem (Mangen et al., 2010) with the extensive use of multicriteria decision analysis (Brookes et al., 2015; Montibeller, Patel, & Del Rio Vilas, 2020). Most of these models assume deterministic impacts employing different multicriteria methods such as: multiattribute value theory (e.g., the prioritization of exotic diseases for the pig industry in Australia by Brookes et al., 2014 and the prioritization of animal health threats to the United Kingdom by Del Rio Vilas, Voller, et al., 2013); the analytic hierarchy process (e.g., the stakeholder prioritization of zoonoses in Japan by Kadohira, Hill, Yoshizaki, Ota, & Yoshikawa, 2015 and the prioritization of zoonotic diseases in Kenya by Munyua et al., 2016); or score‐and‐weight models without decision analysis roots (e.g., the prioritization of communicable diseases in Germany by Balabanova et al., 2011 and the prioritization of diseases and zoonoses in Europe by Humblet et al., 2012).…”

“…Put another way, people will feel more confident in the analysis if they recognize that it was taken in an analytical, data‐driven manner. However, as already mentioned above, the emerging nature of biosecurity threats implies that, in some cases, hard evidence will have to be complemented with the use of expert judgment (Montibeller et al., 2020). Consequently, the risk analysts may raise with the organization the need to adjust ongoing evidence gathering routines to include all relevant evidence, including that based on expert judgment.…”

“…The risk analyst has therefore choices as to the level of sophistication required in a model, depending on the complexity of the prioritization problem, the time and resources available for the analysis, and the results that the client group demands. The development of simple, but theoretically correct, multicriteria models to prioritize health threats is a good example of an option for this design choice (Montibeller et al., 2020).…”

“…Another common metric are quality‐adjusted life years (QALYs), which may be employed to assess health impacts caused by possible outbreaks (Mangen et al., 2010). Other nonhealth‐related human and social impacts lack agreed indicators to assess disutility and may have to be defined in a case‐by‐case way (Montibeller et al., 2020).…”