The limits of forecasting methods in anticipating rare events

Abstract

Publisher's copyright statement:Additional information: Use policyThe full-text may be used and/or reproduced, and given to third parties in any format or medium, without prior permission or charge, for personal research or study, educational, or not-for-prot purposes provided that:• a full bibliographic reference is made to the original source • a link is made to the metadata record in DRO • the full-text is not changed in any way The full-text must not be sold in any format or medium without the formal permission of the copyright holders.Please consult the full DRO policy for further details. The limits of forecasting methods in anticipating rare events AbstractIn this paper we review methods that aim to aid the anticipation of rare, high-impact, events. We evaluate these methods according to their ability to yield well-calibrated probabilities or point forecasts for such events. We first identify six factors that can lead to poor calibration and then examine how successful the methods are in mitigating these factors. We demonstrate that all the extant forecasting methods -including the use of expert judgment, statistical forecasting, Delphi and prediction markets -contain fundamental weaknesses. We contrast these methods with a nonforecasting method that is intended to aid planning for the future -scenario planning. We conclude that all the methods are problematic for aiding the anticipation of rare events and that the only remedies are to either (i) to provide protection for the organization against the occurrence of negatively-valenced events whilst allowing the organization to benefit from the occurrence of positively-valenced events, or (ii) to provide conditions to challenge one"s own thinking -and hence improve anticipation. We outline how use of components of devil"s advocacy and dialectical inquiry can be combined with Delphi and scenario planning to enhance anticipation of rare events.3 3

Forecasts or fortune-telling: When are expert judgements of safety risk valid?

“…This improved performance evaporates when experts are asked to provide estimates in situations where the answer is unknown until after the estimate is made (Goodwin & Wright, 2010). There are two mechanisms undermining the experts.…”

10.4.2 Convergence of Expert Opinion via the Wideband Delphi Method: An Application in Cost Estimation Models

“…There are some topics that do not bode well with this method, especially polarizing issues that challenge a person's values or situations that would be better suited for conflict resolution. Other topics such as rare events are also not suitable for Delphi techniques (Goodwin & Wright 2010). If the intent is to achieve convergence, the ultimate measure of effectiveness of a Wideband Delphi is whether there is a central tendency demonstrated by the group.…”

“…Numerous threats to validity come from cognitive biases in Delphi participants (Goodwin & Wright 2010). Although the Delphi method was developed to eliminate biases that emerge during group decision making, the facilitator must remain vigilant against any that do appear.…”

Using risk model judgements to better understand perceptions of synergistic risks

“…It has been proposed that the decomposition-recomposition approach may fail to improve probability judgements when the task is framed in a manner that is inappropriate to the judgement task or when it renders the task unfamiliar to the judge (Goodwin & Wright, 2010;Wright et al, 2009). …”