Chimeric forecasting: combining probabilistic predictions from computational models and human judgment

Abstract

Forecasts of the trajectory of an infectious agent can help guide public health decision making. A traditional approach to forecasting fits a computational model to structured data and generates a predictive distribution. However, human judgment has access to the same data as computational models plus experience, intuition, and subjective data. We propose a chimeric ensemble-a combination of computational and human judgment forecasts-as a novel approach to predicting the trajectory of an infectious agent. Each month from January, 2021 to June, 2021 we asked two generalist crowds, using the same criteria as the COVID-19 Forecast Hub, to submit a predictive distribution over incident cases and deaths at the US national level either two or three weeks into the future and combined these human judgment forecasts with forecasts from computational models submitted to the COVID-19 Forecasthub into a chimeric ensemble. We find a chimeric ensemble compared to an ensemble including only computational models improves predictions of incident cases and shows similar performance for predictions of incident deaths. A chimeric ensemble is a flexible, supportive public health tool and shows promising results for predictions of the spread of an infectious agent.

Early human judgment forecasts of human monkeypox, May 2022

“…As demonstrated during previous infectious disease outbreaks, crowdsourced human judgment can successfully estimate many different quantities that have epidemiological meaning or are of importance to public health decision makers. [3][4][5][6][7][8] During the 2014-15 and 2015-16 influenza seasons, crowdsourced forecasts of influenza-like illness were collected and found to be among the most accurate when compared to computational forecasts. 3 Beginning on Feb 18, 2020, just before WHO declared COVID-19 a pandemic, experts in infectious disease modelling were called upon to generate predictions of a diverse set of quantities related to COVID-19, including the number of weekly incident cases at the US national level, the cumulative number of deaths by the end of 2020, and counterfactual values (eg, the reported number of cases at the state level under two different potential policy changes).…”

“…5 Additional studies have found that human judgment and computational modelling might complement one another. [6][7][8] We collected 686 unique and revised predictions (358 unique and 328 revised) from May 19 to May 24, 2022, on the Metaculus human judgment crowd sourcing platform (methods, limitations, and TRIPOD reporting guidelines are in the appendix pp 15-19).…”

Utility of human judgment ensembles during times of pandemic uncertainty: A case study during the COVID-19 Omicron BA.1 wave in the USA

“…A more detailed review of techniques prior to COVID-19 can be found in [15]. During the COVID-19 pandemic, this crowd-sourced approach has been used to evaluate vaccine policies [16] and combined with computational models to produce hybrid/ensemble forecasts [17, 18]. Recently, for the emerging monkeypox outbreak, human judgment forecasts were used to estimate cases, deaths, and impact across Europe, US and Canada [19].…”