Characterising information gains and losses when collecting multiple epidemic model outputs

Abstract

Background. Collaborative comparisons and combinations of multiple epidemic models are used as policy-relevant evidence during epidemic outbreaks. Typically, each modeller summarises their own distribution of simulated trajectories using descriptive statistics at each modelled time step. We explored information losses compared to directly collecting a sample of the simulated trajectories, in terms of key epidemic quantities, ensemble uncertainty, and performance against data. Methods. We compared July 2022 projections from the European COVID-19 Scenario Modelling Hub. Using shared scenario assumptions, five modelling teams contributed up to 100 simulated trajectories projecting incidence in Belgium, the Netherlands, and Spain. First, we compared epidemic characteristics including incidence, peaks, and cumulative totals. Second, we drew a set of quantiles from the sampled trajectories for each model at each time step. We created an ensemble as the median across models at each quantile, and compared this to an ensemble of quantiles drawn from all available trajectories at each time step. Third, we compared each trajectory to between 4 and 29 weeks of observed data, using the mean absolute error to weight trajectories in consecutive ensembles. Results. We found that collecting models' simulated trajectories, as opposed to collecting models' quantiles at each time point, enabled us to show additional epidemic characteristics, a wider range of uncertainty, and performance against data. Sampled trajectories contained a right-skewed distribution which was poorly captured by an ensemble of models' quantile intervals. Ensembles weighted by predictive performance narrowed the range of plausible incidence over time, excluding some epidemic shapes altogether. Conclusions. Understanding potential information loss when collecting model projections can support the accuracy, reliability, and communication of collaborative infectious disease modelling efforts. The importance of different information losses may vary with each collaboration's aims, with lesser impact on short term predictions compared to assessing threshold risks and longer term uncertainty.