Evaluation of individual and ensemble probabilistic forecasts of COVID-19 mortality in the United States

Abstract

Significance This paper compares the probabilistic accuracy of short-term forecasts of reported deaths due to COVID-19 during the first year and a half of the pandemic in the United States. Results show high variation in accuracy between and within stand-alone models and more consistent accuracy from an ensemble model that combined forecasts from all eligible models. This demonstrates that an ensemble model provided a reliable and comparatively accurate means of forecasting deaths during the COVID-19 pandemic that exceeded the performance of all of the models that contributed to it. This work strengthens the evidence base for synthesizing multiple models to support public-health action.

Enhancing long-term forecasting: Learning from COVID-19 models

“…The data for the first study come from the CDC collection of COVID-19 forecasts. In April 2020, and with the growing need to forecast COVID-19 trajectories in the United States, the CDC partnered with a research laboratory at the University of Massachusetts Amherst to create a forecast hub that collects and synthesizes COVID-19 trajectory predictions [ 20 ]. As of May 2021, more than 70 modeling groups from academic institutions, research laboratories, and the private sector had contributed by submitting their simulation-based trajectory predictions for various locations in the United States (typically at the state and national levels, but also some county-level predictions).…”

“…the flexibility of likelihood functions used in estimation). While it is feasible to combine point predictions and prediction intervals using proper scoring rules [ 22 ], we maintain our focus distinctly on drivers of point prediction accuracy and omit a comparison of prediction intervals (e.g., see [ 20 , 21 ] for comparing those intervals). We collected all the state- and national-level forecasts we could find on the project website.…”

“…The performance of a median ensemble (of the 61 models of the CDC hub plus the SEIRb set) is also plotted for comparison. Median ensemble is a promising approach to improve predictions: by incorporating information from all predictions from a group of models, while excluding outliers, it significantly improves predictions over any typical model [ 20 , 31 , 32 ], and thus it is a tough standard to beat. SEIRb outperforms the median ensemble.…”

“…We focus on 490,210 point-forecasts for weekly death incidents (across 57 locations); forecast dates over the span of a year (4/13/2020 to 3/29/2021); 20 forecast horizons (1-week-ahead predictions to 20-week-ahead predictions); and 61 models (not every model offers a forecast for every location, forecast date, or horizon). Prior research has used these data to assess the quality of probabilistic forecasts from various models [ 20 , 21 ]. We build on this work by examining the association between the modeling approach and point prediction quality using hand-coded architectural features of forecasting models.…”

An Evaluation of Prospective COVID-19 Modeling: From Data to Science Translation

“…Efforts to date have revealed some of the challenges with predicting the course of the COVID-19 pandemic 1,2 . Critically, individual risk reduction behaviors and policy compliance, which directly impact case growth, are not easily measured.…”