Interval forecasts of weekly incident and cumulative COVID-19 mortality in the United States: A comparison of combining methods

Abstract

Background A combined forecast from multiple models is typically more accurate than an individual forecast, but there are few examples of studies of combining in infectious disease forecasting. We investigated the accuracy of different ways of combining interval forecasts of weekly incident and cumulative coronavirus disease-2019 (COVID-19) mortality. Methods We considered weekly interval forecasts, for 1- to 4-week prediction horizons, with out-of-sample periods of approximately 18 months ending on 8 January 2022, for multiple locations in the United States, using data from the COVID-19 Forecast Hub. Our comparison involved simple and more complex combining methods, including methods that involve trimming outliers or performance-based weights. Prediction accuracy was evaluated using interval scores, weighted interval scores, skill scores, ranks, and reliability diagrams. Results The weighted inverse score and median combining methods performed best for forecasts of incident deaths. Overall, the leading inverse score method was 12% better than the mean benchmark method in forecasting the 95% interval and, considering all interval forecasts, the median was 7% better than the mean. Overall, the median was the most accurate method for forecasts of cumulative deaths. Compared to the mean, the median’s accuracy was 65% better in forecasting the 95% interval, and 43% better considering all interval forecasts. For all combining methods except the median, combining forecasts from only compartmental models produced better forecasts than combining forecasts from all models. Conclusions Combining forecasts can improve the contribution of probabilistic forecasting to health policy decision making during epidemics. The relative performance of combining methods depends on the extent of outliers and the type of models in the combination. The median combination has the advantage of being robust to outlying forecasts. Our results support the Hub’s use of the median and we recommend further investigation into the use of weighted methods.

An ensemble n-sub-epidemic modeling framework for short-term forecasting epidemic trajectories: Application to the COVID-19 pandemic in the USA

“…It will be important to compare the approach with competing methods on other datasets and infectious diseases in the future. Our analysis relied on daily time series data of COVID-19 deaths in the USA, which is inherently noisy due to heterogeneous data reporting at fine spatial scales (i.e., county-level) [ 39 ]. Noisy data complicate the ability of any mathematical model to identify meaningful signals about the impact of transmission dynamics and control interventions.…”

Conventional and frugal methods of estimating COVID-19-related excess deaths and undercount factors

“…Our results reinforce the strength of using multi-method approaches to triangulate the true extent of the impact of the COVID-19 pandemic. By combining conventional and novel frugal methods of estimating pandemic-associated excess mortality in a multi-method approach, we minimized the pitfalls of relying on any particular individual method 86 , 95 – 98 , 117 , 118 . Our findings can have important implications, especially in resource-constrained settings, where robust and resilient data infrastructures tend to be lacking or limited, and in contexts where considerable debate exists about the underlying ground truth 1 – 3 , 12 , 19 , 26 – 28 , 99 – 103 , 119 .…”

“…Finally, we examined the effectiveness of another frugal method—the wisdom of crowds approach—to estimate COVID-19-related excess mortality. Although this approach has been widely used across multiple real-world domains before, including during the COVID-19 pandemic 37 – 86 , 126 , 127 , to our knowledge, this frugal method has not yet been used to estimate COVID-19-related excess mortality. Therefore, our study provides a novel confirmation of the potential of the wisdom of crowds approach as a complementary tool of frugal fact-finding.…”

“…This frugal method has been widely used across multiple domains including neuropsychology, business, finance, economics, election polling, public policy, and global geopolitics 37 , 39 , 59 , 67 – 77 . Specifically, in the context of epidemiology, public health surveillance, and the COVID-19 pandemic, this method has been used to predict future outbreaks, vaccination uptake, disease caseload, infection hotspots, and overall disease severity 72 , 78 – 86 . Despite the potential of cognitive estimation, the utility of this method has not been widely tested to estimate pandemic-associated excess mortality, a gap this study aims to fill.…”

An ensemble n-sub-epidemic modeling framework for short-term forecasting epidemic trajectories: Application to the COVID-19 pandemic in the USA

“…Our study is not exempt of limitations. Our analysis relied on daily time series data of COVID-19 deaths in the USA, which is inherently noisy due to heterogeneous data reporting at fine spatial scales (i.e., county-level) [62]. Noisy data complicate the ability of any mathematical model to identify meaningful signals about the impact of transmission dynamics and control interventions.…”