Accuracy of US CDC COVID-19 Forecasting Models

Abstract

Accurate predictive modeling of pandemics is essential for optimally distributing resources and setting policy. Dozens of case predictions models have been proposed but their accuracy over time and by model type remains unclear. In this study, we analyze all US CDC COVID-19 forecasting models, by first categorizing them and then calculating their mean absolute percent error, both wave-wise and on the complete timeline. We compare their estimates to government-reported case numbers, one another, as well as two baseline models wherein case counts remain static or follow a simple linear trend. The comparison reveals that more than one-third of models fail to outperform a simple static case baseline and two-thirds fail to outperform a simple linear trend forecast. A wave-by-wave comparison of models revealed that no overall modeling approach was superior to others, including ensemble models, and error in modeling has increased over time during the pandemic. This study raises concerns about hosting these models on official public platforms of health organizations including the US-CDC which risks giving them an official imprimatur and further raising concerns if utilized to formulate policy. By offering a universal evaluation method for pandemic forecasting models, we expect this work to serve as the starting point towards the development of more sophisticated models.

Forecasting drug-overdose mortality by age in the United States at the national and county levels

“…Since they rely on the most recent available observational data, model-free recency-type heuristics, such as recency heuristic I and II usually perform well when forecasting over short time horizons. Previous studies have demonstrated that recency heuristic I can provide more accurate forecasts than Google Flu Trends ( 101 ), and that recency heuristic II can compete favorably with CDC and ECDC COVID-19 ensemble forecasting models ( 102 , 104 ). However, simple data-driven heuristics do not typically yield reliable long-term forecasts, as they lack the ability to account for underlying population-level dynamics.…”

Accuracy of COVID-19 evolution models for different forecast horizons

“…A number of works demonstrated the effectiveness of neural network models based on Long Short-Term Memory layers (LSTM) [1,6,8]. A recent study [9] showed that the effectiveness of the existing methods applied to the COVID-19 dynamics forecasting is comparable to the estimates based on the "tomorrow as today" method, which indicates the complexity of the task and the need for further improvement of predictive methods. In our previous works [5,7] it was shown that: 1) the accuracy of machine learning models strongly depends on the number of training samples, 2) the usage of the models of this type is inefficient on short prediction horizons (up to 28 days).…”

Wastewater surveillance beyond COVID-19: a ranking system for communicable disease testing in the tri-county Detroit area, Michigan, USA

“…Some infectious diseases are transmitted through "bites" from insect vectors, while others can be caused by ingesting contaminated food or water (who.int). The WHO, U.S. NIH, U.S. AID, U.S. CDC, and the international scientific community has long recognized the need to develop a comprehensive education, prediction, and prevention system for CDs (13,26,27).…”