Evaluation of an open forecasting challenge to assess skill of West Nile virus neuroinvasive disease prediction

Abstract

Background West Nile virus (WNV) is the leading cause of mosquito-borne illness in the continental USA. WNV occurrence has high spatiotemporal variation, and current approaches to targeted control of the virus are limited, making forecasting a public health priority. However, little research has been done to compare strengths and weaknesses of WNV disease forecasting approaches on the national scale. We used forecasts submitted to the 2020 WNV Forecasting Challenge, an open challenge organized by the Centers for Disease Control and Prevention, to assess the status of WNV neuroinvasive disease (WNND) prediction and identify avenues for improvement. Methods We performed a multi-model comparative assessment of probabilistic forecasts submitted by 15 teams for annual WNND cases in US counties for 2020 and assessed forecast accuracy, calibration, and discriminatory power. In the evaluation, we included forecasts produced by comparison models of varying complexity as benchmarks of forecast performance. We also used regression analysis to identify modeling approaches and contextual factors that were associated with forecast skill. Results Simple models based on historical WNND cases generally scored better than more complex models and combined higher discriminatory power with better calibration of uncertainty. Forecast skill improved across updated forecast submissions submitted during the 2020 season. Among models using additional data, inclusion of climate or human demographic data was associated with higher skill, while inclusion of mosquito or land use data was associated with lower skill. We also identified population size, extreme minimum winter temperature, and interannual variation in WNND cases as county-level characteristics associated with variation in forecast skill. Conclusions Historical WNND cases were strong predictors of future cases with minimal increase in skill achieved by models that included other factors. Although opportunities might exist to specifically improve predictions for areas with large populations and low or high winter temperatures, areas with high case-count variability are intrinsically more difficult to predict. Also, the prediction of outbreaks, which are outliers relative to typical case numbers, remains difficult. Further improvements to prediction could be obtained with improved calibration of forecast uncertainty and access to real-time data streams (e.g. current weather and preliminary human cases). Graphical Abstract

Multi‐Model Prediction of West Nile Virus Neuroinvasive Disease With Machine Learning for Identification of Important Regional Climatic Drivers

“…Among the suite of better-performing models were three baseline models relying only on historical case data: NB-region, NB-hex, and AR(1). In a previous, collaborative WNND forecasting project (Holcomb et al, 2023), a NB model equivalent to the NB-hex model was found to be one of the best predictors of county-level WNND cases in the next year, a slightly different geographical scale but a very similar result to what we found here.…”

“…Models developed for one region often do not translate to others due to ecologic variation or availability of data sources between locations (Keyel et al., 2021). Ongoing WNV forecasting challenges, organized by the Centers for Disease Control and Prevention, aim to provide national and regional predictions of WNV activity yet, to‐date the best performing forecasts have leveraged only local historical incidence to make forecasts (Holcomb et al., 2023). To capture the complexities of WNV transmission and ecology, modelers often include parameters such as human demographics, reported human or veterinary cases, climate, hydrology, avian population dynamics, land use, and mosquito surveillance.…”

Patterns of West Nile Virus in the Northeastern United States Using Negative Binomial and Mechanistic Trait‐Based Models

“…A negative binomial distribution provides a simple method for describing patterns of WNV in the northeast (Keyel & Kilpatrick, 2021 ) and was among the top models in a national WNV forecasting challenge (Holcomb et al., 2023 ). The most important insight for public health is that a series of years with no or few WNV cases is possible even with a constant probability distribution for WNV cases.…”

“…In one instance, a model that worked well in a non‐probabilistic context (e.g., Keyel et al., 2019 ) was not able to outperform a probabilistic negative binomial null model in a predictive context. A negative binomial was also shown to be among the best models in a national WNV forecasting challenge (Holcomb et al., 2023 ). As a consequence, current models for the Northeastern US are very good at describing the range of possible outcomes, but do not provide much information on where in the range of outcomes a particular year will fall.…”

Invited Perspective: The Importance of Models in Preparing for West Nile Virus Outbreaks

“…Identifying weather-related factors associated with the observed spatial structuring of WNV disease is an important step in elucidating the ecological mechanisms that result in persistent trends in annual incidence. 10 This, in turn, can lead to improved predictive capacity and control measures.…”