Comparative assessment of methods for short-term forecasts of COVID-19 hospital admissions in England at the local level

Abstract

Background Forecasting healthcare demand is essential in epidemic settings, both to inform situational awareness and facilitate resource planning. Ideally, forecasts should be robust across time and locations. During the COVID-19 pandemic in England, it is an ongoing concern that demand for hospital care for COVID-19 patients in England will exceed available resources. Methods We made weekly forecasts of daily COVID-19 hospital admissions for National Health Service (NHS) Trusts in England between August 2020 and April 2021 using three disease-agnostic forecasting models: a mean ensemble of autoregressive time series models, a linear regression model with 7-day-lagged local cases as a predictor, and a scaled convolution of local cases and a delay distribution. We compared their point and probabilistic accuracy to a mean-ensemble of them all and to a simple baseline model of no change from the last day of admissions. We measured predictive performance using the weighted interval score (WIS) and considered how this changed in different scenarios (the length of the predictive horizon, the date on which the forecast was made, and by location), as well as how much admissions forecasts improved when future cases were known. Results All models outperformed the baseline in the majority of scenarios. Forecasting accuracy varied by forecast date and location, depending on the trajectory of the outbreak, and all individual models had instances where they were the top- or bottom-ranked model. Forecasts produced by the mean-ensemble were both the most accurate and most consistently accurate forecasts amongst all the models considered. Forecasting accuracy was improved when using future observed, rather than forecast, cases, especially at longer forecast horizons. Conclusions Assuming no change in current admissions is rarely better than including at least a trend. Using confirmed COVID-19 cases as a predictor can improve admissions forecasts in some scenarios, but this is variable and depends on the ability to make consistently good case forecasts. However, ensemble forecasts can make forecasts that make consistently more accurate forecasts across time and locations. Given minimal requirements on data and computation, our admissions forecasting ensemble could be used to anticipate healthcare needs in future epidemic or pandemic settings.

Evaluating an epidemiologically motivated surrogate model of a multi-model ensemble

“…Here, we focus on reported cases and primarily on the European Forecast Hub but our observations hold, in our view, across COVID-19 Forecast Hubs and to a lesser degree targets. We focus on reported cases as these represent the most common forecast target for COVID-19 forecast models (Nixon et al 2022), they are often of the most direct interest due to being a leading indicator for other metrics such as hospitalisations (Meakin et al 2022), and they are generally the most challenging to predict (Sherratt et al 2022). In general, 5 main classes of forecast models are submitted (Bracher et al 2022; Cramer et al 2022), statistical forecasting models such as ARIMA models, mechanistic forecasting models based on the compartmental modelling framework and its generalisations (Srivastava, Xu, and Prasanna 2020; Li et al 2021), semi-mechanistic approaches that blend both of these approaches (Castro et al 2021; Bosse et al 2022), agent-based simulation models (Rakowski et al 2010; Adamik et al 2020), and human insight based forecast models that may also include elements of other methods (Karlen 2020; Bosse et al 2022).…”

The impact of signal variability on epidemic growth rate estimation from wastewater surveillance data

“…Throughout the Covid-19 epidemic the number of reported cases has provided a useful metric to help forecast hospital demand [37]. As the number of reported test results is highly dependent on test-seeking behaviour, wastewater can potentially be a less biased way to provide the same early warning.…”

Combining models to generate consensus medium-term projections of hospital admissions, occupancy and deaths relating to COVID-19 in England

“…For a forecast horizon, h, and projection interval width, 1 − α, the empirical coverage of a model (often referred to also as calibration [28]) is calculated as the proportion of forecast targets (across all forecast dates) for which the projection interval contained the true value; a well calibrated model has empirical coverage equal to the width of the nominal projection interval (i.e. the 50% projection interval should contain the true value 50% of the time).…”