External validation and update of prediction models for unfavorable outcomes in hospitalized patients with COVID-19 in Japan

Abstract

Introduction Most of the currently used prognostic models for COVID-19 are based on Western cohorts, but it is unknown whether any are applicable to patients with COVID-19 in Japan. Methods This retrospective cohort study included 160 patients with COVID-19 who were admitted to the National Center for Global Health and Medicine between January 26, 2020 and July 25, 2020. We searched PubMed for prognostic models for COVID-19. The predicted outcome was initiation of respiratory support or death. Performance of the candidate models was evaluated according to discrimination and calibration. We recalibrated the intercept of each model with our data. We also updated each model by adding β2-microglobulin (β2MG) to the model and recalculating the intercept and the coefficient of β2MG. Results Mean patient age was 49.8 years, 68% were male, 88.7% were Japanese. The study outcomes occurred in 15 patients, including two deaths. Two-hundred sixty-nine papers were screened, and four candidate prognostic models were assessed. The model of Bartoletti et al. had the highest area under receiver operating characteristic curve (AUC) (0.88; 95% confidence interval 0.81–0.96). All four models overestimated the probability of occurrence of the outcome. None of the four models showed statistically significant improvement in AUCs by adding β2MG. Conclusions Our results suggest that the existing prediction models for COVID-19 overestimate the probability of occurrence of unfavorable outcomes in a Japanese cohort. When applying a prediction model to a different cohort, it is desirable to evaluate its performance according to the prevalent health situation in that region.

Development and validation of self-monitoring auto-updating prognostic models of survival for hospitalized COVID-19 patients

“…For the latter, the frequency of updating depends on the dynamics of the performance drift of such models, making the case for monitoring of prospective model performance. While rapid and continuous geographic validation of a newly developed model is challenging 18,[25][26][27] , temporal validation within one site or institution can be feasible, due to continuous recording of patient data as part of the standard of care.…”