Prospective Predictive Performance Comparison between Clinical Gestalt and Validated COVID-19 Mortality Scores

Abstract

Most COVID-19 mortality scores were developed at the beginning of the pandemic and clinicians now have more experience and evidence-based interventions. Therefore, we hypothesized that the predictive performance of COVID-19 mortality scores is now lower than originally reported. We aimed to prospectively evaluate the current predictive accuracy of six COVID-19 scores and compared it with the accuracy of clinical gestalt predictions. 200 patients with COVID-19 were enrolled in a tertiary hospital in Mexico City between September and December 2020. The area under the curve (AUC) of the LOW-HARM, qSOFA, MSL-COVID-19, NUTRI-CoV, and NEWS2 scores and the AUC of clinical gestalt predictions of death (as a percentage) were determined. In total, 166 patients (106 men and 60 women aged 56±9 years) with confirmed COVID-19 were included in the analysis. The AUC of all scores was significantly lower than originally reported: LOW-HARM 0.76 (95% CI 0.69 to 0.84) vs 0.96 (95% CI 0.94 to 0.98), qSOFA 0.61 (95% CI 0.53 to 0.69) vs 0.74 (95% CI 0.65 to 0.81), MSL-COVID-19 0.64 (95% CI 0.55 to 0.73) vs 0.72 (95% CI 0.69 to 0.75), NUTRI-CoV 0.60 (95% CI 0.51 to 0.69) vs 0.79 (95% CI 0.76 to 0.82), NEWS2 0.65 (95% CI 0.56 to 0.75) vs 0.84 (95% CI 0.79 to 0.90), and neutrophil to lymphocyte ratio 0.65 (95% CI 0.57 to 0.73) vs 0.74 (95% CI 0.62 to 0.85). Clinical gestalt predictions were non-inferior to mortality scores, with an AUC of 0.68 (95% CI 0.59 to 0.77). Adjusting scores with locally derived likelihood ratios did not improve their performance; however, some scores outperformed clinical gestalt predictions when clinicians’ confidence of prediction was <80%. Despite its subjective nature, clinical gestalt has relevant advantages in predicting COVID-19 clinical outcomes. The need and performance of most COVID-19 mortality scores need to be evaluated regularly.

Prognostic accuracy of triage tools for adults with suspected COVID-19 in a middle-income setting: an observational cohort study

“…29,33 This highlights the need to validate triage tools in different settings and waves to ensure that provide accurate predictions of risk. 3 Current clinical decision-making to admit patients from the ED in this study cohort achieved a sensitivity of 0.77 (95% CI 0.76 to 0.78) and specificity 0.88 (95% CI 0.87 to 0.88) for the primary outcome. The low prevalence of the primary outcome meant this achieved a negative predictive value (NPV) of 0.99 (95% CI 0.99 to 0.99) and only 14.7% of patients were admitted.…”

“…Although, in keeping with previous research, 38 use of the a triage tool would not outperform clinical decision-making, in LMICs, where disposition decision making is often based on clinician experience and gestalt, use of a triage tool may improve the speed, reproducibility and transparency of decision making, especially for less experienced clinicians. 3 The PRIEST score, PMEWS and NEWS2 use physiological cut-offs which are not routinely used as part of ED triage using SATS in South Africa and LMICs. The PRIEST score and PMEWS also use predictors, such as performance status, which are not routinely collected in this setting.…”

“…In LMICs, disposition decision making is often based on clinician experience and gestalt. 3 Use of risk-stratification tools to allow rapid triage of need for hospitalisation can help prevent hospitals being overwhelmed and assist less experienced clinicians in ensuring those at risk of deterioration receive inpatient treatment (or at least fair resource allocation). For application in low- and middle-income settings, as rapid COVID tests are not available, triage tools must be applied to patients with suspected COVID and must use easily obtainable clinical factors, as opposed to laboratory and other investigations.…”

Ethical Lessons from an Intensivist’s Perspective

“…As in non-pandemic conditions, age, degree of frailty, comorbid conditions, and previous organ dysfunction, are the four most important criteria to help predict patient survival when making triage decisions for COVID-19 patients [9,[11][12][13][14]. These have been combined with other factors in various "prediction of survival" scoring systems for patients with COVID-19 [15]; however, frequent recalibration is likely to be necessary as the pandemic evolves, and although such scores can help inform decisions, clinical judgment should remain an essential component of triage [16,17]. Importantly, the person's preferences must also be taken into account whenever possi-ble, although they may not be known in the intensive care environment if advanced care planning has not previously been discussed or documented.…”

Correspondence on ‘Prospective Predictive Performance Comparison between Clinical Gestalt and Validated COVID-19 Mortality Scores’ by Soto-Mota et al

“…Dear Editor, I read the article 'Prospective predictive performance comparison between clinical gestalt and validated COVID-19 mortality scores' with great interest. 1 The authors compared various COVID-19 mortality prediction models validated in Mexican patients -LOW-HARM, MSL-COVID-19, Nutri-CoV, and neutrophil-to-lymphocyte ratio (NLR) -, qSOFA, and NEWS2 against clinical gestalt to predict mortality among COVID-19 patients admitted to a tertiary hospital, concluding that clinical gestalt was non-inferior. I would like to comment on some issues with this article.…”