Evaluating Prognostic Bias of Critical Illness Severity Scores Based on Age, Gender, and Primary Language in the USA: A Retrospective Multicenter Study

Abstract

Background Although severity scoring systems are used to support decision making and assess ICU performance, the likelihood of bias based on age, gender, and primary language has not been studied. We aimed to identify the potential bias of them such as Sequential Organ Failure Assessment (SOFA) and Acute Physiology and Chronic Health Evaluation IVa (APACHE IVa) by evaluating hospital mortality across subgroups divided by age, gender, and primary language via two large intensive care unit (ICU) databases. Methods This multicenter, retrospective study was conducted using data from the Medical Information Mart for Intensive Care (MIMIC, 2001-2019) database and the electronic ICU Collaborative Research Database (eICU-CRD, 2014-2015). SOFA and APACHE IVa scores were obtained from the first 24 hours of ICU admission. Hospital mortality was the primary outcome. Patients were stratified by age (16-44, 45-64, 64-79, and 80-), gender (female and male), and primary language (English and non-English) then assessed for discrimination and calibration in all subgroups. To evaluate for discrimination, the area under receiver operating characteristic (AUROC) curve and area under precision-recall curve (AUPRC) were used. Standardized mortality ratio (SMR) and calibration belt plot were used to evaluate calibration. Findings A total of 173,930 patient encounters (78,550 MIMIC and 95,380 eICU-CRD) were studied. Measurements of discrimination performed best for the youngest age ranges and worsened with increasing age (AUROC ranging from 0.812 to 0.673 for SOFA and 0.882 to 0.754 for APACHE IVa, p <0.001). There was a significant difference in discrimination between male and female patients, with female patients performing worse. With MIMIC data, patients whose primary language was not English performed worse than English speaking patients (AUROC ranging 0.771 to 0.709 [p <0.001] for SOFA). Measurements of calibration applied to SOFA showed a statistically significant overestimation of mortality in the youngest patients (SMR 0.55-0.6) and underestimation of mortality in the oldest patients (SMR 1.54-1.57). When using SOFA, mortality is overestimated for male patients (SMR 0.92-0.97) and underestimated for female patients (SMR 1.05-1.11) while mortality is overestimated for English-speaking patients (SMR 0.85) and greatly underestimated for non-English speaking patients (SMR 1.4). In contrast, the calibration applied to APACHE-IVa shows underestimation of mortality for all age groups and genders. Interpretation The differences in discrimination and calibration with increasing age, female gender, and non-English speaking patients suggest that illness severity scores are prone to bias in their mortality predictions. Caution must be taken when using these illness severity scores for quality benchmarking across ICUs and decision-making for practices among a diverse population.

Prognosticating the outcome of intensive care in older patients—a narrative review

“…Importantly, the performance of disease severity scores is affected by ongoing demographic shifts, with today's very old patients not being adequately represented in historical samples for training models, and the fact that some conditions, which were imminently fatal in the past, are now managed as chronic conditions over an extended period. On this background, the discriminatory performance of the APACHE IV and SAPS II scores was found to be worse for very old patients [ 16 , 60 ]. In addition to the above issues, the APACHE score requires choosing a principal admission diagnosis, which can lead to ambiguity in older patients with complex multimorbidity and further impair the model's performance.…”