Recalibration and External Validation of the Risk Analysis Index

Abstract

Objective and Background: The Risk Analysis Index (RAI) predicts 30-, 180-, and 365-day mortality based on variables constitutive of frailty. Initially validated, in a single-center Veteran hospital, we sought to improve model performance by recalibrating the RAI in a large, veteran surgical registry, and to externally validate it in both a national surgical registry and a cohort of surgical patients for whom RAI was measured prospectively before surgery. Methods: The RAI was recalibrated among development and confirmation samples within the Veterans Affairs Surgical Quality Improvement Program (VASQIP; 2010–2014; N = 480,731) including major, elective noncardiac surgery patients to create the revised RAI (RAI-rev), comparing discrimination and calibration. The model was tested externally in the American College of Surgeons National Surgical Quality Improvement Program dataset (NSQIP; 2005–2014; N = 1,391,785), and in a prospectively collected cohort from the Nebraska Western Iowa Health Care System VA (NWIHCS; N = 6,856). Results: Recalibrating the RAI significantly improved discrimination for 30-day [c = 0.84–0.86], 180-day [c = 0.81–0.84], and 365-day mortality [c = 0.78–0.82] (P < 0.001 for all) in VASQIP. The RAI-rev also had markedly better calibration (median absolute difference between observed and predicted 180-day mortality: decreased from 8.45% to 1.23%). RAI-rev was highly predictive of 30-day mortality (c = 0.87) in external validation with excellent calibration (median absolute difference between observed and predicted 30-day mortality: 0.6%). The discrimination was highly robust in men (c = 0.85) and women (c = 0.89). Discrimination also improved in the prospectively measured cohort from NWIHCS for 180-day mortality [c = 0.77 to 0.80] (P < 0.001). Conclusions: The RAI-rev has improved discrimination and calibration as a frailty-screening tool in surgical patients. It has robust external validity in men and women across a wide range of surgical settings and available for immediate implementation for risk assessment and counseling in preoperative patients.

Association of Preoperative Frailty and Operative Stress With Mortality After Elective vs Emergency Surgery

“…The RAI is based on the accumulation of deficits model of frailty and uses demographic factors (including age), comorbidities, cognitive decline, residence in a facility, and activities of daily living to quantify frailty, with higher scores indicating greater frailty (eFigure in the Supplement ). 2 , 3 , 4 , 5 The OSS was developed using modified Delphi consensus methods to rate the 565 most common Current Procedural Terminology codes included in Veterans Affairs Surgical Quality Improvement Program on a scale of 1 to 5 by degree of physiologic stress experienced by patients, with higher scores indicating more stress (eTable in the Supplement ). 1 Patients were categorized as robust, normal, frail, and very fail by RAI score (RAI ≤20, 21-29, 30-39, and ≥40, respectively).…”

“… 1 Patients were categorized as robust, normal, frail, and very fail by RAI score (RAI ≤20, 21-29, 30-39, and ≥40, respectively). 1 , 3 The outcomes were mortality at 30, 90, and 180 days. P values were calculated at the 95% significance level.…”

Utility of Frailty Screening Tools in Older Surgical Patients

“…40) Another study that recalibrated and validated the Risk Analysis Index using a national surgical registry reported that the revision improved its discriminatory ability to predict mortality. 41) Based on currently available studies on older surgical patients, frailty screening appears to discriminate and identify high-risk patients regardless of screening tool and type of surgery. The ability to predict outcomes may be improved by combining frailty screening tools with other tools (such as the Surgical Apgar Score), adding more variables to achieve a more comprehensive frailty index, or adapting and validating available frailty tools to individual surgical interventions or clinical settings.…”

“… 40 ) Another study that recalibrated and validated the Risk Analysis Index using a national surgical registry reported that the revision improved its discriminatory ability to predict mortality. 41 )…”

The Importance of Incorporating Frailty Screening Into Surgical Clinical Workflow

“…We would advocate for applying frailty screening in the care of surgical patients a step further, or rather, earlier. Rapid frailty assessment tools, such as the RAI, have been shown to have excellent negative predictive values, 7 ie, if patients are determined not to be frail, then there is a very high probability that the adverse outcome will not happen. In limited-resource settings, an advantage of tools such as the RAI and the proposed MSK-FI is to screen for frailty and rule out those who might not need the full GA. 3 This precludes situations where typically high-value care becomes low-value care when delivered to robust patients who are unlikely to need extra attention and geriatric services, thereby allowing focus and resources to be directed toward high-risk surgical candidates based on an objective measure.…”