External validation of existing prediction models of 30-day mortality after Transcatheter Aortic Valve Implantation (TAVI) in the Netherlands Heart Registration

Abstract

Background: Several mortality prediction models (MPM) are used for predicting early (30-day) mortality following transcatheter aortic valve implantation (TAVI). Little is known about their predictive performance in external TAVI populations. We aim to externally validate established MPMs on a large TAVI dataset from the Netherlands Heart Registration (NHR). Methods: We included data from NHR-patients who underwent TAVI during 2013-2017. We calculated the predicted mortalities per MPM. We assessed the predictive performance by discrimination (Area Under Receiver Operating-characteristic Curve, AU-ROC); the Area Under Precision-Recall Curve, AU-PRC; calibration (using calibration-intercept and calibration-slope); Brier Score and Brier Skill Score. We also assessed the predictive performance among subgroups: tertiles of mortality-risk for non-survivors, gender, and access-route. Results: We included 6177 TAVI-patients with an observed early-mortality rate of 4.5% (n = 280). We applied seven MPMs (STS, EuroSCORE-I, EuroSCORE-II, ACC-TAVI, FRANCE-2, OBSERVANT, and German-AV) on our cohort. The highest AU-ROCs were 0.64 (95%CI 0.61-0.67) for ACC-TAVI and 0.63 (95%CI 0.60-0.67) for FRANCE-2. All MPMs had a very low AU-PRC of ≤0.09. ACC-TAVI had the best calibration-intercept and calibration-

Risk modeling in transcatheter aortic valve replacement remains unsolved: an external validation study in 2946 German patients

“…Reasons for the lowerthan-expected performance may lie in the GAVS-II model being developed in 55% surgical/45% TAVR patients from 2011-2012 (Table 1), which significantly differed in age (74 vs. 81 years) and comorbidities from our population [16]. The OBSERVANT model [14] discriminated numerically worst in the overall cohort and in TF/TA subgroups, and could not match performance in the original population (c-index 0.71), which confirms findings in other external validation studies [12,18,30]. However-of all analyzed models-its mean mortality prediction (5.8%) came closest to observed events (3.7%).…”

“…2) in our patients. The GAVS-II model [16] derived from the German Aortic Valve Registry was expected to be most adapted to German TAVR conditions, but it performed similar to the other models in the overall cohort (c-index 0.63) and in TF and TA subgroups and could not meet the performance in the development cohort (c-index 0.74), similar to the Dutch external validation study [18]. Numerically, its discrimination improved over time and in new vs. old generation devices (Table 4).…”

“…National TAVR-registries have provided platforms for the development of TAVR-specific risk models [14][15][16], which were mostly developed in the "early years" of TAVR (Table 1). Resulting limitations in their performance have already been seen when applied outside of their original populations [30,31], external validation studies with a similar design to our study were performed in the United Kingdom [12] and in the Netherlands [18]-and also retrieved disappointing results.…”

“…However, dedicated TAVR-specific models also disappointed: FRANCE-2 (overall c-index 0.66, 95% CI 0.60-0.71) was better than in the original validation cohort (c-index 0.59 [15]) and in the UK (c-index 0.62 [12]) and Netherlands (c-index 0.63 [18]) external validation studies, but not superior to surgical models and slightly worse than in an Israeli external validation study (c-index 0.71 [30]). The model considerably overestimated risk (Fig.…”

“…Several TAVR-specific risk assessment models have been developed from national registry databases to optimize the predictive performance in the context of TAVR [14][15][16]. However, the optimal approach to risk prediction in TAVR still remains unknown, as their performance has been poor in validation studies [12,17,18].…”