Survival-Inferred Fragility Index of Phase 3 Clinical Trials Evaluating Immune Checkpoint Inhibitors

Bomze, David; Asher, Nethanel; Ali, Omar Hasan; Flatz, Lukas; Azoulay, Daniel; Markel, Gal; Meirson, Tomer

doi:10.1001/jamanetworkopen.2020.17675

Cited by 27 publications

(17 citation statements)

References 79 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Would the coveted statistical significance be lost? In recent years, there has been much interest in a metric-the fragility index (FI), [8][9][10][11] first proposed 3 decades ago 12 -to test the robustness/fragility of statistically significant results. The various applications and implications of the FI are discussed herein.…”

mentioning

confidence: 99%

The Fragility Index for Assessing the Robustness of the Statistically Significant Results of Experimental Clinical Studies

2021

J GEN INTERN MED

View full text Add to dashboard Cite

I n a randomized controlled trial (RCT) comparing treatments A and B, a null hypothesis (H 0 ) of no difference in a primary outcome of interest is defined. Whether any observed difference is statistically significant has traditionally been based on the P value and the confidence interval (CI). For a century, an arbitrary threshold of 0.05 (1/20) has been used to define statistical significance. 1 Because this probability is quite low, we conclude that P ≤ 0.05 suggests that the observed difference between A and B is incompatible with H 0 , and, with a ≥ 95% degree of certainty, that A and B are different. The 95% CI is a lower-upper limit in which the true effect estimate lies within a 95% certainty. A 95% CI that does not include the null effect size indicates the observed difference has reached statistical significance, and H 0 is rejected. The 95% CI provides information about the precision of the estimate and complements the P value. Reporting both is recommended.Use of a P value of 0.05 to dichotomize whether treatments A and B are "truly" different is appealing because of its simplicity. Indeed, for all its many limitations, 2 its use actually increased from 1990 to 2015 in MEDLINE and PubMed Central abstracts and articles. 3 Statistical significance carries considerable weight. Researchers, editors and reviewers, readers, and the press tend to become more excited about positive results. 4 However, a shocking number of scientific studies and meta-analyses are not reproducible or replicable. [5][6][7] A reasonable question to ask is: if a positive study was to be repeated, how "easy" might it be for the results to change from being statistically significant to non-significant (and thus, rightly or wrongly, lose some of their appeal)? What if there was, by sheer chance, one or several more (or less) outcome event(s) in one of the comparative groups? Would the coveted statistical significance be lost? In recent years, there has been much interest in a metric-the fragility index (FI), [8][9][10][11] first proposed 3 decades ago 12 -to test the robustness/fragility of statistically significant results. The various applications and implications of the FI are discussed herein.

show abstract

mentioning

confidence: 99%

The Fragility Index for Assessing the Robustness of the Statistically Significant Results of Experimental Clinical Studies

2021

J GEN INTERN MED

View full text Add to dashboard Cite

show abstract

“…However, in most cases, the statistical conclusions of phase 3 oncology trials were not affected by changing the original design to an unstratified analysis. 8,11 This finding suggests sensitivity of the MAPS results to the statistical modeling.…”

Section: Discussionmentioning

confidence: 90%

“…The survival-inferred fragility index (SIFI) was calculated from the Kaplan-Meier curves by the iterative redesignation of the participants with the longest survival from the intervention to the control group until positive significance was lost. 8 We used the boundary for significance as specified in the statistical design for each study: P = .0476 for MPS; P = .029 for MAPS; and P = .0345 for CM743. For the nonepithelioid group, no statistical boundary was defined.…”

Section: Statistical Analysis Survival Analysismentioning

confidence: 99%

Comparison of 3 Randomized Clinical Trials of Frontline Therapies for Malignant Pleural Mesothelioma

et al. 2022

Self Cite

View full text Add to dashboard Cite

IMPORTANCE Some recently proposed frontline therapies for malignant pleural mesothelioma (MPM) are very costly, yet their impact on quality of life and overall survival of these patients remains arguable. Given the high social toll of this aggressive occupational cancer, it is paramount to establish the real clinical benefit of these treatments. OBJECTIVETo directly compare and analyze the statistical robustness of the 3 randomized clinical trials (RCTs) of frontline therapies recommended for MPM since 2003. DESIGN, SETTING, AND PARTICIPANTS This comparative effectiveness study assessed the following phase 3 RCTs: the Mesothelioma Cisplatin Pemetrexed Study (MPS) of cisplatin plus pemetrexed vs cisplatin; the Mesothelioma Avastin Cisplatin Pemetrexed Study (MAPS) of cisplatin plus pemetrexed plus bevacizumab vs cisplatin plus pemetrexed; and the CheckMate743 (CM743) study of nivolumab plus ipilimumab vs cisplatin plus pemetrexed. Data collection dates for the RCTs ranged from April 1999 to April 2018. Data for this study were analyzed from February to October 2021. MAIN OUTCOMES AND MEASURES Patient selection criteria, superiority of the intervention groups, survival-inferred fragility index, and censoring patterns in each RCT. RESULTS A total of 1501 patients were included in the analysis (1170 men [77.9%]; range of median age for treatment groups, 60 [IQR, 19-84] to 69 [IQR, 65-75] years). A virtual comparison of overall survival in MAPS vs the CM743 study showed no statistically significant difference (hazard ratio [HR], 0.97 [95% CI, 0.79-1.20]; P = .79), and the survival-inferred fragility index in the intention-to-treat (ITT) populations was as low as 0.22% of the total sample size in MPS, −0.45% of the total sample size in MAPS, and 0.99% of the total sample size in the CM743 trial. Moreover, reverse restricted mean survival time (RMST) analysis of overall survival using RMST-difference (RMST-D) demonstrated differential censoring in the ITT population of the CM743 trial favoring the control group (0.56 [95% CI, 0.18-0.94]; P = .004) and in the nonepithelioid group (reverse RMST-D, 0.90 [95% CI, 0.001-1.79]; P = .048). CONCLUSIONS AND RELEVANCEThis comparative effectiveness study found no survival benefit in the CM743 trial over MAPS, despite the inclusion of patients with worse prognosis in the latter trial.Moreover, the statistical conclusions of all the examined trials were shown to be extremely fragile, and the findings of differential censoring in the CM743 trial and in the ITT nonepithelial subset raised additional areas of concern. These findings suggest that selection criteria, fragility, and censoring patterns may affect the original conclusions drawn for the respective trials, casting a shadow on the (continued)

show abstract

“…The latter include the survival-inferred fragility index (SIFI), the restricted mean survival time (RMST), and the spin. The SIFI, is the minimum number of reassignments of the best survivors between arms that would overturn the statistical outcomes (6)(7)(8). The RMST is a statistical calculation of the time to event corresponding to the area under the Kaplan-Meir survival curve limited to a specific time point.…”

Section: Introductionmentioning

confidence: 99%

Systemic review of the robustness of randomized controlled trials for the treatment of cholangiocarcinoma in three domains: survival-inferred fragility index, restricted mean survival time, and the spin effect

Horesh¹,

Bomze²,

Lim³

et al. 2022

Hepatobiliary Surg Nutr

Self Cite

View full text Add to dashboard Cite

Importance: The vast majority of patients with cholangiocarcinoma (CC) have advanced disease at diagnosis and are candidates for palliative treatment only.Objective: The robustness of the randomized controlled trials regarding the treatment of CC are assessed.Evidence Review: A systematic review of all randomized control trials (RCT) of treatments for both intra-and extrahepatic CC between 2010 and 2020 was performed. The survival-inferred fragility index (SIFI, the minimum number of reassignments of the best survivors between arms that would overturn the statistical outcomes) was calculated. In addition, the gain, or loss, in survival in RCTs was evaluated by the restricted mean survival time (RMST) difference. Finally, the level of spin i.e., misrepresentation of study outcomes, was measured in inconclusive studies to assess distorted reporting strategies.Findings: Out of 6,167 studies retrieved, 11 could be retained for full text revision (7 with both intra-and extrahepatic CC, 3 with peri-hilar CC, and 1 with peri-hilar or distal CC). Only 3 studies included resected patients (2 with both intra-and extrahepatic CC and 1 with peri-hilar or distal CC). Nine studies investigated systemic chemotherapy (including 3 after surgical resection), one study evaluated photodynamic therapy, and another investigated the use of an endoscopically inserted stent in the biliary tract. The median SIFI was −2 (IQR −6.25, −0.25) across all studies. Overall, the median RMST difference was 0.56 months (IQR 0.10, 0.95).Finally, for inconclusive studies, the level of spin was high, moderate, and low in respectively 12.5%, 25%, and 62.5% of the studies.Conclusions and Relevance for Reviews: RCTs of CC showed a low degree of robustness with a frequent proportion of associated spin.

show abstract

Survival-Inferred Fragility Index of Phase 3 Clinical Trials Evaluating Immune Checkpoint Inhibitors

Cited by 27 publications

References 79 publications

The Fragility Index for Assessing the Robustness of the Statistically Significant Results of Experimental Clinical Studies

The Fragility Index for Assessing the Robustness of the Statistically Significant Results of Experimental Clinical Studies

Comparison of 3 Randomized Clinical Trials of Frontline Therapies for Malignant Pleural Mesothelioma

Systemic review of the robustness of randomized controlled trials for the treatment of cholangiocarcinoma in three domains: survival-inferred fragility index, restricted mean survival time, and the spin effect

Contact Info

Product

Resources

About