Are potentially clinically meaningful benefits misinterpreted in cardiovascular randomized trials? A systematic examination of statistical significance, clinical significance, and authors’ conclusions

Abstract

BackgroundWhile journals and reporting guidelines recommend the presentation of confidence intervals, many authors adhere strictly to statistically significant testing. Our objective was to determine what proportions of not statistically significant (NSS) cardiovascular trials include potentially clinically meaningful effects in primary outcomes and if these are associated with authors’ conclusions.MethodsCardiovascular studies published in six high-impact journals between 1 January 2010 and 31 December 2014 were identified via PubMed. Two independent reviewers selected trials with major adverse cardiovascular events (stroke, myocardial infarction, or cardiovascular death) as primary outcomes and extracted data on trial characteristics, quality, and primary outcome. Potentially clinically meaningful effects were defined broadly as a relative risk point estimate ≤0.94 (based on the effects of ezetimibe) and/or a lower confidence interval ≤0.75 (based on the effects of statins).ResultsWe identified 127 randomized trial comparisons from 3200 articles. The primary outcomes were statistically significant (SS) favoring treatment in 21% (27/127), NSS in 72% (92/127), and SS favoring control in 6% (8/127). In 61% of NSS trials (56/92), the point estimate and/or lower confidence interval included potentially meaningful effects. Both point estimate and confidence interval included potentially meaningful effects in 67% of trials (12/18) in which authors’ concluded that treatment was superior, in 28% (16/58) with a neutral conclusion, and in 6% (1/16) in which authors’ concluded that control was superior. In a sensitivity analysis, 26% of NSS trials would include potential meaningful effects with relative risk thresholds of point estimate ≤0.85 and/or a lower confidence interval ≤0.65.ConclusionsPoint estimates and/or confidence intervals included potentially clinically meaningful effects in up to 61% of NSS cardiovascular trials. Authors’ conclusions often reflect potentially meaningful results of NSS cardiovascular trials. Given the frequency of potentially clinical meaningful effects in NSS trials, authors should be encouraged to continue to look beyond significance testing to a broader interpretation of trial results.Electronic supplementary materialThe online version of this article (doi:10.1186/s12916-017-0821-9) contains supplementary material, which is available to authorized users.

Robotic Lobectomy is Cost-Effective and Provides Comparable Health Utility Scores to Video-Assisted Lobectomy

“…While the measured actual difference of 0.05 at 12 weeks is lower than the MID of 0.07, it is noted that the MID lies within the 95% CI: (0.01–0.09). It has been shown that clinically meaningful results can be concluded based on the position of the MID within the CI 20,21 . To our knowledge, there are 2 other RCTs comparing robotic to VATS lung resection 22,23 …”

“…It has been shown that clinically meaningful results can be concluded based on the position of the MID within the CI. 20,21 To our knowledge, there are 2 other RCTs comparing robotic to VATS lung resection. 22,23 The ROMAN study enrolled patients with T1-T2, N0-N1 NSCLC who were randomized to either RTS or VATS lung resection (lobectomy and segmentectomy).…”

A randomized controlled trial on the effect of a silver carboxymethylcellulose dressing on surgical site infections after breast cancer surgery

“…There is no guideline providing any strict recommendation on how this interpretation can be achieved. In recent years it has been increasingly recommended by several authors [ 33 – 36 ] to also judge the clinical relevance of study findings. In our study, there is no disadvantage/harm for patients through the intervention and there is proven benefit in terms of patient satisfaction and costs.…”