Transforming the interpretation of significance in heart failure trials

Khan, Muhammad Shahzeb; Irfan, Simra; Khan, Safi U.; Mehra, Mandeep R.; Vaduganathan, Muthiah

doi:10.1002/ejhf.1668

Cited by 3 publications

(1 citation statement)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In a series of studies, Vassar and colleagues examined the impact of changing the threshold on randomized controlled trials (RCTs) published in general medical, orthopaedic trauma, and orthopaedic sports medicine journals [24][25][26]. The results of these studies are summarized in Table 1, along with a study by Thakur and Jha [27] that examined changing the P value threshold on results from 123 RCTs pertaining to chronic rhinosinusitis and a study by Khan et al [28] that focused on 72 RCTS from high impact general medical and cardiology journals. Across the five studies, the range of P values that retained statistical significance with a 0.005 threshold was 38.9-70.7%.…”

Section: Introductionmentioning

confidence: 99%

Impact of redefining statistical significance on P-hacking and false positive rates: An agent-based model

Fitzpatrick,

Gorman,

Trombatore

2024

PLoS ONE

View full text Add to dashboard Cite

In recent years, concern has grown about the inappropriate application and interpretation of P values, especially the use of P<0.05 to denote “statistical significance” and the practice of P-hacking to produce results below this threshold and selectively reporting these in publications. Such behavior is said to be a major contributor to the large number of false and non-reproducible discoveries found in academic journals. In response, it has been proposed that the threshold for statistical significance be changed from 0.05 to 0.005. The aim of the current study was to use an evolutionary agent-based model comprised of researchers who test hypotheses and strive to increase their publication rates in order to explore the impact of a 0.005 P value threshold on P-hacking and published false positive rates. Three scenarios were examined, one in which researchers tested a single hypothesis, one in which they tested multiple hypotheses using a P<0.05 threshold, and one in which they tested multiple hypotheses using a P<0.005 threshold. Effects sizes were varied across models and output assessed in terms of researcher effort, number of hypotheses tested and number of publications, and the published false positive rate. The results supported the view that a more stringent P value threshold can serve to reduce the rate of published false positive results. Researchers still engaged in P-hacking with the new threshold, but the effort they expended increased substantially and their overall productivity was reduced, resulting in a decline in the published false positive rate. Compared to other proposed interventions to improve the academic publishing system, changing the P value threshold has the advantage of being relatively easy to implement and could be monitored and enforced with minimal effort by journal editors and peer reviewers.

show abstract