A review of high impact journals found that misinterpretation of non-statistically significant results from randomized trials was common

Hemming, Karla; Javid, Iqra; Taljaard, Monica

doi:10.1016/j.jclinepi.2022.01.014

Cited by 21 publications

(12 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It commonly occurs when authors claim that a treatment is beneficial, although improvement in the primary end point was not statistically significant. 44,45 Spin may be apparent when authors shift the discussion to more promising secondary endpoints, to any statistically significant results, or when a P value is described as 'trending toward significance'. 46 Unfortunately, spin is quite common in abstracts in all branches of medicine.…”

Section: Problems With Abstractsmentioning

confidence: 99%

Scientific abstracts: Texts, contexts, and subtexts

Lang¹

2022

ESE

View full text Add to dashboard Cite

In their 4000-year history, abstracts have taken several forms and represented a variety of documents. The scientific journal emerged in the 1600s and gave rise to what would become the modern scientific abstract. Here, I describe the contexts in which abstracts evolved, address the subtexts of opinions about their purpose, and review the texts of 12 kinds of abstracts. For most readers, articles do not exist beyond abstracts. However, the quality of abstracts is often poor. Inaccuracies are common, serious, widespread, and long-standing. Abstracts should inform only the choice of what to read and never what to do.

show abstract

Section: Problems With Abstractsmentioning

confidence: 99%

Scientific abstracts: Texts, contexts, and subtexts

Lang¹

2022

ESE

View full text Add to dashboard Cite

show abstract

“…One way of addressing this issue is to consider the whole CI for the measure of treatment efficacy, such as a risk difference, odds ratio, or hazard ratio [HR] . Any parameter value within the CI is deemed compatible with the observed data .…”

Section: Introductionmentioning

confidence: 99%

“…One way of addressing this issue is to consider the whole CI for the measure of treatment efficacy, such as a risk difference, odds ratio, or hazard ratio [HR]. 3,4,8 Any parameter value within the CI is deemed compatible with the observed data. 9 If the CI of a statistically nonsignificant result excludes clinically meaningful values, one might conclude that no clinically meaningful effect exists; however, CIs typically include both clinically meaningful and clinically negligible values of the effectiveness parameter, and such results remain ambiguous.…”

mentioning

confidence: 99%

“…9 If the CI of a statistically nonsignificant result excludes clinically meaningful values, one might conclude that no clinically meaningful effect exists; however, CIs typically include both clinically meaningful and clinically negligible values of the effectiveness parameter, and such results remain ambiguous. 8,10 This study proposes an alternative approach, based on a direct comparison of the null hypothesis of no effect to the alternate hypothesis of effectiveness (as stated by the researchers in their sample size calculation) in light of observed trial data. The reasoning resembles the diagnostic process, when a clinician weighs the probabilities of a disease being absent or present in light of the result of a diagnostic test.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Evidence of Lack of Treatment Efficacy Derived From Statistically Nonsignificant Results of Randomized Clinical Trials

Perneger

Gayet-Ageron

2023

JAMA

View full text Add to dashboard Cite

ImportanceMany randomized clinical trials yield statistically nonsignificant results. Such results are difficult to interpret within the dominant statistical framework.ObjectiveTo estimate the strength of evidence in favor of the null hypothesis of no effect vs the prespecified effectiveness hypothesis among nonsignificant primary outcome results of randomized clinical trials by application of the likelihood ratio.Design, Setting, and ParticipantsCross-sectional study of statistically nonsignificant results for primary outcomes of randomized clinical trials published in 6 leading general medical journals in 2021.Outcome measuresThe likelihood ratio for the null hypothesis of no effect vs the effectiveness hypothesis stated in the trial protocol (alternate hypothesis). The likelihood ratio quantifies the support that the data provide to one hypothesis vs the other.ResultsIn 130 articles that reported 169 statistically nonsignificant results for primary outcomes, 15 results (8.9%) favored the alternate hypothesis (likelihood ratio, &lt;1), and 154 (91.1%) favored the null hypothesis of no effect (likelihood ratio, &gt;1). For 117 (69.2%), the likelihood ratio exceeded 10; for 88 (52.1%), it exceeded 100; and for 50 (29.6%), it exceeded 1000. Likelihood ratios were only weakly correlated with P values (Spearman r, 0.16; P = .045).ConclusionsA large proportion of statistically nonsignificant primary outcome results of randomized clinical trials provided strong support for the hypothesis of no effect vs the alternate hypothesis of clinical efficacy stated a priori. Reporting the likelihood ratio may improve the interpretation of clinical trials, particularly when observed differences in the primary outcome are statistically nonsignificant.

show abstract

“…These case studies illustrate how non‐statistically significant outcomes can be misinterpreted as evidence of no effect; and, moreover, even when results are sufficiently precise to rule out clinically important effects, trialists still persist in interpreting key outcomes based on statistical significance 12‐14 …”

Section: Introductionmentioning

confidence: 99%

A re‐analysis of 150 women's health trials to investigate how the Bayesian approach may offer a solution to the misinterpretation of statistical findings

Hemming

Melo

Luo

et al. 2023

BJOG

Self Cite

View full text Add to dashboard Cite

ObjectiveTo investigate whether a Bayesian interpretation might help prevent misinterpretation of statistical findings and support authors to differentiate evidence of no effect from statistical uncertainty.DesignA Bayesian re‐analysis to determine posterior probabilities of clinically important effects (e.g., a large effect is set at a 4 percentage point difference and a trivial effect to be within a 0.5 percentage point difference). Posterior probabilities greater than 95% are considered as strong statistical evidence, and less than 95% as inconclusive.Sample150 major women's health trials with binary outcomes.Main Outcome MeasuresPosterior probabilities of large, moderate, small and trivial effects.ResultsUnder frequentist methods, 48 (32%) were statistically significant (p‐value ≤ 0.05) and 102 (68%) statistically non‐significant. The frequentist and Bayesian point estimates and confidence intervals showed strong concordance. Of the statistically non‐significant trials (n = 102), the Bayesian approach classified the majority (94, 92%) as inconclusive, neither able to confirm or refute effectiveness. A small number of statistically non‐significant findings (8, 8%) were classified as having strong statistical evidence of an effect.ConclusionsWhilst almost all trials report confidence intervals, in practice most statistical findings are interpreted on the basis of statistical significance, mostly concluding evidence of no effect. Findings here suggest the majority are likely uncertain. A Bayesian approach could help differentiate evidence of no effect from statistical uncertainty.

show abstract

A review of high impact journals found that misinterpretation of non-statistically significant results from randomized trials was common

Cited by 21 publications

References 43 publications

Scientific abstracts: Texts, contexts, and subtexts

Scientific abstracts: Texts, contexts, and subtexts

Evidence of Lack of Treatment Efficacy Derived From Statistically Nonsignificant Results of Randomized Clinical Trials

A re‐analysis of 150 women's health trials to investigate how the Bayesian approach may offer a solution to the misinterpretation of statistical findings

Contact Info

Product

Resources

About