Do not throw the baby out with the bath water: a guide for using non-significant results in practice

Abstract

Acting on results that are not statistically significant is challenging for clinicians. Such results are often interpreted as evidence of lack of association or as useless evidence. We provide a framework for interpreting and applying non-significant results at the point of care using the Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach.It is well known that p values are misused, misunderstood and miscommunicated.1 Much has been written about misleading conclusions based on p values and data dredging; and how this contributes to publication bias and misleading conclusions. However, there is minimal guidance for clinicians interested in practicing evidence-based medicine on how to actually implement results that are not statistically significant in patient care. Many scientific publications have cautioned about the limitations of p values and provided the correct definition and interpretation of them, but a practical guide is needed. To merely tell practitioners that p value is defined as the probability of observing events as extreme or more extreme than the observed data, given that the null hypothesis is true (which is the correct definition), is not very helpful. We provide a framework for interpreting and applying non-significant results at the point of care with an example. This framework is derived from the fields of statistics, evidence-based medicine and patient-centred shared decision-making and is implemented using the GRADE approach.

Level and Prevalence of Spin in Published Cardiovascular Randomized Clinical Trial Reports With Statistically Nonsignificant Primary Outcomes

“…Readers of medical evidence should be aware of these rhetorical techniques in science publications. 28 , 29 , 30 More robust editorial and peer review may help reduce the level of spin. Finally, we propose that medical progress might move faster and waste fewer resources if authors were able to publish neutral findings in higher-impact journals.…”

Cost-effectiveness of a community-based integrated care model compared with usual care for older adults with complex needs: a stepped-wedge cluster-randomised trial

“…Despite positive feedback from participants and their carers, the OPEN ARCH trial achieved a non-statistically significant effect on healthcare utilisation, functional status and quality of life 21. We followed recommendations for presenting non-significant results in practice49 and applied Bayesian CE modelling to further examine the impact of the intervention with 10 000 simulations. Model-based CEA is ideally positioned to explain how well an intervention could work 48.…”

Connecting Female Entertainment Workers in Cambodia to Health Care Services Using mHealth: Economic Evaluation of Mobile Link

“…To estimate the total DALYs averted by Mobile Link, we summed the product of the absolute risk difference (ARD) and the DALYs for each of the five outcomes, represented by the index i in Textbox 1. When interpreting and applying non-significant results, the ARD (also called absolute risk reduction) is recommended to communicate the potential magnitude of effect of an intervention [29].…”

“…To characterize the uncertainty in ARDs, we assigned normal (base-case assumption), beta, and uniform distributions to each group's endline risk and generated 10,000 estimates of the ARD for each outcome. Following recommended procedures [29,31], we derived the lower and upper values of each risk using the confidence intervals of odds ratios reported in the trial (Multimedia Appendix 2 and 3). We assumed that risks reported in the trial were the mean values, and the standard deviation was equal to the difference between the mean value and upper limit divided by 1.96.…”