False-positive and false-negative risks for individual multicentre trials in critical care

Sidebotham, David; Barlow, Christopher Jake

doi:10.1016/j.bjao.2022.100003

Cited by 5 publications

(16 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We used a qualitative scale for the BF 0:1 to grade the strength of the evidence favouring zero effect (Table 1) [9,12]. We used the prop.test function in R to calculate p when p was not reported.…”

Section: Methodsmentioning

confidence: 99%

“…For the post‐test probability of a zero effect, we assigned a prior belief that the intervention had zero effect of 50% (0.5) and, necessarily, 50% (0.5) that the intervention had a non‐zero effect. We calculated the probability the intervention had zero effect after the trial,

p (H_{0}¦ data)

, as,

p (H_{0}¦ data) = \frac{1}{{BF}_{1 : 0} + 1}

We used a qualitative scale for the BF 0:1 to grade the strength of the evidence favouring zero effect (Table 1) [9, 12]. We used the prop.test function in R to calculate p when p was not reported.…”

Section: Methodsmentioning

confidence: 99%

“…However, when power is 20%, we expect to declare an effect in only 15/100 (15%) trials. In critical care, only 5-20% of trials declare an effect for the intervention on the primary outcome [5][6][7][8][9]. This finding has led to much soul searching, and even calls to halt further large randomised trials [10].…”

Section: Introductionmentioning

confidence: 99%

“…We asked the corresponding author of each included study to estimate the cost of running and completing the trial. We have recently reported a similar analysis of multicentre trials in critical care [3, 9].…”

Section: Introductionmentioning

confidence: 99%

“…We can, however, use the Bayes factor to calculate the probability of zero effect after the trial [9,14]. We can write this probability as p(H 0 |data), which in words means the probability there is no effect (i.e.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Multicentre randomised trials in anaesthesia: an analysis using Bayesian metrics

2022

View full text Add to dashboard Cite

Summary Are the results of randomised trials reliable and are p values and confidence intervals the best way of quantifying efficacy? Low power is common in medical research, which reduces the probability of obtaining a ‘significant result’ and declaring the intervention had an effect. Metrics derived from Bayesian methods may provide an insight into trial data unavailable from p values and confidence intervals. We did a structured review of multicentre trials in anaesthesia that were published in the New England Journal of Medicine, The Lancet, Journal of the American Medical Association, British Journal of Anaesthesia and Anesthesiology between February 2011 and November 2021. We documented whether trials declared a non‐zero effect by an intervention on the primary outcome. We documented the expected and observed effect sizes. We calculated a Bayes factor from the published trial data indicating the probability of the data under the null hypothesis of zero effect relative to the alternative hypothesis of a non‐zero effect. We used the Bayes factor to calculate the post‐test probability of zero effect for the intervention (having assumed 50% belief in zero effect before the trial). We contacted all authors to estimate the costs of running the trials. The median (IQR [range]) hypothesised and observed absolute effect sizes were 7% (3–13% [0–25%]) vs. 2% (1–7% [0–24%]), respectively. Non‐zero effects were declared for 12/56 outcomes (21%). The Bayes factor favouring a zero effect relative to a non‐zero effect for these 12 trials was 0.000001–1.9, with post‐test zero effect probabilities for the intervention of 0.0001–65%. The other 44 trials did not declare non‐zero effects, with Bayes factors favouring zero effect of 1–688, and post‐test probabilities of zero effect of 53–99%. The median (IQR [range]) study costs reported by 20 corresponding authors in US$ were $1,425,669 ($514,766–$2,526,807 [$120,758–$24,763,921]). We think that inadequate power and mortality as an outcome are why few trials declared non‐zero effects. Bayes factors and post‐test probabilities provide a useful insight into trial results, particularly when p values approximate the significance threshold.

show abstract

Section: Methodsmentioning

confidence: 99%

p (H_{0}¦ data)

, as,

p (H_{0}¦ data) = \frac{1}{{BF}_{1 : 0} + 1}

Section: Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Multicentre randomised trials in anaesthesia: an analysis using Bayesian metrics

2022

View full text Add to dashboard Cite

show abstract

Interpreting frequentist hypothesis tests: insights from Bayesian inference

Sidebotham,

Barlow,

Martin

et al. 2023

Can J Anesth/J Can Anesth

Self Cite

View full text Add to dashboard Cite

Randomized controlled trials are one of the best ways of quantifying the effectiveness of medical interventions. Therefore, when the authors of a randomized superiority trial report that differences in the primary outcome between the intervention group and the control group are “significant” (i.e., P ≤ 0.05), we might assume that the intervention has an effect on the outcome. Similarly, when differences between the groups are “not significant,” we might assume that the intervention does not have an effect on the outcome. Nevertheless, both assumptions are frequently incorrect. In this article, we explore the relationship that exists between real treatment effects and declarations of statistical significance based on P values and confidence intervals. We explain why, in some circumstances, the chance an intervention is ineffective when P ≤ 0.05 exceeds 25% and the chance an intervention is effective when P > 0.05 exceeds 50%. Over the last decade, there has been increasing interest in Bayesian methods as an alternative to frequentist hypothesis testing. We provide a robust but nontechnical introduction to Bayesian inference and explain why a Bayesian posterior distribution overcomes many of the problems associated with frequentist hypothesis testing. Notwithstanding the current interest in Bayesian methods, frequentist hypothesis testing remains the default method for statistical inference in medical research. Therefore, we propose an interim solution to the “significance problem” based on simplified Bayesian metrics (e.g., Bayes factor, false positive risk) that can be reported along with traditional P values and confidence intervals. We calculate these metrics for four well-known multicentre trials. We provide links to online calculators so readers can easily estimate these metrics for published trials. In this way, we hope decisions on incorporating the results of randomized trials into clinical practice can be enhanced, minimizing the chance that useful treatments are discarded or that ineffective treatments are adopted.

show abstract