Are most randomised trials in anaesthesia and critical care wrong? An analysis using Bayes’ theorem

Abstract

Summary False findings are an inevitable consequence of statistical testing. In this article, I use Bayes’ theorem to estimate the false positive and false negative risks for randomised controlled trials related to our speciality. For small trials in peri‐operative medicine, the false positive risk appears to be at least 50%. For trials reporting weakly significant p values, the risk is even higher. By contrast, large, multicentre trials in critical care appear to have a high false negative risk. These findings suggest much of the evidence that underpins our clinical practice is likely to be wrong.

Bayes' formula: a powerful but counterintuitive tool for medical decision-making

“…16,17 An analysis of hypothesis testing using Bayes' formula is the subject of a separate review by one of the authors. 7…”

“…However, the low specificity means the PPV is low. Assuming a prevalence of HIT of 1%, and a sensitivity and specificity of 90% and 40% respectively, then applying equation (7) to the immunoassay test, we have PPV ¼ 0:01 Â 0:90 0:01 Â 0:90 þ ð1 À 0:01Þ Â ð1 À 0:40Þ ¼ 0:009 0:009 þ 0:594 z0:015:…”

“…In fact, the actual power of published studies is often much less than the reported power of 0.8, particularly for small discovery-orientated trials. 7 Also, for a discovery-orientated trial, investigating a 'long shot' hypothesis, the prior might be much lower than 50%. Both of these effects (low power, low prior) dramatically increase the FPR.…”

“…Small, discovery-orientated RCTs commonly have an FPR greater than 50%, despite using an alpha value of 5%. 7,8 This type of analysis is the basis of John Ioannidis' famous paper, 'Why most published research findings are false'. 9 Bayesian statistical analysis Bayesian inference can be used as an alternative to standard statistical analysis, described in the previous section.…”

Interpreting the results of clinical trials, embracing uncertainty: A Bayesian approach

“…Using a Bayesian approach of inference, new trial results are considered in the context of existing information (please refer to the Appendix for an explanation of Bayes theorem in an A/B test as used in this text). It aims to update current knowledge or the prior probability of trial results gained from previous studies into a posterior probability revised by the new trial result 8‐11 . The belief in the prior probability will, and rightly so, vary among clinicians but the Bayesian approach appears intuitive to the well‐informed clinician eager to consider new trial data 12 .…”