True and false positive rates for different criteria of evaluating statistical evidence from clinical trials

Abstract

BackgroundUntil recently a typical rule that has often been used for the endorsement of new medications by the Food and Drug Administration has been the existence of at least two statistically significant clinical trials favoring the new medication. This rule has consequences for the true positive (endorsement of an effective treatment) and false positive rates (endorsement of an ineffective treatment).MethodsIn this paper, we compare true positive and false positive rates for different evaluation criteria through simulations that rely on (1) conventional p-values; (2) confidence intervals based on meta-analyses assuming fixed or random effects; and (3) Bayes factors. We varied threshold levels for statistical evidence, thresholds for what constitutes a clinically meaningful treatment effect, and number of trials conducted.ResultsOur results show that Bayes factors, meta-analytic confidence intervals, and p-values often have similar performance. Bayes factors may perform better when the number of trials conducted is high and when trials have small sample sizes and clinically meaningful effects are not small, particularly in fields where the number of non-zero effects is relatively large.ConclusionsThinking about realistic effect sizes in conjunction with desirable levels of statistical evidence, as well as quantifying statistical evidence with Bayes factors may help improve decision-making in some circumstances.

Comparing the Evidential Strength for Psychotropic Drugs: A Bayesian Meta-Analysis

“…Previously, Bayesian methods have been proposed for and discussed in the context of the drug development and endorsement. [4,6,7,35–39] In recent years, Bayesian network meta-analyses specifically have become increasingly popular in medical sciences. [37,38,40,41] Our work differs in several respects.…”

“…For the purpose of drug development and endorsement, Bayesian meta-analysis offer several advantages over classical, frequentist meta-analysis, suggested by the FDA. [42] While frequentist meta-analysis is well-equipped to estimate the size of a treatment effect and its uncertainty [6], it cannot differentiate between absence of evidence (uncertainty regarding the effect) and evidence of absence (e.g., evidence for effect = 0; a similar argument is made by [4]). This is especially important in the context of failed or negative trials, which could either indicate insufficient data or non-effectiveness of the drug.…”

“…Previously, Bayesian methods have been proposed for and discussed in the context of the drug development and endorsement. [4,6,7,[35][36][37][38][39] In recent years, Bayesian network meta-analyses specifically have become increasingly popular in medical . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.…”

“…[4,7,8] In contrast to pvalues, BFs quantify evidence in favour of both the null hypothesis (i.e., no treatment effect) and the alternative hypothesis (i.e., a treatment effect) by comparing the relative likelihood of the observed data under either hypothesis. [6,[9][10][11] For instance, a BF10 (where the subscript indicates that the BF quantifies the likelihood of the alternative hypothesis relative to the null hypothesis) of 30 indicates the observed data is thirty times more likely to have occurred under the alternative hypothesis than under the null hypothesis (this is considered strong evidence for the alternative hypothesis). [10] Alternatively, a BF10 of 0.2 (or 1/5) indicates the observed data is five times more likely to have occurred under the null hypothesis than under the alternative hypothesis.…”

“…As such, crucial information such as the number of trials conducted before obtaining two positive trials is ignored. [5,6] Instead, the Bayes Factor ( BF ) has been suggested as a measure to quantify evidence holistically. [4,7,8] In contrast to p -values, BF s quantify evidence in favour of both the null hypothesis (i.e., no treatment effect) and the alternative hypothesis (i.e., a treatment effect) by comparing the relative likelihood of the observed data under either hypothesis.…”