Bayesian Calibration of<i>p</i>‐Values from Fisher's Exact Test

Ott, Manuela; Held, Leonhard

doi:10.1111/insr.12307

Cited by 6 publications

(5 citation statements)

References 54 publications

(103 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…An alternative approach, which allows for sample size adjustments and is also computationally efficient, is to derive approximate data-based Bayes factors in closed form by applying analytical approximations, so called integrated Laplace approximations (Wang & George 2007;Li & Clyde 2016). For example, by applying the Li & Clyde (2016) methodology, an approximate, sample-size adjusted minimum Bayes factor for 2 × 2 contingency tables can be obtained in closed form (Ott & Held 2017). By studying the relationship between this minimum Bayes factor and two-sided P -values from Fisher's exact test, Ott & Held (2017) conclude that the maximal evidence of these P -values is inversely related to sample size.…”

Section: Sample-size Adjusted Bayes Factors In Glmsmentioning

confidence: 99%

“…For example, by applying the Li & Clyde (2016) methodology, an approximate, sample-size adjusted minimum Bayes factor for 2 × 2 contingency tables can be obtained in closed form (Ott & Held 2017). By studying the relationship between this minimum Bayes factor and two-sided P -values from Fisher's exact test, Ott & Held (2017) conclude that the maximal evidence of these P -values is inversely related to sample size. This is the same qualitative relationship as in the linear model, see Section 3.2 and Figure 4.…”

Section: Sample-size Adjusted Bayes Factors In Glmsmentioning

confidence: 99%

See 1 more Smart Citation

On p-Values and Bayes Factors

Held

Ott

2018

Annu. Rev. Stat. Appl.

Self Cite

235

201

View full text Add to dashboard Cite

The p-value quantifies the discrepancy between the data and a null hypothesis of interest, usually the assumption of no difference or no effect. A Bayesian approach allows the calibration of p-values by transforming them to direct measures of the evidence against the null hypothesis, so-called Bayes factors. We review the available literature in this area and consider two-sided significance tests for a point null hypothesis in more detail. We distinguish simple from local alternative hypotheses and contrast traditional Bayes factors based on the data with Bayes factors based on p-values or test statistics. A well-known finding is that the minimum Bayes factor, the smallest possible Bayes factor within a certain class of alternative hypotheses, provides less evidence against the null hypothesis than the corresponding p-value might suggest. It is less known that the relationship between p-values and minimum Bayes factors also depends on the sample size and on the dimension of the parameter of interest. We illustrate the transformation of p-values to minimum Bayes factors with two examples from clinical research.

show abstract

Section: Sample-size Adjusted Bayes Factors In Glmsmentioning

confidence: 99%

Section: Sample-size Adjusted Bayes Factors In Glmsmentioning

confidence: 99%

On p-Values and Bayes Factors

Held

Ott

2018

Annu. Rev. Stat. Appl.

Self Cite

235

201

View full text Add to dashboard Cite

show abstract

“…Bayes factors were calculated using the BayesFactor R package [23]. For comparison, we calculated minimum Bayes factors [6, 24] using the pCalibrate package, also available in R [25]. Minimum Bayes factors quantify the upper bound of evidence against the null hypothesis for a number of priors under the alternative.…”

Section: Methodsmentioning

confidence: 99%

True and false positive rates for different criteria of evaluating statistical evidence from clinical trials

Ravenzwaaij

Ioannidis

2019

BMC Med Res Methodol

View full text Add to dashboard Cite

BackgroundUntil recently a typical rule that has often been used for the endorsement of new medications by the Food and Drug Administration has been the existence of at least two statistically significant clinical trials favoring the new medication. This rule has consequences for the true positive (endorsement of an effective treatment) and false positive rates (endorsement of an ineffective treatment).MethodsIn this paper, we compare true positive and false positive rates for different evaluation criteria through simulations that rely on (1) conventional p-values; (2) confidence intervals based on meta-analyses assuming fixed or random effects; and (3) Bayes factors. We varied threshold levels for statistical evidence, thresholds for what constitutes a clinically meaningful treatment effect, and number of trials conducted.ResultsOur results show that Bayes factors, meta-analytic confidence intervals, and p-values often have similar performance. Bayes factors may perform better when the number of trials conducted is high and when trials have small sample sizes and clinically meaningful effects are not small, particularly in fields where the number of non-zero effects is relatively large.ConclusionsThinking about realistic effect sizes in conjunction with desirable levels of statistical evidence, as well as quantifying statistical evidence with Bayes factors may help improve decision-making in some circumstances.

show abstract

“…The sets differed in the relative frequency of non-zero and zero effects: the first set of simulations had a 25% occurrence rate of null effects, the second set of simulations had a 50% occurrence rate of null effects, the third set of simulations had a 75% occurrence rate of null effects, and the fourth set of simulations did not include any null effects. These different numbers reflect different rates of 'a-priori optimism' of the occurrence of true effects (25,50,75, and 0%) among medications subjected to stage III trials that try to secure licensing. Throughout the paper, we work with effect sizes in standardized form to facilitate computations and allow for comparison across results.…”

Section: Methodsmentioning

confidence: 99%

True and False Positive Rates for Different Criteria of Evaluating Statistical Evidence from Clinical Trials

Ravenzwaaij¹,

Ioannidis²

2018

Preprint

View full text Add to dashboard Cite

Background: Until recently a typical rule that has often been used for the endorsement of new medications by the Food and Drug Administration has been the existence of at least two statistically significant clinical trials favoring the new medication. This rule has consequences for the true positive (endorsement of an effective treatment) and false positive rates (endorsement of an ineffective treatment). Methods: In this paper, we compare true positive and false positive rates for different evaluation criteria through simulations that rely on (1) conventional p-values; (2) confidence intervals based on meta-analyses assuming fixed or random effects; and (3) Bayes factors. We varied threshold levels for statistical evidence, thresholds for what constitutes a clinically meaningful treatment effect, and number of trials conducted. Results: Our results show that Bayes factors, meta-analytic confidence intervals, and p-values often have similar performance. Bayes factors may perform better when the number of trials conducted is high and when trials have small sample sizes and clinically meaningful effects are not small, particularly in fields where the number of non-zero effects is relatively large. Conclusions: Thinking about realistic effect sizes in conjunction with desirable levels of statistical evidence, as well as quantifying statistical evidence with Bayes factors may help improve decision-making in some circumstances.

show abstract

Bayesian Calibration ofp‐Values from Fisher's Exact Test

Cited by 6 publications

References 54 publications

On p-Values and Bayes Factors

On p-Values and Bayes Factors

True and false positive rates for different criteria of evaluating statistical evidence from clinical trials

True and False Positive Rates for Different Criteria of Evaluating Statistical Evidence from Clinical Trials

Contact Info

Product

Resources

About