The Abuse of Power

Hoenig, John M.; Heisey, Dennis M.

doi:10.1198/000313001300339897

Cited by 1,548 publications

(378 citation statements)

References 33 publications

Supporting

Mentioning

368

Contrasting

Unclassified

Order By: Relevance

“…The difference between the amount of underestimation (24%) and overestimation (17%) leads to the inflation of the IC-index. However, a power value of 46% implies that the observed effect is not significant at the conventional p ϭ .05 significance level because 50% power corresponds to a p value of .05 (Hoenig & Heisey, 2001). If the reported results do not include nonsignificant results, it is likely that the reported effect sizes are inflated because even strong effects are likely to produce nonsignificant results in small samples.…”

Section: Computation Of the Incredibility Indexmentioning

confidence: 80%

The ironic effect of significant results on the credibility of multiple-study articles.

Schimmack¹

2012

Psychological Methods

433

416

View full text Add to dashboard Cite

Cohen (1962) pointed out the importance of statistical power for psychology as a science, but statistical power of studies has not increased, while the number of studies in a single article has increased. It has been overlooked that multiple studies with modest power have a high probability of producing nonsignificant results because power decreases as a function of the number of statistical tests that are being conducted (Maxwell, 2004). The discrepancy between the expected number of significant results and the actual number of significant results in multiple-study articles undermines the credibility of the reported results, and it is likely that questionable research practices have contributed to the reporting of too many significant results (Sterling, 1959). The problem of low power in multiple-study articles is illustrated using Bem's (2011) article on extrasensory perception and Gailliot et al.'s (2007) article on glucose and self-regulation. I conclude with several recommendations that can increase the credibility of scientific evidence in psychological journals. One major recommendation is to pay more attention to the power of studies to produce positive results without the help of questionable research practices and to request that authors justify sample sizes with a priori predictions of effect sizes. It is also important to publish replication studies with nonsignificant results if these studies have high power to replicate a published finding.

show abstract

Section: Computation Of the Incredibility Indexmentioning

confidence: 80%

The ironic effect of significant results on the credibility of multiple-study articles.

Schimmack¹

2012

Psychological Methods

433

416

View full text Add to dashboard Cite

show abstract

“…Retrospective power analysis was not performed because it remains controversial and can lead to biased power estimates with low precision [14,17]. Instead, confidence intervals are provided in this study [17]. Strengths of this analysis include the clinical relevance of the research aims.…”

Section: Discussionmentioning

confidence: 99%

“…Because this study is a retrospective study, a priori power analysis, which is universally accepted, is not possible. Retrospective power analysis was not performed because it remains controversial and can lead to biased power estimates with low precision [14,17]. Instead, confidence intervals are provided in this study [17].…”

Section: Discussionmentioning

confidence: 99%

Definitional Differences of ‘Outpatient’ Versus ‘Inpatient’ THA and TKA Can Affect Study Outcomes

Bovonratwet

Webb

Ondeck

et al. 2017

Clinical Orthopaedics &Amp; Related Research

View full text Add to dashboard Cite

show abstract

“…analysis provides useful information, but retrospective power analysis (that uses only the data from a single experiment) provides no additional information that is not already implicit in the p value (e.g., Gerard, Smith, & Weerakkody, 1998;Hoenig & Heisey, 2001;Nakagawa & Foster, 2004;O'Keefe, 2007;Steidl, Hayes, & Schauber, 1997;Sun, Pan, & Wang, 2011;Thomas, 1997). Retrospective power analysis can, however, at least make explicit the probability of achieving various goals in the given experiment, even if that information is not useful for additional inference from the given data.…”

Section: Power Analysis For Bayesian Estimationmentioning

confidence: 99%

Bayesian Estimation Supersedes the t Test

Kruschke¹

2012

PsycEXTRA Dataset

316

412

View full text Add to dashboard Cite

Bayesian estimation for 2 groups provides complete distributions of credible values for the effect size, group means and their difference, standard deviations and their difference, and the normality of the data. The method handles outliers. The decision rule can accept the null value (unlike traditional t tests) when certainty in the estimate is high (unlike Bayesian model comparison using Bayes factors). The method also yields precise estimates of statistical power for various research goals. The software and programs are free and run on Macintosh, Windows, and Linux platforms.Keywords: Bayesian statistics, effect size, robust estimation, Bayes factor, confidence interval One of the most frequently encountered scientific procedures is a comparison of two groups (e.g., du Prel, Röhrig, Hommel, & Blettner, 2010;Fritz, Morris, & Richler, 2012;Wetzels et al., 2011). Given data from two groups, researchers ask various comparative questions: How much is one group different from another? Can we be reasonably sure that the difference is non-zero? How certain are we about the magnitude of difference? These questions are difficult to answer because data are contaminated by random variability despite researchers' efforts to minimize extraneous influences on the data. Because of "noise" in the data, researchers rely on statistical methods of probabilistic inference to interpret the data. When data are interpreted in terms of meaningful parameters in a mathematical description, such as the difference of mean parameters in two groups, it is Bayesian analysis that provides complete information about the credible parameter values. Bayesian analysis is also more intuitive than traditional methods of null hypothesis significance testing (e.g., Dienes, 2011).This article introduces an intuitive Bayesian approach to the analysis of data from two groups. The method yields complete distributional information about the means and standard deviations of the groups. In particular, the analysis reveals the relative credibility of every possible difference of means, every possible difference of standard deviations, and all possible effect sizes. From this explicit distribution of credible parameter values, inferences about null values can be made without ever referring to p values as in null hypothesis significance testing (NHST). Unlike NHST, the Bayesian method can accept the null value, not only reject it, when certainty in the estimate is high. The new method handles outliers by describing the data as heavy tailed distributions instead of normal distributions, to the extent implied by the data. The new method also implements power analysis in both retrospective and prospective forms.The analysis is implemented in the widely used and free programming languages R and JAGS and can be run on Macintosh, Linux, and Windows operating systems. Complete installation instructions are provided, along with working examples. The programs can also be flexibly extended to other types of data and analyses. Thus, the software can be used by virtually ...

show abstract

The Abuse of Power

Cited by 1,548 publications

References 33 publications

The ironic effect of significant results on the credibility of multiple-study articles.

The ironic effect of significant results on the credibility of multiple-study articles.

Definitional Differences of ‘Outpatient’ Versus ‘Inpatient’ THA and TKA Can Affect Study Outcomes

Bayesian Estimation Supersedes the t Test

Contact Info

Product

Resources

About