<i>P</i>-Value Precision and Reproducibility

Boos, Dennis D.; Stefanski, Leonard A.

doi:10.1198/tas.2011.10129

Cited by 162 publications

(125 citation statements)

References 21 publications

Supporting

Mentioning

121

Contrasting

Unclassified

Order By: Relevance

“…To support interpretation of these analyses, null hypothesis significance testing was employed to provide some indication of the strength of evidence for observed patterns, along with r 2 . P-values are typically imprecise and arbitrary cut-offs for declaring statistical significance and are problematic and limiting in several ways (Boos and Stefanski 2011;Halsey et al 2015). Thus, in the present article, the P-value is treated as a continuous variable providing an approximate level of evidence against the null hypothesis (Fisher 1959).…”

Section: Resultsmentioning

confidence: 99%

Ecological Influences and Morphological Correlates of Resting and Maximal Metabolic Rates across Teleost Fish Species

Killen

Glazier

Rezende

et al. 2016

The American Naturalist

208

259

View full text Add to dashboard Cite

Rates of aerobic metabolism vary considerably across evolutionary lineages, but little is known about the proximate and ultimate factors that generate and maintain this variability.Using data for 131 teleost fish species, we performed a large-scale phylogenetic comparative analysis of how interspecific variation in resting and maximum metabolic rates (RMR and MMR, respectively) is related to several ecological and morphological variables. Mass-and temperature-adjusted RMR and MMR are highly correlated along a continuum spanning a 30-to 40-fold range. Phylogenetic generalized least squares models suggest RMR and MMR are higher in pelagic species and that species with higher trophic levels exhibit elevated MMR. This variation is mirrored at various levels of structural organization: gill surface area, muscle protein content, and caudal fin aspect ratio (a proxy for activity) are positively related with aerobic capacity. Muscle protein content and caudal fin aspect ratio are also positively correlated with RMR. Hypoxia-tolerant lineages fall at the lower end of the metabolic continuum. Different ecological lifestyles are associated with contrasting levels of aerobic capacity, possibly reflecting the interplay between selection for increased locomotor performance on one hand and tolerance to low resource availability, particularly oxygen, on the other. These results support the aerobic capacity model of the evolution of endothermy, suggesting elevated body temperatures evolved as correlated responses to selection for high activity levels.

show abstract

Section: Resultsmentioning

confidence: 99%

Ecological Influences and Morphological Correlates of Resting and Maximal Metabolic Rates across Teleost Fish Species

Killen

Glazier

Rezende

et al. 2016

The American Naturalist

208

259

View full text Add to dashboard Cite

show abstract

“…Most researchers recognize that a small sample is less likely to satisfactorily reflect the population that they wish to study, as has been described in the Points of Significance series 21 , but they often do not realize that this effect will influence P values. There is variability in the P value 23 , but this is rarely mentioned in statistics textbooks or in statistics courses.…”

Section: Lewis G Halsey Douglas Curran-everett Sarah L Vowler and Gormentioning

confidence: 99%

“…In most cases, by simply accepting a P value, we ignore the scientific tenet of repeatability. We must accept this inconvenient truth about P values 23 and seek an alternative approach to statistical inference. The natural desire for a single categorical yes-or-no decision should give way to a more mature process in which evidence is graded using a variety of measures.…”

Section: Box 2 Glossarymentioning

confidence: 99%

The fickle P value generates irreproducible results

et al. 2015

View full text Add to dashboard Cite

The reliability and reproducibility of science are under scrutiny. However, a major cause of this lack of repeatability is not being considered: the wide sample-to-sample variability in the P value. We explain why P is fickle to discourage the ill-informed practice of interpreting analyses based predominantly on this statistic.Reproducible research findings are a cornerstone of the scientific method, providing essential validation. There has been recent recognition, however, that the results of published research can be difficult to replicate 1-7 , an awareness epitomized by a series in Nature entitled "Challenges in irreproducible research" and by the Reproducibility Initiative, a project intended to identify and reward reproducible research (http:// validation.scienceexchange.com/#/ reproducibilityinitiative). In a recent meeting at the American Association for the Advancement of Science headquarters involving many of the major journals reporting biomedical science research, a common set of principles and guidelines was agreed upon for promoting transparency and reproducibility 8 . These discussions and initiatives all focused on a number of issues, including aspects of statistical reporting 9 , levels of statistical power (i.e., sufficient statistical capacity to find an effect; a 'statistically significant' finding) 10 and inclusion-exclusion criteria. Yet a fundamental problem inherent in standard statistical methods, one that is pervasively linked to the lack of reproducibility in research, remains to be considered: the wide sample-to-sample variability in the P value. This omission reflects a general lack of awareness about this crucial issue, and we address this matter here.Focusing on the P value during statistical analysis is an entrenched culture [11][12][13] . The P value is often used without the realization that in most cases the statistical power of a study is too low for P to assist the interpretation of the data (Box 1). Among the many and varied reasons for a fearful and hidebound approach to statistical practice, a lack of understanding is prominent 14 . A better understanding of why P is so unhelpful should encourage scientists to reduce their reliance on this misleading concept.Readers may know of the long-standing philosophical debate about the value and validity of null-hypothesis testing [15][16][17] . Although the P value formalizes null-hypothesis testing, this article will not revisit these issues. Rather, we concentrate on how P values themselves are misunderstood.Although statistical power is a central element in reliability 18 , it is often considered only when a test fails to demonstrate a real effect (such as a difference between groups): a 'false negative' result (see Box 2 for a glossary of statistical terms used in this article). Many scientists who are not statisticians do not realize that the power of a test is equally relevant when considering statistically significant results, that is, when the null hypothesis appears to be untenable. This is because the statistical power o...

show abstract

“…This probability has been dubbed the replication (25) or reproducibility probability (26). After a significant result, this probability is typically far lower than most scientists suspect, due to the random variation of the P value.…”

Section: Inferential Reproducibilitymentioning

confidence: 99%