The Insignificance of Statistical Significance Testing

Johnson, Douglas H.

doi:10.2307/3802789

Cited by 932 publications

(644 citation statements)

References 47 publications

Supporting

Mentioning

610

Contrasting

Unclassified

Order By: Relevance

“…Several options are available, and although no one approach is perfect 15 , perhaps the most intuitive and tractable is to report effect size estimates and their precision (95% confidence intervals (95% CIs; see Box 3 for statistical formulae discussed in this article) 29,30 , aided by graphical presentation [31][32][33][34] . This approach to statistical interpretation emphasizes the importance and precision of the estimated effect size, which answers the most frequent question that scientists ask: how big is the difference, or how strong is the relationship or association?…”

Section: Npgmentioning

confidence: 99%

The fickle P value generates irreproducible results

et al. 2015

View full text Add to dashboard Cite

The reliability and reproducibility of science are under scrutiny. However, a major cause of this lack of repeatability is not being considered: the wide sample-to-sample variability in the P value. We explain why P is fickle to discourage the ill-informed practice of interpreting analyses based predominantly on this statistic.Reproducible research findings are a cornerstone of the scientific method, providing essential validation. There has been recent recognition, however, that the results of published research can be difficult to replicate 1-7 , an awareness epitomized by a series in Nature entitled "Challenges in irreproducible research" and by the Reproducibility Initiative, a project intended to identify and reward reproducible research (http:// validation.scienceexchange.com/#/ reproducibilityinitiative). In a recent meeting at the American Association for the Advancement of Science headquarters involving many of the major journals reporting biomedical science research, a common set of principles and guidelines was agreed upon for promoting transparency and reproducibility 8 . These discussions and initiatives all focused on a number of issues, including aspects of statistical reporting 9 , levels of statistical power (i.e., sufficient statistical capacity to find an effect; a 'statistically significant' finding) 10 and inclusion-exclusion criteria. Yet a fundamental problem inherent in standard statistical methods, one that is pervasively linked to the lack of reproducibility in research, remains to be considered: the wide sample-to-sample variability in the P value. This omission reflects a general lack of awareness about this crucial issue, and we address this matter here.Focusing on the P value during statistical analysis is an entrenched culture [11][12][13] . The P value is often used without the realization that in most cases the statistical power of a study is too low for P to assist the interpretation of the data (Box 1). Among the many and varied reasons for a fearful and hidebound approach to statistical practice, a lack of understanding is prominent 14 . A better understanding of why P is so unhelpful should encourage scientists to reduce their reliance on this misleading concept.Readers may know of the long-standing philosophical debate about the value and validity of null-hypothesis testing [15][16][17] . Although the P value formalizes null-hypothesis testing, this article will not revisit these issues. Rather, we concentrate on how P values themselves are misunderstood.Although statistical power is a central element in reliability 18 , it is often considered only when a test fails to demonstrate a real effect (such as a difference between groups): a 'false negative' result (see Box 2 for a glossary of statistical terms used in this article). Many scientists who are not statisticians do not realize that the power of a test is equally relevant when considering statistically significant results, that is, when the null hypothesis appears to be untenable. This is because the statistical power o...

show abstract

Section: Npgmentioning

confidence: 99%

The fickle P value generates irreproducible results

et al. 2015

View full text Add to dashboard Cite

show abstract

“…Perhaps the most common approach to evaluating RSF models, albeit misguided (Yoccoz, 1991;Cherry, 1998;Johnson, 1999;Anderson et al, 2000;Boyce, 2001), is to use probability or significance levels. A small P -value is presumed to give some measure of confidence that the model is not due to chance alone.…”

Section: Statistical Inferencementioning

confidence: 99%

Evaluating resource selection functions

Boyce

Vernier

Nielsen

et al. 2002

Ecological Modelling

2,116

2,037

View full text Add to dashboard Cite

A resource selection function (RSF) is any model that yields values proportional to the probability of use of a resource unit. RSF models often are fitted using generalized linear models (GLMs) although a variety of statistical models might be used. Information criteria such as the Akaike Information Criteria (AIC) or Bayesian Information Criteria (BIC) are tools that can be useful for selecting a model from a set of biologically plausible candidates. Statistical inference procedures, such as the likelihood-ratio test, can be used to assess whether models deviate from random null models. But for most applications of RSF models, usefulness is evaluated by how well the model predicts the location of organisms on a landscape. Predictions from RSF models constructed using presence/absence (used/ unused) data can be evaluated using procedures developed for logistic regression, such as confusion matrices, Kappa statistics, and Receiver Operating Characteristic (ROC) curves. However, RSF models estimated from presence/ available data create unique problems for evaluating model predictions. For presence/available models we propose a form of k -fold cross validation for evaluating prediction success. This involves calculating the correlation between RSF ranks and area-adjusted frequencies for a withheld sub-sample of data. A similar approach can be applied to evaluate predictive success for out-of-sample data. Not all RSF models are robust for application in different times or different places due to ecological and behavioral variation of the target organisms. #

show abstract

“…A 2 test statistic (1 df) can be used to evaluate the likelihood that an observed changepoint is real (Qian and others, in press). However, we only used this statistic to help assess the likelihood that changepoints with relatively wide cumulative probability distributions represented real biological changes, as uncertainty around the changepoint was a much more relevant issue (Suter 1996;Germano 1999;Johnson 1999). Illustration of the cumulative probability of a changepoint estimated for an individual metric in response to surface-water TP.…”

mentioning

confidence: 99%

Integrating Bioassessment and Ecological Risk Assessment: An Approach to Developing Numerical Water-Quality Criteria

King

Richardson

2003

Environmental Management

139

View full text Add to dashboard Cite

/ ioassessment is used worldwide to monitor aquatic health but is infrequently used with risk-assessment objectives, such as supporting the development of defensible, numerical water-quality criteria. To this end, we present a generalized approach for detecting potential ecological thresholds using assemblage-level attributes and a multimetric index (Index of Biological Integrity-IBI) as endpoints in response to numerical changes in water quality. To illustrate the approach, we used existing macroinvertebrate and surface-water total phosphorus (TP) datasets from an observed P gradient and a P-dosing experiment in wetlands of the south Florida coastal plain nutrient ecoregion. Ten assemblage attributes were identified as potential metrics using the observational data, and five were validated in the experiment. These five core metrics were subjected individually and as an aggregated Nutrient-IBI to nonparametric changepoint analysis (nCPA) to estimate cumulative probabilities of a threshold response to TP. Threshold responses were evident for all metrics and the IBI, and were repeatable through time. Results from the observed gradient indicated that a threshold was Ն50% probable between 12.6 and 19.4 g/L TP for individual metrics and 14.8 g/L TP for the IBI. Results from the P-dosing experiment revealed Ն50% probability of a response between 11.2 and 13.0 g/L TP for the metrics and 12.3 g/L TP for the IBI. Uncertainty analysis indicated a low (typically Ն5%) probability that an IBI threshold occurred at Յ 10 g/L TP, while there was Ն95% certainty that the threshold was Յ 17 g/L TP. The weight-of-evidence produced from these analyses implies that a TP concentration Ͼ 12-15 g/L is likely to cause degradation of macroinvertebrate assemblage structure and function, a reflection of biological integrity, in the study area. This finding may assist in the development of a numerical waterquality criterion for TP in this ecoregion, and illustrates the utility of bioassessment to environmental decision-making.Bioassessment has become a widely accepted technique for monitoring aquatic health in streams, lakes, and wetlands throughout the world (Rosenberg and Resh 1993). Bioassessment has a long history in Europe (reviewed by Cairns and Pratt 1993) and has more recently become popular in North America, largely in response to the mandate of §101(a) of the Clean Water Act (CWA) to restore and maintain the biological integrity of the USA's waters (Karr 1981). One bioassessment approach that has received considerable attention in the USA is the multimetric approach (sensu Karr 1981). Multimetric indices, such as the Index of Biological Integrity (e.g., Karr and Chu 1997), are an aggregation of a suite of biological attributes that represent key elements of structure or function of an aquatic assemblage and show a consistent, predictable response to human influence. The strength of multimetric assessments lies in their ability to integrate multiple facets of biological condition (Barbour and others 1995), and thus provid...

show abstract

The Insignificance of Statistical Significance Testing

Cited by 932 publications

References 47 publications

The fickle P value generates irreproducible results

The fickle P value generates irreproducible results

Evaluating resource selection functions

Integrating Bioassessment and Ecological Risk Assessment: An Approach to Developing Numerical Water-Quality Criteria

Contact Info

Product

Resources

About