Marginally Significant Effects as Evidence for Hypotheses

Abstract

Some effects are statistically significant. Other effects do not reach the threshold of statistical significance and are sometimes described as "marginally significant" or as "approaching significance." Although the concept of marginal significance is widely deployed in academic psychology, there has been very little systematic examination of psychologists' attitudes toward these effects. Here, we report an observational study in which we investigated psychologists' attitudes concerning marginal significance by examining their language in over 1,500 articles published in top-tier cognitive, developmental, and social psychology journals. We observed a large change over the course of four decades in psychologists' tendency to describe a p value as marginally significant, and overall rates of use appear to differ across subfields. We discuss possible explanations for these findings, as well as their implications for psychological research.

The earth is flat (p > 0.05): significance thresholds and the crisis of unreplicable research

“…And creativity is increasing. Pritschet, Powell & Horne (2016) searched over 1,500 papers in three journals of psychology for terms like ‘marginally significant’ or ‘approaching significance’ that were used for p -values between 0.05 to 0.1 and up to 0.18. They found that “the odds of an article describing a p -value as marginally significant in 2010 were 3.6 times those of an article published in 1970.” In 2000 and 2010, the proportions of articles describing at least one p -value as marginally significant were 59% and 54%.…”

“…The rules how to label results if p  > 0.05 are usually unwritten (Pritschet, Powell & Horne, 2016), so it is current practice to interpret p -values between 0.05 and 0.1, or even larger p -values, as evidence either against the null hypothesis, or (falsely) in favor of a null effect, or as no evidence at all. This anarchical state of affairs undermines ‘inferential reproducibility’, which might be the most important dimension of reproducibility and “refers to the drawing of qualitatively similar conclusions from either an independent replication of a study or a reanalysis of the original study” (Goodman, Fanelli & Ioannidis, 2016; italics supplied).…”

Clinical Science and the Replicability Crisis

“…Indeed, there are initial indications that replicability problems may be especially marked in certain domains, such as social psychology (Bakker et al, 2012;Fraley & Vazire, 2014;John et al, 2012;Open Science Collaboration 2015;Pritschet, Powell, & Horne, 2016). Specifically, researchers have identified differences among subfields-often pointing to particular problems in social psychology-in the average sample sizes employed in published research (Fraley & Vazire, 2014), in the reported use of certain QRPs (John et al, 2012), in replication success rate (Open Science Collaboration, 2015), and in the use of flexible data interpretation such as characterizing results as "marginally significant" or "trend level" (Pritschet et al, 2016). We are not entirely persuaded by this argument, however.…”

Poor Quality of Sleep in Foster Children Relates to Maltreatment and Placement Conditions

“…According to Cohen, Cohen, West, and Aiken (2003), effect sizes, which provide a better interpretation of the results weight are preferred to statistical significance tests and are as follows: r ≥ .1 represents a small effect size; r ≥ .3 represents a medium effect size, and r ≥ .5 represents a large effect size. Marginal results (ps > .05 < .10; see Pritschet, Powell, & Horne, 2016) with medium to large effect sizes were reported. Table 2 presents the proportions of sleep conditions (habits and disturbances) in our sample.…”