Assessing the Statistical Analyses Used in Basic and Applied Social Psychology After Their p -Value Ban

Abstract

In this article, we assess the 31 articles published in Basic and Applied Social Psychology (BASP) in 2016, which is one full year after the BASP editors banned the use of inferential statistics. We discuss how the authors collected their data, how they reported and summarized their data, and how they used their data to reach conclusions. We found multiple instances of authors overstating conclusions beyond what the data would support if statistical significance had been considered. Readers would be largely unable to recognize this because the necessary information to do so was not readily available.

Effects of peer tutoring on middle school students’ mathematics self-concepts

“…T-test (95% confidence level) was used in order to analyze the differences in gain scores between the experimental and the control group and also the differences between the posttest and the pretest scores within each group for each grade and overall [88]. Analyses of variance (ANOVAs) were used in order to analyze the differences among grades in the experimental group for the pretest scores, posttest scores and gain scores [89]. Effect sizes were calculated [90], and Hedge's g was provided in each case.…”

Moving to a World Beyond “ p < 0.05”

“…The publish-or-perish culture has damaged science, as has the judgment of their work by silly metrics. " Trafimow (2019), who added energy to the discussion of pvalues a few years ago by banning them from the journal he edits (Fricker et al 2019), suggests five "nonobvious changes" to editorial practice. These suggestions, which demand reevaluating traditional practices in editorial policy, will not be trivial to implement but would result in massive change in some journals.…”

“…Challenges for editors and reviewers are many. Here's an example: Fricker et al (2019) observed that when p-values were suspended from the journal Basic and Applied Social Psychology authors tended to overstate conclusions.…”

P-Values in a Post-Truth World

“…Evidence supporting the net negativity of claims of statistical significance is largely missing from ASA I and II, perhaps because the authors believe the evidence is either "self-evident" or otherwise available (though some evidence suggests the opposite [22] ). And while careful reasoning is subjective, it has been extensively studied.…”

“…The core human and systemic problems are not addressed by shifting blame to p-values and pushing alternatives as magic cures-especially alternatives that have been subject to little or no comparative evaluation in either classrooms or practice, implying the need for empirical investigations of these claims, rather than blanket statements. Section 5 also quotes Fricker et al [22] , which empirically evaluated the impact of the journal Basic and Applied Social Psychology banning all of inferential statistics, including the phrase "statistically significant." That study found that when authors were forbidden from using statistical inference, studies tended to overstate, rather than understate their claims, providing evidence that some proposals had the opposite of the desired impact.…”