How Can Significance Tests Be Deinstitutionalized?

Abstract

The purpose of this article is to propose possible solutions to the methodological problem of null hypothesis significance testing (NHST), which is framed as deeply embedded in the institutional structure of the social and organizational sciences. The core argument is that, for the deinstitutionalization of statistical significance tests, minor methodological changes within an unreformed epistemology will be as unhelpful as emotive exaggerations of the ill effects of NHST. Instead, several institutional-epistemological reforms affecting cultural-cognitive, normative, and regulative processes and structures in the social sciences are necessary and proposed in this article. In the conclusion, the suggested research reforms, ranging from greater emphasis on inductive and abductive reasoning to statistical modeling and Bayesian epistemology, are classified according to their practical importance and the time horizon expected for their implementation. Individual-level change in researchers' use of NHST is unlikely if it is not facilitated by these broader epistemological changes.

Publication Bias in the Organizational Sciences

“…In the context of meta-analysis, this may include the use of multiple publication bias methods to estimate the possible range of results rather than relying on a single point estimate. According to Orlitzky (2012), such an approach may be pivotal in advancing the methodological rigor in the organizational sciences. Furthermore, this approach is aligned with customer-centric reporting of results as both researchers and practitioners benefit from understanding the robustness of a meta-analytic estimate (Aguinis et al, 2010).…”

Unpacking the Drivers of Corporate Social Performance: A Multilevel, Multistakeholder, and Multimethod Analysis

“…Hence, our overarching research question is aimed at addressing these multilevel issues. Our overall analytic approach, which is summarized in Hence, the focus of the present study is on the relative empirical importance of factors at different levels of analysis-a question that cannot be answered from the conventionally sizeless and binary hypothesis-testing perspective in the social sciences (Kline, 2004;Orlitzky, 2012;Schmidt, 1996;Ziliak & McCloskey, 2008). The research question implies that our perspective is not only descriptive, but also exploratory because so far no theory has emerged that would specify the magnitude of effect sizes with respect to the different sources of variability in CSP.…”

“…This means that, like seminal studies taking a similar analytic approach in strategic management-with a different outcome variable (e.g., Rumelt, 1991), our study focuses on the magnitude of effect sizes. 2 In general, such a descriptive focus on the magnitude of effect sizes has been recommended as methodological best practice (e.g., Cumming, 2012;Hunter, 1997;Kline, 2004;Orlitzky, 2012;Schmidt, 1992). However, most researchers currently eschew such an emphasis on effect size magnitude in favor of the binary outcomes of null-hypothesis significance tests (Schmidt, 1996;Ziliak & McCloskey, 2008).…”

Testing Point Null Hypothesis of a Normal Mean and the Truth: 21st Century Perspective

“…The most trenchant critics requested significance tests should be abandoned, banned or deinstitutionalized (e.g., Lindley, 1975;Hunter, 1997;Armstrong, 2007;Orlitzky, 2012). The editors of the American Journal of Public Health imposed a ban, although it only lasted two years.…”