PERSPECTIVE—Researchers Should Make Thoughtful Assessments Instead of Null-Hypothesis Significance Tests

Abstract

N ull-hypothesis significance tests (NHSTs) have received much criticism, especially during the last two decades. Yet many behavioral and social scientists are unaware that NHSTs have drawn increasing criticism, so this essay summarizes key criticisms. The essay also recommends alternative ways of assessing research findings. Although these recommendations are not complex, they do involve ways of thinking that many behavioral and social scientists find novel. Instead of making NHSTs, researchers should adapt their research assessments to specific contexts and specific research goals, and then explain their rationales for selecting assessment indicators. Researchers should show the substantive importance of findings by reporting effect sizes and should acknowledge uncertainty by stating confidence intervals. By comparing data with naïve hypotheses rather than with null hypotheses, researchers can challenge themselves to develop better theories. Parsimonious models are easier to understand, and they generalize more reliably. Robust statistical methods tolerate deviations from assumptions about samples.

Good Night, and Good Luck: Perspectives on Luck in Management Scholarship

“…Second, management researchers should incorporate randomness into theory building in order to develop stronger null models when examining hypotheses (Schwab, Abrahamson, Starbuck, & Fidler, 2011;Starbuck, 1994). Explanations relying on randomness might seem unfalsifiable: one could always claim that something was due to chance, but how could such a statement ever be tested?…”

“…Explanations relying on randomness might seem unfalsifiable: one could always claim that something was due to chance, but how could such a statement ever be tested? By making parsimonious assumptions that there is no difference among actors, these 'naïve models' usually make more rigorous and detailed predictions than other theories (Schwab et al, 2011). Many theories in management only make point predictions about the sign of a coefficient in a regression (e.g., we theorize that the effect of x on y is positive...).…”

A Call for Openness in Research Reporting: How to Turn Covert Practices Into Helpful Tools

“…The public and legal institutions regard pharmaceutical companies as acting unethically when they suppress tests that show drugs to have weak or no effects. As well, statistical significance is generally an unreliable indicator of the importance of phenomena because it takes no account of costs or benefits for different stakeholders (Schwab et al 2011;Hubbard, 2015).…”

“…These practices both distort evidence about the usefulness of theories and undermine confidence in the conclusions reported (as summarized in Table 1). The undesirable properties of these practices are wellestablished in the social sciences and the corresponding methods literature (Banks, Rogelberg, et al, 2016;Kepes, Bennett & McDaniel, 2014;Landis & Rogelberg, 2013;Schwab et al, 2011;Simmons et al 2011;Starbuck, 2016a).…”

Rare, outlier and extreme: beyond the Gaussian model and measures

“…Methods for statistical analyses involving rare, outlier and extremes are abundant and lie beyond the scope of this paper; for examples, see Beirlant et al (2004), Coles (2001), Resnick (2007), and Schwertman and de Silva (2007). Also see McKelvey and Andriani (2005), Andriani and McKelvey (2007), and Schwab et al (2011).…”