Shaping Up the Practice of Null Hypothesis Significance Testing

Abstract

Recent criticisms of null hypothesis significance testing (NHST) have appeared in education and psychology research journals (e.g., Cohen, 1990 , 1994 ; Kupfersmid, 1988 ; Rosenthal, 1991 ; Rosnow & Rosenthal, 1989; Shaver, 1985 ; Sohn, 2000 ; Thompson, 1994 , 1997 ; see also Research in the Schools [1998]). In this article we discuss these criticisms for both current use of NHST and plausible future use. We suggest that the historical use of such procedures was reasonable and that current users might spend time profitably reading some of Fisher’s applied work. However, we also believe that modifications to NHST and to the interpretations of its outcomes might better suit the needs of modern science. Our primary conclusion is that NHST is most often useful as an adjunct to other results (e.g., effect sizes) rather than as a stand-alone result. We cite some examples, however, where NHST can be profitably used alone. Last, we find considerable experimental support for a less rigid attitude toward the interpretation of the probability yielded from such procedures.

What is the probability of replicating a statistically significant effect?

“…Although some details of this procedure and its rationale differ slightly between the Fisher and Neyman-Pearson schools of inference (see, e.g., Batanero, 2000;Huberty & Pike, 1999), these common features characterize the behavior of practicing researchers, and the differences are not important for the present purposes (cf. Wainer & Robinson, 2003).…”

“…Scientific theories are built on replicable phenomena (see, e.g., Falk, 1998;Guttman, 1977;Tukey, 1969;Wainer & Robinson, 2003). In sciences with deterministic measurements, the idea of replication is simple: If two researchers measure the same phenomenon using the same instruments and procedures, they should obtain essentially the same results.…”

Factors Influencing the Adoption of ICT by Teachers in Primary Schools in Saudi Arabia

“…After the analysis, the model was evaluated using the co-efficient of determination (R2), estimation of path coefficients (β), and prediction relevance (Q²) [72]. Finally, the model relationship analysis was carried out through alternative hypothesis testing using the p-values [85], [86].…”

In (Partial) Defense of .05

“…That concept (along with the probability of making a Type II error) was yet to come in the Neyman-Pearson approach to hypothesis testing. Also yet to come were several acrimonious arguments between Fisher and W. S. Gosset (who had previously developed the t-test), between Fisher and Karl Pearson, and between Fisher and both Jerzy Neyman and Egon Sharpe Pearson (Karl's son), as documented by Fienberg and Tanur (1966), Cowles andDavis (1982), Inman (1994), Wainer and Robinson (2003), and others.…”