Rationality in psychological research: The good-enough principle.

Abstract

" This article reexamines a number of methodological and procedural issues raised by Meehl (1967, 1978) that seem to question the rationality of psychological inquiry. The first issue concerns the asymmetry in theory testing between psychology and physics and the resulting paradox that, because the psychological null hypothesis is always false, increases in precision in psychology always lead to weaker tests of a theory, whereas the converse is true in physics. The second issue, related to the first, regards the slow progress observed in psychological research and the seeming unwillingness of social scientists to take seriously the Popperian requirements for intellectual honesty. We propose a good-enough principle to resolve Meehl's methodological paradox and appeal to a more powerful reconstruction of science developed by Lakatos (1978a, 1978b) to account for the actual practice of psychological researchers. From time to time every research discipline must reevaluate its method for generating and certifying knowledge. The actual practice of working scientists in a discipline must continually be subjected to severe criticism and be held accountable to standards of intellectual honesty, standards that are themselves revised in light of critical appraisal (Lakatos, 1978a). If, on a metatheoretical level, scientific methodology cannot be defended on rational grounds, then meta-theory must be reconstructed so as to make science rationally justifiable. The history of science is replete with numerous such reconstructions, from the portrayal of science as being inductive and justification-ist, to the more recent reconstructions favored by (naive and sophisticated) methodological falsifica-tionists, such as Popper (1959), Lakatos (1978a), and Zahar (1973). In the last two decades psychology, too, has been subjected to criticism for its research methodology. Of increasing concern is empirical psychology's use of inferential hypothesis-testing techniques and the way in which the information derived from these procedures is used to help us make decisions about the theories under test (e.g., Bakan, 1966; Lykken, 1968; Rozeboom, 1960). In two penetrating essays, Meehl (1967, 1978) has cogently and effectively faulted the use of the traditional null-hypothesis significance test in psychological research. According to Meehl (1978, p. 817), "the almost universal reliance on merely refuting the null hypothesis as the standard method for corroborating substantive theories [in psychology] is a terrible mistake, is basically unsound, poor scientific strategy, and one of the worst things that ever happened in the history of psychology." He maintained that it leads to a methodological paradox when compared to theory testing in physics. In addition, Meehl (1978) pointed to the apparently slow progress in psychological research and the deleterious effect that null-hypothesis testing has had on the detection of progress in the accumulation of psychological knowledge. The cumulative effect of this criticism is to do nothing less than cal...

The Bayesian New Statistics: Hypothesis testing, estimation, meta-analysis, and power analysis from a Bayesian perspective

“…ROPE's go by different names in the literature, including "interval of clinical equivalence," "range of equivalence," "equivalence interval," "indifference zone," "smallest effect size of interest," and "good-enough belt" (e.g. Carlin & Louis, 2009;Freedman, Lowe, & Macaskill, 1984;Hobbs & Carlin, 2008;Lakens, 2014;Schuirmann, 1987;Serlin & Lapsley, 1985Spiegelhalter, Freedman, & Parmar, 1994).…”

Advantages of Computer Programs over Power Charts for the Estimation of Power

“…A second reason is that researchers sometimes wish to retain null hypotheses in cases where treatment effects are small, given that what constitutes a small treatment effect has been specified by the investigator. This idea is mentioned briefly in Keppel (1991, p. 90) and is described in more detail in Greenwald (1975) and Serlin and Lapsley (1985). Specifying small treatment effects that are declared to be consistent with the null hypothesis entails knowing the power to detect such effects, which in turn requires the use of small values.…”

Beyond statistical inference: A decision theory for science

“…In a thoughtful analysis of the implications of our current prejudice against the null hypothesis, Greenwald (1975) suggested, inter alia, that the use of posterior distributions and range null hypotheses would increase the information transmitted by scientific reporting. The present analysis may exploit his suggestions by adding a row to Table 2 for accepting a "minimal effect," or nil hypothesis (Serlin & Lapsley, 1985). A second utility function would be overlaid on the function in Figure 1-presumably, an inverted U centered on 0, such as m |d| , with m measuring the utility accorded this alternative.…”