Tests of significance: Some frequent misunderstandings.

Abstract

Many clinicians and other mental health professionals have difficulty in evaluating and fully comprehending the research literature relevant to their practice. This paper focuses on simple but powerful misunderstandings often attributed to the mainstay of research—significance-testing methodology. The limitations of the method are examined as a means toward better understanding of the contribution empirical research can make to clinical knowledge and practice.

The Statistical versus Clinical Significance Debate

“…Effect Size: As previously described, the use of Fisherian statistical significance testing leads to the loss of valuable information relating to the size of an effect. Often, readers of research reports erroneously assume that small p values necessarily imply large effects (Smith, 1983). However, small p values indicate only that results are less likely to have occurred by chance given the null hypothesis is true; they do not indicate the actual size of the observed effect.…”

How Not to Lie with Statistics: Avoiding Common Mistakes in Quantitative Political Science

“…This sort of methadological retrospective has been done in other disciplines, but although we can learn from some of these, most do not address problems specific enough to political science research. (See, for example, Leamer, 1983a;Smith, 1983;Friedman and Phillips, 198 1;and Hendry, 1980;Gurel, 1968).~ Over three decades ago, Darrell Huff (1954) explained, in a book by the same name, How to Lie With Statistics. Because of the systematic precision required, we should realize by now that it is a lot harder (knowingly or not) to lie (and get away with it) with statistics than without them.…”

Erratum

“…The MSI profiles indicated that there was substantive or clinically significant change (Kazdin, 1992;Smith, 1983) in relationship adjustment for the woman but not for her partner. Both appeared to be highly distressed according to pretreatment scores, which were in the moderately or seriously troubled range of couple adjustment for all subscales.…”