Sound and fury: McCloskey and significance testing in economics

Abstract

For more than 20 years, Deidre McCloskey has campaigned to convince the economics profession that it is hopelessly confused about statistical significance. She argues that many practices associated with significance testing are bad science and that most economists routinely employ these bad practices: 'Though to a child they look like science, with all that really hard math, no science is being done in these and 96 percent of the best empirical economics …' (McCloskey 1999). McCloskey's charges are analyzed and rejected. That statistical significance is not economic significance is a jejune and uncontroversial claim, and there is no convincing evidence that economists systematically mistake the two. Other elements of McCloskey's analysis of statistical significance are shown to be ill-founded, and her criticisms of practices of economists are found to be based in inaccurate readings and tendentious interpretations of those economists' work. Properly used, significance tests are a valuable tool for assessing signal strength, for assisting in model specification, and for determining causal structure.Deidre McCloskey, Stephen Ziliak, statistical significance, economic significance, significance tests, R.A. Fisher, Neyman-Pearson testing, specification search, C10, C12, B41,

Lopsided Reasoning on Lopsided Tests and Multiple Comparisons

“…Like much of the criticism of significance tests over the last half-century, the book blames significance tests themselves for their misuse and misinterpretation by scientists and statisticians. The book is nevertheless worth a read, especially if tempered by perusal of some the strong criticisms of it that have already appeared (Hoover & Siegler 2008;Spanos 2008;). The abundant errors and hyperbolic language in Ziliak & McCloskey (2008) are especially unfortunate in that they distract attention from the two most important and worthwhile messages in the book.…”

Statistical vs. economic significance in economics and econometrics: further comments on McCloskey and Ziliak

“…Underlying McCloskey and Ziliak's claim is a survey analysis of practice in empirical papers published in the American Economic Review during the 1980s and 1990s (see McCloskey and Ziliak 1996;Ziliak and McCloskey 2004). Hoover and Siegler (2008a) criticize McCloskey and Ziliak for erroneously omitting a significant number of relevant articles from the American Economic Review in their two 'questionnaire' surveys, and for inaccurate readings and tendentious interpretations of the articles included. In addition, McCloskey and Ziliak's surveys can be criticized for containing only papers that use statistical tests in connection with regression analysis.…”

“…In any case: how do we secure that our parameter estimates are well defined and have 'good' properties? Hoover and Siegler (2008a) provide further discussion of this issue, and they also criticize McCloskey and Ziliak for sweeping under the carpet potential problems associated with parameter measurement and estimation. 8 As noted in the previous section, McCloskey and Ziliak emphasize many times that 'real error' is more important than pure sampling error.…”

“…The March 2008 issue of Journal of Economic Methodology contains an interesting discussion between Kevin Hoover and Mark Siegler on the one hand, and Deirdre McCloskey and Stephen Ziliak on the other hand, about statistical vs. economic significance in economic research (Hoover andSiegler 2008a, 2008b;McCloskey and Ziliak 2008). The pivot of the discussion is McCloskey and Ziliak's long-lasting critisism of the standard practice of statistical significance testing in applied economic research and in particular their claim that the majority of applied economists and econometricians have failed -and continue to fail -in distinguishing between statistical and economic significance.…”

Understanding Statistical Testing

“…A recent book and related works by Ziliak and McCloskey (McCloskey & Ziliak, 1996, 2008; Ziliak & McCloskey, 2004a, 2004b, 2008a, 2008b) declare statistical significance is invalid for scientific inquiry. Critics responded (Engsted, 2009; Hoover & Siegler, 2008a, 2008b; Spanos, 2008). Ziliak and McCloskey, and their critic Spanos, imply the raw-scale magnitudes of parameters are relevant to hypothesis testing.…”