Statistical vs. economic significance in economics and econometrics: further comments on McCloskey and Ziliak

Abstract

I comment on the controversy between McCloskey and Ziliak and Hoover and Siegler on statistical versus economic significance, in the March 2008 issue of the Journal of Economic Methodology. I argue that while McCloskey and Ziliak are right in emphasizing 'real error', i.e. non-sampling error that cannot be eliminated through specification testing, they fail to acknowledge those areas in economics, e.g. rational expectations macroeconomics and asset pricing, where researchers clearly distinguish between statistical and economic significance and where statistical testing plays a relatively minor role in model evaluation. In these areas models are treated as inherently misspecified and, consequently, are evaluated empirically by other methods than statistical tests. I also criticize McCloskey and Ziliak for their strong focus on the size of parameter estimates while neglecting the important question of how to obtain reliable estimates, and I argue that significance tests are useful tools in those cases where a statistical model serves as input in the quantification of an economic model. Finally, I provide a specific example from economics - asset return predictability - where the distinction between statistical and economic significance is well appreciated, but which also shows how statistical tests have contributed to our substantive economic understanding.statistical and economic significance, statistical hypothesis testing, model evaluation, misspecified models,

Intellectual Capital and Financial Performance: A Meta-Analysis and Research Agenda

“…Indeed, scholars should not be interested only in whether there is an effect (tested by statistical significance) but in how big this effect is (tested by economic significance). However, establishing an order of magnitude in management studies poses significant challenges, as it is difficult to obtain good reliable parameter estimates (Engsted, 2009). This is probably why economic significance is barely discussed in meta-analysis in the strategy or management fields (Doucouliagos & Laroche, 2003).…”

Ceremonial science: The state of Russian economics seen through the lens of the work of ‘Doctor of Science’ candidates

“…All in all the positive spillover effects 45 Different points of view on this topic are discussed in Altman 2004;DeLong and Lang 1992;Engsted 2009;Gigerenzer 2004;Hoover and Siegler 2008;Kraemer 2011;Mayer 2012;McCloskey and Ziliak 1996;Ziliak and McCloskey 2004;2008. generated by the island of excellency established after 1990 seem to be rather limited.…”

Significance Testing in Accounting Research: A Critical Evaluation Based on Evidence

“…Second, the use of the conventional level of significance ‘mechanically and thoughtlessly in each and every application’ is meaningless, as Engsted (, p. 401) points out. This is particularly so when the sample size is large or massive, where an acute imbalance between Type I and Type II error probabilities is expected.…”