Editorial: Errors in the Variables, Unobserved Heterogeneity, and Other Ways of Hiding Statistical Error

Abstract

O ne research function is proposing new scientific theories; another is testing the falsifiable predictions of those theories. Eventually, sufficient observations reveal valid predictions. For the impatient, behold statistical methods, which attribute inconsistent predictions to either faulty data (e.g., measurement error) or faulty theories.Testing theories, however, differs from estimating unknown parameters in known relationships. When testing theories, it is sufficiently dangerous to cure inconsistencies by adding observed explanatory variables (i.e., beyond the theory), let alone unobserved explanatory variables. Adding ad hoc explanatory variables mimics experimental controls when experiments are impractical. Assuming unobservable variables is different, partly because realizations of unobserved variables are unavailable for validating estimates.When different statistical assumptions about error produce dramatically different conclusions, we should doubt the theory, the data, or both. Theory tests should be insensitive to assumptions about error, particularly adjustments for error from unobserved variables. These adjustments can fallaciously inflate support for wrong theories, partly by implicitly under-weighting observations inconsistent with the theory. Inconsistent estimates often convey an important message-the data are inconsistent with the theory! Although adjustments for unobserved variables and ex post information are extraordinarily useful when estimating known relationships, when testing theories, requiring researchers to make these adjustments is inappropriate.

The statistical analysis of crash-frequency data: A review and assessment of methodological alternatives

“…Because each observation has its own parameters, the final model will often provide a statistical fit that is significantly better than a model with traditional fixed parameters. However, random-parameter models are very complex to estimate, they may not necessarily improve predictive capability, and model results may not be transferable to other data sets because the results are observation specific (see Shugan, 2006;Washington et al, 2010).…”

Editorial—The Anna Karenina Bias: Which Variables to Observe?

“…With a strong deductive theory that involves knowledge well beyond the observed data we can predict the nature and direction of the potential bias. Statistical technologies are very good at estimating unknown parameters in known relationships (Shugan 2006). However, those technologies are less helpful for determining which variables to observe and what relationships to expect.…”

Editorial—It’s the Findings, Stupid, Not the Assumptions

“…Most often, assumptions are sufficient conditions that guarantee the validity of the subsequent findings but whose violation by no means necessarily invalidates those findings. Published research often assumes that random error causes unpredicted outcomes, rather than wrong theory (Shugan 2006). Published research often assumes that results are invariant to the time, place, and sample.…”