Inflated false-positive risk in common regression analyses: A combinatorial analysis of model sets

Abstract

Even with a small number of variables researchers can test many possible models of their data thus increasing the risk of false-positive results. Using combinatorics, we show that one key independent variable and three covariates can generate 95 possible models, while six covariates can generate over 2.3 million models. Such large model sets nearly guarantee false-positive results. Using simulation, we show that preregistering a single analysis with a key independent variable heavily reduces the risk of false-positives. However, even so, many models produce false-positive results with a much higher probability than the expected 5%. The worst-case scenario are models with interactions between binary dummy coded variables and omitted main effects. Such models can generate false-positive results up to 34.5% of the time. While preregistration is a crucial step towards reducing false-positive results, researchers need to carefully consider what analyses they plan and we provide recommendations for what analyses to avoid. Our findings also suggest that interpreting p-values in exploratory analyses might be meaningless considering the high false-positive probability.

‘Shaking the Ladder’ reveals how analytic choices can influence associations in nutrition epidemiology: beef intake and coronary heart disease as a case study

“…Indeed, a sample of 150 nutritional epidemiology studies found that 94% did not report a priori covariate selection, and only 20% reported the selection criteria for all covariates (17). Simulations have shown that flexibility in covariate selection can increase the chance of achieving statistical significance (10,(41)(42)(43)(44). Together, the lack of a theoretical framework for any of our varied models raises the question of where on the HR distribution a true causal association may reside.…”