A Critical Review and Best‐Practice Recommendations for Control Variable Usage

Abstract

The use of control variables plays a central role in organizational research due to practical difficulties associated with the implementation of experimental and quasi‐experimental designs. As such, we conducted an in‐depth review and content analysis of what variables, and why such variables are controlled for, in 10 of the most popular research domains (task performance, organizational citizenship behaviors, turnover, job satisfaction, organizational commitment, employee burnout, personality, leader‒member exchange, organizational justice, and affect) in organizational behavior/human resource management (OB/HRM) and applied psychology. Specifically, we examined 580 articles published from 2003 to 2012 in AMJ, ASQ, JAP, JOM, and PPsych. Results indicate that, across research domains with clearly distinct theoretical bases, the overwhelming majority of the more than 3,500 controls identified in our review converge around the same simple demographic factors (i.e., gender, age, tenure), very little effort is made to explain why and how controls relate to focal variables of interest, and control variable practices have not changed much over the past decade. To address these results, we offer best‐practice recommendations in the form of a sequence of questions and subsequent steps that can be followed to make decisions on the appropriateness of including a specific control variable within a particular theoretical framework, research domain, and empirical study. Our recommendations can be used by authors as well as journal editors and reviewers to improve the transparency and appropriateness of practices regarding control variable usage.

A world of difference? The impact of corporate venture capitalists’ investment motivation on startup valuation

“…3.1 and supplemented it with additional information on startups' and CVCs' parent firms' SIC code classifications from the Thomson One database. We further followed Bernerth and Aguinis (2015) and Raudenbush and Bryk (2002) in limiting our predictor variables to those we considered most relevant. Table 3 provides an overview of the underlying variables and their respective definitions.…”

Science’s reproducibility and replicability crisis: International business is not immune

“…Control variables are used very frequently in management and organization studies (Becker, 2005;Carlson & Wu, 2012;Spector & Brannick, 2011). For example, control variables are used by entering them in a hierarchical manner when conducting multiple regression analyses, under the presumption that they eliminate contamination between the predictor and outcome variables (Bernerth & Aguinis, 2016). However, the assumptions and theoretical rationale underlying the use of control variables, namely, that including them provides a ''truer'' test of relations and that the controls used are measured reliably, are seldom tested (Bernerth & Aguinis, 2016).…”

“…For example, control variables are used by entering them in a hierarchical manner when conducting multiple regression analyses, under the presumption that they eliminate contamination between the predictor and outcome variables (Bernerth & Aguinis, 2016). However, the assumptions and theoretical rationale underlying the use of control variables, namely, that including them provides a ''truer'' test of relations and that the controls used are measured reliably, are seldom tested (Bernerth & Aguinis, 2016). Researchers rarely make explicit the reasons why certain variables (and not others) were chosen as controls (Becker, 2005;Spector & Brannick, 2011).…”

“…Researchers rarely make explicit the reasons why certain variables (and not others) were chosen as controls (Becker, 2005;Spector & Brannick, 2011). Finally, control variables reduce the statistical power of the test and the variance associated with the criterion that can potentially be explained by substantive variables (Breaugh, 2008), thereby increasing the chance that the results obtained are an artifact of the choice of control variables used (Bernerth & Aguinis, 2016). Control variables therefore increase systematic capitalization on chance as researchers test several models, including and excluding controls piecemeal until they obtain a desired result (Banks, O'Boyle et al, 2016;Bernerth & Aguinis, 2016;Simmons et al, 2011).…”

“…For example, researchers may include or delete outliers from a manuscript depending on which course of action results in a larger effect-size estimate (Aguinis, Gottfredson, & Joo, 2013). As a second illustration, researchers may capitalize on chance by selecting a particular configuration of control variables after analyzing the impact of several groups of control variables on results and selecting the final set based on which configuration results in better fit indices for a favored model (Bernerth & Aguinis, 2016).…”