Meta-Analytic Choices and Judgment Calls: Implications for Theory Building and Testing, Obtained Effect Sizes, and Scholarly Impact

Abstract

The authors content analyzed 196 meta-analyses including 5,581 effect-size estimates published in Academy of Management Journal, Journal of Applied Psychology, Journal of Management, Personnel Psychology, and Strategic Management Journal from January 1982 through August 2009 to assess the presumed effects of each of 21 methodological choices and judgment calls on substantive conclusions. Results indicate that, overall, the various meta-analytic methodological choices available and judgment calls involved in the conduct of a meta-analysis have little impact on the resulting magnitude of the meta-analytically derived effect sizes. Thus, the present study, based on actual meta-analyses, casts doubt on previous warnings, primarily based on selective case studies, that judgment calls have an important impact on substantive conclusions. The authors also tested the fit of a multivariate model that includes relationships among theory-building and theory-testing goals, obtained effect sizes, year of publication of the meta-analysis, and scholarly impact (i.e., citations per year). Results indicate that the more a meta-analysis attempts to test an existing theory, the larger the number of citations, whereas the more a meta-analysis attempts to build new theory, the lower the number of citations. Also, in support of scientific particularism, as opposed to scientific universalism, the magnitude of the derived effects is not related to the extent to which a meta-analysis is cited. Taken together, the results provide a comprehensive data-based understanding of how meta-analytic reviews are conducted and the implications of these practices for theory building and testing, obtained effect sizes, and scholarly impact.

Paradox Research in Management Science: Looking Back to Move Forward

“…Some principles might need more basic theory development, whereby a grounded-theory approach could help (Wareham, Fox, & Cano Giner, 2014). Finally, meta-analytical techniques could be used to formally test meta-theoretical principles (Aguinis, Dalton, Bosco, Pierce, & Dalton, 2011).…”

An Ounce of Prevention Is Worth a Pound of Cure: Improving Research Quality Before Data Collection

“…An important challenge faced by organizational science researchers is that measures of latent constructs are fallible (Schmidt 1992), and because they are only imperfect indicators of underlying constructs, observed relationships between constructs are usually smaller than the actual relationships. In fact, Aguinis et al (2011) conducted a review of almost 6,000 correlation coefficients reported in the Academy of Management Journal, Journal of Applied Psychology, Journal of Management, Personnel Psychology, and Strategic Management Journal from 1982 through August 2009 and found that the median absolute effect across all types of bivariate relationships is only .23. Interestingly, this value is very close to .20, which is what Lykken (1968) guessed was the expected correlation between any two not necessarily related variables in psychological research.…”

Scholarly Impact Revisited

“…Leung (2007) provided an incisive analysis of scholarly impact from an international-and particularly East Asianperspective. Aguinis, Dalton, Bosco, Pierce, and Dalton (2011) assessed whether meta-analyses that focus on theory building have more impact than meta-analyses that focus on theory testing. And Molina-Azorin (2012) examined whether certain methodological approaches have more impact than others.…”