The problem of effect size heterogeneity in meta-analytic structural equation modeling.

Abstract

Scholars increasingly recognize the potential of meta-analytic structural equation modeling (MASEM) as a way to build and test theory (Bergh et al., 2016). Yet, 1 of the greatest challenges facing MASEM researchers is how to incorporate and model meaningful effect size heterogeneity identified in the bivariate meta-analysis into MASEM. Unfortunately, common MASEM approaches in applied psychology (i.e., Viswesvaran & Ones, 1995) fail to account for effect size heterogeneity. This means that MASEM effect sizes, path estimates, and overall fit values may only generalize to a small segment of the population. In this research, we quantify this problem and introduce a set of techniques that retain both the true score relationships and the variability surrounding those relationships in estimating model parameters and fit indices. We report our findings from simulated data as well as from a reanalysis of published MASEM studies. Results demonstrate that both path estimates and overall model fit indices are less representative of the population than existing MASEM research would suggest. We suggest 2 extension MASEM techniques that can be conducted using online software or in R, to quantify the stability of model estimates across the population and allow researchers to better build and test theory. (PsycINFO Database Record

Transformational leadership and creativity: A meta‐analytic review and identification of an integrated model

“…Many researchers have expressed concern that the presence of heterogeneity in the effect sizes may pose serious problems for MASEM (Sheng, Kong, Cortina, & Hou, ; Yu, Downes, Carter, & O'Boyle, ; Cheung, forthcoming). Particularly, as Murphy () claims, if between‐study heterogeneity exists, we cannot be confident that the observed correlation matrix input in MASEM reflects reality.…”

Meta‐Analytic Research in Management: Contemporary Approaches, Unresolved Controversies, and Rising Standards

“…Finally, it is difficult to test moderator variables (Bergh et al, ). Fortunately, there is a rich literature emerging with suggestions how to handle these problems (Tang and Cheung, ; Yu et al, ).…”

Retraction: ‘The relationship of general mental ability to entrepreneurial firm performance: A meta‐analysis’ by David S. DeGeest, Scott Seibert and Ernest O'Boyle

“…A related statistic, the I 2 value, indicates the ratio of true heterogeneity to sampling error: A value of 0 for the I 2 would indicate that variability is solely due to R e t r a c t e d sampling error (Huedo-Medina et al, 2006). To test our incremental validity hypothesis, we used meta-analytic regression procedures (e.g., Zhao et al, 2010) combined with fullinformation meta-analytic structural equation modelling (FIMASEM; Yu, Downes, Carter, & O'Boyle, 2016), a type of meta-analytic SEM (Viswesvaran & Ones, 1995) that uses SD q estimates to assess credibility intervals for model parameter estimates. 5 To test our mediation hypothesis, we used FIMASEM's (Yu et al, 2016) estimates of indirect effect parameters.…”

“…To test our incremental validity hypothesis, we used meta-analytic regression procedures (e.g., Zhao et al, 2010) combined with fullinformation meta-analytic structural equation modelling (FIMASEM; Yu, Downes, Carter, & O'Boyle, 2016), a type of meta-analytic SEM (Viswesvaran & Ones, 1995) that uses SD q estimates to assess credibility intervals for model parameter estimates. 5 To test our mediation hypothesis, we used FIMASEM's (Yu et al, 2016) estimates of indirect effect parameters. Finally, we opted for a regression approach to test our moderators (Brinckmann et al, 2010;Geyskens et al, 2009).…”

“…These analyses show that GMA has a positive, statistically significant relation with entrepreneurial performance after controlling for the FFM personality traits (b = .33; CV b : [.27, .39]). We also conducted a relative weights analysis (LeBreton & Tonidandel, 2008), which showed GMA to be the most important predictor among the 5 The FIMASEM procedure (Yu et al, 2016) takes into account effect size heterogeneity by incorporating a theoretically and empirically robust assessment of true score variation in the effect sizes used in a meta-analytic SEM matrix. To do so, FIMASEM randomly sample values from the distribution of bivariate effect sizes, which are pooled into multiple effect size matrices.…”