An Examination of G-Theory Methods for Modeling Multitrait–Multimethod Data

Abstract

For nearly three decades, the predominant approach to modeling the latent structure of multitraitmultimethod (MTMM) data in organizational research has involved confirmatory factor analysis (CFA). Despite the frequency with which CFA is used to model MTMM data, commonly used CFA models may produce ambiguous or even erroneous results. This article examines the potential of generalizability theory (G-theory) methods for modeling MTMM data and makes such methods more accessible to organizational researchers. Although G-theory methods have existed for more than half a century, the research literature has yet to provide a clear description and integration of latent models implied by univariate and multivariate G-theory with MTMM data, notions of construct validity, and CFA. To help fill this void, the authors first provide a jargon-free overview of the univariate and multivariate G-theory models and analytically demonstrate linkages between their parameters (variance and covariance components), elements of the MTMM matrices, indices of convergent and discriminant validity, and CFA. The authors conclude with a discussion and empirical illustration of a G-theory-based modeling process that helps clarify the use of G-theory methods for modeling MTMM data.

The internal structure of situational judgement tests reflects candidate main effects: Not dimensions or situations

“…From a covariance perspective, candidate-by-situation (in dimension) interactions reflect between-candidate variance that is specific to a given situation (nested in dimensions) and not variance shared with other situations (nested in dimensions) or variance shared with general judgement. This effect is analogous to CFAbased situation effects (e.g., Westring et al, 2009;Woehr, Putka, et al, 2012). If the situation perspective holds true, then relatively large candidate-by-situation (in dimension) interactions should be evident in SJT scores.…”

“…Secondly, why do dimension-and situation-related effects explain such little variance in SJT responses? On the first question, candidate main effects in SJTs summarize a general judgement factor and the covariance between any underlying dimension and situation factors, if such factors hold psychological meaning (Woehr, Putka, et al, 2012). Thus, a candidate main effect should neither be confused with g in the Spearman tradition (e.g., Gonzalez-Mul e et al, 2014) nor with the first unrotated factor in a PCA (see Jackson et al, 2015;Lance & Jackson, 2015).…”

“…From a covariance perspective, candidate-by-dimension interactions reflect between-candidate variance that is specific to a given dimension and not variance shared with other dimensions or variance shared with general judgement (Putka & Hoffman, 2013). This effect is analogous to the dimension factors typically estimated using CFA (Woehr, Putka, et al, 2012) and is, thus, analogous to the CFA-based dimension effects estimated by Westring et al (2009). If the dimension perspective holds true, then relatively large candidate-by-dimension interactions should be evident in SJTs scores.…”

“…In particular, the assessment centre (AC) literature has grappled with analogous dimensions and their contribution to AC ratings for over six decades (see Sakoda, 1952). While the AC context is different from that presented by SJTs, the AC literature has, nonetheless, utilized innovative analytic approaches to help inform on complex psychometric designs (e.g., Jackson, Michaelides, Dewberry, & Kim, 2016;Woehr, Meriac, & Bowler, 2012;Woehr, Putka, & Bowler, 2012).…”

“…In SJTs, candidate (or person) main effects (r 2 c ) imply that some candidates generally make 'better' judgements than others regardless of the dimension, situation, or response item involved. However, this effect not only summarizes a unidimensional general judgement factor (e.g., Schmitt & Chan, 2006), but also reflects covariation between psychological constructs underlying any dimensions involved in the assessment as well as covariation between situational effects, if such effects have substantive psychological meaning (Meyer, Dalal, & Hermida, 2010;Putka & Hoffman, 2013;Woehr, Putka, et al, 2012). Candidate main effects, with respect to SJTs, are analogous to a general judgement factor, but should not be confused with general mental ability/g (e.g., Gonzalez-Mul e, Mount, & Oh, 2014) or with a dominant general factor as generated through principal components analysis (PCA; see Jackson et al, 2015;Lance & Jackson, 2015).…”