Linear Confirmatory Factor Models to Evaluate Multitrait-Multimethod Matrices: The Effects of Number of Indicators and Correlation Among Methods

Abstract

Two models for confirmatory factor analysis of multitrait-multimethod data (MTMM) were assessed, the correlated traits-correlated methods (CTCM), and the correlated traits-correlated uniqueness models (CTCU). Two Monte Carlo experiments (100 replications per cell) were performed to study the behavior of these models in terms of magnitude and direction of bias, and accuracy of estimates. Study one included a single indicator per trait-method combination, and it manipulated three independent variables: matrix type, from three traits-three methods to six traits-six methods; correlation among method factors, from zero to .6; and model type (CTCM and CTCU). Study two included simulated MTMM matrices with two or more indicators per trait-method combination. Again, three independent variables were manipulated: number of indicators per trait-method combination, from 2 to 5; correlation among methods; and model type, CTCM and CTCU. The results from study one showed that the CTCU model performed very well for MTMM designs with a single indicator per trait-method combination, and consistently better than the CTCM model. However, the results from study two showed that the CTCM model worked reasonably well and better than the CTCU model when more than two indicators per trait-method combination were available. Despite the CTCM model's allowance for correlation between methods, results pointed to better estimates when methods were orthogonal. The main conclusion of the present article is that the use of CTCU models in the situations described in study one and the use of CTCM models in those represented in study two could be recommended.

Bias in the Correlated Uniqueness Model for MTMM Data

“…However, they found evidence of only small positive biases in the CU model (.02 and .047 on average for trait loadings and trait correlations, respectively) and concluded that those biases were trivial. A second simula-tion study found similarly small bias (Tomás, Hontangas, & Oliver, 2000 Despite these findings, we still believe there is the potential for substantial bias in trait estimates under the CU model, especially when there is substantial method variance shared between variables. Tomás et al (2000) were correct to point out the importance of method correlations, but Equations 1a and 1b show that what is really critical is the product of a method correlation and two method factor loadings.…”

“…A second simula-tion study found similarly small bias (Tomás, Hontangas, & Oliver, 2000 Despite these findings, we still believe there is the potential for substantial bias in trait estimates under the CU model, especially when there is substantial method variance shared between variables. Tomás et al (2000) were correct to point out the importance of method correlations, but Equations 1a and 1b show that what is really critical is the product of a method correlation and two method factor loadings. For example, if two methods are highly correlated, but a pair of variables each has a low loading on its measurement method factor, then the correlation between those two variables will not be inflated much (there is little common method bias).…”

“…In particular, to keep the simulation manageable, we included only two values for the sample size variable, N = 125 (i.e., the median value from our review) and N = 500. Although the latter value clearly exceeded the 75th percentile value from our review, we chose this value because it replicated the sample size value chosen by the two earlier simulation studies (Marsh & Bailey, 1991;Tomás et al, 2000). We also included only two values for the matrix size variable, a matrix with three traits and three methods and a MTMM matrix with five traits and three methods.…”

“…In summary, given (a) Lance et al's (2002) conceptual (algebraic) results that suggest that CU model estimates of trait loadings and correlations are potentially biased; (b) the Marsh and Bailey (1991) and Tomás et al (2000) simulation results suggesting that there is little, if any, bias in CU estimates; (c) the unknown generalizability of the previous simulations' results; (d) the relatively common reliance on the CU method to solve convergence problems associated with the CTCM method; and (e) the significance, for the validity of substantive research conclusions, of the assumption that CU parameter estimates are relatively free of bias, further investigation of the accuracy of CU model estimates is needed. Thus, the general purpose of this study was to determine the extent of bias in trait factor loadings and trait intercorrelations under the CU model for a variety of representative sets of MTMM model parameters, including a condition with high method correlations and high method loadings.…”

Evaluation of partial least squares parameter recovery

“…The population model used for this study includes three latent variables, an equal number of items per latent variable, and no cross-loadings. This model was selected for its simplicity and similarity to population models employed by previous research (e.g., Henseler, 2012;Hwang, Malhotra, et al, 2010;Paxton, Curran, bollen, Kirby, & Chen, 2001;Tomás, Hontangas, Oliver, 2000). The population model used for conditions featuring reflective indicator-latent variable relationships is identical to model used by Hwang, Malhotra, et al (2010) and Paxton et al (2001), and is displayed in Figure 1.…”