Internal consistency reliability in measurement: Aggregate and multilevel approaches

Abstract

The purpose of this study was to evaluate the internal consistency reliability of the General Teacher Test assuming clustered and non-clustered data using commercial software (Mplus). Participants were 2,000 testees who were selected using random sampling from a larger pool of examinees (more than 65k). The measure involved four factors, namely: (a) planning for learning, (b) promoting learning, (c) supporting learning, and (d) professional responsibilities, and was hypothesized to comprise a unidimensional instrument assessing generalized skills and competencies. Intra-class correlation coefficients and variance ratio statistics suggested the need to incorporate a clustering variable (i.e., university) when evaluating the factor structure of the measure. Results indicated that single level reliability estimation significantly overestimated the reliability observed across persons and underestimated the reliability at the clustering variable (university). One level reliability was also, at times, lower than the lowest acceptable levels leading to a conclusion of unreliability whereas multilevel reliability was low at the between person level but excellent at the between university level. It was concluded ignoring nesting is associated with distorted and erroneous estimates of internal consistency reliability of an ability measure and the use of MCFA is imperative to account for dependencies between levels of analyses.

Application of multilevel confirmatory factor analysis to compositional organizational constructs

“…This new factor structure warrants further investigation in the context of engagement literature. Finally, recent studies have highlighted that reliability scores are also affected due to clustering of data, and ignoring this clustering can lead to faulty reliability coefficients (Lai, 2021;Sideridis et al, 2018). This can be seen in single-level CFA, where aggregate ω values are much higher than the adjusted, true level-specific ω w and ω b .…”

Versión breve del Cuestionario Revisado de Proceso de Estudio-2 Factores: Un análisis psicométrico en estudiantes universitarios

“…When comparing the internal consistency indices in the whole sample and by country, significant differences were verified for the Deep approach in every case -Omega, ordinal Alpha, and H-, whereas the Surface approach showed a unique difference regarding the H index. Such differences could be attributed, again, to the lack of control in demographic and academic variables (Sideridis, Saddaawi & Al-Harbi, 2018). Further research should analyze and identify variables that may affect the scale's internal consistency.…”

Measurement of parental executive function in early childhood settings: Instrument reliability and validity in community‐led research projects

“…There are a number of limitations to the present study. First, though multilevel approaches to calculating Cronbach's α and EFAs are available using MPlus (e.g., Huang & Cornell, 2016; Sideridis, Saddaawi, & Al‐Harbi, 2018), guidance for fixed effects approaches were neither available for these analyses nor one sample t test. Hence, internal consistency reliability, EFAs, and one sample t‐tests did not account for clustering effects.…”