Using generalizability theory to evaluate the comparative reliability of developmental measures in neurogenetic syndrome and low-risk populations

Abstract

Background: The lack of available measures that can reliably characterize early developmental skills in children with neurogenetic syndromes (NGS) poses a significant challenge for research on early development in these populations. Although syndrome-specific measures may sometimes be necessary, a more cost-and time-efficient solution would be to identify existing measures that are appropriate for use in special populations or optimize existing measures to be used in these groups. Reliability is an important metric of psychometric rigor to consider when auditing and optimizing assessment tools for NGS. In this study, we use Generalizability Theory, an extension of classical test theory, as a novel approach for more comprehensively characterizing the reliability of existing measures and making decisions about their use in the field of NGS research. Methods: We conducted generalizability analyses on a popular early social communication screener, the Communication and Symbolic Behavior Scales-Infant-Toddler Checklist (CSBS-ITC), collected on 172 children (41 Angelman syndrome, 30 Prader-Willi syndrome, 42 Williams syndrome, 59 low-risk controls). Results: Overall, the CSBS-ITC demonstrated at least adequate reliability in the NGS groups included in this study, particularly for the Prader-Willi and Williams syndrome groups. However, the sources of systematic error variance in the CSBS-ITC varied greatly between the low-risk control and NGS groups. Moreover, as unassessed in previous research, the CSBS-ITC demonstrated substantial differences in variance sources among the NGS groups. Reliability of CSBS-ITC scores was highest when averaging across all measurement points for a given child and was generally similar or better in the NGS groups compared to the low-risk control group. Conclusions: Our findings suggest that the CSBS-ITC communicates different information about the reliability of stability versus change, in low-risk control and NGS samples, respectively, and that psychometric approaches like Generalizability Theory can provide more complete information about the reliability of existing measures and inform decisions about how measures are used in research on early development in NGS.

Development, reliability and validity of infectious disease specialist Nurse’s Core competence scale

“…And In the next step, we will analyze the invariance of cross group measurement to figure out the differences in the application in different groups. Besides, we will introduce Generalizability Theory (GT) and apply it in the re-evaluation of the scale so that we can further verify and improve the reliability and validity of the scale [ 35 ].…”

Development and Validation of the Breast Cancer Scale QLICP-BR V2.0 Based on Classical Test Theory and Generalizability Theory

“…It is applied in quantitative research to analyze the influence of patients, items, and interactions between patients and items on the total score of the scale. GT provides a comprehensive and unifying framework that goes beyond the CTT model of a single error term by allowing for the simultaneous analysis of the main and interaction effect source of error variance ( 23 , 24 ). GT subsumes other forms of reliability approaches ( e .…”

Measuring affect in daily life: A multilevel psychometric evaluation of the PANAS-X across four ecological momentary assessment samples.

“…The next step in a GT framework is the “D Study” or “decision” study, which estimates reliability as a proportion of variance using the variance components from the G study. GT can produce multiple forms of between- and within-person reliability estimates that are based on researcher-defined variance components, and can flexibly accommodate multiple nested (e.g., day of study within person) and crossed (e.g., developmental age range; Hamrick et al, 2020) components of variance. The reliability estimates produced (commonly including R k F , R k R , R k RN , R C , R CN ) provide information about the circumstances under which a measure may be reliable.…”