Practical Applications of Generalizability Theory for Designing, Evaluating, and Improving Psychological Assessments

Vispoel, Walter P.; Morris, Carrie A.; Kılınç, Murat

doi:10.1080/00223891.2017.1296455

Cited by 35 publications

(53 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our study provides preliminary evidence that the use of referents can improve the psychometric properties of PME measures [1]. Future studies should investigate the effects of specifying different types of referents on the psychometric properties of PME scales using explanatory item response modeling [54] or generalizability theory [55].…”

Section: Discussionmentioning

confidence: 77%

UNC Perceived Message Effectiveness: Validation of a Brief Scale

Baig

Noar

Gottfredson

et al. 2018

Annals of Behavioral Medicine

View full text Add to dashboard Cite

Background Interventionists commonly identify promising messages for health communication efforts based on audience members’ ratings of perceived message effectiveness (PME). Purpose We sought to validate a new PME measure that improved on existing scales by focusing on the behavior and respondent, being brief, and having strong psychometric properties. Methods Participants were a national convenience sample of 999 adults and national probability samples of 1,692 adults and 869 adolescents recruited in 2015. Smokers and nonsmokers rated up to six brief messages about the chemicals in cigarette smoke on two PME scales. The first was the new three-item University of North Carolina (UNC) PME Scale that assessed effects perceptions. The second was an established six-item PME scale that assessed message perceptions. We examined the UNC PME Scale’s psychometric properties and compared both scales using item factor analysis. Results The UNC PME Scale measured the same construct across multiple chemical messages (all factor loadings ≥ 0.86). It exhibited high reliability (>0.85) over very low to moderate levels of PME (z = −2.5 to 0.2), a range that is useful for identifying more promising messages. Samples of adults and adolescents showed a similar pattern of results. As expected, the UNC PME Scale was strongly positively correlated with message perceptions (r = .84). It also exhibited strong psychometric properties among participants regardless of education, reactance, sex, and smoking status. Discussion The UNC PME Scale reliably and validly measured PME among adults and adolescents from diverse groups. This brief scale may be used to efficiently evaluate candidate antismoking messages and may be suitable for adaptation to other health risk behaviors.

show abstract

Section: Discussionmentioning

confidence: 77%

UNC Perceived Message Effectiveness: Validation of a Brief Scale

Baig

Noar

Gottfredson

et al. 2018

Annals of Behavioral Medicine

View full text Add to dashboard Cite

show abstract

“…Generalizability theory [GT] [2,3] extends classical test theory [CTT] [4] by differentiating between a (theoretically) unlimited number of independent sources of measurement error, rather than a single conflated error term. Beyond its value for scale development [5], the versatility of GT has been demonstrated by applications in a wide variety of psychological domains, such as clinical/counseling [6][7][8], personality assessment [9,10], and education sciences [11][12][13]. GT can simultaneously quantify reliability of several types that are of common interest in psychological disciplines, including scale reliability, test-retest reliability, and interrater reliability (IRR).…”

Section: Introductionmentioning

confidence: 99%

How to Estimate Absolute-Error Components in Structural Equation Models of Generalizability Theory

Jorgensen

2021

Psych

View full text Add to dashboard Cite

Structural equation modeling (SEM) has been proposed to estimate generalizability theory (GT) variance components, primarily focusing on estimating relative error to calculate generalizability coefficients. Proposals for estimating absolute-error components have given the impression that a separate SEM must be fitted to a transposed data matrix. This paper uses real and simulated data to demonstrate how a single SEM can be specified to estimate absolute error (and thus dependability) by placing appropriate constraints on the mean structure, as well as thresholds (when used for ordinal measures). Using the R packages lavaan and gtheory, different estimators are compared for normal and discrete measurements. Limitations of SEM for GT are demonstrated using multirater data from a planned missing-data design, and an important remaining area for future development is discussed.

show abstract

“…G theory provides a multifaceted approach for estimating score reliability (Brennan, 2001(Brennan, , 2010Cronbach, Gleser, Nanda, & Rajaratnum, 1972;Shavelson & Webb, 1991;Shavelson, Webb, & Rowley, 1989;Vispoel, Morris, & Kilinc, 2018a, 2018bWebb, Shavelson, & Haertel, 2006). The application of G theory to psychophysiological research and its advantages over classical test theory for ERP research have been described elsewhere (Baldwin et al, 2015;Clayson et al, 2020;Clayson & Miller, 2017a, 2017b.…”

Section: Conceptual Frameworkmentioning

confidence: 99%

Using generalizability theory and the ERP Reliability Analysis (ERA) Toolbox for assessing test-retest reliability of ERP scores part 1: Algorithms, framework, and implementation

Clayson

Carbine

Baldwin

et al. 2021

International Journal of Psychophysiology

View full text Add to dashboard Cite

The reliability of event-related brain potential (ERP) scores depends on study context and how those scores will be used, and reliability must be routinely evaluated. Many factors can influence ERP score reliability; generalizability (G) theory provides a multifaceted approach to estimating the internal consistency and temporal stability of scores that is well suited for ERPs. G-theory's approach possesses a number of advantages over classical test theory that make it ideal for pinpointing sources of error in scores. The current primer outlines the G-theory approach to estimating internal consistency (coefficients of equivalence) and test-retest reliability (coefficients of stability). This approach is used to evaluate the reliability of ERP measurements.The primer outlines how to estimate reliability coefficients that consider the impact of the number of trials, events, occasions, and groups. The uses of two different G-theory reliability coefficients (i.e., generalizability and dependability) in ERP research are elaborated, and a dataset from the companion manuscript, which examines N2 amplitudes to Go/NoGo stimuli, is used as an example of the application of these coefficients to ERPs. The developed algorithms are implemented in the ERP Reliability Analysis (ERA) Toolbox, which is open-source software designed for estimating score reliability using G theory. The toolbox facilitates the application of G theory in an effort to simplify the study-by-study evaluation of ERP score reliability. The formulas provided in this primer should enable researchers to pinpoint the sources of measurement error in ERP scores from multiple recording sessions and subsequently plan studies that optimize score reliability.

show abstract

Practical Applications of Generalizability Theory for Designing, Evaluating, and Improving Psychological Assessments

Cited by 35 publications

References 40 publications

UNC Perceived Message Effectiveness: Validation of a Brief Scale

UNC Perceived Message Effectiveness: Validation of a Brief Scale

How to Estimate Absolute-Error Components in Structural Equation Models of Generalizability Theory

Using generalizability theory and the ERP Reliability Analysis (ERA) Toolbox for assessing test-retest reliability of ERP scores part 1: Algorithms, framework, and implementation

Contact Info

Product

Resources

About