A Unified Comparison of IRT‐Based Effect Sizes for DIF Investigations

Chalmers, R. Philip

doi:10.1111/jedm.12347

Cited by 6 publications

(2 citation statements)

References 52 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This is advantageous for its use in PCtrees because it does not require to fit a psychometric model as part of the computation of γ, resulting in faster computation times (cf. Chalmers, 2022).…”

Section: Model-based Recursive Partitioning For Polytomous Datamentioning

confidence: 99%

Partial credit trees meet the partial gamma coefficient for quantifying DIF and DSF in polytomous items

Henninger,

Radek,

Sengewald

et al. 2024

Preprint

View full text Add to dashboard Cite

Partial credit trees (PCtree) from the model-based recursive partitioning framework combine the partial credit measurement model for polytomous items with decision trees from machine learning. This method allows researchers to investigate measurement invariance by detecting differential item and differential step functioning (DIF/DSF) in a data driven way. In this manuscript, we extend PCtrees by an effect size measure for DIF/DSF in polytomous items, the partial gamma coefficient from psychometrics. We evaluate this extension of PCtrees in a series of simulation studies. Our results show that the partial gamma coefficient supports researchers in evaluating whether splits in the tree are meaningful, identifying DIF and DSF items, and can stop the tree from growing in case of negligible effect sizes. Furthermore, we assess and implement a correction for item-wise testing that is particularly crucial in longer tests. Finally, we illustrate the extension of PCtrees using data from the LISS panel to showcase its enhanced interpretability.

show abstract

Section: Model-based Recursive Partitioning For Polytomous Datamentioning

confidence: 99%

Partial credit trees meet the partial gamma coefficient for quantifying DIF and DSF in polytomous items

Henninger,

Radek,

Sengewald

et al. 2024

Preprint

View full text Add to dashboard Cite

show abstract

“…Our approach could be complemented with a quantification of effect size, which, unlike in CTT, is often difficult to estimate in IRT. Recently, Meade (2010), Chalmers et al (2016), andChalmers (2022) proposed two methods for quantifying the size of the DTF between the subgroups. The signed DTF estimates the systematic bias resulting from one subgroup scoring higher consistently across θ, thus providing the overall curve differences on θ.…”

mentioning

confidence: 99%

Item response theory approach to ethnocentrism

Monaghan,

Bizumic

2023

Front. Polit. Sci.

View full text Add to dashboard Cite

IntroductionAlthough ethnocentrism is one of the fundamental concepts in the social sciences, its study has been impeded by a diversity of conceptualizations and measures. In recent years, a growing number of political scientists and psychologists have undertaken in-depth research into ethnocentrism. In addition, researchers have recently proposed a comprehensive reconceptualization of ethnocentrism and developed a new Ethnocentrism scale. There is strong evidence for this scale's reliability and validity in indexing ethnocentrism, but like most measures in psychology and political science, this scale is based on classical test theory. Item response theory (IRT) is a powerful psychometric technique that can provide a much more sophisticated test of test performance and is currently under-utilized in research.MethodsWe performed IRT to assess the psychometric properties of the Ethnocentrism scale on a sample of 4,187 participants.ResultsThe scale's items had strong psychometric properties to capture the ethnocentrism latent construct, particularly in the below average to above average range. Men required marginally lower levels of ethnocentrism to endorse less socially acceptable items than women (items relating to superiority, purity, or exploitativeness). When compared to liberals, conservatives responded more readily to nearly all ethnocentrism items. Given this variation, the IRT approach highlighted that future measurements must adjust for differential item functioning, albeit more for political orientation than gender identity.DiscussionThe findings detail how IRT can enhance measurement in political science and demonstrate the implications for how gender and political ideology may affect the differential performance of items.

show abstract