Examining the item response process to personality measures in high‐stakes situations: Issues of measurement validity and predictive validity

Abstract

We conducted two experimental studies with between-subjects and within-subjects designs to investigate the item response process for personality measures administered in high-versus low-stakes situations. Apart from assessing measurement validity of the item response process, we examined predictive validity; that is, whether or not different response models entail differential selection outcomes. We found that ideal point response models fit slightly better than dominance response models across high-versus low-stakes situations in both studies. Additionally, fitting ideal point models to the data led to fewer items displaying differential item functioning compared to fitting dominance models. We also identified several items that functioned as intermediate items in both the faking and honest conditions when ideal point models were fitted, suggesting that ideal point model is "theoretically" more suitable across these contexts for personality inventories. However, the use of different response models (dominance vs. ideal point) did not have any substantial impact on the validity of personality measures in high-stakes situations, or the effectiveness of selection decisions such as mean performance or percent of fakers selected. These findings are significant in that although prior research supports the importance and use of ideal point models for measuring personality, we find that in the case of personality faking, though ideal point models seem to have slightly better measurement validity, the use of dominance models may be adequate with no loss to predictive validity. K E Y W O R D S dominance model, faking, ideal point model, item response process, personality assessment Personnel Psychology. 2020;73:305-332.wileyonlinelibrary.com/journal/peps

An Item-Level Analysis for Detecting Faking on Personality Tests: Appropriateness of Ideal Point Item Response Theory Models

“…Considering GGUM's mathematical complexity for estimation difficulties, some studies related to detect faking used other methods, for example, techniques based on reaction times, and scored invalidity scales (Sellbom and Bagby, 2010;Monaro et al, 2018;Roma et al, 2018;Mazza et al, 2019), generally obtained superior accurate outcomes. Finally, practically speaking, the use of ideal point models seems not to result in any improvement for predictive validity, if comparing with dominance models (Zhang et al, 2019). Hence there are still some issues with ideal point models when used for modeling faking response data.…”

Reduction of faking with the use of a forced‐choice personality test: Cross‐cultural comparisons between South Korea and the United States

“…More recent studies utilized IRT‐based approaches to validate forced‐choice tests (P. Lee et al, 2018; Morillo et al, 2019; Ng et al, 2021; Walton et al, 2019; Wetzel & Frick, 2020; Zhang et al, 2020) and to demonstrate the robustness of forced‐choice formats to faking (Usami et al, 2016; Wetzel et al, 2021). However, it should be noted that this claim was based on less inflation of forced‐choice mean scores than those from the Likert‐scale format, and faking was still observed in forced‐choice formats (Pavlov et al, 2019; Wetzel et al, 2021).…”

The Fakeability of Personality Measurement with Graded Paired Comparisons

“…We estimated honest and faking factor scores in a common model to estimate both types of scores on the same scale. Only to test whether the assumption of identical item parameters in both conditions affects the results notably, we estimated one model for each of the two conditions (Zhang et al, 2020), resulting in highly correlated factor loadings in both models (r = .98). All further analyses are based on the joint model.…”