Comparing Personality Test Formats and Warnings: Effects on criterion‐related validity and test‐taker reactions

Abstract

The potential for applicant response distortion on personality measures remains a major concern in high-stakes testing situations. Many approaches to understanding response distortion are too transparent (e.g., instructed faking studies) -or are too subtle (e.g., correlations with social desirability measures as indices of faking). Recent research reveals more promising approaches in two methods: using forced-choice (FC) personality test items and warning against faking. The present study examined effects of these two methods on criterion-related validity and test-taker reactions. Results supported incremental validity for an FC and Likert-scale measure in warning and no-warning conditions, above and beyond cognitive ability. No clear differences emerged between the FC vs Likert measures or warning vs no-warning conditions in terms of validity. However, some evidence suggested that FC measures and warnings may produce negative test-taker reactions. We conclude with implications for implementation in selection settings.

Impact of electronic warnings on online personality scores and test-taker reactions in an applicant simulation

“…Examining procedural justice perceptions, McFarland (2003) found no significant difference between warned and unwarned test-takers. Similarly, Converse et al (2008) found that neither positive nor negative warnings influenced perceived test ease, test-taker expectations, satisfaction with the testing process, or belief in the tests. However, Converse et al (2008) did find that the negative warnings resulted in a higher level of test-taker anxiety.…”

“…Similarly, Converse et al (2008) found that neither positive nor negative warnings influenced perceived test ease, test-taker expectations, satisfaction with the testing process, or belief in the tests. However, Converse et al (2008) did find that the negative warnings resulted in a higher level of test-taker anxiety. In our study, we wanted to focus on the effects of an extreme negative warning, an accusation of faking, on test anxiety and perceptions of fairness.…”

“…Strong evidence suggests that blatant distortion will be reduced (Landers et al, 2011) and the overall impact of the situational pressure to fake will be blunted ; however, this might not influence criterion related validity coefficients (Converse et al, 2008). Furthermore, use of these statements might also make test users more anxious and might result in the degradation of the construct validity of the personality measures (Robson et al, 2008;Vasilopoulos, Cucina, & McElreath, 2005) and inaccurate scores for honest test-takers (Ellingson et al, 2012;Fan et al, 2012).…”

Impact of Elaboration on Responding to Situational Judgment Test Items

“…Imposing forced response formats on test-takers has received renewed attention as a second proactive approach. Although a multidimensional forced-choice response format was effective for reducing score inflation at the group level Christiansen, Burns, & Montgomery, 2005;Converse, Oswald, Imus, Hedricks, Roy, & Butera, 2008;Jackson, Wroblewski, & Ashton, 2000), it was affected by faking to the same degree as a traditional Likert scale at the individual level of analysis (Heggestad, Morrison, Reeve, & McCloy, 2006).…”

Applicant Perspectives During Selection: A Review Addressing “So What?,” “What’s New?,” and “Where to Next?”

“…This was made possible, in large part, by the development of the Selection Procedural Justice Scale (SPJS; Bauer, Truxillo, Sanchez, Craig, Ferrara, & Campion, 2001) and Social Process Questionnaire on Selection (Derous, Born, & De Witte, 2004), which enabled more explicit measurement and testing of Gilliland's rules. Second, this new era of applicant reactions boasts a broader focus, with scholars incorporating a wide range of theoretical perspectives that extend beyond Gilliland's framework (e.g., expectations theory, attribution theory; e.g., Converse, Oswald, Imus, Hedricks, Roy, & Butera, 2008). This theoretical expansion has advanced our knowledge about the mechanisms underlying applicant reactions.…”