The Multidimensional Forced-Choice Format as an Alternative for 					Rating Scales

Wetzel, Eunike; Frick, Susanne A.; Greiff, Samuel

doi:10.1027/1015-5759/a000609

Cited by 18 publications

(8 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The precision of trait estimates has been thoroughly pointed out as a main weakness in the use of FCQs (Brown & Maydeu-Olivares, 2011;Kreitchmann et al, 2019;Meade, 2004;Wetzel et al, 2020). Accordingly, the aim of this study was to investigate the effect of the assembly of pairwise FCQs on the recovery of trait estimates, presenting an efficient automated optimization procedure.…”

Section: Discussionmentioning

confidence: 99%

A genetic algorithm for optimal assembly of pairwise forced-choice questionnaires

Abad

Sorrel

2021

Behav Res

View full text Add to dashboard Cite

The use of multidimensional forced-choice questionnaires has been proposed as a means of improving validity in the assessment of non-cognitive attributes in high-stakes scenarios. However, the reduced precision of trait estimates in this questionnaire format is an important drawback. Accordingly, this article presents an optimization procedure for assembling pairwise forced-choice questionnaires while maximizing posterior marginal reliabilities. This procedure is performed through the adaptation of a known genetic algorithm (GA) for combinatorial problems. In a simulation study, the efficiency of the proposed procedure was compared with a quasi-brute-force (BF) search. For this purpose, five-dimensional item pools were simulated to emulate the real problem of generating a forced-choice personality questionnaire under the five-factor model. Three factors were manipulated: (1) the length of the questionnaire, (2) the relative item pool size with respect to the questionnaire’s length, and (3) the true correlations between traits. The recovery of the person parameters for each assembled questionnaire was evaluated through the squared correlation between estimated and true parameters, the root mean square error between the estimated and true parameters, the average difference between the estimated and true inter-trait correlations, and the average standard error for each trait level. The proposed GA offered more accurate trait estimates than the BF search within a reasonable computation time in every simulation condition. Such improvements were especially important when measuring correlated traits and when the relative item pool sizes were higher. A user-friendly online implementation of the algorithm was made available to the users.

show abstract

Section: Discussionmentioning

confidence: 99%

A genetic algorithm for optimal assembly of pairwise forced-choice questionnaires

Abad

Sorrel

2021

Behav Res

View full text Add to dashboard Cite

show abstract

“…Some advantages of the proposed fuzzy IRT-map are as follows. First, since the procedure does not require a dedicated measurement setting, it is applicable over a wide range of survey situations including different rating formats (e.g., Likert-type, forced-choice, funnel response-format) [73,99].…”

Section: Discussionmentioning

confidence: 99%

A psychometric modeling approach to fuzzy rating data

Calcagnì,

Cao,

Rubaltelli

et al. 2021

Preprint

View full text Add to dashboard Cite

Modeling fuzziness and imprecision in human rating data is a crucial problem in many research areas, including applied statistics, behavioral, social, and health sciences. Because of the interplay between cognitive, affective, and contextual factors, the process of answering survey questions is a complex task, which can barely be captured by standard (crisp) rating responses.Fuzzy rating scales have progressively been adopted to overcome some of the limitations of standard rating scales, including their inability to disentangle decision uncertainty from individual responses. The aim of this article is to provide a novel fuzzy scaling procedure which uses Item Response Theory trees (IRTrees) as a psychometric model for the stage-wise latent response process. In so doing, fuzziness of rating data is modeled using the overall rater's pattern of responses instead of being computed using a single-item based approach. This offers a consistent system for interpreting fuzziness in terms of individual-based decision uncertainty. A simulation study and two empirical applications are adopted to assess the characteristics of the proposed model and provide converging results about its effectiveness in modeling fuzziness and imprecision in rating data.

show abstract

“…In relation to cross‐cultural settings, Wetzel et al (2020) claimed the feasibility of forced‐choice format as an alternative to Likert‐scale formats in their preliminary conclusions. However, the benefit of using forced‐choice formats for cross‐cultural selection settings is not yet clear because there is a lack of scholarly evidence on the performance of forced‐choice formats with respect to faking occurring in different cultures.…”

Section: Research Backgroundsmentioning

confidence: 99%

Reduction of faking with the use of a forced‐choice personality test: Cross‐cultural comparisons between South Korea and the United States

Lee

2022

Int J Selection Assessment

View full text Add to dashboard Cite

Forced-choice format tests have been suggested as an alternative to Likert-scale measures for personnel selection due to robustness to faking and response styles. This study compared degrees of faking occurring in Likert-scale and forced-choice five-factor personality tests between South Korea and the United States. Also, it was examined whether the forced-choice format was effective at reducing faking in both countries. Data were collected from 396 incumbents participating in both honest and applicant conditions (N SK = 179, N US = 217). Cohen's d values for within-subjects designs (ds within ) for between the two conditions were utilized to measure magnitudes of faking occurring in each format and country. In both countries, the degrees of faking occurring in the Likert-scale were larger than those from the forced-choice format, and the magnitudes of faking across five personality traits were larger in South Korea by from 0.07 to 0.12 in ds within .The forced-choice format appeared to successfully reduce faking for both countries as the average ds within decreased by 0.06 in both countries. However, the patterns of faking occurring in the forced-choice format varied between the two countries. In South Korea, degrees of faking in Openness and Conscientiousness increased, whereas those in Extraversion and Agreeableness were substantially decreased. Potential factors leading to trait-specific faking under the forced-choice format were discussed in relation to cultural influence on the perception of personality traits and score estimation in Thurstonian item response theory (IRT) models. Finally, the adverse impact of using forced-choice formats on multicultural selection settings was elaborated.

show abstract

The Multidimensional Forced-Choice Format as an Alternative for Rating Scales

Cited by 18 publications

References 28 publications

A genetic algorithm for optimal assembly of pairwise forced-choice questionnaires

A genetic algorithm for optimal assembly of pairwise forced-choice questionnaires

A psychometric modeling approach to fuzzy rating data

Reduction of faking with the use of a forced‐choice personality test: Cross‐cultural comparisons between South Korea and the United States

Contact Info

Product

Resources

About