Taking the Test Taker’s Perspective: Response Process and Test Motivation in Multidimensional Forced-Choice Versus Rating Scale Instruments

Sass, Rachelle; Frick, Susanne A.; Reips, Ulf‐Dietrich; Wetzel, Eunike

doi:10.1177/1073191118762049

Cited by 32 publications

(32 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…At the same time, several studies have investigated different aspects of the model's validity. In a think-aloud task when completing FC items, respondents indeed reported making pairwise comparisons between all items of a triplet, i.e., a block of three items (Sass, Frick, Reips, & Wetzel, 2018). Thus, one of the key components of T-IRT models, the transformation of rankings into pairwise comparisons, seems to describe respondents' underlying decision-making processes reasonably well.…”

Section: Previous Validation Studiesmentioning

confidence: 90%

“…However, this study also draws attention to a limitation of this modeling technique: only 76% of the participants reported having no difficulty in keeping the information for all statements in mind in order to appraise the utility of an item relative to the utilities of all other items in the block. Further, as the Sass et al (2018) study only used triplets, this limitation likely leads to severe violations of the model assumptions when blocks with more items are employed.…”

Section: Previous Validation Studiesmentioning

confidence: 99%

“…While they are known to be problematic in single stimulus items already, the issue becomes more severe in FC questionnaires due to the increased cognitive load they provoke: Assuming that the Law of Comparative Judgement describes the response processes in FC questionnaires reasonably well, a block of n items leads to the respondent having to makeñ = n(n − 1)/2 comparisons when answering a single block of items. Recall that only 76% of respondents reported having no difficulties in processing the three comparisons necessary for blocks with three items (Sass et al, 2018); but blocks with four items require participants to process six comparisons. A second problem of including negatively keyed items in a block is their potential to result in substantial methodological variance (Dueber, Love, Toland, & Turner, 2018).…”

Section: Equally Vs Unequally Keyed Itemsmentioning

confidence: 99%

See 2 more Smart Citations

On the Statistical and Practical Limitations of Thurstonian IRT Models

Bürkner¹,

Schulte²,

Holling³

2018

Preprint

View full text Add to dashboard Cite

Forced-choice questionnaires have been proposed to avoid common response biases typically associated with rating scale questionnaires. To overcome ipsativity issues of trait scores obtained from classical scoring approaches of forced-choice items, advanced methods from item response theory (IRT) such as the Thurstonian IRT model have been proposed. For convenient model specification, we introduce the thurstonianIRT R package, which uses Mplus, lavaan, and Stan for model estimation. Based on practical considerations, we establish that items within one block need to be equally keyed to achieve similar social desirability, which is essential for creating force-choice questionnaires that have the potential to resist faking intentions. According to extensive simulations, measuring up to 5 traits using blocks of only equally keyed items does not yield sufficiently accurate trait scores and inter-trait correlation estimates, neither for frequentist nor Bayesian estimation methods. As a result, persons' trait scores remain partially ipsative and, thus, do not allow for valid comparisons between persons. However, we demonstrate that trait scores based on only equally keyed blocks can be improved substantially by measuring a sizeable number of traits. More specifically, in our simulations of 30 traits, scores based on only equally keyed blocks were non-ipsative and highly accurate. We conclude that in high-stakes situations where persons are motivated to give fake answers, Thurstonian IRT models should only be applied to tests measuring a sizeable number of traits.

show abstract

Section: Previous Validation Studiesmentioning

confidence: 90%

Section: Previous Validation Studiesmentioning

confidence: 99%

Section: Equally Vs Unequally Keyed Itemsmentioning

confidence: 99%

See 1 more Smart Citation

On the Statistical and Practical Limitations of Thurstonian IRT Models

Bürkner¹,

Schulte²,

Holling³

2018

Preprint

View full text Add to dashboard Cite

show abstract

“…However, this study also draws attention to a limitation of this modeling technique: Only 76% of the participants reported having no difficulty in keeping the information for all statements in mind to appraise the utility of an item relative to the utilities of all other items in the block. Furthermore, as the Sass et al (2018) study only used triplets, this limitation likely leads to severe violations of the model assumptions when blocks with more items are employed.…”

Section: Previous Validation Studiesmentioning

confidence: 99%

On the Statistical and Practical Limitations of Thurstonian IRT Models

Bürkner

Schulte

Benedek

2019

Educational and Psychological Measurement

111

View full text Add to dashboard Cite

Forced-choice questionnaires have been proposed to avoid common response biases typically associated with rating scale questionnaires. To overcome ipsativity issues of trait scores obtained from classical scoring approaches of forced-choice items, advanced methods from item response theory (IRT) such as the Thurstonian IRT model have been proposed. For convenient model specification, we introduce the thurstonianIRT R package, which uses Mplus, lavaan, and Stan for model estimation. Based on practical considerations, we establish that items within one block need to be equally keyed to achieve similar social desirability, which is essential for creating forced-choice questionnaires that have the potential to resist faking intentions. According to extensive simulations, measuring up to five traits using blocks of only equally keyed items does not yield sufficiently accurate trait scores and inter-trait correlation estimates, neither for frequentist nor for Bayesian estimation methods. As a result, persons’ trait scores remain partially ipsative and, thus, do not allow for valid comparisons between persons. However, we demonstrate that trait scores based on only equally keyed blocks can be improved substantially by measuring a sizable number of traits. More specifically, in our simulations of 30 traits, scores based on only equally keyed blocks were non-ipsative and highly accurate. We conclude that in high-stakes situations where persons are motivated to give fake answers, Thurstonian IRT models should only be applied to tests measuring a sizable number of traits.

show abstract

“…However, psychometric models make different assumptions about response processes for FC and SS measures (Sass, Frick, Reips, & Wetzel, 2018). These theoretical differences raise a fundamental question: Are scales using FC and SS formats measuring the same construct?…”

mentioning

confidence: 99%

Though Forced, Still Valid: Psychometric Equivalence of Forced-Choice and Single-Statement Measures

Zhang

Sun

Drasgow

et al. 2019

Organizational Research Methods

View full text Add to dashboard Cite

Forced choice (FC) measures are gaining popularity as an alternative assessment format to single statement (SS) measures due to their potential in reducing the impact of various response styles and faking. However, a fundamental question remains to be answered: do FC and SS instruments measure the same underlying constructs? In addition, FC measures are theorized to be more cognitively challenging, so how would this feature influence respondents' reactions to FC measures compared to SS? Two studies were designed to answer these questions. Study 1 results showed that FC measures scored by the Multi-unidimensional Pairwise Preference Model (MUPP) and SS measures scored with an ideal point model yielded similar factor structures and almost identical criterion-related validity across 12 criteria. Both formats also had similar pattern of marginal reliabilities and test-retest reliabilities. Study 1 findings were replicated in Study 2.In addition, we found strong evidence for convergent validity between the two formats. Though the FC format was perceived to be more difficult, respondents showed no differential preference and expressed similar level of emotional and cognitive reactions to the two formats.

show abstract

Taking the Test Taker’s Perspective: Response Process and Test Motivation in Multidimensional Forced-Choice Versus Rating Scale Instruments

Cited by 32 publications

References 37 publications

On the Statistical and Practical Limitations of Thurstonian IRT Models

On the Statistical and Practical Limitations of Thurstonian IRT Models

On the Statistical and Practical Limitations of Thurstonian IRT Models

Though Forced, Still Valid: Psychometric Equivalence of Forced-Choice and Single-Statement Measures

Contact Info

Product

Resources

About