Thurstonian modeling of ranking data via mean and covariance structure analysis

2013

Psychological Methods

165

185

In multidimensional forced-choice (MFC) questionnaires, items measuring different attributes are presented in blocks, and participants have to rank-order the items within each block (fully or partially). Such comparative formats can reduce the impact of numerous response biases often affecting single-stimulus items (aka, rating or Likert scales). However, if scored with traditional methodology, MFC instruments produce ipsative data, whereby all individuals have a common total test score. Ipsative scoring distorts individual profiles (it is impossible to achieve all high or all low scale scores), construct validity (covariances between scales must sum to zero), criterion related validity (validity coefficients must sum to zero), and reliability estimates.We argue that these problems are caused by inadequate scoring of forced-choice items, and advocate the use of item response theory (IRT) models based on an appropriate response process for comparative data, such as Thurstone's Law of Comparative Judgment. We show that by applying Thurstonian IRT modeling (Brown & Maydeu-Olivares, 2011), even existing forcedchoice questionnaires with challenging features can be scored adequately and that the IRTestimated scores are free from the problems of ipsative data. Assessments of personality, social attitudes, interests, motivation, psychopathology and well-being largely rely on respondent-reported measures. Most such measures employ the socalled single-stimulus format, where respondents evaluate one question (or item) at a time, often in relation to a rating scale (i.e. Likert-type items). Because the respondents rate each item separately from other items, they make absolute judgments about the extent to which the item describes their personality, attitudes, etc. Simple to answer and score and therefore popular with test takers and test users, the single-stimulus format makes several assumptions about the respondents' rating behaviors that are often unrealistic. For instance, the use of rating scales relies on the assumption that respondents interpret category labels in the same way. This assumption is very rarely tested in practice, but research available on the issue suggests that interpretation and meaning of response categories vary from one respondent to another (Friedman & Amoo, 1999). Furthermore, individual response styles may vary (Van Herk, Poortinga & Verhallen, 2004) so that some respondents avoid extreme categories (central tendency responding), whereas others prefer them (extreme responding). Sometimes respondents tend to agree with both positive and negative statements as presented (acquiescence bias).Another common problem is getting respondents to differentiate between ratings they give to single-stimulus items. When rating another person's attributes or behavior (as in the 360-degree feedback), respondents commonly give either high or low ratings on all behaviors (halo/horn effect) depending on whether they judge the person to score high or low on a single important dimension. Typically, respon...

Section: Item Response Modeling Of Forced-choice Questionnairesmentioning

confidence: 99%

How IRT can solve problems of ipsative data in forced-choice questionnaires.

Brown

2013

Psychological Methods

165

185

“…This is because for each block there are r 0 n(n -1)(n -2)/6 redundancies among the thresholds and tetrachoric correlations estimated from the binary outcome variables (Maydeu-Olivares, 1999). With p ranking blocks in the questionnaire, the number of redundancies is p × r. Thus, when n > 2, one needs to subtract p × r from the degrees of freedom given by Mplus to obtain the correct p value for the test of exact fit.…”

Section: Model Identificationmentioning

confidence: 99%

Fitting a Thurstonian IRT model to forced-choice data using Mplus

Brown

2012

Behav Res

To counter response distortions associated with the use of rating scales (a.k.a. Likert scales), items can be presented in a comparative fashion, so that respondents are asked to rank the items within blocks (forced-choice format). However, classical scoring procedures for these forced-choice designs lead to ipsative data, which presents psychometric challenges that are well described in the literature. Recently, Brown and Maydeu-Olivares (Educational and Psychological Measurement 71: 460-502, 2011a) introduced a model based on Thurstone's law of comparative judgment, which overcomes the problems of ipsative data. Here, we provide a step-by-step tutorial for coding forcedchoice responses, specifying a Thurstonian item response theory model that is appropriate for the design used, assessing the model's fit, and scoring individuals on psychological attributes. Estimation and scoring is performed using Mplus, and a very straightforward Excel macro is provided that writes full Mplus input files for any forced-choice design. Armed with these tools, using a forced-choice design is now as easy as using ratings.

“…Most frequently, it has been given the connotation of "desiredness," which is also the meaning that we apply here (see also, Kahneman, 2003). Bentler (1998) and Maydeu-Olivares (1999). However, these authors used a specialized estimation and testing approach that is cumbersome to use in applied work.…”

mentioning

confidence: 99%

Structural Equation Modeling of Paired-Comparison and Ranking Data.

Böckenholt

2005

Psychological Methods

Self Cite

107

L. L. Thurstone's (1927) model provides a powerful framework for modeling individual differences in choice behavior. An overview of Thurstonian models for comparative data is provided, including the classical Case V and Case III models as well as more general choice models with unrestricted and factor-analytic covariance structures. A flow chart summarizes the model selection process. The authors show how to embed these models within a more familiar structural equation modeling (SEM) framework. The different special cases of Thurstone's model can be estimated with a popular SEM statistical package, including factor analysis models for paired comparisons and rankings. Only minor modifications are needed to accommodate both types of data. As a result, complex models for comparative judgments can be both estimated and tested efficiently.Keywords: Lisrel, preference data, choice models, random utility models, factor analysisThe methods of ranking and paired comparison play an essential role in the measurement of preferences, attitudes, and values. In a ranking task, respondents are presented with a set of alternatives (which, in the literature, are also referred to as options, stimuli, or items) and are asked to order them from most to least preferred. In a paired-comparison task, respondents are presented with pairs selected from the set of available alternatives and are instructed to select the more preferred alternative from each pair. The popularity of paired-comparison and ranking methods can be traced back to three reasons. First, by asking for a comparison of choice alternatives, these methods impose minimal constraints on the response behavior of a respondent. Especially when differences between choice alternatives are small, these methods are likely to provide more information about individual preferences than is obtainable by rating methods. Second, internal consistency checks are available that facilitate the identification of respondents who do not have well-defined preferences, values, or attitudes. If respondents can be shown to be consistent in their judgments, one can have much greater confidence in the obtained measurements and the predictive value of the derived scales in further applications. Third, paired-comparison and ranking data provide a rich source of information about the effects of individual differences and perceived similarity relationships among choice alternatives. This article presents two applications that illustrate this important feature.Because of their versatility, the methods of paired comparison and rankings are used in a wide range of studies.