The calibrated sigma method: An efficient remedy for between-group differences in response category use on Likert scales

Abstract

The authors propose a procedure, labeled the calibrated sigma method, which is designed to correct for between-group differences in endorsement likelihood of response categories that are unrelated to the content of the items. The method is especially useful in cross-cultural research where group differences may reflect variation in scale usage rather than substantive differences. However, the procedure is also relevant in other situations, for example, when different data collection modes or different experimental manipulations affect respondents' perception of the meaning of the scale labels. The calibrated sigma method uses information derived from heterogeneous control items (calibration items) to reweight the responses to substantive items in a group-specific way. The advantages of the calibrated sigma method are that it avoids the arbitrariness in the assignment of particular numerical values to response categories; that it is compatible with the linear model, which is used by most marketing researchers; and that it does not require the use of complex nonlinear models involving the estimation of many additional measurement model parameters. The authors validate the calibrated sigma method on a simulated cross-linguistic data set pertaining to 12 different languages; an empirical data set collected from respondents of the same nationality but from two different language groups; and an experimental data set consisting of responses to two different response scale formats. The findings demonstrate that the proposed procedure controls for artefactual scale use differences across groups but does not eliminate substantive differences. It is particularly efficient for marketing research agencies, panel providers and other marketing researchers who analyze surveys involving multiple language groups, different scale formats, multiple modes of data collection, or different manipulations affecting the meaning of the response category labels.

The Impact of Green Open Space on Community Attachment—A Case Study of Three Communities in Beijing

“…The other section was designed to investigate the perception of people to all aspects of green open space. Responses were categorized by using Likert scales [32] …”

“…The other section was designed to investigate the perception of people to all aspects of green open space. Responses were categorized by using Likert scales [32] from the community attachment and the degree of satisfaction, as well as the perceived area of green open space, accessibility and perceived importance of green open space. Four closed questions (i.e., I do not want to live in other communities (agree or disagree); I have a strong community attachment (agree or disagree); the community is special to me (agree or disagree); I will be very sad when I leave (agree or disagree)) were used to measure community attachment.…”

When Cheap Isn't the Same as Not Expensive: Generic Price Terms and Their Negations

“…As a consequence, when respondents rate products with relatively low prices, they will be more likely to use the most extreme scale positions referring to lower prices when the endpoint is labeled not expensive rather than cheap. The reason is that less extreme response options are more readily endorsed because they are less exceptional (de Langhe, Puntoni, Fernandes, & van Osselaer, ; Weijters, Baumgartner, & Geuens, ). In contrast, the endorsement rates for the scale positions referring to higher prices should not differ regardless of whether the endpoints are labeled not cheap or expensive, because these terms are equivalent in meaning and therefore equally extreme.…”

Cross Cultural Issues in Consumer Science and Consumer Psychology

“…Researchers need to pay special attention to the issue of designing response scale formats that are cross-culturally and cross-linguistically equivalent, because often a common response scale is used throughout the questionnaire, which can introduce systematic between-group biases (Weijters, Baumgartner, & Geuens, 2016;Weijters, Geuens, & Baumgartner, 2013). Consider a situation in which a researcher uses the response category label 'strongly agree' in English and is wondering how to translate this label into French.…”

“…Since language is often confounded with culture, such differences can easily be misconstrued as evidence for substantive cross-cultural variation. This is especially true since measurement invariance testing is often unable to detect uniform bias (i.e., bias that is the same across multiple items, which is typically the case with response scale effects) (Weijters, Baumgartner, et al, 2016).…”