Methods for Assessing and Calibrating Response Scales Across Countries and Languages

Abstract

IntroductionScientific research rests on the reliable and consistent measurement of phenomenon. In cross-national or cross-cultural survey research between countries or social groups that speak different languages, the goal of replicative measurement is greatly complicated by the necessity of designing and administering questionnaires in two or more languages. Only by assuring that the items in all languages and questionnaires are equivalent both in meaning and response scales can comparable measurement be obtained and valid inferences drawn. But the complexity of both survey measurement and of languages makes the goal of equivalency an extremely difficult challenge (Glick, et al.

The Effect of Familiarity with the Response Category Labels on Item Response to Likert Scales

“…We focused on positive labels only since prior research has shown that modifiers are typically balanced (i.e., "completely agree" and "completely disagree" have reciprocal values; see Smith et al 2009). …”

“…Cliff (1959) found that amplifiers such as "decidedly," "very" or "extremely" multiplied the intensity of the adjectives they modified, and he showed that "extremely" had a stronger intensifying effect than "very," which in turn had a stronger effect than "decidedly". Smith et al (2009) also noted differences in the perceived intensity of the amplifiers investigated in their research (e.g., "completely," "definitely," "strongly," "very much"). However, these authors did not test whether differences in the intensity of the endpoint labels induced differences in the extent to which the associated categories were selected.…”

“…For scaling labels in terms of intensity, Smith et al (2009) recommend the use of a direct rating method in which respondents rate the intensity of labels on a numeric scale. To provide evidence of convergent validity, we supplement this direct rating method with a pairwise comparison technique in which respondents compare labels two at a time and for each pair choose the one that they perceive to convey a stronger sense of agreement.…”

The calibrated sigma method: An efficient remedy for between-group differences in response category use on Likert scales

“…In particular, response category labels used in different languages may vary in intensity. For example, in a comparison of response category labels in the U.S. and Germany, Smith, et al (2005) found subtle intensity differences between apparently equivalent labels such as 'definitely agree' and its German translation, 'stimme bestimmt zu' (i.e., 'definitely' is a stronger term in English than 'bestimmt' is in German). Furthermore, Weijters et al (2013) recently demonstrated that a category label may be more idiomatic and thus more familiar in one language than another.…”

“…However, in certain situations scale usage heterogeneity may occur primarily at the group level, in which case it is more appropriate to model differential scale usage at the group level. For example, when Likert-type rating scales anchored by labels such as 'strongly (dis)agree' or 'completely (dis)agree' are used in different languages, the meaning of the response category labels may subtly but systematically vary across languages, which can lead to differences in scale usage at the group level (Skevington & Tucker, 1999;Smith, Mohler, Harkness, & Onodera, 2005;Szabo, Orley, & Saxena, 1997;Weijters, Geuens, & Baumgartner, 2013). Similarly, data collection modes or experimental manipulations may affect the perceived meaning of the category labels and thus induce scale usage heterogeneity (Jordan, Marcus, & Reeder, 1980;Weijters, Schillewaert, & Geuens, 2008).…”