Enhancing the Validity and Cross-Cultural Comparability of Measurement in Survey Research

Abstract

T he discipline of political science is built on theory, including a rough agreement on normative theories preferring freedom, democracy, and political equality, among others, and the development of positive theories focused on understanding the causes and consequences of these variables. Empirical political science, in turn, is devoted in large part to making causal inferences about these same variables. UnYou may be interested in our Anchoring Vignettes Web site, which, as a companion to this paper, provides software to implement the methods here, answers to frequently asked questions, example vignettes, and other materials (see http://gking.harvard.edu/vign/). Our names on this paper are ordered alphabetically. Our thanks go to John Aldrich, Jim Alt, Larry Bartels, Neal Beck, David Cutler, Federico Girosi, Dan Ho, Kosuke Imai, Stanley Feldman, Michael Herron, Mel Hinich, Simon Jackman, Orit Kedar, Jeff Lewis, Jeffrey Liu, John Londregan, Joe Newhouse, Keith Poole, Sid Verba, Jonathan Wand, and Chris Winship for helpful discussions; Ken Benoit, Debbie Javeline, and Karen Ferree for help in writing vignettes; three anonymous referees and the editor for exceptionally helpful suggestions (One of our reviewers, who we now know is Henry Brady, wrote an extraordinary 20 page single-spaced review that greatly improved our work.);

The framing of risks and the communication of subjective probabilities for victimizations

“…Furthermore, Nakao and Axelrod (1983) show such group differences between doctors and patients in interpreting verbal frequency expressions like "infrequent" or "not frequently". Finally, King et al (2004) demonstrate cultural differences in using response scales. Wänke (2002) highlighted the importance of reference groups.…”

Analysis of the validity of the vignette approach to correct for heterogeneity in reporting health system responsiveness

“…7 An analysis of the more common alternative patterns of vignette ordering can provide information about the relative importance of the problem of measurement error versus the problems of multidimensionality and variation in the cultural construct of a domain. Measurement error is generally associated with a large number of alternative orderings (due to chance).…”

“…[3][4][5][6][7]), health system responsiveness [2,[8][9][10], happiness and job satisfaction [11,12], national identity [13] and state effectiveness [14]. Despite the growing popularity of the vignette methodology to address the issue of reporting heterogeneity, the formal evaluation of the validity of the approach remains a topic of research [15][16][17][18].…”

“…1 A limited number of other studies have tried to assess the validity of this assumption. These were focussed on self-reports of the ratings of work disability [5], mobility [19], visual acuity and political efficacy [7,21], job satisfaction [11] and life satisfaction for income [21], largely making use of non-parametric methods using tests based on the global ordering of the vignettes. Our study explores the validity of the vignette equivalence assumption, making reference to the concept of responsiveness and using data from the WHS.…”

“…30.3 6. The average is computed assigning the same weight to each country within a group 7. As an example ''running a marathon'' could be viewed as a multidimensional construct.…”