Enhancing the Validity and Cross-Cultural Comparability of Measurement in Survey Research

King, Gary; Murray, Christopher J L; Salomon, Joshua A.; Tandon, Ajay

doi:10.1017/s000305540400108x

Cited by 733 publications

(216 citation statements)

References 35 publications

Supporting

Mentioning

206

Contrasting

Unclassified

Order By: Relevance

“…It uses a generalization of the ordered probit model to distribute vector-valued responses according to the proportion of "similar" respondents who choose the categories spanned by the vector. As this approach is not central in the current study, further explanation can be found in King and colleagues (Hopkins & King, 2010;King et al, 2004;.…”

Section: The Technique Of Anchoring Vignettesmentioning

confidence: 83%

“…The technique of anchoring vignettes was introduced to correct for measurement bias-in particular, the so-called reference-group effect-in Likert-type responses (King, Murray, Salomon, & Tandon, 2004). The reference-group effect stems from different standards (i.e., a reference group) that respondents use to evaluate themselves and their own behaviors, and may be related to various response styles, which refer to the tendency to use certain response options on some basis other than the target construct (Paulhus, 1991).…”

Section: The Technique Of Anchoring Vignettesmentioning

confidence: 99%

See 1 more Smart Citation

Effects of Anchoring Vignettes on Comparability and Predictive Validity of Student Self-Reports in 64 Cultures

Buchholz

Klieme

2017

Journal of Cross-Cultural Psychology

View full text Add to dashboard Cite

He, Jia; Buchholz, Janine; Klieme, Eckhard Effects of anchoring vignettes on comparability and predictive validity of student self-reports in 64 cultures formal und inhaltlich überarbeitete Version der Originalveröffentlichung in: formally and content revised edition of the original source in:Journal of cross-cultural psychology 48 (2017) 3, S. 319-334, 10.1177 We grant a non-exclusive, non-transferable, individual and limited right to using this document. This document is solely intended for your personal, non-commercial use. Use of this document does not include any transfer of property rights and it is conditional to the following limitations: All of the copies of this documents must retain all copyright information and other information regarding legal protection. You are not allowed to alter this document in any way, to copy it for public or commercial purposes, to exhibit the document in public, to perform, distribute or otherwise use the document in public. AbstractAnchoring vignettes are item batteries especially designed for correcting responses that might be affected by incomparability. This article investigates the effects of anchoring vignettes on the validity of student self-report data in 64 cultures. Using secondary data analysis from the 2012 Programme for International Student Assessment (PISA), we checked the validity of ratings on vignette questions, and investigated how rescaled item responses of two student scales, Teacher Support and Classroom Management, enhanced comparability and predictive validity. The main findings include that (a) responses to vignette questions represent valid individual and cultural differences; in particular, violations in these responses (i.e., misorderings) are related to low socioeconomic status and low cognitive sophistication; (b) the rescaled responses tend to show higher levels of comparability; and (c) the associations of rescaled Teacher Support and Classroom Management with math achievement, Student-Oriented Instruction, and Teacher-Directed Instruction are slightly different from raw scores of the two target constructs, and the associations with rescaled scores seem to be more in line with the literature. Namely, the associations among all self-report Likerttype scales are weaker with rescaled scores, presumably reducing common method variance, and both rescaled scale scores are more positively related to math achievement. The country ranking also changes substantially; in particular, Asian cultures top the ranking on Teacher Support after rescaling. However, anchoring vignettes are not a cure-all in solving measurement bias in crosscultural surveys; we discuss the technicality and directions for further research on this technique.

show abstract

Section: The Technique Of Anchoring Vignettesmentioning

confidence: 83%

Section: The Technique Of Anchoring Vignettesmentioning

confidence: 99%

Effects of Anchoring Vignettes on Comparability and Predictive Validity of Student Self-Reports in 64 Cultures

Buchholz

Klieme

2017

Journal of Cross-Cultural Psychology

View full text Add to dashboard Cite

show abstract

“…More robust approaches are required, drawing on experimental designs in which some key elements of the consultation-interaction can be accounted for in the analysis, for example through the use of standardised consultations and video vignettes. 81,82 Using patient survey data to improve care…”

Section: Patients' Varying Experiences Of Carementioning

confidence: 99%

Improving patient experience in primary care: a multimethod programme of research on the measurement and improvement of patient experience

Burt

Campbell

Abel

et al. 2017

Programme Grants Appl Res

View full text Add to dashboard Cite

BackgroundThere has been an increased focus towards improving quality of care within the NHS in the last 15 years; as part of this, there has been an emphasis on the importance of patient feedback within policy, through National Service Frameworks and the Quality and Outcomes Framework. The development and administration of large-scale national patient surveys to gather representative data on patient experience, such as the national GP Patient Survey in primary care, has been one such initiative. However, it remains unclear how the survey is used by patients and what impact the data may have on practice.ObjectivesOur research aimed to gain insight into how different patients use surveys to record experiences of general practice; how primary care staff respond to feedback; and how to engage primary care staff in responding to feedback.MethodsWe used methods including quantitative survey analyses, focus groups, interviews, an exploratory trial and an experimental vignette study.Results(1)Understanding patient experience data. Patients readily criticised their care when reviewing consultations on video, although they were reluctant to be critical when completing questionnaires. When trained raters judged communication during a consultation to be poor, a substantial proportion of patients rated the doctor as ‘good’ or ‘very good’. Absolute scores on questionnaire surveys should be treated with caution; they may present an overoptimistic view of general practitioner (GP) care. However, relative rankings to identify GPs who are better or poorer at communicating may be acceptable, as long as statistically reliable figures are obtained. Most patients have a particular GP whom they prefer to see; however, up to 40% of people who have such a preference are unable regularly to see the doctor of their choice. Users of out-of-hours care reported worse experiences when the service was run by a commercial provider than when it was run by a not-for profit or NHS provider. (2)Understanding patient experience in minority ethnic groups. Asian respondents to the GP Patient Survey tend to be registered with practices with generally low scores, explaining about half of the difference in the poorer reported experiences of South Asian patients than white British patients. We found no evidence that South Asian patients used response scales differently. When viewing the same consultation in an experimental vignette study, South Asian respondents gave higher scores than white British respondents. This suggests that the low scores given by South Asian respondents in patient experience surveys reflect care that is genuinely worse than that experienced by their white British counterparts. We also found that service users of mixed or Asian ethnicity reported lower scores than white respondents when rating out-of-hours services. (3)Using patient experience data. We found that measuring GP–patient communication at practice level masks variation between how good individual doctors are within a practice. In general practices and in out-of-hours centres, staff were sceptical about the value of patient surveys and their ability to support service reconfiguration and quality improvement. In both settings, surveys were deemed necessary but not sufficient. Staff expressed a preference for free-text comments, as these provided more tangible, actionable data. An exploratory trial of real-time feedback (RTF) found that only 2.5% of consulting patients left feedback using touch screens in the waiting room, although more did so when reminded by staff. The representativeness of responding patients remains to be evaluated. Staff were broadly positive about using RTF, and practices valued the ability to include their own questions. Staff benefited from having a facilitated session and protected time to discuss patient feedback.ConclusionsOur findings demonstrate the importance of patient experience feedback as a means of informing NHS care, and confirm that surveys are a valuable resource for monitoring national trends in quality of care. However, surveys may be insufficient in themselves to fully capture patient feedback, and in practice GPs rarely used the results of surveys for quality improvement. The impact of patient surveys appears to be limited and effort should be invested in making the results of surveys more meaningful to practice staff. There were several limitations of this programme of research. Practice recruitment for our in-hours studies took place in two broad geographical areas, which may not be fully representative of practices nationally. Our focus was on patient experience in primary care; secondary care settings may face different challenges in implementing quality improvement initiatives driven by patient feedback. Recommendations for future research include consideration of alternative feedback methods to better support patients to identify poor care; investigation into the factors driving poorer experiences of communication in South Asian patient groups; further investigation of how best to deliver patient feedback to clinicians to engage them and to foster quality improvement; and further research to support the development and implementation of interventions aiming to improve care when deficiencies in patient experience of care are identified.FundingThe National Institute for Health Research Programme Grants for Applied Research programme.

show abstract

“…To attenuate challenges to inference caused by non-equivalence in measurement, researchers should be careful to evaluate (and report) evidence for the validity of measures before assuming crosscultural equivalence. Such evidence can include techniques modeling response differences (e.g., King et al 2003) Byrne and Campbell 1999). For a more in-depth review, we refer to other work on cross-cultural measurement (e.g., Hui and Triandis 1985;Leung 1997, 2000).…”

Section: Cross-cultural Non-equivalence Of Measurementmentioning

confidence: 99%

What Can Cross-Cultural Correlations Teach Us about Human Nature?

Pollet

Tybur

Frankenhuis

et al. 2014

Hum Nat

143

111

View full text Add to dashboard Cite

Many recent evolutionary psychology and human behavioral ecology studies have tested hypotheses by examining correlations between variables measured at a group level (e.g., state, country, continent). In such analyses, variables collected for each aggregation are often taken to be representative of the individuals present within them, and relationships between such variables are presumed to reflect individual-level processes. There are multiple reasons to exercise caution when doing so, including: (1) the ecological fallacy, whereby relationships observed at the aggregate level do not accurately represent individual-level processes; (2) non-independence of data points, which violates assumptions of the inferential techniques used in null hypothesis testing; and (3) cross-cultural non-equivalence of measurement (differences in construct validity between groups). We provide examples of how each of these gives rise to problems in the context of testing evolutionary hypotheses about human behavior, and we offer some suggestions for future research.Keywords Ecological fallacy . Cross-cultural research . Research methods . Simpson's paradox . Non-independence . Measurement equivalence Many hypothesis-testing techniques used by evolutionary behavioral scientists involve examining degrees of covariation between two or more variables and making inferences based on such relationships (e.g., analysis of variance, Pearson correlation). This approach may be applied to a number of different research designs that examine variation at different, hierarchically organized levels (e.g., individuals versus groups;

show abstract

Enhancing the Validity and Cross-Cultural Comparability of Measurement in Survey Research

Cited by 733 publications

References 35 publications

Effects of Anchoring Vignettes on Comparability and Predictive Validity of Student Self-Reports in 64 Cultures

Effects of Anchoring Vignettes on Comparability and Predictive Validity of Student Self-Reports in 64 Cultures

Improving patient experience in primary care: a multimethod programme of research on the measurement and improvement of patient experience

What Can Cross-Cultural Correlations Teach Us about Human Nature?

Contact Info

Product

Resources

About