Analysis of the validity of the vignette approach to correct for heterogeneity in reporting health system responsiveness

Abstract

Despite the growing popularity of the vignette methodology to deal with self-reported, categorical data, the formal evaluation of the validity of this methodology is still a topic of research. Some critical assumptions need to hold in order for this method to be valid. In this paper we analyse the assumption of "vignette equivalence" using data on health system responsiveness contained within the World Health Survey. We perform several tests to check the assumption of vignette equivalence. First, we use a test based on the global ordering of the vignettes. A minimal condition for the assumption of vignette equivalence to hold is that individual responses are consistent with the global ordering of vignettes. Secondly, using the hierarchical ordered probit model (HOPIT) model on the pool of countries, we undertake sensitivity analyses, stratifying countries according to the Inglehart-Welzel scale and the Human Development Index. The results of this analysis are robust, suggesting that the vignette equivalence assumption is not contradicted. Thirdly, we model the reporting behaviour of the respondents through a two-step regression procedure to evaluate whether the vignettes construct is perceived by respondents in different ways. Overall, across the analyses the results do not contradict the assumption of vignette equivalence and accordingly lend support to the use of the vignette methodology when analysing self-reported data and health system responsiveness.

Are Americans Really Less Happy with Their Incomes?

“…In order to be able to do that, comparability of the measures used is an obvious necessary condition. This paper illustrates that correcting income satisfaction 11 A third type of tests of vignette equivalence in Rice et al (2011) uses country groupings. We cannot perform these tests since we only have two countries.…”

Racial, Gender, and Socioeconomic Status Bias in Senior Medical Student Clinical Decision-Making: A National Survey

“…Another concern might be the use of vignettes to search for evidence of stereotyping in decision-making, although several studies have shown vignettes to be accurate in reflecting actual clinical practice. [39][40][41][42][43] The survey response rate among students at participating schools (40.2 %) is low enough that some might question the validity of the sample. Although our data show relatively few differences among survey respondents and the larger population of senior medical students as depicted by the AAMC survey, suggesting sample validity, it is possible that non-participants and students from non-participating schools may demonstrate different decision-making tendencies than those reflected in these data.…”

Health systems' responsiveness and reporting behaviour: Multilevel analysis of the influence of individual-level factors in 64 countries

“…1. Previous studies of the ranking of these vignettes across all eight domains have demonstrated their validity in relation to distinguishing between different levels of service standards (from best state to worst state), as measured by high inter-individual Spearman's rank ordered coefficients (Rice et al, 2011(Rice et al, , 2009). This implies that respondents understood the vignettes in the same way, and hence the data did not violate the essential 'vignette equivalence' assumption.…”

“…The WHS vignettes have been successfully tested for eliciting a common understanding across respondents (vignette equivalence) (Rice et al, 2011;Rice et al, 2012). The data have been used to publish papers quite widely (King et al, 2013;Szwarcwald et al, 2010), which include studies on their psychometric properties (Valentine et al, 2010).…”