Using Cognitive Interviewing and Behavioral Coding to Determine Measurement Equivalence across Linguistic and Cultural Groups

Abstract

The present study aimed to examine and compare results from two questionnaire pretesting methods (i.e., behavioral coding and cognitive interviewing) in order to assess systematic measurement bias in survey questions for adult smokers across six countries (USA, Australia, Uruguay, Mexico, Malaysia and Thailand). Protocol development and translation involved multiple bilingual partners in each linguistic/cultural group. The study was conducted with convenience samples of 20 adult smokers in each country. Behavioral coding and cognitive interviewing methods produced similar conclusions regarding measurement bias for some questions; however, cognitive interviewing was more likely to identify potential response errors than behavioral coding. Coordinated survey qualitative pretesting (or post-survey evaluation) is feasible across cultural groups, and can provide important information on comprehension and comparability. Cognitive interviewing appears a more robust technique than behavioral coding, although combinations of the two might be even better.

Promoting cessation resources through cigarette package warning labels: a longitudinal survey with adult smokers in Canada, Australia and Mexico

“…The Spanish language translations for the Mexico survey were developed with a committee translation process,18 with cognitive interviewing to ensure intended comprehension for some questions 19…”

Item comparability in cross-national surveys: results from asking probing questions in cross-national web surveys about attitudes towards civil disobedience

“…Implementing cognitive techniques in a cross-national survey itself (similar to Schuman's "random probes", 1966) or conducting cognitive interviews "after the fact" to explain problems in the dataset is still rare. Recent exceptions, however, are the post-survey cognitive interviewing studies by Thrasher et al (2011);Reeve et al (2011) or Morren et al (Forthcoming). Second, the comparative cognitive studies that are conducted mostly apply to different ethnic groups in only one country, mostly the U.S.…”

“…Experienced cognitive interviewers may not be available in all countries, and even if they were, it would be necessary to standardize procedures across countries for reasons of comparability. For instance, different house styles in recruiting respondents or different guidelines specifying the conduct of interviews would need to be harmonized, at least to some extent (Miller et al 2011;Thrasher et al 2011). Third, the number of cases per country in cross-national studies is usually too small to draw more generalizable conclusions on the differences between country-specific answer patterns (Thrasher et al 2011).…”

“…Second, the application of cognitive interviewing techniques does not have to be based on personal interviews. The critical issues of (non-)availability of cognitive interviewers across countries or of harmonization of probing procedures (Thrasher et al 2011) could be circumvented by implementing probing techniques in self-administered surveys. Third, the dominant assumption that low case numbers will be sufficient when using cognitive interviewing techniques is counterproductive.…”

“…For instance, different house styles in recruiting respondents or different guidelines specifying the conduct of interviews would need to be harmonized, at least to some extent (Miller et al 2011;Thrasher et al 2011). Third, the number of cases per country in cross-national studies is usually too small to draw more generalizable conclusions on the differences between country-specific answer patterns (Thrasher et al 2011). One of the few cross-national cognitive studies, in which seven countries participated and in which, among others, ESS questions were tested, conducted not more than ten interviews in three and not more than 20 in another two countries ).…”