Improving Social Measurement by Understanding Interaction in Survey Interviews

Abstract

Many of the official statistics and leading indicators that inform policy decisions are created from aggregating data collected in scientific survey interviews. What happens in the back-and-forth of those interviews-whether a sampled member of the public agrees to participate or not, whether a respondent comprehends questions in the way they were intended or not, whether the interview is spoken or texted-can thus have far-reaching consequences. But the landscape for social measurement is rapidly changing: Participation rates are declining, and people's daily communication patterns are evolving with new technologies (text messaging, video chatting, social media posting, etc.). New analyses of survey interactions are demonstrating aspects of interviewer speech that can substantially affect survey participation, which is vital if social measurement is to be trustworthy. Findings also suggest that, once a survey interview starts, the risks of misunderstanding and miscommunication are greater than one might expect, potentially jeopardizing the accuracy of survey results; different approaches to interviewing that allow clarification dialogue can improve respondents' comprehension and thus survey data quality. Analyses of text messaging and voice interviews on smartphones demonstrate the importance of adapting scientific social measurement to new patterns of communication, adding ways for people to contribute their data at a time and in a mode that is convenient for them even when they are mobile or multitasking.

Talk or Text? Evaluating Response Rates by Remote Survey Method during COVID-19

“…6 Second, our findings contribute to a rich literature that examines the effects of self-administration on data quality and reporting related to sensitive survey items, such as intimate-partner violence (Aguero and Frisancho, 2021;Cullen, 2020); contraceptive use (Greenleaf et al, 2020); and other health issues (Kays et al, 2012). 7 Initial research suggests that interviews conducted via text messaging on mobile devices essentially mimic self-administration in terms of higher reporting of sensitive behaviors and higher data quality in general (Brenner and DeLamater, 2014;Cocco and Tuzzi, 2013;Schober and Conrad, 2015). Interviews using text messages also remove the potential for interviewer effects on responses due to observable characteristics of the interviewers, on which there is ample research (Schaeffer et al, 2010).…”

When Do Misunderstandings Matter? Evidence From Survey Interviews About Smoking

“…The phenomenon of interpretive variability we observe in these survey questions may well be related to interpretive variability in language more generally (see Kurtz & Schober, , for evidence on how fiction readers' interpretation of themes in short stories can be surprisingly divergent). But survey interviews have particular features (Houtkoop‐Steenstra, ; Schaeffer, ; Schaeffer & Maynard, , ; Schober & Conrad, ) that make them distinct from other kinds of dialog or language comprehension settings, beyond the fact that they are consequential for social measurement (Schober & Conrad, ). In particular, unlike in references to objects in physical settings, in which interlocutors can have immediate evidence about what their partner means and when understanding has gone wrong, the autobiographical circumstances (behaviors and opinions) about which survey respondents answer are not immediately visible to the researcher.…”

“…In this study, our aim was to explore the prevalence of different types of misunderstandings and the consequences for survey estimates, as well as how attempts to repair misunderstanding succeed and fail, in a full‐length U.S. government survey. Our strategy was to ask respondents in the laboratory to participate in a telephone interview using an actual questionnaire deployed in its entirety (rather than excerpting questions from multiple surveys as in, e.g., Conrad & Schober, ; Lind, Schober, Conrad, & Reichert, ; Schober et al., ), and answering questions about their own lives rather than fictional scenarios (as in, e.g., Conrad et al., ; Schober & Conrad, ; Schober et al., ). Using an actual complete questionnaire that included not only behavioral but also opinion questions allowed us to better quantify the potential consequences of misunderstanding in real‐world social measurement, where the outcomes of the survey can have major implications for policy (Schober & Conrad, ).…”

“…Our strategy was to ask respondents in the laboratory to participate in a telephone interview using an actual questionnaire deployed in its entirety (rather than excerpting questions from multiple surveys as in, e.g., Conrad & Schober, ; Lind, Schober, Conrad, & Reichert, ; Schober et al., ), and answering questions about their own lives rather than fictional scenarios (as in, e.g., Conrad et al., ; Schober & Conrad, ; Schober et al., ). Using an actual complete questionnaire that included not only behavioral but also opinion questions allowed us to better quantify the potential consequences of misunderstanding in real‐world social measurement, where the outcomes of the survey can have major implications for policy (Schober & Conrad, ). The laboratory setting allowed us to examine respondents' interpretations of survey concepts in a more detailed way than has been done previously (e.g., in Conrad & Schober, ), using post‐interview measures that assess (a) respondents' conceptualizations and (b) how their answers might change if they used a standard definition.…”

Comprehension and engagement in survey interviews with virtual agents

“…Results from these interviews can have far-reaching consequences: even small changes in reported US unemployment rates, for example, can affect world financial markets, and results from the ESS make “a major contribution to the creation of effective social and economic policies in Europe” (Geoghegan-Quinn, 2012 ). So understanding what leads to accurate responses, and to participants' willingness to engage in such surveys, is societally important (Schober and Conrad, 2015 ).…”