Asking Probing Questions in Web Surveys

Abstract

Cognitive interviewing is a well-established method for evaluating and improving a questionnaire prior to fielding. However, its present implementation brings with it some challenges, notably in terms of small sample sizes or the possibility of interviewer effects. In this study, the authors test web surveys through nonprobability online panels as a supplemental means to implement cognitive interviewing techniques. The overall goal is to tackle the above-mentioned challenges. The focus in this article is on methodological features that pave the way for an eventual successful implementation of category-selection probing in web surveys. The study reports on the results of 1,023 respondents from Germany. In order to identify implementation features that lead to a high number of meaningful answers, the authors explore the effects of (1) different panels, (2) different probing variants, and (3) different numbers of preceding probes on answer quality. The overall results suggest that category-selection probing can indeed be implemented in web surveys. Using data from two panels—a community panel where members can actively get involved, for example, by creating their own polls, and a “conventional” panel where answering surveys is the members' only activity—the authors find that high community involvement does not increase the likelihood to answer probes or produce longer statements. Testing three probing variants that differ in wording and provided context, the authors find that presenting the context of the probe (i.e., the probed item and the respondent’s answer) produces a higher number of meaningful answers. Finally, the likelihood to answer a probe decreases with the number of preceding probes. However, the word count of those who eventually answer the probes slightly increases with an increasing number of probes.

Pretesting Survey Questions Via Web Probing – Does it Produce Similar Results to Face-to-Face Cognitive Interviewing?

“…This can result in more satisficing response behavior of the respondents (K rosnick 1991) who then do not provide the same depth of information as participants in a f2f cognitive interview (Meitinger and Behr 2016). Nevertheless, Behr and colleagues have shown that web respondents give meaningful answers to open-ended probing questions (Behr et al 2012), and Meitinger and Behr (2016) found that there is an extensive overlap between the results of both methods with respect to identified error types and uncovered themes although cognitive interview respondents provided, on average, more indications of errors than web probing respondents.…”

“…This, in turn, allows researchers to quantify their pretest findings (Behr et al 2012). Second, recruiting participants via the Internet enhances the radius of the regional accessibility.…”

“…As web probing does not involve a cognitive interviewer, respondents have to answer the probing questions in a self-administered form. The method of web probing has recently been recognized as a promising tool for evaluating survey questions, both during the post-survey assessment of item validity (Behr et al 2012(Behr et al , 2013 and as a pretesting method to collect data about response strategies (Edgar 2012).…”

Item comparability in cross-national surveys: results from asking probing questions in cross-national web surveys about attitudes towards civil disobedience

“…The general feasibility of asking probing questions on the Web has already been tested and confirmed by Behr et al (2012) and Behr et al (Forthcoming). In these studies, data collection and substantive analyses have been restricted to Germany only.…”

Mixed Methods in Value Research: An Analysis of the Validity of the Russian Version of the Schwartz Value Survey (SVS) Using Cognitive Interviews, Multidimensional Scaling (MDS), and Confirmatory Factor Analysis (CFA)

“…Furthermore the use of small samples in cognitive pretesting seems problematic, as the results may be too specifi c for the samples chosen. On-line cognitive pretest studies using probing techniques with larger samples ( N > 400) seems be a more valid approach (Behr & Kaczmirek, & Bandilla 2012).…”