The Effect of Probing “Don’t Know” Responses on Measurement Quality and Nonresponse in Surveys

Abstract

In survey interviews, "Don't know" (DK) responses are commonly treated as missing data. One way to reduce the rate of such responses is to probe initial DK answers with a follow-up question designed to encourage respondents to give substantive, non-DK responses. However, such probing can also reduce data quality by introducing additional or differential measurement error. We propose a latent variable model for analyzing the effects of probing on responses to survey questions. The model makes it possible to separate measurement effects of probing from true differences between respondents who do and do not require probing. We analyze new data from an experiment which compared responses to two multi-item batteries of questions with and without probing. In this study, probing reduced the rate of DK responses by around a half. However, it also had substantial measurement effects, in that probed answers were often weaker measures of constructs of interest than were unprobed answers. These effects were larger for questions on attitudes than for pseudo-knowledge questions on perceptions of external facts. The results provide evidence against the use of probing of "Don't know" responses, at least for the kinds of items and respondents considered in this study.

Understanding ‘Don't know’ answers to survey questions - an International comparative analysis using interview paradata

“…The removal of 'Don't know' options from attitudinal scales has been shown to encourage some respondents to provide directional responses that they would have otherwise withheld, but which were consistent with answers given to questions about similar issues when a 'Don't know' option was included (Bradburn and Sudman, 1988;Gilljam and Granberg, 1993). Research by Kuha et al, (2018) has revealed how follow up probes can help respondents give a directional response. However there is a time cost to this and there can be a measurement effect as the probe can affect the response given.…”

Inflation Target Credibility in Times of High Inflation

“…Researchers may probe survey participants that "do not know" to obtain more information from those participants that might have been too comfortable to answer seriously in the first round. However, follow-up probing questions may lead to a distorted measure of what people truly think about future inflation, see Kuha et al (2018). In some application, it make sense to omit the "do not know" option from the list of possible answers.…”

Comparing Self-Reported and Partnership-Inferred Sexual Orientation in Household Surveys

“…First, in cases where respondents have difficulties in understanding a question, interviewers can clarify questions and answer options, thus helping respondents to provide a valid answer. Second, interviewers may actively probe and argue in order to obtain valid responses (Kuha et al 2014). Third, many respondents likely perceive that not providing an answer is an undesired behavior in front of the interviewer because it runs counter to the main purpose of the interview of collecting valid information.…”