The Evaluation of Statistical Process Control Methods to Monitor Interview Duration During Survey Data Collection

Abstract

Despite general agreement regarding the usefulness of statistical process control (SPC) tools for monitoring paradata, using SPC from an early phase of the survey fieldwork is rather rare. This study focuses on one type of paradata-interview duration-to fill this void. First, we establish a procedure based on the idea of enabling fieldwork monitoring for the seventh round of the European Social Survey in Belgium from its start. The impact of respondent characteristics on interview duration is controlled for by multiple regression. Moreover, we simulate the real conditions of an ongoing survey data collection process by cumulating data and repeating the identification of problematic interviews each week, on the basis that "new" data are available. Second, for each interview we record and track the results with regard to whether or not it is problematic over the fieldwork period, to examine the consistency of our findings. We find that as more data becomes available, the results concerning whether an interview is problematic changes in only 0.3% of the cases. Out of the 27 interviews identified as problematic when all information was available, 25 were immediately identified once relevant information was available. Overall, these findings suggest that SPC tools are reliable and efficient in a survey context, and accordingly have great potential for allowing survey practitioners to focus on the interviews for which further examination is needed immediately, rather than when the data collection has been completed.

Assessing Response Quality by Using Multivariate Control Charts for Numerical and Categorical Response Quality Indicators

“…However, care should be taken when including indicators that are expected and "allowed" to vary across respondents (or interviewers) or across time (e.g., interview duration). We would suggest that before being used to identify outliers, the impact of influencing factors on these indicators should be controlled for using multiple regression models (Jin et al 2019), multilevel models (Couper and Kreuter 2012), or penalized spline models (Schafer 2013).…”

SPC-based model for evaluation of training processes in industrial context

“…• Demonstration of the usefulness of SPC for monitoring paradata for assessing the quality of data obtained in surveys (Jin, Vandenplas & Loosveldt, 2019).…”

Nonparametric multivariate control chart for numerical and categorical variables

“…Out of its available tools, the control chart is the most important and most widely used. Although initially applied in industrial fields, the control chart has found uses in many non-industrial fields, such as public health (see the review article by Thor et al 2007), finance (e.g., Kovářık and Sarga 2014;Bodnar and Schmid 2011;Bilson et al 2010), and even surveys (Jin and Loosveldt 2020;Jin et al 2019;Sirkis et al 2011). It calculates control limits using statistical equations, and graphically presents the fluctuations of the quality characteristic of a process.…”