Using Mouse Movements to Predict Web Survey Response Difficulty

Abstract

A key goal of survey interviews is to collect the highest quality data possible from respondents. In practice, however, it can be difficult to achieve this goal because respondents do not always understand particular survey questions as designers intended. Researchers have used a variety of indicators to identify and predict respondent confusion and difficulty in answering questions in different modes. In web surveys, it is possible to automatically detect response difficulty in real time. The research to date has focused on response latencies—mostly long response times—as evidence of difficulty. In addition to response latencies, however, web surveys offer rich behavioral data, which may predict respondent confusion and difficulty more directly than response times. This article focuses on one such behavior, mouse movements. We examine a set of mouse movements participants engage in when answering questions about experimental scenarios whose difficulty has been manipulated (as confirmed by respondent ratings). This approach makes it possible to determine which movements are general movements, demonstrating how a person interacts with a computer, and which movements are related to response difficulty. We find not only that certain mouse movements are highly predictive of difficulty but also that such movements add considerable value when used in conjunction with response times. The approach developed in this article may be useful in delivering help to confused respondents in real time and as a diagnostic tool to identify confusing questions.

Predicting Question Difficulty in Web Surveys: A Machine Learning Approach Based on Mouse Movement Features

“…These might be some of the reasons why our accuracies are only moderately high. A complementary approach could measure participants' subjective difficulty for a given question and use it as the outcome for prediction (Horwitz, Kreuter, & Conrad, 2017). Also, future research could include additional difficulty manipulations or manipulate different types of difficulty within the same question in a crossed design.…”

Knowledge extraction from pointer movements and its application to detect uncertainty

“…The classification of certainty versus uncertainty was correct in 89% of the cases. The estimated performance of the model was, therefore, better than that of ( Horwitz et al, 2017 ). This improvement might relate to the choice of features used to indicate uncertainty.…”

“…The trajectory has been assessed in terms of horizontal direction inversions ( Zushi et al, 2012 ) and deviation from the idealized straight-line trajectory ( Schneider et al, 2015 ). More recently, Horwitz et al (2017) , used mouse cursor trajectories to predict response difficulty, achieving a performance accuracy of between 74% and 79%. Significant predictors of uncertainty were horizontal directional inversions, hovering the mouse cursor over a question for more than 2s, and marking a response option for more than 2s ( Horwitz et al, 2017 ).…”

“…More recently, Horwitz et al (2017) , used mouse cursor trajectories to predict response difficulty, achieving a performance accuracy of between 74% and 79%. Significant predictors of uncertainty were horizontal directional inversions, hovering the mouse cursor over a question for more than 2s, and marking a response option for more than 2s ( Horwitz et al, 2017 ).…”

How Effective Are Eye-Tracking Data in Identifying Problematic Questions?

“…Therefore, survey researchers have to check for potential cognitive hurdles and their underlying causes by evaluating their draft questions (Fowler, 2013;Miller, 2014). There is a large variety of question testing tools available, such as cognitive interviews, response latency measurement, expert reviews (Presser et al, 2004), or paradata such as mouse movements (Horwitz, Kreuter, & Conrad, 2017). Additionally, the analysis of eye-movement data is an apparently promising technique for identifying problematic survey questions (Kamoen, Holleman, Mak, Sanders, & Van Den Bergh, 2017;.…”