Detection of Careless Responses in Online Surveys Using Answering Behavior on Smartphone

Abstract

Some respondents make careless responses due to the "satisficing," which is an attempt to complete a questionnaire as quickly and easily as possible. To obtain results that reflect a fact, detecting satisficing and excluding the responses with satisficing from the analysis targets are required. One of the devised methods detects satisficing by adding questions that check violations of instructions and inconsistencies. However, this approach may cause respondents to lose their motivation and prompt them to satisficing. Additionally, a deep learning model that automatically answers these questions was reported. This threatens the reliability of the conventional method. To detect careless responses without inserting such screening questions, machine learning (ML) detection using data obtained from answer results was attempted in a previous study, with a detection rate of 55.6%, which is not sufficient from the viewpoint of practicality. Therefore, we hypothesized that a supervised ML model with a higher detection rate could be constructed by using on-screen answering behavior as features. However, (1) no existing questionnaire system can record on-screen answering behavior and (2) even if the answering behavior can be recorded, it is unclear which answering behavior features are associated with satisficing. We developed an answering behavior recording plug-in for LimeSurvey, an online questionnaire system used all over the world, and collected a large amount of data (from 5,692 people) in Japan. Then, a variety of features were examined and generated from answering behavior, and we constructed ML models to detect careless responses. We call this detection method the ML-ABS (ML-based answering behavior scale). Evaluation by cross-validation demonstrated that the detection rate for careless responses was 85.9%, which is much higher than the previous ML method. Among the various features we proposed, we found that reselecting the Likert scale and scrolling particularly contributed to the detection of careless responses.

Identifying Careless Responding in Web-Based Surveys

“…The BiLSTM model with 25 AAOIs yielded an F1 score of 0.90, indicating an excellent balance between precision and recall, performing markedly better than models based on classical C/IER indices or models that combine classical indices with mouse movement indices (accuracy at 80% or 84%, respectively). The achieved accuracy was higher than in similar previous studies, e.g., Gogami et al (2021), Fernandez-Fontelo et al (2023), or Schroeders et al (2022), in which the model with responses and response times reached only 70% accuracy (and a model with only responses showed even lower accuracy of 59%).…”

“…Their model performed well on simulated data but did not markedly improve careless response detection in empirical data compared to popular C/IER indicators (straightlining, even-odd, etc.). Gogami et al (2021), who used supervised ML to identify C/IER but did not achieve a sufficiently high detection rate, obtained similar results. However, when combined with log data indices (scrolling and response editing), their supervised ML model showed a markedly improved C/IER detection rate.…”

“…Schroeders et al, 2022). Moreover, it showed that including log data can improve the identification of aberrant patterns in the data (Gogami et al, 2021; Ulitzsch, Pohl, et al, 2022). ML models seem adequate tools to model log data information due to their high-dimensional nature that it is often difficult to represent using a set of simple indicators (Leiner, 2019; Pokropek et al, 2023).…”

“…Gogami et al (2021) demonstrated the feasibility of utilizing log data from mobile devices to study C/IER. Future studies should work on improving paradata collection and analysis in this context.…”

“…Response latencies (Leiner, 2019), click counts (Buchanan & Scofield, 2018), and mouse/cursor move trajectories (Horwitz et al, 2017; Pokropek et al, 2023) are examples of C/IER indices based on paradata. Log data contain additional information on survey behavior that is sometimes impossible to deduce from the response vectors alone (Fernandez-Fontelo et al, 2023; Gogami et al, 2021).…”