Will the Questions Ever End? Person-Level Increases in Careless Responding During Questionnaire Completion

Bowling, Nathan A.; Gibson, Anthony; Houpt, Joseph W.; Brower, Cheyna Katherine

doi:10.1177/1094428120947794

Cited by 72 publications

(66 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Costs (e.g., time or cognitive effort) and benefits (e.g., financial gratification or intrinsic motivation) are set off against each other to evaluate the willingness to further participate at a specific point of time. That is, not only the willingness to answer superficially increases but also the probability to abort the questionnaire ( Bowling et al, 2020 ). Thus, it seems plausible to consider the phenomena of attrition and careless responding in tandem ( Meade et al, 2017 ).…”

Section: Discussionmentioning

confidence: 99%

Detecting Careless Responding in Survey Data Using Stochastic Gradient Boosting

Schroeders

Schmidt²,

Gnambs

2021

Educational and Psychological Measurement

View full text Add to dashboard Cite

Careless responding is a bias in survey responses that disregards the actual item content, constituting a threat to the factor structure, reliability, and validity of psychological measurements. Different approaches have been proposed to detect aberrant responses such as probing questions that directly assess test-taking behavior (e.g., bogus items), auxiliary or paradata (e.g., response times), or data-driven statistical techniques (e.g., Mahalanobis distance). In the present study, gradient boosted trees, a state-of-the-art machine learning technique, are introduced to identify careless respondents. The performance of the approach was compared with established techniques previously described in the literature (e.g., statistical outlier methods, consistency analyses, and response pattern functions) using simulated data and empirical data from a web-based study, in which diligent versus careless response behavior was experimentally induced. In the simulation study, gradient boosting machines outperformed traditional detection mechanisms in flagging aberrant responses. However, this advantage did not transfer to the empirical study. In terms of precision, the results of both traditional and the novel detection mechanisms were unsatisfactory, although the latter incorporated response times as additional information. The comparison between the results of the simulation and the online study showed that responses in real-world settings seem to be much more erratic than can be expected from the simulation studies. We critically discuss the generalizability of currently available detection methods and provide an outlook on future research on the detection of aberrant response patterns in survey research.

show abstract

Section: Discussionmentioning

confidence: 99%

Detecting Careless Responding in Survey Data Using Stochastic Gradient Boosting

Schroeders

Schmidt²,

Gnambs

2021

Educational and Psychological Measurement

View full text Add to dashboard Cite

show abstract

“…Since then, it has been shown that satisficing degrades the quality of survey results [2], [3], [10]- [13]. In particular, in online questionnaires, this problem is a factor that significantly contributes to quality deterioration [14]. Oppenheimer et al [2] proposed the instructional manipulation check (IMC), which inserts instructions such as "Please move to the next page without answering any question" in the instructional text, and conducted an online questionnaire using the IMC.…”

Section: Related Work and Challengesmentioning

confidence: 99%

“…The intra-subject deviation is the difference between the baseline measured on pages 5 and 6 and the mean of all pages. The reason for adding this deviation is that careless responses increase as the respondents proceed through the questionnaire [14]. Based on this fact, we assumed that the feature difference between the beginning and end of the questionnaire might contribute to the detection of careless responses.…”

Section: A Machine Learning Modelsmentioning

confidence: 99%

Detection of Careless Responses in Online Surveys Using Answering Behavior on Smartphone

et al. 2021

View full text Add to dashboard Cite

Some respondents make careless responses due to the "satisficing," which is an attempt to complete a questionnaire as quickly and easily as possible. To obtain results that reflect a fact, detecting satisficing and excluding the responses with satisficing from the analysis targets are required. One of the devised methods detects satisficing by adding questions that check violations of instructions and inconsistencies. However, this approach may cause respondents to lose their motivation and prompt them to satisficing. Additionally, a deep learning model that automatically answers these questions was reported. This threatens the reliability of the conventional method. To detect careless responses without inserting such screening questions, machine learning (ML) detection using data obtained from answer results was attempted in a previous study, with a detection rate of 55.6%, which is not sufficient from the viewpoint of practicality. Therefore, we hypothesized that a supervised ML model with a higher detection rate could be constructed by using on-screen answering behavior as features. However, (1) no existing questionnaire system can record on-screen answering behavior and (2) even if the answering behavior can be recorded, it is unclear which answering behavior features are associated with satisficing. We developed an answering behavior recording plug-in for LimeSurvey, an online questionnaire system used all over the world, and collected a large amount of data (from 5,692 people) in Japan. Then, a variety of features were examined and generated from answering behavior, and we constructed ML models to detect careless responses. We call this detection method the ML-ABS (ML-based answering behavior scale). Evaluation by cross-validation demonstrated that the detection rate for careless responses was 85.9%, which is much higher than the previous ML method. Among the various features we proposed, we found that reselecting the Likert scale and scrolling particularly contributed to the detection of careless responses.

show abstract

“…Additionally, certain scales, such as those that measure socially desirable traits, may facilitate non-random IER whereas other scales that do not necessarily have a socially desirable aspect, may facilitate random IER. It may be that participants read the first few items and construct a schema on how to respond to the items but engage in less effort as they progress through the survey, regardless of pattern ( Bowling et al, 2020 ). Ultimately, while it may be that direct IER is the least-related form of IER to response styles, it can still be assessed within a larger battery of IER assessment, contributing to the holistic determination of IER for participants.…”

Section: Discussionmentioning

confidence: 99%

The Relationship of Insufficient Effort Responding and Response Styles: An Online Experiment

Alarcon

Lee

2022

Front. Psychol.

View full text Add to dashboard Cite

While self-report data is a staple of modern psychological studies, they rely on participants accurately self-reporting. Two constructs that impede accurate results are insufficient effort responding (IER) and response styles. These constructs share conceptual underpinnings and both utilized to reduce cognitive effort when responding to self-report scales. Little research has extensively explored the relationship of the two constructs. The current study explored the relationship of the two constructs across even-point and odd-point scales, as well as before and after data cleaning procedures. We utilized IRTrees, a statistical method for modeling response styles, to examine the relationship between IER and response styles. To capture the wide range of IER metrics available, we employed several forms of IER assessment in our analyses and generated IER factors based on the type of IER being detected. Our results indicated an overall modest relationship between IER and response styles, which varied depending on the type of IER metric being considered or type of scale being evaluated. As expected, data cleaning also changed the relationships of some of the variables. We posit the difference between the constructs may be the degree of cognitive effort participants are willing to expend. Future research and applications are discussed.

show abstract

Will the Questions Ever End? Person-Level Increases in Careless Responding During Questionnaire Completion

Cited by 72 publications

References 29 publications

Detecting Careless Responding in Survey Data Using Stochastic Gradient Boosting

Detecting Careless Responding in Survey Data Using Stochastic Gradient Boosting

Detection of Careless Responses in Online Surveys Using Answering Behavior on Smartphone

The Relationship of Insufficient Effort Responding and Response Styles: An Online Experiment

Contact Info

Product

Resources

About