Identifying careless responses in survey data.

Abstract

When data are collected via anonymous Internet surveys, particularly under conditions of obligatory participation (such as with student samples), data quality can be a concern. However, little guidance exists in the published literature regarding techniques for detecting careless responses. Previously several potential approaches have been suggested for identifying careless respondents via indices computed from the data, yet almost no prior work has examined the relationships among these indicators or the types of data patterns identified by each. In 2 studies, we examined several methods for identifying careless responses, including (a) special items designed to detect careless response, (b) response consistency indices formed from responses to typical survey items, (c) multivariate outlier analysis, (d) response time, and (e) self-reported diligence. Results indicated that there are two distinct patterns of careless response (random and nonrandom) and that different indices are needed to identify these different response patterns. We also found that approximately 10%-12% of undergraduates completing a lengthy survey for course credit were identified as careless responders. In Study 2, we simulated data with known random response patterns to determine the efficacy of several indicators of careless response. We found that the nature of the data strongly influenced the efficacy of the indices to identify careless responses. Recommendations include using identified rather than anonymous responses, incorporating instructed response items before data collection, as well as computing consistency indices and multivariate outlier analysis to ensure high-quality data.

Amazon’s Mechanical Turk as a Viable Source for Organizational and Occupational Health Research

“…In total, the Time 1 survey was completed by 1516 participants. To address potential participant careless responding (Meade and Craig 2012), items were included to detect insufficient effort responding (IER). Though a number of terms have been used in the past (e.g., random responding), IER refers to "a response set in which the respondent answers a survey measure with low or little motivation to comply with survey instructions, correctly interpret item content, and provide accurate responses" (Huang et al 2012, p. 100).…”

Reversed item bias: An integrative model.

“…The term careless responding is sometimes used very broadly to refer to any type of random or nonrandom response to survey questions that is not based on content (see Meade & Craig, 2012, for a recent example). From a psychological perspective, such a broad conception of careless responding is problematic because many different mechanisms could lead to responding that is not sufficiently sensitive to content.…”

The communality-bonus effect for male transformational leaders – leadership style, gender, and promotability

“…While data for Study 1a were collected with paper-pencil questionnaires distributed at the university, we collected Study 1b online by posting the study link on a professional networking website. Due to the greater susceptibility of rash or careless responses in online surveys that pose a threat to data quality, we followed the suggestions of various researchers and added several checks for inappropriate responses in Study 1b (Huang, Liu, & Bowling, 2015;Meade & Craig, 2012). To ensure data quality, we employed three robust manipulation and data quality checks in Study 1b and excluded participants who did not meet these criteria (Meade & Craig, 2012).…”

“…Due to the greater susceptibility of rash or careless responses in online surveys that pose a threat to data quality, we followed the suggestions of various researchers and added several checks for inappropriate responses in Study 1b (Huang, Liu, & Bowling, 2015;Meade & Craig, 2012). To ensure data quality, we employed three robust manipulation and data quality checks in Study 1b and excluded participants who did not meet these criteria (Meade & Craig, 2012). We initially collected data from 389 working adults, but participants were only included in the final sample if they answered the manipulation checks correctly (see later) as well as a test question ("It is important to us that you read all questions.…”