Insufficient effort responding: Examining an insidious confound in survey data.

Abstract

Insufficient effort responding (IER; Huang, Curran, Keeney, Poposki, & DeShon, 2012) to surveys has largely been assumed to be a source of random measurement error that attenuates associations between substantive measures. The current article, however, illustrates how and when the presence of IER can produce a systematic bias that inflates observed correlations between substantive measures. Noting that inattentive responses as a whole generally congregate around the midpoint of a Likert scale, we propose that Mattentive, defined as the mean score of attentive respondents on a substantive measure, will be negatively related to IER's confounding effect on substantive measures (i.e., correlations between IER and a given substantive measure will become less positive [or more negative] as Mattentive increases). Results from a personality questionnaire (Study 1) and a simulation (Study 2) consistently support the hypothesized confounding influence of IER. Using an employee sample (Study 3), we demonstrated how IER can confound bivariate relationships between substantive measures. Together, these studies indicate that IER can inflate the strength of observed relationships when scale means depart from the scale midpoints, resulting in an inflated Type I error rate. This challenges the traditional view that IER attenuates observed bivariate correlations. These findings highlight situations where IER may be a methodological nuisance, while underscoring the need for survey administrators and researchers to deter and detect IER in surveys. The current article serves as a wake-up call for researchers and practitioners to more closely examine IER in their data.

Methods for the detection of carelessly invalid responses in survey data

“…One type of this error comes from respondents who provide invalid data to survey questions. Fortunately, this is error that can be removed from the survey data collection process in order to produce more stable and consistent results (Huang, Liu, & Bowling, 2015;Maniaci & Rogge, 2014).…”

“…Unfortunately, some participants will simply not put in the effort required to respond accurately or thoughtfully to all questions asked of them. The inclusion of these responses into a set of otherwise accurate data can have a host of unexpected and undesired effects on relationships being examined (Huang et al, 2015).…”

Amazon’s Mechanical Turk as a Viable Source for Organizational and Occupational Health Research

“…Though a number of terms have been used in the past (e.g., random responding), IER refers to "a response set in which the respondent answers a survey measure with low or little motivation to comply with survey instructions, correctly interpret item content, and provide accurate responses" (Huang et al 2012, p. 100). Considering IER can produce a systematic bias in survey research that inflates observed relationships between measures (see Huang et al 2015), we included four instructed response items (e.g., "Please select strongly agree for this item") to remove this potential confound. Three participants responded incorrectly to all four IER items, four participants responded incorrectly to three IER items, 11 participants responded incorrectly to two IER items, and 80 participants responded incorrectly to one IER items, resulting in a sample of n = 1418 (93.54% of study respondents) to answer Research Questions 1 and 2.…”

The communality-bonus effect for male transformational leaders – leadership style, gender, and promotability

“…While data for Study 1a were collected with paper-pencil questionnaires distributed at the university, we collected Study 1b online by posting the study link on a professional networking website. Due to the greater susceptibility of rash or careless responses in online surveys that pose a threat to data quality, we followed the suggestions of various researchers and added several checks for inappropriate responses in Study 1b (Huang, Liu, & Bowling, 2015;Meade & Craig, 2012). To ensure data quality, we employed three robust manipulation and data quality checks in Study 1b and excluded participants who did not meet these criteria (Meade & Craig, 2012).…”

“…Participants in the autocratic (M 1a = 6.28, SD 1a = .67; M 1b = 5.75, SD 1b = .90) as compared to the transformational (M 1a = 4.99, SD 1a = .98; M 1b = 3.10, SD 1b = .99) leadership condition perceived leaders to show significantly more autocratic leadership, t 1a (83) = −7.07, p < .001, t 1b (183) = −18.81, p < .001. In Study 1b, we employed three additional manipulation checks and excluded participants who did not answer the checks correctly to ensure that all participants in our final sample understood the manipulation as intended (cf., Casler, Bickel, & Hackett, 2013;Huang et al, 2015). 2 We were strict about excluding participants with careless responses and those who failed the manipulation checks, because we were specifically interested in reactions to the target leader dependent on whether his or her behaviour was indeed perceived to be autocratic or transformational.…”