Difference in Response Effort Across Sample Types: Perception or Reality?

Abstract

Landers and Behrend (2015) question organizational researchers’ stubborn reliance on sample source to infer the validity of research findings, and they challenge the arbitrary distinctions researchers often make between sample sources. Unconditional favoritism toward particular sampling strategies (e.g., organizational samples) can restrict choices in methodology, which in turn may limit opportunities to answer certain research questions. Landers and Behrend (2015) contend that no sampling strategy is inherently superior (or inferior), and therefore, all types of samples warrant careful consideration before any validity-related conclusions can be made. Despite sound arguments, the focal article focuses its consideration on external validity and deemphasizes the potential influence of sample source on internal validity. Agreeing with the position that no samples are the “gold standard” in organizational research and practice, we focus on insufficient effort responding (IER; Huang, Curran, Keeney, Poposki, & DeShon, 2012) as a threat to internal validity across sample sources.

The Effects of Inattentive Responding on Construct Validity Evidence When Measuring Social–Emotional Learning Competencies

“…Moreover, evidence of unidimensionality improved and coefficient alpha increased slightly after removing respondents flagged for IER (Huang et al., ), and coefficient alpha and multiple regression R 2 values were higher for attentive respondents (Maniaci & Rogge, ). Despite this evidence and perceptions that IER has moderate impacts on survey results (Liu, Bowling, Huang, & Kent, ), the use of IER detection methods is rare in published research (Ran, Liu, Marchiondo, & Huan, ).…”

Untitled

“…It is imperative to have similar, reliable, and valid sample sizes. To deem a study valid, results must be similar among multiple samples regardless of sampling strategies (Ran, Liu, Marchiondo, & Huang, 2015). Online studies have been shown to typically have a higher level of carelessness and insufficient effort responding than paper and pencil studies (Huang, Bowling, Liu, & Li, 2014).…”

Insufficient Effort Survey Responding: An Under‐Appreciated Problem in Work and Organisational Health Psychology Research

“…As noted, organisational researchers generally do not report treatment of IER in published studies (Ran et al, ). This may reflect an overall inattention to the issue of IER (see Liu et al, ), or simply reflect researchers’ omission in reporting such screening.…”

“…Liu, Bowling, Huang, and Kent () surveyed Society for Industrial and Organizational Psychology members and results indicated that only 37 per cent reported checking for IER “a lot” or “almost all the time”. Ran, Liu, Marchiondo, and Huang () examined data‐screening procedures pertaining to IER detection for articles published from 2012 to 2014 in the Journal of Applied Psychology , Journal of Management , and Academy of Management Journal , and found that IER screening procedures were reported in only three of 463 survey data samples.…”