It is an accepted fact in survey research that not all participants will respond to items with the thoughtful introspection required to produce a valid response. When participants respond without sufficient effort their responses are considered to be careless, and these responses represent error. Many methods exist for the detection of these individuals (Huang, Curran, Keeney, Poposki, & Deshon, 2012; Johnson, 2005; Meade & Craig, 2012), and several techniques exist for testing their effectiveness. These techniques often involve generating careless responses through some process, then attempting to detect those known cases in otherwise normal data. One method to produce these data is through the simulation of data with varying degrees of randomness. Despite the common use of this technique, we know little about how it actually maps onto real world data. The purpose of this paper is to compare simulated data with real world data on commonly used careless response metrics. Results suggest that care should be applied when simulating data, and that decisions researchers make when generating this data can have large effects on the apparent effectiveness of these metrics. Despite these potential limitations, it appears that with proper use and continued research simulation techniques can still be quite valuable.

Goldammer et al. (2020) examined the performance of careless response detection indices by experimentally manipulating survey instructions to induce careless responding, then compared the ability of various indices to detect these induced careless responses. Based on these analyses, Goldammer et al. concluded that metrics designed to detect overly consistent response patters (i.e. longstring and IRV) were ineffective. In this comment, we critique this conclusion by highlighting critical problems with the experimental manipulation used. Specifically, Goldammer et al.’s manipulations only encouraged overly inconsistent, or random, responding and thus did not induce the full range of careless response behavior that is present in natural careless responding. As such, it is unsurprising that metrics designed to detect overly consistent responding did not appear to be effective. Because the full range of careless behavior was not induced, Goldammer et al.’s study cannot address the utility of longstring or similar metrics. We offer recommendations for alternative experimental manipulations that may produce more naturalistic and diverse careless responding.

Yarkoni (2020) highlights patterns of overgeneralization in psychology research. In this comment, we note that such challenges also pertain to applied psychological and organizational research and prac-tice. We use two examples—cross-cultural generalizability and implicit bias training—to illustrate common practices of overgeneralization from narrow research samples to broader operational popula-tions. We conclude with recommendations for research and practice.