I Just Want My Research Credit: Frequency of Suboptimal Effort in a Non-Clinical Healthy Undergraduate Sample

Abstract

Although performance validity testing is becoming fairly routine in clinical settings, research protocols involving neuropsychological tests infrequently include assessments of performance validity. The current study utilized an embedded measure of effort over two administrations of CNS Vital Signs to determine the frequency of poor effort in non-clinical healthy undergraduate students participating in a research study for course credit. Results indicate that more than 1 in 10 college students participating in a cognitive test battery for research showed test scores consistent with inadequate effort, which was associated with poor performance on testing across many domains. This conclusion was supported by poor performance on many other subtests. Healthy college students with suboptimal effort (n = 11) had an overall score in the 15th percentile on average compared to the 48th percentile in the rest of the students (n = 66). Those who failed validity indicators on the baseline administration were more likely to fail validity indicators on the repeat administration. Those who were tested in the morning were also more likely to fail validity indicators. The current study provides evidence for the potential limitations of conducting research using neuropsychological tests with healthy college student volunteers in the absence of performance validity testing. Revised college-level cutoffs are proposed.

Methods for the detection of carelessly invalid responses in survey data

“…The removal of these invalid responders has been shown to reduce error and provide more valid results. Best guesses put the average inclusion of C/IE responders at around 10% (Curran et al, 2010;DeRight & Jorgensen, 2015;Maniaci & Rogge, 2014;Meade & Craig, 2012), but with potentially high variance on that estimate in normal research studies (e.g. Burns et al, 2014;Gough & Bradley, 1996).…”

“…Practical estimates of C/IE responders in typical studies range from as low as about 1% (Gough & Bradley, 1996) to as high as about 30% (Burns, Christiansen, Morris, Periard, & Coaster, 2014), with some consensus among research studies suggesting that the modal rate is somewhere near 8-12% (Curran, Kotrba, & Denison, 2010;DeRight & Jorgensen, 2015;Maniaci & Rogge, 2014;Meade & Craig, 2012). Inclusion of even a low proportion of these responses in datasets impact the usefulness of those data (Curran & Kotrba, 2012;Huang et al, 2015;Maniaci & Rogge, 2014;Woods, 2006), and it is clear that efforts should be made to either deter such data or remove it following collection.…”

Developing a Cognition Endpoint for Traumatic Brain Injury Clinical Trials

“…We know that this assumption is sometimes violated, especially in TBI-related disability claim contexts but in other settings as well. [90][91][92] Several relatively brief performance-based measures of response bias are now available, and could be included in a research battery.…”

Methods for the detection of carelessly invalid responses in survey data

“…The removal of these invalid responders has been shown to reduce error and provide more valid results. Best guesses put the average inclusion of C/IE responders at around 10% (Curran et al, 2010;DeRight & Jorgensen, 2015;Maniaci & Rogge, 2014;Meade & Craig, 2012), but with potentially high variance on that estimate in normal research studies (e.g. Burns, Christiansen, Morris, Periard, & Coaster, 2014;Gough & Bradley, 1996).…”

“…Practical estimates of C/IE responders in typical studies range from as low as about 1% (Gough & Bradley, 1996) to as high as about 30% (Burns, Christiansen, Morris, Periard, & Coaster, 2014), with some consensus among research studies suggesting that the modal rate is somewhere near 8-12% (Curran, Kotrba, & Denison, 2010;DeRight & Jorgensen, 2015;Maniaci & Rogge, 2014;Meade & Craig, 2012). Inclusion of even a low proportion of these responses in datasets impact the usefulness of those data (Curran & Kotrba, 2012;Huang et al, 2015;Maniaci & Rogge, 2014;Woods, 2006), and it is clear that efforts should be made to either deter such data or remove it following collection.…”