Performance Validity in Collegiate Football Athletes at Baseline Neurocognitive Testing

Abstract

Objective: To assess the prevalence of invalid performance on baseline neurocognitive testing using embedded measures within computerized tests and individually administered neuropsychological measures, and to examine the influence of incentive status and performance validity on neuropsychological test scores. Setting: Sport-related concussion management program at a regionally accredited university. Participants: A total of 83 collegiate football athletes completing their preseason baseline assessment within the University's concussion management program and a control group of 140 nonathlete students. Design: Cross-sectional design based on differential incentive status: motivated to do poorly to return to play more quickly after sustaining a concussion (athletes) versus motivated to do well due to incentivizing performance (students). Main Measures: Immediate Post-Concussion and Cognitive Testing (ImPACT), performance validity tests, and measures of cognitive ability. Results: Half of the athletes failed at least 1 embedded validity indicator within ImPACT (51.8%), and the traditional neuropsychological tests (49.4%), with large effects for performance validity on cognitive test scores (d: 0.62-1.35), incentive status (athletes vs students; d: 0.36-1.15), and the combination of both factors (d: 1.07-2.20) on measures of attention and processing speed. Conclusion: Invalid performance on baseline assessment is common (50%), consistent across instruments (ImPACT or neuropsychological tests) and settings (one-on-one or group administration), increases as a function of incentive status (risk ratios: 1.3-4.0) and results in gross underestimates of the athletes' true ability level, complicating the clinical interpretation of the postinjury evaluation and potentially leading to premature return to play.

Flipping the Script: Measuring Both Performance Validity and Cognitive Ability with the Forced Choice Recognition Trial of the RCFT

“…This surprising finding underlines the importance of empirically verifying even intuitive and logically appealing assumptions. Our results supported, instead, previously voiced theoretical concerns ( Giromini et al., 2019 ; McWhirter et al., 2019 ; van Helvoort et al., 2019 ) and published data ( Abeare et al., 2019 ; An et al., 2019 ; Hurtubise et al., 2020 ; Roye et al., 2019 ) about the epistemological ambiguity around the incentive status of control groups comprised of undergraduate research volunteers. Researchers long assumed that cognitively healthy university students assigned to the control condition would demonstrate their highest ability level by default.…”

“…Our data were largely consistent with the domain specificity effect, in that similarity in the cognitive domain (attention, memory, processing speed, verbal reasoning) or sensory modality (auditory, visual, tactile) between predictor and criterion PVTs influenced classification accuracy ( Abeare, Sabelli, et al., 2019 ; Erdodi, 2019 ; Schroeder et al., 2019 ). AUC values for the RCFT based EVIs were consistently higher against the EI-5 MEM (the modality-congruent validity composite) than the EI-5 PSP (the modality-incongruent validity composite), suggesting that instrumentation artifacts may exert a subtle but detectable influence on signal detection analyses.…”

“…When relevant, we computed the base rate of failure (BR Fail ; i.e., the percent of the sample that failed a given cutoff). The prevalence of the condition of interest (in this context, BR Fail ) is a descriptive statistic that is important for understanding classification accuracy in general (Wald & Bestwick, 2014) and in the context of performance validity assessment specifically (Abeare, Messa, et al, 2019).…”

“…When relevant, we computed the base rate of failure (BR Fail ; i.e., the percent of the sample that failed a given cutoff). The prevalence of the condition of interest (in this context, BR Fail ) is a descriptive statistic that is important for understanding classification accuracy in general ( Wald & Bestwick, 2014 ) and in the context of performance validity assessment specifically ( Abeare, Messa, et al., 2019 ). Although area under the curve (AUC) is useful for comparing overall classification accuracy across models ( Altman & Bland, 1994 ; Fawcett, 2006 ; Marzban, 2004 ), its clinical relevance has been called into question ( Hand, 2009 ; Lobo et al., 2008 ; Wald & Bestwick, 2014 ).…”

“…The unexpectedly high BR Fail on the free-standing PVTs (9.0%-29.2%) in the incentivized control group puts the low BR Fail on the Yes/No Recognition and RCFT FCR trials (1.2%) in perspective. Namely, it neutralizes arguments that EVIs inevitably conflate genuine impairment and non-credible responding (Glassmire et al., 2019; Messa et al., 2020 ) when compared to free-standing PVTs that are, by design, robust to genuine and severe cognitive impairment ( Abeare et al., 2019 ; Erdodi & Rai, 2017 ; Whitney et al., 2013 ). If this finding is replicated by future research, it would further enhance the appeal of the RCFT recognition trials as EVIs.…”