Validity and Reliability of Baseline Testing in a Standardized Environment

Abstract

Objective -The Immediate Postconcussion Assessment and Cognitive Testing (ImPACT) is a computerized neuropsychological test battery commonly used to determine cognitive recovery from concussion based on comparing post-injury scores to baseline scores. This model is based on the premise that ImPACT baseline test scores are a valid and reliable measure of optimal cognitive function at baseline. Growing evidence suggests that this premise may not be accurate and a large contributor to invalid and unreliable baseline test scores may be the protocol and environment in which baseline tests are administered. This study examined the effects of a standardized environment and administration protocol on the reliability and performance validity of athletes' baseline test scores on ImPACT by comparing scores obtained in two different group-testing settings. Method -Three hundred-sixty one Division 1 cohort-matched collegiate athletes' baseline data were assessed using a variety of indicators of potential performance invalidity; internal reliability was also examined. Results -Thirty-one to thirty-nine percent of the baseline cases had at least one indicator of low performance validity, but there were no significant differences in validity indicators based on environment in which the testing was conducted. Internal consistency reliability scores were in the acceptable to good range, with no significant differences between administration conditions. Conclusions -These results suggest that athletes may be reliably performing at levels lower than their best effort would produce.

Navigating uncertainty: exploring parents' knowledge of concussion management and neuropsychological baseline testing

“…Consistent with concerns raised in the literature (28)(29)(30), participants were also concerned with the ability for young athletes to purposefully score lower on a NBT to try and RTP faster. B3 expressed these concerns with NBT as a strategy because of the way young athletes could game the system knowing that the baseline scores they provide would make it easier for them to pass the test if they were to suffer a SRC.…”

“…Moreover, NBT demonstrates low test re-test reliability (24)(25)(26)(27); this is particularly problematic in youth sports where cognitive development is dynamic (4,5). Natural variability in cognitive abilities due to personal and contextual factors (e.g., academic stress) and purposefully underperforming (i.e., "sandbagging") on baseline testing also compromises the validity of baseline testing (28)(29)(30). Overemphasis on baseline testing may overshadow other critical aspects of concussion management, such as graded RTP and learning and psychological readiness to return-to-sport.…”

Acute evaluation of sport-related concussion and implications for the Sport Concussion Assessment Tool (SCAT6) for adults, adolescents and children: a systematic review

“…In contrast, decline on one SCAT5 test at the 0.95 CI (12.9% of sample) or two or more tests at the 0.80 CI (8.3% of sample) is sufficiently rare to warrant clinical concern supporting possible concussion diagnosis. Documented methods for improving the reliability/accuracy of cognitive measures in sports included: use of multiple baselines74 75; development of cognitive composite scores76–79; use of tests that minimise test ceiling/floor effects; and incorporating measures/procedures to ensure performance validity at baseline 80–85…”

Valid, invalid, or somewhere in between? Baseline Immediate Post-Concussion Assessment and Cognitive Testing (ImPACT) and stand-alone performance validity testing in collegiate athletes.

“…The large discrepancies between invalidity identified by PVTs and EVIs suggest significant over-or underidentification (and possibly both) and that the needed level of nuance in validity evaluation in baseline cognitive testing may not have a simple answer. If as large a percentage of baseline data has questionable validity as has been suggested by this and other research (Higgins et al, 2018), the baseline testing model quickly becomes untenable. The answer may not be to do away with baseline testing, but to change the paradigms that we use to evaluate baseline functioning and performance validity.…”