A systematic review of validity procedures used in neuropsychological batteries.

Abstract

This study presents a systematic review of validity evidence for neuropsychological batteries. Studies published in international databases between 2005 and 2012 were examined. Considering the specificity of neuropsychological batteries, the aim of the study was to review the statistical analyses and procedures that have been used to validate these instruments. A total of 1,218 abstracts were read, of which 147 involved studies of neuropsychological batteries or tests that evaluated at least three cognitive processes. The full text of each article was analyzed according to publication year, focal instrument of the study, sample type, sample age range, characterization of the participants, and procedures and analyses used to provide evidence of validity. The results showed that the studies primarily analyzed patterns of convergence and divergence by correlating the instruments with other tests. Measures of reliability, such as internal consistency and test-retest reliability, were also frequently employed. To provide evidence of relationships between test scores and external criteria, the most common procedures were evaluations of sensitivity and specificity, and comparisons were made between contrasting groups. The statistical analyses frequently used were Receiver Operating Characteristic analysis, Pearson correlation, and Cronbach's alpha. We discuss the necessity of incorporating both classic and modern psychometric procedures and presenting a broader scope of validity evidence, which would represent progress in this field. Finally, we hope our findings will help researchers better plan the validation process for new neuropsychological instruments and batteries.

Cognitive Screening (TRIACOG) for adults with cerebrovascular diseases: Construction process and validity evidence.

“…The TRIACOG presented evidence of criterion validity as statistically significant differences were found for all the tasks when comparing contrasting groups (healthy and poststroke individuals). This type of evidence is important in neuropsychological assessments to ensure that the instrument is capable of identifying participants who present indicators of cognitive impairment (Pawlowski et al, 2013; Urbina, 2007). Therefore, evidence of an instrument’s criterion validity includes its ability to predict external variables (or criteria) to ensure that the tool is capable of identifying deficits that are consistent with certain conditions and diagnoses (AERA, APA, & NCME, 2014; Urbina, 2007), in this case, forms of impairment after a stroke.…”

“…To construct an appropriate cognitive screening measure, it is recommended that instruments to be used by health professionals be able to be administered to most patients in 5–20 min, include a large number of cognitive domains, and be sensitive to the clinical conditions under which it will be applied (Malloy et al, 1997). Furthermore, the construction of neuropsychological instruments must follow specific steps and provide ample evidence of validity and reliability (American Educational Research Association, American Psychological Association, & National Council on Measurement in Education [AERA, APA, & NCME], 2014; Pawlowski, Segabinazi, Wagner, & Bandeira, 2013).…”

Rationale and Design of the National Neuropsychology Network

“…Lengthy assessment protocols further limit neuropsychology access because fewer patients can be tested each day, and long appointment wait times preclude timely evaluations which detracts from efficient patient management. Clinical test validation has also been constrained by diagnostic criteria variability and selection biases associated with samples of convenience in which the base rates of performance patterns are unknown (Pawlowski, Segabinazi, Wagner, & Bandeira, 2013). Thus, many "rules of thumb" for diagnostic decision-making are insufficiently validated across the spectrum of clinical conditions that may be referred for neuropsychological evaluation (Chaytor & Schmitter-Edgecombe, 2003;Duff, Suhrie, Dalley, Anderson, & Hoffman, 2019;Hoogland et al, 2018;Raspall et al, 2005).…”

Using Multiverse Analysis to Highlight Differences in Convergent Correlation Outcomes Due to Data Analytical and Study Design Choices

“…The focus of this article is on convergent validity as one of the most common methods to investigate the validity of neuropsychological assessment in general ( Pawlowski et al, 2013 ), and particularly so with regards to the example of a Multiple Errands Tasks we will present here ( Webb et al, 2021 ). The 2014 Standards for Educational and Psychological Testing (Standards 1.12 and1.16) state that if a cognitive process is being assessed via a test, in this case executive functioning, then evidence must be generated that the test does assess that cognitive process ( American Educational Research Association et al, 2014 ), and to generate evidence using well construct-validated tools.…”

“…A 2013 systematic review of validation procedures for neuropsychological batteries aimed at clinical populations, examined 147 studies that were published between 2005 and 2012 ( Pawlowski et al, 2013 ). The clinical populations in the reviewed studies predominantly included dementia and Alzheimer’s disease, mild cognitive impairment, and acquired brain injury, though other clinical groups were also present.…”