Validation of an integrated method for determining cognitive ability: Implications for routine assessments and clinical trials

Abstract

Introduction Although accurate diagnosis of deficit of mild intensity is critical, various methods are used to assess, dichotomize and integrate performance, with no validated gold standard. This study described and validated a framework for the analysis of cognitive performance. Methods This study was performed by using the GREFEX database (724 controls and 461 patients) examined by 7 tests assessing executive functions. The first phase determined the criteria for the cutoff scores, the second phase, the effect of test number on diagnostic accuracy and the third phase, the best methods for combining test scores into an overall summary score. Four validation criteria were used: determination of impaired performance as compared to expected one, false-positive rate ≤5%, detection of both single and multiple impairments with optimal sensitivity. Results The procedure based on 5th percentile cutoffs determined from standardized residuals was the most appropriate procedure. Although AUC increased with the number of scores (p=.0001), the false-positive rate also increased (p=.0001), resulting in suboptimal sensitivity for detecting selective impairment. Two overall summary scores, the average of the seven process scores and the IRT score, had significantly (p=.0001) higher AUCs, even for patients with a selective impairment, and provided higher resulting prevalence of dysexecutive disorders (p=.0001). Conclusions The present study provides and validates a generative framework for the interpretation of cognitive data. Two overall summary score met all 3 validation criteria. A practical consequence is the need to profoundly modify the analysis and interpretation of cognitive assessments for both routine use and clinical research.

Systematic Review and Meta-Analysis of Prevalence in Post-Stroke Neurocognitive Disorders in Hospital-Based Studies

“…To our knowledge, this major effect has never been previously reported and is consistent with previous studies demonstrating the role of four methodological factors influencing the resulting prevalence of cognitive impairment: control for demographic factors (particularly age and education), procedures controlling for the false-positive rate (adjustment of the number of unpaired scores, computation of domain scores or global score), the size of the control group used to determine the cutoff score, and the threshold used to categorize cognitive score as normal versus impaired [45-47]. Eight studies used norms based on a control group: only 6 of these studies indicated the sample size of this group, which was greater than 500 in only 1 study, indicating that at least 5, and possibly 7, of the 8 studies were at risk of inaccurate determination of the cutoff and consequently inaccurate estimation of the prevalence of post-stroke NCD [45-58]. The effect of post-stroke interval supports the previous observation of an increased incidence of dementia according to post-stroke interval [2].…”

“…The second concerns the adoption of a standardized cognitive assessment according to the proposed neuropsychological battery assessing 5 cognitive domains (language, visuo-constructive abilities, memory, action speed, and executive functions), in addition to depression and behavioral changes [49]. The third source of improvement concerns the standardized analysis of a combination of cognitive scores, as proposed in a recent validation study [45]. Finally, common criteria for mild and major post-stroke NCD must be adopted, as recently proposed by the VASCOG group [50].…”

Operationalizing mild cognitive impairment criteria in small vessel disease: the VMCI‐Tuscany Study

“…A recent study found that summary scores, such as averaging of z-scores and item response theory score, provided a more accurate determination of the prevalence of cognitive impairment in a very large sample of 461 patients and 724 controls [38]. Although the use of the number of impaired cognitive scores has been demonstrated to be less sensitive than summary scores, the relatively small sample in our study did not allow the use of such sophisticated methods; that, however, will have to be implemented in further studies on the optimization of operationalization of criterion of mild cognitive impairment Q2 .…”

Executive Functions Deficits After Severe Traumatic Brain Injury

“…1 Briefly, the 19 transformed scores (log transformation for the completion time in the Trail Making and Stroop tests; Box-Cox transformation for the other scores) 39 of the 7 cognitive tests were entered in a series of linear regression analysis (1 per score) with age, educational level, gender, and an interaction term (age × education level) and significant factors in healthy controls were retained. Regression coefficients computed in healthy controls were used to calculate standardized residuals, that is, z scores (poor performance corresponded to a negative z score).…”

“…30 For this purpose, we used previously validated diagnostic criteria of executive functions deficits and a recently described statistical methodology permitting to control the effect of confounders on test performance on a large sample of patients with TBI compared with healthy individuals. 1 Briefly, the transformed scores of cognitive and behavioral measures were entered in a series of linear regression analysis with age, educational level, gender, and an interaction term (age × education level) and significant factors in healthy controls were retained. Regression coefficients computed in healthy controls were used to calculate standardized residuals, that is, z scores.…”