Cognitive screening instruments for dementia: comparing metrics of test limitation

Abstract

Cognitive screening instruments (CSIs) for dementia and mild cognitive impairment are usually characterised in terms of measures of discrimination such as sensitivity, specificity, and likelihood ratios. However, CSIs also have limitations. Several metrics exist which may be used to denote test limitations but they are seldom examined. Data from several pragmatic test accuracy studies of CSIs were interrogated to calculate various measures of limitation, namely: misclassification rate; net harm to net benefit ratio; and the likelihood to be diagnosed or misdiagnosed. Intra- and inter-test performance for measures of discrimination and limitation were compared. The study found that some tests with very high sensitivity but low specificity for dementia fared poorly on measures of limitation, with high misclassification rates, low net harm to net benefit ratios, and low likelihoods to be diagnosed or misdiagnosed; some had likelihoods favouring misdiagnosis over diagnosis. Tests with a better balance of sensitivity and specificity fared better on measures of limitation. When choosing which CSIs to administer, measures of test limitation should be considered as well as measures of test discrimination. Although high test sensitivity may be desirable to avoid false negatives, false positives also have a cost. Identification of tests having high misclassification rate, low net harm to net benefit ratio, and low likelihood to be diagnosed or misdiagnosed, may have implications for their use in clinical practice.

Communicating Risk: Developing an “Efficiency Index” for Dementia Screening Tests

“…EI may also be compared to other unitary measures, including the critical success index, F measure, area under the receiver operating characteristic curve (AUC ROC), and Matthews’ correlation coefficient (MCC) [ 4 , 33 , 35 ]. Critical success index and F measure ignore TN values, unlike EI.…”

Communicating risk to patients: developing an “Efficiency Index” for diagnostic and screening tests

“…28 Other unitary measures which have been used to summarise diagnostic test performance, and with which EI should be compared, include the Youden index (Y), diagnostic odds ratio (DOR), critical success index, F measure, area under the receiver operating curve (AUC ROC), and Matthews' correlation coefficient (MCC). 4,31,32 Being based on Acc and Inacc, EI treats FN and FP as equally undesirable, as is also the case for Y and DOR, an assumption which is often not the case in clinical practice where FN may be considered more costly. As Acc is calculated using values from both columns of the 2x2 contingency table, EI is dependent on disease prevalence.…”