MACE for Diagnosis of Dementia and MCI: Examining Cut-Offs and Predictive Values

Abstract

The definition of test cut-offs is a critical determinant of many paired and unitary measures of diagnostic or screening test accuracy, such as sensitivity and specificity, positive and negative predictive values, and correct classification accuracy. Revision of test cut-offs from those defined in index studies is frowned upon as a potential source of bias, seemingly accepting any biases present in the index study, for example related to sample bias. Data from a large pragmatic test accuracy study examining the Mini-Addenbrooke’s Cognitive Examination (MACE) were interrogated to determine optimal test cut-offs for the diagnosis of dementia and mild cognitive impairment (MCI) using either the maximal Youden index or the maximal correct classification accuracy. Receiver operating characteristic (ROC) and precision recall (PR) curves for dementia and MCI were also plotted, and MACE predictive values across a range of disease prevalences were calculated. Optimal cut-offs were found to be a point lower than those defined in the index study. MACE had good metrics for the area under the ROC curve and for the effect size (Cohen’s d) for both dementia and MCI diagnosis, but PR curves suggested the superiority for MCI diagnosis. MACE had high negative predictive value at all prevalences, suggesting that a MACE test score above either cut-off excludes dementia and MCI in any setting.

Screening for dementia: Q* index as a global measure of test accuracy revisited

“…AUC by integration was 0.886 (95% confidence interval 0.856-0.916). 11 Calculated AUC showed a maximum of 0.941 at the optimal cut-off of ≤23/30 ( Table 1, 4 th column). The calculated AUC most closely corresponding to the AUC by integration was between the cut-offs of ≤19/30 and ≤20/30, the optima defined by d and Y respectively.…”

“…13 Q* index was measured graphically, by reading off the point of intersection of the ROC plot with the anti-diagonal. 7,11 Q* index was also calculated at each MACE cut-off, encompassing the range of MACE cut-offs defined in the index study, 12…”

New unitary metrics for dementia test accuracy studies

“…SUI might be given by some kind of summation, but clearly not by a ratio as this would give the relative of value of the test for ruling in or ruling out a diagnosis. Exploring different methods, 18,19 SUI was defined as the sum of (CUI+ + CUI-), as for the 'gain in certainty' unitary measure derived from sensitivity and specificity. 20 Hence SUI may range from 0 to 2, and is desirably as close to 2 as possible.…”

Applying Kraemer’s Q (Positive Sign Rate): Some Implications for Diagnostic Test Accuracy Study Results

“…The purpose of this study was to examine QSN and QSP in comparison to Sens and Spec in a large dataset from a screening test accuracy study, and also to examine other metrics based on Sens and Spec, both standard paired (likelihood ratios, clinical utility indexes) and unitary (Youden index, diagnostic odds ratio, number needed to diagnose) measures [7], and also some recently described unitary metrics (likelihood to diagnose or misdiagnose [LDM], the summary utility index [SUI] and the number needed for screening utility [NNSU] [8,9]).…”