Screening for dementia: Q^* index as a global measure of test accuracy revisited

Abstract

Receiver operating characteristic (ROC) curves intersect the downward diagonal through ROC space at a point, the Q* index, where by definition sensitivity and specificity are equal. Aside from its use in meta-analysis, Q* index has also been suggested as a possible global parameter summarising test accuracy of cognitive screening instruments and as a definition for optimal test cut-off. Area under the ROC curve (AUC ROC) is a recognised measure of test accuracy. This study compared different methods for determining Q* index (both graphical and calculation from diagnostic odds ratio) and AUC ROC (integration and calculation from diagnostic odds ratio) using the dataset of a prospective screening test accuracy study of the Mini-Addenbrooke's Cognitive Examination. The different methods did not agree. DOR-based calculations gave a very sensitive cut-off but with poorer global metrics than the graphical method. DOR-based calculations are not recommended for defining optimal test cut-off or test accuracy.

Receiver Operating Characteristic Plot and Area Under the Curve With Binary Classifiers: Pragmatic Analysis of Cognitive Screening Instruments

“…From the ROC plot constructed for MACE as a continuous scale (Figure 3), AUC calculated by DOR method was found to be greater than that by rank-sum method (Table 2, row 5), as previously shown, 26 with some consequent change in qualitative classification of AUC (in 1/3 schemata).…”

“…Results from two studies of discrete binary classifiers, examining the applause and AW signs, confirmed that AUC calculated from DOR provided more optimistic values than the usual rank-sum method, as was previously shown for MACE. 26 The validity of the simplification of AUC calculation for binary categorical data, to the equation AUC = ½. (Sens + Spec), 11 was also confirmed.…”

“…Previous analyses have established values of AUC for MACE, both by rank-sum and DOR methods. 25,26 In this study, MACE was analysed both as a continuous ordinal scale and as a binary scale by using a previously defined optimal test cut-off (defined by maximal Youden index) of ≤20/30. 25 Demographics of the three studies are shown in Table 1.…”

ROC plot and AUC with binary classifiers: pragmatic analysis of cognitive screening instruments

“…Previous analyses have established values of AUC for MACE, both by rank-sum and DOR methods. 25,26 In this study, MACE was analysed both as a continuous ordinal scale and as a binary scale by using a previously defined optimal test cut-off (defined by maximal Youden index) of ≤20/30. 25…”