Decision curve analysis for assessing the usefulness of tests for making decisions to treat: An application to tests for prodromal psychosis.

Abstract

For a test to be considered useful for making treatment decisions, it is necessary that making treatment decisions based on the results of the test be a preferable strategy to making treatment decisions without the test. Decision curve analysis is a framework for assessing when a test would be expected to be useful, which integrates evidence of a test's performance characteristics (sensitivity and specificity), condition prevalence among at-risk patients, and patient preferences for treatment. We describe decision curve analysis generally and illustrate its potential through an application to tests for prodromal psychosis. Clinical psychosis is often preceded by a prodromal phase, but not all those with prodromal symptoms proceed to develop full psychosis. Patients identified as at risk for developing psychosis may be considered for proactive treatment to mitigate development of clinically defined psychosis. Tests exist to help identify those at-risk patients most likely to develop psychosis, but it is uncertain when these tests would be considered useful for making proactive treatment decisions. We apply decision curve analysis to results from a systematic review of studies investigating clinical tests for predicting the development of psychosis in at-risk populations, and present resulting decision curves that illustrate when the tests may be expected to be useful for making proactive treatment decisions.

A Primer on Receiver Operating Characteristic Analysis and Diagnostic Efficiency Statistics for Pediatric Psychology: We Are Ready to ROC

“…Neither of these strategies is useful in most clinical applications (cf. Kraemer, 1992;Pulleyblank, Chuma, Gilbody, & Thompson, 2013;Youngstrom, 2013). Ideally, there would be a cut score or threshold on the predictor that would separate those with the diagnosis from those without it.…”

“…These are probably appropriate for engineering and some biomedical applications, but in the context of mental health diagnoses, they are less representative. The AUC is constrained by the reliability and validity of the reference standard: If the criterion diagnosis is imperfect, then it is impossible for the AUC to reach 1.00 (Kraemer, 1992;Pepe, 2003). In practice, many of the best-performing behavior checklists and inventories currently available deliver AUC estimates in the 0.7-0.8 range under clinically realistic conditions and with valid reference standard diagnoses.…”

“…If the goal is to use an index test as a screener, then high sensitivity is more important than specificity, because the goal is to avoid missing cases that truly have the target diagnosis; and conversely, applications using the index test as diagnostic confirmation would put more of a premium on specificity (Kraemer, 1992). The base rate directly affects the overall accuracy of classifications, as well as the positive and negative predictive powers of the test, whereas sensitivity and specificity are algebraically unrelated to base rate (Pepe, 2003) (see Glossary as well). Positive predictive power describes the percentage of cases testing positive that actually have the diagnosis, and negative predictive power is the accuracy rate of negative test results.…”

“…Without all of the cost and benefit utilities available, there are three pragmatic approaches: (1) pick a desired sensitivity, and evaluate the rest of the test performance around that, or conversely start with an desired specificity and work from there (Pepe, 2003); (2) select a threshold based on the maximum kappa, recognizing that the kappa estimate itself is tied to the base rate, and will not generalize to settings with different rates (Kraemer, 1992); or (3) divide the index test score into multiple ranges, and estimate the diagnostic efficiency separately for each range (Straus et al, 2011). Because the raw Internalizing score performed significantly better than the Affective Problems scale and the T-score, we use it to illustrate different ways of evaluating the cut scores.…”

Development and validation of a prognostic nomogram to predict recurrence in high-risk gastrointestinal stromal tumour: A retrospective analysis of two independent cohorts

“…The study applied the decision curve analysis (DCA) for evaluating the clinical utility of the nomogram based on net benefits at different threshold probabilities in the training and validation datasets. 16 Additionally, we analysed the potency of the nomogram to stratify patients with high-risk GIST.…”