Evidence-Based Assessment From Simple Clinical Judgments to Statistical Learning: Evaluating a Range of Options Using Pediatric Bipolar Disorder as a Diagnostic Challenge

Abstract

Reliability of clinical diagnoses is often low. There are many algorithms that could improve diagnostic accuracy, and statistical learning is becoming popular. Using pediatric bipolar disorder as a clinically challenging example, we evaluated a series of increasingly complex models ranging from simple screening to a supervised LASSO regression in a large (=550) academic clinic sample. We then externally validated models in a community clinic (=511) with the same candidate predictors and semi-structured interview diagnoses, providing high methodological consistency; the clinics also had substantially different demography and referral patterns. Models performed well according to internal validation metrics. Complex models degraded rapidly when externally validated. Naïve Bayesian and logistic models concentrating on predictors identified in prior meta-analyses tied or bettered LASSO models when externally validated. Implementing these methods would improve clinical diagnostic performance. Statistical learning research should continue to invest in high quality indicators and diagnoses to supervise model training.

Integrating Multiple Informants’ Reports: How Conceptual and Measurement Models May Address Long-Standing Problems in Clinical Decision-Making

“…Given this leaning to remove reports and the findings reported in this article, we suspect that one’s hypothesis about the meaning underlying informant discrepancies (e.g., they reflect measurement error, one informant provides “better” reports) might result in idiosyncratic decision-making for important clinical tasks such as diagnosis and treatment selection. A large body of research demonstrates that these idiosyncrasies not only commonly occur in clinical work but also that statistical- or actuarial-based decision-making results in relative improvements in accuracy and effectiveness (Grove, Zald, Lebow, Snitz, & Nelson, 2000; Lilienfeld & Lynn, 2014; Meehl, 1954; Youngstrom, Halverson, Youngstrom, Lindheim, & Findling, 2018). Future research is needed to understand how the trait-score approach can move clinical decision-making when using multi-informant reports from an idiosyncratic, dismissive stance on specific information sources to an evidence-based, inclusive stance on multiple information sources.…”

Real-Time Monitoring of Suicide Risk among Adolescents: Potential Barriers, Possible Solutions, and Future Directions

“…The first problem is overfitting, which occurs when models prioritize prediction of already-known cases in training data rather than generalizability to future data. This is a problem well known within statistics and computer science (Dietterich, 1995) and addressed in some areas of psychology (e.g., screening for pediatric bipolar disorder; Youngstrom, Halverson, Youngstrom, Lindhiem, & Findling, 2018) but possibly less frequently acknowledged within suicide research. Overfitting is especially problematic for creating algorithms to predict suicide risk because of the costs associated with a false positive (e.g., someone may be given an intervention for suicide risk despite not needing one) and especially high cost associated with a false negative (e.g., someone at risk may die by suicide; Gianfrancesco, Tamang, Yazdany, & Schmajuk, 2018).…”

Is the Finding Too Good to Be True? Moving from “More Is Better” to Thinking in Terms of Simple Predictions and Credibility

“…The more clinically complex the setting, and the more different the demographics, the more that we should expect the effect size to shrink (Konig, Malley, Weimar, Diener, & Ziegler, 2007). Internal cross-validation is not a substitute for finding data that closely resemble where we will need to use the measure (Youngstrom, Halverson, Youngstrom, Lindhiem, & Findling, 2018).…”