Illuminating the Black Box: Interpreting Deep Neural Network Models for Psychiatric Research

Abstract

Psychiatric research is often confronted with complex abstractions and dynamics that are not readily accessible or well-defined to our perception and measurements, making data-driven methods an appealing approach. Deep neural networks (DNNs) are capable of automatically learning abstractions in the data that can be entirely novel and have demonstrated superior performance over classical machine learning models across a range of tasks and, therefore, serve as a promising tool for making new discoveries in psychiatry. A key concern for the wider application of DNNs is their reputation as a "black box" approach-i.e., they are said to lack transparency or interpretability of how input data are transformed to model outputs. In fact, several existing and emerging tools are providing improvements in interpretability. However, most reviews of interpretability for DNNs focus on theoretical and/or engineering perspectives. This article reviews approaches to DNN interpretability issues that may be relevant to their application in psychiatric research and practice. It describes a framework for understanding these methods, reviews the conceptual basis of specific methods and their potential limitations, and discusses prospects for their implementation and future directions.

Modern Views of Machine Learning for Precision Psychiatry

“…XAI has found emergent applications in medicine, finance, economy, security, and defense [328], [329]. In psychiatry, the mission of XAI is to help clarify the link between neural circuits to behavior, and to improve our understanding of therapeutic strategies to enhance cognitive, affective, and social functions [330], [331]. Notably, XAI distinguishes the standard AI in two important ways: (i) promote transparency, interpretability, and generalizability; (ii) transform classical "black box" ML models into "glass box" models, while achieving similar or improved performance.…”

“…This is also an important dimension to improve "precision" in mental health. The choice of the interpretation method [331], like model-specific (such as analyzing attention weights of a transformer), or model agnostic (such as local interpretable model-agnostic explanations (LIME)), is very specific to the nature of the problem. While various interpretation methods can be used to learn about model functioning, it is important to note that the interpretation results can only be trusted as long as the challenges of generalizability and data quality are addressed.…”

Development of Biologically Interpretable Multimodal Deep Learning Model for Cancer Prognosis Prediction

“…Furthermore, network architecture designs are largely selected based on experimental results alone and are not intuitively explainable to medical practitioners. For example, deep neural networks (DNNs), which are widely used in biomedical machine learning applications, can become a "black box," which is difficult for practitioners to understand intuitively as layers are added [26]. The potential for real-world adoption suffers as a result, as a lack of understandability erodes trust and the likelihood of adoption for machine learning tools [27].…”

Intelligent Palliative Care Based on Patient-Reported Outcome Measures

“…The DL model achieved ROC AUC figures of 0.978, 0.956 and 0.943, respectively. Topics extracted from health notes identified possible predictive topics such as cognitive function and laboratory testing in the absence of interpretability of DL models, which are considered 'black box' because of a lack of transparency and interpretability concerning how input data are transformed to output (15). More interpretable ML decision trees (gradient-boosted trees) used electronic health records for chemotherapy patients, including palliative care patients, to predict 30-day mortality after start of a new regimen.…”