Methodological and Quality Flaws in the Use of Artificial Intelligence in Mental Health Research: Systematic Review

Abstract

Background Artificial intelligence (AI) is giving rise to a revolution in medicine and health care. Mental health conditions are highly prevalent in many countries, and the COVID-19 pandemic has increased the risk of further erosion of the mental well-being in the population. Therefore, it is relevant to assess the current status of the application of AI toward mental health research to inform about trends, gaps, opportunities, and challenges. Objective This study aims to perform a systematic overview of AI applications in mental health in terms of methodologies, data, outcomes, performance, and quality. Methods A systematic search in PubMed, Scopus, IEEE Xplore, and Cochrane databases was conducted to collect records of use cases of AI for mental health disorder studies from January 2016 to November 2021. Records were screened for eligibility if they were a practical implementation of AI in clinical trials involving mental health conditions. Records of AI study cases were evaluated and categorized by the International Classification of Diseases 11th Revision (ICD-11). Data related to trial settings, collection methodology, features, outcomes, and model development and evaluation were extracted following the CHARMS (Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies) guideline. Further, evaluation of risk of bias is provided. Results A total of 429 nonduplicated records were retrieved from the databases and 129 were included for a full assessment—18 of which were manually added. The distribution of AI applications in mental health was found unbalanced between ICD-11 mental health categories. Predominant categories were Depressive disorders (n=70) and Schizophrenia or other primary psychotic disorders (n=26). Most interventions were based on randomized controlled trials (n=62), followed by prospective cohorts (n=24) among observational studies. AI was typically applied to evaluate quality of treatments (n=44) or stratify patients into subgroups and clusters (n=31). Models usually applied a combination of questionnaires and scales to assess symptom severity using electronic health records (n=49) as well as medical images (n=33). Quality assessment revealed important flaws in the process of AI application and data preprocessing pipelines. One-third of the studies (n=56) did not report any preprocessing or data preparation. One-fifth of the models were developed by comparing several methods (n=35) without assessing their suitability in advance and a small proportion reported external validation (n=21). Only 1 paper reported a second assessment of a previous AI model. Risk of bias and transparent reporting yielded low scores due to a poor reporting of the strategy for adjusting hyperparameters, coefficients, and the explainability of the models. International collaboration was anecdotal (n=17) and data and developed models mostly remained private (n=126). Conclusions These significant shortcomings, alongside the lack of information to ensure reproducibility and transparency, are indicative of the challenges that AI in mental health needs to face before contributing to a solid base for knowledge generation and for being a support tool in mental health management.

Asynchronous Technologies in Mental Health Care and Education

“…We discuss recent use cases of asynchronous technologies from both educational and clinical perspectives, which are relevant to shaping the current and future standards of mental health care and training. We hope that this inspires the ongoing exploration of safe and practical implementation [ 11 ] of technologies in psychiatric care.…”

Fuzzy Cognitive Maps for Algorithmic Decision Making in eMental Health

“…This makes FCMs a viable modelling approach for situations in which no training data is available, which is a common challenge in mental health research. Indeed, as recently investigated in a review of the methodological and quality flaws in AI research for mental health [6], current research efforts mostly use private data, aiming to develop new AI models rather than validating existing models using public data sets. The availability of public mental health data sets is likely cause and effect in maintaining a preference for private data in AI research for mental health.…”

“…A second advantage (2) of FCMs, in their traditional form, is their explainability [25]. The black-box nature of conventional and especially, deep AI models has been linked to clinicians' reluctance to trust the predictions of AI models and consequently, their reluctance to adopt them in clinical practice [25,45,6].…”

“…However, AI-based DMHS introduce new challenges of their own, including the availability of high quality clinical data sets to train models, model transparency, and the smooth integration of AI tools into clinicians' workflows, which in turn affect clinicians' trust in AI technology and their willingness to use it in practice [3]. The most prominent purposes of AI tools in mental health are quality assessments of mental health treatments [6] and the detection of symptoms and risks of mental illness, and subsequent mental disorder diagnosis [7,6]. All in all, the current state of AI in mental health is still considered to be in the phase of proof-of-concepts and showcasing technical feasibility rather than providing systems that are ready to create clinical impact [7].…”

The EPIDIA4Kids protocol for a digital epidemiology study on brain functioning in children, based on a multimodality biometry tool running on an unmodified tablet

“…This multimodality approach ( 73 ) reflects the recent advances in both quantity and quality of data available since Information & Communication Technology (ICT) and Internet of Things (IoT) brings high potential for improved accuracy at cost-efficiency ( 24 , 28 ). Machine learning and AI applied to NDDs may open valuable route to examine heterogenous symptoms in pediatric population and at individual level, by integrating multimodality dimensions in prediction models, such as social, environmental, and structural determinants ( 74–76 ). Furthermore, the models can handle for missing data and larger numbers of interactive predictors.…”