Feeding the machine: challenges to reproducible predictive modeling in resting-state connectomics

Abstract

Machine learning offers a promising set of prediction tools that have enjoyed more recent application in network neuroscience. Computer algorithms hold the potential to uncover hidden patterns and guide scientists and practitioners alike. In this NETN Perspectives, we examine the current application of predictive models, e.g., classifiers trained using machine learning (ML), within the clinical network neurosciences. Our primary goal is to summarize how ML is being applied and critically assess the most common practices. Our review covers 118 studies published using ML and functional MRI (fMRI) to infer various dimensions of the human functional connectome. We identify several important methodological challenges in this literature. More than half of the studies focused almost exclusively on maximizing the accuracy of classifying brain functional connectomes into one of several predetermined categories (e.g., disease versus healthy), with significantly less emphasis on offering insights into the how features of the functional connectome combine to orchestrate complex brain functions. There was also a concerning lack of transparency across many of the key steps in training and evaluating predictive models using machine learning. The summary of this literature underscores the importance of external validation (i.e., lockbox or test-set data) and highlights several methodological pitfalls that can be addressed by the imaging community. We argue that ideally studies can be motivated by the reproducibility and generalizability of the findings, and the potential clinical significance of the insights. We offer recommendations that might move the community toward a more principled integration of machine learning into clinical neuroscience with the goal of advancing imaging biomarkers of brain disorders, understanding causative determinants for health risks and parsing heterogeneous patient outcomes.