Decoding and mapping task states of the human brain via deep learning

Wang, Xiaoxiao; Liang, Xiao; Jiang, Zhoufan; Nguchu, Benedictor Alexander; Zhou, Yawen; Wang, Yanming; Wang, Huijuan; Li, Yu; Zhu, Yuying; Wu, Feng; Gao, Jia‐Hong; Qiu, Bensheng

doi:10.1002/hbm.24891

Cited by 78 publications

(105 citation statements)

References 83 publications

Supporting

Mentioning

100

Contrasting

Order By: Relevance

“…For example, the anterior tempo-451 ral lobe and temporal parietal regions are selected for the social task, which are 452 typically associated with social cognition [30,37]. Our findings also have overlaps 453 with the task decoding results in recent works [46]. cates the membership score of the region i for community j.…”

supporting

confidence: 51%

“…Also, our GNN design facilitates model inter- 43 pretability by regulating intermediate outputs with a novel loss term, which 44 provides the flexibility to choose between individual-level and group-level expla- 45 nations. 46 A preliminary version of this work explored the regularization on the pooling layer of standards GNN for fMRI analysis is provisionally accepted but have not 48 published at the 22st International Conference on Medical Image Computing 49 and Computer Assisted Intervention. This paper extends this work by designing 50 novel graph convolutional layers, justifying the loss terms regarding the pooling 51 layer, and testing our methods on additional datasets.…”

mentioning

confidence: 99%

See 1 more Smart Citation

BrainGNN: Interpretable Brain Graph Neural Network for fMRI Analysis

Zhou

Gao

et al. 2020

Preprint

View full text Add to dashboard Cite

The brain is an exceptionally complex system and understanding it's functional 2 organization is the goal of modern neuroscience. Using fMRI, large strides in 3 understanding this organization have been made by modeling the brain as a 4 graph-a mathematical construct describing the connections or interactions (i.e. 5 edges) between different discrete objects (i.e. nodes). To create these graphs, 6 nodes are defined as brain regions of interest (ROIs) and edges are defined as the 7 functional connectivity between those ROIs, computed as the pairwise correla-8 tions of functional magnetic resonance imaging (fMRI) time series, as illustrated 9 in Fig. 1. Traditional graph-based analyses for fMRI have focused on using graph 10 theoretical metrics to summarize the functional connectivity for each node into 11 a single number [45,23]. However, these methods do not consider higher-order 12 interactions between ROIs, as these interactions cannot be preserved in a sin-13 gle number. Additionally, due to the high dimensionality of fMRI data, usually 14 ROIs are clustered into highly connected communities to reduce dimensionality. 15Then, features are extracted from these smaller communities for further analysis 16 [31,12]. For these two-stage methods, if the results from the first stage are not 17 reliable, significant errors can be induced in the second stage. 18The past few years have seen the growing prevalence of the use of graph 19 neural networks (GNN) for end-to-end graph learning applications. GNNs are 20 the state-of-the-art deep learning methods for most graph-structured data anal-21 ysis problems. They combine node features, edge features, and graph structure 22 by using a neural network to embed node information and pass information 23 through edges in the graph. As such, they can be viewed as a generalization of 24 the traditional convolutional neural networks (CNN) for images. Due to their 25 high performance and interpretability, GNNs have been a widely applied graph 26 analysis method. [26,25,49,28,50]. Most existing GNNs are built on graphs that 27 do not have correspondence between the nodes of different instances, such as 28 social networks and protein networks, limiting interpretability. These methods 29-including the current GNN methods for fMRI analysis -use the same ker-30 nel over different nodes, which implicitly assumes brain graphs are translation 31 invariant. However, nodes in the same brain graph have distinct locations and 32 unique identities. Thus, applying the same kernel over all nodes is problematic. 33In addition, few GNN studies have explored both individual-level and group-level 34 explanations, which are critical in neuroimaging research. 35In this work, we propose a graph neural network-based framework for map-36 ping regional and cross-regional functional activation patterns for classification 37 tasks, such as classifying neurodisorder patients versus healthy control subjects 38 and performing cognitive task decoding. Our framework jointly learns ROI clus-39 tering and the downstr...

show abstract

supporting

confidence: 51%

mentioning

confidence: 99%

BrainGNN: Interpretable Brain Graph Neural Network for fMRI Analysis

Zhou

Gao

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…Recently, promising results on brain decoding have been shown by using deep artificial neural networks (DNNs). For instance, multiple cognitive domains can be distinguished by applying convolutional neural networks on the whole-brain hemodynamic response (Wang et al , 2019) . But the temporal dependence of hemodynamic response was interrupted by choosing random time points from the entire fMRI scan.…”

Section: Domain-general Brain Decodingmentioning

confidence: 99%

“…An alternative way is to train classifiers directly from a large set of fMRI data of a large population, for example the Human Connectome Project (HCP), that provides a detailed mapping of cognitive functions consisting of experimental conditions spanning seven cognitive domains (1 hour per subject) Van Essen et al , 2013) . Based on this powerful resource, several deep artificial neural networks (DNNs) have been recently proposed to map human cognition from recorded brain activity, for instance using the well-known convolutional (Wang et al , 2019) and recurrent neural network architectures (Li and Fan, 2019) . But these studies simplified the decoding task by either distinguishing the seven cognitive domains, or only focusing on experimental conditions from a single cognitive domain at a time.…”

Section: Introductionmentioning

confidence: 99%

Functional Annotation of Human Cognitive States using Deep Graph Convolution

Tetrel

Thirion

Bellec

2020

Preprint

View full text Add to dashboard Cite

A key goal in neuroscience is to understand brain mechanisms of cognitive functions. An emerging approach is "brain decoding", which consists of inferring a set of experimental conditions performed by a participant, using pattern classification of brain activity. Few works so far have attempted to train a brain decoding model that would generalize across many different cognitive tasks drawn from multiple cognitive domains. To tackle this problem, we proposed a domain-general brain decoder that automatically learns the spatiotemporal dynamics of brain response within a short time window using a deep learning approach. By leveraging our prior knowledge on network organization of human brain cognition, we constructed deep graph convolutional neural networks to annotate cognitive states by first mapping the task-evoked fMRI response onto a brain graph, propagating brain dynamics among interconnected brain regions and functional networks, and generating state-specific representations of recorded brain activity.We evaluated the decoding model on a large population of 1200 participants, under 21 different experimental conditions spanning 6 different cognitive domains, acquired from the Human Connectome Project task-fMRI database. Using a 10s window of fMRI response, the 21 cognitive states were identified with a test accuracy of 89% (chance level 4.8%). Performance remained good when using a 6s window (82%). It was even feasible to decode cognitive states from a single fMRI volume (720ms), with the performance following the shape of the hemodynamic response. Moreover, a saliency map analysis demonstrated that the high decoding performance was driven by the response of biologically meaningful brain regions. Together, we provide an automated tool to annotate human brain activity with fine temporal resolution and fine cognitive granularity. Our model shows potential applications as a reference model for domain adaptation, possibly making contributions in a variety of domains, including neurological and psychiatric disorders.the decoding model was constrained by the shape of the hemodynamic response. The stability of the decoding model was finally evaluated by changing the number of subjects used for model training.mixture of hemodynamic responses evoked by different task events. Early attempts have been made by adding independent regressors with delayed onsets (Nishimoto et al. , 2011) . But the simple linear model only generates a blurred image from the average prediction of each category.One possible solution to this problem is to use a multi-label decoding model based on GCN.Specifically, given a short-series of fMRI signals, the model predicts a set of cognitive states instead of one single task condition. Due to the delay effect of hemodynamic response that reaches plateau around 6s past stimulus, we can modify the label matrix by prolonging each event duration until 8s after the task onset and allow multiple labels assigned to the same time point.An interesting potential application of our work would be transfer lear...

show abstract

“…Deep learning‐based approaches require a large number (usually to the order of hundreds of thousands) of examples for effective training and parameter tuning. In order to transfer a pretrained CNN on fMRI data, a three‐dimensional CNN is required to be trained on a large set of imaging data to extract the necessary features for fMRI analysis (Hossain, Umar, Alsulaiman, & Muhammad, ; Jang, Plis, Calhoun, & Lee, ; Kamnitsas et al, ; Liu et al, ; Pinaya et al, ; Sarraf & Tofighi, ; Wang et al, ). However, using deep CNNs requires a certain level of expertise to interpret the high level features and to fine‐tune the network for the specific task of fMRI classification, and often comes with significant computational complexity (Heinsfeld, Franco, Craddock, Buchweitz, & Meneguzzi, ; Liu et al, ).…”

Section: Introductionmentioning

confidence: 99%

A heuristic information cluster search approach for precise functional brain mapping

Asadi

Wang

Olson

et al. 2020

Human Brain Mapping

View full text Add to dashboard Cite

Detection of the relevant brain regions for characterizing the distinction between cognitive conditions is one of the most sought after objectives in neuroimaging research. A popular approach for achieving this goal is the multivariate pattern analysis which is currently conducted through a number of approaches such as the popular searchlight procedure. This is due to several advantages such as being automatic and flexible with regards to size of the search region. However, these approaches suffer from a number of limitations which can lead to misidentification of truly informative regions which in turn results in imprecise information maps. These limitations mainly stem from several factors such as the fact that the information value of the search spheres are assigned to the voxel at the center of them (in case of searchlight), the requirement for manual tuning of parameters such as searchlight radius and shape, and high complexity and low interpretability in commonly used machine learning‐based approaches. Other drawbacks include overlooking the structure and interactions within the regions, and the disadvantages of using certain regularization techniques in analysis of datasets with characteristics of common functional magnetic resonance imaging data. In this article, we propose a fully data‐driven maximum relevance minimum redundancy search algorithm for detecting precise information value of the clusters within brain regions while alleviating the above‐mentioned limitations. Moreover, in order to make the proposed method faster, we propose an efficient algorithmic implementation. We evaluate and compare the proposed algorithm with the searchlight procedure as well as least absolute shrinkage and selection operator regularization‐based mapping approach using both real and synthetic datasets. The analysis results of the proposed approach demonstrate higher information detection precision and map specificity compared to the benchmark approaches.

show abstract

Decoding and mapping task states of the human brain via deep learning

Cited by 78 publications

References 83 publications

BrainGNN: Interpretable Brain Graph Neural Network for fMRI Analysis

BrainGNN: Interpretable Brain Graph Neural Network for fMRI Analysis

Functional Annotation of Human Cognitive States using Deep Graph Convolution

A heuristic information cluster search approach for precise functional brain mapping

Contact Info

Product

Resources

About