Transfer learning of deep neural network representations for fMRI decoding

Abstract

BackgroundDeep neural networks have revolutionised machine learning, with unparalleled performance in object classification. However, in brain imaging (e.g. fMRI), the direct application of Convolutional Neural Networks (CNN) to decoding subject states or perception from imaging data seems impractical given the scarcity of available data. New methodIn this work we propose a robust method to transfer information from deep learning (DL) features to brain fMRI data with the goal of decoding. By adopting Reduced Rank Regression with Ridge Regularisation we establish a multivariate link between imaging data and the fully connected layer (fc7) of a CNN. We exploit the reconstructed fc7 features by performing an object image classification task on two datasets: one of the largest fMRI databases, taken from different scanners from more than two hundred subjects watching different movie clips, and another with fMRI data taken while watching static images, ResultsThe fc7 features could be significantly reconstructed from the imaging data, and led to significant decoding performance. Comparison with existing methodsThe decoding based on reconstructed fc7 outperformed the decoding based on imaging data alone. ConclusionIn this work we show how to improve fMRI-based decoding benefiting from the mapping between functional data and CNN features. The potential advantage of the proposed method is twofold: the extraction of stimuli representations by means of an automatic procedure (unsupervised) and the embedding of highdimensional neuroimaging data onto a space designed for visual object discrimination, leading to a more manageable space from dimensionality point of view.A long-standing goal of cognitive neuroscience is to unravel the brain mechanisms associated with sensory perception. Cognitive neuroscientists often conduct empirical research using non-invasive imaging techniques, among which functional Magnetic Resonance Imaging (fMRI) or Electroencephalography (EEG), to validate computational theories and models by relating sensory experiences, like watching images and videos, to the observed brain activity. Establishing such relationship is not trivial, due to our partial understanding of the neural mechanisms involved, the limited view offered by current imaging techniques, and the high dimensions in both imaging and sensorial spaces.A large amount of statistical approaches have been proposed in the literature to accomplish this task; in particular, in the last two decades great attention has been given to generative (also referred to as encoding) and discriminative (decoding) models, that have different aims, strengths and limitations (see [1]). Encoding models aim at characterising single units response harnessing the richness of the stimulus representation in a suitable space, and can thus be used to model the brain response to new stimuli, provided that a suitable decomposition is available. On the other hand, decoding models solve a "simpler" problem of discriminating between specific stimulus types and are better sui...

Transfer Learning in Magnetic Resonance Brain Imaging: A Systematic Review

“…The second most popular strategy to apply transfer learning was fine-tuning certain parameters in a pretrained CNN [ 34 , 127 , 128 , 129 , 130 , 131 , 132 , 133 , 134 , 135 , 136 , 137 , 138 , 139 , 140 , 141 , 142 , 143 , 144 , 145 , 146 ]. The remaining approaches first optimized a feature extractor (typically a CNN or a SVM), and then trained a separated model (SVMs [ 30 , 45 , 147 , 148 , 149 ], long short-term memory networks [ 150 , 151 ], clustering methods [ 148 , 152 ], random forests [ 70 , 153 ], multilayer perceptrons [ 154 ], logistic regression [ 148 ], elastic net [ 155 ], CNNs [ 156 ]). Additionally, Yang et al [ 157 ] ensembled CNNs and fine-tuned their individual contribution.…”

Attention module improves both performance and interpretability of four‐dimensional functional magnetic resonance imaging decoding neural network

“…Most fMRI experiments comprise tens to hundreds of participants due to experimental costs or participant selection. It is natural to use transfer learning to alleviate the data scarcity problem in the target domain (e.g., small sample datasets) by utilizing the knowledge acquired in the source domain (e.g., large cohorts; Gao, Zhang, Wang, Guo, & Zhang, 2019; Svanera et al, 2019; Thomas, Müller, & Samek, 2019; X. Wang et al, 2020). The fMRI data vary across datasets (e.g., scanner, scanning parameters, task design, template space), so it remains an open question how far the DNN can transfer‐learn in fMRI.…”

A novel and simple machine learning algorithm for preoperative diagnosis of acute appendicitis in children

“…For diagnosis and classification purposes, simple conceptual decision-making models that can learn are widely used. [15][16][17][18] The decision tree is a reliable and effective decision-making technique which in general provides high classification accuracy; moreover, interpretation and implementation of these models are quite easy compared to most classification algorithms.…”