Multimodal deep neural decoding of visual object representation in humans

Abstract

Perception and categorization of objects in a visual scene are essential to grasp the surrounding situation. However, it is unclear how neural activities in spatially-distributed brain regions, especially in the aspect of temporal dynamics, represent visual objects. To address this issue, we explored spatial and temporal organization of visual object representations using concurrent functional magnetic resonance imaging (fMRI) and electroencephalography (EEG), combined with neural decoding by deep neural networks (DNNs). Visualization of the fMRI DNN revealed that visual categorization (faces or non-face objects) occurred in brain-wide cortical regions, including the ventral temporal cortex. Interestingly, the EEG DNN valued the earlier phase of neural responses for categorization and the later phase of neural responses for sub-categorization. Combination of both DNNs improved classification performance for both categorization and sub-categorization. These deep learning-based results demonstrate a categorization principle in which visual objects are represented in a spatially organized and coarse-to-fine manner.

Contribution of image statistics and semantics in local vs. distributed EEG decoding of rapid serial visual presentation

“…The interpretation of EEG decoding models could also be benefited by their application to other sensory domains, as well as to the implementation of transfer learning between modalities (Hebart, Contier et al 2023, Watanabe, Miyoshi et al 2023. In particular, the latter possibility is interesting since the low-level regularities present in images do not represent a confound for the categorization of sounds, and vice-versa; however, this task is likely more difficult than the classification of unimodal percepts, since it may require the involvement of higher order trans-modal cortical regions, potentially implicating the consolidation of conscious perception, and thus rendering the RSVP paradigm suboptimal for this purpose (Del Cul, Baillet et al 2007).…”