Spatiotemporal cortical dynamics for rapid scene recognition as revealed by EEG decoding

Abstract

The human visual system rapidly recognizes the categories and global properties of complex natural scenes. The present study investigated the spatiotemporal dynamics of neural signals involved in ultra-rapid scene recognition using electroencephalography (EEG) decoding. We recorded visual evoked potentials from 11 human observers for 232 natural scenes, each of which belonged to one of 13 natural scene categories (e.g., a bedroom or open country) and had three global properties (naturalness, openness, and roughness). We trained a deep convolutional classification model of the natural scene categories and global properties using EEGNet. Having confirmed that the model successfully classified natural scene categories and the three global properties, we applied Grad-CAM to the EEGNet model to visualize the EEG channels and time points that contributed to the classification. The analysis showed that EEG signals in the occipital lobes at short latencies (~88 ms) contributed to the classifications other than roughness, whereas those in the frontal lobes at relatively long latencies (~250 ms) contributed to the classification of naturalness and the individual scene category. These results suggest that different global properties are encoded in different cortical areas and with different timings, and that the encoding of scene categories shifts from the occipital to the frontal lobe over time.

Neural basis of perceptual surface qualities: Evidence from EEG decoding

“…4, that style information characterizing surface perception strongly correlated with the VEPs within 200 ms after the stimulus onset, and that the visual impression of surface images were successfully reconstructed only by the VEPs via the MVAE models. These results are consistent with the fact that the global information processed in a single glance contributes to the rough perception of scenes and objects as well as natural surfaces (Potter, 1976; Thorpe et al 2001; Greene & Oliva, 2009; Greene & Hansen, 2020; Jagadeesh & Gardner, 2022; Orima & Motoyoshi, 2023).…”

“…The MVAE models developed in the present study reconstructed very high-quality visual stimuli from simple VEPs alone. However, the realistic images such as reconstructed images obtained in the present study were not generated by the VEPs for natural scene images (obtained in Orima & Motoyoshi, 2023). Previous studies have found that the perception of texture, including natural surface images used in the present study, is largely based on low-level statistical features such as image statistics (Bergen & Adelson, 1988; Heeger & Bergen, 1995; Zipser et al 1996; Portilla & Simoncelli, 2000; Baker & Mareschal, 2001; Motoyoshi et al 2007; Freeman & Simoncelli, 2011; Freeman et al 2013; Ziemba et al, 2019; Orima & Motoyoshi, 2021), but natural scenes have many aspects that cannot be explained by low-level features alone (Groen et al, 2013; Epstein & Baker, 2019; Greene & Hansen, 2020).…”

“…The observer’s eye movements were controlled by the pre-experiment instruction (cf. Orima & Motoyoshi, 2021, 2023). This session was repeated 5 times for each of 5 variants, 25 times in total for each observer.…”

“…Recent study has developed a model called EEGNet, which yields high accuracy in EEG-based classification (Lawhern et al, 2018). In the other previous study, the EEGNet model is used to visualize spatiotemporal cortical mapping for natural scene perception (Orima & Motoyoshi, 2023). By using a two-dimensional network like EEGNet model instead of a one-dimensional convolutional network, it may be possible to reconstruct the original image more realistically and to visualize the contribution of VEPs to the visual impression in a spatiotemporal manner.…”