A deep neural network model of the primate superior colliculus for emotion recognition

Abstract

Although sensory processing is pivotal to nearly every theory of emotion, the evaluation of the visual input as ‘emotional’ (e.g. a smile as signalling happiness) has been traditionally assumed to take place in supramodal ‘limbic’ brain regions. Accordingly, subcortical structures of ancient evolutionary origin that receive direct input from the retina, such as the superior colliculus (SC), are traditionally conceptualized as passive relay centres. However, mounting evidence suggests that the SC is endowed with the necessary infrastructure and computational capabilities for the innate recognition and initial categorization of emotionally salient features from retinal information. Here, we built a neurobiologically inspired convolutional deep neural network (DNN) model that approximates physiological, anatomical and connectional properties of the retino-collicular circuit. This enabled us to characterize and isolate the initial computations and discriminations that the DNN model of the SC can perform on facial expressions, based uniquely on the information it directly receives from the virtual retina. Trained to discriminate facial expressions of basic emotions, our model matches human error patterns and above chance, yet suboptimal, classification accuracy analogous to that reported in patients with V1 damage, who rely on retino-collicular pathways for non-conscious vision of emotional attributes. When presented with gratings of different spatial frequencies and orientations never ‘seen’ before, the SC model exhibits spontaneous tuning to low spatial frequencies and reduced orientation discrimination, as can be expected from the prevalence of the magnocellular (M) over parvocellular (P) projections. Likewise, face manipulation that biases processing towards the M or P pathway affects expression recognition in the SC model accordingly, an effect that dovetails with variations of activity in the human SC purposely measured with ultra-high field functional magnetic resonance imaging. Lastly, the DNN generates saliency maps and extracts visual features, demonstrating that certain face parts, like the mouth or the eyes, provide higher discriminative information than other parts as a function of emotional expressions like happiness and sadness. The present findings support the contention that the SC possesses the necessary infrastructure to analyse the visual features that define facial emotional stimuli also without additional processing stages in the visual cortex or in ‘limbic’ areas. This article is part of the theme issue ‘Cracking the laugh code: laughter through the lens of biology, psychology and neuroscience’.

Analysis of convolutional neural networks reveals the computational properties essential for subcortical processing of facial expression

“…Note that we estimated this value on an assumption of the image size of 30.5° based on our DoG parameters, the filter resolution, and the RF size of superior colliculus neurons representing the foveal region. The range of DoG-filter width corresponds to that applied in the models for the superior colliculus in a recent simulation study (Méndez et al, 2022).…”

“…The present study provides the first computational model for facial expression processing along the subcortical pathway (see Méndez et al, 2022 for a model of face processing in the superior colliculus). Despite the celebrated success of DNNs in modeling visual processing in the ventral cortical pathway, it has remained unclear whether and how the CNN architecture can be adapted to processing in the subcortical pathway.…”

“…Similarities between neural networks regarding expression-specific performance may vary according to these differences. The relatively poor ability to distinguish between sad and neutral faces was also observed in another CNN with the first layer of DoG filtering and average pooling (Méndez et al, 2022). This CNN was constructed to simulate facial processing in the superior colliculus, and was trained to discriminate three facial expressions: happy, sad, and neutral.…”

Different Biometrical Features for Detecting Human Intrusion Using Artificial Intelligence: Literature Review

“…While the researcher Thai Hoang Le proposed a hybrid intelligent method that combines two algorithms (AdaBoost, Artificial Neural Network) to distinguish faces after determining the geometric features of the face alignment [23]. While other researchers used the method of determining other geometric features of the human face, such as the locations of the eyebrows, eyes, the location of the nose, mouth, ears, and others, with the intelligent algorithm (Radial Basis Function (RBF)), which is one of the types of neural networks [25][63] [65]. The researchers (Kolhandai Yesu and others) also relied on the geometric measurements of the face, eyes, nose and mouth as inputs to the intelligent feed-forward neural network [32].…”

Cracking the laugh code: laughter through the lens of biology, psychology and neuroscience

“…[ 12 ]), and computational (Celeghin et al . [ 13 ]), clinical (Sessa et al . [ 14 ]) and system neuroscience (Zauli [ 15 ], Sun [ 16 ]).…”