Motivated attention and task relevance in the processing of cross-modally associated faces: Behavioral and electrophysiological evidence

Abstract

It has repeatedly been shown that visually presented stimuli can gain additional relevance by their association with affective stimuli. Studies have shown effects of associated affect in event-related potentials (ERP) like the early posterior negativity (EPN), late positive complex (LPC), and even earlier components as the P1 or N170. However, findings are mixed as to the extent associated affect requires directed attention to the emotional quality of a stimulus and which ERP components are sensitive to task instructions during retrieval. In this preregistered study (https://osf.io/ts4pb), we tested cross-modal associations of vocal affect-bursts (positive, negative, neutral) to faces displaying neutral expressions in a flash-card-like learning task, in which participants studied face-voice pairs and learned to correctly assign them to each other. In the subsequent EEG test session, we applied both an implicit (“old-new”) and explicit (“valence-classification”) task to investigate whether the behavior at retrieval and neurophysiological activation of the affect-based associations were dependent on the type of motivated attention. We collected behavioral and neurophysiological data from 40 participants who reached the preregistered learning criterium. Results showed EPN effects of associated negative valence after learning and independent of the task. In contrast, modulations of later stages (LPC) by positive and negative associated valence were restricted to the explicit, i.e., valence-classification, task. These findings highlight the importance of the task at different processing stages and show that cross-modal affect can successfully be associated to faces.

Additive effects of emotional expression and stimulus size on the perception of genuine and artificial facial expressions: an ERP study

“…Finally, although the stimuli were homogenized in some of their low-level features, physical variations in emotional expressions were inherent to the stimuli, as identical stimuli cannot express different emotions. However, faces have been shown to elicit emotion-related effects when associated with emotional expressions of the face and voice 44 , 80 , 81 . To achieve full control of physical stimulus features across valence categories, future studies could use faces with neutral expressions but associated with emotional relevance and presented at different sizes.…”

“…The common mode sense (CMS) active electrode and the driven right leg (DLR) passive electrode were used as reference electrodes. Based on previous studies in the lab 35,44 and consistent with typical ERP studies on emotional face processing 4 , we determined the time windows and electrode regions (ROIs) for the ERP components of interest, which were confirmed by www.nature.com/scientificreports/ pilot data (N = 4) that were not included in the analysis. The following preregistered ERPs were extracted for the visual (face-locked) components: P1: Peak amplitudes and peak latency, 80-120 ms; occipital electrode cluster: A8, A9, A10, A15 (O1), A16, A17, A28 (O2), A29, A30, B5, B6, B7; N170: Peak amplitudes and peak latency, 130-200 ms; occipitotemporal electrode cluster: D32 (P9), A10 (PO7), A11, A12, B10 (P10), B7 (PO8), B8, B9; EPN: mean amplitudes, 250-300 ms; occipito-temporal cluster: A10 (PO7), A11, A12, A14, A15 (O1), D32 (P9), A24, A25, A26, A27, A28 (O2), B7 (PO8), B8, B9, B10(P10); LPC: mean amplitudes, 400-600 ms; occipitoparietal electrode cluster: A4, A19 (Pz), A20, A21 (POz), A5, A32, A18, A31, A17 (PO3), A30 (PO4).…”

“…Emotion-related effects on the EPN, the early posterior negativity, have been reported not only for faces but also for other stimulus domains 41 , 42 , which is why it is generally assumed to reflect selective attention to hedonic and arousing stimuli 43 . While the EPN has been suggested to reflect rather automatic processing, emotion effects on the late positive complex/potential (LPC/LPP) have been reported more robustly for tasks requiring attention to the affective content of stimuli 4 and are known to be sensitive to task demands 6 , 44 . To encourage the processing of facial expressions without explicitly deciding on their emotional valence, in the present study, we implemented a task in which participants had to judge the naturalness of facial expressions.…”

Gender congruence and emotion effects in cross‐modal associative learning: Insights from ERPs and pupillary responses

“…Cross-modal associations have also been demonstrated in fear-conditioning studies (Miskovic & Keil, 2012), with some of them using aversive screams (Bruchmann et al, 2021;Glenn et al, 2012;Schindler et al, 2022). Effects of cross-modally associated valence through affect bursts have been shown for a valence-implicit and valence-explicit retrieval task, although participants were not instructed to attend to a specific stimulus feature during learning (Ziereis & Schacht, 2023), indicating that also less intense vocal stimuli can be successfully associated to faces.…”

“…Typical ERP modulations for classical or evaluative conditioning and instrumental learning paradigms were reported at latencies from 100 ms on, during short-(P1, N170), mid-(EPN), and long-(LPC) latencies. Enhanced amplitudes for faces with associated relevance have been reported for the P1 (monetary reward: Hammerschmidt et al, 2017; facial expressions of emotion: Aguado et al, 2012), N170 (fear-conditioning: Camfield et al, 2016;Schellhaas et al, 2020;Sperl et al, 2021;aversive screams: Bruchmann et al, 2021;person knowledge: Luo et al, 2016;Schindler et al, 2021; facial expressions of emotion: Aguado et al, 2012), EPN (vocal emotional expressions: Ziereis & Schacht, 2023;fear-conditioning: Bruchmann et al, 2021;de Sá et al, 2018;Schellhaas et al, 2020;person knowledge: Abdel Rahman, 2011;Luo et al, 2016;Suess et al, 2014;Xu et al, 2016;affective communication: Wieser, Gerdes, et al, 2014), and the LPC (fear-conditioning: de Sá et al, 2018;Panitz et al, 2015;Rehbein et al, 2018;Sperl et al, 2021;Wiemer et al, 2021;aversive screams: Bruchmann et al, 2021;monetary reward: Hammerschmidt, Kulke, et al, 2018;person knowledge: Abdel Rahman, 2011;Baum et al, 2020;Kissler & Strehlow, 2017;Schindler et al, 2021;Xu et al, 2016). Despite this evidence, there is still uncertainty about the boundary conditions under which associated effects occur, for example, regarding the need for explicit awareness of a CS-US contingency for stable associations (Mertens & Engelhard, 2020).…”