The interplay between audiovisual temporal synchrony and semantic congruency in the cross‐modal boost of the visual target discrimination during the attentional blink

Abstract

The visual attentional blink can be substantially reduced by delivering a task‐irrelevant sound synchronously with the second visual target (T2), and this effect is further modulated by the semantic congruency between the sound and T2. However, whether the cross‐modal benefit originates from audiovisual interactions or sound‐induced alertness remains controversial, and whether the semantic congruency effect is contingent on audiovisual temporal synchrony needs further investigation. The current study investigated these questions by recording event‐related potentials (ERPs) in a visual attentional blink task wherein a sound could either synchronize with T2, precede T2 by 200 ms, be delayed by 100 ms, or be absent, and could be either semantically congruent or incongruent with T2 when delivered. The behavioral data showed that both the cross‐modal boost of T2 discrimination and the further semantic modulation were the largest when the sound synchronized with T2. In parallel, the ERP data yielded that both the early occipital cross‐modal P195 component (192–228 ms after T2 onset) and late parietal cross‐modal N440 component (424–448 ms) were prominent only when the sound synchronized with T2, with the former being elicited solely when the sound was further semantically congruent whereas the latter occurring only when that sound was incongruent. These findings demonstrate not only that the cross‐modal boost of T2 discrimination during the attentional blink stems from early audiovisual interactions and the semantic congruency effect depends on audiovisual temporal synchrony, but also that the semantic modulation can unfold at the early stage of visual discrimination processing.

The stimulus‐driven and representation‐driven cross‐modal attentional spreading are both modulated by audiovisual temporal synchrony

“…Follow-up ttests showed that responses were significantly faster in the simultaneous VA condition than in the remaining three conditions [V + A + SIM vs. V + : t( 29 . These data indicate that presenting semantically congruent sounds simultaneously with visual targets brought the greatest information redundancy and thus resulted in a behavioral gain even when the sounds were task-irrelevant, consistent with the literature (e.g., Zhao et al, 2022;Zimmer, Itthipanyanan, et al, 2010;Zimmer, Roberts, et al, 2010). The hit rate collapsed across target types was above 95%, and no significant main effect of target type was found, although the Bayes factor did not provide substantial evidence for the null hypothesis [F(3, 87) = 2.378, p = .075, η 2 p = 0.076, BF 10 = 0.636; V + A + SIM : 95.93 ± 0.59%; V + A + D100 : 97.10 ± 0.51%; V + A + D300 : 96.11 ± 0.58%; V + : 95.93 ± 0.57%].…”

“…Follow‐up t ‐tests showed that responses were significantly faster in the simultaneous VA condition than in the remaining three conditions [V + A + SIM vs. V + : t (29) = −3.201, p FDR = 0.010, Cohen's d = −0.584, BF 10 = 11.620; V + A + SIM vs. V + A + D100 : t (29) = −2.909, p FDR = 0.014, d = −0.531, BF 10 = 6.180; V + A + SIM vs. V + A + D300 : t (29) = −3.987, p FDR = 0.002, d = −0.728, BF 10 = 72.395], whereas the remaining three conditions did not differ substantially [V + A + D100 vs. V + : t (29) = −0.325, p FDR = 0.747, d = −0.059, BF 10 = 0.204; V + A + D100 vs. V + A + D300 : t (29) = −1.442, p FDR = 0.240, d = −0.263, BF 10 = 0.494; V + A + D300 vs. V + : t (29) = 1.103, p FDR = 0.335, d = 0.201, BF 10 = 0.338]. These data indicate that presenting semantically congruent sounds simultaneously with visual targets brought the greatest information redundancy and thus resulted in a behavioral gain even when the sounds were task‐irrelevant, consistent with the literature (e.g., Zhao et al., 2022; Zimmer, Itthipanyanan, et al., 2010; Zimmer, Roberts, et al., 2010). The hit rate collapsed across target types was above 95%, and no significant main effect of target type was found, although the Bayes factor did not provide substantial evidence for the null hypothesis [ F (3, 87) = 2.378, p = .075, η 2 p = 0.076, BF 10 = 0.636; V + A + SIM : 95.93 ± 0.59%; V + A + D100 : 97.10 ± 0.51%; V + A + D300 : 96.11 ± 0.58%; V + : 95.93 ± 0.57%].…”

“…Electroencephalogram (EEG) was recorded from 64 Ag/ AgCl electrodes mounted in a custom-designed elastic cap. These electrode sites were positioned according to a modified 10-10 system montage (for details, see Zhao et al, 2022). An electrode placed at the left mastoid was used as the reference electrode (M1) during data acquisition.…”

Audiovisual Emotional Congruency Modulates the Stimulus-Driven Cross-Modal Spread of Attention

“…Electroencephalographic (EEG) signals were continuously recorded with a NeuroScan SynAmp amplifier (NeuroScan, Inc., El Paso, TX, USA) and a custom-built 64-electrode elastic cap on which the electrodes were positioned in accordance with a modified 10-10 system montage (for details, see [ 39 ]). Two additional electrodes, AFz and M1 (left mastoid), served as the ground and reference electrodes, respectively, for online EEG recording.…”

Pitch Improvement in Attentional Blink: A Study across Audiovisual Asymmetries

“…The elimination of ABs is currently an important topic in the research on attention. A growing body of research has found that an AB is improved when auditory stimulation accompanies the experiment [9][10][11][12][13][14]. However, in focusing on the auditory improvement of ABs, these studies failed to further explore the effects of auditory stimulus material representations on ABs.…”