Semantic and spatial congruency mould audiovisual integration depending on perceptual awareness

Abstract

Information integration is considered a hallmark of human consciousness. Recent research has challenged this tenet by showing multisensory interactions in the absence of awareness. This psychophysics study assessed the impact of spatial and semantic correspondences on audiovisual binding in the presence and absence of visual awareness by combining forward–backward masking with spatial ventriloquism. Observers were presented with object pictures and synchronous sounds that were spatially and/or semantically congruent or incongruent. On each trial observers located the sound, identified the picture and rated the picture’s visibility. We observed a robust ventriloquist effect for subjectively visible and invisible pictures indicating that pictures that evade our perceptual awareness influence where we perceive sounds. Critically, semantic congruency enhanced these visual biases on perceived sound location only when the picture entered observers’ awareness. Our results demonstrate that crossmodal influences operating from vision to audition and vice versa are interactively controlled by spatial and semantic congruency in the presence of awareness. However, when visual processing is disrupted by masking procedures audiovisual interactions no longer depend on semantic correspondences.

Evaluating the Effect of Semantic Congruency and Valence on Multisensory Integration

“…Additionally, there was an interaction effect between the semantic and spatial congruences where the effect of the semantic congruence on the audiovisual binding was increased when the sound was presented bilaterally (Delong & Noppeney, 2021). In experiment 2, they found a significant positive effect of semantically congruent stimuli on picture identification (in line with previous studies such as Chen and Spence (2010)).…”

“…In experiment 2, they found a significant positive effect of semantically congruent stimuli on picture identification (in line with previous studies such as Chen and Spence (2010)). Experiment 2 also found a significant interaction effect of semantic and spatial congruences (as in experiment 1) but only for the visible conditions (conditions where the participant identified that they saw and recognized the image) as opposed to the invisible conditions (where the participants reported that they could not identify the image that was presented) (Delong & Noppeney, 2021). Ultimately, this study further reinforces the impact that semantic congruency has on our multisensory integration and demonstrates that it is related to other factors such as spatial congruency.…”

“…In both experiments, participants were asked to identify where the sound came from (left or right) and in experiment 2 they were additionally asked to identify which image(s) they saw. In experiment 1, they found that semantically congruent stimuli significantly improved audiovisual binding and increased the impact of spatial congruence (Delong & Noppeney, 2021).…”

“…They conducted two experiments using realistic images and sounds as well as masking in experiment 2. In each trial participants were presented with one or two images that varied in location on the screen and either a semantically congruent or incongruent sound from either the right or left side (or bilaterally) (Delong & Noppeney, 2021). In both experiments, participants were asked to identify where the sound came from (left or right) and in experiment 2 they were additionally asked to identify which image(s) they saw.…”

“…Frequently, surrounding stimuli can be conflicting; therefore, understanding how the central nervous system (CNS) is able to resolve conflicts of sensory inputs when making judgements is a large area of study. The binding of separate stimuli (multisensory integration) is known to be impacted by factors such as spatial and temporal proximity and has been more recently shown to be impacted by the semantic (higher level meaning of stimuli) congruency of the stimuli (Barutchu et al, 2018;Delong & Noppeney, 2021;Doehrmann & Naumer, 2008;Spilcke-Liss et al, 2019).…”