Evaluating the Effect of Semantic Congruency and Valence on Multisensory Integration

Letts, Elyse; Basharat, Aysha; Barnett‐Cowan, Michael

doi:10.1163/22134808-bja10073

Cited by 3 publications

(1 citation statement)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To date, it is strongly supported that this significant difference in RTs is due to the semantic congruency between the two modality signals (Laurienti et al, 2004; Molholm et al, 2004; Mastroberardino, Santangelo, & Macaluso, 2015; Tsilionis & Vatakis, 2016). However, Letts, Basharat, and Barnett-Cowan (2022) bring the parameter of valence being another significant factor in multisensory integration. Our results provide further evidence that semantic congruency cannot be a critical factor on its own for determining multisensory perception.…”

Section: Discussionmentioning

confidence: 99%

The effect of semantic content on the perception of audiovisual movieclips

Kesoglou,

Mikellidou

2024

Preprint

View full text Add to dashboard Cite

Our brain is skilled with the ability to perceive and process multimodal stimuli. This process known as crossmodal perceptual integration, has been in the research spotlight for a long time, providing evidence for the integration of information coming from different modalities. Prior research mostly utilized pictures and focused on the semantic content of a single sound or word. The present study aims to investigate crossmodal perceptual integration in realistic conditions using short movieclips (1500ms) and auditory meaningful three-word sentences to evaluate target detection in terms of accuracy and response times. Two experimental tasks were developed using PsychoPy, where participants had to indicate whether a target (noun for Experiment 1, verb for Experiment 2) was present or absent. In trials without a target, target-related information was always present, either through one of the two senses (vision or audition; incongruent condition) or through both senses (congruent condition). We observed superior performance when the target was absent generally (Mexp1 =93.9%, SDexp1 =0.04, Mexp2 =84.2%, SDexp2 =0.182) compared to when it was present (Mexp1 =82.3%, SDexp1 =0.143, Mexp2 =73.7%, SDexp2 =0.184). Moreover, superior performance was noted in incongruent target-related movieclips, which significantly decreased during congruent target-related movieclips. In Experiment 1, we observed that in the audio condition when the target-related word was a noun, participant performance was superior compared to when it was a verb (M=99.4% vs. M=86.7%; tincVerb vs. incNoun =-8.428, p=.001). In Experiment 2, the judgment scores were similarly high in incongruent movieclips and significantly lower in congruent ones regardless of whether the target-related information presented was a verb or noun. The present results provide evidence regarding the role of complexity of semantics, and especially the diverse role verbs and nouns could play in crossmodal perceptual integration in more realistic situations. Our findings can enrich the content of learning techniques, as well as the design of AI models, by taking advantage of the supporting role of semantic audiovisual information, while taking into consideration the potential confusion that the complexity of semantic information can induce to perception experience.

show abstract

Section: Discussionmentioning

confidence: 99%

The effect of semantic content on the perception of audiovisual movieclips

Kesoglou,

Mikellidou

2024

Preprint

View full text Add to dashboard Cite

show abstract

Harmony of senses: Exploring the impact of sound aesthetic features' on taste imagery

Velasco,

Corradi,

Motoki

2023

Food Quality and Preference

View full text Add to dashboard Cite

Differential fMRI neural synchrony associated with migraine during emotionally arousing naturalistic stimuli

Klamer,

Craig,

Haines

et al. 2024

Preprint

View full text Add to dashboard Cite

Migraine is a common neurological disorder that impacts approximately 12% of the general population and is characterized by moderate to severe headaches, nausea, mood changes, and fatigue. It impacts lower-level visual and auditory processing, causing hypersensitivities that lead to heightened audiovisual multisensory integration. However, the impact of migraine on the processing of complex, audiovisual stimuli is still unclear. Additionally, migraine may induce hypersensitivities to emotional arousal and valence, though the relative significance of these factors remains unknown. The current study seeks to identify how migraine impacts synchronous neural processing of complex, audiovisual stimuli, and how this differs based on the emotional arousal and valence of the stimulus. To do so, we collected functional magnetic resonance imaging data from 17 migraineurs and 25 healthy controls during the passive viewing of three audiovisual films of differing emotional arousal and valence. We identified that, in response to high arousal emotional stimuli, the migraine group showed greater neural synchrony in regions associated with multisensory integration, including the posterior superior temporal gyrus. In response to the negative valence film, the migraine group had heightened neural synchrony in bilateral cuneus and the right angular gyrus. These findings suggest that migraine involves hypersensitivity to audiovisual movies as a function of both emotional arousal and valence, with high arousal driving greater synchrony in multisensory integration and negative valence enhancing synchrony in peripheral visual processing. Overall, this research highlights distinct pathways through which emotional stimuli impacts neural processing in migraine.

show abstract

Evaluating the Effect of Semantic Congruency and Valence on Multisensory Integration

Cited by 3 publications

References 37 publications

The effect of semantic content on the perception of audiovisual movieclips

The effect of semantic content on the perception of audiovisual movieclips

Harmony of senses: Exploring the impact of sound aesthetic features' on taste imagery

Differential fMRI neural synchrony associated with migraine during emotionally arousing naturalistic stimuli

Contact Info

Product

Resources

About