Semantic content affects crossmodal integration in audiovisual movieclips

Abstract

Semantic content affects crossmodal perceptual integration. To date, research focusing on this effect has not investigated in depth real-world aspects, such as language. The present study is the first to investigate how parts of speech affect crossmodal integration using audiovisual movieclips, comprising actions and auditory three-word sentences. We introduce for the first time target nouns (Experiment 1) and target verbs (Experiment 2), as well as target-absent trials where a target-related verb or noun is always presented through one sense (incongruent movieclips), or both senses (congruent movieclips). Performance was superior in incongruent target-related movieclips, and significantly deteriorated in congruent target-related movieclips. Analysis of the grammatical category of target-related information led to different results between the two experiments, showing that different parts of speech can influence crossmodal integration differently. We demonstrate the significant role of semantic complexity, and the valence of cognitive schemata linked with verbs on crossmodal perceptual confusion.