An unconscious study of audiovisual semantic consistency based on printed words

Abstract

Chen & Spence found in a previous study the time-courses and categorical specificity of crossmodal semantic congruency effects in pictures and printed words. To explore whether the time-courses and semantic consistency of audio and pictures can affect human's visual processing unconsciously, we developed a Python-based audio-visual semantic consistency program to study the impact of auditory cues on the breakthrough timing of printed words under the Two-Alternative Forced Choice continuous flash suppression (2AFC-CFS) paradigm. Specifically, auditory cues were presented at 5 different (-1000ms, -750ms, -500ms, -250ms, 0ms) stimulus onset asynchronies (SOAs) with respect to the visual targets. In addition, there were 5 different match types in our auditory cues and printed words: congruent, incongruent, correlated, noisy and no-sound. The results of the study shows that SOA and congruency have the main effect in the unconscious condition. At the same time, spoken words produced greater facilitation than naturalistic sounds auditory stimuli. When leading by 500ms or more, spoken words induced a slowly emerging congruency effect in the congruent. In all cases, however, auditory stimuli sped up recognition compared to no sound. These results therefore suggests that the neural representations associated with visual and auditory stimuli can interact in a shared semantic system.