Does Timbre Modulate Visual Perception? Exploring Crossmodal Interactions

Abstract

Musical timbre is often described using terms from non-auditory senses, mainly vision and touch; but it is not clear whether crossmodality in timbre semantics reflects multisensory processing or simply linguistic convention. If multisensory processing is involved in timbre perception, the mechanism governing the interaction remains unknown. To investigate whether timbres commonly perceived as “bright-dark” facilitate or interfere with visual perception (darkness-brightness), we designed two speeded classification experiments. Participants were presented consecutive images of slightly varying (or the same) brightness along with task-irrelevant auditory primes (“bright” or “dark” tones) and asked to quickly identify whether the second image was brighter/darker than the first. Incongruent prime-stimulus combinations produced significantly more response errors compared to congruent combinations but choice reaction time was unaffected. Furthermore, responses in a deceptive identical-image condition indicated subtle semantically congruent response bias. Additionally, in Experiment 2 (which also incorporated a spatial texture task), measures of reaction time (RT) and accuracy were used to construct speed-accuracy tradeoff functions (SATFs) in order to critically compare two hypothesized mechanisms for timbre-based crossmodal interactions, sensory response change vs. shift in response criterion. Results of the SATF analysis are largely consistent with the response criterion hypothesis, although without conclusively ruling out sensory change.

Music and Time Perception in Audiovisuals: Arousing Soundtracks Lead to Time Overestimation No Matter Their Emotional Valence

“…Such integration involves both bottom-up (sensory-perceptual) and top-down (expectative) processes: on the one hand, a recipient perceives information using their senses; on the other, one integrates this information using previous knowledge and cognitive schemas stored in the long-term memory [82]. Other studies have already revealed that, under the influence of differently valenced soundtracks for the same video, not only do the viewers generate diverse plot expectations [20,83] and alter their recall of the scene [84,85] (i.e., high-level processing), but they can also be driven and even deceived in a way that impacts their visual perception (i.e., low-level processing) [18][19][20][21]. Therefore, it can be assumed that such a to-be-processed integration, only present in the music conditions, could be the cause of time overestimation in accordance with the memory-based model of time perception.…”

“…As we will see, aside from the research traditionally focusing on music (i.e., perceived duration of musical excerpts), some work has pointed at the domain of audiovisual stimuli [15] and film-induced mood [16], with some interest in the dynamic interaction between the auditory (i.e., musical) and visual elements [17]. For example, it has been found that music is capable of modulating and altering visual perception [18][19][20] due to phenomena such as auditory driving [21].…”

Learning to play a musical instrument in the middle school is associated with superior audiovisual working memory and fluid intelligence: A cross-sectional behavioral study

“…Moreover, general musical sophistication was found to correlate with self-reported music training ( Müllensiefen et al, 2014 ). As the students of the musical curriculum are chosen through aptitude tests based on imitation and recognition of rhythm, melody and pitch, we aimed to statistically control for pre-existing individual differences in musicality to better see the effect of music training, similarly to other studies where some Gold-MSI subscales were used as covariates ( Kang and Williamson, 2012 ; Farrugia et al, 2015 ; Ma et al, 2021 ; Wallmark et al, 2021 ; Krause et al, 2022 ).…”

Roughness perception: A multisensory/crossmodal perspective

“…In a series of two experiments, Wallmark et al ( 2021 ) asked whether the timbre of a musical note (an acoustic prime) would affect the subsequent visual perception of, in the first experiment, brightness (dark–bright dimension) and, in the second experiment, both brightness and spatial texture (smooth–rough dimension). To this end, they used a speeded-response paradigm in which the participants had to identify a shift in roughness/brightness between two consecutively-presented target squares of subtly contrasting levels (rougher/brighter, smoother/darker, or the same).…”

“…Moreover, strong differences might exist in the emotional response to felt textures depending on individual factors. However, Wallmark et al ( 2021 ) claim that the use of the term rough to talk about musical timbre and visual objects might be more than just be an arbitrary linguistic convention: “Perhaps we see brightness and spatial texture not just through eyes, but also (albeit faintly) through timbrally attuned ears” (p. 16). This drives us back at the core question at stake here, i.e., the possibility for intersensory, or amodal, stimuli quality to exist (see Spence & Di Stefano, 2022b ).…”