Cross-Modal Associations Between Harmonic Dissonance and Visual Roughness

Abstract

Visual associations with auditory stimuli have been the subject of numerous studies. Colour, shape, size, and several other parameters have been linked to musical elements like timbre and pitch. In this article, we aim to examine the relationship between harmonisations with varying degrees of dissonance and visual roughness. Based on past research in which high sensory dissonance was associated with angular shapes, we argued that nontonal and highly dissonant harmonisations will be associated with angular and rough images, while more consonant stimuli will be associated with the images of low visual roughness. A fixed melody was harmonised in 7 different styles, including highly tonal, nontonal, and random variations. Through a listening task, musically trained participants rated the stimuli in terms of enjoyment, familiarity, and matched them to images of variable roughness. The overall consonance of the stimuli was calculated using two distinct models (Harrison & Pearce, 2020; Wang et al., 2013 ) and a variant of the aggregate dyadic consonance index ( Huron, 1994 ). Our results demonstrate that dissonance, as calculated by all models, was highly correlated with visual roughness, and enjoyment and familiarity followed expected patterns compared to tonal and nontonal stimuli. In addition to sensory dissonance, however, it appears that other factors, such as the typicality of chord progressions and the sense of tonality may also influence this cross-modal interaction.

Roughness perception: A multisensory/crossmodal perspective

“…This, in turn, supports the amodal nature of roughness perception, at least if defined in terms of the pick up by two or more senses (i.e., vision and touch). Such a conclusion is suggested also by the existence of consistent crossmodal correspondences involving roughness, at least across specific sensory domains and regarding specific features (auditory roughness-visual shapes, see Giannos et al, 2021 ; Liew et al, 2018 ; Liew et al, 2017 ; hue-tactile roughness, see Ludwig & Simner, 2013 ; Slobodenyuk et al, 2015 ; see Table 2 ). It is perhaps also worth noting here that the seemingly natural use of the term roughness across a diverse range of experiences, and its seemingly easy interpretation by those who come across the term being used outside of a purely auditory/tactile context, might be taken to support that the term identifies an amodal or intermodal concept.…”

“…Several recent studies have examined the relationship between auditory roughness and shapes, showing that listeners tend to match more dissonant sounds to spikier and rougher objects/shapes (and vice versa; Liew et al, 2018 ; Liew et al, 2017 ). Giannos et al ( 2021 ) extended the investigation from isolated sounds to the harmonic context, hypothesizing that nontonal and highly dissonant harmonic stimuli would have been associated with rough images, while more consonant stimuli would be associated with the images of low visual roughness. To test this hypothesis, they harmonized a fixed melody in seven different styles, including highly tonal, non-tonal, and random variations, asking their participants to match the melodies to images of variable roughness (i.e., black and white 2D and 3D images that represented surfaces with different degrees smoothness/roughness).…”

“…Lower row. Visual stimuli used by Giannos et al ( 2021 ) to study crossmodal associations between melodies harmonized with different degrees of roughness and 3D surface textures …”

“…Auditory dissonance correlates with visual roughness and spikier images (Giannos et al, 2021 ; Liew et al, 2017 ; Liew et al, 2018 ).…”

Cross-modal associations between paintings and sounds: Effects of embodiment

“…Researchers continue to delineate associations between sensations from different modalities ( Giannos et al, 2021 ; Mok et al, 2019 ; Motoki et al, 2020 ; Saluja & Stevenson, 2018 ). They also attempt to understand the mechanisms which underly these associations.…”