How Soundtracks Shape What We See: Analyzing the Influence of Music on Visual Scenes Through Self-Assessment, Eye Tracking, and Pupillometry

Abstract

This article presents two studies that deepen the theme of how soundtracks shape our interpretation of audiovisuals. Embracing a multivariate perspective, Study 1 (N = 118) demonstrated, through an online between-subjects experiment, that two different music scores (melancholic vs. anxious) deeply affected the interpretations of an unknown movie scene in terms of empathy felt toward the main character, impressions of his personality, plot anticipations, and perception of the environment of the scene. With the melancholic music, participants felt empathy toward the character, viewing him as more agreeable and introverted, more oriented to memories than to decisions, while perceiving the environment as cozier. An almost opposite pattern emerged with the anxious music. In Study 2 (N = 92), we replicated the experiment in our lab but with the addition of eye-tracking and pupillometric measurements. Results of Study 1 were largely replicated; moreover, we proved that the anxious score, by increasing the participants' vigilance and state of alert (wider pupil dilation), favored greater attention to minor details, as in the case of another character who was very hard to be noticed (more time spent on his figure). Results highlight the pervasive nature of the influence of music within the process of interpretation of visual scenes.

Music and Time Perception in Audiovisuals: Arousing Soundtracks Lead to Time Overestimation No Matter Their Emotional Valence

“…Such integration involves both bottom-up (sensory-perceptual) and top-down (expectative) processes: on the one hand, a recipient perceives information using their senses; on the other, one integrates this information using previous knowledge and cognitive schemas stored in the long-term memory [82]. Other studies have already revealed that, under the influence of differently valenced soundtracks for the same video, not only do the viewers generate diverse plot expectations [20,83] and alter their recall of the scene [84,85] (i.e., high-level processing), but they can also be driven and even deceived in a way that impacts their visual perception (i.e., low-level processing) [18][19][20][21]. Therefore, it can be assumed that such a to-be-processed integration, only present in the music conditions, could be the cause of time overestimation in accordance with the memory-based model of time perception.…”

“…This method allowed us to present all the possible combinations of valence and arousal (Table 2). Similarly to [20], the four pieces were chosen by considering the findings of [71], and the subsequent studies enumerated by [72] concerning a plethora of psychoacoustic parameters associated with emotional expression in music. Two of the pieces evoked negative affects but differed in the arousal dimension: the After Celan track's soft tone and morbid intensity fosters sadness and tenderness [73].…”

“…As we will see, aside from the research traditionally focusing on music (i.e., perceived duration of musical excerpts), some work has pointed at the domain of audiovisual stimuli [15] and film-induced mood [16], with some interest in the dynamic interaction between the auditory (i.e., musical) and visual elements [17]. For example, it has been found that music is capable of modulating and altering visual perception [18][19][20] due to phenomena such as auditory driving [21].…”

Scaring away anxiety: Therapeutic avenues for horror fiction to enhance treatment for anxiety symptoms

“…Although specific details of horror content may be uncertain, the tropes of horror provide a sense of "certain uncertainty" for the viewer, as they set a template of what the viewer can expect if certain elements are present. For example, the viewer may not know when a jump-scare will occur, but music in the movie can notify the audience that something frightening is imminent and direct them to be vigilant (Ansani et al, 2020). Another example is the mirror trope, in which a character looks in a mirror or reflective surface, there is a cutaway, and then the camera returns to the mirror to reveal that the killer or monster is behind the character.…”

The effect of background music on the aesthetic experience of a visual artwork in a naturalistic environment

“…Growing evidence suggests that musical emotions can bias judgments of the emotion of complex visual stimuli, such as dynamic visual scenes and film (Ansani et al, 2020; Cohen, 2001, 2013; Herget, 2021; Steffens, 2020), as well as faces (Hanser et al, 2015; Jeong et al, 2011; Jomori et al, 2013; Logeswaran & Bhattacharya, 2009; Marin et al, 2017; Quarto et al, 2014), geometric shapes/figures (Bhattacharya & Lindsen, 2016; Marshall & Cohen, 1988; Weinreich & Gollwitzer, 2016), and pictures (Arriaga et al, 2014; Baumgartner et al, 2006; Campos-Bueno et al, 2015; Marin et al, 2012). It has been shown, for instance, that emotional judgment of neutral facial stimuli can be systematically biased toward the direction of the emotional valence of the music presented a few seconds before the visual stimuli (Logeswaran & Bhattacharya, 2009), and that brightness judgments of geometric shapes can be affected by the emotional valence of the musical primes (Bhattacharya & Lindsen, 2016).…”