Seeing music performance: Visual influences on perception and experience

Abstract

Drawing from ethnographic, empirical, and historical / cultural perspectives, we examine the extent to which visual aspects of music contribute to the communication that takes place between performers and their listeners. First, we introduce a framework for understanding how media and genres shape aural and visual experiences of music. Second, we present case studies of two performances, and describe the relation between visual and aural aspects of performance. Third, we report empirical evidence that visual aspects of performance reliably influence perceptions of musical structure (pitch related features) and a¤ective interpretations of music. Finally, we trace new and old media trajectories of aural and visual dimensions of music, and highlight how our conceptions, perceptions and appreciation of music are intertwined with technological innovation and media deployment strategies.

A comparison of the McGurk effect for spoken and sung syllables

“…For example, body movements can change the perception of musical tension and expressiveness (Dahl & Friberg, 2007;Davidson, 1993;Vines, Krumhansl, Wanderley, & Levitin, 2006), tone duration (Schutz & Lipscomb, 2007), and plucked versus bowed judgments (Saldaña & Rosenf blum, 1993). Facial cues can influence the perception of happiness and sadness conveyed by music (Thompson, Graham, & Russo, 2005;Thompson, Russo, & Quinto, 2008) and the size of melodic intervals (Thompson, Russo, & Livingstone, 2010). The above studies reflect a simple compromise between auditory and visual cues rather than a wholly independent fused percept.…”

Interaction of Sight and Sound in the Perception and Experience of Musical Performance

“…In order to investigate the relative contributions of auditory and visual performance cues to observers' evaluations, an experimental method is needed in which the expressivity conveyed by the visual and auditory components of a performance can be manipulated independently, so as to result in matched and mismatched audiovisual pairings. Such experimental designs have been successfully used to investigate the interaction of auditory and visual components in the perception of note duration (Schutz & Lipscomb, 2007), loudness (Rosenblum & Fowler, 1991), timbre (Saldaña & Rosenblum, 1993), pitch (Thompson, Graham, & Russo, 2005), and interval affect (Thompson, Russo, & Quinto, 2008), demonstrating that visual information can significantly alter the perception of various auditory features. However, the difficulty with applying such a design to a complex action such as musical performance is that musicians find it very difficult to control expressivity in one modality independent of the other (e.g., Thompson & Luck, 2012), and the temporal properties of a musical performance also vary greatly from one performance to the next.…”

Musicians have enhanced subcortical auditory and audiovisual processing of speech and music

“…Lip and facial movements provide timing or segmentation cues (e.g., of consonant and vowels), as well as more complex information, such as emotional state, that improve the listener's reaction time and recognition of speech (13)(14)(15)(16)(17). Similarly, a musician's face and body movements convey cues for time-varying features of music, such as rhythm and phrasing (e.g., the grouping of notes into a division of a composition), the emotional content of the piece (17), and changes to and from consonant and dissonant musical passages (18). Audiovisual perception of speech and music share some commonalities.…”