Listeners as spectators? Audio-visual integration improves music performer identification

Abstract

Listeners take for granted their capacity to distinguish between musical instruments, and their ability to discriminate between performers playing the same instrument by their sound alone. Sound perception is usually considered a purely auditory process, but there is significant debate on how auditory and visual information are integrated during listening. Two experiments examined how listeners perceive individual performers. Saxophonists (n = 5) performed three jazz standards for an audio and video recording. Experiment 1 investigated listeners' ability to identify individual saxophonists by ear. Listeners heard one saxophonist, and were later asked to identify him from a lineup of saxophonists. Only 52% of listeners could correctly identify their target. Experiment 2 explored the integration of cross-modal sensory experiences (audio and visual) in saxophonist identification. Participants either watched a silent video clip of a saxophonist playing and matched it to an audio clip of the same performer, or heard an audio clip of a saxophonist and matched it to a silent video clip. Listener/viewers reliably identified their target saxophonists, and results suggest that listeners combine cross-modal perceptions to identify individual performers, and can use the information about a performer in one modality and match it to the same performer in another modality.

Listeners' and Performers' Shared Understanding of Jazz Improvisations

“…For example, the facts that eye contact between performers and audience members (e.g., Antonietti et al, 2009) or visual characteristics of the performers (Davidson, 2001, 2012; Thompson et al, 2005; Mitchell and MacDonald, 2012; Morrison et al, 2014) can affect audience members' judgments do not clearly point to whether audience members' interpretations will therefore be more similar to each other as a result, or more similar to performers'. Similarly, the fact that audience members can react differently to comparable live vs. recorded performances (see Barker, 2013; Katevas et al, 2015) doesn't clearly predict in which situation they are more likely to share understanding with performers or each other.…”

“…Our focus is on listeners' understanding during solo listening to an audiorecording of a live performance, rather than how listeners experience a live (or audio- or videorecorded) performance in which they are physically copresent with and can be affected by the reactions of other listeners as an audience in a shared space (e.g., Pennebaker, 1980; Mann et al, 2013; Koehler and Broughton, 2016; Zadeh, in press). Because listeners are not copresent with the performers nor do they see video of the performers, additional factors that can affect audience experience, like eye contact between performers and audience members (e.g., Antonietti et al, 2009) or visual characteristics of the performers (Davidson, 2001, 2012; Thompson et al, 2005; Mitchell and MacDonald, 2012; Morrison et al, 2014), do not come into play.…”

What you see is what you hear: The importance of visual priming in music performer identification

“…This is an important finding as it highlights how sensitive listeners are to performers' visual cues. Listener/viewers have already shown they are proficient at identifying target speakers, and indeed saxophonists, that they see or hear from forced-choice pairs in the alternate sensory domain (Kamachi et al, 2003;Lachs & Pisoni, 2004;Mitchell & MacDonald, 2014). This novel line-up design indicated that while increasing the number of distractors affected listener/viewers' ability to identify individual performers, they could still consistently and reliably match audio and visual presentations of individual saxophonists.…”

The Relative Importance of Aural and Visual Information in the Evaluation of Western Canon Music Performance by Musicians and Nonmusicians

“…In addition to performers' movements being linked to the music and their expressive intentions, audio and visual information are also integrated in audience members' perceptions of performance: both melodic cues and facial expressions influence observers' judgements of the emotional valance of sung intervals, which suggests that visual aspects of performance are integrated with aural content (Thompson, Russo, & Quinto, 2008). This perspective is supported by the discovery that audiovisual integration is important in the successful identification of individual musicians (Mitchell & MacDonald, 2014). Furthermore, there is evidence that audience members integrate audio and visual information in judgements of note duration, and that those ratings of note duration vary as a function of changes in visual information but not audio information (Schutz & Lipscomb, 2007).…”