Audio-visual integration of emotional cues in song

Visual determinants of a cross-modal illusion

“…Audiences can visually ascertain a performer's emotional intentions (Dahl & Friberg, 2007) and the relative size of sung intervals (Thompson & Russo, 2007), as well as extract certain structural and expressive aspects of a musical composition from an accompanying choreographed dance (Krumhansl & Schenck, 1997). Furthermore, vision can influence the perception of pitch (Gillespie, 1997;Thompson et al, 2005), loudness (Rosenblum & Fowler, 1991), and timbre (Saldaña & Rosenblum, 1993), as well as affect (Thompson, Russo, & Quinto, 2008), expressivity (Davidson, 1993(Davidson, , 1994, audience interest (Broughton & Stevens, 2009), phrasing (Vines, Krumhansl, Wanderley, & Levitin, 2006), performance quality (Wapnick, Darrow, Kovacs, & Dalrymple, 1997;Wapnick, Mazza, & Darrow, 1998), and lyric comprehension (Hidalgo-Barnes & Massaro, 2007). Because of the widespread nature of vision's role in music, a further understanding of the perceptual consequences of musicians' physical gestures will be useful to performers and audiences alike.…”

A comparison of the McGurk effect for spoken and sung syllables

“…For example, body movements can change the perception of musical tension and expressiveness (Dahl & Friberg, 2007;Davidson, 1993;Vines, Krumhansl, Wanderley, & Levitin, 2006), tone duration (Schutz & Lipscomb, 2007), and plucked versus bowed judgments (Saldaña & Rosenf blum, 1993). Facial cues can influence the perception of happiness and sadness conveyed by music (Thompson, Graham, & Russo, 2005;Thompson, Russo, & Quinto, 2008) and the size of melodic intervals (Thompson, Russo, & Livingstone, 2010). The above studies reflect a simple compromise between auditory and visual cues rather than a wholly independent fused percept.…”

Interaction of Sight and Sound in the Perception and Experience of Musical Performance

“…In order to investigate the relative contributions of auditory and visual performance cues to observers' evaluations, an experimental method is needed in which the expressivity conveyed by the visual and auditory components of a performance can be manipulated independently, so as to result in matched and mismatched audiovisual pairings. Such experimental designs have been successfully used to investigate the interaction of auditory and visual components in the perception of note duration (Schutz & Lipscomb, 2007), loudness (Rosenblum & Fowler, 1991), timbre (Saldaña & Rosenblum, 1993), pitch (Thompson, Graham, & Russo, 2005), and interval affect (Thompson, Russo, & Quinto, 2008), demonstrating that visual information can significantly alter the perception of various auditory features. However, the difficulty with applying such a design to a complex action such as musical performance is that musicians find it very difficult to control expressivity in one modality independent of the other (e.g., Thompson & Luck, 2012), and the temporal properties of a musical performance also vary greatly from one performance to the next.…”