Sight vs. Sound Judgments of Music Performance Depend on Relative Performer Quality: Cross-cultural Evidence From Classical Piano and Tsugaru Shamisen Competitions

Abstract

Which information dominates in evaluating performance in music? Both experts and laypeople consistently report believing that sound should be the most important domain when judging music competitions, but experimental studies of Western participants rating video-only vs. audio-only versions of 6-second excerpts of Western classical performances have shown that in at least some cases visual information can play a stronger role. However, whether this phenomenon applies generally to music competitions or is restricted to specific repertoires or contexts is disputed. In this Registered Report, we focus on testing the generalizability of sight vs. sound effects by replicating previous studies of classical piano competitions with Japanese participants, while also expanding the same paradigm using new examples from competitions of a traditional Japanese folk musical instrument: the Tsugaru shamisen. For both classical piano and Tsugaru shamisen, we ask participants to choose the winner between the 1st- and 2nd- placing performers in 5 competitions and the 1st-place and low-ranking performers in 5 competitions (i.e., 40 performers total from 10 piano and 10 shamisen competitions). We tested the following three predictions twice each (once for piano and once for shamisen): 1) an interaction was predicted between domain (video-only vs. audio-only) and variance in quality (choosing between 1st and 2nd place vs. choosing between 1st and low-placing performers); 2) visuals were predicted to trump sound when variation in quality is low (1st vs. 2nd place); and 3) sound was predicted to trump visuals when variation in quality is high (1st vs. low-placing). Our experiments (n = 155 participants) confirmed our first predicted interaction between audio/visual domain and relative performer quality for both piano and shamisen conditions, suggesting that this interaction is cross-culturally general. In contrast, the second prediction was only supported for the piano stimuli and the third prediction was only supported for the shamisen condition, suggesting culturally dependent factors in the specific balance between sight and sound in the judgment of musical performance. Our results resolve discrepancies and debates from previous sight-vs-sound studies by replicating and extending them to include non-Western participants and musical traditions. Our findings may also have practical applications to evaluation criteria for performers, judges, and organizers of competitions, concerts, and auditions.

The influence of sensory modality on aesthetic judgments of poetry

“…While there has not been any work comparing written to spoken poems, or the effect of combining these two modalities, research in music has provided some hints about the differences between audio only and combined audiovisual presentations of musical performances. Prior work has looked at the effect of audio-visual information on the aesthetic and emotional appeal of musical performances (e.g., Chiba et al, 2023;Coutinho & Scherer, 2017;Pan et al, 2019;Tsay, 2013). A number of these findings suggest that the combination of audio and visual information can enhance emotional perception when compared to audio-only or visual-only presentations (Thompson et al, 2008).…”

“…That is, more recent data has indicated that when looking at musical performers of a similar quality, visual cues have been shown to help enhance and differentiate aesthetic judgements when in a competitive context (Chiba et al, 2023). When looking at visual information from musical performances, performances with high visual expressiveness were rated to be more engaging, enjoyable, and emotionally expressive when compared to performances with low visual expressiveness (Thompson et al, 2008).…”

The influence of sensory modality on aesthetic judgments of poetry.

“…That is, recent data has indicated that when looking at musical performers of a similar quality, visual cues have been shown to help enhance and differentiate aesthetic judgments when in a competitive context (Chiba et al, 2023). When looking at visual information from musical performances, performances with high visual expressiveness were rated to be more engaging, enjoyable, and emotionally expressive when compared to performances with low visual expressiveness (Thompson et al, 2008).…”

“…While there has not been any work comparing written to spoken poems, or the effect of combining these two modalities, research in music can provide some hints about the differences between audio-only and combined audiovisual presentations of musical performances. Prior work has looked at the effect of audiovisual information on the aesthetic and emotional appeal of musical performances (e.g., Chiba et al, 2023; Coutinho & Scherer, 2017; Pan et al, 2019; Tsay, 2013). A number of these findings suggest that the combination of audio and visual information can enhance emotional perception when compared to audio-only or visual-only presentations (Thompson et al, 2008).…”