Perception and Modeling of Affective Qualities of Musical Instrument Sounds across Pitch Registers

Abstract

Composers often pick specific instruments to convey a given emotional tone in their music, partly due to their expressive possibilities, but also due to their timbres in specific registers and at given dynamic markings. Of interest to both music psychology and music informatics from a computational point of view is the relation between the acoustic properties that give rise to the timbre at a given pitch and the perceived emotional quality of the tone. Musician and nonmusician listeners were presented with 137 tones produced at a fixed dynamic marking (forte) playing tones at pitch class D# across each instrument's entire pitch range and with different playing techniques for standard orchestral instruments drawn from the brass, woodwind, string, and pitched percussion families. They rated each tone on six analogical-categorical scales in terms of emotional valence (positive/negative and pleasant/unpleasant), energy arousal (awake/tired), tension arousal (excited/calm), preference (like/dislike), and familiarity. Linear mixed models revealed interactive effects of musical training, instrument family, and pitch register, with non-linear relations between pitch register and several dependent variables. Twenty-three audio descriptors from the Timbre Toolbox were computed for each sound and analyzed in two ways: linear partial least squares regression (PLSR) and nonlinear artificial neural net modeling. These two analyses converged in terms of the importance of various spectral, temporal, and spectrotemporal audio descriptors in explaining the emotion ratings, but some differences also emerged. Different combinations of audio descriptors make major contributions to the three emotion dimensions, suggesting that they are carried by distinct acoustic properties. Valence is more positive with lower spectral slopes, a greater emergence of strong partials, and an amplitude envelope with a sharper attack and earlier decay. Higher tension arousal is carried by brighter sounds, more spectral variation and more gentle attacks. Greater energy arousal is associated with brighter sounds, with higher spectral centroids and slower decrease of the spectral slope, as well as with greater spectral emergence. The divergences between linear and nonlinear approaches are discussed.

All about that bass: Audio filters on basslines determine groove and liking in electronic dance music

“…Presenting recorded instruments playing a single pitch (D#4), Eerola, Ferrer Flores, and Alluri (2012) found a significant negative correlation between high-frequency to low-frequency energy ratio and valence ratings; in this particular experiment, the correlation between valence (pleasant/unpleasant) and preference (like/dislike) ratings was so high ( r = 0.97) as to warrant treating the two constructs synonymously. In a similar study, McAdams, Douglas, and Vampala (2017) found a significant correlation between instrument family and liking across multiple pitch registers. Finally, Wallmark, Iacoboni, Deblieck, and Kendall (2018) found a significant negative correlation between subjective ratings of “noisiness” and liking (referred to in the study as “valence”) in 400-ms excerpts of popular music.…”

Embodied Listening and Timbre

“…The finding that noisy timbre drives limbic activity more so than normal timbres produced by the same sound generators may have implications for our understanding of the role of timbre in affective response to music more generally (Eerola et al, 2012;McAdams et al, 2017). We theorize that part of the pleasure derived from music with prominent noisy timbres may be a case of ''limbic reversal'' as theorized by Huron (2006).…”

“…Filipic et al (2010), for example, found no difference between musicians and non-musicians in the perception of brief timbral stimuli. On the other hand, McAdams et al (2017) reported a three-way interaction between training, pitch register, and instrument family in perceptual evaluations of timbre. Although our stimuli were controlled for pitch, it is possible that familiarity affected behavioral evaluations and neural activity (Margulis et al, 2009).…”

“…In a recent behavioral study, McAdams, Douglas, and Vempala (2017) found that listeners evaluate affective connotations of isolated instrumental timbres consistently, even across differing pitch registers. Eerola, Ferrer, and Alluri (2012) reported similar results with pitch controlled (D#4).…”

Modeling Timbre Similarity of Short Music Clips

“…On the contrary, there is a rich literature on audio features associated with computer-based instrument identification (Joder et al, 2009 ), genre classification (e.g., Andén and Mallat, 2011 ), the prediction of affective qualities (Laurier et al, 2009 ; McAdams et al, 2017 ), or more general aspects of the perception of audio excerpts (Alluri and Toiviainen, 2010 ). Audio features are most commonly derived from the Short-Time Fourier Transform of the music signal, from which spectral or temporal statistics are computed.…”