Hierarchical temporal structure in music, speech and animal vocalizations: jazz is like a conversation, humpbacks sing like hermit thrushes

Abstract

Humans talk, sing and play music. Some species of birds and whales sing long and complex songs. All these behaviours and sounds exhibit hierarchical structure-syllables and notes are positioned within words and musical phrases, words and motives in sentences and musical phrases, and so on. We developed a new method to measure and compare hierarchical temporal structures in speech, song and music. The method identifies temporal events as peaks in the sound amplitude envelope, and quantifies event clustering across a range of timescales using Allan factor (AF) variance. AF variances were analysed and compared for over 200 different recordings from more than 16 different categories of signals, including recordings of speech in different contexts and languages, musical compositions and performances from different genres. Non-human vocalizations from two bird species and two types of marine mammals were also analysed for comparison. The resulting patterns of AF variance across timescales were distinct to each of four natural categories of complex sound: speech, popular music, classical music and complex animal vocalizations. Comparisons within and across categories indicated that nested clustering in longer timescales was more prominent when prosodic variation was greater, and when sounds came from interactions among individuals, including interactions between speakers, musicians, and even killer whales. Nested clustering also was more prominent for music compared with speech, and reflected beat structure for popular music and self-similarity across timescales for classical music. In summary, hierarchical temporal structures reflect the behavioural and social processes underlying complex vocalizations and musical performances.

Rhythm in speech and animal vocalizations: a cross‐species perspective

“…In Ref. 152 and two subsequent studies (Ref. 153 and Schneider, Ramirez-Aristizabal, Gavilan, and Kello, unpublished data) dozens of recordings of various types of conversational inter-actions, in both English and Spanish, have all yielded AF functions with a common slope and bend.…”

“…Kello and colleagues also applied AF analysis to a wide range of speech, music, and animal vocalization recordings. Results were consistent with those of Falk and Kello and also extended them by showing that nested clustering is enhanced by musical composition.…”

“…Each AF function for each sound recording from Ref. was quantified in terms of a linear coefficient that corresponded to the degree of hierarchical temporal structure, and a quadratic coefficient that corresponded to the amount and direction of change in hierarchical temporal structure as a function of timescale. Four different categories of sound recordings were analyzed—animal vocalization, speech, classical music, and popular music—each with four subcategories (see the legend).…”

The Bursts and Lulls of Multimodal Interaction: Temporal Distributions of Behavior Reveal Differences Between Verbal and Non‐Verbal Communication

“…The burstiness framework is particularly useful for the current study's purposes because it (a) provides an index of the temporal structures of verbal and non-verbal communicative behavior estimated from a distribution of interevent intervals of behavioral events and (b) can also provide a simplified view into the non-homogeneous processes that are features of complex, dynamical systems. Although there have been many documented observations of non-homogeneous processes generating bursty or clustered dynamics in human communicative behavior (e.g., Abney, Paxton, Dale, & Kello, 2014;Altmann, Cristadoro, & Degli Esposti, 2012;Falk & Kello, 2017;Kello et al, 2017), there is also evidence that the temporal structure of communicative behavior is rhythmic and periodic (Cummins & Port, 1998;Dauer, 1983;Kohler, 2009;Tilsen & Arvaniti, 2013). It is, therefore, an open question regarding where verbal and non-verbal communicative behaviors fall on a spectrum of temporal structure.…”

Evolution of speech rhythm: a cross-species perspective

“…With this result in hand, Kello and colleagues 75 applied AF analysis to a wide range of speech, music, and animal vocalization recordings. Results further supported the relationship between hierarchical temporal structure and prosodic exaggeration, in that synthesized speech with impoverished prosodic cues contained less nested clustered compared with natural speech.…”