Personalised interactive sonification of musical performance data

Abstract

In this article, we describe methods and consequences for giving audience members interactive control over the real-time sonification of performer movement data in electronic music performance. We first briefly describe how to technically implement a musical performance in which each audience member can interactively construct and change their own individual sonification of performers' movements, heard through headphones on a personal WiFi-enabled device, while also maintaining delay-free synchronization between performer movements and sound. Then, we describe two studies we conducted in the context of live musical performances with this technology. These studies have allowed us to examine how providing audience members with the ability to interactively sonify performer actions impacted their experiences, including their perceptions of their own role and engagement with the performance. These studies also allowed us to explore how audience members with different levels of expertise with sonification and sound, and different motivations for interacting, could be supported and influenced by different sonification interfaces. This work contributes to a better understanding of how providing interactive control over sonification may alter listeners' experiences, of how to support everyday people in designing and using bespoke sonifications, and of new possibilities for musical performance and participation.

Sonification of Complex Spectral Structures

“…Musical sonification has been applied in movement rehabilitation (Kantan et al, 2021 ), in rehabilitation after stroke (Nikmaram et al, 2019 ), in gate retraining (Lorenzoni et al, 2018 ), and in rehabilitation of Parkinsonian dysgraphia (Véron-Delor et al, 2018 ). Music has been also applied for the sonification of visual artworks (Adhitya and Kuuskankare, 2012 ), for inter-active sonification of drummers' gestures for personalizing the resulting performance (Wolf and Fiebrink, 2019 ), and for the sonification of climate temperature data mapping them C major scale notes played by string instruments (George et al, 2017 ). Also, the spectrum of the light emitted by galaxies has been sonified by using harmonic sequences (Ballora, 2014 ).…”

Ritualistic Approach to Sonic Interaction Design: A Poetic Framework for Participatory Sonification

“…By making collective participation necessary for the functioning of the sonification system proposed in this paper, we want to enclose the artwork to the domain of rituals. Hermann's and Hunt's definition of interactive sonification [8] and KatieAnna Wolf's work on personalized interactive sonification [9] provide a solid starting point to support such statement, however not entirely aligned with it. While the former sees interactive sonification as a virtual musical instrument, played primarily to learn more about the data, the latter tries to reconcile the role of sonification in music performance presenting it as a medium to enrich audience members' experience of music.…”

Introduction to the special issue on interactive sonification

“…Wolf and Fiebrink presented a platform which allows audiences of a musical performance to interact with the performance by sonifying the movements produced by musicians while playing an air-drum [8]. The bidirectional and personalised approach promotes a stronger enjoyment and engaging experience.…”