Traces of Modal Synergy: Studying Interactive Musical Sonification of Images in General-audience Use

Abstract

Photone is an interactive installation combining color images with musical sonification. The musical expression is generated based on the syntactic (as opposed to semantic) features of an image as it is explored by the user’s pointing device, intending to catalyze a holistic user experience we refer to as modal synergy where visual and auditory modalities multiply rather than add. We collected and analyzed two months’ worth of data from visitors’ interactions with Photone in a public exhibition at a science center. Our results show that a small proportion of visitors engaged in sustained interaction with Photone, as indicated by session times. Among the most deeply engaged visitors, a majority of the interaction was devoted to visually salient objects, i.e., semantic features of the images. However, the data also contains instances of interactive behavior that are best explained by exploration of the syntactic features of an image, and thus may suggest the emergence of modal synergy.

The design of Datascapes: toward a design framework for sonification for anomaly detection in AI-supported networked environments

“…In these fields, the use of sound as a substitute or as a complement to visual dashboards has attracted the interest of the research community in the past two decades. Examples of sonification systems (i.e., solutions that use acoustic and psychoacoustic dimensions e.g., pitch, loudness, timbre to represent and communicate values in a dataset) designed to support experts in monitoring tasks cover various fields, including the continuous monitoring of medical applications in healthcare (Ballora et al, 2000(Ballora et al, , 2004Ziemer and David, 2017;Shackleton et al, 2023); data monitoring in finance (Nesbitt and Barrass, 2004;Worrall, 2010), cybersecurity (Axon et al, 2017(Axon et al, , 2019(Axon et al, , 2020Lenzi et al, 2019), situational awareness in digital (Vickers et al, 2014(Vickers et al, , 2017Debashi and Vickers, 2018), physical (Rönnberg et al, 2016) and IoT networks (Roddy, 2018), process monitoring in industrial production (Hermann et al, 2015;Hildebrandt and Rinderle-Ma, 2015). These solutions share a similar use case i.e., sonification is used as a complement to existing visualizations to provide a real-time peripheral monitoring solution while reducing the cognitive overload of human operators provoked by an excess of visual information (Roetzel, 2019).…”

Flexible Automation in Air Traffic Control Through Adaptation of Human-Automation Collaboration

“…The fact that there is already a strong focus on the visual modality in the targeted control room environments could have been an argument to investigate other modalities. There are some studies of how to use sonification as a complement to visual information in ATC, for example, in a basic ATC environment by Ngo, Pierce, and Spence [56], in a radar ATC environment by Rönnberg, Lundberg, and Löwgren [73], and more recently in a tower environment by Elmquist et al [18] to enhance the awareness of aircraft's positions. While these are innovative and possible good complements to the visual modality, there are also reasons for continuing to focus on the visual modality.…”

Artefact, Participant and Interaction in Auditory Experiences

“…This way, the artefact engages with the participant as a performer in a co-operative dialogue, enabling the development of the player's action repertoire. Interactive sonifcations are an example, as the Photone installation, which combines colored images with musical sonifcation [30] or a multi-touch surface to explore model-based sonifcations [31]. Evaluations should take into consideration the communicability of a base layer of the composition, but also how effective are the artefact's attempts at directing attention, amplifying perceptions and designed interpretations and appropriations.…”