An Embodied Sonification Model for Sit-to-Stand Transfers

Abstract

Interactive sonification of biomechanical quantities is gaining relevance as a motor learning aid in movement rehabilitation, as well as a monitoring tool. However, existing gaps in sonification research (issues related to meaning, aesthetics, and clinical effects) have prevented its widespread recognition and adoption in such applications. The incorporation of embodied principles and musical structures in sonification design has gradually become popular, particularly in applications related to human movement. In this study, we propose a general sonification model for the sit-to-stand (STS) transfer, an important activity of daily living. The model contains a fixed component independent of the use-case, which represents the rising motion of the body as an ascending melody using the physical model of a flute. In addition, a flexible component concurrently sonifies STS features of clinical interest in a particular rehabilitative/monitoring situation. Here, we chose to represent shank angular jerk and movement stoppages (freezes), through perceptually salient pitch modulations and bell sounds. We outline the details of our technical implementation of the model. We evaluated the model by means of a listening test experiment with 25 healthy participants, who were asked to identify six normal and simulated impaired STS patterns from sonified versions containing various combinations of the constituent mappings of the model. Overall, we found that the participants were able to classify the patterns accurately (86.67 ± 14.69% correct responses with the full model, 71.56% overall), confidently (64.95 ± 16.52% self-reported rating), and in a timely manner (response time: 4.28 ± 1.52 s). The amount of sonified kinematic information significantly impacted classification accuracy. The six STS patterns were also classified with significantly different accuracy depending on their kinematic characteristics. Learning effects were seen in the form of increased accuracy and confidence with repeated exposure to the sound sequences. We found no significant accuracy differences based on the participants' level of music training. Overall, we see our model as a concrete conceptual and technical starting point for STS sonification design catering to rehabilitative and clinical monitoring applications.

Online reach adjustments induced by real-time movement sonification

“…However, this perspective extends beyond thinking of movement sonification as sensory replacement for individuals with deficits, to thinking of it as augmentation for individuals with intact sensory systems. For example, Kantan et al (2022, p. 3) say, "Regardless of the approach or application, the function of the sonification is to serve as a communication channel for task-relevant information streams." Thus, the idea is that a movement sonification system, with its acoustic feedback, constitutes a new (artificial) sensory system which can deliver movement-related information to the brain in a way which enhances perception of the body and bodily movement, e.g., affording more accurate perception of knee angle (Dyer et al 2015(Dyer et al , 2017aGhai et al 2019).…”

Sound-Guided 2-D Navigation: Effects of Information Concurrency and Coordinate System

“…In the future, we plan to test sequential Cartesian guidance in the training of the sit-to-stand transition, which involves a clear horizontal component (leaning forward) followed by a vertical one (rising). This will build on past work on providing auditory feedback on this movement [21]. An important limitation of our polar system design was that the bearing was always computed from the perspective of the center of the space.…”

Orchestrating Motion: Real-Time Sonification Tools and Methods to Support Movement Rehabilitation

“…). [113] As shown in Fig. 8, the participants classified the STS patterns accurately (86.67 ± 14.69% correct responses with the fixed + flexible component, 71.56% overall), with confidence (64.95 ± 16.52% self-reported rating), and relatively rapidly (response time: 4.28 ± 1.52 s).…”

“…As such, good design involves "finding the right way to represent the kinds of data or facts that one wishes to reveal", which is crucial given that "sonification may be ineffective if the rendered sound appears arbitrary to the listener in relation to the underlying data" [242]. Bakogiannis et al [15] emphasized that a sonification model is well designed when the representation is structure-preserving (preserves the main elements, connections and relations in the information) and inference-preserving (enables the listeners to draw conclusions about the phenomenon) [15] (also reviewed in Paper C [113]). Supper et al [222] have asserted that the crucial topic of meaning-making in sonification has remained somewhat neglected, with the community focusing more on developing technical solutions to facilitate sonification design.…”

“…The authors concluded that inter-modality feedback congruence is an important criterion in determining how human actions are externally modulated by multimodal interactive systems [224]" (quoted portion adapted from additional paper J [116]). Based on the above, it appears important to consider how an otherwise silent movement should best be sonified so as to maximize the semantic congruence between the interacting modalities when the sonification is applied.…”