Head gesture sonification for supporting social interaction

Abstract

In this paper we introduce two new methods for real-time sonification of head movements and head gestures. Head gestures such as nodding or shaking the head are important non-verbal back-channelling signals which facilitate coordination and alignment of communicating interaction partners. Visually impaired persons cannot interpret such non-verbal signals, same as people in mediated communication (e.g. on the phone), or cooperating users whose visual attention is focused elsewhere. We introduce our approach to tackle these issues, our sensing setup and two di↵erent sonification methods. A first preliminary study on the recognition of signals shows that subjects understand the gesture type even without prior explanation and can estimate gesture intensity and frequency with no or little training.

A Review of Data Gathering Methods for Evaluating Socially Assistive Systems

“…A total of 26 studies used questionnaires to collect data. These studies were targeted at older adult, i.e., [ 17 , 18 , 58 , 60 , 61 , 62 , 63 , 64 , 65 , 66 , 69 ], people with visual impairments, i.e., [ 19 , 28 , 29 , 33 , 34 , 77 , 80 , 83 , 85 , 87 ], deafness, i.e., [ 43 , 46 , 48 ], and physical disabilities, i.e., [ 50 , 54 ]. A total of 34 questionnaires ( Table 3 ) from 26 studies consisted of two categories: (a) standardized questionnaires (N = 21 questionnaires) and (b) self-developed questionnaires (N = 13 questionnaires).…”

“…For example, in [ 69 ], the researchers developed a questionnaire to measure the qualities of life of participants such as psychological aspect and social environment. Ten self-developed questionnaires in [ 18 , 28 , 29 , 34 , 48 , 50 , 60 , 77 , 83 , 85 ], measured usability-related aspects of SASs. Usability in these questionnaires consists of several dimensions, such as ease of use in [ 34 , 48 ], learnability [ 29 ], user experience in [ 83 , 85 ], and user satisfaction in [ 60 ].…”

“…Ten studies collected data through user task performance. These studies targeted people with visual impairments (as in [ 28 , 30 , 31 , 32 , 77 , 79 ]), older adults (as in [ 17 , 70 ]), children with ASD (as in [ 36 ]), and people with social communication disorders (SCD) (as in [ 56 ]). Two types of user task performance were identified as follows:…”

Sonification for supporting joint attention in dyadic augmented reality-based cooperation

“…Participants filled out a questionnaire containing statements and questions, and a 7-point Likert scale ranging from 1 ('false') to 7 ('true') (resp. 'no' to 'yes') 2 . We also collected basic data such as age, sex and profession as well as information about experience with computers and musical instruments and possible issues related to sound awareness.…”

“…An earlier system of this project made use of head gesture sonifications such as nodding and shaking the head: as the head-mounted displays allow either to look on the desk or to look to the interaction partner, but not simultaneously, the sonification of head gestures conveys analogic and subtle information to support interaction [2]. Furthermore, enhancing and augmenting object sounds with informative or aesthetic acoustic additions is a well established approach in Sonic Interaction Design [4], yet so far rarely considered for collaborative applications.…”