User-centred process for the definition of free-hand gestures applied to controlling music playback

Abstract

Music is a fundamental part of most cultures. Controlling music playback has commonly been used to demonstrate new interaction techniques and algorithms. In particular, controlling music playback has been used to demonstrate and evaluate gesture recognition algorithms. Previous work, however, used gestures that have been defined based on intuition, the developers' preferences, and the respective algorithm's capabilities. In this paper we propose a refined process for deriving gestures from constant user feedback. Using this process every result and design decision is validated in the subsequent step of the process. Therefore, comprehensive feedback can be collected from each of the conducted user studies. Along the process we develop a set of free-hand gestures for controlling music playback. The situational context is analysed to shape the usage scenario and derive an initial set of necessary functions. In a successive user study the set of functions is validated and proposals for gestures are collected from participants for each function. Two gesture sets containing static and dynamic gestures are derived and analysed in a comparative evaluation. The comparative evaluation shows the suitability of the identified gestures and allows further refinement. Our results indicate that the proposed process, that includes validation of each design decision, improves the final results. By using the process to identify gestures for controlling music playback we not only show that the refined process can successfully be applied, but we also provide a consistent gesture set that can serve as a realistic benchmark for gesture recognition algorithms.

Frame-Based Elicitation of Mid-Air Gestures for a Smart Home Device Ecosystem

“…Most importantly, a comparison of the results of the studies reveals that the proposed gestures are quite different, therefore a "universal gesture set" for mid-air interaction with smart TVs has not been obtained. Other home devices or activities investigated include music playback control [10] and "entertainment environment" [6].…”

“…The need for mid-air control of home devices can be argued for many use cases including user convenience (i.e., immediate (fast) control of a device, over remote control, which requires another control device), as well as for people that have temporary injuries or permanent mobility impairments which affect their mobility inside the home. Mid-air interactions have been investigated in the smart home mainly to identify and craft gestures for single device interactions, like the TV [3][4][5][6][7][8][9] or various media (e.g., [6,10]). However, a smart home environment consists of several interconnected devices with variable sensing capabilities, which may be controlled and manipulated remotely.…”

Gesture Elicitation Studies for Mid-Air Interaction: A Review

“…From our review, it appears that two studies (2 of 47, 4%) adopted the general approach of Nielsen's "Intuitive and Ergonomic" Method. In particular, [23] conducted an intuitive and ergonomic method to investigate and compare gestures proposed by experts and novices using a vision-based Anaesthesia-related system within an operating room, while [26], adopted Nielsen's approach to elicit the set of commands of a music player and the gestures that are more appropriate for each task.…”

“…Many other dimensions of the appropriateness of a mid-air gesture were examined, such as comfort [11,23,27,36,44,60], perceived fatigue [24,36,40,50], discoverability [34,55,58] learnability (how easy gestures can be learned) [24,27,50], gesture simplicity [26,45,61], body parts suitability [19,73], concurrent gestures during intense gameplay [19] and unimanual or bimanual gestures [48,53]. Finally, a few studies (7 out of 47, 14.9%) did not mentioned any specific focus of the gestures proposed.…”

“…The number of participants in the elicitation studies examined is shown in Table 6. Two studies employed only four participants [47,55] and another one employed 9 participants [26], while four studies employed from 35 to 89 users [22,29,53,57]. All other studies recruited between 10 to 30 participants.…”