Mid-Air Gesture Control of Multiple Home Devices in Spatial Augmented Reality Prototype

Abstract

Touchless, mid-air gesture-based interactions with remote devices have been investigated as alternative or complementary to interactions based on remote controls and smartphones. Related studies focus on user elicitation of a gesture vocabulary for one or a few home devices and explore recommendations of respective gesture vocabularies without validating them by empirical testing with interactive prototypes. We have developed an interactive prototype based on spatial Augmented Reality (AR) of seven home devices. Each device responds to touchless gestures (identified from a previous elicitation study) via the MS Kinect sensor. Nineteen users participated in a two-phase test (with and without help provided by a virtual assistant) according to a scenario that required from each user to apply 41 gestural commands (19 unique). We report on main usability indicators: task success, task time, errors (false negative/positives), memorability, perceived usability, and user experience. The main conclusion is that mid-air interaction with multiple home devices is feasible, fairly easy to learn and apply, and enjoyable. The contributions of this paper are (a) validation of a previously elicited gesture set; (b) development of a spatial AR prototype for testing of mid-air gestures, and (c) extensive assessment of gestures and evidence in favor of mid-air interaction in smart environments.

Exploring Methods to Optimize Gesture Elicitation Studies: A Systematic Literature Review

“…The framework also facilitated end-users in recommending gestures consistent with the scenario without increasing disagreements among end-users. Similar conclusions were also drawn in Vogiatzidakis et al's experiment [15], which was conducted within generic scenario as a framework. The results show participants achieved high consistency in gesture proposals.…”

“…Although GES has shown great potential in practice, current literature indicates that there are still research gaps remain unsolved, particularly the lack of a recognized best practice process. As a result, many researchers have proposed modifications and improvements based on their specific needs [12], including but not limited to: changes to the elicitation process [13], the display of referents [12], elicitation environment [7], evaluation metrics for elicitation [14], and assessment of elicitation effects [15]. This article attempts to summarize the latest research progress through a systematic literature review, to assist HCI researchers and designers in better designing and conducting gesture elicitation studies, or further developing this useful tool.…”

“…Behavioral observation: The effectiveness of the final gesture set can also be evaluated through observing behavioral performance during gesture interaction tasks, Vogiatzidakis et al [15] measured task completion time, errors (false positives and false negatives), and task success. Tan et al [64] measured reaction time to examine the performance of individual microgestures.…”

Gesture Recognition Based on Deep Learning: A Review

“…Gesture recognition can be used in smart home systems, allowing users to control home devices [20] and systems through gestures, such as adjusting lighting, temperature and volume, etc., improving the intelligence and convenience of furniture [21,22].…”

Investigating the Performance of Gesture-Based Input for Mid-Air Text Entry in a Virtual Environment: A Comparison of Hand-Up versus Hand-Down Postures

“…The results of gesture sets, such as panning and rotating, could guide the development of gesture interaction and immersive maps. For home devices in smart environments, Vogiatzidakis and Koutsabasis [ 37 ] developed a spatial AR prototype to test the mid-air gesture and found the interaction between mid-air and home devices was possible.…”