MagicHand: Interact with IoT Devices in Augmented Reality Environment

Abstract

We present an Augmented Reality (AR) visualization and interaction tool for users to control Internet of Things (IoT) devices with hand gestures. Today, smart IoT devices are becoming increasingly ubiquitous with diverse forms and functions, yet most user controls over them are still limited to mobile devices and web interfaces. Recently, AR has been developed rapidly, and provided immersive solutions to enhance user experience of applications in many fields. Its capability to create immersive interactions allows AR to improve the way smart devices are controlled via more direct visual feedback. In this paper, we create a functional prototype of one such system, enabling seamless interactions with sound and lighting systems through the use of augmented hand-controlled interaction panels. To interpret users' intentions, we implement a standard 2D convolution neural network (CNN) for real-time hand gesture recognition and deploy it within our system. Our prototype is also equipped with a simple but effective object detector which can identify target devices within a proper range by analyzing visual and geometric features. We evaluate the performance of our system qualitatively and quantitatively and demonstrate it on two smart devices.

Multimodal Interaction Systems Based on Internet of Things and Augmented Reality: A Systematic Literature Review

“…AR is used to visualize graphical objects on a view of the real world, where a real-time camera feed on a device or head-mounted display (HMD) typically represents the view. AR objects can not only be used as simple data representations [9,10], but can also be combined with an input modality, such as touch on a mobile device [11], as part of a user interface (UI) to allow an agent to interact with an MI system [12,13]. This property distinguishes between applications that only utilize AR as a data visualization tool and those that use AR as part of their UI to interact with the underlying MI system.…”

“…The combination of AR and IoT technologies makes it possible to bridge the real world, virtual world (i.e., digital content), and ambient environment (i.e., information from IoT sensors), thereby increasing the richness of information and opening possibilities for context-aware application development. For example, visualized data of an ambient environment can serve as digital content that is superimposed on the real world [10,12,13]. However, the use and development of multimodal interaction in applications that combine AR and IoT has remained a little researched area.…”

“…IoT and AR are used differently in MI systems, depending on the intended purpose. For example, Dodevska et al [10] utilized AR as a data visualization tool by displaying data from IoT sensors that were attached to a centrifuge, whereas Sun et al [12] created an AR interface that could be manipulated by an agent's hand gestures to control IoT devices. The agent was able to send commands to control an IoT device in both studies.…”

“…An interaction device is a platform that visualizes AR content to the agent and it may also receive input from the agent. The interaction device can have many forms, such as a mobile device (e.g., smartphone, tablet PC) [10,11,[33][34][35][36][37]39,40,42,44,46,48,[50][51][52], HMD [12,38,41,45,49], PC/TV monitor [41,43,47], and projector [41].…”

Towards Industrial IoT-AR Systems using Deep Learning-Based Object Pose Estimation

“…Fusing IoT and AR is not entirely novel, and has been demonstrated in [3]- [5]. However, AR has not been adopted to the fullest in a manufacturing context.…”