Real-time hand interaction for augmented reality on mobile phones

Abstract

Over the past few years, Augmented Reality has become widely popular in the form of smart phone applications, however most smart phone-based AR applications are limited in user interaction and do not support gesture-based direct manipulation of the augmented scene. In this paper, we introduce a new AR interaction methodology, employing users' hands and fingers to interact with the virtual (and possibly physical) objects that appear on the mobile phone screen. The goal of this project was to support different types of interaction (selection, transformation, and fine-grain control of an input value) while keeping the methodology for hand detection as simple as possible to maintain good performance on smart phones. We evaluated our methods in user studies, collecting task performance data and user impressions about this direct way of interacting with augmented scenes through mobile phones.

Multimodal feedback for finger-based interaction in mobile augmented reality

“…In a mobile context, utilization of the touch screen is also commonplace, yet, suffers from issues, too -such as occlusion of the screen and ergonomics [7]. Researchers therefore started exploring finger tracking for handheld AR interaction as well (e.g., [5,6,7]; see [13] for an overview of different scenarios, including but not limited to handheld AR). When comparing touch versus finger tracking, Hürst et al [7] showed that the latter often suffers in performance, likely due to a lack of haptic feedback.…”

Augmented Reality based monitoring of the remote-lab

“…AR tries to solve these issues by combining real and overlaying information and visualising scenes in a proper way to understand with the interaction function. Nevertheless, there is still need to improve the interaction method in a way that will provide the reality of interaction meaning with objects [7]. Supporting the AR technique needs to be proved by applications and tangible novel devices, such as smart overlay grid.…”

“…Supporting the AR technique needs to be proved by applications and tangible novel devices, such as smart overlay grid. Most of the applications do not support gesture-base direct manipulating of the augmented scene that is responsible for user interaction with objects for more real interaction efficiency [7]. Therefore, our study has come up with solutions that enable AR interaction which includes new methodology, deploying user's smart selection tool in both virtually and the possibility of physical interaction with objects that will be shown to the end-user.…”

Towards a Natural User Interface to Support People with Visual Impairments in Detecting Colors

“…objects, surfaces, etc) in the real world. There are many applications, already on the market, aimed at supporting this task; the user points the device camera towards the desired item and then the app "reads aloud" the color in the image center or in the point where the user taps 8 . This interaction is suitable for colorblind users and/or for users with residual vision, who can still see shadows and distinguish between objects.…”

“…The reason for this simplification is that this contribution focuses on the evaluation of system usability and ergonomics. Finger detection will be added as a future work: it is a non-trivial but doable task, as proved by many papers addressing this problem [7,8].…”