Gesture-based interaction via finger tracking for mobile augmented reality

Abstract

The goal of this research is to explore new interaction metaphors for augmented reality on mobile phones, i.e. applications where users look at the live image of the device's video camera and 3D virtual objects enrich the scene that they see. Common interaction concepts for such applications are often limited to pure 2D pointing and clicking on the device's touch screen. Such an interaction with virtual objects is not only restrictive but also difficult, for example, due to the small form factor. In this article, we investigate the potential of finger tracking for gesture-based interaction. We present two experiments evaluating canonical operations such as translation, rotation, and scaling of virtual objects with respect to performance (time and accuracy) and engagement (subjective user feedback). Our results indicate a high entertainment value, but low accuracy if objects are manipulated in midair, suggesting great possibilities for leisure applications but limited usage for serious tasks.

Real-time hand interaction for augmented reality on mobile phones

“…This might mean that gestures that requires longer interaction times could face acceptance problems. For instance, one of the possible solutions that Hürst et al [8] suggested was to focus on applications that requires gestures less frequently.…”

“…Other works, such as [8,17] partially addressed encumbrance issues due to multi-marker use by attaching color stickers on fingertips. Recently, Hürst et al [8] implemented various interaction capabilities on mobile phones by employing the system's sensor data, and color markers on fingertips.…”

“…Recently, Hürst et al [8] implemented various interaction capabilities on mobile phones by employing the system's sensor data, and color markers on fingertips. They showcased single-and dual-finger interactions and conducted user studies to evaluate their techniques on the mobile phone.…”

“…through gestures on a phone's touch screen, the possible scope of applications was extended. Our motivation stems from the successful demonstration of direct free-hand gestures in AR [2,3,8,13,19], and our aim is to enable such direct interaction in the "magic lens" of smart-phone based AR.…”

A study on improving close and distant device movement pose manipulation for hand-held augmented reality

“…While it is a very interesting method it is unable to move the VO and also rotating the head for manipulation can be tiresome. Hürst and van Wezel, [Hürst and van Wezel 2013] presented their research on potential of finger tracking for gesturebased interaction through two experiments. Their results indicated a high entertainment value, but low accuracy.…”