Lightweight palm and finger tracking for real-time 3D gesture control

Abstract

We present a novel technique implementing barehanded interaction with virtual 3D content by employing a time-of-flight camera. The system improves on existing 3D multi-touch systems by working regardless of lighting conditions and supplying a working volume large enough for multiple users. Previous systems were limited either by environmental requirements, working volume, or computational resources necessary for realtime operation. By employing a time-of-flight camera, the system is capable of reliably recognizing gestures at the finger level in real-time at more than 50 fps with commodity computer hardware using our newly developed precision hand and finger-tracking algorithm. Building on this algorithm, the system performs gesture recognition with simple constraint modeling over statistical aggregations of the hand appearances in a working volume of more than 8 cubic meters. Two iterations of user tests were performed on a prototype system, demonstrating the feasibility and usability of the approach as well as providing first insights regarding the acceptance of true barehanded touch-based 3D interaction. INTRODUCTIONMulti-touch interaction techniques have become widely available recently, being used for instance in table top systems such as Microsoft's Surface [30] projection-based systems such as CityWall [24], desktop systems such as HP's TouchSmart series as well as in several mobile devices, in particular smartphones such as the Google Nexus and, of course, the iPhone. The introduction of multi-touch interaction techniques has probably been the most important change to user input since the introduction of the mouse.To date, multi-touch interaction typically is surface based. Selection of objects is required before actually manipulating them by touching the corresponding surface with the hands.Freehand multi-touch has been explored within various approaches, e.g. Oblong's g-speak 1 , after initially having been introduced in the popular Hollywood movie "Minority Report". They usually depended heavily on hand-worn gloves, markers, wrists, or other input devices, and typically did not achieve the intuitiveness, simplicity and efficiency of surface (2D) based multi-touch techniques 2 . In contrast to those approaches, the goal of our approach was to use barehanded interaction as a replacement for surface based interaction.Only a vision-based approach will allow for freehand and barehanded 3D multi-touch interaction. The system must also provide sufficient solutions for the following four steps: detection of hand position without prior knowledge of existence; for each appearance determine pose from image cues; track appearances over time; recognize gestures based on their trajectory and pose information. Various approaches exist to solve all four problems, each featuring different advantages and disadvantages.Barehanded 3D interaction has recently been presented by Mygestyk 3 and provides the basis for Microsoft's Kinect interface for Xbox 360 4 . However, these approaches are limited either to a sing...

A Survey on Vision-based Dynamic Gesture Recognition

“…Model-based approaches [21], [22], [18] estimate the current hand state by matching a 3D hand model to the observed image features. The difficulty of this approach is in feature extraction because human hands are texture less and do not provide reliable edges internally [23].…”

“…The main problem with this modeling technique is that it deals with all the parameters of hand which are of huge dimensionality. Skeletal models overcome this problem of volumetric hand parameters models by limiting the set of parameters in modeling hand shapes from 3D structures [22].…”

“…Virtual reality interaction uses the hand gesture to manipulate the virtual movements using one or two hands for 2D and 3D interactions display [23]. In [22], they provided 3D pointing gesture recognition for natural human computer Interaction HCI in a real-time from binocular views. Some virtual reality applications are available in [56], [57], [58].…”

Sequential Classification of Palm Gestures Based on A* Algorithm and MLP Neural Network for Quadrocopter Control

“…So far, many attempts were made to control various systems and devices, using biomedical signals. For example, partially paralyzed people can still use EEG [1], EMG [2], EOG [3] signals, as an HMI interface to perform basic actions, or visually impaired people can interact with a computer using palm gestures [4].…”

Estimate Hand Poses Efficiently from Single Depth Images

“…Vision-based hand interpretation plays important roles in diverse applications including humanoid animation (Sueda et al 2008;Wang and Popović 2009), robotic control (Gustus et al 2012), and human-computer interaction (Hackenberg et al 2011;Melax et al 2013), among others. In its core lies the nevertheless challenging problem of 3D hand pose estimation (Erol et al 2007;Gorce et al 2011), owing mostly to the complex and dexterous nature of hand articulations (Gustus et al 2012).…”