Fingertip detection and tracking for recognition of air-writing in videos

Abstract

Air-writing is the process of writing characters or words in free space using finger or hand movements without the aid of any hand-held device. In this work, we address the problem of mid-air finger writing using web-cam video as input. In spite of recent advances in object detection and tracking, accurate and robust detection and tracking of the fingertip remains a challenging task, primarily due to small dimension of the fingertip. Moreover, the initialization and termination of mid-air finger writing is also challenging due to the absence of any standard delimiting criterion. To solve these problems, we propose a new writing hand pose detection algorithm for initialization of air-writing using the Faster R-CNN framework for accurate hand detection followed by hand segmentation and finally counting the number of raised fingers based on geometrical properties of the hand. Further, we propose a robust fingertip detection and tracking approach using a new signature function called distance-weighted curvature entropy. Finally, a fingertip velocity-based termination criterion is used as a delimiter to mark the completion of the air-writing gesture. Experiments show the superiority of the proposed fingertip detection and tracking algorithm over state-of-the-art approaches giving a mean precision of 73.1 % while achieving real-time performance at 18.5 fps, a condition which is of vital importance to air-writing. Character recognition experiments give a mean accuracy of 96.11 % using the proposed air-writing system, a result which is comparable to that of existing handwritten character recognition systems. IntroductionWith the emergence of virtual and augmented reality, the need for the development of natural human-computer interaction (HCI) systems to replace the traditional HCI approaches is increasing rapidly. In particular, interfaces incorporating hand gesturebased interaction have gained popularity in many fields of application viz. automotive interfaces (Ohn-Bar and Trivedi, 2014), human activity recognition (Rohrbach et al., 2016) and several state-of-the-art hand gesture recognition approaches have been developed (Molchanov et al., 2015;Rautaray and Agrawal, 2015). However, hand motion gestures as such are not sufficient to input text. This necessitates the need for the development of touch-less air-writing systems which may replace touch and electromechanical input panels leading to a more natural human-computer interaction (HCI) approach.A vision-based system for the recognition of mid-air finger-writing trajectories is not a new problem and substantial work has been done in the past two decades. One of the early works by Oka et al. (Oka et al., 2002) used a sophisticated device with an infrared and color sensor for fingertip tracking and recognition of simple geometric shapes trajectories. In (Amma et al., 2012), inertial sensors attached to a glove were used for continuous spotting and recognition of air-writing. Recently, Misra et al. (Misra et al., 2017) have developed a hand gesture recognition framewor...

Segregation of meaningful strokes, a pre‐requisite for self co‐articulation removal in isolated dynamic gestures

“…It was found experimentally that the four immediate neighbours contributed to self co-articulations in most cases. For polarity, the gradient of all the gesture coordinates was calculated using (17) and (18), and by (19), gestures with normal strokes were obtained. Then, to check the performance of global measure, using (20), the MER was calculated for 'ED', 'IV e ', and 'Po'.…”

Graph Spectral Domain Feature Learning With Application to in-Air Hand-Drawn Number and Shape Recognition

“…Recent work on in-air handwritten number recognition can be classified into two groups: image-based representation [4]- [13] and node-based representation [14]- [20]. In image-based representation, the numbers are saved as an image with two types of pixels: number path pixels and background pixels, as in static number recognition applications.…”

“…Later, further improvements have been achieved by normalizing the hand-path by picking out a specific number of points [5]. Deep learning-based techniques employ one dimensional convolutional neural networks to learn the features from image-based representations [9]- [13]. It must be also noted that some methods were concerned with recognizing words written on a touch pad using finger strokes [9]- [11], as opposed to in-air drawing as in robotic applications.…”

“…Recently, in-air hand-drawn number recognition has gained strong interest [4]- [20] due to applications-related importance and the interesting challenges in the problem. These methods are either based on image representation of numbers and shapes [4]- [13] followed by shape matching or the path representation of hand movement [14]- [20] followed by shape matching. Some methods are based on deep learning for shape matching [9]- [13].…”

Moving-Object Tracking Algorithm Based on PCA-SIFT and Optimization for Underground Coal Mines

“…Research and application of moving-target positioning is an important part of intelligent video perception, and is being studied in China and other parts of the world. Mukherjee et al [1] proposed a position-tracking method to distinguish and track multiple targets in a given scenario, based on the colors of the moving targets. Christiansen et al [2] used the moving target segmentation method based on the points on the contour of The associate editor coordinating the review of this manuscript and approving it for publication was Kun Mean Hou.…”