Egocentric-View Fingertip Detection for Air Writing Based on Convolutional Neural Networks

Abstract

This research investigated real-time fingertip detection in frames captured from the increasingly popular wearable device, smart glasses. The egocentric-view fingertip detection and character recognition can be used to create a novel way of inputting texts. We first employed Unity3D to build a synthetic dataset with pointing gestures from the first-person perspective. The obvious benefits of using synthetic data are that they eliminate the need for time-consuming and error-prone manual labeling and they provide a large and high-quality dataset for a wide range of purposes. Following that, a modified Mask Regional Convolutional Neural Network (Mask R-CNN) is proposed, consisting of a region-based CNN for finger detection and a three-layer CNN for fingertip location. The process can be completed in 25 ms per frame for 640×480 RGB images, with an average error of 8.3 pixels. The speed is high enough to enable real-time “air-writing”, where users are able to write characters in the air to input texts or commands while wearing smart glasses. The characters can be recognized by a ResNet-based CNN from the fingertip trajectories. Experimental results demonstrate the feasibility of this novel methodology.

Next Generation Computing and Communication Hub for First Responders in Smart Cities

“…Any innovation can be further improved. For example, hand gesture detectors can be improved by 'air-writing' fingertips [66], which is extremely useful in some specific emergency scenarios when voice communication is impossible (e.g., loss of voice as a trauma or related disabilities). The apps for recognition of individuals with disabilities, identification of the type of impairment, and interaction with these individuals in order to provide help are considered in Section 7.…”

Deep Learning Based Air-Writing Recognition with the Choice of Proper Interpolation Technique

“…They built a synthetic dataset using Unity3D and proposed a modified mask regional convolutional neural network. Their method could detect fingertip for air-writing in a minimal length of time for each frame [23]. Kim et al experimented with the WiTA dataset, which contains air-writing data for Korean and English alphabets collected by RGB cameras [24].…”

Fingertip Detection Algorithm Based on Maximum Discrimination HOG Feature in Complex Background

“…In [11], the YOLO algorithm was used to obtain the hand area, and then the fingertip detection was regressed by the VGG16 full convolutional neural network. Y.-H.Chen et al [12] proposed an improved masked region convolutional neural network (Mask R-CNN), which uses a region-based CNN network to detect fingers and achieves fingertip detection through a three-layer CNN network. Deep learning-based algorithms (such as YOLO, etc.)…”