Recognition of Symbolic Gestures Using Depth Information

Abstract

Symbolic gestures are the hand postures with some conventionalized meanings. They are static gestures that one can perform in a very complex environment containing variations in rotation and scale without using voice. The gestures may be produced in different illumination conditions or occluding background scenarios. Any hand gesture recognition system should find enough discriminative features, such as hand-finger contextual information. However, in existing approaches, depth information of hand fingers that represents finger shapes is utilized in limited capacity to extract discriminative features of fingers. Nevertheless, if we consider finger bending information (i.e., a finger that overlaps palm), extracted from depth map, and use them as local features, static gestures varying ever so slightly can become distinguishable. Our work here corroborated this idea and we have generated depth silhouettes with variation in contrast to achieve more discriminative keypoints. This approach, in turn, improved the recognition accuracy up to 96.84%. We have applied Scale-Invariant Feature Transform (SIFT) algorithm which takes the generated depth silhouettes as input and produces robust feature descriptors as output. These features (after converting into unified dimensional feature vectors) are fed into a multiclass Support Vector Machine (SVM) classifier to measure the accuracy. We have tested our results with a standard dataset containing 10 symbolic gesture representing 10 numeric symbols (0-9). After that we have verified and compared our results among depth images, binary images, and images consisting of the hand-finger edge information generated from the same dataset. Our results show higher accuracy while applying SIFT features on depth images. Recognizing numeric symbols accurately performed through hand gestures has a huge impact on different Human-Computer Interaction (HCI) applications including augmented reality, virtual reality, and other fields.

Depth data and fusion of feature descriptors for static gesture recognition

“…Depth map provided by the Kinect camera is the preferred cue [8, 17] to segment the closest object in an image as it aids in addressing the following challenges present in gesture recognition‐cluttered background, light source with varying illumination levels, varying skin tones of the user's hands performing the gesture and occlusion of hands with each other and with the face.…”

“…Thus, to work on complex datasets, many authors combined two or more cues from the Kinect camera for more reliable and accurate gesture recognition. Mahnud et al [8] used both RGB and depth map for hand segmentation and extraction of features. Plouffe and Cretu [9] utilised both the depth map and skeleton information for hand segmentation and DTW for classification.…”

Analysis of Application of Cluster Descriptions in Space of Characteristic Image Features

“…The "bag of features" [12,13] method is a "bag of words" analog, which is used in computer vision applications to build strong descriptions of an image. This approach usually includes key point feature detection (typically SIFT), quantization (clustering) of these features (typically with k-means) and distribution of key point features in the space of cluster centers in the form of a histogram (we refer to this procedure as "screening").…”

“…A lot of different modifications are proposed for the abovementioned common method, including classification with support vector machines [13] and smart additional processing of key point descriptions [13].…”

Understanding intuitive gestures in wearable mixed reality environments

“…Results of this dissertation suggest that user experience developers and designers should consider the specified MR interaction scenario. New MR users require additional affordances, guidance, and feedback when interacting in MR in order to have successful user experiences(Mahmud, Hasan, Abdullah-Al-Tariq, Kabir & Mottalib, 2018). Also, data from this dissertation suggests that new gestures and additional testing are needed in order to best utilize the tools provided by MR HUDs, with the goal of achieving a natural telepresence in MR through the interaction of MR HUD UIs.…”