Hand Shape Recognition Using Very Deep Convolutional Neural Networks

Abstract

This work examines the application of modern deep convolutional neural network architectures for classification tasks in the sign language domain. Transfer learning is performed by pre-training the models on the ImageNet dataset. After fine-tuning on the ASL fingerspelling and the 1 Million Hands datasets the models outperform state-of-the-art approaches on both hand shape classification tasks. Introspection of the trained models using Saliency Maps is also performed to analyze how the networks make their decisions. Finally, their robustness is investigated by occluding selected image regions.

DDaNet: Dual-Path Depth-Aware Attention Network for Fingerspelling Recognition Using RGB-D Images

“…Table 4 lists the average classification accuracy among the five subjects obtained from LOOCV evaluation. Some state-of-the-art methods use handcrafted features based on prior knowledge [2-7, 9, 11, 14, 16, 19, 22, 24] while others automatically learn discriminative descriptors [21,27,28,32,35,36]. The proposed DDaNet outperforms the other methods in terms of accuracy (93.53%), demonstrating the benefits of learning discriminative features related to letter signs through a deep neural network with an attention module.…”

“…When compared to the deep-learning-based methods that jointly extract features and classify the letter signs [7,21,32,36], the proposed DDaNet ach ieves the highest precision (94.10%), recall (93.48%), and F score (93.26%). Note that only four methods provide results of precision, recall, and F score.…”

“…Deep learning techniques handle this limitation by automatically learning discriminative features from a large dataset containing scene variations. Table 1 shows that most studies have introduced convolutional neural networks (CNNs) to jointly learn hand descriptors and letter sign classifiers [21,28,32,35,36]. In some studies, deep neural networks have been designed for specific purposes, such as feature extraction [27] and classification [7].…”

“…Feature Classifier Year [3] Scale-invariant features SCF 2012 [5] SP-EMD TM 2014 [7] Depth features DBN 2014 [9] Point cloud HMM 2019 [2] Gabor filters RF 2011 [4] Shape function descriptor RF 2013 [14] Oriented radial distribution RF 2014 [8] Depth comparison feature RF 2015 [19] Random line segments RF 2017 [6] Hand dominant line SVM 2013 [11] Shape matrix SVM 2015 [22] Local binary patterns SVM 2015 [24] 3D local shape descriptor SVM 2015 [16] Contour fragments features SVM 2016 [27] PCANet SVM 2019 [29] FuseNet * 2016 [21] CNN * 2017 [32] ResNet * 2018 [36] CNN + multi-view * 2018 [37] Two-stream CNN * 2019 [38] RFBNet * 2019 [28] Gabor texture features + CNN* 2019 [35] Deep eigenspace * 2020 * Feature representations and classifiers jointly learned using deep learning.…”

One for All: An End-to-End Compact Solution for Hand Gesture Recognition

“…1. Architectural comparison between existing (a) [1], [2] -(b) [3], [4] two stage network and proposed one for all: an end-to-end solution for HGR. The visual representation implies the proposed frameworks efficacy to handle the all kind of challenges: complex signs, illumination variations, complex and cluttered backgrounds.…”