Hand Pose Guided 3D Pooling for Word-level Sign Language Recognition

Hosain, Al Amin; Santhalingam, Panneer Selvam; Pathak, Parth H.; Rangwala, Huzefa; Košecká, Jana

doi:10.1109/wacv48630.2021.00347

Cited by 32 publications

(7 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Firstly, several 2D-CNN-LSTM models have been trained separately using RGB, depth, and optical flow data and then these features are fused at the classification level using the best 2D-CNN-LSTM model. In [11], both motion and hand shape cues have been used as input features and fed to a 3D-CNN. A pose-guided 3D pooling mechanism is used to fuse the prediction score during test time.…”

Section: Hybrid/multi-modal Methodsmentioning

confidence: 99%

“…The latest deep learning models make use of ConvNets to extract spatial cues and Recurrent neural networks to model temporal dependencies. Some of these models make use of 3D ConvNets [11] to fuse spatial and temporal cues. However, appearance-based methods have distinctively higher computational complexity originating from higher data dimensionality.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Signgraph: An Efficient and Accurate Pose-Based Graph Convolution Approach Toward Sign Language Recognition

et al. 2023

View full text Add to dashboard Cite

Sign language recognition (SLR) enables the deaf and speech-impaired community to integrate and communicate effectively with the rest of society. Word level or isolated SLR is a fundamental yet complex task with the main objective of using models to correctly recognize signed words. Sign language consists of very fast and complex hand, body, face movements, and mouthing cues that make the task very challenging. Several input modalities; RGB, optical Flow, RGB-D, and pose/skeleton have been proposed for SLR. However, the complexity of these modalities and the state-of-the-art (SOTA) methodologies tend to be exceedingly sophisticated and over-parametrized. In this paper, our focus is to use the hands and body poses as an input modality. One major problem in pose-based SLR is extracting the most valuable and distinctive features for all skeleton joints. In this regard, we propose an accurate, efficient, and lightweight pose-based pipeline leveraging a graph convolution network (GCN) along with residual connections and a bottleneck structure. The proposed architecture not only facilitates efficient learning during model training providing significantly improved accuracy scores but also alleviates computational complexity. With the proposed architecture in place, we are able to achieve improved accuracies on three different subsets of the WLASL dataset and the LSA-64 dataset. Our proposed model outperforms previous SOTA pose-based methods by providing a relative improvement of 8.91%, 27.62%, and 26.97% for WLASL-100, WLASL-300, and WLASL-1000 subsets. Moreover, our proposed model also outperforms previous SOTA appearance-based methods by providing a relative improvement of 2.65% and 5.15% for WLASL-300 and WLASL-1000 subsets. For the LSA-64 dataset, our model is able to achieve 100% test recognition accuracy. We are able to achieve this improved performance with far less computational cost as compared to existing appearancebased methods.

show abstract

Section: Hybrid/multi-modal Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Signgraph: An Efficient and Accurate Pose-Based Graph Convolution Approach Toward Sign Language Recognition

et al. 2023

View full text Add to dashboard Cite

show abstract

“…The overall dataset's performance of the Top-1, Top-2, and Top-3 percentages was analyzed in terms of accuracy for each dataset group. The performance of the hDNN-SLR was evaluated with the other cutting-edge models such as I3D [43], Pose-TGCN [43], Pose-GRU [43], GCN-BERT [26], Multi-Stream [44], and Fusion-3 [45]. Table 2 shows the performance evaluation of hDNN-SLR with other baseline architectures in terms of Accuracy measures using the WLASL100, WLASL300, WLASL1000 and WLASL2000 dataset consisting of 100, 300, 1000 and 2000 word sign videos.…”

Section: E Experimental For Islr Using a Benchmark Datasetmentioning

confidence: 99%

Multi-Semantic Discriminative Feature Learning for Sign Gesture Recognition Using Hybrid Deep Neural Architecture

et al. 2023

View full text Add to dashboard Cite

The speech and hearing-impaired community use sign language as the primary means of communication. It is quite challenging for the general population to interpret or learn sign language completely. A sign language recognition system must be designed and developed to address this communication barrier. Most current sign language recognition systems rely on wearable sensors, keeping the recognition system unaffordable for most individuals. Moreover, the existing vision-based sign recognition frameworks do not consider all of the spatial and temporal information required for accurate recognition. A novel vison-based hybrid deep neural net methodology is proposed in this study for recognizing Indian and Russian sign gestures. The proposed framework is aimed to establish a single framework for tracking and extracting multisemantic properties, such as non-manual components and manual co-articulations. Furthermore, spatial feature extraction from the sign gestures is deployed using a 3D deep neural net with atrous convolutions. The temporal and sequential feature extraction is carried out by employing attention-based Bi-LSTM. In addition, the distinguished abstract feature extraction is done using the modified autoencoders. The discriminative feature extraction for differentiating the sign gestures from unwanted transition gestures is done by leveraging the hybrid attention module. The experimentation of the proposed model has been carried out on the novel multi-signer Indo-Russian sign language dataset. The proposed sign language recognition framework with hybrid neural net yields better results than other state-of-the-art frameworks.

show abstract

“…They achieve 65.89%, 84.11%, and 89.92% for top-1, top-5, and top-10 accuracies, respectively, on WLASL with 100 sign classes, and 56.14%, 79.94%, 86.98% on WLASL with 300 classes. In other work that uses motion and hand shapes to guide the pooling for a three-dimensional CNN on RGB video frames, Hosain et al [87] achieve top-1, top-5, and top-10 accuracies on WLASL 100 of 75.67%, 86.83%, and 90.91%, respectively, and 68.30%, 84.19%, 87.06% on WLASL 300, using their Fusion-2 and Fusion-3 models.…”

Section: Related Workmentioning

confidence: 99%

Modelling Sign Language with Encoder-Only Transformers and Human Pose Estimation Keypoint Data

Woods

Rana

2023

Mathematics

View full text Add to dashboard Cite

We present a study on modelling American Sign Language (ASL) with encoder-only transformers and human pose estimation keypoint data. Using an enhanced version of the publicly available Word-level ASL (WLASL) dataset, and a novel normalisation technique based on signer body size, we show the impact model architecture has on accurately classifying sets of 10, 50, 100, and 300 isolated, dynamic signs using two-dimensional keypoint coordinates only. We demonstrate the importance of running and reporting results from repeated experiments to describe and evaluate model performance. We include descriptions of the algorithms used to normalise the data and generate the train, validation, and test data splits. We report top-1, top-5, and top-10 accuracy results, evaluated with two separate model checkpoint metrics based on validation accuracy and loss. We find models with fewer than 100k learnable parameters can achieve high accuracy on reduced vocabulary datasets, paving the way for lightweight consumer hardware to perform tasks that are traditionally resource-intensive, requiring expensive, high-end equipment. We achieve top-1, top-5, and top-10 accuracies of 97%, 100%, and 100%, respectively, on a vocabulary size of 10 signs; 87%, 97%, and 98% on 50 signs; 83%, 96%, and 97% on 100 signs; and 71%, 90%, and 94% on 300 signs, thereby setting a new benchmark for this task.

show abstract

Hand Pose Guided 3D Pooling for Word-level Sign Language Recognition

Cited by 32 publications

References 35 publications

Signgraph: An Efficient and Accurate Pose-Based Graph Convolution Approach Toward Sign Language Recognition

Signgraph: An Efficient and Accurate Pose-Based Graph Convolution Approach Toward Sign Language Recognition

Multi-Semantic Discriminative Feature Learning for Sign Gesture Recognition Using Hybrid Deep Neural Architecture

Modelling Sign Language with Encoder-Only Transformers and Human Pose Estimation Keypoint Data

Contact Info

Product

Resources

About