Enhanced Weak Spatial Modeling Through CNN-Based Deep Sign Language Skeletal Feature Transformation

Abstract

Recent sign language skeletal-based feature models (SLSm) consist of various distracting coordinates that lead to complex deep-learning modeling. However, SLSm is not purely a spatial-temporal coordinate arrangement problem; it is also limited by human dynamics and feature aggregations. The objectives of this work are twofold: (a) to transform the skeletal features of the SLSm model to address the problem of variations in viewpoint and changes across features of repeated signs due to human dynamics, and (b) to exploit the potential of exhaustive searching in dropping distracting features to prevent complex deep learning modeling. Method: We propose a transformed skeletal feature-based model (SCT) from a feature thresholding theory. We first extract the hand-skeletal joint-related features relevant to the coordinates and positions of the hand transcription that efficiently capture human dynamics. The extracted features are transformed into a subset of a predefined threshold and fed into the proposed ensemble exhaustive feature searching. The searched features are transformed into their equivalent deep input image sequences. Outcomes: By leveraging the skeletal-based transformed and deep spatial features, the proposed method demonstrates robust performance in sign language recognition, surpassing recent deep learning models in accuracy and simplicity. The proposed skeletal features demonstrate superiority in learning complex hand gestures of public data sets, improving accuracy by more than 2%.INDEX TERMS Human-computer interaction, End-to-end deep neural network, Multimodal data interaction, Hand gestures, Sign language recognition, and Pattern recognition.