Recognition of Non-Manual Content in Continuous Japanese Sign Language

Abstract

The quality of recognition systems for continuous utterances in signed languages could be largely advanced within the last years. However, research efforts often do not address specific linguistic features of signed languages, as e.g., non-manual expressions. In this work, we evaluate the potential of a single video camera-based recognition system with respect to the latter. For this, we introduce a two-stage pipeline based on two-dimensional body joint positions extracted from RGB camera data. The system first separates the data flow of a signed expression into meaningful word segments on the base of a frame-wise binary Random Forest. Next, every segment is transformed into image-like shape and classified with a Convolutional Neural Network. The proposed system is then evaluated on a data set of continuous sentence expressions in Japanese Sign Language with a variation of non-manual expressions. Exploring multiple variations of data representations and network parameters, we are able to distinguish word segments of specific non-manual intonations with 86% accuracy from the underlying body joint movement data. Full sentence predictions achieve a total Word Error Rate of 15.75%. This marks an improvement of 13.22% as compared to ground truth predictions obtained from labeling insensitive towards non-manual content. Consequently, our analysis constitutes an important contribution for a better understanding of mixed manual and non-manual content in signed communication.

Continuous Sign Language Recognition and Its Translation into Intonation-Colored Speech

“…CSLR is challenging due to the complexity and variability of sign language, as well as the need for accurate and reliable detection and recognition of hand, body, lip, and emotional movements. To overcome these problems, researchers are studying a number of approaches, including machine learning algorithms [ 5 , 6 ] and neural network models [ 7 , 8 , 9 , 10 , 11 , 12 ]. Recent advances in deep learning and neural networks have shown promising results in improving the accuracy and speed of CSLR systems.…”

“…A review of CSLR literature [ 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10 , 11 , 12 , 13 , 14 , 15 , 16 , 17 ] reveals a growing body of research focused on developing robust and efficient recognition systems. However, no work related to CSLR with intonation-colored speech has been carried out for the Kazakh language.…”

A Comprehensive Review of Sign Language Recognition: Different Types, Modalities, and Datasets

“…They trained the classifier with over a million hand shape sign data. Brock et al [84] performed Continuous Japanese SLR using CNN. They used frame-wise binary Random Forest for segmentation.…”

Air-CSL: Chinese Sign Language Recognition Based on the Commercial WiFi Devices

“…Sign language recognition systems based on multiple methods can break the barrier that exists between deaf people and people who do not know sign language. For example, some sign language recognition systems use cameras [5][6][7][8][9] or Kinect [10,11] to capture sign language gesture data literature. However, the data acquisition equipment is affected by the lighting conditions, while the camera captures data in such a way that violates the privacy of the person.…”