Automatic Hand Sign Recognition: Identify Unusuality Through Latent Cognizance

Abstract

Sign language is a main communication channel among hearing disability community. Automatic sign language transcription could facilitate better communication and understanding between hearing disability community and hearing majority. As a recent work in automatic sign language transcription has discussed, effectively handling or identifying a non-sign posture is one of the key issues. A non-sign posture is a posture unintended for sign reading and does not belong to any valid sign. A non-sign posture may arise during sign transition or simply from an unaware posture. Confidence ratio has been proposed to mitigate the issue. Confidence ratio is simple to compute and readily available without extra training. However, confidence ratio is reported to only partially address the problem. In addition, confidence ratio formulation is susceptible to computational instability. This article proposes alternative formulations to confidence ratio, investigates an issue of non-sign identification for Thai Finger Spelling recognition, explores potential solutions and has found a promising direction. Not only does this finding address the issue of non-sign identification, it also provide some insight behind a well-learned inference machine, revealing hidden meaning and new interpretation of the underlying mechanism. Our proposed methods are evaluated and shown to be effective for non-sign detection.

Automatic Thai Finger Spelling Transcription

“…A subsequent study [10] employing a Yolo-based Darknet-19 [11] to simultaneously locate a hand and read a sign has reported mAP 82.06 % also on a single-image setting, but with a complex background. Another study [8] has addressed FSR on a plain background using VGG-16 [12] and reported mAP 97.59 % on a single-image setting, as well as discovered Latent Cognizance-an innovative mechanism to allow an out-of-context awareness.…”

“…Therefore, we may refer that TFS has 27 multi-posture signings later on). The fact that TFS strongly relies on a multi-posture scheme makes a transcription task ambiguous, e.g., a sequence of TFS signs: 4, 7, 8, and 14 can be transcribed as [4,7,8,14], which all are treated as single postures, resulting "ตหวน" in Thai script, or transcribed as [4,7,8,14], whose first two signs are treated as a multi-posture signing, resulting "ทวน" in Thai script. Our work proposes to address this through a sequence model.…”

“…The SSC stage may represent sign labels simply by one-hot coding (denoted LO) or it may even elaborate the label representation through word embedding [22] (denoted LE). Optionally, the SSC stage can reach further and take a penultimate vector [8]-a value vector before softmax calculation-(denoted PV) or a feature vector (denoted FV) deeper inside the SR stage. Figure 5 illustrates these 4 alternatives.…”

“…Liwicki and Everingham [6] use a supervised model to identify a nonsign frame. Ingeniously, Nakjai and Katanyukul [8] use a probabilistic inference-Latent Cognizancetaking advantage of what the model has already learned. Some use image transition to identify non-sign frames.…”

“…Resorting to various signing schemes, TFS poses a challenge on automatically disambiguating a word from a signing video. Previous works [7,8] frame TFS recognition as an image classification.…”