Recent developments in visual sign language recognition

Abstract

Research in the field of sign language recognition has made significant advances in recent years. The present achievements provide the basis for future applications with the objective of supporting the integration of deaf people into the hearing society. Translation systems, for example, could facilitate communication between deaf and hearing people in public situations. Further applications, such as user interfaces and automatic indexing of signed videos, become feasible. The current state in sign language recognition is roughly 30 years behind speech recognition, which corresponds to the gradual transition from isolated to continuous recognition for small vocabulary tasks. Research efforts were mainly focused on robust feature extraction or statistical modeling of signs. However, current recognition systems are still designed for signer-dependent operation under laboratory conditions. This paper describes a comprehensive concept for robust visual sign language recognition, which represents the recent developments in this field. The proposed recognition system aims for signer-independent operation and utilizes a single video camera for data acquisition to ensure userfriendliness. Since sign languages make use of manual and facial means of expression, both channels are employed for recognition. For mobile operation in uncontrolled environments, sophisticated algorithms were developed that robustly extract manual and facial features. The extraction of manual features relies on a multiple hypotheses tracking approach to resolve ambiguities of hand positions. For facial feature extraction, an active appearance model is applied which allows identification of areas of interest such as the eyes and mouth region. In the next processing step, a numerical description of the facial expression, head pose, line of sight, and lip outline is computed. The system employs a resolution strategy for dealing with mutual overlapping of the signer's hands and face. Classification is based on hidden Markov models which are able to compensate time and amplitude variances in the articulation of a sign. The classification stage is designed for recognition of isolated signs, as well as of continuous sign language. In the latter case, a stochastic language model can be utilized, which considers uni-and bigram probabilities of single and successive signs. For statistical modeling of reference models each sign is represented either as a whole or as a composition of smaller subunits-similar to phonemes in spoken languages. While recognition based on word models is limited to rather small vocabularies, subunit models open the door to large vocabularies. Achieving signer-independence constitutes a challenging problem, as the articulation of a sign is subject to high interpersonal variance. This problem cannot be solved by simple feature normalization and must be addressed at the classification level. Therefore, dedicated adaptation methods known from speech recognition were implemented and modified to consider the specifics of sign languages. F...

Sign Language Recognition

“…Despite these problems recent uses of SLR include translation to spoken language, or to another sign language when combined with avatar technology [3,25]. Sign video data once recognised can be compressed using SLR into an encoded form (e.g.…”

“…They have shown that their system works very well with signer dependence, but recognition rates drop considerably in real world situations. In [3] Von Agris et al presented a comprehensive SLR system using techniques from speech recognition to adapt the signer features and classification, making the recognition task signer independent. In other work [1], they demonstrated how three approaches to speaker adaptation in speech recognition can be successfully applied to the problem of signer adaptation for signer independent sign language recognition.…”

“…Results showed that some signs of Deutsche Gebrdensprache/German Sign Language (DGS) can be recognised based on non-manual features alone, but generally the recognition rate increases by between 1.5% and 6% upon inclusion of non-manual features. In [3], Von Agris et al present a comprehensive sign language recognition system using images from a single camera. The system was developed to use manual and non-manual features in a PaHMM to recognise signs, and furthermore, statistical language modelling is applied and compared.…”

A survey on mouth modeling and analysis for Sign Language recognition

“…The phonemes are mainly expressed through combinations of manual features, such as shape, posture (orientation), location and motion of the hands. However, SLs convey much of their prosody through non-manual signs [45], [67], [47], [68], [56], [41], such as the pose of head and torso, facial expressions (combinations of eyes, eyebrows, cheeks and lips) and mouth movements. These non-manual articulators play an important role in lexical distinction, grammatical structure and adjectival or adverbial content.…”

“…In contrast to speech recognition, which is now ubiquitous in realworld HCI and other applications, ASLR is still far from being a mature technology. Nevertheless, during the last decades, there have been some significant developments in ASLR (please see [45], [67], [15] for some general surveys). However, the most of the research effort is oriented towards the SL recognition using hands-based features.…”