Parallel hidden Markov models for American sign language recognition

Abstract

The major challenge that faces American Sign Language (ASL) recognition now is to develop methods that will scale well with increasing vocabulary size. Unlike in spoken languages, phonemes can occur simultaneously in ASL. The number of possible combinations of phonemes after enforcing linguistic constraints is approximately 5:5 10 8 : Gesture recognition, which is less constrained than ASL recognition, suffers from the same problem.Thus, it is not feasible to train conventional hidden Markov models (HMMs) for large-scale ASL applications. Factorial HMMs and coupled HMMs are two extensions to HMMs that explicitly attempt to model several processes occuring in parallel. Unfortunately, they still require consideration of the combinations at training time.In this paper we present a novel approach to ASL recognition that aspires to being a solution to the scalability problems. It is based on parallel HMMs (PaHMMs), which model the parallel processes independently. Thus, they can also be trained independently, and do not require consideration of the different combinations at training time.We develop the recognition algorithm for PaHMMs and show that it runs in time polynomial in the number of states, and in time linear in the number of parallel processes. We run several experiments with a 22 sign vocabulary and demonstrate that PaHMMs can improve the robustness of HMM-based recognition even on a small scale. Thus, Pa-HMMs are a very promising general recognition scheme with applications in both gesture and ASL recognition.

Sign Language Recognition

“…Due to the relative disadvantages of HMMs (poor performance when training data is insufficient, no method to weight features dynamically and violations of the stochastic independence assumptions), they coupled the HMM recogniser with motion analysis using computer vision techniques to improve combined recognition rates [100]. In their following work, Vogler and Metaxas [101] demonstrated that Parallel Hidden Markov Models (PaHMMs) are superior to regular HMMs, Factorial HMMs and Coupled HMMs for the recognition of sign language due the intrinsic parallel nature of the phonemes. The major problem though is that regular HMMs are simply not scalable in terms of handling the parallel nature of phonemes present in sign.…”

“…While their later work [101], used PaHMMs on both hand shape and motion sub-units, as proposed by the linguist Stokoe [95]. Work has also concentrated on learning signs from low numbers of examples.…”

A System for Large Vocabulary Sign Search

“…The majority of existing methods are model-based, using Hidden Markov Models [6][7][8][9] or alternative approaches such as recursive partition trees [10], boosted volumetric features [11], and hidden conditional random fields [12]. Such methods typically use ten or more training examples per gesture or sign.…”

Untethered gesture acquisition and recognition for virtual world manipulation

“…Hence it was of limited use in tracking natural pointing gestures, although it was able to recognize parametric gestures defined by the relative position of both hands [33] using a variation of Hidden Markov Models. Other HMM approaches to gesture recognition include [19,32]. Bobick and Davis [11] created a view-dependent approach to gesture recognition using temporal templates.…”