A Linguistic Feature Vector for the Visual Interpretation of Sign Language

Abstract

Abstract. This paper presents a novel approach to sign language recognition that provides extremely high classification rates on minimal training data. Key to this approach is a 2 stage classification procedure where an initial classification stage extracts a high level description of hand shape and motion. This high level description is based upon sign linguistics and describes actions at a conceptual level easily understood by humans. Moreover, such a description broadly generalises temporal activities naturally overcoming variability of people and environments. A second stage of classification is then used to model the temporal transitions of individual signs using a classifier bank of Markov chains combined with Independent Component Analysis. We demonstrate classification rates as high as 97.67% for a lexicon of 43 words using only single instance training outperforming previous approaches where thousands of training examples are required.

Sign Language Recognition

“…This relies on a large training set for the base features (120 signs by 75 people) yet subsequently allows a new sign classifier to be trained using one shot learning. Bowden et al [12] also presented a sign language recognition system capable of correctly classifying new signs given a single training example. Their approach used a 2 stage classifier bank, the first of which used hard coded classifiers to detect hand shape, arrangement, motion and position sub-units.…”

“…The task of recognition is often simplified by forcing the possible word sequence to conform to a grammar which limits the potential choices and thereby improves recognition rates [91,104,12,45]. N-Gram grammars are often used to improve recognition rates, most often bi-gram [83,44,34] but also uni-gram [10].…”

Automated extraction of signs from continuous sign language sentences using Iterated Conditional Modes

“…Sign language research in the vision community has primarily focused on improving recognition rates of signs either by improving the motion representation and similarity measures [25,2,23,4,14] or by adding linguistic clues during the recognition process [6,9]. Ong and Ranganath [18] presented a review of the automated sign language research and also point out one important issue in continuous sign language recognition.…”

“…Following the success of Hidden Markov Models (HMMs) in speech recognition, they were used by sign language researchers [22,20,6,4,21] for representing and recognizing signs. But HMMs need a large number of training data and unlike speech, data from native signers is not yet as easily available as speech data.…”

Efficient approximations to model-based joint tracking and recognition of continuous sign language

“…Frequently, the hands are signing in front of the face, overlap, and may temporarily disappear. In early work on sign language recognition, the problem of hand tracking is facilitated by using special gloves [1,15]. Other systems require the user to start every gesture from a predefined 'home' position [10].…”