Cued Speech automatic recognition in normal-hearing and deaf subjects

Abstract

This article discusses the automatic recognition of Cued Speech in French based on hidden Markov models (HMMs). Cued Speech is a visual mode which, by using hand shapes in different positions and in combination with lip patterns of speech, makes all the sounds of a spoken language clearly understandable to deaf people. The aim of Cued Speech is to overcome the problems of lipreading and thus enable deaf children and adults to understand spoken language completely. In the current study, the authors demonstrate that visible gestures are as discriminant as audible orofacial gestures. Phoneme recognition and isolated word recognition experiments have been conducted using data from a normal-hearing cuer. The results obtained were very promising, and the study has been extended by applying the proposed methods to a deaf cuer. The achieved results have not shown any significant differences compared to automatic Cued Speech recognition in a normal-hearing subject. In automatic recognition of Cued Speech, lip shape and gesture recognition are required. Moreover, the integration of the two modalities is of great importance. In this study, lip shape component is fused with hand component to realize Cued Speech recognition. Using concatenative feature fusion and multi-stream HMM decision fusion, vowel recognition, consonant recognition, and isolated word recognition experiments have been conducted. For vowel recognition, an 87.6% vowel accuracy was obtained showing a 61.3% relative improvement compared to the sole use of lip shape parameters. In the case of consonant recognition, a 78.9% accuracy was obtained showing a 56% relative improvement compared to the use of lip shape only. In addition to vowel and consonant recognition, a complete phoneme recognition experiment using concatenated feature vectors and Gaussian mixture model (GMM) discrimination was conducted, obtaining a 74.4% phoneme accuracy. Isolated word recognition experiments in both normalhearing and deaf subjects were also conducted providing a word accuracy of 94.9% and 89%, respectively. The obtained results were compared with those obtained using audio signal, and comparable accuracies were observed.

Inner lips feature extraction based on CLNF with hybrid dynamic template for Cued Speech

“…61.8% accuracy on average is obtained for the recognition of the 13 French vowels in CS. This result is comparable with the state of the art [12] in automatic CS recognition. It is another validation of our method for automatic tracking of inner lips parameters.…”

“…In the previous study of CS lips feature extraction, Heracleous et al [12] and Aboutabit et al [13] extracted the A and B parameters by painting blue color on the subject's lips. Firstly, the gray level image was subtracted from the blue component of the RGB image.…”

Interaction between articulatory gestures and inner speech in a counting task

“…4). A ChromaKey system, which transforms blue areas into pure black, automatically detects the corresponding zones, which allows to quantitatively analyze the movements of the lips, mandible (Lallouache, 1990) or hand/finger (Heracleous et al, 2010).…”

Re-Synchronization Using the Hand Preceding Model for Multi-Modal Fusion in Automatic Continuous Cued Speech Recognition

“…In the literature on the automatic CS recognition, video images were recorded with artifices applied to the CS speaker (blue sticks on the lips, blue marks on the hand and forehead) to mark the pertinent information and make their further feature extraction easier. For example, in [17], [18], lips feature was extracted by tracking the color marks on speaker's lips. Then a threshold was applied to the gray level images to segment the blue lips.…”

“…In the existing literature, CS feature modalities are assumed to be synchronous in the recognition task. In [17], [18], direct feature fusion (i.e., direct concatenation of the features) was applied to the isolated 1 CS recognition without taking into account the asynchrony issue. In the state-of-theart [19], a tandem architecture that combines convolutional neural network (CNN) [20], [21] with multi-stream hidden markov model (MSHMM) [22] was used for the continuous CS recognition, and is referred as S 3 in this work.…”