Multimodal speech recognition of a person with articulation disorders using AAM and MAF

Abstract

We investigated the speech recognition of a person with articulation disorders resulting from athetoid cerebral palsy. The articulation of speech tends to become unstable due to strain on speech-related muscles, and that causes degradation of speech recognition. Therefore, we use multiple acoustic frames (MAF) as an acoustic feature to solve this problem. Further, in a real environment, current speech recognition systems do not have sufficient performance due to noise influence. In addition to acoustic features, visual features are used to increase noise robustness in a real environment. However, there are recognition problems resulting from the tendency of those suffering from cerebral palsy to move their head erratically. We investigate a pose-robust audio-visual speech recognition method using an Active Appearance Model (AAM) to solve this problem for people with articulation disorders resulting from athetoid cerebral palsy. AAMs are used for face tracking to extract pose-robust facial feature points. Its effectiveness is confirmed by word recognition experiments on noisy speech of a person with articulation disorders.

Audio-Visual Speech Recognition Using Convolutive Bottleneck Networks for a Person with Severe Hearing Loss

“…In Ref. [10], we used multiple acoustic frames (MAF) as an acoustic dynamic feature to improve the recognition rate of a person with an articulation disorder, especially in speech recognition using dynamic features only.…”

Data collection for mobile audio-visual speech recognition in various environments

“…This application can be launched on Android ™ cell phones and tablet computers. Here, model training and recognition methods of the recog nizer which accepts connected-digit utterances are described: In the visual modality, several features have been proposed; for example, discrete-cosine-transform results and optical flow-based parameters [3] as pixel-based features, alternatively, Active Appearance Model (AAM) parameters [5] as model based features. These features have been investigated and com pared in [13].…”

“…Face detection, that is realized as a hardware function of the camera, is conducted in each picture to obtain a face region. A monochrome captured image with face detection results are then stored into a visual Name/Issuer AT R [1] Tokyo Tech [3] Tokyo Tech [16] M2TINIT [9] Tokyo Tech [17] Tokyo Tech [18] Kobe Univ [5] CENSREC-I-AV [15] CENSREC-2-AV [19] frame. ii.…”

“…Most multi-modal speech recognition schemes employ visual information: face, mouth or lip images. Audio-Visual ASR (AVASR) has been investigated by many researchers [1], [2], [3], [4], [5], [6]. To day many mobile computers have not only microphone but also embedded camera to capture ones' user; such equipments are often used for video communication applications.…”

A preliminary demonstration of exemplar-based voice conversion for articulation disorders using an individuality-preserving dictionary

“…In [9], we proposed robust feature extraction based on principal component analysis (PCA) with more stable utterance data instead of DCT. In [10], we used multiple acoustic frames (MAF) as an acoustic dynamic feature to improve the recognition rate of a person with an articulation disorder, especially in speech recognition using dynamic features only. In spite of these efforts, the recognition rate for articulation disorders is still lower than that of physically unimpaired persons.…”