Re-Synchronization Using the Hand Preceding Model for Multi-Modal Fusion in Automatic Continuous Cued Speech Recognition

Abstract

Cued Speech (CS) is an augmented lip reading system complemented by hand coding, and it is very helpful to the deaf people. Automatic CS recognition can help communications between the deaf people and others. Due to the asynchronous nature of lips and hand movements, fusion of them in automatic CS recognition is a challenging problem. In this work, we propose a novel re-synchronization procedure for multi-modal fusion, which aligns the hand features with lips feature. It is realized by delaying hand position and hand shape with their optimal hand preceding time which is derived by investigating the temporal organizations of hand position and hand shape movements in CS. This re-synchronization procedure is incorporated into a practical continuous CS recognition system that combines convolutional neural network (CNN) with multi-stream hidden markov model (MSHMM). A significant improvement of about 4.6% has been achieved retaining 76.6% CS phoneme recognition correctness compared with the state-of-the-art architecture (72.04%), which did not take into account the asynchrony issue of multi-modal fusion in CS. To our knowledge, this is the first work to tackle the asynchronous multi-modal fusion in the automatic continuous CS recognition.

Exponential Multi-Modal Discriminant Feature Fusion for Small Sample Size

“…As compared to single-modal data, multi-modal data contains more information on a single target. Multi-modal fusion is divided into three different fusion levels [2]: pixel-level fusion, feature-level fusion, and decision-level fusion. In the three-level fusions, pixel-level fusion [3] is located at the lowest level, which can retain the original information to the greatest extent.…”

Multistream neural architectures for cued-speech recognition using a pre-trained visual feature extractor and constrained CTC decoding

“…A feature extractor based on a multi-stream convolutional neural network (CNN) processing the raw regions of interest of hand and lips was combined with an HMM-GMM phonetic decoder. One of the major challenges in continuous ACSR is to deal with the asynchrony between hand and lip [7], i.e., the configuration of the hand pertaining to a certain phoneme can precede (or follow) that of the lips for the same phoneme by a variable delay ranging from a few milliseconds to several hundred milliseconds [8]. In [6], this issue has been addressed with a simple heuristic by considering the hand configuration observed at the beginning of the previous phoneme.…”

Backdoor Attack Against Speaker Verification

“…Acoustics signal processing [1,2,3], especially speaker verification [4,5,6], has been widely and successfully adopted in our daily life. Speaker verification aims at determining whether a given utterance belongs to a specific speaker.…”