Multimodal Fusion and Sequence Learning for Cued Speech Recognition from Videos

Papadimitriou, Katerina; Parelli, Maria; Sapountzaki, Galini; Pavlakos, Georgios; Maragos, Petros; Potamianos, Gerasimos

doi:10.1007/978-3-030-78095-1_21

Cited by 2 publications

(3 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The technique is the best suited for lipreading applications and attained more than 80% in CS recognition [25]. Deep learning was also utilized to analyze hand shape and lip features using CNN models for CS recognition from videos [26]. A Malay CSR system was introduced to help the deaf learn and practice the basics of cued speech consonants and vowels using hand gestures [27].…”

Section: Multimodal Integration For Hard Of Hearingmentioning

confidence: 99%

Multimodal Based Audio-Visual Speech Recognition for Hard-of-Hearing: State of the Art Techniques and Challenges

Bhaskar

Madathil

2022

IJEEI

View full text Add to dashboard Cite

Multimodal Integration (MI) is the study of merging the knowledge acquired by the nervous system using sensory modalities such as speech, vision, touch, and gesture. The applications of MI expand over the areas of Audio-Visual Speech Recognition (AVSR), Sign Language Recognition (SLR), Emotion Recognition (ER), Bio Metrics Applications (BMA), Affect Recognition (AR), Multimedia Retrieval (MR), etc. The fusion of modalities such as hand gestures-facial, lip-hand position, etc., are mainly used as sensory modalities to develop hard-of-hearing multimodal systems. This paper encapsulates an overview of multimodal systems available within the literature for hard-ofhearing studies. This paper also discusses some of the studies related to hardof-hearing acoustic analysis. It is observed that very few algorithms have been developed for hard-of-hearing AVSR than normal hearing. Thus, the study of audio-visual-based speech recognition systems for hard-of-hearing is highly demanded by people trying to communicate with natively speaking languages. This paper also highlights the state-of-the-art techniques in AVSR and the challenges faced by the researchers in the development of AVSR systems.

show abstract

Section: Multimodal Integration For Hard Of Hearingmentioning

confidence: 99%

Multimodal Based Audio-Visual Speech Recognition for Hard-of-Hearing: State of the Art Techniques and Challenges

Bhaskar

Madathil

2022

IJEEI

View full text Add to dashboard Cite

show abstract

“…Text2sign [21] HLSTM [22] ESN [23] DiffGAN [24] RG [25] SEEG [26] HA2G [27] Paul Duchnowski [28] Gérard Bailly [29] Wav2lip [30] Audio2head [31] AD-NERF [32] DiffTalk [33] Rule-based [34] LBP [35] SDF [36] PTSLP [37] Syn [38] MMFSL [39] Re-Syn [40] CMML [41] DTW [42] HMMs [43] FCN [44] RL [45] CS [1] MCCS [46] SLreview [2] UnorgSign [47] CoSreview [6] CoSreview [3] SE [48] THreview [4] THE [49] VHTHG [50] Fig. 3: Structured taxonomy of the existing BL research which includes three genres.…”

Section: Sign Language Recognitionmentioning

confidence: 99%

“…More recently, traditional DL-based methods (i.e., CNNbased, LSTM-based) have been developed to alleviate the aforementioned problem. For instance, Sankar et al [39] propose a novel RNN model trained with a Connectionist Temporal Classification (CTC) loss [225]. Papatimitriou et al [170] propose a fully convolutional model with a time-depth separable block and attention-based decoder.…”

mentioning

confidence: 99%

Multi-Modal Fusion Emotion Recognition Method of Speech Expression Based on Deep Learning

Liu

Wang

et al. 2021

Front. Neurorobot.

View full text Add to dashboard Cite

The redundant information, noise data generated in the process of single-modal feature extraction, and traditional learning algorithms are difficult to obtain ideal recognition performance. A multi-modal fusion emotion recognition method for speech expressions based on deep learning is proposed. Firstly, the corresponding feature extraction methods are set up for different single modalities. Among them, the voice uses the convolutional neural network-long and short term memory (CNN-LSTM) network, and the facial expression in the video uses the Inception-Res Net-v2 network to extract the feature data. Then, long and short term memory (LSTM) is used to capture the correlation between different modalities and within the modalities. After the feature selection process of the chi-square test, the single modalities are spliced to obtain a unified fusion feature. Finally, the fusion data features output by LSTM are used as the input of the classifier LIBSVM to realize the final emotion recognition. The experimental results show that the recognition accuracy of the proposed method on the MOSI and MELD datasets are 87.56 and 90.06%, respectively, which are better than other comparison methods. It has laid a certain theoretical foundation for the application of multimodal fusion in emotion recognition.

show abstract

Multimodal Fusion and Sequence Learning for Cued Speech Recognition from Videos

Cited by 2 publications

References 25 publications

Multimodal Based Audio-Visual Speech Recognition for Hard-of-Hearing: State of the Art Techniques and Challenges

Multimodal Based Audio-Visual Speech Recognition for Hard-of-Hearing: State of the Art Techniques and Challenges

Multi-Modal Fusion Emotion Recognition Method of Speech Expression Based on Deep Learning

Contact Info

Product

Resources

About