Different Contextual Window Sizes Based RNNs for Multimodal Emotion Detection in Interactive Conversations

Lai, Helang; Chen, Hongying; Wu, Shuangyan

doi:10.1109/access.2020.3005664

Cited by 10 publications

(3 citation statements)

References 50 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Neural networks are used to filter this invalid data from videos and physiological signals by using continuity and semantic compatibility. H. Lai et al [53] proposed a model for emotion recognition in interactive conversations. In this model, different RNN architectures with variable contextual window sizes differentiate various aspects of contexts in conversations to improve the accuracy.…”

Section: Multimodal Emotion Recognition (Mmer)mentioning

confidence: 99%

A Review on Methods and Applications in Multimodal Deep Learning

Summaira¹,

Li²,

Shoib³

et al. 2022

Preprint

View full text Add to dashboard Cite

Deep Learning has implemented a wide range of applications and has become increasingly popular in recent years. The goal of multimodal deep learning (MMDL) is to create models that can process and link information using various modalities. Despite the extensive development made for unimodal learning, it still cannot cover all the aspects of human learning. Multimodal learning helps to understand and analyze better when various senses are engaged in the processing of information. This paper focuses on multiple types of modalities, i.e., image, video, text, audio, body gestures, facial expressions, and physiological signals. Detailed analysis of the baseline approaches and an in-depth study of recent advancements during the last five years (2017 to 2021) in multimodal deep learning applications has been provided. A fine-grained taxonomy of various multimodal deep learning methods is proposed, elaborating on different applications in more depth. Lastly, main issues are highlighted separately for each domain, along with their possible future research directions.CCS Concepts: • Computing methodologies → Machine learning; • Information systems → Multimedia and multimodal retrieval.

show abstract

Section: Multimodal Emotion Recognition (Mmer)mentioning

confidence: 99%

A Review on Methods and Applications in Multimodal Deep Learning

Summaira¹,

Li²,

Shoib³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Neural networks are used to filter this invalid data from videos and physiological signals by using continuity and semantic compatibility. H. Lai et al [75] proposed a model for emotion recognition in interactive conversations. In this model, different RNN architectures with variable contextual window sizes differentiate various aspects of contexts in conversations to improve the accuracy.…”

Section: Other MMDL Applicationsmentioning

confidence: 99%

Recent Advances and Trends in Multimodal Deep Learning: A Review

Summaira¹,

Li²,

Shoib³

et al. 2021

Preprint

View full text Add to dashboard Cite

Deep Learning has implemented a wide range of applications and has become increasingly popular in recent years. The goal of multimodal deep learning is to create models that can process and link information using various modalities. Despite the extensive development made for unimodal learning, it still cannot cover all the aspects of human learning. Multimodal learning helps to understand and analyze better when various senses are engaged in the processing of information. This paper focuses on multiple types of modalities, i.e., image, video, text, audio, body gestures, facial expressions, and physiological signals. Detailed analysis of past and current baseline approaches and an in-depth study of recent advancements in multimodal deep learning applications has been provided. A fine-grained taxonomy of various multimodal deep learning applications is proposed, elaborating on different applications in more depth. Architectures and datasets used in these applications are also discussed, along with their evaluation metrics. Last, main issues are highlighted separately for each domain along with their possible future research directions.CCS Concepts: • Computing methodologies → Machine learning; • Information systems → Multimedia and multimodal retrieval.

show abstract

“…The next generation of revolutionary HCI technology may not impact the whole industry, such as the emergence of graphical interfaces and touch technology, but may use data-driven intelligence to realize the potential revolution of HCI [8]. The vigorous development of artificial intelligence has greatly promoted the intelligence of machines, and the in-depth study of the interaction between humans and machines has promoted new gesture interaction technology and automatic speech recognition technology [9][10][11].…”

Section: Introductionmentioning

confidence: 99%

Deep Learning for Intelligent Human–Computer Interaction

Poiesi

Dong

et al. 2022

Applied Sciences

View full text Add to dashboard Cite

In recent years, gesture recognition and speech recognition, as important input methods in Human–Computer Interaction (HCI), have been widely used in the field of virtual reality. In particular, with the rapid development of deep learning, artificial intelligence, and other computer technologies, gesture recognition and speech recognition have achieved breakthrough research progress. The search platform used in this work is mainly the Google Academic and literature database Web of Science. According to the keywords related to HCI and deep learning, such as “intelligent HCI”, “speech recognition”, “gesture recognition”, and “natural language processing”, nearly 1000 studies were selected. Then, nearly 500 studies of research methods were selected and 100 studies were finally selected as the research content of this work after five years (2019–2022) of year screening. First, the current situation of the HCI intelligent system is analyzed, the realization of gesture interaction and voice interaction in HCI is summarized, and the advantages brought by deep learning are selected for research. Then, the core concepts of gesture interaction are introduced and the progress of gesture recognition and speech recognition interaction is analyzed. Furthermore, the representative applications of gesture recognition and speech recognition interaction are described. Finally, the current HCI in the direction of natural language processing is investigated. The results show that the combination of intelligent HCI and deep learning is deeply applied in gesture recognition, speech recognition, emotion recognition, and intelligent robot direction. A wide variety of recognition methods were proposed in related research fields and verified by experiments. Compared with interactive methods without deep learning, high recognition accuracy was achieved. In Human–Machine Interfaces (HMIs) with voice support, context plays an important role in improving user interfaces. Whether it is voice search, mobile communication, or children’s speech recognition, HCI combined with deep learning can maintain better robustness. The combination of convolutional neural networks and long short-term memory networks can greatly improve the accuracy and precision of action recognition. Therefore, in the future, the application field of HCI will involve more industries and greater prospects are expected.

show abstract

Different Contextual Window Sizes Based RNNs for Multimodal Emotion Detection in Interactive Conversations

Cited by 10 publications

References 50 publications

A Review on Methods and Applications in Multimodal Deep Learning

A Review on Methods and Applications in Multimodal Deep Learning

Recent Advances and Trends in Multimodal Deep Learning: A Review

Deep Learning for Intelligent Human–Computer Interaction

Contact Info

Product

Resources

About