Multi-attention Recurrent Network for Human Communication Comprehension

Zadeh, Amir; Liang, Paul Pu; Poria, Soujanya; Vij, Prateek; Wang, Zhaoxia; Morency, Louis–Philippe

doi:10.48550/arxiv.1802.00923

Cited by 7 publications

(9 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Data is partitioned into predetermined train (12787 points), validation (3634 points) and test (1438 points) splits. Raw features for CMU-MOSI and CMU-MOSEI are obtained from CMU-Multimodal SDK [18] .…”

Section: Test Data Setsmentioning

confidence: 99%

Multi-Modal Sentiment Analysis Using Deep Canonical Correlation Analysis

Sun¹,

Sarma²,

Sethares³

et al. 2019

Interspeech 2019

View full text Add to dashboard Cite

This paper learns multi-modal embeddings from text, audio, and video views/modes of data in order to improve upon downstream sentiment classification. The experimental framework also allows investigation of the relative contributions of the individual views in the final multi-modal embedding. Individual features derived from the three views are combined into a multi-modal embedding using Deep Canonical Correlation Analysis (DCCA) in two ways i) One-Step DCCA and ii) Two-Step DCCA. This paper learns text embeddings using BERT, the current state-of-the-art in text encoders. We posit that this highly optimized algorithm dominates over the contribution of other views, though each view does contribute to the final result. Classification tasks are carried out on two benchmark data sets and on a new Debate Emotion data set, and together these demonstrate that the one-Step DCCA outperforms the current state-of-the-art in learning multi-modal embeddings.

show abstract

Section: Test Data Setsmentioning

confidence: 99%

Multi-Modal Sentiment Analysis Using Deep Canonical Correlation Analysis

Sun¹,

Sarma²,

Sethares³

et al. 2019

Interspeech 2019

View full text Add to dashboard Cite

show abstract

“…Details of each type will be introduced in the following sections. 1) Extracted feature from audio clips (Ex0, Ex04): The spoken system of modern Chinese is named 'Hanyu Pinyin', abbreviated to 'pinyin' 3 . It is the official romanization system for mandarin in mainland China [51].…”

Section: Learning Phonetic Featuresmentioning

confidence: 99%

“…In recent years, sentiment analysis has become increasingly popular for processing social media data on online communities, blogs, wikis, microblogging platforms, and other online collaborative media [1]. Sentiment analysis is a branch of affective computing research [2], [3] that aims to classify text -but sometimes also audio and video [4], [5] -into either positive or negative -but sometimes also neutral [6]. Sentiment analysis techniques can be broadly categorized into symbolic and sub-symbolic approaches: the former include the use of lexicons [7], ontologies [8], and semantic networks [9] to encode the polarity associated with words and multiword expressions; the latter consist of supervised [10], semi-supervised [11] and unsupervised [12] machine learning techniques that perform sentiment classification based on word co-occurrence frequencies.…”

Section: Introductionmentioning

confidence: 99%

Phonetic-enriched Text Representation for Chinese Sentiment Analysis with Reinforcement Learning

Peng¹,

Ma²,

Poria

et al. 2019

Preprint

Self Cite

View full text Add to dashboard Cite

“…The dataset was composed of over 23,500 spoken sentence videos, totaling 65 hours, 53 minutes, and 36 seconds. The dataset had been segmented at the sentence level; the sentences had been transcribed, and audio, visual, and textual features had been generated and released as part a public Zadeh et al (2018b) software development kit (SDK). Additionally, raw videos were available for download.…”

Section: Moseimentioning

confidence: 99%

“…This large quantity of data comes from real-world expressions of sentiment, offering a unique opportunity to train and test model performance and generalization on a large dataset. Additionally, Zadeh et al (2018b) released a software development kit (SDK) for training and testing models on the CMU-MOSEI dataset, with future work focusing on addition of other multimodal datasets. These releases culminated in a challenge focused on human multimodal language with the opportunity to train a model and evaluate it on a held-out challenge test set.…”

Section: Introductionmentioning

confidence: 99%

Getting the subtext without the text: Scalable multimodal sentiment classification from visual and acoustic modalities

Blanchard¹,

Moreira²,

Bharati³

et al. 2018

Proceedings of Grand Challenge and Workshop on Human Multimodal Language (Challenge-Hml)

View full text Add to dashboard Cite

In the last decade, video blogs (vlogs) have become an extremely popular method through which people express sentiment. The ubiquitousness of these videos has increased the importance of multimodal fusion models, which incorporate video and audio features with traditional text features for automatic sentiment detection. Multimodal fusion offers a unique opportunity to build models that learn from the full depth of expression available to human viewers. In the detection of sentiment in these videos, acoustic and video features provide clarity to otherwise ambiguous transcripts. In this paper, we present a multimodal fusion model that exclusively uses high-level video and audio features to analyze spoken sentences for sentiment. We discard traditional transcription features in order to minimize human intervention and to maximize the deployability of our model on at-scale real-world data. We select high-level features for our model that have been successful in nonaffect domains in order to test their generalizability in the sentiment detection domain. We train and test our model on the newly released CMU Multimodal Opinion Sentiment and Emotion Intensity (CMU-MOSEI) dataset, obtaining an F 1 score of 0.8049 on the validation set and an F 1 score of 0.6325 on the held-out challenge test set. Figure 1: A blindspot in multimodal sentiment analysis is the inclusion of human-transcriptions of spoken sentiment, which limits model applicability. We address this by using only prosodic and visual features for sentiment classification.

show abstract

Multi-attention Recurrent Network for Human Communication Comprehension

Cited by 7 publications

References 0 publications

Multi-Modal Sentiment Analysis Using Deep Canonical Correlation Analysis

Multi-Modal Sentiment Analysis Using Deep Canonical Correlation Analysis

Phonetic-enriched Text Representation for Chinese Sentiment Analysis with Reinforcement Learning

Getting the subtext without the text: Scalable multimodal sentiment classification from visual and acoustic modalities

Contact Info

Product

Resources

About