A Co-Memory Network for Multimodal Sentiment Analysis

Fusion-Extraction Network for Multimodal Sentiment Analysis

“…However, they only consider the visual information for textual representation, and ignore the mutual promotion of text and image. Thus, Xu et al [23] propose a co-memory attentional mechanism to interactively model the interaction between text and image. Though taking the mutual influence of text and image into consideration, Xu et al [23] adopt a coarsegrained attention mechanism which may not have enough capacity to extract sufficient information.…”

“…Thus, Xu et al [23] propose a co-memory attentional mechanism to interactively model the interaction between text and image. Though taking the mutual influence of text and image into consideration, Xu et al [23] adopt a coarsegrained attention mechanism which may not have enough capacity to extract sufficient information. Furthermore, they simply concatenate the visual representation and the textual representation for final sentiment classification.…”

“…The above encoding representation only considers their single modality, and the attention mechanism is often applied to capture the interactions between different modality representations. However, previous works [22,23] adopt coarsegrained attention which may cause information loss, as the text contains multiple words and the image presentation contains multiple feature maps. In contrast, as shown in the middle part of Fig.…”

“…The later has 19600 text-image pairs labeled by three annotators. For fair comparison, we process the original two MVSA datasets on the same way used in [22,23]. We randomly split the datasets into training set, validation set and test set by using the split ratio 8:1:1.…”

“…Xu et al [22] only take the single-direction influence of image to text into consideration and ignore interactive promotion between visual and textual information. A co-memory network [23] then is proposed to model the interactions between visual contents and textual words iteratively. Nevertheless, the co-memory network only applies a weighted textual/visual vector as the guide to learn attention weights on visual/textual representation.…”