Visual and Textual Sentiment Analysis of a Microblog Using Deep Convolutional Neural Networks

Yu, Yuhai; Lin, Hongfei; Meng, Junmin; Zhao, Zhehuan

doi:10.3390/a9020041

Cited by 114 publications

(52 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Inspired by the work of [28,32,33], we train five networks with the same structure for each of three textual models and combine them separately (TCNN5, TLSTM5, and TGRU5, see Table 1) to improve performance further. For example, among three types of textual models, the textual convolutional neural network (TCNN5) obtains the performance of 82.30% in ImageCLEF2015 and 89.81% in ImageCLEF2016, which are both higher than textual baseline of 78.34% and 88.13% (Baseline_Text).…”

Section: Textual Resultsmentioning

confidence: 99%

“…After training the networks with 10-fold cross validation (10FCV) on the training set, we test our trained models on the test set. Many codes have been modified from our previous work [28,32,33] and are implemented with the neural network library of Keras, running on top of TensorFlow. All default parameters are used, except for those parameters mentioned in Section 2.…”

Section: Methodsmentioning

confidence: 99%

“…The model of TCNNs, shown in Figure 2, is similar to the CNN architecture [19,28] with an embedding layer with a dropout of 0.25, a convolutional layer with a kernel size of 3 and 250 feature maps, a max-over-time pooling layer with a max-pool size of 2, a vanilla layer with 250 neurons, a dropout of 0.25, and a ReLU (Rectified Linear Unit) activation function, and a full-connected layer with 2 neurons. The softmax function is implemented at the final layer to output the prediction probabilities of two classes.…”

Section: Textual Convolutional Neural Networkmentioning

confidence: 99%

“…The model of VCNNs is a deep CNN similar to [28,32,33]. The first two convolutional layers contain 32 kernels of size 3 × 3, and the second two convolutional layers have 32 kernels of size 3 × 3.…”

Section: Visual Convolutional Neural Networkmentioning

confidence: 99%

See 3 more Smart Citations

Assembling Deep Neural Networks for Medical Compound Figure Detection

Lin

Meng

et al. 2017

Information

Self Cite

View full text Add to dashboard Cite

Abstract:Compound figure detection on figures and associated captions is the first step to making medical figures from biomedical literature available for further analysis. The performance of traditional methods is limited to the choice of hand-engineering features and prior domain knowledge. We train multiple convolutional neural networks (CNNs), long short-term memory (LSTM) networks, and gated recurrent unit (GRU) networks on top of pre-trained word vectors to learn textual features from captions and employ deep CNNs to learn visual features from figures. We then identify compound figures by combining textual and visual prediction. Our proposed architecture obtains remarkable performance in three run types-textual, visual and mixed-and achieves better performance in ImageCLEF2015 and ImageCLEF2016.

show abstract

Section: Textual Resultsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Section: Textual Convolutional Neural Networkmentioning

confidence: 99%

Section: Visual Convolutional Neural Networkmentioning

confidence: 99%

See 2 more Smart Citations

Assembling Deep Neural Networks for Medical Compound Figure Detection

Lin

Meng

et al. 2017

Information

Self Cite

View full text Add to dashboard Cite

show abstract

“…This CNN has only six weight layers similar to [22,35,36]. The first two convolutional layers contain 32 kernels of size 3 × 3, and the second two convolutional layers have 64 kernels of size 3 × 3.…”

Section: Convolutional Neural Networkmentioning

confidence: 99%

Deep Transfer Learning for Modality Classification of Medical Images

et al. 2017

Self Cite

View full text Add to dashboard Cite

Medical images are valuable for clinical diagnosis and decision making. Image modality is an important primary step, as it is capable of aiding clinicians to access required medical image in retrieval systems. Traditional methods of modality classification are dependent on the choice of hand-crafted features and demand a clear awareness of prior domain knowledge. The feature learning approach may detect efficiently visual characteristics of different modalities, but it is limited to the number of training datasets. To overcome the absence of labeled data, on the one hand, we take deep convolutional neural networks (VGGNet, ResNet) with different depths pre-trained on ImageNet, fix most of the earlier layers to reserve generic features of natural images, and only train their higher-level portion on ImageCLEF to learn domain-specific features of medical figures. Then, we train from scratch deep CNNs with only six weight layers to capture more domain-specific features. On the other hand, we employ two data augmentation methods to help CNNs to give the full scope to their potential characterizing image modality features. The final prediction is given by our voting system based on the outputs of three CNNs. After evaluating our proposed model on the subfigure classification task in ImageCLEF2015 and ImageCLEF2016, we obtain new, state-of-the-art results-76.87% in ImageCLEF2015 and 87.37% in ImageCLEF2016-which imply that CNNs, based on our proposed transfer learning methods and data augmentation skills, can identify more efficiently modalities of medical images.

show abstract

Various syncretic co‐attention network for multimodal sentiment analysis

Cao

Zhu

Gao

et al. 2020

Concurrency and Computation

View full text Add to dashboard Cite

The multimedia contents shared on social network reveal public sentimental attitudes toward specific events. Therefore, it is necessary to conduct sentiment analysis automatically on abundant multimedia data posted by the public for real-world applications. However, approaches to single-modal sentiment analysis neglect the internal connections between textual and visual contents, and current multimodal methods fail to exploit the multilevel semantic relations of heterogeneous features. In this article, the various syncretic co-attention network is proposed to excavate the intricate multilevel corresponding relations between multimodal data, and combine the unique information of each modality for integrated complementary sentiment classification. Specifically, a multilevel co-attention module is constructed to explore localized correspondences between each image region and each text word, and holistic correspondences between global visual information and context-based textual semantics. Then, all the single-modal features can be fused from different levels, respectively. Except for fused multimodal features, our proposed VSCN also considers unique information of each modality simultaneously and integrates them into an end-to-end framework for sentiment analysis. The superior results of experiments on three constructed real-world datasets and a benchmark dataset of Visual Sentiment Ontology (VSO) prove the effectiveness of our proposed VSCN. Especially qualitative analyses are given for deep explaining of our method.

show abstract

Visual and Textual Sentiment Analysis of a Microblog Using Deep Convolutional Neural Networks

Cited by 114 publications

References 21 publications

Assembling Deep Neural Networks for Medical Compound Figure Detection

Assembling Deep Neural Networks for Medical Compound Figure Detection

Deep Transfer Learning for Modality Classification of Medical Images

Various syncretic co‐attention network for multimodal sentiment analysis

Contact Info

Product

Resources

About