Multi-modality helps in crisis management: An attention-based deep learning approach of leveraging text for image classification

Ahmad, Zishan; Jindal, Raghav; N.S., Mukuntha; Ekbal, Asif; Bhattachharyya, Pushpak

doi:10.1016/j.eswa.2022.116626

Cited by 11 publications

(2 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…An efficient DL algorithm was presented in [21] to manipulate multimodal information sources (words and photos) and disseminate helpful information during natural disasters. The programme divided the tweets into seven crucial and actionable groups, including reports of "hurt or dead individuals" and "infrastructure damage".…”

Section: Related Workmentioning

confidence: 99%

“…Research by [32][33][34][35][36][37] similarly highlighted the multimodal model utilised in various textual, image and video datasets for different domains. Likewise, [15,17,[20][21][22] employed multimodality to develop DL-based models in facilitating disaster management and recovery. Table I depicts the comparison of different classification algorithms and their effects on each model.…”

Section: Fig 3 the Experiments Frameworkmentioning

confidence: 99%

See 1 more Smart Citation

Social Media Multimodal Information Analysis based on the BiLSTM-Attention-CNN-XGBoost Ensemble Neural Network

Jixian¹,

An²,

Su³

et al. 2022

IJACSA

View full text Add to dashboard Cite

Social media users internalise information in a multimodal context. Social media functions as a primary information source for disaster situational awareness encompassing texts, photographs, videos, and other multimodal information widely used in emergency management. Applying ensemble learning to social media sentiment analysis has garnered much scholarly attention, albeit with limited research on rescue and its sub-domain, which is characterised as a major complexity. A multimodal information categorisation model based on hierarchical feature extraction was proposed in this study. The information of multiple modes is first mapped to a unified text vector space in modelling the semantic content at the sentence and multimodal information levels in the multimodal information. Multiple deep learning (DL) models were subsequently applied to model the semantic content at the aforementioned levels. This study offers a BiLSTM-Attention-CNN-XGBOOST ensemble neural network model to acquire extensive multimodal information characteristics. Based on the empirical outcomes, this method precisely extracted multimodal information features with an accuracy exceeding 85% and 95% for Chinese-and English-language datasets, respectively.

show abstract

Section: Related Workmentioning

confidence: 99%