A computationally efficient multi-modal classification approach of disaster-related Twitter images

Rizk, Yara; Jomaa, Hadi S.; Awad, Mariette; Castillo, Carlos

doi:10.1145/3297280.3297481

Cited by 23 publications

(11 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A few recent works by Alam et al (2017) and Nguyen et al (2017b) used visual features only in finding informative images in case of disaster. Rizk et al (2019) and Mouzannar et al (2018) used both textual and visual features related to build-infrastructure damage, nature damage, and fire for estimating the damage due to disaster. To the best of our knowledge, no work has been reported where tweet text and images are used together to filter informative tweets from massive social media contents.…”

Section: Introductionmentioning

confidence: 99%

“…They used Nepal Earthquake, Ecuador Earthquake, Hurricane Matthew, Typhoon Ruby, and Google Images datasets and trained event-specific as well as cross-event classifier. Their CNN model outperformed Bag-of-Visual-Words (BoVW) techniques and achieved the F1 scores in the range of 0.67 to 0.89.Recently, researchers have proposed multi-modal systems utilizing the tweet text and images both for finding relevant information from social media Rizk et al (2019). developed a multi-modal disaster-related classifier to classify Twitter data into the builtinfrastructure damage and nature damage classes.…”

mentioning

confidence: 99%

See 1 more Smart Citation

A deep multi-modal neural network for informative Twitter content classification during emergencies

Kumar

Singh

Dwivedi³

et al. 2020

Ann Oper Res

View full text Add to dashboard Cite

People start posting tweets containing texts, images, and videos as soon as a disaster hits an area. The analysis of these disaster-related tweet texts, images, and videos can help humanitarian response organizations in better decision-making and prioritizing their tasks. Finding the informative contents which can help in decision making out of the massive volume of Twitter content is a difficult task and require a system to filter out the informative contents. In this paper, we present a multi-modal approach to identify disaster-related informative content from the Twitter streams using text and images together. Our approach is based on long-short-term-memory (LSTM) and VGG-16 networks that show significant improvement in the performance, as evident from the validation result on seven different disaster-related datasets. The range of F1-score varied from 0.74 to 0.93 when tweet texts and images used together, whereas, in the case of only tweet text, it varies from 0.61 to 0.92. From this result, it is evident that the proposed multi-modal system is performing significantly well in identifying disaster-related informative social media contents.

show abstract

Section: Introductionmentioning

confidence: 99%

mentioning

confidence: 99%

A deep multi-modal neural network for informative Twitter content classification during emergencies

Kumar

Singh

Dwivedi³

et al. 2020

Ann Oper Res

View full text Add to dashboard Cite

show abstract

“…They used the Inception pre-trained model for visual feature extraction and designed a CNN architecture for textual features. Similarly, Rizk et al [35] proposed a multimodal architecture to classify the Twitter data into infrastructure and natural damage categories. Ofli et al [8] also presented a multimodal approach for classifying the tweets into two categories: informative task (e.g., informative vs. non-informative) and humanitarian task (e.g., affected individuals, rescue volunteering or donation effort, infrastructure and utility damage).…”

Section: B Multimodal Approachesmentioning

confidence: 99%

A Deep Attentive Multimodal Learning Approach for Disaster Identification From Social Media Posts

Hossain

Hoque

et al. 2022

IEEE Access

View full text Add to dashboard Cite

Microblogging platforms such as Twitter have become indispensable for disseminating valuable information, especially at times of natural and man-made disasters. Often people post multimedia contents with images and/or videos to report important information such as casualties, damages of infrastructure, and urgent needs of affected people. Such information can be very helpful for humanitarian organizations for planning adequate response in a time-critical manner. However, identifying disaster information from a vast amount of posts is an arduous task, which calls for an automatic system that can filter out the actionable and non-actionable disaster-related information from social media. While many studies have shown the effectiveness of combining text and image contents for disaster identification, most previous work focused on analyzing only the textual modality and/or applied traditional recurrent neural network (RNN) or convolutional neural network (CNN) which might lead to performance degradation in case of long input sequences. This paper presents a multimodal disaster identification system that utilizes both visual and textual data in a synergistic way by conjoining the influential word features with the visual features to classify tweets. Specifically, we utilize a pretrained convolutional neural network (e.g., ResNet50) to extract visual features and a bidirectional long-term memory (BiLSTM) network with attention mechanism to extract textual features. We then aggregate both visual and textual features by leveraging a feature fusion approach followed by applying the softmax classifier. The evaluations demonstrate that the proposed multimodal system enhances the performance over the existing baselines including both unimodal and multimodal models by attaining approximately 1% and 7% of performance improvement, respectively.

show abstract

“…On the imagery content, they achieved an F1 score of 87.74% using XGboost [14]. The study in [56] propose a simple, computationally inexpensive, multi-modal two-stage framework to classify tweets (text and image) with built-infrastructure damage vs. nature-damage. The study investigated their approach using a home-grown dataset and the SUN dataset [71].…”

Section: Multimodality (Image and Text)mentioning

confidence: 99%

Robust Training of Social Media Image Classification Models for Rapid Disaster Response

Alam,

Imran

et al. 2021

Preprint

View full text Add to dashboard Cite

Images shared on social media help crisis managers in terms of gaining situational awareness and assessing incurred damages, among other response tasks. As the volume and velocity of such content are really high, therefore, real-time image classification became an urgent need in order to take a faster response. Recent advances in computer vision and deep neural networks have enabled the development of models for real-time image classification for a number of tasks, including detecting crisis incidents, filtering irrelevant images, classifying images into specific humanitarian categories, and assessing the severity of the damage. For developing real-time robust models, it is necessary to understand the capability of the publicly available pretrained models for these tasks. In the current state-of-art of crisis informatics, it is under-explored. In this study, we address such limitations. We investigate ten different architectures for four different tasks using the largest publicly available datasets for these tasks. We also explore the data augmentation, semi-supervised techniques, and a multitask setup. In our extensive experiments, we achieve promising results.

show abstract

A computationally efficient multi-modal classification approach of disaster-related Twitter images

Cited by 23 publications

References 19 publications

A deep multi-modal neural network for informative Twitter content classification during emergencies

A deep multi-modal neural network for informative Twitter content classification during emergencies

A Deep Attentive Multimodal Learning Approach for Disaster Identification From Social Media Posts

Robust Training of Social Media Image Classification Models for Rapid Disaster Response

Contact Info

Product

Resources

About