Disaster Image Classification by Fusing Multimodal Social Media Data

Abstract

Social media datasets have been widely used in disaster assessment and management. When a disaster occurs, many users post messages in a variety of formats, e.g., image and text, on social media platforms. Useful information could be mined from these multimodal data to enable situational awareness and to support decision making during disasters. However, the multimodal data collected from social media contain a lot of irrelevant and misleading content that needs to be filtered out. Existing work has mostly used unimodal methods to classify disaster messages. In other words, these methods treated the image and textual features separately. While a few methods adopted multimodality to deal with the data, their accuracy cannot be guaranteed. This research seamlessly integrates image and text information by developing a multimodal fusion approach to identify useful disaster images collected from social media platforms. In particular, a deep learning method is used to extract the visual features from social media, and a FastText framework is then used to extract the textual features. Next, a novel data fusion model is developed to combine both visual and textual features to classify relevant disaster images. Experiments on a real-world disaster dataset, CrisisMMD, are performed, and the validation results demonstrate that the method consistently and significantly outperforms the previously published state-of-the-art work by over 3%, with a performance improvement from 84.4% to 87.6%.

Classification of Video Observation Data for Volcanic Activity Monitoring Using Computer Vision and Modern Neural NetWorks (on Klyuchevskoy Volcano Example)

“…Considering the described peculiarities, we proposed to use methods based on the convolutional neural networks (CNN) as a tool for classifying images, which allows to analyse the whole frame, without highlighting the area of interest and preliminary feature vector extraction. With the help of a trained neural network, the continuous image stream can be automatically distributed over a given set of classes [26][27][28], solving the problem of input control of data informativity. The most important condition for training a neural network is the presence of labelled image dataset.…”

Robust Training of Social Media Image Classification Models

“…For disaster-related image classification, there have been studies where transfer-learning-based models have been used either as feature extractors or for fine-tuning the model. Such studies include flood detection from social media multimodal content [43], disaster-related tasks in a multitask learning [44], real-time system for disaster image classification during hurricane [45], sentiment analysis from disaster images [46], aerial image classification for disaster response [47], and deep features with multimodal training [48].…”

Natural Disaster Tweets Classification Using Multimodal Data

“…Their approach of training an image and text classifier and combining the two outperforms the baselines. Zou et al (2021) proposed a method through which they integrated image and text information to identify disaster images collected from different social media platforms. They use a deep learning method and FastText framework to extract visual and textual features respectively.…”