Multimodal Ground-Based Cloud Classification Using Joint Fusion Convolutional Neural Network

Abstract

The accurate ground-based cloud classification is a challenging task and still under development. The most current methods are limited to only taking the cloud visual features into consideration, which is not robust to the environmental factors. In this paper, we present the novel joint fusion convolutional neural network (JFCNN) to integrate the multimodal information for ground-based cloud classification. To learn the heterogeneous features (visual features and multimodal features) from the ground-based cloud data, we designed the proposed JFCNN as a two-stream structure which contains the vision subnetwork and multimodal subnetwork. We also proposed a novel layer named joint fusion layer to jointly learn two kinds of cloud features under one framework. After training the proposed JFCNN, we extracted the visual and multimodal features from the two subnetworks and integrated them using a weighted strategy. The proposed JFCNN was validated on the multimodal ground-based cloud (MGC) dataset and achieved remarkable performance, demonstrating its effectiveness for ground-based cloud classification task.

Multi-Evidence and Multi-Modal Fusion Network for Ground-Based Cloud Recognition

“…In addition, since cloud types change over time, appropriate fusion of multi-modal information and cloud visual information could improve the classification performance. The JFCNN [32] achieved excellent performance with the accuracy of 93.37% by learning ground-based cloud images and multi-modal information jointly. However, the dataset used in [32] only contains 3711 labeled cloud samples, and it is randomly split into the training set and the test set with the ratio of 2:1, which means there may exist high dependence between training and test samples.…”

“…The comparison results between the proposed MMFN and other methods, such as [32,62,63], are summarized in Table 2. Firstly, most results in the right part of the table are more competitive than those in the left part, which indicates that the multi-modal information contains useful information for ground-based cloud recognition.…”

“…Such a strategy thoroughly investigates the correlations between the visual features and the multi-modal information and takes into consideration the complementary and supplementary information between them as well as their relative importance for the recognition task. 77.95 DMF [31] 79.05 DCAFs [25] 82.67 DCAFs + MI 82.97 CloutNet [26] 79.92 CloutNet + MI 80.37 JFCNN [32] 84.13 DTFN [62] 86.48 HMF [63] 87.90 MMFN 88.63…”

“…Then the deep features are integrated with multimodal features with weight. Liu et al [32] propounded a two-stream network to learn ground-based cloud images and multi-modal information jointly, and then employed a weighted strategy to fuse these two kinds of information. In spite of these efforts, recognizing ground-based cloud using both cloud images and multi-modal information still remains an open issue.…”

Ground‐Based Cloud Classification Using Task‐Based Graph Convolutional Network

“…Cloud height, cloud coverage, and cloud type are three major aspects of cloud observation and have been extensively studied (Davies, ; Fu et al, ; Liu et al, ; Zhang et al, ; Zhou et al, ). However, due to the variability and diversity of cloud appearances, cloud type classification is extremely challenging and still under development.…”