A Study of Weather-Image Classification Combining VIT and a Dual Enhanced-Attention Module

Abstract

A weather-image-classification model combining a VIT (vision transformer) and dual augmented attention module is proposed to address the problems of the insufficient feature-extraction capability of traditional deep-learning methods with the recognition accuracy still to be improved and the limited types of weather phenomena existing in the dataset. A pre-trained model vision transformer is used to acquire the basic semantic feature representation of weather images. Dual augmented attention combined with convolutional self-attention and Atrous self-attention modules are used to acquire the low-level and high-level deep-image semantic representations, respectively, and the feature vectors are spliced and fed into the linear layer to obtain the weather types. Experimental validation is performed on the publicly available standard weather-image datasets MWD (Multi-class Weather Database) and WEAPD (Weather Phenomenon Database), and the two datasets are combined to enhance the comprehensiveness of the model for weather-phenomena recognition. The results show that the model achieves the highest F1 scores of 97.47%, 87.69% and 92.73% on the MWD, WEAPD and merged datasets, respectively. These scores are higher than the scores of recent deep-learning models with excellent performance in the experimental comparison, thereby, proving the effectiveness of the model.

A Study on Meteorological Recognition With a Fusion Enhanced Model Based on EVA02 and LinearSVC

“…The results in Table 5 demonstrate that the proposed model achieved the highest performance. Specifically, on the MWD weather dataset, the model outperformed other excellent deep learning models, including AlexNet-feature fusion, VGG16-TL, MeteCNN, VGG16-GNet-SNN [11], L-CT, and VIT-DA [20], with improvements in F1-scores of 2.75%, 2.63%, 1.9%, 1.75%, 2.29%, and 0.75%, respectively. On the WEAPD weather dataset, the model exhibited F1-score improvements of 12.51%, 12.34%, 11.81%, 11.19%, 11.05%, and 9.89% compared to the same set of models.…”

“…Li et al [20] conducted experiments and discovered that Transformer-based network architectures perform better on weather image datasets compared to CNN-based models. This research hypothesized that weather image data may have certain commonalities that can be learned more effectively by ViT-like networks.…”

“…Toğaçar et al [11] combined features extracted from GoogLeNet and VGG-16 models and provided them as inputs to SNNs for classification; Xiao et al [18] considered that recognizing fewer weather categories cannot encompass the majority of existing weather phenomena, thus constructing an 11category weather dataset and proposed modifying the VGG-19 network by incorporating SE modules and replacing fully connected layers with max-pooling layers for classification. However, the convolution-based feature extraction approach is susceptible to factors such as light, reflections and other factors that do not have structural information, so meteorological recognition research began to use the attention mechanism to focus on fine-grained features of weather images, and Yu et al [19] proposes the network GSLSNet, which utilizes the Global Similarity Module (GSM) to identify the overall weather conditions and uses the Local Significance Module (LSM) to focus on the details of the weather on roads; Li et al [20] introduces the Convolutional Self-Attention Mechanism and Atrous Self-Attention Mechanism based on VisionTransformer (ViT) [21] to obtain the low and high level deep image semantic representations, respectively, and fuses the features to obtain the weather type after classification, which has higher performance metrics than recent high-performing deep learning models on publicly available datasets, but these models have relatively simple classification layers. Furthermore, existing recognition models in the meteorological domain face the challenges of excessively long neural network training times, high hardware requirements, and substantial computational expenses.…”

“…Models using modified convolutional neural networks combined with various classification strategies [11], [18] are limited by the shortcomings of their underlying neural networks in feature extraction, and they cannot extract more semantic features. However, for models that combine global and local features of images and use attention mechanisms for fine-grained feature extraction [19], [20], they intend to achieve an end-toend network model. The drawback is that their classification capabilities are relatively simple, and we found that they all involve training the entire model composed of feature extraction layers and classification layers.…”

Evaluation of Vision Transformer on Weather Image Recognition

“…They concluded that their implementation of Vision Transformer performed better than other current unsupervised techniques on splicing items. Li et al [11] implemented a model that combined a vision transformer and augmentation of images on the weather image classification problem. The research aimed to resolve the problems that lack the capability for feature extraction resulting from traditional deep learning models.…”