MSP-Former: Multi-Scale Projection Transformer for Single Image Desnowing

Abstract

Image restoration of snow scenes in severe weather is a difficult task. Snow images have complex degradations and are cluttered over clean images, changing the distribution of clean images. The previous methods based on CNNs are challenging to remove perfectly in restoring snow scenes due to their local inductive biases' lack of a specific global modeling ability. In this paper, we apply the vision transformer to the task of snow removal from a single image. Specifically, we propose a parallel network architecture split along the channel, performing local feature refinement and global information modeling separately. We utilize a channel shuffle operation to combine their respective strengths to enhance network performance. Second, we propose the MSP module, which utilizes multi-scale avgpool to aggregate information of different sizes and simultaneously performs multi-scale projection self-attention on multi-head self-attention to improve the representation ability of the model under different scale degradations. Finally, we design a lightweight and simple local capture module, which can refine the local capture capability of the model. In the experimental part, we conduct extensive experiments to demonstrate the superiority of our method. We compared the previous snow removal methods on three snow scene datasets. The experimental results show that our method surpasses the state-of-the-art methods with fewer parameters and computation. We achieve substantial growth by 1.99dB and SSIM 0.03 on the CSD test dataset. On the SRRS and Snow100K datasets, we also increased PSNR by 2.47dB and 1.62dB compared with the Transweather approach and improved by 0.03 in SSIM. In the visual comparison section, our MSP-Former also achieves better visual effects than existing methods, proving the usability of our method.

Dual Gradient Based Snow Attentive Desnowing

“…Cheng et al [17] accurately removed snow areas from images using on advanced snow masks.Ye et al [18] considered the similarity between dehazing and desnowing problems to solve both problems. MSP-Former [19], LMQFormer [20], and SnowFormer [21] employed transformer-based methods, which enabled to produce accurate desnowing results.…”

RSFDM-Net: Real-time Spatial and Frequency Domains Modulation Network for Underwater Image Enhancement

“…Underwater imaging plays a significant role in underwater robotics [1], providing essential information for perceiving and understanding underwater environments. Recently, more and more works [2][3][4][5][6] have paid attention to realw-world image restoration problems with challenging degradations. According to the Jaffe-McGlamey imaging model [7,8], underwater imaging consists of a linear superposition of direct, back scattered, and forward scattered components.…”

Restoring Snow-Degraded Single Images With Wavelet in Vision Transformer

“…1 The particles of snow in a scene occur in various shapes, sizes, densities, and stochastic distribution [1] and result in complex pixel variations that obscure the latent information in an image [2]. As depicted in Figure 1, this problem can introduce dire consequences in critical computer vision algorithms purposed for object detection, classification, and even segmentation applications [1], [2], [3], [4], [5], [6], [7], [8], [9], which often requires clean input image samples for optimal performance. Therefore, the need to restore snow-degraded images is…”

“…technique for vision processing as a result of its eminent outcome improvements over previous methods. Despite these accolades, the limited receptive field of the CNN's convolving kernels impedes its ability to effectively model long-range feature dependencies [9], [16]. We attribute this limitation to account for why many existing CNN-based single image desnow methods seemingly struggle to effectively remove multi-scale snow syntheses when evaluated on different varying snow datasets [1], [6], [7] despite being fine-tuned for the specific task.…”

“…On the other hand, the recent vision transformer (ViT) [17] and its subsequent modified variants [4], [9], [16] have managed to demonstrate superior performance advantage over the CNN counterpart in various vision processing tasks, including image denoising. By virtue of the transformer's innate multi-head self-attention (MHSA) and patch-based feature embedding mechanisms, the ViT is capable of facilitating global long-range feature representation superior to its predecessors [18].…”