TransWeather: Transformer-based Restoration of Images Degraded by Adverse Weather Conditions

Valanarasu, Jeya Maria Jose; Yasarla, Rajeev; Patel, Vishal M.

doi:10.1109/cvpr52688.2022.00239

Cited by 158 publications

(96 citation statements)

References 56 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As transformer possesses long-range modelling capability and adaptability to input content, they were adopted in various high-level CV tasks such as object classi cation, detection, tracking, segmentation and pose estimation. For image restoration, networks which adopted transformer are Restormer [19], U-former [20], Swin-IR [21], U2-former [22] and Transweather [27]. However, these networks perform poor on real-rain images which are affected by high-density rainfall.…”

Section: Vision Transformersmentioning

confidence: 99%

DeTformer: A Novel Efficient Transformer Framework for Image Deraining

Ragini

Cheruku

Kodali

2023

Preprint

View full text Add to dashboard Cite

Captured rainy images severely degrade outdoor vision systems performance, such as semi-autonomous or autonomous driving systems and video surveillance systems. Consequently, removing heavy and complex rain streaks i.e., undesirable rainy artifacts from a rainy image plays a crucial role for many high-level computer vision tasks and has drawn researchers’ attention from the past few years. The main drawbacks of Convolutional neural networks: have smaller receptive field, lack the model’s ability to capture long-range dependencies and complicated rainy artifacts, non-adaptive to input content and also computational complexity grows quadratically with input image size. These factors limit the deraining model performance improvement further. Recently, transformer has achieved better performance in both Natural language processing (NLP) and high-level computer vision (CV). We cannot adopt transformer directly to image deraining task as it has following limitations: a) although the transformer possesses powerful long-range computational capability, it lacks the ability to model local features b) to process input image, transformer uses fixed patch size, therefore pixels at the patch edges cannot use local features of surrounding pixels while removing heavy rain streaks. To address these issues, in single image deraining, we proposed a novel and efficient De-raining Transformer (DeTformer). In DeTformer, we designed a “Gated-Depth-wise Convolution Feed-forward Network” (GDWCFN) to address the first issue and applied depthwise convolution to improve the modelling capability of local features and suppress unnecessary features and allow only useful information further. Also, the second issue was addressed, by introducing multi-resolution features in our network, and we applied progressive learning in the transformer and thus it allows the edge pixels to utilize local features effectively. Furthermore, to integrate the extracted multi-scale features and provide feature interaction across channel dimensions, we introduced a “Multi-head Depth-wise Convolution Transposed Attention” (MDWCTA) module. The proposed model experimented with various de-rained datasets and compared with various state-of-the-art models. The experimental results show that DeTformer network achieves superior performance compared to state-of-the-art networks on synthetic and real-world rain datasets.

show abstract

Section: Vision Transformersmentioning

confidence: 99%

DeTformer: A Novel Efficient Transformer Framework for Image Deraining

Ragini

Cheruku

Kodali

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Most recently, Valanarasu. et al propose an alternative state-of-the-art solution to this problem with TransWeather (Valanarasu et al, 2022). As an end-to-end vision transformer (Dosovitskiy et al, 2021) based multi-weather image restoration model, it exhibits more powerful versatility.…”

Section: Image Enhancementmentioning

confidence: 99%

“…Due to its success in high-level tasks such as image classification, segmentation, and detection, the transformer has been used in low-level vision tasks. Valanarasu et al proposed Transweather, an end-to-end multi-weather image restoration model, as an alternative solution to multi-encoders for the same application scenario (Valanarasu et al, 2022). Li et al also proposed a unified framework capable of recovering images with unknown degradation types, which has demonstrated its effectiveness in image enhancement affected by natural weather (Li et al, 2022).…”

Section: Introductionmentioning

confidence: 99%

All-in-one aerial image enhancement network for forest scenes

et al. 2023

View full text Add to dashboard Cite

Drone monitoring plays an irreplaceable and significant role in forest firefighting due to its characteristics of wide-range observation and real-time messaging. However, aerial images are often susceptible to different degradation problems before performing high-level visual tasks including but not limited to smoke detection, fire classification, and regional localization. Recently, the majority of image enhancement methods are centered around particular types of degradation, necessitating the memory unit to accommodate different models for distinct scenarios in practical applications. Furthermore, such a paradigm requires wasted computational and storage resources to determine the type of degradation, making it difficult to meet the real-time and lightweight requirements of real-world scenarios. In this paper, we propose an All-in-one Image Enhancement Network (AIENet) that can restore various degraded images in one network. Specifically, we design a new multi-scale receptive field image enhancement block, which can better reconstruct high-resolution details of target regions of different sizes. In particular, this plug-and-play module enables it to be embedded in any learning-based model. And it has better flexibility and generalization in practical applications. This paper takes three challenging image enhancement tasks encountered in drone monitoring as examples, whereby we conduct task-specific and all-in-one image enhancement experiments on a synthetic forest dataset. The results show that the proposed AIENet outperforms the state-of-the-art image enhancement algorithms quantitatively and qualitatively. Furthermore, extra experiments on high-level vision detection also show the promising performance of our method compared with some recent baselines.

show abstract

“…This All-in-One method was tested across three datasets of rainy, hazy, and snowy images and achieved better or comparable performance than dedicated adverse weather removal models. Jeya et al [ 33 ] proposed a transformer-based encoder–decoder network called TransWeather. Through fine filtering, they created a dataset combining the Snow100K, Raindrop, and Outdoor-Rain corpora.…”

Section: Related Workmentioning

confidence: 99%

Framework for Generation and Removal of Multiple Types of Adverse Weather from Driving Scene Images

Yang

Carballo

Zhang

et al. 2023

Sensors

View full text Add to dashboard Cite

Weather variation in the distribution of image data can cause a decline in the performance of existing visual algorithms during evaluation. Adding additional samples of target domain to training data or using pre-trained image restoration methods such as de-hazing, de-raining, and de-snowing, to improve the quality of input images are two promising solutions. In this work, we propose Multiple Weather Translation GAN (MWTG), a CycleGAN-based, dual-purpose framework that simultaneously learns weather generation and its removal from image data. MWTG consists of four GANs constrained using cycle consistency that carry out domain translation tasks between hazy, rainy, snowy, and clear weather, using an asymmetric approach. To increase network capacity, we employ a spatial feature transform (SFT) layer to fuse the features extracted from the weather layer, which contains high-level domain information from the previous generators. Further, we collect an unpaired, real-world driving dataset recorded under various weather conditions called Realistic Driving Scenes under Bad Weather (RDSBW). We qualitatively and quantitatively evaluate MWTG using the RDSBW and the variation of Cityscapes that synthesize weather effects, eg., FoggyCityscape. Our experimental results suggest that MWTG can generate realistic weather in clear images and also accurately remove noise from weather images. Furthermore, the SOTA pedestrian detector ASCP is shown to achieve an impressive gain in detection precision after image restoration using the proposed MWTG method.

show abstract

TransWeather: Transformer-based Restoration of Images Degraded by Adverse Weather Conditions

Cited by 158 publications

References 56 publications

DeTformer: A Novel Efficient Transformer Framework for Image Deraining

DeTformer: A Novel Efficient Transformer Framework for Image Deraining

All-in-one aerial image enhancement network for forest scenes

Framework for Generation and Removal of Multiple Types of Adverse Weather from Driving Scene Images

Contact Info

Product

Resources

About