Hybrid CNN-Transformer Feature Fusion for Single Image Deraining

Abstract

Since rain streaks exhibit diverse geometric appearances and irregular overlapped phenomena, these complex characteristics challenge the design of an effective single image deraining model. To this end, rich local-global information representations are increasingly indispensable for better satisfying rain removal. In this paper, we propose a lightweight Hybrid CNN-Transformer Feature Fusion Network (dubbed as HCT-FFN) in a stage-by-stage progressive manner, which can harmonize these two architectures to help image restoration by leveraging their individual learning strengths. Specifically, we stack a sequence of the degradation-aware mixture of experts (DaMoE) modules in the CNN-based stage, where appropriate local experts adaptively enable the model to emphasize spatially-varying rain distribution features. As for the Transformer-based stage, a background-aware vision Transformer (BaViT) module is employed to complement spatially-long feature dependencies of images, so as to achieve global texture recovery while preserving the required structure. Considering the indeterminate knowledge discrepancy among CNN features and Transformer features, we introduce an interactive fusion branch at adjacent stages to further facilitate the reconstruction of high-quality deraining results. Extensive evaluations show the effectiveness and extensibility of our developed HCT-FFN. The source code is available at https://github.com/cschenxiang/HCT-FFN.

Learning an ensemble dehazing network for visible remote sensing images

“…Due to its significant performance, transformer has also been introduced into low-level vision tasks. [30][31][32][33][34] SwinIR 14 designed a residual network by Swin transformer blocks and achieve state-of-the-art results in image restoration. Wang et al 35 presented an efficient transformer-based network for image restoration named Uformer.…”

Learning degradation-aware visual prompt for maritime image restoration under adverse weather conditions

“…We have witnessed the rapid advancement of CNNs in image dehazing and deraining (Li et al, 2020;Zhou et al, 2021;Chen et al, 2022). However, due to the inherent characteristics of convolution operations, specifically the use of local receptive fields and the independence of input content, CNNs struggle to effectively model spatially-long feature dependencies of images (Chen et al, 2023c).…”