Video Rain Streak Removal by Multiscale Convolutional Sparse Coding

Abstract

Video rain/snow removal from surveillance videos is an important task in the computer vision community since rain/snow existed in videos can severely degenerate the performance of many surveillance system. Various methods have been investigated extensively, but most only consider consistent rain/snow under stable background scenes. Rain/snow captured from practical surveillance camera, however, is always highly dynamic in time with the background scene transformed occasionally. To this issue, this paper proposes a novel rain/snow removal approach, which fully considers dynamic statistics of both rain/snow and background scenes taken from a video sequence. Specifically, the rain/snow is encoded as an online multi-scale convolutional sparse coding (OMS-CSC) model, which not only finely delivers the sparse scattering and multi-scale shapes of real rain/snow, but also well encodes their temporally dynamic configurations by real-time ameliorated parameters in the model. Furthermore, a transformation operator imposed on the background scenes is further embedded into the proposed model, which finely conveys the dynamic background transformations, such as rotations, scalings and distortions, inevitably existed in a real video sequence. The approach so constructed can naturally better adapt to the dynamic rain/snow as well as background changes, and also suitable to deal with the streaming video attributed its online learning mode. The proposed model is formulated in a concise maximum a posterior (MAP) framework and is readily solved by the ADMM algorithm. Compared with the state-of-the-art online and offline video rain/snow removal methods, the proposed method achieves better performance on synthetic and real videos datasets both visually and quantitatively. Specifically, our method can be implemented in relatively high efficiency, showing its potential to real-time video rain/snow removal.Index Terms-multi-scale, convolutional sparse coding, rain/snow removal, online learning, alignment method.

Uncertainty Guided Multi-Scale Residual Learning-Using a Cycle Spinning CNN for Single Image De-Raining

“…We use the estimated residual map and the feature maps as input to the Confidence map Network (CN) to compute the confidence measure at every pixel, which indicates how sure the network is about the residual value at each pixel. CN consists of the following sequence of convolutional layers, Convblock(67,16)-Convblock(16,16)-Convblock (16,3) as shown in the Figure 4(c). Given the estimated residual map and the corresponding feature maps as input to the confidence map network, it estimates c ×4 and c ×2 .…”

“…A number of different techniques have been developed in the literature to address this problem. These algorithms can be clustered into two main groups -(i) video based algorithms [36,9,25,20,16], and (ii) single imagebased algorithms [35,8,18,31,37]. Algorithms corresponding to the first category assume temporal consistency among the image frames, and use this assumption for deraining.…”

“…32)-AvgPool-ConvBlock(32,32)-AvgPool-Convblock(32,32)-AvgPool-ConvBlock(32,32)-AvgPool-ConvBlock(32,32)-UpSample-ConvBlock(64,32)-UpSample-ConvBlock(67,32)-UpSample-ConvBlock(67,16)-ConvBlock(16,16)…”

Removing Rain in Videos: A Large-Scale Database and a Two-Stream ConvLSTM Approach

“…After the rain detection subnet converges, these hyperparameters are set as α = 0.01 and β = 1 to train the rain removal subnet. Quantitative evaluation We evaluate the performance of our approach by comparing with 5 state-of-the-art approaches: deep detail network (DDN) [11], stochastic encoding (SE) [14], matrix decomposition (MD) [23], multiscale convolutional sparse coding (MS) [17], and joint recurrent rain removal and reconstruction (J4R) [19]. Among them, [11] is a latest image-based approach, while others are video-based approaches.…”

Advancing Image Understanding in Poor Visibility Environments: A Collective Benchmark Study

“…The early works formulate rain streaks with more flexible and intrinsic characteristics, including rain modeling [10,[114][115][116][117][118][119][120][121][122][123][124]. The presence of learning-based method [125][126][127][128][129][130][131], with improved modeling capacity, brings new progress. The emergence of deep learning-based methods push performance of video deraining to a new level.…”