Uncertainty Guided Multi-Scale Residual Learning-Using a Cycle Spinning CNN for Single Image De-Raining

Abstract

Single image de-raining is an extremely challenging problem since the rainy image may contain rain streaks which may vary in size, direction and density. Previous approaches have attempted to address this problem by leveraging some prior information to remove rain streaks from a single image. One of the major limitations of these approaches is that they do not consider the location information of rain drops in the image. The proposed Uncertainty guided Multi-scale Residual Learning (UMRL) network attempts to address this issue by learning the rain content at different scales and using them to estimate the final de-rained output. In addition, we introduce a technique which guides the network to learn the network weights based on the confidence measure about the estimate. Furthermore, we introduce a new training and testing procedure based on the notion of cycle spinning to improve the final de-raining performance. Extensive experiments on synthetic and real datasets to demonstrate that the proposed method achieves significant improvements over the recent state-of-the-art methods. Code is available at: https://github.com/rajeevyasarla/ UMRL--using-Cycle-Spinning

Confidence Measure Guided Single Image De-Raining

“…Additionally, these confidence scores enable QuDeC to perform soft labeling of the distortion levels in the location-quality-label-map s, which makes QuDeC more robust to location quality label errors and effectively use this information as a prior while estimatingr. To computer andŝ we start with UMRL [16] and add Decoder D2 to obtain the network architecture of QuDeC as shown in Figure 6.…”

“…The encoder and decoder D1 networks are similar to the the encoder and decoder networks of UMRL [16] where a convolutional block (ConvBlock as shown in Figure 7(a)) is used as the building block. The encoder network is described as follows, ConvBlock(3,32)-AvgPool-ConvBlock(32,32)-AvgPool-Convblock(32,32)-AvgPool-ConvBlock(32,32)-AvgPool where AvgPool is the average pooling layer, UpSample is the upsampling convolution layer, and ConvBlock(i, j) indicates ConvBlock with i input channels and j output channels.…”

“…We use the estimated residual map and the feature maps as inputs to the Confidence map Network (CN) to compute the confidence measure at every pixel, which indicates how sure the network is about the residual value at each pixel. CN consists of the following sequence of convolutional layers, Convblock(67,16)-Convblock(16,16)-Convblock (16,3) as shown in Figure 7(c). Given the estimated residual map and the corresponding feature maps as inputs to the confidence map network, it estimates c ×4 and c ×2 .…”

“…To this end, we use the Natural Image Quality Evaluator (NIQE) no-reference image quality score [17] and formulate a joint task of estimating the level of distortion at each spatial location of the image and computing rain streak information to reconstruct the clean image. This is achieved by adding a decoder network which computes the level of distortion at each location, to our UMRL approach in [16]. This additional decoder network labels the distortion at each location into three classes and also computes the confidence scores for each patch (which indicates how confident the decoder is about those obtained labels) as shown in the Figure 2.…”

“…• We formulate a joint task of estimating the level of distortion at each spatial location of the image and compute rain streak information by adding a new decoder to UMRL [16]. • A new loss function is proposed for estimating the confidence scores for both residual maps and distortion quality-label maps.…”