Raindrop Removal and Background Information Recovery in Coastal Wave Video Imagery using Generative Adversarial Networks

Abstract

In this paper, we propose a video enhancement method using generative adversarial networks to remove raindrops and restore the background information on the removed region in the coastal wave video imagery distorted by raindrops during rainfall. Two experimental models are implemented: Pix2Pix network widely used for image-to-image translation and Attentive GAN, which is currently performing well for raindrop removal on a single images. The models are trained with a public dataset of paired natural images with and without raindrops and the trained models are evaluated their performance of raindrop removal and background information recovery of rainwater distortion of coastal wave video imagery. In order to improve the performance, we have acquired paired video dataset with and without raindrops at the real coast and conducted transfer learning to the pretrained models with those new dataset. The performance of fine-tuned models is improved by comparing the results from pretrained models. The performance is evaluated using the peak signal-to-noise ratio and structural similarity index and the finetuned Pix2Pix network by transfer learning shows the best performance to reconstruct distorted coastal wave video imagery by raindrops.

Wave-Tracking in the Surf Zone Using Coastal Video Imagery with Deep Neural Networks

“…For the Pix2Pix network pre-training, we use an open dataset including paired images with and without raindrops in the land environment collected by Qian et al [9]. After the pre-training, we train the pre-trained model using a dataset paired with and without raindrops, which was acquired directly using two CCTVs installed side by side in Anmok beach [10]. 17,002 paired images are extracted from the 10 min.…”