Simultaneous Enhancement and Super-Resolution of Underwater Imagery for Improved Visual Perception

Abstract

In this paper, we introduce and tackle the simultaneous enhancement and super-resolution (SESR) problem for underwater robot vision and provide an efficient solution for near real-time applications. We present Deep SESR, a residualin-residual network-based generative model that can learn to restore perceptual image qualities at 2×, 3×, or 4× higher spatial resolution. We supervise its training by formulating a multi-modal objective function that addresses the chrominancespecific underwater color degradation, lack of image sharpness, and loss in high-level feature representation. It is also supervised to learn salient foreground regions in the image, which in turn guides the network to learn global contrast enhancement. We design an end-to-end training pipeline to jointly learn the saliency prediction and SESR on a shared hierarchical feature space for fast inference. Moreover, we present UFO-120, the first dataset to facilitate large-scale SESR learning; it contains over 1500 training samples and a benchmark test set of 120 samples. By thorough experimental evaluation on UFO-120 and several other standard datasets, we demonstrate that Deep SESR outperforms the existing solutions for underwater image enhancement and super-resolution. We also validate its generalization performance on several test cases that include underwater images with diverse spectral and spatial degradation levels, and also terrestrial images with unseen natural objects. Lastly, we analyze its computational feasibility for single-board deployments and demonstrate its operational benefits for visually-guided underwater robots.

SFA-GAN: Structure-frequency-aware generative adversarial network for underwater image enhancement

“…The data that support the findings of this study are openly available [39][40] [41]. The codes are available from the corresponding author on reasonable request.…”

“…Each pair consists of a good quality ground truth image, a ground truth saliency map and a poor quality distorted image. To achieve unpaired UIE, we random shuffle combinations of poor quality and good quality images in the training set and keep pairing unchanged in test set [39].…”

Attention-guided hybrid transformer-convolutional neural network for underwater image super-resolution

“…The multiple degradation modalities contained in the UFO-120 dataset 33 facilitate the validation of the generalization ability of our AHTCN. For the sake of fairness, this experiment only compares with the conventional underwater SR methods.…”

“…USR-248 is a large dataset 10 consisting of paired instances for supervised training of 2×, 4×, and 8× SR models. The UFO-120 dataset 33 contains 1500 training samples and a test set consisting of 120 samples. We employ the ADAM optimizer 34 with β 1 ¼ 0.9, β 2 ¼ 0.99, and ε ¼ 10 −8 to train the model.…”

DGC‐UWnet: Underwater image enhancement based on computation‐efficient convolution and channel shuffle

“…The dataset with the reference image, named EUVP‐150, contains 150 pairs of images selected from the EUVP. The dataset without reference image, named RUIE‐150 dataset and UFO‐150, contain 150 underwater images from the RUIE (Real‐world Underwater Image Enhancement) dataset [35] and UFO‐120 dataset [36] respectively. All the images in validation datasets were resized to 256 × 256.…”