The Effects of Super-Resolution on Object Detection Performance in Satellite Imagery

Abstract

We explore the application of super-resolution techniques to satellite imagery, and the effects of these techniques on object detection algorithm performance. Specifically, we enhance satellite imagery beyond its native resolution, and test if we can identify various types of vehicles, planes, and boats with greater accuracy than native resolution. Using the Very Deep Super-Resolution (VDSR) framework and a custom Random Forest Super-Resolution (RFSR) framework we generate enhancement levels of 2×, 4×, and 8× over five distinct resolutions ranging from 30 cm to 4.8 meters. Using both native and super-resolved data, we then train several custom detection models using the SIMRDWN object detection framework. SIMRDWN combines a number of popular object detection algorithms (e.g. SSD, YOLO) into a unified framework that is designed to rapidly detect objects in large satellite images. This approach allows us to quantify the effects of super-resolution techniques on object detection performance across multiple classes and resolutions. We also quantify the performance of object detection as a function of native resolution and object pixel size. For our test set we note that performance degrades from mean average precision (mAP) = 0.53 at 30 cm resolution, down to mAP = 0.11 at 4.8 m resolution. Super-resolving native 30 cm imagery to 15 cm yields the greatest benefit; a 13 − 36% improvement in mAP. Superresolution is less beneficial at coarser resolutions, though still provides a small improvement in performance.

Small-Object Detection in Remote Sensing Images with End-to-End Edge-Enhanced GAN and Object Detector Network

“…Therefore, we create the low-resolution images which do not precisely correspond to low-resolution images. However, improvement of resolution through deep learning always improves object detection performance on remote sensing images (for both artificial and real low-resolution images), as discussed in the introduction and related works section of this paper [5]. There are some impressive works [61,71] to create realistic low-resolution images from high-resolution images.…”

“…There are many methods on detecting and locating objects from images that are captured using satellites or drones, but detection performance is not satisfactory on noisy and low-resolution images, especially when the objects are small [4]. Even on high-resolution imagery, the detection performance of small objects is lower compared to large objects [5].…”

“…To improve detection accuracy on remote sensing images, researchers have used deep convolutional neural network (CNN)-based super-resolution techniques to generate artificial high-resolution images and then detect objects [5,12]. Deep CNN-based super-resolution techniques such as image super-resolution using deep convolutional networks (SRCNN) [17] and accurate image super-resolution using very deep convolutional networks (VDSR) [18] show excellent results on generating realistic high-resolution imagery from low-resolution input data.…”

“…The positive effects of super-resolution on object detection tasks is discussed in [5] where the authors also use remote sensing datasets for their experiments. Simultaneous CNN-based image enhancement with object detection using single-shot multi-box detector (SSD) [10] is done in [59].…”

Two-Stream Action Recognition-Oriented Video Super-Resolution

“…To address the resolution problem, most of existing works choose interpolation to simply re-scale the input while some recent works in other areas propose superresolution (SR) as an alternative solution. For example, [26] investigated the effects of SR on object detection and [37] proposed a dataset for assessing the impact of image restoration and enhancement on image classification. Most of them stop at a preliminary experimental study on existing SR methods without proposing approaches targeting on their tasks, e.g.…”