MADNet 2.0: Pixel-Scale Topography Retrieval from Single-View Orbital Imagery of Mars Using Deep Learning

Abstract

The High-Resolution Imaging Science Experiment (HiRISE) onboard the Mars Reconnaissance Orbiter provides remotely sensed imagery at the highest spatial resolution at 25–50 cm/pixel of the surface of Mars. However, due to the spatial resolution being so high, the total area covered by HiRISE targeted stereo acquisitions is very limited. This results in a lack of the availability of high-resolution digital terrain models (DTMs) which are better than 1 m/pixel. Such high-resolution DTMs have always been considered desirable for the international community of planetary scientists to carry out fine-scale geological analysis of the Martian surface. Recently, new deep learning-based techniques that are able to retrieve DTMs from single optical orbital imagery have been developed and applied to single HiRISE observational data. In this paper, we improve upon a previously developed single-image DTM estimation system called MADNet (1.0). We propose optimisations which we collectively call MADNet 2.0, which is based on a supervised image-to-height estimation network, multi-scale DTM reconstruction, and 3D co-alignment processes. In particular, we employ optimised single-scale inference and multi-scale reconstruction (in MADNet 2.0), instead of multi-scale inference and single-scale reconstruction (in MADNet 1.0), to produce more accurate large-scale topographic retrieval with boosted fine-scale resolution. We demonstrate the improvements of the MADNet 2.0 DTMs produced using HiRISE images, in comparison to the MADNet 1.0 DTMs and the published Planetary Data System (PDS) DTMs over the ExoMars Rosalind Franklin rover’s landing site at Oxia Planum. Qualitative and quantitative assessments suggest the proposed MADNet 2.0 system is capable of producing pixel-scale DTM retrieval at the same spatial resolution (25 cm/pixel) of the input HiRISE images.

Subpixel-Scale Topography Retrieval of Mars Using Single-Image DTM Estimation and Super-Resolution Restoration

“…MARSGAN [30] and MADNet [29] are both based on the relativistic GAN framework [72,73] that involves training of a generator network and a relativistic discriminator network in parallel. For MARSGAN SRR, the generator network is trained to produce potential SRR estimations, whilst the discriminator network is trained in parallel (and updated in an alternating manner with the generator network) to estimate the probability of the given training higher-resolution images being relatively more realistic than the generated SRR images on average (within a small batch), whereas for MADNet SDE, the generator network is trained to produce per-pixel relative heights, and the discriminator network is trained to distinguish the predicted heights from the ground-truth heights.…”

“…Apart from the globally available lower resolution (~463 m/pixel) Mars Orbiter Laser Altimeter (MOLA) DTM [11,12], higher resolution Mars DTMs are typically produced from the 12.5-50 m/pixel Mars Express High Resolution Stereo Camera (HRSC) images [13], the 6 m/pixel Mars Reconnaissance Orbiter (MRO) Context Camera (CTX) images [14], the ~4.6 m/pixel (4 m/pixel nominal resolution) ExoMars Trace Gas Orbiter (TGO) Colour and Stereo Surface Imaging System (CaSSIS) images [15,16], and the ~30 cm/pixel (25 cm/pixel nominal resolution) MRO High Resolution Imaging Science Experiment (HiRISE) images [17]. The DTM products derived from these imaging sources often have different effective resolutions and spatial coverage, depending on the properties of the input images and the DTM retrieval methods, which include traditional photogrammetric methods [18][19][20][21], photoclinometry methods [22][23][24][25], and deep learning-based methods [26][27][28][29].…”

“…It has been a common understanding that DTMs derived from a particular imaging dataset can only achieve a lower, or at the best, similar effective spatial resolution compared to the input images, due to the various approximations and/or filtering processes introduced by the photogrammetric or photoclinometric pipelines. With recent successes in deep learning techniques, it has now become more practical and effective to improve the effective resolution of an image using super-resolution restoration (SRR) networks [30], retrieving pixel-scale topography using single-image DTM estimation (SDE) networks [27,29], and eventually, combining the two techniques to potentially produce subpixel-scale topography from only a single-view input image.…”

“…In parallel, SDE refers to the process of deriving heights or depths from a given singleview image, using traditional "shape from shading" (photoclinometry) techniques [22][23][24]47,48], or learning the "height/depth cues" from a training dataset using deep learning techniques. Such deep learning-based SDE methods, using residual networks [49][50][51], conditional random fields [52][53][54][55], attention-based networks [56,57], GANs [58,59], and U-nets [60,61], have been fairly successful in recent years, not only in the field of indoor/outdoor scene reconstruction but also in the field of remote sensing for topographic retrieval using planetary orbital images [26][27][28][29].…”

“…In this paper, we propose combining the use of SRR and SDE to boost the effective resolution of optical single-image-based DTMs to subpixel-scale. The in-house implementations of the MARSGAN (multi-scale adaptive-weighted residual super-resolution generative adversarial network) SRR system [30,35] and the MADNet (multi-scale generative adversarial U-net based single-image DTM estimation) SDE system [27,29] are employed for this study. Our study site is within the 3-sigma ellipses of the Rosalind Franklin ExoMars rover's planned landing site (centred near 18.275 • N, 335.368 • E) at Oxia Planum [28,62,63].…”