Disentangled Latent Transformer for Interpretable Monocular Height Estimation

Abstract

Monocular height estimation (MHE) from remote sensing imagery has high potential in generating 3D city models efficiently for a quick response to natural disasters. Most existing works pursue higher performance. However, there is little research exploring the interpretability of MHE networks. In this paper, we target at exploring how deep neural networks predict height from a single monocular image. Towards a comprehensive understanding of MHE networks, we propose to interpret them from multiple levels: 1) Neurons: unit-level dissection. Exploring the semantic and height selectivity of the learned internal deep representations; 2) Instances: object-level interpretation. Studying the effects of different semantic classes, scales and spatial contexts on height estimation; 3) Attribution: pixel-level analysis. Understanding which input pixels are important for the height estimation. Based on the multi-level interpretation, a disentangled latent Transformer network is proposed towards a more compact, reliable and explainable deep model for monocular height estimation. Furthermore, a novel unsupervised semantic segmentation task based on height estimation is first introduced in this work. Additionally, we also construct a new dataset for joint semantic segmentation and height estimation. Our work provides novel insights for both understanding and designing MHE models. The dataset and code are publicly available at https://github.com/ShadowXZT/DLT-Height-Estimation.pytorch.

HeightNet: Monocular Object Height Estimation

“…You et al first determined the depth selectivity of some hidden units, which showed good insights for interpreting monocular depth estimation models [18]. Zhi et al proposed a novel disentangled latent Transformer model based on the multi-level interpretation [19]. Santo et al proposed a deep photometric stereo network (DPSN) by applying a photometric stereo method based on deep learning away from simplified image formation models such as the general Lambert model [20].…”

HTC-DC Net: Monocular Height Estimation From Single Remote Sensing Images

“…2) Multi-task Learning Networks: Multi-task learning networks introduce auxiliary tasks in addition to height predictions, with the expectation that both tasks support each other during training. Usually, based on the assumption that heights and semantics are highly correlated [33], semantic segmentation can be regarded as an auxiliary task for height estimation. For example, Srivastava et al [34] were the first to showcase the gains the auxiliary semantic segmentation head brings.…”

Change-Aware Visual Question Answering

“…With the advent of deep learning, various multi-modal learning networks [12,13] are proposed for both the computer vision and remote sensing communities [14,15]. VQA for natural images has been developed for many years [16,17], and VQA for remote sensing images has also been a hot research topic in recent years [18,19].…”