Self-supervised Monocular Depth Estimation: Solving the Dynamic Object Problem by Semantic Guidance

Klingner, Marvin; Termöhlen, Jan-Aike; Mikolajczyk, Jonas; Fingscheidt, Tim

doi:10.1007/978-3-030-58565-5_35

Cited by 266 publications

(129 citation statements)

References 60 publications

Supporting

Mentioning

129

Contrasting

Order By: Relevance

“…Our method does not require MVS depth as input, only camera poses, and does not require training a fusion module. In addition, some methods [Klingner et al 2020;Patil et al 2020] use semantic information to guide depth prediction and identify moving objects during training. Those methods usually focuses on specific scenarios such as autonomous driving.…”

Section: Related Workmentioning

confidence: 99%

Consistent depth of moving objects in video

et al. 2021

View full text Add to dashboard Cite

show abstract

Section: Related Workmentioning

confidence: 99%

Consistent depth of moving objects in video

et al. 2021

View full text Add to dashboard Cite

show abstract

“…For some ablation experiments, we also use the PSPNet architecture [80], using a pyramid pooling module as decoder without skip connections, and the ERFNet architecture [81], using efficient residual connections and factorized convolutions. We simply double the decoder architecture for the second decoder head to obtain a multi-task network as described in [52], while adapting the full-resolution multi-scale depth loss formulation proposed by [4]. We apply the gradient scaling described by ( 9) at all connections (particularly, also at skip connections) between the encoder and the decoder with a scaling factor of λ = 0.1.…”

Section: B Experimental Setupmentioning

confidence: 99%

Online Performance Prediction of Perception DNNs by Multi-Task Learning With Depth Estimation

Klingner

Fingscheidt

2021

IEEE Trans. Intell. Transport. Syst.

Self Cite

View full text Add to dashboard Cite

Online performance prediction (or: observation) of deep neural networks (DNNs) in highly automated driving presents an unsolved task until now, as most DNNs are evaluated offline requiring datasets with ground truth labels. In practice, however, DNN performance depends on the used camera type, lighting and weather conditions, and on various other kinds of domain shift. Also, the input to DNN-based perception systems can be perturbed by adversarial attacks requiring means to detect these input perturbations. In this work we propose a method to mitigate these problems by a multi-task learning approach with monocular depth estimation as a secondary task, which enables us to predict the DNN's performance for various other (primary) tasks by evaluating only the depth estimation task with a physical depth measurement provided, e.g., by a LiDAR sensor. We show the effectiveness of our method for the primary task of semantic segmentation using various training datasets, test datasets, model architectures, and input perturbations. Our method provides an effective way to predict (observe) the performance of DNNs for semantic segmentation even on a single-image basis and is transferable to other primary DNN-based perception tasks in a straightforward manner.

show abstract

“…The introduction of semantic cues into depth estimation has been explored by prior works. One direction is to use segmentation masks to mask out pixels belonging to dynamic classes so that they would not contribute to the optimization process [6,8]. The other one is coupling a semantic segmentation network into the depth estimation task to take advantage of semantic features.…”

Section: Incorporating Semantic Informationmentioning

confidence: 99%

“…The proposed method recomposes the training objective with semantic cues and geometry constraints in a self-supervised manner, making it more credible towards ambiguous pixels. Meanwhile, since the segmentation network only contributes in training phase and no intermediate output is fused into depth network, the proposed method does not introduce extra inference complexity compared to prior works [6,7,8,9]. Our main contribution can be concluded as:…”

Section: Introductionmentioning

confidence: 99%

Rethinking Training Objective For Self-Supervised Monocular Depth Estimation: Semantic Cues To Rescue

2021

2021 IEEE International Conference on Image Processing (ICIP)

View full text Add to dashboard Cite

Monocular depth estimation finds a wide range of applications in modeling 3D scenes. Since it is expensive to collect ground truth labels to supervise training, plenty of works have been done in a self-supervised manner. A common practice is to train the network optimizing a photometric objective (i.e., view synthesis ) due to its effectiveness. However, this training objective is sensitive to optical changes and lacks a consideration of object-level cues, which leads to sub-optimal results in some cases, e.g., artifacts in complex regions and depth discontinuities around thin structures. We summarize them as depth ambiguities.In this paper, we propose an easy yet effective architecture, introducing semantic cues into supervision to solve problems mentioned above. First through our study on the problems we figure out that they are due to the limitation of the commonly applied photometric reconstruction training objective. Then we come up with our method using semantic cues to encode the geometry constraint behind view synthesis. The proposed novel objective is more credible towards confusing pixels, also takes an object-level perception. Experiments show that without introducing extra inference complexity, our method alleviates depth ambiguities greatly and performs comparably with state-of-the-art methods on KITTI benchmark.

show abstract

Self-supervised Monocular Depth Estimation: Solving the Dynamic Object Problem by Semantic Guidance

Cited by 266 publications

References 60 publications

Consistent depth of moving objects in video

Consistent depth of moving objects in video

Online Performance Prediction of Perception DNNs by Multi-Task Learning With Depth Estimation

Rethinking Training Objective For Self-Supervised Monocular Depth Estimation: Semantic Cues To Rescue

Contact Info

Product

Resources

About