DADA: Depth-Aware Domain Adaptation in Semantic Segmentation

Abstract

Figure 1: We propose a novel depth-aware domain adaptation framework (DADA) to efficiently leverage depth as privileged information in the unsupervised domain adaptation setting. This example shows how semantic segmentation of a scene from the target domain benefits from the proposed approach, in comparison to state-of-the-art domain adaptation with no use of depth. In figure's top, we use different background colors (blue and red) to represent source and target information that are available during training. Here, annotated source domain data come from the synthetic SYNTHIA dataset and un-annotated target domain images are real scenes from Cityscapes. The cyclist highlighted by the yellow box is a good qualitative illustration of the improvement we obtain. AbstractUnsupervised domain adaptation (UDA) is important for applications where large scale annotation of representative data is challenging. For semantic segmentation in particular, it helps deploy, on real "target domain" data, models that are trained on annotated images from a different "source domain", notably a virtual environment. To this end, most previous works consider semantic segmentation as the only mode of supervision for source domain data, while ignoring other, possibly available, information like depth. In this work, we aim at exploiting at best such a privileged information while training the UDA model. We propose a unified depth-aware UDA framework that leverages in several complementary ways the knowledge of dense depth in the source domain. As a result, the performance of the trained semantic segmentation model on the target domain is boosted. Our novel approach indeed achieves state-of-the-art performance on different challenging synthetic-2-real benchmarks. Code and models are available at https://github.com/ valeoai/DADA.

Online Performance Prediction of Perception DNNs by Multi-Task Learning With Depth Estimation

“…Multi-task learning (MTL) is defined as the joint learning process of several tasks at once by either learning a shared feature representation [43], [44], or by implementing cross-task consistency checks into the training process [45], [46]. Depth estimation as used in this work has been shown to take and give profit through MTL with other tasks as, e.g., semantic segmentation [44], [47], [48], domain adaptation [7], optical flow estimation [49], [50], or 3D pose estimation [4], [24]. Particularly, self-supervised depth estimation has been combined with semantic segmentation [46], [51]- [53] or instance segmentation [36], [54] to mitigate the effect of moving objects, which violate the static world assumption made during training of such models.…”

“…which considers the number of true positives (T P s ), false negatives (F N s ), and false positives (F P s ) between the estimated segmentation mask m and the ground truth segmentation mask m for each class s. Note that T P s , F N s and F P s are calculated over an entire test set and only afterwards the mIoU is obtained according to (7). For online performance observation, however, the segmentation performance has to be predicted and evaluated on a single-image basis (image index n) in order to be real-time capable.…”

“…For online performance observation, however, the segmentation performance has to be predicted and evaluated on a single-image basis (image index n) in order to be real-time capable. Therefore, for the scope of this work, we calculate the mIoU metric imagewise (mIoU n ) according to (7), and afterwards the average value of the mIoU n over all images is taken as mIoU . This makes it possible to predict and evaluate the semantic segmentation performance on a single image 1 x, without changing the interpretation of the metric when reporting the overall performance on a whole test set.…”

“…For example, when the predicted performance is too low, the high-level planning could decide not to trust the current information from the environment perception. This is of special importance when considering that in practice the DNN performance is often very sensitive to changes of the environment, which have not been included into the dataset, the neural network was trained on [7], [8]. Such changes include, e.g., a different camera type, different lighting or weather conditions, various other kinds of domain shift, or even directed adversarial attacks [9], [10], which are difficult to detect on the input image.…”

Generalized Sparse Convolutional Neural Networks for Semantic Segmentation of Point Clouds Derived from Tri-Stereo Satellite Imagery

“…There are also recent improvements in language understanding with Bidirectional Encoder Representations from Transformers (BERT) [9] and A Robustly Optimized BERT Pretraining Approach (RoBERTa) [10], and, adding to that, the recent breakthrough in task agnostic transfer learning by Howard et al [11]. In CV, DL has advanced, inter alia, the tasks of image classification [12,13], object-detection [14][15][16], object-tracking [17], pose estimation [18][19][20][21], superresolution [22], and semantic segmentation [23][24][25][26][27][28]. These advancements give rise to new applications in, e.g., solid-state materials science and chemical sciences [29,30], meteorology [31], medicine [32][33][34][35][36][37][38][39], seismology [40][41][42], biology [43], life sciences in general [44], chemistry [45], and physics [46][47][48][49][50][51][52]…”