Unsupervised Domain Adaptation for Semantic Segmentation of Urban Scenes

Abstract

The semantic understanding of urban scenes is one of the key components for an autonomous driving system. Complex deep neural networks for this task require to be trained with a huge amount of labeled data, which is difficult and expensive to acquire. A recently proposed workaround is the usage of synthetic data, however the differences between real world and synthetic scenes limit the performance. We propose an unsupervised domain adaptation strategy to adapt a synthetic supervised training to real world data. The proposed learning strategy exploits three components: a standard supervised learning on synthetic data, an adversarial learning strategy able to exploit both labeled synthetic data and unlabeled real data and finally a self-teaching strategy working on unlabeled data only. The last component is guided by the segmentation confidence, estimated by the fully convolutional discriminator of the adversarial learning module, helping to further reduce the domain shift between synthetic and real data. Furthermore we weighted this loss on the basis of the class frequencies to enhance the performance on less common classes. Experimental results prove the effectiveness of the proposed strategy in adapting a segmentation network trained on synthetic datasets, like GTA5 and SYNTHIA, to a real dataset as Cityscapes.

Adversarial Learning and Self-Teaching Techniques for Domain Adaptation in Semantic Segmentation

“…Another possible source of errors during the training procedure could be related to the well distinguishable Dirac distributed segmentation ground truth data from other distributions generated by G. We have investigated this issue and in general G produces segmentation maps very close to the Dirac distribution and this forces D to capture also other statistical properties of the two different types of input data. Notice that this issue has been investigated also in [3], [5] with similar conclusions. The discriminator D is used to implement the second loss function for the training of G, L s,t G,2 .…”

“…We present an unsupervised domain adaptation strategy for road driving scenes able to adapt an initial learning performed on synthetic data to the real world case. The domain adaptation strategy presented in this work is based on adversarial learning E-mail: umberto.michieli@dei.unipd.it and is an extension of our previous work introduced in [3]: here we further improve the self-teaching strategy and we present a more robust experimental evaluation.…”

“…This key component is based on the idea introduced in [5] that the output of the discriminator can be also used as a measure of the reliability of the network estimations to be exploited in a self-teaching framework. However, this component has been greatly improved in this work, both with respect to [5] and to [3]. First of all, the output of the discriminator has been considered as a weight to be applied to the loss function of the self-teaching component at each location (in place of the hard threshold used in previous work [3]).…”

“…However, this component has been greatly improved in this work, both with respect to [5] and to [3]. First of all, the output of the discriminator has been considered as a weight to be applied to the loss function of the self-teaching component at each location (in place of the hard threshold used in previous work [3]). Then, a novel region growing scheme is introduced in order to extend and better represent the shape of reliable regions (the approaches of [3], [5] tend to almost always discard edge regions and small objects).…”

“…First of all, the output of the discriminator has been considered as a weight to be applied to the loss function of the self-teaching component at each location (in place of the hard threshold used in previous work [3]). Then, a novel region growing scheme is introduced in order to extend and better represent the shape of reliable regions (the approaches of [3], [5] tend to almost always discard edge regions and small objects). Finally, since the various classes have different frequencies, we also weighted the loss coming from unlabeled data in proportion to the frequency of the various classes in the dataset thus obtaining a better balance of the results between the different classes and avoiding the dramatic drop in performance on less common classes (typically corresponding to small objects and structures that represent the critical elements for an autonomous vehicle).…”