Contrastive Learning and Self-Training for Unsupervised Domain Adaptation in Semantic Segmentation

Abstract

Deep convolutional neural networks have considerably improved state-of-the-art results for semantic segmentation. Nevertheless, even modern architectures lack the ability to generalize well to a test dataset that originates from a different domain. To avoid the costly annotation of training data for unseen domains, unsupervised domain adaptation (UDA) attempts to provide efficient knowledge transfer from a labeled source domain to an unlabeled target domain. Previous work has mainly focused on minimizing the discrepancy between the two domains by using adversarial training or self-training. While adversarial training may fail to align the correct semantic categories as it minimizes the discrepancy between the global distributions, self-training raises the question of how to provide reliable pseudo-labels. To align the correct semantic categories across domains, we propose a contrastive learning approach that adapts category-wise centroids across domains. Furthermore, we extend our method with selftraining, where we use a memory-efficient temporal ensemble to generate consistent and reliable pseudo-labels. Although both contrastive learning and self-training (CLST) through temporal ensembling enable knowledge transfer between two domains, it is their combination that leads to a symbiotic structure. We validate our approach on two domain adaptation benchmarks: GTA5 → Cityscapes and SYNTHIA → Cityscapes. Our method achieves better or comparable results than the state-of-the-art. We will make the code publicly available.

Continual Unsupervised Domain Adaptation for Semantic Segmentation using a Class-Specific Transfer

“…2) Self-Training: A very successful strategy, also found in several state-of-the-arts methods for UDA, is self-training (ST) [7], [12], [17], [33], [35]. It basically converts the target predictions into pseudo-labels to minimize the cross-entropy.…”

“…However, ST only works if high quality pseudo-labels (PL) can be provided. One line of work attempts to improve their quality by using an ensemble of three models [7], a memoryefficient temporal ensemble [12], or an average of different outputs of the same network [34]. Nevertheless, most ST-based approaches try to filter out the noisy target predictions by using reliability measures.…”

“…UDA methods attempt to transfer knowledge from a labeled source domain to an unlabeled target domain. This is achieved by mitigating the discrepancy between the source and target distribution in the input space [5]- [8], the feature space [9]- [12], the output space [13]- [16], or even in several spaces in parallel. Nevertheless, the goal of UDA is not sufficient because it neglects the performance on the source domain and focuses only on the target results.…”

“…Nevertheless, the goal of UDA is not sufficient because it neglects the performance on the source domain and focuses only on the target results. Consequently, UDA is particularly useful for the well-studied synthetic-to-real scenario [12], [13], [17], where the perception system never encounters the synthetic domain again. In the context of automated driving, however, the environment to which the model has just been adapted may return to a previously seen target domain or the source domain.…”

Hard-aware Instance Adaptive Self-training for Unsupervised Cross-domain Semantic Segmentation

“…These meth- ods aim to minimize a series of adversarial losses to learn invariant representations across domains, thereby aligning source and target feature distributions. More recently, an alternative research line to reduce domain shifts focuses on building schemes based on the self-training (ST) framework [16], [17], [18], [19], [20], [21], [22], [23], [24], [25], [26], [27], [28], [29], [30], [31], [32]. These works iteratively train the model by using both the labeled source domain data and generated pseudo-labels of unlabeled target domain data, thus achieving alignment between source and target domains.…”