Content-Consistent Matching for Domain Adaptive Semantic Segmentation

Abstract

This paper considers the adaptation of semantic segmentation from the synthetic source domain to the real target domain. Different from most previous explorations that often aim at developing adversarialbased domain alignment solutions, we tackle this challenging task from a new perspective, i.e., content-consistent matching (CCM). The target of CCM is to acquire those synthetic images that share similar distribution with the real ones in the target domain, so that the domain gap can be naturally alleviated by employing the content-consistent synthetic images for training. To be specific, we facilitate the CCM from two aspects, i.e., semantic layout matching and pixel-wise similarity matching. First, we use all the synthetic images from the source domain to train an initial segmentation model, which is then employed to produce coarse pixel-level labels for the unlabeled images in the target domain. With the coarse/accurate label maps for real/synthetic images, we construct their semantic layout matrixes from both horizontal and vertical directions and perform the matrixes matching to find out the synthetic images with similar semantic layout to real images. Second, we choose those predicted labels with high confidence to generate feature embeddings for all classes in the target domain, and further perform the pixel-wise matching on the mined layout-consistent synthetic images to harvest the appearance-consistent pixels. With the proposed CCM, only those content-consistent synthetic images are taken into account for learning the segmentation model, which can effectively alleviate the domain bias caused by those content-irrelevant synthetic images. Extensive experiments are conducted on two popular domain adaptation tasks, i.e., GTA5− →Cityscapes and SYNTHIA− →Cityscapes. Our CCM yields consistent improvements over the baselines and performs favorably against previous state-of-the-arts.

Map-Guided Curriculum Domain Adaptation and Uncertainty-Aware Evaluation for Semantic Nighttime Image Segmentation

“…artificial light sources casting very different illumination patterns at night). A major class of adaptation approaches, including [37], [38], [39], [40], [41], [42], [43], [44], [45], [46], [47], [48], [49], [50], involves adversarial confusion or feature alignment between domains. The general concept of curriculum learning has been successfully applied to domain adaptation by ordering tasks [51], target-domain pixels [52], or domains [10], [11], [35], [53].…”

Domain Adaptive Semantic Segmentation without Source Data

“…For example, Zou et al [52] proposed a class-balanced selection strategy and selected the high-confidence pseudo-labeled target samples, while Zheng et al [51] estimated the uncertainty via an auxiliary classifier and selected the rectified pseudo labels through the dynamic threshold. Unlike the aforementioned methods, Li et al [17] proposed to select the source images that share similar distribution with the real ones in the target domain to alleviate the domain gap in another perspective.…”

Pixel-level Intra-domain Adaptation for Semantic Segmentation

“…The goal of unsupervised domain adaptive segmentation is to train a segmentation network that achieves good performance in an unlabeled target domain when only the source domain data are annotated. Methodology-wise, existing methods build on three techniques: 1) adversarial learning [7,22,32,33,36,42,44,46], 2) image-toimage translation [2,10,12,14,17,25,31,40] and 3) self-training [1,15,18,19,24,36,43,47,49,53].…”