Prototypical Cross-domain Self-supervised Learning for Few-shot Unsupervised Domain Adaptation

Abstract

Unsupervised Domain Adaptation (UDA) transfers predictive models from a fully-labeled source domain to an unlabeled target domain. In some applications, however, it is expensive even to collect labels in the source domain, making most previous works impractical. To cope with this problem, recent work performed instance-wise cross-domain self-supervised learning, followed by an additional fine-tuning stage. However, the instance-wise selfsupervised learning only learns and aligns low-level discriminative features. In this paper, we propose an endto-end Prototypical Cross-domain Self-Supervised Learning (PCS) framework for Few-shot Unsupervised Domain Adaptation (FUDA) 1 . PCS not only performs cross-domain low-level feature alignment, but it also encodes and aligns semantic structures in the shared embedding space across domains. Our framework captures category-wise semantic structures of the data by in-domain prototypical contrastive learning; and performs feature alignment through cross-domain prototypical self-supervision. Compared with state-of-the-art methods, PCS improves the mean classification accuracy over different domain pairs on FUDA by 10.

Multi-source Few-shot Domain Adaptation

“…To address this task, we propose to learn the latent feature-space clustering of each source and target domain, and align clusters with the same category across different domains in a self-supervised manner. Specifically, we use a ProtoNCE [39] loss to learn the semantic feature of a single domain as it has been shown to semantically align data across a single source and target domain [33]. We further extend it into the multi-source adaptation scenario to learn better discriminative and domain-invariant features across all domains.…”

“…To learn a well-clustered semantic structure in the feature space, it is problematic to apply ProtoNCE on a mixed dataset with different distributions, because images of different categories from different domains may be incorrectly aggregated into the same cluster. As a result, due to the domain shift between sources and target, we cannot directly apply ProtoNCE to M i=1 (S i ∪ S u i ) ∪ T as in [39], and due to the domain shift among different sources, we cannot apply ProtoNCE to the source M i=1 (S i ∪ S u i ) and target T separately as [33]. Instead, we perform prototypical contrastive learning in each source S i ∪ S u i ) and target T .…”

“…Recently, self-supervised learning methods [35,33] have been proposed to perform domain alignment between two domains. One trivial extension is combining all source domains first, and then perform domain alignment between M i=1 S i ∪ S u i and T .…”

“…However, during the early stages of the pandemic, there were few domain experts that could provide such annotations and even obtaining fully labeled source data was impractical. As another example, each retinal image in the Diabetic Retinopathy dataset [32] is annotated by a panel of 7+ U.S. board-certified ophthalmologists, with a total group of 54 doctors used for annotation [32,33]. Thus it is practically too stringent to assume the availability of abundant labels across domains.…”

“…In MFDA, however, with a limited number of labels in each source, it is much harder to learn discriminative features for both source and target. Some recent works [35,33] perform few-shot adaptation with a single source, and in this work, we build upon these contributions and investigate the multiple source scenario.…”