Semi-supervised Models are Strong Unsupervised Domain Adaptation Learners

Zhang, Yabin; Zhang, Haojian; Deng, Bin; Li, Shuai; Jia, Kui; Zhang, Lei

doi:10.48550/arxiv.2106.00417

Cited by 7 publications

(9 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Inspired by some previous semi-supervised domain adaptation methods [33,35], we also add FixMatch to GVB, and it can achieve remarkable performance improvement. Such results are consistent with [77] which reveals that the semi-supervised models are strong unsupervised domain adaptation learners. We evaluate our proposed method by applying PCL it to GVB and GVB with Fix-Match.…”

Section: Results On Unsupervised Domain Adaptationsupporting

confidence: 88%

See 1 more Smart Citation

Probabilistic Contrastive Learning for Domain Adaptation

Li¹,

Zhang²,

Wang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Recent feature contrastive learning (FCL) has shown promising performance in self-supervised representation learning. For domain adaptation, however, FCL cannot show overwhelming gains since the class weights are not involved during optimization, which does not guarantee the produced features to be clustered around the class weights learned from source data. To tackle this issue, we propose a novel probability contrastive learning (PCL) in this paper, which not only produces compact features but also enforces them to be distributed around the class weights. Specifically, we propose to use the output probabilities after softmax to perform contrastive learning instead of the extracted features and remove the 2 normalization in the traditional FCL. In this way, the probability will approximate the onehot form, thereby narrowing the distance between the features and the class weights. Our proposed PCL is simple and effective. We conduct extensive experiments on two domain adaptation tasks, i.e., unsupervised domain adaptation and semi-supervised domain adaptation. The results on multiple datasets demonstrate that our PCL can consistently get considerable gains and achieves the state-of-theart performance. In addition, our method also obtains considerable gains on semi-supervised tasks when labeled data is scarce.

show abstract

Section: Results On Unsupervised Domain Adaptationsupporting

confidence: 88%

“…It contains 152,397 synthetic images for the source domain and 55,388 real-world images for the target domain. Following the standard transductive setting [77] for UDA, we use all labeled source data and all unlabeled target data, and test on the same unlabeled target data. Experimental Details.…”

Section: Results On Unsupervised Domain Adaptationmentioning

confidence: 99%

Probabilistic Contrastive Learning for Domain Adaptation

Li¹,

Zhang²,

Wang³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…In fact, top performing approaches from the VisDA2018 and VisDA2019 competitions use semi-supervised learning techniques explicitly in their UDA approaches. 11,12 Most similar to our work, Zhang et.al 13 recently demonstrated that SSL algorithms on their own and in combination with modern UDA techniques are strong baselines for UDA problems, when compared to UDA specific algorithms. Berthelot et.al 10 explicitly unify SSL and UDA with the AdaMatch algorithm which extends prior SSL work, FixMatch, 14 to SSDA applications by adding additional steps to the FixMatch training process, including "logit interpolation, distribution alignment, and a relative confidence threshold".…”

Section: Background and Related Worksupporting

confidence: 76%

Semi-supervised domain transfer for robust maritime satellite image classification

Emmanuel

Ratto

Drenkow

et al. 2023

Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications V

View full text Add to dashboard Cite

Naval intelligence plays a critical role in multi-domain operations by identifying and tracking vessels of interest, especially suspected "dark ships" operating in an emissions-controlled (EMCON) state. While applying machine learning (ML) to maritime satellite imagery could enable an automated open-ocean search capability for dark ships, ensuring the robustness of ML models to environmental variations in the maritime domain remains a challenge because training sets do not encapsulate all possible environmental conditions. To address the challenge of unsupervised domain adaptation (UDA) in ship classification, i.e. transferring a ML model from a labeled source domain to an unlabeled target domain, we propose employing combinations of semi-supervised learning (SSL) techniques with standalone UDA approaches. Specifically, we incorporate combinations of FixMatch, minimum class confusion, gradient reversal, and mixup augmentation into the standard cross-entropy supervised loss function. These interventions were compared in two domain shift settings, one in which the source and target domains are both comprised of simulated data, and another in which the source domain consists of only simulated data, and the target domain consists of only real data. Experimental results comparing the combinations of interventions to a regularized fine-tuning baseline demonstrate that the greatest improvements in model robustness were achieved when combinations of our SSL strategy (FixMatch) and UDA algorithms were incorporated into training.However, ensuring the robustness of ML models for maritime imagery remains a challenge. We define "robustness" as the performance of a ML model on data generated from a counterfactually-altered version of the data-generating process. 5 In maritime imagery, the data-generating process is defined by numerous environmental and operating factors that produce non-i.i.d. data. For instance, the brightness and texture of the ocean's surface under a clear sky is a complex function of the viewing angle, solar angle, and wind speed. 6 Collecting training data over the full range of conditions can be difficult, so datasets may be biased toward ocean conditions with clear skies and calm seas. Furthermore, the observable features of a ship and its wake can change as a function its velocity relative to the waves, sun, and viewpoint, 7 and collected datasets may not be able to sufficiently sample this space.A potential "robustness tactic", 5 or intervention, to improve maritime image classifier robustness is to employ synthetic training data that accommodates physics-based models for the relevant optics and ocean dynamics that define the data-generating process. However, even the most sophisticated models cannot employ all of the relevant factors that impact the imagery (e.g. clouds, biologics). Naively training on simulated data can impose a different bias in the ML model than if it was trained on "cherry-picked" real-world data. Recently, many unsupervised domain adaptation (UDA) techniques 8,9 have been propos...

show abstract

“…Inspired by previous works [32,88,93] in domain adaptation, we add FixMatch [63] to the existing method to construct a stronger baseline. Specifically, let T (x) and T (x) denote the weakly and strongly augmented views for x ∈ D tu , respectively.…”

Section: Methods 41 a Stronger Baselinementioning

confidence: 99%

Low-confidence Samples Matter for Domain Adaptation

Zhang¹,

Li²,

Wang³

2022

Preprint

View full text Add to dashboard Cite

Domain adaptation (DA) aims to transfer knowledge from a label-rich source domain to a related but labelscarce target domain. The conventional DA strategy is to align the feature distributions of the two domains. Recently, increasing researches have focused on self-training or other semi-supervised algorithms to explore the data structure of the target domain. However, the bulk of them depend largely on confident samples in order to build reliable pseudo labels, prototypes or cluster centers. Representing the target data structure in such a way would overlook the huge low-confidence samples, resulting in suboptimal transferability that is biased towards the samples similar to the source domain. To overcome this issue, we propose a novel contrastive learning method by processing low-confidence samples, which encourages the model to make use of the target data structure through the instance discrimination process. To be specific, we create positive and negative pairs only using low-confidence samples, and then re-represent the original features with the classifier weights rather than directly utilizing them, which can better encode the task-specific semantic information. Furthermore, we combine cross-domain mixup to augment the proposed contrastive loss. Consequently, the domain gap can be well bridged through contrastive learning of intermediate representations across domains. We evaluate the proposed method in both unsupervised and semi-supervised DA settings, and extensive experimental results on benchmarks reveal that our method is effective and achieves state-ofthe-art performance. The code can be found in https: //github.com/zhyx12/MixLRCo.

show abstract

Semi-supervised Models are Strong Unsupervised Domain Adaptation Learners

Cited by 7 publications

References 32 publications

Probabilistic Contrastive Learning for Domain Adaptation

Probabilistic Contrastive Learning for Domain Adaptation

Semi-supervised domain transfer for robust maritime satellite image classification

Low-confidence Samples Matter for Domain Adaptation

Contact Info

Product

Resources

About