Self-Adaptation for Unsupervised Domain Adaptation

Abstract

Lack of labelled data in the target domain for training is a common problem in domain adaptation. To overcome this problem, we propose a novel unsupervised domain adaptation method that combines projection and self-training based approaches. Using the labelled data from the source domain, we first learn a projection that maximises the distance among the nearest neighbours with opposite labels in the source domain. Next, we project the source domain labelled data using the learnt projection and train a classifier for the target class prediction. We then use the trained classifier to predict pseudo labels for the target domain unlabelled data. Finally, we learn a projection for the target domain as we did for the source domain using the pseudo-labelled target domain data, where we maximise the distance between nearest neighbours having opposite pseudo labels. Experiments on a standard benchmark dataset for domain adaptation show that the proposed method consistently outperforms numerous baselines and returns competitive results comparable to that of SOTA including self-training, tri-training, and neural adaptations.

Neural Unsupervised Domain Adaptation in NLP---A Survey

“…The two approaches differ in the specifics of the method to construct the shared space. SCL uses auxiliary functions inspired by (Ando and Zhang, 2005) AdaptaBERT (Li et al, 2019) background embeddings (Gururangan et al, 2020) multi-phase pre-training Hybrid: (Ruder and Plank, 2018) Mt-Tri (Jia et al, 2019) cross-domain LM (Rotman and Reichart, 2019) deep self-training (Cui and Bollegala, 2019) SelfAdapt (Peng and Dredze, 2017) MTL-DA (Guo et al, 2020) DistanceNet-Bandit domain-specific and domain-general features is the key idea of EasyAdapt (Daum III, 2007) a seminal supervised DA method. A recent line of work (Ziser and Reichart, 2017;Ziser and Reichart, 2018a;Ziser and Reichart, 2018b;Ziser and Reichart, 2019) brings SCL back to neural networks.…”

“…Data-centric methods focus on the data aspect and either involve pseudo-labeling (or bootstrapping) to bridge the domain gap, e.g. (Abney, 2007;Zhu and Goldberg, 2009;Ruder and Plank, 2018;Cui and Bollegala, 2019), data selection, e.g. (Axelrod et al, 2011;Plank and van Noord, 2011;Ruder and Plank, 2017) and pre-training methods, e.g.…”

“…Work on the intersection of data-centric and model-centric methods can be plentiful. It currently includes combining semi-supervised objectives with an adversarial loss (Lim et al, 2020;Alam et al, 2018b), combining pivot-based approaches with pseudo-labeling (Cui and Bollegala, 2019), or work that uses multi-task approaches, e.g., to combine in-domain data with auxiliary tasks such as language modeling (Jia et al, 2019). Depending on the specifics, the auxiliary-task pre-training methods discussed in Section 5.3 can be seen as a form of hybrid approach as well.…”

Neural Unsupervised Domain Adaptation in NLP—A Survey

“…Work on the intersection of data-centric and model-centric methods can be plentiful. It currently includes combining semi-supervised objectives with an adversarial loss (Lim et al, 2020;Alam et al, 2018b), combining pivot-based approaches with pseudo-labeling (Cui and Bollegala, 2019) and very recently with contextualized word embeddings (Ben-David et al, 2020), and combining multi-task approaches with domain shift (Jia et al, 2019), multi-task learning with pseudo-labeling (multi-task tritraining) (Ruder and Plank, 2018), and adaptive ensembling (Desai et al, 2019), which uses a studentteacher network with a consistency-based self-ensembling loss and a temporal curriculum. They apply adaptive ensembling to study temporal and topic drift in political data classification (Desai et al, 2019).…”

“…Model-centric methods target approaches to augment the feature space, alter the loss function, the architecture or model parameters (Blitzer et al, 2006;Pan et al, 2010;Ganin et al, 2016). Data-centric methods focus on the data aspect and either involve pseudo-labeling (or bootstrapping) to bridge the domain gap (Abney, 2007;Zhu and Goldberg, 2009;Ruder and Plank, 2018;Cui and Bollegala, 2019), data selection (Axelrod et al, 2011;Plank and van Noord, 2011;Ruder and Plank, 2017) and pre-training methods (Han and Eisenstein, 2019;Guo et al, 2020). As some approaches take elements of both, we include a hybrid category.…”