Transfer Learning with Dynamic Distribution Adaptation

Abstract

Transfer learning aims to learn robust classifiers for the target domain by leveraging knowledge from a source domain. Since the source and the target domains are usually from different distributions, existing methods mainly focus on adapting the cross-domain marginal or conditional distributions. However, in real applications, the marginal and conditional distributions usually have different contributions to the domain discrepancy. Existing methods fail to quantitatively evaluate the different importance of these two distributions, which will result in unsatisfactory transfer performance. In this paper, we propose a novel concept called Dynamic Distribution Adaptation (DDA), which is capable of quantitatively evaluating the relative importance of each distribution. DDA can be easily incorporated into the framework of structural risk minimization to solve transfer learning problems. On the basis of DDA, we propose two novel learning algorithms: (1) Manifold Dynamic Distribution Adaptation (MDDA) for traditional transfer learning, and (2) Dynamic Distribution Adaptation Network (DDAN) for deep transfer learning. Extensive experiments demonstrate that MDDA and DDAN significantly improve the transfer learning performance and setup a strong baseline over the latest deep and adversarial methods on digits recognition, sentiment analysis, and image classification. More importantly, it is shown that marginal and conditional distributions have different contributions to the domain divergence, and our DDA is able to provide good quantitative evaluation of their relative importance which leads to better performance. We believe this observation can be helpful for future research in transfer learning.

Sketching an AI Marketplace: Tech, Economic, and Regulatory Aspects

“…Furthermore, the original training dataset provided by the customer may itself be from multiple private sources (e.g., mobile crowdsensing) and may follow some multimodal distribution. The AI marketplace should also be able to help the customer by allowing the aggregation of multiple alternative datasets from other data owners in the marketplace while also ensuring the aggregate dataset follows a similar distribution (e.g., using a transfer learning approach [18]) as the validation dataset. This is important since AI learning algorithms suffer from major model quality loss (or even divergence) when trained on non-IID data [19].…”

Cross Domain Mean Approximation for Unsupervised Domain Adaptation

“…Joint Distribution Analysis (JDA) [27] improves TCA by considering not only the marginal distribution shift but also the conditional distribution shift with the pseudo-labels of the target domain. Wang et al [53,54] improves JDA by adaptively leveraging the marginal and conditional distributions. Li et al [55][56][57] adopted MMD to eliminate the discrepancy of features and distributions between the source and target domains under the heterogeneous domain adaptation.…”

Transfer Learning for Wireless Fingerprinting Localization Based on Optimal Transport

“…General feature-based transfer learning methods include Transfer Component Analysis (TCA) [27], Joint Distribution Adaptation (JDA) [28], Balanced boundary Distribution Adaptation (BDA) [29], etc. These methods are types of unsupervised transfer learning, where there is no label information in the target domain.…”

“…The results can be constructed with dimension reduction method, that is, solve the first m eigenvalues of (KLK + µI) −1 KHK, reducing the computational cost of solving SDP. The TCA method is the base method in feature-based transfer learning, many other methods are extended upon it, such as JDA [28] and BDA [29].…”