Robust Domain Adaptation: Representations, Weights and Inductive Bias

On the inductive biases of deep domain adaptation

“…Fact 2: the feature extractor can be harmful. This fact has been highlighted by at least three previous publications [8,9,10] in a form of a no-free-lunch theorem. The gist of this result relies on the observation that while the first two terms of the upper-bound are minimized, the third one can go out of control.…”

“…It is therefore clear, that other explanations should be put forward to understand what makes a transfer amenable to a favorable alignment. In this article, we elaborate on the role of (hidden) inductive biases, as advocated by [10].…”

“…However, further analyses of [8,9,10] have shown that practical domain-alignment algorithms are in fact largely inconsistent with the theory from which they claim to be derived and may even be counterproductive in some cases. Even if alternative methods [11,12] begin to emerge, domainalignment remains prevalent in the literature and gives decent results on various transfers.…”

“…An inductive bias is a set of hidden assumptions that condition the behavior of the model on unseen data. Although [10] has already introduced the notion of inductive bias for domain alignment, this paper is to go further by giving a more comprehensive presentation of existing biases. Finally, as a complement to this expository work, we also propose a strategy to establish such inductive biases by meta-learning.…”