Budget-Aware Adapters for Multi-Domain Learning

Abstract

Multi-Domain Learning (MDL) refers to the problem of learning a set of models derived from a common deep architecture, each one specialized to perform a task in a certain domain (e.g. photos, sketches, paintings). This paper tackles MDL with a particular interest in obtaining domain-specific models with an adjustable budget in terms of the number of network parameters and computational complexity. Our intuition is that, as in real applications the number of domains and tasks can be very large, an effective MDL approach should not only focus on accuracy but also on having as few parameters as possible. To implement this idea we derive specialized deep models for each domain by adapting a pre-trained architecture but, differently from other methods, we propose a novel strategy to automatically adjust the computational complexity of the network. To this aim, we introduce Budget-Aware Adapters that select the most relevant feature channels to better handle data from a novel domain. Some constraints on the number of active switches are imposed in order to obtain a network respecting the desired complexity budget. Experimentally, we show that our approach leads to recognition accuracy competitive with state-of-the-art approaches but with much lighter networks both in terms of storage and computation. * Work done when with MHUG (DISI, University of Trento). 2

Boosting binary masks for multi-domain learning through affine transformations

“…Since including batch-normalization layers affects the performances, for the sake of presenting a fair comparison, we report also the results of Piggyback [43] obtained as a special case of our model with separate BN parameters per domain for ResNet-50 and DenseNet-121. Moreover, we report the results of the Budget-Aware adapters (BA 2 ) method in [4]. This method relies on binary masks applied not per-parameter but per-channel, with a budget constraint allowing to further squeeze the network complexity.…”

“…This method relies on binary masks applied not per-parameter but per-channel, with a budget constraint allowing to further squeeze the network complexity. As in our method, also in [4] domain-specific BN layers are used.…”

“…In this section, we report the results obtained on the Visual Decathlon Challenge. We compare our model with the baseline method Piggyback [43] (PB), the budget-aware adapters of [4] (BA 2 ), the improved version of the winning entry of the 2017 edition of the challenge [58] (DAN), the network with domain-specific parallel adapters [56] (PA), the domain-specific attention modules of [39] (MTAN), the covariance normalization approach [35] (CovNorm) and SpotTune [19]. We additionally report the baselines proposed by the authors of the challenge [55].…”

“…This score function takes into account the performances of a model on all 10 classes, preferring models with good performances on all of them compared to models outperforming by a large margin the baseline in just a few. Following [4], we use this metric also for the first benchmark, keeping the same upper-bound of 1000 points for each domain. Moreover, as in [4], we report the ratio among the score obtained and the parameters used, denoting it as S p .…”

“…Following [4], we use this metric also for the first benchmark, keeping the same upper-bound of 1000 points for each domain. Moreover, as in [4], we report the ratio among the score obtained and the parameters used, denoting it as S p . This metric allows capturing the trade-off among the performances and model size.…”