PAC-Bayesian Analysis for a Two-Step Hierarchical Multiview Learning Approach

Abstract

We study a two-level multiview learning with more than two views under the PAC-Bayesian framework. This approach, sometimes referred as late fusion, consists in learning sequentially multiple view-specific classifiers at the first level, and then combining these view-specific classifiers at the second level. Our main theoretical result is a generalization bound on the risk of the majority vote which exhibits a term of diversity in the predictions of the view-specific classifiers. From this result it comes out that controlling the trade-off between diversity and accuracy is a key element for multiview learning, which complements other results in multiview learning. Finally, we experiment our principle on multiview datasets extracted from the Reuters RCV1/RCV2 collection. 1 The fusion of descriptions, resp. of models, is sometimes called Early Fusion, resp. Late Fusion. 2 Our notion of hyper-prior and hyper-posterior distributions is different than the one proposed for lifelong learning [Pentina and Lampert, 2014], where they basically consider hyper-prior and hyper-posterior over the set of possible priors: The prior distribution P over the voters' set is viewed as a random variable.

PAC-Bayes and domain adaptation

“…The question of finding a relevant prior in a domain adaptation situation is an exciting direction which could also be exploited when some few target labels are available. Moreover, this notion of prior distribution could model information learned from previous tasks as pointed out by Pentina and Lampert [61], or from other views/representations of the data [62]. This suggests that we can extend our analyses to multisource domain adaptation [63,64,27,65] and lifelong learning where the objective is to perform well on future tasks, for which no data has been observed so far [66].…”

Multiview Boosting by Controlling the Diversity and the Accuracy of View-specific Voters

“…In this section, we give a general multiview PAC-Bayesian theorem Goyal et al [2017] that takes the form of a generalization bound for the Gibbs risk in the context of a two-level hierarchy of distributions. A key step in PAC-Bayesian proofs is the use of a change of measure inequality [McAllester, 2003], based on the Donsker-Varadhan inequality [Donsker and Varadhan, 1975].…”

“…Since the classes are unbalanced, we report the accuracy along with F1-measure for the methods and all the scores are averaged over all the one-vs-all classification problems. We consider two multiview learning algorithms based on our two-step hierarchical strategy, and compare the PB-MVBoost algorithm described in Section 4, with a previously developed multiview learning algorithm Goyal et al [2017], based on classifier late fusion approach Snoek et al [2005], and referred to as Fusion all Cq . Concretely, at the first level, this algorithm trains different view-specific linear SVM models with different hyperparameter C values (12 values between 10 −8 and 10 3 ).…”

“…In the next section, we discuss some related works. In Section 3, we present the PAC-Bayesian multiview learning framework Goyal et al [2017]. In Section 4, we derive our multiview learning algorithm PB-MVBoost.…”

Fast and Provably Effective Multi-view Classification with Landmark-Based SVM

“…Basically, they train multiple view-specific models either by alternatively optimizing them, "teaching" one another, or by fostering their smoothness in predictions. The final step of such techniques consists in aggregating the predictions of the view-specific classifiers, for instance by majority vote [21,11] or by weighted majority vote [19,13]. Note that these methods usually face the following issues: their performances are degraded by the computational overload of training and testing multiple learners; also, by usually making the assumption that the views' common information is the only worth keeping, they boil down to denoising the single views from their uncorrelated information.…”