PAC-Bayesian learning of linear classifiers

Germain, Pascal; Lacasse, Alexandre; Laviolette, François; Marchand, Mario

doi:10.1145/1553374.1553419

Cited by 132 publications

(207 citation statements)

References 6 publications

Supporting

Mentioning

198

Contrasting

Unclassified

Order By: Relevance

“…Many other ways to choose h t could be studied, namely a combination of a subset of weak classifiers, or the choice between the weak classifier of the major view and a combination of the classifiers on minor views, etc. Hence, many alternate selections deserve to be studied, both theoretically (for example, in the PAC-Bayes framework [23]) and empirically.…”

Section: Discussion and Improvementsmentioning

confidence: 99%

A Boosting Approach to Multiview Classification with Cooperation

Koço

Capponi

2011

Machine Learning and Knowledge Discovery in Databases

View full text Add to dashboard Cite

Abstract. In many fields, such as bioinformatics or multimedia, data may be described using different sets of features (or views) which carry either global or local information. Some learning tasks make use of these several views in order to improve overall predictive power of classifiers through fusion-based methods. Usually, these approaches rely on a weighted combination of classifiers (or selected descriptions), where classifiers are learned independently. One drawback of these methods is that the classifier learned on one view does not communicate its failures within the other views. This paper deals with a novel approach to integrate multiview information. The proposed algorithm, named Mumbo, is based on boosting. Within the boosting scheme, Mumbo maintains one distribution of examples on each view, and at each round, it learns one weak classifier on each view. Within a view, the distribution of examples evolves both with the ability of the dedicated classifier to deal with examples of the corresponding features space, and with the ability of classifiers in other views to process the same examples within their own description spaces. Hence, the principle is to slightly remove the hard examples from the learning space of one view, while their weights get higher in the other views. This way, we expect that examples are urged to be processed by the most appropriate views, when possible. At the end of the iterative learning process, a final classifier is computed by a weighted combination of selected weak classifiers.This paper provides the Mumbo algorithm in a multiclass and multiview setting, based on recent theoretical advances in boosting. The boosting properties of Mumbo are proved, as well as some results on its generalization capabilities. Several experimental results are reported which point out that complementary views may actually cooperate under some assumptions.

show abstract

Section: Discussion and Improvementsmentioning

confidence: 99%

A Boosting Approach to Multiview Classification with Cooperation

Koço

Capponi

2011

Machine Learning and Knowledge Discovery in Databases

View full text Add to dashboard Cite

show abstract

“…PAC-Bayes (McAllester 1999;Seeger 2002;McAllester 2003;Langford 2006;Lacasse et al 2006;Germain et al 2009;Seldin et al 2012;Tolstikhin and Seldin 2013) is a theory for bounding the generalization error of classifiers. A variety of PAC-Bayes generalization bounds (McAllester 1999;Seeger 2002;McAllester 2003;Langford 2006;Lacasse et al 2006;Germain et al 2009;Seldin et al 2012;Tolstikhin and Seldin 2013) have been proposed for different classifiers such as deterministic classifiers, Gibbs classifiers (McAllester 1999), linear classifiers or nonlinear classifiers (e.g.…”

Section: Pac-bayes Generalization Boundsmentioning

confidence: 99%

“…PAC-Bayes theory (McAllester 1999;Seeger 2002;McAllester 2003;Langford 2006;Lacasse et al 2006;Germain et al 2009;Seldin et al 2012;Tolstikhin and Seldin 2013) potentially can provide a framework to learn feature mappings and classifiers jointly, allowing the fine tuning of feature mapping. PAC-Bayes is a theory proposed to bound the generalization error of classifiers, where classifiers are learned by minimizing the generalization bound with respect to the parameters of the classifiers over the training set.…”

Section: Introductionmentioning

confidence: 99%

“…In this paper, we propose an approach based on the PAC-Bayes theory (McAllester 1999;Seeger 2002;McAllester 2003;Langford 2006;Lacasse et al 2006;Germain et al 2009;Seldin et al 2012;Tolstikhin and Seldin 2013) to integrate the complementary strengths of generative and discriminative models. First we derive a stochastic feature mapping which is a function over the observed and hidden variables of generative models.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Stochastic feature mapping for PAC-Bayes classification

Wang

Liu

et al. 2015

Mach Learn

View full text Add to dashboard Cite

Hidden information derived from probabilistic generative models of data distributions can be used to construct features for discriminative classifiers. This observation has motivated the development of approaches that attempt to couple generative and discriminative models together for classification. However, existing approaches typically feed features derived from generative models to discriminative classifiers, and do not refine the generative models or the feature mapping functions based on classification results. In this paper, we propose a coupling mechanism developed under the PAC-Bayes framework that can fine-tune the generative models and the feature mapping functions iteratively to improve the classifier's performance. In our approach, a stochastic feature mapping, which is a function over the random variables of a generative model, is derived to generate feature vectors for a stochastic classifier. We construct a stochastic classifier over the feature mapping and derive the PAC-Bayes generalization bound for the classifier, for both supervised and semi-supervised learning. This allows us to jointly learn the feature mapping and the classifier by minimizing the bound with an EM-like iterative algorithm using labeled and unlabeled data. The resulting framework integrates the learning of the discriminative classifier and the generative model and allows iterative fine-tuning of the generative models, and the feedforward feature mappings based on task performance feedback. Our experiments show, in three distinct applications, this new framework produces a general classification tool with state-of-the-art performance.

show abstract

“…Germain et al [7] recently show a simplified PAC-Bayes generalization proof technique for linear classifiers in a more general setting.…”

Section: Generalization Errormentioning

confidence: 99%

Human Action Segmentation and Recognition Using Discriminative Semi-Markov Models

et al. 2010

View full text Add to dashboard Cite

A challenging problem in human action understanding is to jointly segment and recognize human actions from an unseen video sequence, where one person performs a sequence of continuous actions.In this paper, we propose a discriminative semiMarkov model approach, and define a set of features over boundary frames, segments as well as neighboring segments. This enable us to conveniently capture a combination of local and global features that best represent a specific action type. To efficiently solve the inference problem of simultaneously segmentation and recognition, we devise a Viterbi-like dynamic programming algorithm, which is able to process 20 frames per second in practice. Moreover, the model is discriminatively learned from large margin principle, and is formulated as an optimization problem with exponentially many constraints. To solve it efficiently, we present two different optimization algorithms, namely cutting plane method and bundle method, and demonstrate that each can be alternatively deployed in a "plug and play" fashion. From its theoretical aspect, we also analyze the generalization error of the proposed approach and provide a PAC-Bayes bound.A preliminary version has been published at [28].Qingfeng Shi NICTA and ANU, Canberra, Australia E-mail: qinfeng.shi@rsise.anu.edu.au Li Wang Southeast University, Nanjing, China E-mail: wang.li.seu.nj@gmail.com Li Cheng TTI-Chicago, USA E-mail: licheng@tti-c.orgThe proposed approach is evaluated on a variety of datasets, and is shown to perform competitively to the state-of-the-art methods. For example, on KTH dataset, it achieves 95% ± 0.01 recognition accuracy, where the best known result on this dataset is 92% ± 0.03 [8].

show abstract

PAC-Bayesian learning of linear classifiers

Cited by 132 publications

References 6 publications

A Boosting Approach to Multiview Classification with Cooperation

A Boosting Approach to Multiview Classification with Cooperation

Stochastic feature mapping for PAC-Bayes classification

Human Action Segmentation and Recognition Using Discriminative Semi-Markov Models

Contact Info

Product

Resources

About