We present a general PAC-Bayes theorem from which all known PAC-Bayes risk bounds are obtained as particular cases. We also propose different learning algorithms for finding linear classifiers that minimize these bounds. These learning algorithms are generally competitive with both AdaBoost and the SVM.

We propose a novel active learning strategy based on the compression framework of [9] for label ranking functions which, given an input instance, predict a total order over a predefined set of alternatives. Our approach is theoretically motivated by an extension to ranking and active learning of Kääriäinen's generalization bounds using unlabeled data [7], initially developed in the context of classification. The bounds we obtain suggest a selective sampling strategy provided that a sufficiently, yet reasonably large initial labeled dataset is provided. Experiments on Information Retrieval corpora from automatic text summarization and question/answering show that the proposed approach allows to substantially reduce the labeling effort in comparison to random and heuristic-based sampling strategies.

We propose algorithms for producing weighted majority votes that learn by probing the empirical risk of a randomized (uniformly weighted) majority vote-instead of probing the zero-one loss, at some margin level, of the deterministic weighted majority vote as it is often proposed. The learning algorithms minimize a risk bound which is convex in the weights. Our numerical results indicate that learners producing a weighted majority vote based on the empirical risk of the randomized majority vote at some finite margin have no significant advantage over learners that achieve this same task based on the empirical risk at zero margin. We also find that it is sufficient for learners to minimize only the empirical risk of the randomized majority vote at a fixed number of voters without considering explicitly the entropy of the distribution of voters. Finally, our extensive numerical results indicate that the proposed learning algorithms are producing weighted majority votes that generally compare favorably to those produced by AdaBoost.

We propose new PAC-Bayes bounds for the risk of the weighted majority vote that depend on the mean and variance of the error of its associated Gibbs classifier. We show that these bounds can be smaller than the risk of the Gibbs classifier and can be arbitrarily close to zero even if the risk of the Gibbs classifier is close to 1/2. Moreover, we show that these bounds can be uniformly estimated on the training data for all possible posteriors Q. Moreover, they can be improved by using a large sample of unlabelled data.

We provide a PAC-Bayesian bound for the expected loss of convex combinations of classifiers under a wide class of loss functions (which includes the exponential loss and the logistic loss). Our numerical experiments with Adaboost indicate that the proposed upper bound, computed on the training set, behaves very similarly as the true loss estimated on the testing set.