Deviation optimal learning using greedy $Q$-aggregation

Abstract

Given a finite family of functions, the goal of model selection aggregation is to construct a procedure that mimics the function from this family that is the closest to an unknown regression function. More precisely, we consider a general regression model with fixed design and measure the distance between functions by the mean squared error at the design points. While procedures based on exponential weights are known to solve the problem of model selection aggregation in expectation, they are, surprisingly, sub-optimal in deviation. We propose a new formulation called Q-aggregation that addresses this limitation; namely, its solution leads to sharp oracle inequalities that are optimal in a minimax sense. Moreover, based on the new formulation, we design greedy Q-aggregation procedures that produce sparse aggregation models achieving the optimal rate. The convergence and performance of these greedy procedures are illustrated and compared with other standard methods on simulated examples.

Optimal exponential bounds for aggregation of estimators for the Kullback-Leibler loss

“…For the fixed design regression setting, [25] considers all three aggregation problems in the context of generalized linear models and gives constrained likelihood maximization methods which are optimal in both expectation and deviation with respect to the Kullback-Leibler loss. More recently, [12] extends the results of [25] for model selection by introducing the Q-aggregation method and giving a greedy algorithm which produces a sparse aggregate achieving the optimal rate in deviation for the L 2 loss. More general properties of this method applied to other aggregation problems as well are discussed in [11].…”

PAC-Bayesian Aggregation of Affine Estimators

“…Proposition 1 (Dai et al [17]). If pen(t) = 2σ 2 Tr(A t ), and β ≥ 4σ 2 16, then for all η > 0, with probability at least 1 − η,…”

“…Indeed, the Exponential Weighting Aggregation is not optimal anymore in probability. Dai et al [16] have indeed proved the sub-optimality in deviation of exponential weighting, not allowing to obtain a sharp oracle inequality in probability. Under strong assumptions and independent noise, Bellec [4] provides a sharp oracle inequality with optimal rate for another aggregation procedure called Q-aggregation.…”

Solution of linear ill-posed problems by model selection and aggregation

“…The first estimator is obtained by minimizing penalized empirical risk with a penalty on model M proportional to j∈M ψ 2 j . The second one is based on a Q-aggregation type procedure that is specifically designed for solution of linear ill-posed problems and is different from that of Dai et al (2012) developed for solution of direct problems. We establish oracle inequalities for both estimators that hold with high probabilities and in expectation.…”

“…Dai et al (2012) considered Pen(θ) proportional to the Kullback-Leibler divergence KL(θ, π) for some prior π on Θ M . For direct problems, they derived sharp oracle inequalities both in expectation and with high probability for Q-aggregation with such penalty.…”