Hyper-sparse optimal aggregation

Abstract

In this paper, we consider the problem of hyper-sparse aggregation. Namely, given a dictionary F = {f1, . . . , fM } of functions, we look for an optimal aggregation algorithm that writes f = P M j=1 θjfj with as many zero coefficients θj as possible. This problem is of particular interest when F contains many irrelevant functions that should not appear in f . We provide an exact oracle inequality for f , where only two coefficients are non-zero, that entails f to be an optimal aggregation algorithm. Since selectors are suboptimal aggregation procedures, this proves that 2 is the minimal number of elements of F required for the construction of an optimal aggregation procedures in every situations. A simulated example of this algorithm is proposed on a dictionary obtained using LARS, for the problem of selection of the regularization parameter of the LASSO. We also give an example of use of aggregation to achieve minimax adaptation over anisotropic Besov spaces, which was not previously known in minimax theory (in regression on a random design).

Mirror averaging with sparsity priors

“…Before closing this section, let us mention the recent work [26], where some interesting results on the aggregation of estimators in sparse regression are obtained.…”

“…In view of (26), for any prior π and any β > 0 the MA aggregate f n satisfies the inequality Using the Taylor formula, we get…”