Elastic-net regularization in learning theory

Abstract

a b s t r a c tWithin the framework of statistical learning theory we analyze in detail the so-called elastic-net regularization scheme proposed by Zou and Hastie [H. Zou, T. Hastie, Regularization and variable selection via the elastic net, J. R. Stat. Soc. Ser. B, 67(2) (2005) 301-320] for the selection of groups of correlated variables. To investigate the statistical properties of this scheme and in particular its consistency properties, we set up a suitable mathematical framework. Our setting is random-design regression where we allow the response variable to be vector-valued and we consider prediction functions which are linear combinations of elements (features) in an infinite-dimensional dictionary. Under the assumption that the regression function admits a sparse representation on the dictionary, we prove that there exists a particular ''elastic-net representation'' of the regression function such that, if the number of data increases, the elastic-net estimator is consistent not only for prediction but also for variable/feature selection. Our results include finite-sample bounds and an adaptive scheme to select the regularization parameter. Moreover, using convex analysis tools, we derive an iterative thresholding algorithm for computing the elastic-net solution which is different from the optimization procedure originally proposed in the above-cited work.

Concentration estimates for learning with unbounded sampling

“…Most results are stated under the standard uniform boundedness assumption for the output that for some constant M > 0, |y| ≤ M almost surely. This standard assumption is abandoned in [5,9,22]. In [5,9], it is assumed that the output satisfies the condition…”

Sparse Learning of the Disease Severity Score for High-Dimensional Data

“…In fact, since the model imposes both 1 and 2 norms on the feature vector , it resembles the elastic net regularization [13], which has an advantage of achieving higher stability with respect to random sampling [14].…”

Adaptive Kernel Methods Using the Balancing Principle

“…This latter case is of interest for estimators in RKHS since estimates in various other norms, e.g., uniform norm or Sobolev norms, can be easily obtainedsee [35]. Also, it is of interest in sparse learning-see [13] and references thereinwhere one is interested in estimating the coefficients obtained expanding f H on a given dictionary.…”

“…Example 4 (Elastic-Net Regularization) The elastic-net algorithm proposed in [45], is studied in [13] in the context of learning with an infinite dimensional overcomplete dictionary of features (ψ γ ) γ ∈Γ . In this case, we let 2 (Γ ) be the space of β = (β γ ) γ ∈Γ such that γ ∈Γ |β γ | 2 < ∞ and look for an estimator of the form γ ∈Γ β γ ψ γ .…”

“…In this setting one is often interested in error estimates on the coefficients since they give information on which features are most important. If we assume the target function to have an expansion f H = γ ∈Γ β * γ ψ γ such that γ ∈Γ |β * γ | < ∞, then, for λ > 1/ √ n, it is possible to prove [13] that with probability at least 1 − η…”