Total stability of kernel methods

Abstract

Regularized empirical risk minimization using kernels and their corresponding reproducing kernel Hilbert spaces (RKHSs) plays an important role in machine learning. However, the actually used kernel often depends on one or on a few hyperparameters or the kernel is even data dependent in a much more complicated manner. Examples are Gaussian RBF kernels, kernel learning, and hierarchical Gaussian kernels which were recently proposed for deep learning. Therefore, the actually used kernel is often computed by a grid search or in an iterative manner and can often only be considered as an approximation to the "ideal" or "optimal" kernel. The paper gives conditions under which classical kernel based methods based on a convex Lipschitz loss function and on a bounded and smooth kernel are stable, if the probability measure P, the regularization parameter λ, and the kernel k may slightly change in a simultaneous manner. Similar results are also given for pairwise learning. Therefore, the topic of this paper is somewhat more general than in classical robust statistics, where usually only the influence of small perturbations of the probability measure P on the estimated function is considered.if |y − t| ≥ 0 for some τ > 0 for quantile regression. We refer for details and more examples of kernels.Definition 2.3. The loss function L is called Lipschitz continuous, if there exists a constant |L| 1 < ∞ such that |L(x, y, t 1 ) − L(x, y, t 2 )| ≤ |L| 1 |t 1 − t 2 | ∀x ∈ X , y ∈ Y, t 1 , t 2 ∈ R.(2.2) Assumption 2.4. Let L be a convex with respect to the last argument and Lipschitz continuous loss function with Lipschitz constant |L| 1 ∈ (0, ∞).Assumption 2.5. For all (x, y) ∈ X × Y, let L(x, y, ·) be differentiable and its derivative be Lipschitz continuous with Lipschitz constant |L ′ | 1 ∈ (0, ∞).The moment condition E P L(X, Y, 0) < ∞ excludes heavy-tailed distributions such as the Cauchy distribution and many other stable distributions used in financial or actuarial problems. We avoid the moment condition by shifting the loss with by the term L(x, y, 0). This trick is wellknown in the literature on robust statistics, see, e.g., Huber (1967), Christmann et al. (2009), and Christmann and Zhou (2016). Denote the shifted loss function of L by L ⋆ (x, y, t) := L(x, y, t) − L(x, y, 0), (x, y, t) ∈ X × Y × R.The shifted loss function L ⋆ still shares the properties of L specified in Assumption 2.4 and Assumption 2.5, see Christmann et al. (2009, Proposition 2), in particular, if L is convex, differentiable, and Lipschitz continuous with Lipschitz constant |L| 1 with respect to the third argument, then L ⋆ inherits convexity, differentiability and Lipschitz continuity from L with identical Lipschitz constant |L ⋆ | 1 = |L| 1 . Additionally, if the derivative L ′ satisfies Lipschitz continuity with Lipschitz constant |L ′ | 1 , so does (L ⋆ ) ′ with the identical Lipschitz constant |(L

Total Stability of SVMs and Localized SVMs

“…We will now state our first main result about the stability of SVMs. As mentioned in the introduction, this is a generalization of a result (Theorem 2.7) by Christmann et al (2018): First of all, we eliminated an additional condition on L that was required by Christmann et al (2018). Previously, L did not only need to be convex and Lipschitz continuous but also differentiable.…”

“…In this section, we will show stability of SVMs with respect to slight changes in the triple (P, λ, k) consisting of probability measure, regularization parameter and kernel. Our notion of stability will be similar to that of (4), with the slight difference that we additionally need to consider ||k 1 − k 2 || ∞ as an exchange for the result considerably generalizing the referenced theorem by Christmann et al (2018), that is, the result being applicable to arbitrary positive regularization parameters and a larger class of loss functions. Thus, it will be of the type…”

“…Returning to the reduced conditions on the loss function L compared to the referenced theorem by Christmann et al (2018), we only require L to be convex as well as Lipschitz continuous. Here, convexity refers to convexity in the last argument, i.e.…”

“…Christmann et al, 2009, Theorem 8). In order to circumvent this problem, Christmann et al (2018) additionally provided another result (Theorem 2.10) in which λ 1 and λ 2 are allowed to be arbitrarily close to zero and instead…”

“…Bousquet and Elisseeff, 2002;Steinwart, 2004, 2007;Christmann and van Messem, 2008;Hable and Christmann, 2011, among others). We will, however, base many of our considerations on Christmann et al (2018), where the authors investigated the effect of simultaneous slight changes in the whole triple (P, λ, k) instead of only in P and obtained results like…”