Sparse Multiresolution Representations With Adaptive Kernels

Abstract

Reproducing kernel Hilbert spaces (RKHSs) are key elements of many non-parametric tools successfully used in signal processing, statistics, and machine learning. In this work, we aim to address three issues of the classical RKHSbased techniques. First, they require the RKHS to be known a priori, which is unrealistic in many applications. Furthermore, the choice of RKHS affects the shape and smoothness of the solution, thus impacting its performance. Second, RKHSs are illequipped to deal with heterogeneous degrees of smoothness, i.e., with functions that are smooth in some parts of their domain but vary rapidly in others. Finally, the computational complexity of evaluating the solution of these methods grows with the number of data points, rendering these techniques infeasible for many applications. Though kernel learning, local kernel adaptation, and sparsity have been used to address these issues, many of these approaches are computationally intensive or forgo optimality guarantees. We tackle these problems by leveraging a novel integral representation of functions in RKHSs that allows for arbitrary centers and different kernels at each center. To address the complexity issues, we then write the function estimation problem as a sparse functional program that explicitly minimizes the support of the representation leading to low complexity solutions. Despite their non-convexity and infinite dimensionality, we show these problems can be solved exactly and efficiently by leveraging duality, and we illustrate this new approach in simulated and real data.Index Terms-multikernel learning, RKHS

Constrained Learning with Non-Convex Losses

“…The next result shows that under mild condition on the distributions D i , this equality holds even if the i are non-convex. We note that, besides being the first step in the construction of Theorem 1, this result has also been used in other contexts (see, e.g., [46], [53], [54]).…”

Federated Classification using Parsimonious Functions in Reproducing Kernel Hilbert Spaces

“…The class function C and the complexity measure ρ(f ) constrain the variability of f and dictate how it generalizes to unobserved samples x. In this paper we adopt the use of low complexity kernel representations as introduced in [25].…”

“…We are therefore searching for a function f that can be specified by as few kernels as possible while still passing within of all the elements of the training set. When we search for kernels to add to the representation, the search is over kernel centers s ∈ S and kernel parameters w ∈ W. The latter allows, e.g., a search over kernel widths -see [25] for details.…”

“…Their success is due to the richness of these spaces, which allows them to model a large class of functions. Moreover, the functions found in RKHS methods can be represented as a possibly infinite linear combination of functions called reproducing kernels [20], [22], [25]. The integral representation of RKHS methods presented in [25] is used for this federated classification because it both learns a low complexity representation and detects the critical samples to the learning problem.…”

“…Moreover, the functions found in RKHS methods can be represented as a possibly infinite linear combination of functions called reproducing kernels [20], [22], [25]. The integral representation of RKHS methods presented in [25] is used for this federated classification because it both learns a low complexity representation and detects the critical samples to the learning problem. Indeed, the integral representation allows for a modified L 0norm for functions which measures the support of the function.…”