Parsimonious Online Learning with Kernels via sparse projections in function space

Abstract

Despite their attractiveness, popular perception is that techniques for nonparametric function approximation do not scale to streaming data due to an intractable growth in the amount of storage they require. To solve this problem in a memory-affordable way, we propose an online technique based on functional stochastic gradient descent in tandem with supervised sparsification based on greedy function subspace projections. The method, called parsimonious online learning with kernels (POLK), provides a controllable tradeoff between its solution accuracy and the amount of memory it requires. We derive conditions under which the generated function sequence converges almost surely to the optimal function, and we establish that the memory requirement remains finite. We evaluate POLK for kernel multi-class logistic regression and kernel hinge-loss classification on three canonical data sets: a synthetic Gaussian mixture model, the MNIST hand-written digits, and the Brodatz texture database. On all three tasks, we observe a favorable trade-off of objective function evaluation, classification performance, and complexity of the nonparametric regressor extracted the proposed method.

Decentralized Online Learning With Kernels

“…We define the model order as number of data points M i,t in the dictionary of agent i at time t (the number of columns of X t ). FSGD is such that M i,t = t − 1, and hence grows unbounded with iteration index t. Next we address this intractable memory growth such that we may execute stochastic descent through low-dimensional projections of the stochastic gradient, inspired by [31]. First, we clarify the motivation for the choice of the penalty function (9).…”

“…κ(d M , ·)], and K D,D as the resulting kernel matrix from this dictionary. We enforce function parsimony by selecting dictionaries D i with M i,t << O(t) for each i [31].…”

“…In contrast, a method was recently proposed that combines greedily constructed [30] sparse subspace projections with functional stochastic gradient method and guarantees exact convergence to the minimizer of the average risk functional. This technique, called parsimonious online learning with kernels (POLK), tailors the parameterization compression to preserve the descent properties of the underlying RKHS-valued stochastic process [31], and inspires the approach considered here.…”

“…In this work, we extend the ideas in [31] to multi-agent settings. Multiple tools from distributed optimization may be used to do so; however, we note that the Representer Theorem [18] has not been established for general stochastic saddle point problems in RKHSs.…”

Sparse Multiresolution Representations With Adaptive Kernels

“…So far we have shown that the complexity of the formulation can be reduced by moving centers in addition to moving the width. To further explore the effect of kernel centers on the complexity of the solution, we compare the performance of (PII ′ ) to that of kernel orthogonal matching pursuit (KOMP) with pre-fitting (see [9], [30]), for a simulated signal as in (23). KOMP takes an initial function and a set of sample points and tries to estimate it by a parsimonious function of a lower complexity.…”