Efficient KLMS and KRLS algorithms: A random fourier feature perspective

Abstract

We present a new framework for online Least Squares algorithms for nonlinear modeling in RKH spaces (RKHS). Instead of implicitly mapping the data to a RKHS (e.g., kernel trick), we map the data to a finite dimensional Euclidean space, using random features of the kernel's Fourier transform. The advantage is that, the inner product of the mapped data approximates the kernel function. The resulting "linear" algorithm does not require any form of sparsification, since, in contrast to all existing algorithms, the solution's size remains fixed and does not increase with the iteration steps. As a result, the obtained algorithms are computationally significantly more efficient compared to previously derived variants, while, at the same time, they converge at similar speeds and to similar error floors.

Adaptive Graph Filters in Reproducing Kernel Hilbert Spaces: Design and Performance Analysis

“…This increase makes this model unsuitable for real-time applications and large-scale networks. The growing dimensionality of the dictionary is a well-known issue in single-node kernel methods [40], [41], [49]- [53], where several solutions have been Algorithm 1: GKLMS Input: step size µ Initialization: α 0 = empty vector; %Learning for each time instant n do Input:…”

“…In conventional DSP, several approaches to nonlinear system modeling exist in the literature [30]- [38]. In particular, methods based on reproducing kernel Hilbert spaces (RKHS) have gained popularity due to their efficacy and mathematical simplicity [36]- [53]. There is extensive literature on function estimation in RKHS for both single-and multi-node networks, see, e.g., [39]- [57].…”

A Polarized Random Fourier Feature Kernel Least-Mean-Square Algorithm

“…The problem is a joint optimization of {λ k , θ k } N w k=1 , which is obviously not a convex problem. Meanwhile, for commonly used RFF-based KLMS algorithms [4], [14] in kernel adaptive filtering, samples {w i } N w i=1 drawn from the distribution p(w) and samples {d i } N w i=1 drawn from a uniform distribution may not be optimal for the given learning task because of the uncertainty of random sampling. A set of optimized {θ k } N w k=1 is essential to be obtained before the risk function can be a convex problem over {λ k } N w k=1 .…”

Adaptation and Learning Over Networks for Nonlinear System Modeling

“…In particular, it is possible to show that this class of kernels can be easily approximated with very simple stochastic mappings. [37] was the first to apply this idea explicitly to kernel filters, and similar algorithms were independently reintroduced in [38]. Since Chapter 8 is entirely devoted to this idea, we will not go further into it.…”