Analyzing Sparse Dictionaries for Online Learning With Kernels

Abstract

Many signal processing and machine learning methods share essentially the same linear-in-the-parameter model, with as many parameters as available samples as in kernel-based machines. Sparse approximation is essential in many disciplines, with new challenges emerging in online learning with kernels. To this end, several sparsity measures have been proposed in the literature to quantify sparse dictionaries and constructing relevant ones, the most prolific ones being the distance, the approximation, the coherence and the Babel measures. In this paper, we analyze sparse dictionaries based on these measures. By conducting an eigenvalue analysis, we show that these sparsity measures share many properties, including the linear independence condition and inducing a well-posed optimization problem. Furthermore, we prove that there exists a quasi-isometry between the parameter (i.e., dual) space and the dictionary's induced feature space. Index Terms-Sparse approximation, adaptive filtering, kernelbased methods, Gram matrix, machine learning, pattern recognition. Paul Honeine (M'07) was born in Beirut, Lebanon, on October 2, 1977. He received the Dipl.-Ing. degree in mechanical engineering in 2002 and the M.Sc. degree in industrial control in 2003, both from the Faculty of Engineering, the Lebanese University, Lebanon. In 2007, he received the Ph.D. degree in Systems Optimisation and Security from the University of Technology of Troyes, France, and was a Postdoctoral Research associate with the Systems Modeling and Dependability Laboratory, from 2007 to 2008. Since September 2008, he has been an assistant Professor at the University of Technology of Troyes, France. His research interests include nonstationary signal analysis and classification, nonlinear and statistical signal processing, sparse representations, machine learning. Of particular interest are applications to (wireless) sensor networks, biomedical signal processing, hyperspectral imagery and nonlinear adaptive system identification. He is the co-author (with C. Richard) of the 2009 Best Paper Award at the IEEE Workshop on Machine Learning for Signal Processing. Over the past 5 years, he has published more than 100 peerreviewed papers.

An Incremental Learning Method for L1- Regularized Kernel Machine in WSN

“…To solve problem (7), Alternating Direction Method of Multiplies (ADMM) [13] is used. In ADMM form, the problem (7) can be reformulated as (8), and the resulting iterations as (9), (10) and (11): (11) In (9), I is an identity matrix, and (1) T c ρ ++ KKI is always invertible, since 0 ρ > . S in (10) is the soft thresholding operator, and is defined as (12):…”

“…Most similar to the research presented here are for SVM [6][7][8][9][10][11]. However, the sparse characteristic of SVM is determined by Hinge loss function, the support vectors will gradually increase with the sequentially executing of incremental learning method on training samples, so it will rapidly increase the computational cost and the need for larger memory space on subsequent training samples.…”

“…When there are large amount of training samples or training samples are available in sequence, the batch learning method performed on sensor node will be infeasible. Alternatively, the incremental learning method of classifiers or regression machines is initiated and researched for Support Vector Machine (SVM) to deal with large-scale datasets [6][7][8][9][10][11]. Kernel method or kernel machine, short for machine learning method based on kernel function, attracts widely attention and research because of the successful application of SVM and statistical learning theory, and has become the mainstream method in machine learning.…”

Mixed Integer Programming For Sparse Coding: Application to Image Denoising

“…To measure the quality of the dictionaries, we consider the coherence (correlation measured with the inner product) between the atoms of each dictionary, thus measuring how much two atoms in the dictionary are similar. This fundamental information allows to define more powerful measures, such as the coherence and Babel function [41], [42]. The coherence measure of a given dictionary, defined by the maximum absolute inner product between two distinct atoms, provides strong insights on the capacity of the dictionary to recover sparse signals.…”

Improving sparsity in kernel adaptive filters using a unit-norm dictionary

“…where the set D i = {s j } Ni is known as dictionary at time i and the vectors s j as centres. Sparsification criteria for KAFs ensure finite-sized dictionary when the signal lies on a compact set [17], however, when the signal is monotonically increasing these criteria keep including more centres in the dictionary, thus resulting in higher computational complexity due to having dictionary members that do not necessarily improve predictions.…”