On the number of iterations for the matching pursuit algorithm

Abstract

We address the problem of selecting, from a given dictionary, a subset of predictors whose linear combination provides the best description for the vector of measurements. To this end, we apply the well-known matching pursuit algorithm (MPA). Even if there are theoretical results on the performance of MPA, there is no widely accepted rule for stopping the algorithm. In this work, we focus on stopping rules based on information theoretic criteria (ITC). The key point is to evaluate the degrees of freedom (df) for the model produced at each iteration. This is traditionally done by computing the trace of the hat matrix which maps the data vector to its estimate. We prove some theoretical results concerning the hat matrix. One of them provides an upper bound on the increase of df from the m-th to the (m + 1)-th iteration. Based on the properties of the hat matrix, we propose novel ITC for selecting the number of iterations. All of them are obtained by modifying criteria designed for variable selection in the classical linear model. For assessing the performance of the novel criteria, we conduct a simulation study.Index Terms-Matching pursuit algorithm, hat matrix, projector, information theoretic criteria, number of iterations

On the selection of predictors by using greedy algorithms and information theoretic criteria

“…It follows that either CV or IT criteria should be used in order to select the 'optimal' linear model from the m ub models generated when running the algorithm. The use of CV is discussed in Bühlmann & van de Geer (2011), but it has been argued in (Li et al . 2017(Li et al .…”

“…From statistical viewpoint, the linear models produced during the MPA iterations are more important than the model that corresponds to the ε-approximate solution. Due to this reason, it is enough to run MPA for a single value of ν (see Bühlmann & van de Geer 2011, Section 12.6.2.1 and Li et al . 2017).…”

“…For ease of comparison, we set m ub = 20000 for all the algorithms. Based on the findings from Li et al . (2017), we choose the shrinkage parameter for MPA to be ν = 0.1.…”

“…Exploiting the relationship between BIC and MDL, two other extended criteria can be defined (see also Li et al . 2017Li et al .…”

Optimal Estimation of Wavelet Decomposition Level for a Matching Pursuit Algorithm

“…One can use several criteria [ 14 ] for stopping the algorithm, such as iterations limit, the steady error value, the absence of unused atoms, etc. Here we should note several important properties, as follows: The algorithm converges (i.e., ) for any f that is in the space spanned by the dictionary and the error decreases monotonically.…”