On the selection of predictors by using greedy algorithms and information theoretic criteria

Abstract

Summary We discuss the use of the following greedy algorithms in the prediction of multivariate time series: Matching Pursuit Algorithm (MPA), Orthogonal Matching Pursuit (OMP), Relaxed Matching Pursuit (RMP), Frank–Wolfe Algorithm (FWA) and Constrained Matching Pursuit (CMP). The last two are known to be solvers for the lasso problem. Some of the algorithms are well‐known (e.g. OMP), while others are less popular (e.g. RMP). We provide a unified presentation of all the algorithms, and evaluate their computational complexity for the high‐dimensional case and for the big data case. We show how 12 information theoretic (IT) criteria can be used jointly with the greedy algorithms. As part of this effort, we derive new theoretical results that allow modification of the IT criteria such that to be compatible with RMP. The prediction capabilities are tested in experiments with two data sets. The first one involves air pollution data measured in Auckland (New Zealand) and the second one concerns the House Price Index in England (the United Kingdom).

On the Application of the Stability Methods to Time Series Data

“…The main difference is that all the entries of β that correspond to the columns of X selected at the current and past iterations are updated by least squares, hence the IT criteria for Gaussian linear regression can be used. For example, in [8], 12 different IT criteria have been employed for selecting the best model from the candidates yielded by OMP.…”

“…Relying on the idea of the complementary pairs from [15], we consider the subset O = O \ O [see (7) and (8) for the definitions of O and O ]. It is straightforward to obtain another estimator, S |O | , for the subset S of 'signal' variables by replacing O with O in (9) and then applying Lasso for solving the optimization problem.…”

“…Some entries on the off-diagonal locations of M are randomly selected and forced to be zero (the number of zero entries is restricted to be less than 0.6 • K 2 y ); (iii) for 1 ≤ i ≤ p y , the element-wise product of M and A i gives the matrix A i . The methodology utilized for producing the artificial data has a certain degree of similarity with the one used for simulating the data in an experiment reported in [8]. There are two differences between the approach in this work and the one in [8].…”

“…The methodology utilized for producing the artificial data has a certain degree of similarity with the one used for simulating the data in an experiment reported in [8]. There are two differences between the approach in this work and the one in [8]. The first difference stems from the fact that, in [8], all the odd blocks have the same cardinality and all the even blocks have the same cardinality, but |O 1 | can be different from |E 1 |.…”

“…The first difference stems from the fact that, in [8], all the odd blocks have the same cardinality and all the even blocks have the same cardinality, but |O 1 | can be different from |E 1 |. The second difference is that, in [8], after generating the blocks O 1 , . .…”