Greedy algorithms for prediction

Abstract

In many prediction problems, it is not uncommon that the number of variables used to construct a forecast is of the same order of magnitude as the sample size, if not larger. We then face the problem of constructing a prediction in the presence of potentially large estimation error. Control of the estimation error is either achieved by selecting variables or combining all the variables in some special way. This paper considers greedy algorithms to solve this problem. It is shown that the resulting estimators are consistent under weak conditions. In particular, the derived rates of convergence are either minimax or improve on the ones given in the literature allowing for dependence and unbounded regressors. Some versions of the algorithms provide fast solution to problems such as Lasso.Comment: Published at http://dx.doi.org/10.3150/14-BEJ691 in the Bernoulli (http://isi.cbs.nl/bernoulli/) by the International Statistical Institute/Bernoulli Society (http://isi.cbs.nl/BS/bshome.htm

On the number of iterations for the matching pursuit algorithm

“…The experimental settings are similar to those in [4,Sec. 3.3]; the sample size is n = 100, and the dictionary is complete (p n = 100).…”

“…is the vector of linear parameters corresponding to the model selected by a particular IT criterion, in trial r, then we compute the mean integrated square error as follows [4,Sec. 3.3]:…”

“…For testing the predictive power of each IT criterion, we use the same method as in [4,Sec. 3.3]: For each trial, the same algorithm as the one used to generate the dictionary is applied in order to produce a matrix X out,r whose size is…”

“…When MPA is used for high-dimensional linear models, existing theoretical results do not provide a stopping rule for the algorithm (see, for example, [4]). In practical applications the number of iterations is selected either by using cross-validation (CV) or an information theoretic (IT) criterion [2].…”

Optimal data collection for randomized control trials

“…See Temlyakov (2011) for a detailed discussion of greedy algorithms in the context of approximation theory. Tropp (2004), Tropp and Gilbert (2007), Barron, Cohen, Dahmen, and DeVore (2008), Zhang (2009), Huang, Zhang, and Metaxas (2011), Ing and Lai (2011), and Sancetta (2016, among many others, demonstrate the usefulness of greedy algorithms for signal recovery in information theory, and for the regression problem in statistical learning. We use a variant of OGA that can allow for selection of groups of variables (see, for example, Huang, Zhang, and Metaxas (2011)).…”