Memory-universal prediction of stationary random processes

Abstract

We consider the problem of one-step-ahead prediction of a real-valued, stationary, strongly mixing random process fX i g 1 i=01. The best mean-square predictor of X 0 is its conditional mean given the entire infinite past fX i g 01 i=01. Given a sequence of observations X 1 X 2 1 1 1 X N , we propose estimators for the conditional mean based on sequences of parametric models of increasing memory and of increasing dimension, for example, neural networks and Legendre polynomials. The proposed estimators select both the model memory and the model dimension, in a data-driven fashion, by minimizing certain complexity regularized least squares criteria. When the underlying predictor function has a finite memory, we establish that the proposed estimators are memory-universal: the proposed estimators, which do not know the true memory, deliver the same statistical performance (rates of integrated mean-squared error) as that delivered by estimators that know the true memory. Furthermore, when the underlying predictor function does not have a finite memory, we establish that the estimator based on Legendre polynomials is consistent.

Untitled

“…Finally, the optimal balance between the complexity of the model and the memory size used for prediction must be determined. We observe that our results bear strong affinities to the approach taken by Modha and Masry (1998), while deviating from them in scope and methodology; see Remark 7 in Section 6 for a detailed comparison. An additional related work is the one by Campi and Kumar (1998), which deals with the problem of learning dynamical systems in a stationary environment.…”

“…One approach to incorporating temporal structure in order to form better predictors, by more appropriate complexity regularization, is described in this work. In particular, the optimal memory size that should be used in order to form a predictor is in principle derivable from the procedure (see also (Modha & Masry, 1998)), given information about the mixing nature of the time series (see Section 4 for a definition of mixing). It is thus hoped that many of the successful Machine Learning approaches to modeling static data will be extended to time series, with the benefit of a solid mathematical framework.…”

Untitled

“…Implicitly, while establishing the rates of convergence results in Theorem 2.1 and Corollary 2.1, we assumed that these least-squares estimators can indeed be computed. Such an assumption is the very basis for applying Vapnik's empirical risk minimization theory to neural networks, and has been widely used in the literature dealing with rates of convergence results for neural networks and other models, see, for example, Barron (1994), Barron, Birgé, & Massart (1996), Breiman (1993), Haussler (1992), Kearns (1997), Lugosi & Nobel (1995), Lugosi & Zeger (1996), McCaffrey & Gallant (1994), Modha & Masry (1996, 1998, Vapnik (1982Vapnik ( , 1995, and White (1989).…”

“…In this case, it is known that the complexityregularized regression estimator is order-universal, that is, the complexity-regularized regression estimator, which does not know the true dimension m , delivers the same rate of integrated mean-squared error as that delivered by an estimator that knows the true dimension. See Barron (1994), Barron, Birgé, & Massart (1996), and Modha & Masry (1998). Establishing order-universality results for prequential and cross-validated regression estimators currently remains an open problem.…”

“…Roughly speaking, a model with a larger dimension has a smaller approximation error but a larger estimation error, whereas a model with a smaller dimension has a smaller estimation error but a larger approximation error. The problem of model selection is to empirically select the finite-dimensional parametric model (from the permissible collection of models {S m } m ∈ M n ) that achieves the best tradeoff between the competing approximation error and estimation error components-and, consequently, achieves the smallest possible statistical risk in estimating s. For previous theoretical work on model selection in the context of nonparametric regression estimation, see, for example, Barron (1991Barron ( , 1994, Barron, Birgé, & Massart (1996), Birgé & Massart (1994b), Baum & Haussler (1989), Haussler (1992), Lugosi & Nobel (1995), Lugosi & Zeger (1996), McCaffrey & Gallant (1994), Modha & Masry (1996, 1998, Rissanen (1989), Shen & Wong (1994), Vapnik (1982Vapnik ( , 1995, and White (1989).…”