Non-parametric Residual Variance Estimation in Supervised Learning

Abstract

Abstract. The residual variance estimation problem is well-known in statistics and machine learning with many applications for example in the field of nonlinear modelling. In this paper, we show that the problem can be formulated in a general supervised learning context. Emphasis is on two widely used non-parametric techniques known as the Delta test and the Gamma test. Under some regularity assumptions, a novel proof of convergence of the two estimators is formulated and subsequently verified and compared on two meaningful study cases.

Autoregressive time series prediction by means of fuzzy inference systems using nonparametric residual variance estimation

“…For a proof of convergence, refer to [28,29]. DT is an unbiased and asymptotically perfect estimator with a relatively fast convergence [29] and is useful for evaluating nonlinear correlations between two random variables, namely, input-output pairs.…”

“…DT can be seen as part of a more general NNE framework known as the Gamma Test [24]. Despite the simplicity of DT, it has been shown to be a robust method in real world applications [28]. This method will be used in the next sections for a priori input selection.…”

“…These methods select inputs a priori, i.e., the selection stage is based only on the dataset and does not require to build models. Thus, the computational cost of DT based selection is lower than that of the model dependent cases, in which input selection is addressed as a generalization error minimization problem, using leave-one-out, bootstrap or other resampling techniques [28].…”

Fuzzy inference based autoregressors for time series prediction using nonparametric residual variance estimation

“…For a proof of convergence, refer to [6]. DT has been shown to be a robust method for estimating the lowest possible mean squared error (MSE) of a nonlinear model without overfitting.…”

xftsp: A tool for time series prediction by means of fuzzy inference systems

“…where YNN(i) is the output corresponding to XNN(i)' For a proof of convergence, refer to [5]. DT has been shown to be a robust method for estimating the lowest possible mean squared error (MSE) of a nonlinear model without overfitting.…”