Residual variance estimation in machine learning

Abstract

The problem of residual variance estimation consists of estimating the best possible generalization error obtainable by any model based on a finite sample of data. Even though it is a natural generalization of linear correlation, residual variance estimation in its general form has attracted relatively little attention in machine learning.In this paper, we examine four different residual variance estimators and analyze their properties both theoretically and experimentally to understand better their applicability in machine learning problems. The theoretical treatment differs from previous work by being based on a general formulation of the problem covering also heteroscedastic noise in contrary to previous work, which concentrates on homoscedastic and additive noise.In the second part of the paper, we demonstrate practical applications in input and model structure selection. The experimental results show that using residual variance estimators in these tasks gives good results often with a reduced computational complexity, while the nearest neighbor estimators are simple and easy to implement.

Fast Feature Selection in a GPU Cluster Using the Delta Test

“…The difference is in the extra hyper-parameter present in the Gamma Test (the number of neighbors), while the DT uses only the first nearest neighbor, providing a fully non-parametric method. The reduction to only the closest neighbor still gives the unbiased estimator of the noise variance in the limit N → ∞ [10].…”

“…In order to evaluate the goodness of an individual, the delta test (DT) [5] value obtained using the combination of variables is used, as it has been shown to be an adequate criterion [6,10].…”

The delta test: The 1-NN estimator as a feature selection criterion

“…This is a well-known estimator of Var(ε) and it has been showne.g., in [26]-that the estimate converges to the true value of the noise variance in the limit M → ∞. Although it is not considered to be the most accurate noise estimator, its advantages include reliability, simplicity, and computational efficiency [22].…”

“…It is shown in [26] that the variance of the Delta test converges to 0 with increasing M . As the expectation of the Delta Test under the above assumptions is strictly minimised by the "best" selection, this means that the probability of the method choosing this selection generally increases by increasing the number of available samples.…”

“…The Tecator data set consists of 240 samples of near infrared spectra with the task being to model the fat content 0.4901 MI 4,5,8,9,32,33,35,40,48 0.6009 LARS 1-01 8 0.9372 LARS 1-02 8,10 0.8270 LARS 1-03 8,10,11 0.7988 LARS 1-04 8,10,11,40 0.7909 LARS 1-05 8,10,11,33,40 0.6319 LARS 1-06 8,10,11,33,40,42 0.6502 LARS 1-07 8,10,11,33,40,42,54 0.6508 LARS 1-08 8,10,11,26,33,40,42,54 0.6498 LARS 1-09 8,10,11,26,33,36,…”