Framework for making better predictions by directly estimating variables’ predictivity

Abstract

We propose approaching prediction from a framework grounded in the theoretical correct prediction rate of a variable set as a parameter of interest. This framework allows us to define a measure of predictivity that enables assessing variable sets for, preferably high, predictivity. We first define the prediction rate for a variable set and consider, and ultimately reject, the naive estimator, a statistic based on the observed sample data, due to its inflated bias for moderate sample size and its sensitivity to noisy useless variables. We demonstrate that the I-score of the PR method of VS yields a relatively unbiased estimate of a parameter that is not sensitive to noisy variables and is a lower bound to the parameter of interest. Thus, the PR method using the I-score provides an effective approach to selecting highly predictive variables. We offer simulations and an application of the I-score on real data to demonstrate the statistic's predictive performance on sample data. We conjecture that using the partition retention and I-score can aid in finding variable sets with promising prediction rates; however, further research in the avenue of sample-based measures of predictivity is much desired.prediction | variable selection | high-dimensional data | predictivity P rediction is a highly important goal for many scientists and has become increasingly difficult as the quantity and complexity of available data have grown. Complex and highdimensional data particularly demand attention. However, the literature on prediction does not yet have a clear theoretical framework that allows for characterizing a variable's predictivity directly [see A Brief Literature Review on VS for a brief review on the literature of variable selection (VS)]. Rather, VS for variable sets in the context of prediction is currently conducted in two common ways. The first is VS through identification of variables correlated with the outcome, measured through tests of statistical significance-such as the chi-square test. The second is through VS of variables that seem to do well in an independent set of test data, as measured through testing sample error rates. The first approach is still very much in use for predicting health outcomes (see ref. 1, among others) but its prediction performance has been disappointing (e.g., refs. 1 and 2). We show in our related work (3) how and why the popular filter approach of VS through statistical significance does not serve the purpose of prediction well. For an intuitive illustration of the relationship between predictive and significant sets of variables, see Fig. 1. Under a significance-test-based search setting, the set of variables found to be significant expands as the sample size grows (Fig. 1, widening orange dotted ovals). However, the set of predictive variables (Fig. 1, blue circle) is not susceptible to sample-size changes in the same way-because predictivity is a population parameter-and overlaps, but is not perfectly aligned with, significant sets. It is easy to see that in this scenario targe...

Empirical Methods for Remote Sensing of Nitrogen in Drylands May Lead to Unreliable Interpretation of Ecosystem Function

“…It is tempting to discuss this in terms of ecology and the association between canopy N and LAI; however, it is more likely a statistical problem. An explanation is that statistically significant explanatory variables (e.g., spectral bands) that have an association with a target variable might not necessarily carry the most predictive power, and the most predictive variables are not necessarily the most significant ones [57], [58]. A key distinction that makes a variable significant or predictive lies in the properties of their underlying distribution.…”

SNP Selection using Variable Ranking and Sequential Forward Floating Selection with Two Optimality Criteria

“…If − 1 variables are influential in the sense that any smaller subset of variables is less influential, then the removal of a variable to size − 2 will decrease the I-score. Thus, the I-score has a natural tendency to "peak" at variable set(s) that lead to high predictive power in the face of noisy variables under the current sample size [15]. For high-dimensional variable selection problem, one way to thin out the candidates, i.e., to reduce the search space is to apply the I-score to one explanatory variable at a time, and to focus on those which indicate strong marginal observable effects [14].…”

Kernel‐based measures of association

“…It has also been shown, from a predictive modeling perspective, that test‐derived statistics are not reliable measures , even though they may have adequate power in detecting departures from independence (Lo et al, ). On the other hand, PR's I has been shown to correlate well with the actual extent of association between X variables and Y (in this case, class labels) (Lo, Chernoff, Zheng, & Lo, ), with the true association parameterized by how well X can predict Y .…”