To Correct or Not To Correct? Reliability of Psychological Tests and Incremental Predictive Validity from a Statistical Learning Perspective. (This paper was recently published in Behaviometrika: "An out-of-sample perspective on the assessment of incremental predictive validity")

Abstract

When validating a new test, its incremental predictive validity (IV) is often evaluatedfrom a classical approach using in-sample estimates. In this study, to evaluate IV, weadopt a statistical learning (SL) approach that uses out-of-sample estimates. Followingthis approach, we identify two important aspects to consider: 1) evaluate the predictionrules based on the out-of-sample prediction error, and 2) obtain the prediction rulesusing different estimation methods (e.g., OLS, SIMEX, and ridge). We performed asimulation study to investigate the effect of using in- and out-of-sample measures on IV,and to assess the influence of the reliabilities of tests on these two estimates of IV.Results showed that the two approaches differed, such that in-sample IV is on averagelarger than out-of-sample IV, but that the difference decreases as the size of thecalibration sample increases. In- and out-of-sample IV were both affected by thereliability of the added test, that is, as reliability increases IV also increases. Inaddition, if collinearity exists between two tests, the reliability of the first test has anegative relationship with the IV of the second test. Worthy of note is that thedirection of these effects did not differ between the two approaches (in/out-of-sample).It was further shown that using ridge regression enhances IV defined as out-of-samplewhen the size of the calibration sample is small. We also showed how to apply the newprinciples in practice using empirical data from a student selection procedure. Weconcluded that in addition to establishing strong reliability for the new test, validationshould be based on out-of-sample prediction accuracy and that users should consider ashrinkage method to estimate optimal rules for prediction for new persons.