A simulation of the impacts of machine learning to combine psychometric employee selection system predictors on performance prediction, adverse impact, and number of dropped predictors

Abstract

We compare modern machine learning (MML) techniques to ordinary least squares (OLS) regression on out‐of‐sample (OOS) operational validity, adverse impact, and dropped predictor counts within a common selection scenario: the prediction of job performance from a battery of diverse psychometrically‐validated tests. In total, scores from 1.2 billion validation study participants were simulated to describe outcomes across 31,752 combinations selection system design and scoring decisions. The most consistently valuable improvement from adopting MML over traditional regression was from dropping predictors rather than by improving prediction. On average, MML improved prediction of performance from psychometric scale composites only when the ratio of sample size to scale count was less than approximately 3, although algorithm choice, predictor count, and selection ratio affected outcomes as well. We also simulated the effects of design choices when combining item scores, which showed consistent, superior predictive accuracy for several MML algorithms, especially elastic net and random forest, over traditional regression. Given these results, we suggest the potential of machine learning for employee selection is unlikely to be realized in selection systems focusing on the combination of scale composites from previously validated psychometric tests. Instead, it will be realized in unconventional design scenarios, such as the use of individual items to make multiple trait inferences, or with novel data formats like text, image, audio, video, and behavioral traces. We therefore recommend researchers focus on the potential value of MML in future selection contexts rather than continuing to focus on the current value of MML in current selection contexts.

Machine learning applications to personnel selection: Current illustrations, lessons learned, and future research

“…When examining ML algorithms compared to traditional methods with large samples, the improvement in prediction is probably not going to be large in most instances. This is illustrated by Koenig et al (2023) Studies 3-5, and especially by Landers et al (2023). These studies were relevant for the special issue specifically because they did not report exceptional findings, which leads to overestimates in our literature and future meta-analyses.…”

“…Another concern is that building an ML model based on test items rather than total test scores or scales violates the inferences we can make about the construct validity, which are based on the total test score or scale. Landers et al (2023) suggest that you cannot infer construct validity from a scale to its individual items but, if all the items are included in the model and scored separately, to what extent can we infer the model has the construct validity of the scale or total score that includes all the items? This issue may be even more complex if items differ not only in the weights they receive in a model but also if curvilinear relationships of items scored in the model.…”

“…b) What contexts might benefit from ML? Aside from contexts involving lower n/k ratios (Landers et al 2023), curvilinear relationships, or when a decision tree ML model is better (e.g., classification contexts or categorical criteria), perhaps ML would be beneficial when research is truly exploratory (e.g., data mining situations). Putka et al (2018) describe situations with structured data where various techniques may help.…”

“…Numeric and text Wide range of supervised ML to optimally predict outcomes (e.g., job performance) and may include many variables and curvilinear relationships (e.g., Koenig et al, 2023, Study 5;Landers et al, 2023).…”