Differential Prediction and the Use of Multiple Predictors: The Omitted Variables Problem.

Abstract

Moderated regression is widely used to examine differential prediction by race or gender. When using multiple predictors in a selection system, guidance as to whether differential prediction analysis should be conducted on each predictor individually, or on the set of predictors in combination, is lacking. Analyzing predictors individually creates the possibility of an omitted variable problem. Army Project A data were used to examine differential prediction by race with the use of personality measures for 79 predictor-criterion combinations. Traditional analysis indicated predictive bias by intercept in 45 instances and by slope in 7 instances; the inclusion of an Armed Services Vocational Aptitude Battery general factor as an additional predictor changed the conclusion in 32 cases for the intercept and in 3 cases for the slope.

Developments in the Criterion‐related Validation of Selection Procedures: A Critical Review and Recommendations for Practice

“…This problem can also occur when examining differential prediction if the omitted predictor is related to both the criterion and subgroup membership. Using data from Project A, Sackett, Laczo, and Lippe (2003) found that inclusion of a previously omitted predictor can change conclusions of differential prediction analyses. For example, existence of significant intercept differences when personality variables were used to predict core task performance dropped from 100% to 25% when the omitted predictor (i.e., a measure of general mental ability) was included in the model.…”

“…If there is evidence of differential prediction, examine whether it appears to be due to the predictors, criteria, or both. Avoid the omitted variables problem by including all relevant predictors in the MMR model. Furthermore, if a composite of predictors is to be used, then the composite (and not the individual predictors that comprise it) should be the focus of the differential prediction analyses (Sackett et al, 2003). Use power analysis to determine the sample size required to draw valid inferences regarding differential prediction, and report the actual level of power for all relevant analyses.…”

“…Avoid the omitted variables problem by including all relevant predictors in the MMR model. Furthermore, if a composite of predictors is to be used, then the composite (and not the individual predictors that comprise it) should be the focus of the differential prediction analyses (Sackett et al, 2003).…”

Personality and Cognitive Ability as Predictors of Effective Performance at Work

“…Sackett et al (2003) underscored the importance, when addressing the bias in the target variable using regression analyses, of including omitted variables that are correlated with both the criterion and the target predictor. Not doing so is likely to suggest bias in the target variable when the bias is a function of some other omitted variable.…”

Relational Differential Prediction

“…But, given that in both cases we have two populations that have been subjected to an external agent, we argue that the concepts and techniques originally developed for uplift marketing can, and should, apply to the task of differential prediction (and vice versa). Differential prediction has been studied extensively in the context of multi-attribute data [30,28]. One approach is to generate different classifiers for each sub-population, and to look for differences between the classifiers.…”