Validating Synthetic Validation: Comparing Traditional and Synthetic Validity Coefficients

Abstract

We describe a unique application of a synthetic validation technique to a selection system development project in a large organization. Job analysis data were collected from 4,725 job incumbents and 619 supervisors, and were used to identify 11 job families and 27 job components. We developed 12 tests to predict performance on these job components and conducted a concurrent validation study to collect test and job component data for 1,926 incumbents. We created a test composite for each job component and then chose a test battery for each job family based on its relevant job components. Synthetic validity coefficients were computed for each test battery and compared to traditional validity coefficients that were computed within job families with large sample sizes. The synthetic validity coefficient was very close to the within-family validity coefficient for most job families and was within the bounds of sampling error for all job families. Validities tended to be highest when predictors were weighted according to the number of job components to which they were relevant and job component criterion measures were unit weighted.

Personality and Cognitive Ability as Predictors of Effective Performance at Work

“…Van Iddekinge and Ployhart did report a few studies on maximum and typical performance; most studies have found that these performance indices are not highly correlated and that they do have different correlates (e.g., Marcus et al 2007). Johnson & Carter (2010) provided an excellent case study of the application of synthetic validity. Using job analysis data, the authors identified 11 job families and 27 job components.…”

Validation Is Like Motor Oil: Synthetic Is Better

“…This vector defines how the job component performance scores should be weighted and combined to arrive at a definition of overall performance. A typical source of job component weights is mean importance ratings from the job analysis, although alternative weighting strategies such as unit weights may perform just as well (Johnson & Carter, in press).…”

Using O*NET to Develop a Framework of Job Characteristics to Potentially Improve the Predictive Validity of Personality Measures

“…In theory, these predictors should thus be valid predictors for any job that contains the job components identified, and the validity of tests that measure these predictors should be similar across job domains as long as they include these components. Some extant research evidence suggests that validity coefficients produced through synthetic validation are similar to coefficients produced through local validity studies (Johnson & Carter, 2010;Peterson, Wise, Arabian, & Hoffman, 2001). …”

“…The current study bears some resemblance to "synthetic" validation approaches (e.g., Scherbaum, 2005), which involve several steps, including (generally): conducting job analyses to identify components of jobs that may be common across several jobs, measuring important predictors (often personality traits) and job performance outcomes, and choosing the most valid predictors of job components identified by job analyses (Johnson & Carter, 2010). In theory, these predictors should thus be valid predictors for any job that contains the job components identified, and the validity of tests that measure these predictors should be similar across job domains as long as they include these components.…”