Validation Is Like Motor Oil: Synthetic Is Better

Johnson, Jeff W.; Steel, Piers; Scherbaum, Charles A.; Hoffman, Calvin C.; Jeanneret, P. Richard; Foster, Jeff

doi:10.1111/j.1754-9434.2010.01245.x

Cited by 46 publications

(39 citation statements)

References 83 publications

(113 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…More interesting and meaningful results will be found by elucidating the conditions under which person-variables are more versus less influential (e.g., Chatman, 1989;Mischel, 1977). Examining situational strength in a more thorough and meaningful manner will, therefore, improve psychologists' understanding of disposition-situation interactions, and thereby substantially improve personnel-selection systems by allowing researchers and practitioners to more accurately estimate criterion-related validities (Johnson, Steel, Scherbaum, Hoffman, Jeanneret, & Foster, 2010;Meyer, Dalal, & Bonaccio, 2009;Meyer, Dalal, & Hermida, 2010;Peterson & Bownas, 1982).…”

Section: Theor Ymentioning

confidence: 99%

“…These estimates could then be made publicly available through an online database similar to O*NET (the Occupational Information Network; http://online.onetcenter.org). Such a database would allow researchers and practitioners to improve human resources systems in the Army by more accurately assessing the likely validity of noncognitive predictors (see Johnson et al, 2010, for arguments in support of such a system), and by better estimating the likelihood of deleterious outcomes based on situational-strength misfit. For example, selecting Soldiers on the basis of non-cognitive predictors may be more conducive to performance in some MOSs than others.…”

Section: Futur E Resear Chmentioning

confidence: 99%

See 1 more Smart Citation

Dissecting Situational Strength: Theoretical Analysis and Empirical Tests

Dalal¹,

Meyer²,

Jose³

et al. 2012

View full text Add to dashboard Cite

It has been argued that the most important characteristic of situations is their "strength"-and that dispositional forces predict behavior in "weak," but not "strong," situations. Until recently, however, there was little consensus regarding how to measure this construct. As a consequence, several behavioral predictions are less accurate than they could be. The current research presents a theoretically unified conceptualization of this construct (consisting of the four facets of clarity, consistency, constraints, and consequences), provides tests of its primary postulates, and assesses the effects of misfit between preferred and perceived levels of strength on the job. We found that situational strength influenced the impact of personality (conscientiousness and agreeableness) on citizenship and counterproductive (deviant) behavior at work. We also found that the deleterious impact of misfit between preferred and perceived levels of situational strength was worse for certain types of misfit than for others. We suggest fruitful avenues for future research as well as practical implications. Because recent changes in the Army's nature and structure have "weakened" the situations Soldiers commonly experience, the current research has considerable implications for Army research in Soldier selection, classification, and retention. Overall, the current research sheds considerable light on the phenomenon of situational strength.

show abstract

Section: Theor Ymentioning

confidence: 99%

Section: Futur E Resear Chmentioning

confidence: 99%

Dissecting Situational Strength: Theoretical Analysis and Empirical Tests

Dalal¹,

Meyer²,

Jose³

et al. 2012

View full text Add to dashboard Cite

show abstract

“…The fact that O*NET is based on the Standard Occupational Classification (SOC) title taxonomy, which rates approximately one-tenth the number of occupations described in the DOT, arguably limits the usefulness of the O*NET database as a vehicle for developing JCV models to predict the Strength ratings required by SSA and others when adjudicating disability claims (in the sense that DOT data provide greater granularity with respect to matching a specific person's past job title to an occupational title). However, past JCV studies have deemed the SOC level of analysis to be adequate for JCV (e.g., LaPolice et al, 2008), and the Johnson et al (2010) likewise advocated strongly for the expanded use of O*NET data in JCV applications.…”

Section: The Present Studymentioning

confidence: 99%

“…This study focused on one of the concerns described by Harvey (2010): specifically, the question of whether derivation-sample Rs of the size typically seen in past O*NET JCV studies are of sufficient magnitude to allow researchers and practitioners to safely conclude that applied uses of these equations will exhibit adequate utility and precision. Based on the enthusiastic recommendations offered by Jeanneret and Strong (2003), LaPolice et al (2008), and Johnson et al (2010), practitioners might well conclude that the mere presence of such sizableappearing multiple Rs provides all the empirical evidence that is necessary to justify the use of JCV prediction models when making applied personnel decisions.…”

mentioning

confidence: 99%

Deriving Synthetic Validity Models: Is R = .80 Large Enough?

Harvey¹

2011

PsycEXTRA Dataset

View full text Add to dashboard Cite

Many view the .50 --.80 Rs typically seen for job-component validation (JCV) models as adequate to justify inferences of validity/utility. This study derived JCV models predicting DOT Strength from O*NET dimensions; although a R = .82 was achieved, errors of prediction were too large to justify its applied use. advanced optimistic conclusions regarding the quality and utility of O*NET data, including its suitability as a data source for quantifying the job activities and demands that are used as predictors when deriving JCV models that predict worker attribute requirements. For example, LaPolice et al. (2008) showed that when predicting the criterion of occupational literacy levels, JCV models based on O*NET data produced multiple Rs ranging from .80 to .83.Although other JCV studies (e.g., Jeanneret & Strong, 2003) have reported derivation-sample Rs that are much lower in magnitude than those found by LaPolice et al. (ranging as low as the .30's, .40's and .50's), JCV proponents remain highly upbeat in their assessments of the potential of JCV to revolutionize the test validation process. For example, Johnson, Steel, Scherbaum, Hoffman, Jeanneret, and Foster (2010) asserted that although "there are still several details to hammer out, especially the number of job components to be used [as predictors] and possibly modifying the O*NET to include aspects gleaned from PAQ-based synthetic validity successes .... there are no theoretical obstacles left" (p. 325, emphasis added), and that JCV "has great potential to advance the science and practice of industrial and organizational psychology" (p. 305, emphasis added).LaPolice et al. (2008) concluded that "these multiple Rs are all very close to the maximum correlation for each dependent variable, suggesting that our models are approaching the best possible prediction" (p. 435, emphasis added). They went on to conclude that "the fact that the multiple Rs are all close to the maximum possible provides evidence for the construct validity of O*NET" (p. 435, emphasis added), and that "in summary, this study suggests that O*NET descriptor data predict literacy scores with a high degree of accuracy, providing evidence for the construct validity of O*NET" (p. 437, emphasis added). Jeanneret and Strong (2003) were similarly optimistic regarding the suitability of O*NET and JCV for mission-critical selection and placement applications.In contrast, others have raised questions regarding the quality of Oinclude the questionable quality and accuracy of data collected using single-item holistic judgments, the lack of utility of O*NET's abstract occupational title taxonomy (which rates a far smaller number of occupational units, or OUs, than the more detailed occupations rated in the DOT), the questionable defensibility and quality of data collected from untrained job incumbents or analysts lacking first-hand job knowledge, and the lack of common-metric ratings of moderate specificity work activities. Regarding the latter, concerns have been raised regarding the appropriateness of ...

show abstract

“…Steel, Huffcutt, and Kammeyer‐Mueller (2006) and Johnson (2007) have presented techniques for implementing synthetic validation strategies, further increasing its accessibility and applicability to selection situations. Johnson et al (2010) recently called for much more widespread application of synthetic validation, including the development of a national database linking predictor and criterion constructs to support synthetic validation.…”

mentioning

confidence: 99%

Validating Synthetic Validation: Comparing Traditional and Synthetic Validity Coefficients

Johnson¹,

Carter²

2010

Personnel Psychology

Self Cite

View full text Add to dashboard Cite

We describe a unique application of a synthetic validation technique to a selection system development project in a large organization. Job analysis data were collected from 4,725 job incumbents and 619 supervisors, and were used to identify 11 job families and 27 job components. We developed 12 tests to predict performance on these job components and conducted a concurrent validation study to collect test and job component data for 1,926 incumbents. We created a test composite for each job component and then chose a test battery for each job family based on its relevant job components. Synthetic validity coefficients were computed for each test battery and compared to traditional validity coefficients that were computed within job families with large sample sizes. The synthetic validity coefficient was very close to the within-family validity coefficient for most job families and was within the bounds of sampling error for all job families. Validities tended to be highest when predictors were weighted according to the number of job components to which they were relevant and job component criterion measures were unit weighted.

show abstract

Validation Is Like Motor Oil: Synthetic Is Better

Cited by 46 publications

References 83 publications

Dissecting Situational Strength: Theoretical Analysis and Empirical Tests

Dissecting Situational Strength: Theoretical Analysis and Empirical Tests

Deriving Synthetic Validity Models: Is R = .80 Large Enough?

Validating Synthetic Validation: Comparing Traditional and Synthetic Validity Coefficients

Contact Info

Product

Resources

About