Validity Generalization and Situational Specificity: An Analysis of the Prediction of First—Year Grades in Law School

Abstract

Results from 726 validity studies were analyzed to determine the degree of validity generalization of the Law School Admission Test for predicting firstyear grades in law school. Four validity generalization procedures were used and their results compared. As much as 70% of the variance in observed validity coefficients could be accounted for by differences in the within-study variability of LSAT scores, simple sampling error, and between-study differences in criterion reliability. The 90% credibility value for the true validities was estimated to be .45, and the average true validity was estimated to be .54. Despite the substantial degree of validity generalization, law school and the year the study was conducted explained significant portions of the residual variance in validities. Thus, some degree of situational specificity of validity remained. Hundreds of predictive validity studies areconducted each year at academic institutions.Grades for the first semester or year at a college or professional school are correlated with test scores and previous grade average or rank in class. Usually the results for a particular class at a particular institution are viewed in isolation with little, if any, regard for results obtained in other years or at other institutions. Situational specificity of results seems to be implicitly assumed.

Establishing the criterion validity of the Graduate Medical School Admissions Test (GAMSAT)

“…The results in Table 3 show that only a handful of the relationships are statistically significant, that significant estimates are mostly associated with GAMSAT III and average GPA, that different model specifications lead to different estimates, and that the estimates bounce around quite a bit across institutions. Such cross‐institutional variability has been noted by others 8,22–25 . The predictive power of each model is captured in the variance‐explained statistics, which also fluctuate across models and universities.…”

“…Such cross-institutional variability has been noted by others. 8,[22][23][24][25] The predictive power of each model is captured in the variance-explained statistics, which also fluctuate across models and universities.…”

Armed Services Vocational Aptitude Battery (ASVAB): Predicting Military Criteria from General and Specific Abilities

“…The situational specificity of employment tests for prediction of job performance or proficiency has been assumed by many psychologists over the years. Schmidt, Hunter, and Pearlman (1981) used meta-analytic techniques to show that most of the apparent variability in test validity is due to statistical artifacts such as samplinc' error, unreliability of measurement, unrcliability of criterion measures, and restriction in the range of abilities in specific samples A number of other researchers (Guzzo, Jette, & Katzell 1985;Hyde, 1981;Linn, Harnisch & Dunbar, 1981;Schmidt, Gast-Roseiberg, & Hunter, 1980;Schmidt, Hunter, & Caplan, 1981;Steele & Ovalle, 1981) have also applied meta-analytic techniques to !a~gp numbers of validity studies across scores of different jubs and found itt;e variance in validity coefficients 'hat could not be explained by these four statistical sources of variance (Schmidt, Hunter, Pearlman, & Shane, 1979). Little, if any, variance was explained by factors specific to a given type of criterion.…”

Generalization of Sat® Validity Across Colleges

“…Thus the variance of the square roots of the criterion reliabilities overestimates the amount of validity variance that is accounted for by variation in those reliabilities. Linn et al, (1981) have proposed a plausible distribution of criterion reliabilities, which was used to compute the needed variance.…”