A Note on Presenting What Predictive Validity Numbers Mean

Abstract

Descriptions of validity results based solely on correlation coefficients or percent of the variance accounted for are not merely difficult to interpret, they are likely to be misinterpreted. Predictors that apparently account for a small percent of the variance may actually be highly important from a practical perspective. This study combined two existing data sets to demonstrate alternative methods of showing the value of the Graduate Record Examination General Test (GRE) as an indicator of first-year graduate grades. The combined data sets contained 4,451 students in six graduate fields: biology, chemistry, education, English, experimental psychology, and clinical psychology. Students within a department were divided into quartiles based on GRE scores and on undergraduate grade point average (UGPA), and the percent of students in the top and bottom quartiles earning a 3.8 or higher GPA in their first year of graduate study was noted. Even after controlling for undergraduate GPA quartiles (i.e., looking at GRE quartile differences within GPA quartiles), substantial differences related to GRE quartile remained.Results of predictive validity studies typically are summarized in terms of simple correlations, multiple correlations, and increments in multiple correlations with additional variables. Although these coefficients provide convenient summaries, they are difficult for lay audiences (and even trained researchers) to interpret. To anyone who is unfamiliar with correlations in the social sciences, a correlation (r)

Predicting Performance in the First-Year of Pharmacy School

“…The uncorrected correlation values between PCAT composite scores and first-year pharmacy GPAs shown in Table 2 compare favorably with those from a study that found similar uncorrected correlation values of about 0.30 between the Graduate Record Examination (GRE) general test and first-year graduate school GPAs, and to the 90% credibility interval of 0.32-0.68 determined for PCAT total in a meta-analysis of PCAT predictive validity studies. 1,6 However, these correlations are somewhat lower than those found in a previous study conducted by the publisher of the PCAT. 7 The higher mean PCAT scores, entering GPAs, and first-year pharmacy GPAs found in the current study may reflect higher admission standards, which would result in greater range restriction and somewhat lower correlations between entering variables and first-year GPAs.…”

“…6 Table 3 shows that the addition of PCAT scores to entering GPAs represents an incremental variance of 6%. While accurate, such conventional portrayals of incremental variance may appear misleadingly small and may not clearly suggest the value that PCAT scores add to entering GPAs.…”

“…9 The difference in R 2 values between PCAT scores and entering GPAs is similar to the increase in explained variance (incremental variance) found for the GRE and for other professional and school admission tests, which is typically less than 10%. 6,10 The unstandardized regression coefficients shown in Table 3 indicate the amount of change in first-year pharmacy GPA predicted for each 1-point increase in a variable. The significant positive contributions of PCAT biology and chemistry scores in Models 1 and 3 and the stronger contribution of entering science GPA in Models 2 and 3 are consistent with the correlation findings presented in Table 2, which show that scores on the scienceoriented PCAT subtests and the entering science GPAs were the factors most related to subsequent first-year pharmacy GPAs.…”

New Perspectives on the Validity of the GRE® General Test for Predicting Graduate School Grades

“…Bridgeman, Burton, and Cline () suggested a valuable comparison of outcomes for top versus bottom GRE score quartiles of enrolled students to illustrate predictive validity. Limitations of this approach include omitting part of the score distribution, communicating validity for enrolled students rather than the applicant pool, and conveying only univariate information when decision makers might simultaneously use more than one predictor.…”

How the unintended consequences of organizational interventions complicate the assessment of economic utility

“…In addition to studies of methods for estimating the economic impact of organizational interventions, there has been a lively literature dealing with the credibility and meaning of these estimates to users (e.g., Bridgeman, Burton and Cline, 2009;Hazer and Highhouse, 1997;Whyte and Latham, 1997). One conclusion from this literature is that simply presenting financial projections to end users without a clear and compelling description of how interventions in organizations actually deliver these benefits tends to undermine the credibility of projections of financial benefits.…”