Changing abilities vs. changing tasks: Examining validity degradation with test scores and college performance criteria both assessed longitudinally.

Dahlke, Jeffrey A.; Kostal, Jack W.; Sackett, Paul R.; Kuncel, Nathan R.

doi:10.1037/apl0000316

Cited by 9 publications

(10 citation statements)

References 50 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Second, it is also possible that the criterion, career choice, changes over time (e.g., Dahlke, Kostal, Sackett, & Kuncel, 2018). In particular, individuals may gravitate toward more congruent environments over the course of their careers (Judge, Higgins, Thoresen, & Barrick, 1999; Schneider, 1987).…”

Section: Discussionmentioning

confidence: 99%

How accurate are interest inventories? A quantitative review of career choice hit rates.

Hanna¹,

Rounds²

2020

Psychological Bulletin

View full text Add to dashboard Cite

Every year, millions of people use interest inventories to help them make educational and career choices. The present meta-analysis examines the criterion-related validity of interest inventories in predicting career choice outcomes, such as college major choice and occupational membership. This analysis of predictive hit rates incorporates over 75 years of research investigating the accuracy of interest inventories. Using a binomial-normal meta-analytic model to quantitatively estimate the overall hit rate, the present analysis found that measured interests attain an estimated predictive accuracy rate of 50.3% in successfully predicting career choice.Due to a substantial amount of true heterogeneity in effect sizes, we tested several potential moderators. In particular, the hit rate accuracy was moderated by concurrent versus predictive criterion assessment, criterion interest category, interest inventory, type of interest inventory scale, career choice criterion, method used to match criterion to a scale, and hit rate method.Additionally, the present study is an attempt to reintroduce base rates into the evaluation of predictive accuracy. Implications for future research are discussed.iii TABLE OF CONTENTS

show abstract

Section: Discussionmentioning

confidence: 99%

How accurate are interest inventories? A quantitative review of career choice hit rates.

Hanna¹,

Rounds²

2020

Psychological Bulletin

View full text Add to dashboard Cite

show abstract

“…Note that the changing-person perspective is not what H1 is about, but it is important to keep in mind the fact that declining validity has more than one interpretation. We note however that the changing-task perspective gained compelling support from a recent study (Dahlke, Kostal, Sackett, & Kuncel, 2018) in which predictors (cognitive test scores) and performance criteria (college performance) were measured over multiple occasions rather than just once, thus making it possible to disentangle the two interpretations of declining validity. This study showed that the validity changes were mostly due to the time at which the criterion was measured and were barely influenced by the time at which the predictor was assessed.…”

Section: Which Abilities Dictate Success On the Task At Different Stamentioning

confidence: 91%

Mechanisms of working memory training: Insights from individual differences

2019

View full text Add to dashboard Cite

Computerized working-memory training (WM), despite typically yielding large practice effects in the training task, transfers reliably almost only to similar tasks and barely transfers to Fluid Intelligence (Gf). We hypothesized that WM training tasks gradually become less related to Gf due to the development of task-specific skills that reduce reliance on WM. As a result, what is being trained in the advanced stages of training is weakly related to WM and Gf. This hypothesis leads to predicting that with training progression, there would be a gradual change in the rank-ordering of individuals (quasi-simplex) in the per-session scores of the training task coupled with a trend in reduction in Gf loadings of these scores. We reanalyzed individual differences in per-session scores in the training task from two moderately large-scale published studies. Results show that, as predicted, the correlations between per-session scores decreased with increasing temporal distance between sessions, suggesting a quasi-simplex pattern indicative of a gradual change in the rank-ordering of individuals. However, contrary to the prediction, the training tasks maintained or even tended to increase their Gf loading with training progression. We provide post-hoc accounts for these results, some which challenge prevalent assumptions beyond the attempt to improve Gf through improving WM.

show abstract

“…These data were nonrandomly sampled by virtue of being collected from schools that used the SAT as a selection device and that consented to provide students’ grade data to the College Board for research purposes. The database we analyzed has been used in other published research (e.g., Beatty et al, 2015; Dahlke et al, 2018, 2019; Higdem et al, 2016; Kostal et al, 2015; Shewach et al, 2017; Yu et al, 2016), but the present study reports unique analyses of these data and draws novel insights.…”

Section: Methodsmentioning

confidence: 99%

On the assessment of predictive bias in selection systems with multiple predictors.

Dahlke¹,

Sackett²

2022

Journal of Applied Psychology

Self Cite

View full text Add to dashboard Cite

There is a long history of examining assessments used in college admissions or personnel selection for predictive bias, also called differential prediction, to determine whether a selection system predicts comparable levels of performance for individuals from different demographic groups who have the same assessment scores. We expand on previous research that has considered predictive bias in individual predictor variables to (a) examine magnitudes of differential prediction in multipredictor selection systems and (b) explore how differences in prediction generalize across samples. We also share updated methods for computing standardized effect sizes for categorically moderated regression models that facilitate the metaanalysis of differential prediction effects. Our findings highlight the importance of analyzing composite predictors when testing for predictive bias in compensatory selection systems and demonstrate the generalizability of long-observed differential prediction trends by race/ethnicity.

show abstract

Changing abilities vs. changing tasks: Examining validity degradation with test scores and college performance criteria both assessed longitudinally.

Cited by 9 publications

References 50 publications

How accurate are interest inventories? A quantitative review of career choice hit rates.

How accurate are interest inventories? A quantitative review of career choice hit rates.

Mechanisms of working memory training: Insights from individual differences

On the assessment of predictive bias in selection systems with multiple predictors.

Contact Info

Product

Resources

About