A Missing Data Approach to Correct for Direct and Indirect Range Restrictions with a Dichotomous Criterion: A Simulation Study

Pfaffel, Andreas; Kollmayer, Marlene; Schober, Barbara; Spiel, Christiane

doi:10.1371/journal.pone.0152330

Cited by 21 publications

(21 citation statements)

References 50 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The first correction method we used is Thorndike’s case C formula for indirect selection [37] which estimates predictive validity as a function of the diminution of the standard deviation caused by selection, and the correlations between z HAMNat , zEduAttain and zOutcomeOverall (the influence of zEduAttain renders the selection by HAM-Nat indirect). The second method is a Bayesian type of estimation: Multiple Imputation by Chained Equations (= MICE, [38]).…”

Section: Methodsmentioning

confidence: 99%

Selection and academic success of medical students in Hamburg, Germany

et al. 2019

View full text Add to dashboard Cite

BackgroundStudent selection at Hamburg medical school is based on the combination of a natural science knowledge test (HAM-Nat) and pre-university educational attainment.MethodOf the 1565 medical students enrolled in Hamburg from 2012 to 2015 about half were admitted by an entrance test, half by quotas. First, we analysed sociodemographic determinants of entrance test performance. Then, we used regression analysis to describe the interplay of variables in the prediction of study outcome, the role of sociodemographic factors, and differences in the calibration of educational attainment specific to German federal states.ResultsBetter performance in the entrance test was associated with age over 21, male gender, German nationality, first language German and both parents holding an academic degree – effect sizes were small. No differences were found for the birthplaces of parents (a proxy for migration background). Study outcome differed considerably among admission paths: Students admitted by entrance test or the quota for excellent pre-university educational attainment performed markedly better during the first 3 terms than students admitted by the waiting list quota and the quota for foreign students. Gender differences in study outcome were slight with better performance by males. The relation of pre-university educational attainment to study outcome was moderated by the federal state in which secondary schooling took place. Methods for the equating of state-specific grades are explored. The predictive validity of the HAM-Nat after correction for range restriction was r = .31. The relatively low value of this coefficient may be attributed to 3 factors: 1. self-selection of applicants which leads to a validity-enhancing effect that is not revealed by the predictor-outcome correlation, 2. reduction of variance due to a high selection ratio, and 3. high test difficulty, exceeding the demands of the medical curriculum.ConclusionThe HAM-Nat achieves a small amount of incremental validity over pre-university educational attainment. This effect, obtained from correlational analysis, underestimates the validity of the test, because it does not reflect the role of self-selection and other validity enhancing features of the selection process.

show abstract

Section: Methodsmentioning

confidence: 99%

Selection and academic success of medical students in Hamburg, Germany

et al. 2019

View full text Add to dashboard Cite

show abstract

“…Importantly, the relation obtained from a range-restricted data set underestimates the relation that would be obtained from the (not available) unrestricted data set. This bias must be corrected to provide a more valid population estimate (e.g., Pfafel, Kollmayer, Schober, & Spiel, 2016) -which none of the studies included here did.…”

Section: Study Limitationsmentioning

confidence: 99%

Smart teachers, successful students? A systematic review of the literature on teachers’ cognitive abilities and teacher effectiveness

Bardach¹,

Klassen²

2019

Preprint

View full text Add to dashboard Cite

This study provides a systematic review of the literature on teachers’ cognitive abilities (intelligence test scores and proxies of cognitive abilities such as college entrance exam scores and basic skills test scores) and teacher effectiveness. Twenty-seven studies conducted between 2000 and 2019 constitute the sample for this review. Studies using intelligence test scores were rare, with the results indicating no or negative associations with teacher effectiveness. Studies on proxies of cognitive abilities yielded, at most, small positive relations with teacher effectiveness. However, behind these overall results regarding proxies of cognitive abilities lie interesting heterogeneities, as several studies analyzing different test domains uncover a differentiated pattern of findings. We also identify key limitations related to construct measurement, sampling approaches, statistical analyses and the interpretation and reporting of the included studies, and outline a path for future research on teachers’ cognitive abilities and teacher effectiveness.

show abstract

“…A multitude of methods have been developed to achieve this feat in a variety of testing situations [ 1 ]. The menu of suitable methods comprises multiple correction formulas using different sets of assumptions as well as maximum likelihood methods of missing value imputations such as Bayesian Monte Carlo methods [ 15 – 17 ].…”

Section: Illustrative Scenariomentioning

confidence: 99%

“…4. The MICE algorithm is the most recent and according to its advocates the most accurate approach [ 15 ]. The acronym stands for Multiple Imputation by Chained Equations A multiple imputation analysis consists of three distinct steps: the imputation phase, the analysis phase, and the pooling phase.…”

Section: Illustrative Scenariomentioning

confidence: 99%

“…The smallest confidence interval of estimations for is delivered by EM and the largest by the formulas for direct and indirect range restriction. With the simulation problem at hand, MICE did not perform better than EM although MICE is regarded as the most advanced method for coefficients estimation when data are missing [ 15 ].…”

Section: Study 1: Monte Carlo Simulationmentioning

confidence: 99%

See 1 more Smart Citation

Correcting the predictive validity of a selection test for the effect of indirect range restriction

Zimmermann

Hampe²

2017

BMC Med Educ

View full text Add to dashboard Cite

BackgroundThe validity of selection tests is underestimated if it is determined by simply calculating the predictor-outcome correlation found in the admitted group. This correlation is usually attenuated by two factors: (1) the combination of selection variables which can compensate for each other and (2) range restriction in predictor and outcome due to the absence of outcome measures for rejected applicants.MethodsHere we demonstrate the logic of these artifacts in a situation typical for student selection tests and compare four different methods for their correction: two formulas for the correction of direct and indirect range restriction, expectation maximization algorithm (EM) and multiple imputation by chained equations (MICE). First we show with simulated data how a realistic estimation of predictive validity could be achieved; second we apply the same methods to empirical data from one medical school.ResultsThe results of the four methods are very similar except for the direct range restriction formula which underestimated validity.ConclusionFor practical purposes Thorndike’s case C formula is a relatively straightforward solution to the range restriction problem, provided distributional assumptions are met. With EM and MICE more precision is obtained when distributional requirements are not met, but access to a sophisticated statistical package such as R is needed. The use of true score correlation has its own problems and does not seem to provide a better correction than other methods.

show abstract

A Missing Data Approach to Correct for Direct and Indirect Range Restrictions with a Dichotomous Criterion: A Simulation Study

Cited by 21 publications

References 50 publications

Selection and academic success of medical students in Hamburg, Germany

Selection and academic success of medical students in Hamburg, Germany

Smart teachers, successful students? A systematic review of the literature on teachers’ cognitive abilities and teacher effectiveness

Correcting the predictive validity of a selection test for the effect of indirect range restriction

Contact Info

Product

Resources

About