Evaluating the validity of the selection measures used for the UK’s foundation medical training programme: a national cohort study

Abstract

ObjectivesCurrently relative performance at medical school (educational performance measure (EPM) decile), additional educational achievements and the score on a situational judgement test (SJT) are used to rank applicants to the UK Foundation Years postgraduate medical training programme. We sought to evaluate whether these three measures were predictive of subsequent successful completion of the programme, and thus were valid selection criteria.MethodsData were obtained from the UK Medical Education Database (UKMED) on 14 131 UK applicants to the foundation programme starting in 2013 and 2014. These data included training outcomes in the form of Annual Reviews of Competency Progression (ARCPs), which indicated whether the programme was successfully completed. The relationship between applicants’ performance on the three selection measures to the odds of successful programme completion were modelled.ResultsOn univariable analyses, all three measures were associated with the odds of successful completion of the programme. Converting the SJT score to deciles to compare the effect sizes suggested that one decile increase in the EPM increased the odds of completing the programme by approximately 15%, whereas the equivalent value was 8% for the SJT scores. On multivariable analyses (with all three measures included in the model), these effects were only independently and statistically significant for EPM decile (OR 1.14, 95% CI 1.10 to 1.18, p<0.001) and SJT z-score decile (OR 1.05, 95% CI 1.01 to 1.09, p=0.02).ConclusionsThe EPM decile and SJT scores may be effective selection measures for the foundation programme. However, educational achievements does not add value to the other two measures when predicting programme completion. Thus, its usefulness in this context is less clear. Moreover, our findings suggest that the weighting for the EPM decile score, relative to SJT performance, should be increased.

Situational judgement test validity for selection: A systematic review and meta‐analysis

“…A total of 10 were cross-sectional studies, [27][28][29][30][31][32][33][34][35]55 where the outcome was measured at the same time or in the same selection cycle as taking the SJT. A total of 17 were cohort studies 17,[36][37][38][39][40][41][42][43][44][45][46][47][48][49][50][51] that had a follow-up period before the outcome of interest was measured. Three studies employed a mixture of F I G U R E 1 PRISMA (preferred reporting items for systematic reviews and meta-analyses) flowchart for the systematic review Abbreviations: CINAHL, Cumulative Index to Nursing and Allied Health Literature; EMBASE, Excerpta Medica Database; ERIC, Educational Resources Information Center; EThos, Electronic Theses Online Service; UCAT, University Clinical Aptitude Test cross-sectional and more distal outcomes.…”

“…[52][53][54] The length of followup across the cohort studies varied from 1 to 9 years after taking the SJT. Full details of the included studies are listed in Table S1. A total of 11 studies 17,28,29,35,[40][41][42][43]47,49,50 looked at undergraduate selection for medical school entry, five studies 31,36,37,45,51 at entry to Foundation Year training programmes (the first 2 years of post-qualification training in the United Kingdom [UK]) and 14 studies 27,30,[32][33][34]38,39,44,46,48,[52][53][54][55] at entry to specialty training. The youngest participant mean age was 17.9 years and the oldest was 34.0 years.…”

A scoping review of admission criteria and selection methods in nursing education

“…Therefore, it is more cost-effective than multiple mini interview and selection centers. Despite the use of situational judgment tests for student selection in some health care professions (91)(92)(93)(94)(95), no research evidence was found regarding the use of this method for nursing student selection…”

Situational judgement tests for selection: Traditional vs construct‐driven approaches

“…Moreover, a previous review of the use of SJTs for the evaluation of non‐academic traits in general concluded that there was evidence of cost‐effectiveness compared to other approaches (eg interviews) . Emerging evidence also indicates that, to date, in a variety of undergraduate and postgraduate medical selection settings, at least modest correlations are observed between performance on these traditional SJTs and subsequent relevant outcomes that reflect aspects of social functioning . This suggests, at least in this context, such selection SJTs are generally estimating, to some extent, knowledge of interpersonal functioning.…”

“…Indeed, there may be little relationship between performance on such instruments and academic achievement in the early years of medical undergraduate education . Evidence for the validity of SJTs in evaluating non‐academic traits for medical selection purposes has also been sought by linking the test scores to outcomes that may require a degree of interpersonal competence, such as performance in high‐fidelity simulations of clinical practice in primary care physicians, successful completion of the first stage of postgraduate medical training or performance in the first year as a general practitioner . Cross‐sectional evidence that supports the concept of SJTs measuring constructs relevant to interpersonal functioning also exists, in the form of correlations with multiple mini‐interviews .…”