Evaluating the complementary roles of an SJT and academic assessment for entry into clinical practice

Abstract

Although there is extensive evidence confirming the predictive validity of situational judgement tests (SJTs) -017-9755-4 performance. This is an important area of enquiry as despite it being common practice to use both types of methods within a selection system, there is currently no evidence that this approach translates into increased predictive validity of the selection system as a whole, over that achieved by the use of a single selection method. In this preliminary study, the majority of the range of scores achieved by successful applicants to the UK Foundation Programme provided a unique opportunity to address both of these areas of enquiry. Sampling targeted high ([80th percentile) and low (\20th percentile) scorers on the SJT. Supervisors rated 391 trainees' in-role performance, and incidence of remedial action was collected. SJT and academic performance scores correlated with supervisor ratings (r = .31 and .28, respectively). The relationship was stronger between the SJT and in-role performance for the low scoring group (r = .33, high scoring group r = .11), and between academic performance and in-role performance for the high scoring group (r = .29, low scoring group r = .11). Trainees with low SJT scores were almost five times more likely to receive remedial action. Results indicate that an SJT for entry into trainee physicians' first role in clinical practice has good predictive validity of supervisor-rated performance and incidence of remedial action. In addition, an SJT and a measure of academic performance appeared to be complementary to each other. These initial findings suggest that SJTs may be more predictive at the lower end of a scoring distribution, and academic attainment more predictive at the higher end.Adv in Health Sci Educ (2017) 22:401-413 DOI 10.1007/s10459

Selection methods in healthcare professions: where are we now and where next?

“…In this issue alone, professional skills (Cousans et al 2016;Dore et al 2016), innovation (Patterson and Zibarras 2016) and personality (Ferguson and Lievens 2016) are reported. Elsewhere, researchers have focussed on values (e.g.…”

“…In this issue, Cousans et al (2016) provide preliminary evidence for the situational judgement test (SJT, non-academic) and academic performance measures used in postgraduate medical selection in the UK as being complementary in predicting subsequent in-role performance. Research suggests that SJTs may be more predictive at the lower end of a scoring distribution, and academic attainment more predictive at the higher end.…”

Lessons learned from 15 years of non-grades-based selection for medical school

“…A related difficulty is how to find credible measures of performance in future practice that can be related to the personal qualities assessed during selection. Currently, measures used include supervisor ratings and incidence of remedial action in postgraduate training or results on national licensure examinations, with promising outcomes. Nevertheless, it remains extremely difficult to predict at the time of admission to medical school who will be a good doctor many years later.…”

Situational judgment tests reliably measure professional attributes important for clinical practice

“…These results are made available publicly each year in an annual technical report published on the Foundation Programme website,6 and it would be prudent to also refer to full sample data as the basis for commentary. In recent studies, importantly the Foundation SJT positively correlates with subsequent in-training supervisor ratings10 and so provides good evidence of predictive validity.The Foundation SJT is specifically designed to measure nonacademic attributes important for trainees entering clinical practice, and it is important that these attributes are measured reliably at the point of entry to the Foundation Programme in addition to a measure of academic performance. We agree with Najim et al in highlighting that examinations up until this point are largely based on academic performance from General Certificate of Secondary Education (GCSE) exams to medical school finals.…”

“…These results are made available publicly each year in an annual technical report published on the Foundation Programme website,6 and it would be prudent to also refer to full sample data as the basis for commentary. In recent studies, importantly the Foundation SJT positively correlates with subsequent in-training supervisor ratings10 and so provides good evidence of predictive validity.…”