Predictive validity of the UKCAT for medical school undergraduate performance: a national prospective cohort study

Abstract

BackgroundThe UK Clinical Aptitude Test (UKCAT) has been shown to have a modest but statistically significant ability to predict aspects of academic performance throughout medical school. Previously, this ability has been shown to be incremental to conventional measures of educational performance for the first year of medical school. This study evaluates whether this predictive ability extends throughout the whole of undergraduate medical study and explores the potential impact of using the test as a selection screening tool.MethodsThis was an observational prospective study, linking UKCAT scores, prior educational attainment and sociodemographic variables with subsequent academic outcomes during the 5 years of UK medical undergraduate training. The participants were 6812 entrants to UK medical schools in 2007–8 using the UKCAT. The main outcome was academic performance at each year of medical school. A receiver operating characteristic (ROC) curve analysis was also conducted, treating the UKCAT as a screening test for a negative academic outcome (failing at least 1 year at first attempt).ResultsAll four of the UKCAT scale scores significantly predicted performance in theory- and skills-based exams. After adjustment for prior educational achievement, the UKCAT scale scores remained significantly predictive for most years. Findings from the ROC analysis suggested that, if used as a sole screening test, with the mean applicant UKCAT score as the cut-off, the test could be used to reject candidates at high risk of failing at least 1 year at first attempt. However, the ‘number needed to reject’ value would be high (at 1.18), with roughly one candidate who would have been likely to pass all years at first sitting being rejected for every higher risk candidate potentially declined entry on this basis.ConclusionsThe UKCAT scores demonstrate a statistically significant but modest degree of incremental predictive validity throughout undergraduate training. Whilst the UKCAT could be considered a fairly crude screening tool for future academic performance, it may offer added value when used in conjunction with other selection measures. Future work should focus on the optimum role of such tests within the selection process and the prediction of post-graduate performance.Electronic supplementary materialThe online version of this article (doi:10.1186/s12916-016-0682-7) contains supplementary material, which is available to authorized users.

Situational judgement test validity for selection: A systematic review and meta‐analysis

“…6 Moreover, the predictor-outcome coefficients we observed in the present review are frequently similar in magnitude, and sometimes larger, compared to those cited for cognitive assessments used to evaluate problem-solving ability in medical school applicants. [59][60][61][62] There Over half (n = 17) of the studies 17,27,[30][31][32]37,38,[40][41][42][43]45,46,48,51,52,55 identified attempted to estimate the incremental validity of the SJT being evaluated, above and beyond other selection assessments, such as tests of cognitive ability, academic performance and clinical knowledge. Of those that did, only two 31,45 did not report any evidence of incremental validity.…”

“…In this regard, numerical simulation methods, such as those previously applied to medical selection situations, may be useful. 5,60 As the widespread use of SJTs in under- More generally, it should be highlighted that all the included studies were from high-income countries. Previous research has found that SJT methodology is typically transportable for use in recruitment settings in other countries.…”

A comparison of undergraduate outcomes for students from gateway courses and standard entry medicine courses

“…A levels are Graded A* to F, with A* being the highest grade and typically medical schools require three A levels with A* or A grades. Studies have reported academic attainment and UCAT scores to be significant predictors of medical school outcome [13,14]. Mwandigha et al, who found UCAT scores added some predictive value in addition to secondary school achievement for undergraduate academic performance, supports this [15].…”

2018 Ottawa consensus statement: Selection and recruitment to the healthcare professions

“…Undergraduate Medical and Health Sciences Aptitude Test; UMAT) versus hybrids of the two types (e.g., the Medical College Admissions Test MCAT 2015). Performance on aptitude tests has been compared to mainly midpoint or endpoint training outcome measures, such as exam performance early in the curriculum (McManus, Dewberry, Nicholson, & Dowell, 2013) and at the end of medical school (Tiffin et al, 2016). There is equivocal evidence that aptitude tests may or may not favour certain types of candidates, which may have implications for fairness and widening access to medicine (Tiffin, Dowell, & McLachlan, 2012;Lievens, Patterson, Corstjens, Martin, & Nicholson, 2016;Kumar, Roberts, Bartle, & Eley, 2018).…”