Assessing the predictive validity of the UCAT—A systematic review and narrative synthesis

Bala, Laksha; Pedder, Stephen; Sam, Amir H.; Brown, Celia

doi:10.1080/0142159x.2021.1998401

Cited by 8 publications

(17 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The standardised coefficients of BMAT Section 2 scores were comparable to those found for UCAT predictive validities (mainly small [0.1–0.29] but some moderate [0.3–0.49] coefficients 16,19 ) and similar to, or exceeding, UMAT predictive validity (variable coefficients, ranging from non‐predictive to coefficients of around 0.2 42,43 ). BMAT Section 2 predictive validity falls below the moderate‐large validity coefficients found for the new MCAT (~0.5–0.7) 44 and for the pre‐2015 MCAT for final exams (~0.5 for the science sections) 41 .…”

Section: Discussionsupporting

confidence: 65%

“…We (and others) posit that BMAT Section 1 seems to test fluid intelligence 15,17,18,20 . Yet in these cohorts it predicted performance at medical school less than the fluid intelligence components of, for example, UCAT 16,19,39,40 …”

Section: Discussionmentioning

confidence: 95%

“…Section 1, 'thinking skills', measures skills in 'problem-solving, argument comprehension, data and graphical interpretation and inference'. 16 Section 2, 'scientific knowledge and applications', assesses knowledge of core mathematics, biology, chemistry and physics to the level of age-16 qualifications. Section 3, the 'writing task', requires test takers to answer one of three short essay questions in 30 min.…”

Section: Box 1 the Biomedical Admissions Test (Bmat)mentioning

confidence: 99%

“…However, there are no studies examining the association between BMAT scores and performance in other medical schools, on clinical examinations or performance in the later years of medical school. This paucity of research contrasts sharply with that reporting on other commercially available selection tests, such as the UCAT, formerly the UK Clinical Aptitude Test (the most widely used admissions test in the UK) 16,19 . Moreover, the two published BMAT studies referred to above and BMAT generally have been criticised for lacking reliability and validity data, with several authors calling for more rigorous and independent research examining the use and properties of BMAT 20,21 …”

Section: Introductionmentioning

confidence: 98%

“…This paucity of research contrasts sharply with that reporting on other commercially available selection tests, such as the UCAT, formerly the UK Clinical Aptitude Test (the most widely used admissions test in the UK). 16,19 Moreover, the two published BMAT studies referred to above and BMAT generally have been criticised for lacking reliability and validity data, with several authors calling for more rigorous and independent research examining the use and properties of BMAT. 20,21 Although these calls for more research were more than 10 years ago, no more studies or technical reports were published by the test company, and no independent research was commissioned.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

BMAT's predictive validity for medical school performance: A retrospective cohort study

et al. 2022

Self Cite

View full text Add to dashboard Cite

Background Although used widely, there is limited evidence of the BioMedical Admissions Test's (BMAT) predictive validity and incremental validity over prior educational attainment (PEA). We investigated BMAT's predictive and incremental validity for performance in two undergraduate medical schools, Imperial College School of Medicine (ICSM), UK, and Lee Kong Chian School of Medicine (LKCMedicine), Singapore. Our secondary goal was to compare the evidence collected with published evidence relating to comparable tools. Methods This was a retrospective cohort study of four ICSM (1188 students, entering 2010–2013) and three LKCMedicine cohorts (222 students, 2013–2015). We investigated associations between BMAT Section 1 (‘Thinking Skills’), Section 2 (‘Scientific Knowledge and Applications’) and Section 3a (‘Writing Task’) scores, with written and clinical assessment performance across all programme years. Incremental validity was investigated over PEA (A‐levels) in a subset of ICSM students. Results When BMAT sections were investigated independently, Section 2 scores predicted performance on all written assessments in both institutions with mainly small effect sizes (standardised coefficient ranges: ICSM: 0.08–0.19; LKCMedicine: 0.22–0.36). Section 1 scores predicted Years 5 and 6 written assessment performance at ICSM (0.09–0.14) but nothing at LKCMedicine. Section 3a scores only predicted Year 5 clinical assessment performance at ICSM with a coefficient <0.1. There were no positive associations with standardised coefficients >0.1 between BMAT performance and clinical assessment performance. Multivariable regressions confirmed that Section 2 scores were the most predictive. We found no clear evidence of incremental validity for any BMAT section scores over A‐level grades. Discussion Schools who wish to assess scientific knowledge independently of A‐levels may find BMAT Section 2 useful. Comparison with previous studies indicates that, overall, BMAT seems less useful than comparable tools. Larger scale studies are needed. Broader questions regarding why institutions adopt certain admissions tests, including those with little evidence, need consideration.

show abstract

Section: Discussionsupporting

confidence: 65%

Section: Discussionmentioning

confidence: 95%

Section: Box 1 the Biomedical Admissions Test (Bmat)mentioning

confidence: 99%

Section: Introductionmentioning

confidence: 98%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

BMAT's predictive validity for medical school performance: A retrospective cohort study

et al. 2022

Self Cite

View full text Add to dashboard Cite

show abstract

Bridging Validity Frameworks in Assessment: Beyond Traditional Approaches in Health Professions Education

Carrillo-Avalos,

Leenen,

Trejo-Mejía

et al. 2023

Teaching and Learning in Medicine

View full text Add to dashboard Cite

Is the awarding gap at UK medical schools influenced by ethnicity and medical school attended? A retrospective cohort study

Brown,

Goss,

Sam

2023

BMJ Open

Self Cite

View full text Add to dashboard Cite

ObjectivesTo better characterise the Awarding Gap (AG) between black, Asian and other minority ethnic (BAME) and white students in UK undergraduate medical education by examining how it affects eight minority ethnicity subgroups (Bangladeshi, black, Chinese, Indian, Pakistani, mixed, other Asian background and other ethnic background) and whether the AG varies by medical school attended.DesignRetrospective cohort study.SettingData extracted from the UK Medical Education Database on students enrolled at 33 UK medical schools in the academic years starting 2012, 2013 and 2014.Participants16 020 ‘Home’ tuition fee status students who sat the University Clinical Aptitude Test on entry to university and obtained a UK Foundation Programme (UKFP) application score on exit.Primary outcome measureUKFP Z-scores on exit from medical school.ResultsThere were significant differences in UKFP Z-scores between ethnicity subgroups. After white students, mixed ethnicity students performed best (coefficient −0.15 standard deviations [SD]) compared with white students, (95% confidence interval [CI] −0.23 to −0.08, p<0.001) and Pakistani students scored lowest (coefficient −0.53 SD, 95% CI −0.60 to −0.46, p<0.001). In pairwise comparisons of scores between all nine individual ethnicity subgroups, 15/36 were statistically significant. The AG varied considerably across medical schools. The largest gap showed the coefficient for BAME was −0.83 SD compared with white students (95% CI −1.18 to −0.49, p<0.001), while the smallest demonstrated no statistically significant difference in performance between BAME and white students (+0.05 SD, 95% CI −0.32 to 0.42, p=0.792).ConclusionsBAME students are significantly disadvantaged by the current UK medical education system. There are clear differences in medical school outcomes between students from different ethnicity subgroups, and the size of the AG also varies by medical school attended. Urgent and effective action must be taken to address the AG and achieve an equal learning environment for our future doctors.

show abstract

Assessing the predictive validity of the UCAT—A systematic review and narrative synthesis

Cited by 8 publications

References 28 publications

BMAT's predictive validity for medical school performance: A retrospective cohort study

BMAT's predictive validity for medical school performance: A retrospective cohort study

Bridging Validity Frameworks in Assessment: Beyond Traditional Approaches in Health Professions Education

Is the awarding gap at UK medical schools influenced by ethnicity and medical school attended? A retrospective cohort study

Contact Info

Product

Resources

About