A meta-analytic perspective on the valid use of subjective human judgement to make medical school admission decisions

Abstract

While medical educators appear to believe that admission to the medical school should be governed, at least in part, by human judgement, there has been no systematic presentation of evidence suggesting it improves selection. From a fair testing perspective, legal, ethical, and psychometric considerations, all dictate that the scientific evidence regarding human judgement in selection should be given consideration. To investigate the validity of using human judgements in admissions, multi-disciplinary meta-analytic research evidence from the wider literature is combined with studies from within medical education to provide evidence regarding the fairness and validity of using interviews and holistic review in medical school admissions. Fourteen studies, 6 of which are meta-analytic studies that summarized 292 individual studies, were included in the final review. Within these studies, a total of 33 studies evaluated the reliability of the traditional interview. These studies reveal that the interview has low to moderate reliability (~.42) which significantly limits its validity. This is confirmed by over 100 studies examining interview validity which collectively show interview scores to be moderately correlated with important outcome variables (corrected value ~.29). Meta-analyses of over 150 studies demonstrate that mechanical/formula-based selection decisions produce better results than decisions made with holistic/clinical methods (human judgement). Three conclusions regarding the use of interviews and holistic review are provided by these meta-analyses. First, it is clear that the traditional interview has low reliability and that this significantly limits its validity. Second, the reliable variance from interview scores appears moderately predictive of outcomes that are relevant to consider in medical school admission. And third, the use of holistic review as a method of incorporating human judgement is not a valid alternative to mechanical/statistical approaches as the evidence clearly indicates that mechanistic methods are more predictive, reliable, cost efficient, and transparent.

Holistic Admissions Review in Nursing Education: A Concept Analysis Using Rodgers’ Evolutionary Method

“…A student’s ability to perform the interpersonal and interprofessional tasks of the discipline dictates the success of the future professional. Standardized scores cannot assess in-person communication styles, quality of caring for patients (Kreiter et al, 2018), student leadership abilities (Glazer et al, 2015), community involvement (Glazer et al, 2015; Grabowski, 2018), ethics (Kreiter et al, 2018), or adherence to discipline-specific values. Holistic review considers student cohort teams and determines whether groups will fit well together, academically or professionally (Katsufrakis et al, 2016; Love et al, 2016).…”

“…Medical schools consistently described legal guidelines or rulings regarding holistic review implementation (Grabowski, 2018; Thomas & Dockter, 2019). Numerous sources described concerns regarding problematic bias, reliability, or validity of subjective human decision-making during holistic review processes (Kreiter et al, 2018; Sims & Lynch, 2016; Wilson et al, 2019), and some described the importance of standardizing assessment data to enhance reliability or validity (Love et al, 2016; Marshall et al, 2020). Wilson et al (2019) noted that faculty may experience preferences related to personal or social similarities with applicants and feel legitimized by their expert experiences as objective judges, even when evidence shows some degree of bias.…”

“…Faculty or residents may also view holistic review initiatives as biased against students who are not underrepresented in medicine (Garrick et al, 2019). Additional concerns included time investments, training and development costs, and increased faculty workload (Grabowski, 2018; Katsufrakis et al, 2016; Kreiter et al, 2018).…”

Association between Overall Academic Achievement, Cognitive, and Non-Cognitive Variables among Medical and Allied Health Students: A 10-Year Review

“…The selection process for medical and allied college applicants involves assessing their cognitive (secondary school certificate grades and pre-admission aptitude test score) and non-cognitive characteristics (demographic details). Where selection based on cognitive elements is governed by test score metrics, selection based on non-cognitive elements is governed by human judgment (Kreiter et al, 2018). Consequently, some researchers suggest that pre-admission aptitude tests alone may not be comprehensive enough, as they often rely on the same constructs as secondary school certificate examinations (Roach et al, 2019;Mwandigha et al, 2018).…”

Main and Regional Campus Assessments of Applicants to a Rural Physician Leadership Program

“…[3][4][5] For example, holistic approaches advocate consideration of multiple factors other than academic performance, [6][7][8] including an individual's "fit" with a medical program's social mission. [9][10] Yet, underlying most selection criteria or methods are individual ratings or judgments of some personal quality, aptitude, or behavior 11 is information limited by the reliability of the ratings and the representativeness of the encounter. 12,13 For programs with specific foci, such as those accepting applicants directly into rural paths, tracks, or concentrations, the admission process may accommodate supplemental values, interests, or stakeholder perspectives.…”