Objectives
Global, Covid-driven restrictions around face-to-face interviews for healthcare student selection have forced admissions staff to rapidly adopt adapted online systems before supporting evidence is available. We have developed, what we believe is, the first fully automated interview grounded in Multiple Mini-Interview methodology. This study aimed to explore test re-test reliability, acceptability and usability of the system.
Design, setting and participants
Mixed-methods feasibility study in Physician Associate programmes from two United Kingdom and one United States university during 2019 to 2020.
Primary, secondary outcomes
Feasibility measures (test retest reliability acceptability and usability) were assessed using intra-class correlation, descriptive statistics, thematic and content analysis.
Methods
Volunteers took (Test 1), then repeated (Test 2), the automated MMI, with a seven-day interval, then completed an evaluation questionnaire. Admissions staff participated in focus group discussions.
Results
Sixty-two students and seven admission staff participated; 34 students and four staff from UK and 28 students and three staff from US universities.
Good-excellent test-retest reliability was observed with Test 1 and Test 2 ICC between 0.62-0.81 p< 0.001 when assessed by individual total scores (range 80.6-119), station total scores 0.6-0.91, p< 0.005, individual site (all ICC ≥0.76, p<0.001) and mean test retest across sites 0.82, p<0.001 (95% CI 0.7-0.9).
Admissions staff reported potential to reduce resource costs and bias through a more objective screening tool for pre-selection or to replace some MMI stations in a hybrid model. Maintaining human interaction through touch points was considered essential.
Users positively evaluated the system, stating it was intuitive with an accessible interface. Concepts chosen for dynamic probing needed to be appropriately tailored.
Conclusion
These preliminary findings suggest that the system is reliable, generating consistent scores for candidates and is acceptable to end-users provided human touchpoints are maintained. Thus, there is evidence for the potential of such an automated system to augment healthcare student selection processes.