Reliability of a Structured Interview Scoring Instrument for a Canadian Postgraduate Emergency Medicine Training Program

Abstract

The reliability of the overall interview scores was very high. The intraclass correlations for each interviewer's VAS scores were also high, but interrater correlations within interview teams were moderate and not higher than those across interview teams. This study suggests that an interview assessment instrument can be highly reliable overall and that interviewers base scores on an overall global impression.

Selection for postgraduate training

“…The interview, however, is a less valid and less reliable method that mostly assesses knowledge 7 . As a result, modifications to the interview, such as the structured standardised interview, 8 and the multiple mini‐interview (MMI), 3 have been tried with considerable success. However, unless modified to assess live interpersonal interactions and practical skills, as in some MMI stations, the classical interview format cannot assess practical skills.…”

AAIM Recommendations to Promote Equity and Inclusion in the Internal Medicine Residency Interview Process

“… 51 MMI may mitigate bias by reducing the variability of questions and limiting the impact of both the interview context and interviewer biases (e.g., gender, race, or leniency/stringency tendency). 51 , 54 , 55 , 56 MMIs include scenarios that may provide insight into an applicant's non-cognitive domains, such as interpersonal and communication skills. 51 Studies have shown improved reliability as well as perceived fairness, effectiveness, and acceptability, 55 , 56 , 57 , 58 although some applicants felt it difficult to establish a connection with interviewers during MMIs.…”

A transparent and defensible process for applicant selection within a Canadian emergency medicine residency program

“…We incorporated strong assessment practices such as criterion-referenced rating scales 22 and multiple assessments from diverse raters. 23 Furthermore, we assessed characteristics that are associated with success in residency (reference letters and publication numbers 17 ) and have demonstrated reliability (MMIs 24 consisting of structured interviews with predefined assessment tools 25 ). Informal communications with other programs suggest that our use of simultaneous cloud-based rating, Z-scores, and heat maps are innovations that are not commonly used.…”