Exploring the use of rating scales with entrustment anchors in workplace‐based assessment

Abstract

Purpose Competency‐based medical education (CBME) has prompted widespread implementation of workplace‐based assessment (WBA) tools using entrustment anchors. This study aimed to identify factors that influence faculty's rating choices immediately following assessment and explore their experiences using WBAs with entrustment anchors, specifically the Ottawa Surgical Competency Operating Room Evaluation scale. Method A convenience sample of 50 semi‐structured interviews with Emergency Medicine (EM) physicians from a single Canadian hospital were conducted between July and August 2019. All interviews occurred within two hours of faculty completing a WBA of a trainee. Faculty were asked what they considered when rating the trainee's performance and whether they considered an alternate rating. Two team members independently analysed interview transcripts using conventional content analysis with line‐by‐line coding to identify themes. Results Interviews captured interactions between 70% (26/37) of full‐time EM faculty and 86% (19/22) of EM trainees. Faculty most commonly identified the amount of guidance the trainee required as influencing their rating. Other variables such as clinical context, trainee experience, past experiences with the trainee, perceived competence and confidence were also identified. While most faculty did not struggle to assign ratings, some had difficulty interpreting the language of entrustment anchors, being unsure whether their assessment should be retrospective or prospective in nature, and if/how the assessment should change whether they were ‘in the room’ or not. Conclusions By going to the frontline during WBA encounters, this study captured authentic and honest reflections from physicians immediately engaged in assessment using entrustment anchors. While many of the factors identified are consistent with previous retrospective work, we highlight how some faculty consider factors outside the prescribed approach and struggle with the language of entrustment anchors. These results further our understanding of ‘in‐the‐moment’ assessments using entrustment anchors and may facilitate effective faculty development regarding WBA in CBME.

Exploring how feedback reflects entrustment decisions using artificial intelligence

“…This finding corroborates the work of Dolan et al and Dudek et al on the effects of using an entrustment scale on feedback, as well as several studies concluding that constructive feedback should be specific and contain future direction 45–48 . In addition, our results support Robinson et al's recent finding from qualitative interviews that the amount of guidance a supervisor provided to a trainee was the primary factor that influenced their entrustment decision, where an increasing need for guidance led to lower ES levels 11 . While our results about low ES levels are thus expected in the context of these prior studies, it is somewhat surprising that the level of narrative detail was also increased at the highest ES levels.…”

“…[45][46][47][48] In addition, our results support Robinson et al's recent finding from qualitative interviews that the amount of guidance a supervisor provided to a trainee was the primary factor that influenced their entrustment decision, where an increasing need for guidance led to lower ES levels. 11 While our results about low ES levels are thus expected in the context of these prior studies, it is somewhat surprising that the level of narrative detail was also increased at the highest ES levels. This finding may relate to how supervisors justified their choice of higher entrustment and the availability of evidence to support it.…”

“…9,14 These factors were largely identified using supervisors' recall of prior entrustment decisions or projections of how they would approach future encounters. 15,16 While a recent study mitigated potential recall bias, by performing interviews with supervisors shortly after in-the-moment (or 'ad-hoc') 15,17 entrustment decisions, 11 a potential for distortion may still remain, because supervisor self-reports during interviews may not necessarily reflect their accountability to trainee stakeholders. The interpersonal factors highlight the importance of communication between supervisor and trainee in navigating entrustment, as is also reflected in Telio et al's work on the 'educational alliance'.…”

“…Because entrustment decisions are influenced by several key factors, it could be inferred that guidance for how trainees can achieve different levels of entrustment would also be shaped by those factors 8–10 . Prior research explored these factors by examining supervisors' interpretations of their own thought processes around entrustment but not in the context of feedback interactions with trainees 11–13 . Entrustment‐determining factors include the complexity of the task and the clinical context in which the task is to be completed.…”

“…[8][9][10] Prior research explored these factors by examining supervisors' interpretations of their own thought processes around entrustment but not in the context of feedback interactions with trainees. [11][12][13] Entrustment-determining factors include the complexity of the task and the clinical context in which the task is to be completed. Interpersonal factors include the trustworthiness of the trainee for the task (including competence and experience), the supervisor's tendency to trust (including tolerance of risk and ability provide support) and the relationship between the trainee and supervisor (including their mutual trust in each other).…”