Randomized Controlled Pilot Study of Video Self‐assessment for Resident Mastoidectomy Training

Abstract

A prospective randomized controlled pilot study was performed to determine if video self-assessment improves competency in mastoidectomy and to assess interrater agreement between expert and resident evaluations of recorded mastoidectomy. Sixteen otolaryngology residents were recorded while performing cadaveric mastoidectomy and randomized into video self-assessment and control groups. All residents performed a second recorded mastoidectomy. Performance was evaluated by blinded experts with a validated assessment scale. Video self-assessment did not lead to greater skill improvement between the first and second mastoidectomy. Interrater agreement was fair to substantial between the expert evaluators and between resident self-evaluations by recall and video review. Agreement between experts and residents was only slight to fair; residents consistently rated their performance higher than experts (P < .05). In conclusion, 1 session of video self-review did not lead to improved competence in mastoidectomy over standard practice. While experts agree on assessments, residents may overestimate their competency in performing cadaveric mastoidectomy.

What are the learning objectives in surgical training – a systematic literature review of the surgical competence framework

“…Learners with greater experience (higher postgraduate year level) tended to have higher self as well as preceptor ratings Technical skills, non-technical skills Learning by reflection Yes Quick et al (2017) [ 67 ] Residents, N = 14 Gastro surgery OR procedure n/a Performance assessment (other report), self-report OSATS Junior residents believe they are more skilled technically; however, they maintain a humble attitude toward less technical aspects, such as operative flow, knowledge, and overall comfort. Technical skills, non-technical skills, knowledge Learning by doing, learning by reflection Yes Jethwa et al (2018) [ 68 ] Residents, N = 16 Otolaryngology Simulation (human cadaveric) RCT Performance assessment (other report), self-report Task-based checklist and global rating scale One session of video self-review did not improve competence over standard practice. Technical skills Learning by doing, learning by reflection No …”

Supplementing Intraoperative Mastoidectomy Teaching With Video-Based Coaching

“…35,36 In studies on cadaveric temporal bone dissection, single timepoint interventions have proven ineffective at improving mastoidectomy competence, whereas, studies of virtual reality mastoidectomies demonstrate improved competence over several sessions. 10,[37][38][39] While the majority of VBC studies have demonstrated improvement in skill acquisition after single preoperative, perioperative or postoperative sessions, only 2 studies have evaluated skills in the operating room and only one of which implemented comparable postoperative VBC. 15,29,40 In this study, significant improvement of surgical skill was demonstrated after a series of coaching sessions.…”

“…23,35,46 In a pilot study on resident mastoidectomy performance, after one session of video self-review, there was no improvement in mastoidectomy competence, and residents' self-assessments remained higher than attendings. 37 Another alternative could be review of expert-generated videos, but in most studies comparing VBC to watching expert-generated videos, surgeons performed better after VBC. 13,15,[47][48][49] Thus, while it is likely that residents obtain some benefit from self-or expert-video review, [49][50][51][52][53] attending presence and feedback remain critical.…”

Examining the Validity of Pediatric Resident Self-Assessment in the Delivery Room

“…Jethwa found that inter-rater agreement between expert and resident assessments was slight to fair with residents consistently rating their performance higher than observers. 18 Isenberg et al demonstrated that medical students underestimated their proficiency in technical skills and overestimated their proficiency in general clinical skills and suggested that self-assessments cannot be used as a sole measure of performance but can provide supplemental information on proficiency even in early training. 19 Overestimating performance may be due to self-serving conscious biases, while underestimating performance may be due to modesty bias or imposter syndrome.…”