Multi-Institutional Development of a Mastoidectomy Performance Evaluation Instrument

Abstract

Objective A method for rating surgical performance of a mastoidectomy procedure that is shown to apply universally across teaching institutions has not yet been devised. This work describes the development of a rating instrument created from a multi-institutional consortium. Design Using a participatory design and a modified Delphi approach, a multi-institutional group of expert otologists constructed a 15 element task-based checklist for evaluating mastoidectomy performance. This instrument was further refined into a 14 element checklist focusing on the concept of safety after using it to rate a large and varied population of performances. Setting Twelve Otolaryngological surgical training programs in the United States. Participants 14 surgeons from 12 different institutions took part in the construction of the instrument. Results By using 14 experts from 12 different institutions and a literature review, individual metrics were identified, rated as to the level of importance and operationally defined to create a rating scale for mastoidectomy performance. Initial use of the rating scale showed modest rater agreement. The operational definitions of individual metrics were modified to emphasize “safe” as opposed to “proper” technique. A second rating instrument was developed based on this feedback. Conclusions Using a consensus building approach with multiple rounds of communication between experts is a feasible way to construct a rating instrument for mastoidectomy. Expert opinion alone using a Delphi method provides face and content validity evidence, however, this is not sufficient to develop a universally acceptable rating instrument. A continued process of development and experimentation to demonstrate evidence for reliability and validity making use of a large population of raters and performances is necessary to achieve universal acceptance.

Cross-Institutional Evaluation of a Mastoidectomy Assessment Instrument

“…The content aspect concerns the fact that the assessment would cover all parts of the domain in question. The development of the test 4 involved experts considering all aspects of the technical skills used in mastoidectomy and paring down the list to the ones they collectively considered the most important.…”

“…We used the two rating instruments described in Kerwin et al 4 The instruments cover the technical skills used in two-handed surgical tool manipulation and bone removal in a mastoidectomy. Ratings did not happen at the same time: all ratings from the first instrument were collected several months before the ratings from the second instrument.…”

“…Using a virtual reality temporal bone simulator system, we administered the original instrument and also altered it to make a second instrument that focused on safety. 4 We acquired a wide selection of mastoidectomies by experts, residents and medical students from 12 institutions, and those mastoidectomies were evaluated using the two instruments.…”