Construct validity of cadaveric temporal bones for training and assessment in mastoidectomy

Awad, Zaid; Tornari, Chrysostomos; Ahmed, Shahanaz; Tolley, Neil

doi:10.1002/lary.25310

Cited by 22 publications

(33 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Our results for individual items are similar to those in the reports of Laeeq et al 11 and Awad et al 12 Other instruments, such as the Welling Scale and the one by Zirkle et al 13 have also been developed, and a review can be read in Sethia et al 7 . The checklists in the current work are similar to those of the Johns Hopkins assessment.…”

Section: Comparison With Other Studiessupporting

confidence: 89%

Cross-institutional evaluation of a mastoidectomy assessment instrument

Kerwin

Hittle

Stredney

et al. 2017

Preprint

View full text Add to dashboard Cite

ObjectiveThe objective of this work is to obtain validity evidence for an evaluation instrument used to assess the performance level of a mastoidectomy. The instrument has been previously described and had been formulated by a multi-institutional consortium. DesignMastoidectomies were performed on a virtual temporal bone system and then rated by experts using a previously described 15 element task-based checklist. Based on the results, a second, similar checklist was created and a second round of rating was performed. SettingTwelve otolaryngological surgical training programs in the United States. Participants65 mastoidectomy performances were evaluated coming from 37 individuals with a variety of temporal bone dissection experience, from medical students to attending physicians. Raters were attending surgeons from 12 different institutions. ResultsIntraclass correlation (ICC) scores varied greatly between items in the checklist with some being low and some being high. Percentage agreement scores were similar to previous rating instruments. There is strong evidence that a high score on the task-based checklist is necessary for a rater to consider a mastoidectomy to be performed at the level of an expert but a high score is not a sufficient condition. Rewording of the instrument items to focus on safety does not result in increased reliability of the instrument. The strong result of the Necessary Condition Analysis suggests that going beyond simple correlation measures can give extra insight into grading results. Additionally, we suggest using a multiple point scale instead of a binary pass/fail question combined with descriptive mastery levels. Conclusions

show abstract

Section: Comparison With Other Studiessupporting

confidence: 89%

Cross-institutional evaluation of a mastoidectomy assessment instrument

Kerwin

Hittle

Stredney

et al. 2017

Preprint

View full text Add to dashboard Cite

show abstract

“…Fifty‐one studies were included in the review, of which 47 were full‐text original research articles and four were conference posters. The main characteristics of studies, including OCEBM and mean MERSQI scores are shown in Tables S2 – S5 (supporting information).…”

Section: Resultsmentioning

confidence: 99%

“…Various score‐based methods were also used, including procedure scores, global rating scale (GRS), OSATS (Objective Structured Assessment of Technical Skills in Surgery) and the GOALS (Global Operative Assessment of Laparoscopic Skills) scale. Seven of the 23 studies were RCTs and 16 were cohort studies. Of the seven RCTs, three compared cadaveric simulation with no simulation training, and four compared cadaveric simulation with low‐fidelity simulation.…”

Section: Resultsmentioning

confidence: 99%

Systematic review of the current status of cadaveric simulation for surgical training

James

Chapman

Pattison

et al. 2019

British Journal of Surgery

View full text Add to dashboard Cite

BackgroundThere is growing interest in and provision of cadaveric simulation courses for surgical trainees. This is being driven by the need to modernize and improve the efficiency of surgical training within the current challenging training climate. The objective of this systematic review is to describe and evaluate the evidence for cadaveric simulation in postgraduate surgical training.MethodsA PRISMA‐compliant systematic literature review of studies that prospectively evaluated a cadaveric simulation training intervention for surgical trainees was undertaken. All relevant databases and trial registries were searched to January 2019. Methodological rigour was assessed using the widely validated Medical Education Research Quality Index (MERSQI) tool.ResultsA total of 51 studies were included, involving 2002 surgical trainees across 69 cadaveric training interventions. Of these, 22 assessed the impact of the cadaveric training intervention using only subjective measures, five measured impact by change in learner knowledge, and 23 used objective tools to assess change in learner behaviour after training. Only one study assessed patient outcome and demonstrated transfer of skill from the simulated environment to the workplace. Of the included studies, 67 per cent had weak methodology (MERSQI score less than 10·7).ConclusionThere is an abundance of relatively low‐quality evidence showing that cadaveric simulation induces short‐term skill acquisition as measured by objective means. There is currently a lack of evidence of skill retention, and of transfer of skills following training into the live operating theatre.

show abstract

“…Both of the scales use a list of evaluation items with ratings of one to five. Work by Laeeq et al 7 and Awad et al 8 show some validity evidence for that instrument but in only a small number of institutions.…”

Section: Previous Workmentioning

confidence: 94%

Multi-Institutional Development of a Mastoidectomy Performance Evaluation Instrument

Kerwin

Hittle

Stredney

et al. 2017

Journal of Surgical Education

View full text Add to dashboard Cite

Objective A method for rating surgical performance of a mastoidectomy procedure that is shown to apply universally across teaching institutions has not yet been devised. This work describes the development of a rating instrument created from a multi-institutional consortium. Design Using a participatory design and a modified Delphi approach, a multi-institutional group of expert otologists constructed a 15 element task-based checklist for evaluating mastoidectomy performance. This instrument was further refined into a 14 element checklist focusing on the concept of safety after using it to rate a large and varied population of performances. Setting Twelve Otolaryngological surgical training programs in the United States. Participants 14 surgeons from 12 different institutions took part in the construction of the instrument. Results By using 14 experts from 12 different institutions and a literature review, individual metrics were identified, rated as to the level of importance and operationally defined to create a rating scale for mastoidectomy performance. Initial use of the rating scale showed modest rater agreement. The operational definitions of individual metrics were modified to emphasize “safe” as opposed to “proper” technique. A second rating instrument was developed based on this feedback. Conclusions Using a consensus building approach with multiple rounds of communication between experts is a feasible way to construct a rating instrument for mastoidectomy. Expert opinion alone using a Delphi method provides face and content validity evidence, however, this is not sufficient to develop a universally acceptable rating instrument. A continued process of development and experimentation to demonstrate evidence for reliability and validity making use of a large population of raters and performances is necessary to achieve universal acceptance.

show abstract

Construct validity of cadaveric temporal bones for training and assessment in mastoidectomy

Abstract: N/A.

Cited by 22 publications

References 17 publications

Cross-institutional evaluation of a mastoidectomy assessment instrument

Cross-institutional evaluation of a mastoidectomy assessment instrument

Systematic review of the current status of cadaveric simulation for surgical training

Multi-Institutional Development of a Mastoidectomy Performance Evaluation Instrument

Contact Info

Product

Resources

About