Do Attending and Trainee Surgeons Agree on What Happens in the Operating Room During Septoplasty?

Ryan, John F.; Malpani, Anand; Naz, Hajira; Boahene, Kofi; Papel, Ira D.; Kontis, Theda C.; Maxwell, Jessica H.; Creighton, Francis X.; Byrne, Patrick J.; Wanamaker, John R.; Hager, Gregory D.; Vedula, S. Swaroop; Malekzadeh, Sonya; Ishii, Lisa E.; Ishii, Masaru

doi:10.1089/fpsam.2021.0327

Cited by 1 publication

(2 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Another reason for unprepared surgeons is that assessment of trainees' skill is affected by rater effects. Indirect evidence of the impact of rater effects on learning is seen in studies that show disagreement between attendings and trainees on trainees' skill 15 …”

Section: Discussionmentioning

confidence: 99%

“…Indirect evidence of the impact of rater effects on learning is seen in studies that show disagreement between attendings and trainees on trainees' skill. 15 Rater effects introduce variance that is not relevant to the construct being assessed (e.g., surgical skill). They lead to poor reliability (e.g., inter-rater agreement) and consequently, poor validity of assessments.…”

Section: Ethnicitymentioning

confidence: 99%

See 1 more Smart Citation

Correcting for Rater Effects in Operating Room Surgical Skills Assessment

Chou,

Naz,

Boahene

et al. 2024

The Laryngoscope

Self Cite

View full text Add to dashboard Cite

ObjectiveTo estimate and adjust for rater effects in operating room surgical skills assessment performed using a structured rating scale for nasal septoplasty.MethodsWe analyzed survey responses from attending surgeons (raters) who supervised residents and fellows (trainees) performing nasal septoplasty in a prospective cohort study. We fit a structural equation model with the rubric item scores regressed on a latent component of skill and then fit a second model including the rating surgeon as a random effect to model a rater‐effects‐adjusted latent surgical skill. We validated this model against conventional measures including the level of expertise and post‐graduation year (PGY) commensurate with the trainee's performance, the actual PGY of the trainee, and whether the surgical goals were achieved.ResultsOur dataset included 188 assessments by 7 raters and 41 trainees. The model with one latent construct for surgical skill and the rater as a random effect was the best. Rubric scores depended on how severe or lenient the rater was, sometimes almost as much as they depended on trainee skill. Rater‐adjusted latent skill scores increased with attending‐estimated skill levels and PGY of trainees, increased with the actual PGY, and appeared constant over different levels of achievement of surgical goals.ConclusionOur work provides a method to obtain rater effect adjusted surgical skill assessments in the operating room using structured rating scales. Our method allows for the creation of standardized (i.e., rater‐effects‐adjusted) quantitative surgical skill benchmarks using national‐level databases on trainee assessments.Level of EvidenceN/A Laryngoscope, 2024

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Ethnicitymentioning

confidence: 99%