Toolbox of assessment tools of technical skills in otolaryngology–head and neck surgery: A systematic review

Abstract

Our study reports on published tools for assessing technical skills for OTL-HNS residents during core procedures conducted in the OR. These assessment tools could facilitate the provision of timely feedback to trainees including specific goals for improvement. However, the paucity of publications suggests little agreement on how to best perform work-based direct-observation assessment for core surgical procedures in OTL-HNS. The sparsity of tools specific to OTL-HNS may become a barrier to a fluid transition to competency-based medical education. Laryngoscope, 128:1571-1575, 2018.

Validity evidence as a key marker of quality of technical skill assessment in OTL–HNS

“…We performed a secondary analysis of data generated through a systematic review conducted by our group in 2016. As reported in the initial study, we searched for all OTL–HNS assessment tools evaluating technical skills of residents in the operating room (OR). The complete review methodology, as well as the Preferred Reporting Items for Systematic Reviews and Meta‐Analyses diagram, is detailed in the initial study …”

“…For this secondary analysis, we included all 16 articles from the previous review and focused on validity evidence reported in each article. Data extraction included sample size, validation setting (e.g., OR, cadaver laboratory), assessment process (e.g., direct observation, recordings), and type of assessor(s).…”

“…The complete review methodology, as well as the Preferred Reporting Items for Systematic Reviews and Meta-Analyses diagram, is detailed in the initial study. 8 For this secondary analysis, we included all 16 articles from the previous review 8 and focused on validity evidence reported in each article. Data extraction included sample size, validation setting (e.g., OR, cadaver laboratory), assessment process (e.g., direct observation, recordings), and type of assessor(s).…”

“…Current assessment approaches primarily involve the monitoring of case number with end‐of‐rotation global faculty judgments of resident competence, which, if not supported by frequent assessments during the rotation, have been shown to lack validity and reliability . To address the new assessment requirements in CBME, including an increased reliance on workplace‐based observation, assessment, and feedback, a number of direct‐observation rater‐based assessment tools for specific OTL–HNS procedures are being reported in the literature …”

“…[4][5][6][7] To address the new assessment requirements in CBME, including an increased reliance on workplace-based observation, assessment, and feedback, a number of direct-observation rater-based assessment tools for specific OTL-HNS procedures are being reported in the literature. 8 Quality monitoring of assessment practices should be a priority in all residency programs to ensure decisions based on assessment scores are as defensible as possible. More specifically, to better support judgments of performance or decisions regarding resident competency, assessment tools should be used according to their intended purpose, as well as monitored to ensure they are generating defensible and actionable scores.…”