Blinded Evaluation of Interrater Reliability of an Operative Competency Assessment Tool for Direct Laryngoscopy and Rigid Bronchoscopy

Ishman, Stacey L.; Benke, James R.; Johnson, Kaalan; Zur, Karen B.; Jacobs, Ian N.; Thorne, Marc C.; Brown, David J.; Lin, Sandra Y.; Bhatti, Nasir I.; Deutsch, Ellen S.

doi:10.1001/2013.jamaoto.115

Cited by 20 publications

(30 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The investigators adapted version of an OSATS module using a criterion-reference rating scale findings were similar to previous airway skills performance assessments. [13,16,17] The inter-rater reliability results show that each of the three OSATS tools have face validity. [1] Each OSATS tool enabled faculty-evaluators to focus on task completion and provide immediate feedback.…”

Section: Discussionmentioning

confidence: 91%

“…Inter-rater reliability was "acceptable". [13] Feasibility was found with both the checklist and global rating scale when completion times were between three and five minutes. Interrater reliability for both the checklist and global rating tools were excellent.…”

Section: Review Of the Literaturementioning

confidence: 99%

“…The authors concluded that the evaluators were able to provide immediate feedback to residents regarding their skills performance. [13] VanHeest et al developed, tested, and published the first OSATS tool using a procedure specific detailed checklist, a validated global rating scale, and a pass/fail assessment for 27 residents performing surgery on a cadaveric specimen at three stations involving trigger finger release (TFR), open carpal tunnel release (CTR), and distal radius fracture fixation (DTFF). Construct validity was found between the year in training and the following assessments: procedure-specific detailed checklist scores for TFR and open CTR; global rating scores for TFR and DTFF; and, pass/fail assessment for TFR.…”

Section: Review Of the Literaturementioning

confidence: 99%

See 2 more Smart Citations

Testing an objective structured assessment technical skills tool: A pilot study

Dixon¹,

Burns²

2015

JNEP

View full text Add to dashboard Cite

Purpose: The purpose of this prospective pilot study was to test the feasibility, reliability and validity of an adapted version of an objective structured assessment technical skills (OSATS) tool using a criterion-reference rating scale while evaluating student registered nurse anesthetists (SRNAs) performing mask ventilation, laryngeal mask airway (LMA) insertion, and endotracheal intubation. Methods: Prior to testing, three OSATS evaluation tools were developed with nurse anesthesia faculty input. Six facultyevaluators pilot tested an adapted version of three OSATS tools for feasibility and validity with 23 SRNAs performing mask ventilation, LMA insertion, and endotracheal intubation with an airway management trainer. SRNAs had 15 minutes; five-minute each to perform mask ventilation, insert an LMA, and perform endotracheal intubation. Paired observations from six facultyevaluators determined inter-rater reliability. Separate survey questionnaires were created for faculty-evaluators and SRNAs; using three four-point Likert-type rating scales to measure feasibility and validity. Results: Descriptive statistics revealed that faculty-evaluators' found the OSATs tools to be feasible based on six out of seven somewhat favorable to strongly favorable responses. The Kendall's Coefficient of Concordance W showed statistical significance for mask ventilation and endotracheal intubation (p-value of .041 and .036). Six faculty-evaluators found the OSATs tools to have face and content validity based on six out of seven somewhat favorable responses. Twenty-three SRNAs found the OSATS tools to have face validity based on somewhat strongly favorable responses. Conclusions: An adapted version of OSATS tool for evaluating SRNAs performing mask ventilation, LMA insertion, and endotracheal intubation demonstrated feasibility, reliability, and validity. The inter-rater values obtained from Kendall's Coefficient of Concordance W in this project correlated with the Cook and Beckman report.

show abstract

Section: Discussionmentioning

confidence: 91%

Section: Review Of the Literaturementioning

confidence: 99%

Section: Review Of the Literaturementioning

confidence: 99%

See 1 more Smart Citation

Testing an objective structured assessment technical skills tool: A pilot study

Dixon¹,

Burns²

2015

JNEP

View full text Add to dashboard Cite

show abstract

“…2 Several studies have introduced objective procedure-specific assessment tools for tonsillectomy, endoscopic sinus surgery, direct laryngoscopy and bronchoscopy, and mastoidectomy. [5][6][7][8] A logical initial step in implementation of procedural competency assessments is to determine the approximate level of training at which residents should be expected to demonstrate competence in key procedures.…”

Section: Discussionmentioning

confidence: 99%

Attainment of surgical competence in otolaryngology training

et al. 2014

View full text Add to dashboard Cite

show abstract

“…A task-specific portion was developed by dividing the operative procedure into the specific tasks deemed to be necessary for the completion of a safe and efficient surgery, based upon previously developed anchors. 18 A global-operative performance portion of the scale was developed based on the American College of Surgeons Global rating scale. Task-specific performance and global-operative performance each had six specific aspects that were rated on a five-point Likert scale, for a total score of 30 in each of the two categories.…”

Section: Osats Developmentmentioning

confidence: 99%