Development and pilot testing of an operative competency assessment tool for pediatric direct laryngoscopy and rigid bronchoscopy

Ishman, Stacey L.; Brown, David J.; Boss, Emily F.; Skinner, Michael; Tunkel, David E.; Stavinoha, Rose; Lin, Sandra Y.

doi:10.1002/lary.21067

Cited by 53 publications

(73 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Face and content validity were found when five of the six facultyevaluators responded favorable to the OSATS tools comprehensibility, ability to differentiate level of performance, assess other clinical training activities, and complete the tool and provide immediate feedback. [4] Kendall's Coefficient of Concordance W (inter-rater reliability) yielded a statistical significance for mask ventilation and ETT intubation (w = 0.80, χ 2 = 35.3, p-value = .036) (see Table 8). The Kendall's Coefficient of Concordance W was not significant for LMA insertion (w = 0.64, χ 2 = 28.3, p-value was .166) (see Table 8).…”

Section: Resultsmentioning

confidence: 98%

“…[10,11,20] The favorable responses by SRNAs indicate that each OSATS tool enhanced faculty-evaluators abilities to provide formal, real-time and immediate feedback. [4,19] The investigators were able to standardize practical skills for evaluating airway management skills performances for simulation laboratory testing. [2] The 50/50 survey responses by the SRNAs to questions asking whether their skills were assessed objectively or subjectively is unknown but, may be due to a perception that the tools were biased, unrealistic, and did not reflect their previous graded skills performance.…”

Section: Discussionmentioning

confidence: 99%

“…Construct validity was found with both the task specific and the global rating scale (p < .001). [4] Laeeq et al developed and evaluated the reliability and validity of an training objective assessment tool using a global rating scale and a checklist for endoscopic sinus surgery skills in the operating room. The authors found both the global rating scale and checklist instruments possessed a high internal consistency and construct validity.…”

Section: Review Of the Literaturementioning

confidence: 99%

“…[22] The Accreditation Council for Graduate Medical Education (ACGME) proposed that specific outcomes should assess competency and improve education and performance feedback. [4] 2. METHODS…”

Section: Review Of the Literaturementioning

confidence: 99%

“…[3] An assessment tool without a criterion-reference rating scale, the evaluator may exhibit personal bias, present a wide range of opinions, or have personality issues with a student. [4] Subjective evaluations do not always provide detailed feedback or identify proficiencies or deficiencies in a timely manner. [5] According to Hall, standardize models for evaluating student nurses' skills performance are not in existence and clinical evaluations are usually subjective.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Testing an objective structured assessment technical skills tool: A pilot study

Dixon¹,

Burns²

2015

JNEP

View full text Add to dashboard Cite

Purpose: The purpose of this prospective pilot study was to test the feasibility, reliability and validity of an adapted version of an objective structured assessment technical skills (OSATS) tool using a criterion-reference rating scale while evaluating student registered nurse anesthetists (SRNAs) performing mask ventilation, laryngeal mask airway (LMA) insertion, and endotracheal intubation. Methods: Prior to testing, three OSATS evaluation tools were developed with nurse anesthesia faculty input. Six facultyevaluators pilot tested an adapted version of three OSATS tools for feasibility and validity with 23 SRNAs performing mask ventilation, LMA insertion, and endotracheal intubation with an airway management trainer. SRNAs had 15 minutes; five-minute each to perform mask ventilation, insert an LMA, and perform endotracheal intubation. Paired observations from six facultyevaluators determined inter-rater reliability. Separate survey questionnaires were created for faculty-evaluators and SRNAs; using three four-point Likert-type rating scales to measure feasibility and validity. Results: Descriptive statistics revealed that faculty-evaluators' found the OSATs tools to be feasible based on six out of seven somewhat favorable to strongly favorable responses. The Kendall's Coefficient of Concordance W showed statistical significance for mask ventilation and endotracheal intubation (p-value of .041 and .036). Six faculty-evaluators found the OSATs tools to have face and content validity based on six out of seven somewhat favorable responses. Twenty-three SRNAs found the OSATS tools to have face validity based on somewhat strongly favorable responses. Conclusions: An adapted version of OSATS tool for evaluating SRNAs performing mask ventilation, LMA insertion, and endotracheal intubation demonstrated feasibility, reliability, and validity. The inter-rater values obtained from Kendall's Coefficient of Concordance W in this project correlated with the Cook and Beckman report.

show abstract

Section: Resultsmentioning

confidence: 98%

Section: Discussionmentioning

confidence: 99%

Section: Review Of the Literaturementioning

confidence: 99%

Section: Review Of the Literaturementioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Testing an objective structured assessment technical skills tool: A pilot study

Dixon¹,

Burns²

2015

JNEP

View full text Add to dashboard Cite

show abstract

Blinded Evaluation of Interrater Reliability of an Operative Competency Assessment Tool for Direct Laryngoscopy and Rigid Bronchoscopy

Ishman¹,

Benke²,

Johnson³

et al. 2012

Arch Otolaryngol Head Neck Surg

Self Cite

View full text Add to dashboard Cite

Assessment of operative competency for thyroidectomy: Comparison of resident self‐assessment vs attending surgeon assessment

et al. 2020

View full text Add to dashboard Cite

Background: Self-awareness of skill, essential for progression as a surgeon, has direct bearing on postresidency practice; however, studies have supported that residents achieve self-perceived competence later than believed by program directors. This study compares residents' self-perception of operative competency to attending surgeon's evaluation using Computer Enhanced Visual Learning, a validated online hemithyroidectomy-specific assessment tool. Methods: Eleven otolaryngology-Head and Neck Surgery (HNS) residents completed a preoperative module and postoperative survey, later reviewed by an attending surgeon. Eighty-three performances were assessed for inter-rater reliabilities of key surgical steps. Results: Almost perfect agreement (Kappa = 0.81-1.00) was shown in 11 of 18 parameters. Substantial agreement (Kappa = 0.61-0.80) was demonstrated in the remaining seven parameters. Conclusions: Otolaryngology-HNS trainees have high self-awareness of their performance at each step in a hemithyroidectomy. Standardized assessment tools can allow for documentation of procedural performance and serve as guides for improvement. This is the only study to examine otolaryngology-HNS trainees' self-perceived skill compared to an attending surgeon's assessment for hemithyroidectomy.

show abstract

Development and pilot testing of an operative competency assessment tool for pediatric direct laryngoscopy and rigid bronchoscopy

Cited by 53 publications

References 10 publications

Testing an objective structured assessment technical skills tool: A pilot study

Testing an objective structured assessment technical skills tool: A pilot study

Blinded Evaluation of Interrater Reliability of an Operative Competency Assessment Tool for Direct Laryngoscopy and Rigid Bronchoscopy

Assessment of operative competency for thyroidectomy: Comparison of resident self‐assessment vs attending surgeon assessment

Contact Info

Product

Resources

About