Use of Critical Items in Determining Point-of-Care Ultrasound Competence

Abstract

We previously developed a workplace-based tool for assessing point of care ultrasound (POCUS) skills and used a modified Delphi technique to identify critical items (those that learners must successfully complete to be considered competent). We performed a standard setting procedure to determine cut scores for the full tool and a focused critical item tool. This study compared ratings by 24 experts on the two checklists versus a global entrustability rating. All experts assessed three videos showing an actor performing a POCUS exam on a patient. The performances were designed to show a range of competences and one included potentially critical errors. Interrater reliability for the critical item tool was higher than for the full tool (intraclass correlation coefficient = 0.84 [95% confidence interval [CI] 0.42–0.99] vs. 0.78 [95% CI 0.25–0.99]). Agreement with global ratings of competence was higher for the critical item tool (κ = 0.71 [95% CI 0.55–0.88] vs 0.48 [95% CI 0.30–0.67]). Although sensitivity was higher for the full tool (85.4% [95% CI 72.2–93.9%] vs. 81.3% [95% CI 67.5–91.1%]), specificity was higher for the critical item tool (70.8% [95% CI 48.9–87.4%] vs. 29.2% [95% CI 12.6–51.1%]). We recommend the use of critical item checklists for the assessment of POCUS competence.

Development of a multisystem point of care ultrasound skills assessment checklist

“…Checklists are perceived as being easier to use, especially for non-expert assessors, and having better interrater reliability [ 17 ]. However, checklists may focus more on thoroughness rather than overall competency and may not capture a summative assessment of one’s performance [ 18 , 19 ]. One approach to overcome this limitation is increasing the point-value of critical checklist items, or identifying checklist items that result in immediate disqualification from competency if performed incorrectly [ 18 , 20 ].…”

“…However, checklists may focus more on thoroughness rather than overall competency and may not capture a summative assessment of one’s performance [ 18 , 19 ]. One approach to overcome this limitation is increasing the point-value of critical checklist items, or identifying checklist items that result in immediate disqualification from competency if performed incorrectly [ 18 , 20 ]. By comparison, global rating scales provide an overall assessment of a learner’s skills and can differentiate learner levels with high reliability and sensitivity, particularly when performed by content experts [ 21 – 23 ].…”