Direct Observation Assessment of Ultrasound Competency Using a Mobile Standardized Direct Observation Tool Application With Comparison to Asynchronous Quality Assurance Evaluation

Abstract

Objectives: Competency assessment is a key component of point-of-care ultrasound (POCUS) training. The purpose of this study was to design a smartphone-based standardized direct observation tool (SDOT) and to compare a faculty-observed competency assessment at the bedside with a blinded reference standard assessment in the quality assurance (QA) review of ultrasound images. Methods: In this prospective, observational study, an SDOT was created using SurveyMonkey containing specific scoring and evaluation items based on the Council of Emergency Medicine Residency-Academy of Emergency Ultrasound: Consensus Document for the Emergency Ultrasound Milestone Project. Ultrasound faculty used the mobile phone-based data collection tool as an SDOT at the bedside when students, residents, and fellows were performing one of eight core POCUS examinations. Data recorded included demographic data, examination-specific data, and overall quality measures (on a scale of 1-5, with 3 and above being defined as adequate for clinical decision making), as well as interpretation and clinical knowledge. The POCUS examination itself was recorded and uploaded to QPath, a HIPAA-compliant ultrasound archive. Each examination was later reviewed by another faculty blinded to the result of the bedside evaluation. The agreement of examinations scored adequate (3 and above) in the two evaluation methods was the primary outcome.

Tools for Measuring Clinical Ultrasound Competency: Recommendations From the Ultrasound Competency Work Group

“…Incorporating an SDOT on a scheduled basis ensures consistency of the evaluation across learners. A recent study demonstrated that measuring clinical ultrasound competency via in situ clinical observation as measured by SDOT correlated well with assessment via asynchronous image review …”

The Ultrasound Competency Assessment Tool (UCAT): Development and Evaluation of a Novel Competency‐based Assessment Tool for Point‐of‐care Ultrasound

“…While structured assessment methods for POCUS assessment exist, few have undergone robust evaluation, and none are in widespread use. [9][10][11][12][13][14] In conjunction with the content experts on the CAEP EUC, we used modified Delphi methodology to create a universally applicable, simple, entrustment-based, POCUS competency assessment tool and subsequently piloted it in an OSCE scenario.…”

A comparison of handheld ultrasound versus traditional ultrasound for acquisition of RUSH views in healthy volunteers

“…Although there are no prior data comparing these 2 devices, prior literature has demonstrated an interpretability of 89% in resident ultrasonographers. The variance was not reported 8 . Assuming a variance of 10% in this cohort, a sample size of 50 would be able to detect a 6% difference in interpretability between handheld and traditional groups with power of 0.8 and alpha of 0.05.…”