Collecting Validity Evidence for Simulation‐Based Assessment of Point‐of‐Care Ultrasound Skills

Jensen, Jesper Kørup; Dyre, Liv; Jørgensen, Mads Emil; Andreasen, Lisbeth Anita; Tolsgaard, Martin G.

doi:10.1002/jum.14292

Cited by 19 publications

(16 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this light, e‐learning provides an opportunity to expose learners to an image interpretation learning experience (i.e., part‐task) that could complement the resource intensive face‐to‐face teaching and learning at the bedside that addresses all facets of POCUS (i.e., whole‐task) . However, to date, most POCUS simulation and online education focuses on image acquisition, specific conditions (e.g., cardiac tamponade), or adult applications . As such, existing teaching tools are limited in being able to significantly improve learner exposure to the part‐task of pediatric POCUS image interpretation.…”

mentioning

confidence: 99%

The Variable Journey in Learning to Interpret Pediatric Point‐of‐care Ultrasound Images: A Multicenter Prospective Cohort Study

Kwan

Pusic

Pecaric³

et al. 2019

AEM Education and Training

View full text Add to dashboard Cite

Objectives: To complement bedside learning of point-of-care ultrasound (POCUS), we developed an online learning assessment platform for the visual interpretation component of this skill. This study examined the amount and rate of skill acquisition in POCUS image interpretation in a cohort of pediatric emergency medicine (PEM) physician learners.Methods: This was a multicenter prospective cohort study. PEM physicians learned POCUS using a computerbased image repository and learning assessment system that allowed participants to deliberately practice image interpretation of 400 images from four pediatric POCUS applications (soft tissue, lung, cardiac, and focused assessment sonography for trauma [FAST]). Participants completed at least one application (100 cases) over a 4week period.Results: We enrolled 172 PEM physicians (114 attendings, 65 fellows). The increase in accuracy from the initial to final 25 cases was 11.6%, 9.8%, 7.4%, and 8.6% for soft tissue, lung, cardiac, and FAST, respectively. For all applications, the average learners (50th percentile) required 0 to 45, 25 to 97, 66 to 175, and 141 to 290 cases to reach 80, 85, 90, and 95% accuracy, respectively. The least efficient (95th percentile) learners required 60 to 288, 109 to 456, 160 to 666, and 243 to 1040 cases to reach these same accuracy benchmarks. Generally, the soft tissue application required participants to complete the least number of cases to reach a given proficiency level, while the cardiac application required the most.Conclusions: Deliberate practice of pediatric POCUS image cases using an online learning and assessment platform may lead to skill improvement in POCUS image interpretation. Importantly, there was a highly variable rate of achievement across learners and applications. These data inform our understanding of POCUS image interpretation skill development and could complement bedside learning and performance assessments.From the

show abstract

mentioning

confidence: 99%

The Variable Journey in Learning to Interpret Pediatric Point‐of‐care Ultrasound Images: A Multicenter Prospective Cohort Study

Kwan

Pusic

Pecaric³

et al. 2019

AEM Education and Training

View full text Add to dashboard Cite

show abstract

“…Future studies could assess the time and number of attempts the novices need to reach the pass/fail level and to compare this to the number of attempts and time needed to reach the learning curve plateau. This will highlight whether novices acquire additional learning beyond the pass/fail level, as previous studies have suggested [33, 34]. …”

Section: Discussionmentioning

confidence: 88%

“…Previous studies show that there is a difference between pass/fail level and learning curve plateau, indicating that the trainees might be proficiently trained when reaching the pass/fail level, but still have potential to reach a higher level of competence until reaching a learning curve plateau [30, 33, 34]. When reaching the learning curve plateau the trainees would not benefit from further simulation training.…”

Section: Discussionmentioning

confidence: 99%

Ensuring Basic Competence in Thoracentesis

et al. 2019

View full text Add to dashboard Cite

Background: Trocar pigtail catheter thoracentesis (TPCT) is a common procedure often performed by junior physicians. Simulation-based training may effectively train physicians in the procedure prior to performing it on patients. An assessment tool with solid validity evidence is necessary to ensure sufficient procedural competence. Objectives: Our study objectives were (1) to collect evidence of validity for a newly developed pigtail catheter assessment tool (Thoracentesis Assessment Tool [ThorAT]) developed for the evaluation of TPCT performance and (2) to establish a pass/fail score for summative assessment. Methods: We assessed the validity evidence for the ThorAT using the recommended framework for validity by Messick. Thirty-four participants completed two consecutive procedures and their performance was assessed by two blinded, independent raters using the ThorAT. We compared performance scores to test whether the assessment tool was able to discern between the two groups, and a pass/fail score was established. Results: The assessment tool was able to discriminate between the two groups in terms of competence level. Experienced physicians received significantly higher test scores than novices in both the first and second procedure. A pass/fail score of 25.2 points was established, resulting in 4 (17%) passing novices and 1 (9%) failing experienced participant in the first procedure. In the second procedure 9 (39%) novices passed and 2 (18%) experienced participants failed. Conclusions: This study provides a tool for summative assessment of competence in TPCT. Strong validity evidence was gathered from five sources of evidence. A simulation-based training program using the ThorAT could ensure competence before performing thoracentesis on patients.

show abstract

“…To our knowledge, no studies have gathered validity evidence for a practical test in LUS, even though there has been an increased focus on the use of ultrasound for thoracic assessment, including the accuracy and eligibility of LUS [28]. Assessment tools for the evaluation of ultrasound skills in other organ systems have been published within the last decade [29-32]. …”

Section: Discussionmentioning

confidence: 99%

Developing and Gathering Validity Evidence for a Simulation-Based Test of Competencies in Lung Ultrasound

et al. 2018

View full text Add to dashboard Cite

Background: Clinical lung ultrasound (LUS) is a fast bedside diagnostic tool which can assist clinicians in decisions regarding the treatment and monitoring of patients with respiratory symptoms. LUS training and education differ widely, and is often done in a clinical setting, with potential risks for patients if decisions are made based on the wrong interpretations. No clear guidelines or recommendations for objective and standardized assessment of LUS skills exist, and those that do are often based on a fixed time-frame or an arbitrary number of examinations performed; this does not ensure adequate competencies. Objectives: The study aimed to develop and gather validity evidence for a practical, simulation-based test in LUS. Methods: Nine cases were developed in collaboration with 3D Systems Healthcare, Littleton, CO, USA, representing the most common diagnosis and sonographic findings in patients with respiratory symptoms. Thirty-six participants with different levels of competence in LUS, completed the test. The participants were divided into groups, i.e., novices, intermediates, and experienced, according to their experience with LUS, the number of examinations they had performed, and any research they had conducted. Their answers were used for item analyses. Results: The intraclass correlation coefficient, Cronbachs’ α, was 0.69 summarized, and there was a statistically significant difference (p < 0.001) between the novices and the trained participants (intermediates and experienced). A pass/fail score of 16 points was calculated according to the contrasting-groups method. Conclusion: We developed a test for the assessment of clinical competencies in LUS. The test proved solid validity evidence, and a pass/fail standard without any false-negatives, and only 2 explained false-positives.

show abstract

Collecting Validity Evidence for Simulation‐Based Assessment of Point‐of‐Care Ultrasound Skills

Abstract: The performance of FAST examinations can be assessed in a simulated setting using defensible performance standards, which have both good reliability and validity.

Cited by 19 publications

References 21 publications

The Variable Journey in Learning to Interpret Pediatric Point‐of‐care Ultrasound Images: A Multicenter Prospective Cohort Study

The Variable Journey in Learning to Interpret Pediatric Point‐of‐care Ultrasound Images: A Multicenter Prospective Cohort Study

Ensuring Basic Competence in Thoracentesis

Developing and Gathering Validity Evidence for a Simulation-Based Test of Competencies in Lung Ultrasound

Contact Info

Product

Resources

About