Feasibility and preliminary validity evidence for remote video-based assessment of clinicians in a global health setting

Abstract

Background Serious childhood illnesses (SCI), defined as severe pneumonia, severe dehydration, sepsis, and severe malaria, remain major contributors to amenable child mortality worldwide. Inadequate recognition and treatment of SCI are factors that impact child mortality in Botswana. Skills assessments of providers caring for SCI have not been validated in low and middle-income countries. Objective To establish preliminary inter-rater reliability, validity evidence, and feasibility for an assessment of providers who care for SCI using simulated patients and remote video capture in community clinic settings in Botswana. Methods This was a pilot study. Four scenarios were developed via a modified Delphi technique and implemented at primary care clinics in Kweneng, Botswana. Sessions were video captured and independently reviewed. Response process and internal structure analysis utilized intra-class correlation (ICC) and Fleiss’ Kappa. A structured log was utilized for feasibility of remote video capture. Results Eleven subjects participated. Scenarios of Lower Airway Obstruction (ICC = 0.925, 95%CI 0.695–0.998) and Hypovolemic Shock from Severe Dehydration (ICC = 0.892, 95%CI 0.596–0.997) produced excellent ICC among raters while Lower Respiratory Tract Infection (LRTI, ICC = 0, 95%CI -0.034–0.97) and LRTI + Distributive Shock from Sepsis (0.365, 95%CI -0.025–0.967) were poor. Oxygen therapy (0.707), arranging transport (0.706), and fluid administration (0.701) demonstrated substantial task reliability. Conclusions Initial development of an assessment tool demonstrates many, but not all, criteria for validity evidence. Some scenarios and tasks demonstrate excellent reliability among raters, but others may be limited by manikin design and study implementation. Remote simulation assessment of some skills by clinic-based providers in global health settings is reliable and feasible.

Pediatric Resuscitation Education in the Age of COVID-19

“…Though more data is needed to make recommendations, this pilot study adds to the growing evidence that videoconferencing can be an effective tool for teaching complex topics, and could be routinely deployed in a global health setting (8,(10)(11)(12)14). Future studies should compare changes in confidence level following in-person courses versus videoconferencing courses, as well as long-term follow-ups to track participant knowledge and skill retention.…”

“…Videoconferencing has played a valuable role in global health work and is cited as a component in many courses 8,9 . Previous pilots showed videoconferencing as a successful tool for aspects of medical education courses in low and middle-income countries (LMIC) including patient simulation and ultrasound training, with equivalently accurate evaluation of hands-on testing compared to in-person scoring 10,11 . In 2008, a domestic U.S.-based study found similar outcomes of conducting a PALS course in-person and via videoconferencing, indicating the effectiveness of virtual learning with a PALS course 12 .…”

Development, validation and feasibility study of a remote basic skills assessment for wheelchair service providers

“…In particular, two studies have revealed remote skills test approaches are feasible. A study by Smith et al (2019) conducted a feasibility study on a remote video-based assessment for clinicians in a global health setting. This pilot study established preliminary inter-rater reliability, validity evidence, and feasibility for an assessment of spinal cord injury (SCI) care providers using simulated patients and remote video capture in community clinic settings.…”

“…The literature review suggests a feasibility study is an appropriate method to identify whether remote skills tests are appropriate for particular fields and settings (Cohen, 2013;Smith, 2019). This study therefore aimed to conduct a feasibility study of all skills assessment approaches (modalities 1, 2 and 3).…”

“…Several studies (Kirby et al, 2019, Munera et al, 2019, Smith et al, 2019, and Cohen et al, 2013 offered examples of remote skills testing strategies for WSP and other rehabilitation providers. They concluded that a remote testing format is reliable and feasible to assess WSP and other rehabilitation care providers" skills.…”