The Agreement Between Virtual Patient and Unannounced Standardized Patient Assessments in Evaluating Primary Health Care Quality: Multicenter, Cross-sectional Pilot Study in 7 Provinces of China

Abstract

Background The unannounced standardized patient (USP) is the gold standard for primary health care (PHC) quality assessment but has many restrictions associated with high human and resource costs. Virtual patient (VP) is a valid, low-cost software option for simulating clinical scenarios and is widely used in medical education. It is unclear whether VP can be used to assess the quality of PHC. Objective This study aimed to examine the agreement between VP and USP assessments of PHC quality and to identify factors influencing the VP-USP agreement. Methods Eleven matched VP and USP case designs were developed based on clinical guidelines and were implemented in a convenience sample of urban PHC facilities in the capital cities of the 7 study provinces. A total of 720 USP visits were conducted, during which on-duty PHC providers who met the inclusion criteria were randomly selected by the USPs. The same providers underwent a VP assessment using the same case condition at least a week later. The VP-USP agreement was measured by the concordance correlation coefficient (CCC) for continuity scores and the weighted κ for diagnoses. Multiple linear regression was used to identify factors influencing the VP-USP agreement. Results Only 146 VP scores were matched with the corresponding USP scores. The CCC for medical history was 0.37 (95% CI 0.24-0.49); for physical examination, 0.27 (95% CI 0.12-0.42); for laboratory and imaging tests, –0.03 (95% CI –0.20 to 0.14); and for treatment, 0.22 (95% CI 0.07-0.37). The weighted κ for diagnosis was 0.32 (95% CI 0.13-0.52). The multiple linear regression model indicated that the VP tests were significantly influenced by the different case conditions and the city where the test took place. Conclusions There was low agreement between VPs and USPs in PHC quality assessment. This may reflect the “know-do” gap. VP test results were also influenced by different case conditions, interactive design, and usability. Modifications to VPs and the reasons for the low VP-USP agreement require further study.

Learning and clinical reasoning experience of second-year medical pharmacology students and teachers with virtual patients developed using OpenLabyrinth

“…However, not all investigators have found substantial benefits with the use of virtual case scenarios [20]. It was conducted a multicentric pilot study assessing the agreement between standardized patient and virtual patient assessments for evaluating primary health care quality [21]. The agreement between the two evaluation methods was low with the virtual patient results being affected by the design, usability and case conditions [21].…”

Artificial Intelligence and Healthcare Simulation: The Shifting Landscape of Medical Education

“…The investigators found that working with VPs seemed to preferentially enhance students’ skills for interviewing and delivering treatment. By contrast, the student’s success working with human SPs correlated more closely with the student’s basic fund of knowledge in science and clinical areas, as it might be called upon for taking Steps 1 and 2, respectively, of the United States Medical Licensing Examination (USMLE) [ 40 ]. In short, these results suggested that a human performance system (HPS) called for a greater draw of basic and clinical facts, while VPs might be better suited to practicing bedside communication skills.…”

Virtual Standardized Patients Versus Traditional Academic Training for Improving Clinical Competence Among Traditional Chinese Medicine Students: Prospective Randomized Controlled Trial

“…Our team also used OSPs and student standardized patients (SSPs) in early-stage teaching, which yielded positive student learning outcomes [ 8 , 9 ]. However, OSPs and SSPs require costly personnel and material resources, significantly limiting their promotion, whereas VSPs are a low-cost tool [ 21 ]. The Chengdu University of Traditional Chinese Medicine (CDUTCM) is one of the few TCM universities that launched the VSP-TCM project.…”