Implementation and evaluation of a virtual objective structured clinical examination for pharmacy students

Abstract

IntroductionThe University of California, San Francisco (UCSF) School of Pharmacy employs Objective Structured Clinical Examinations (OSCEs) to assess students' skills. The Coronavirus disease 2019 (COVID‐19) pandemic made in‐person OSCEs infeasible. This article describes the implementation and evaluation of a two‐station virtual OSCE (vOSCE) for first‐year pharmacy students.MethodsA summative vOSCE was conducted in June 2020 using Zoom and ExamSoft. Faculty and residents served as standardized patients (SPs) and evaluated student performance. Students and SPs were surveyed on their perceptions post‐vOSCE. A mixed methods analysis was conducted, incorporating thematic analysis of survey responses and quantitative analyses of student performance on the virtual vs in‐person OSCE.ResultsNinety‐six (75%) students and 17 SPs (94.4%) completed the survey. Students and SPs (69% vs 88%, P = .10) felt that vOSCEs were acceptable alternatives for demonstrating knowledge, communication, and telehealth skills. SPs felt that in‐person OSCEs were more effective to assess communication skills and ability to conduct in‐person encounters (P < .05). Qualitative comments from students and SPs corroborated these differences, emphasizing limitations of vOSCEs for demonstrating nonverbal skills, and additional considerations regarding exam administration and assessment. Students who failed an OSCE generally had lower median communication scores (P < .01). Technological difficulties, before OSCE performance, and before telehealth experience were not associated with passing the vOSCE (P > 0.05).DiscussionThe virtual environment did not seem to negatively impact student performance. However, students and SPs felt vOSCEs may be less effective to demonstrate and assess nonverbal communication. To address these limitations, future implementation of vOSCEs may include providing students with instruction on nonverbal telehealth skills and SP training on virtual assessment of nonverbal communication skills.

Implementation of virtual OSCE in health professions education: A systematic review

“…Others struggled to develop rapport because of the reduction in non‐verbal communication cues over the screen. One student commented it was ‘difficult to maintain eye contact and generate rapport’, 50 and another found it ‘difficult to be empathetic’ because of a ‘disconnection’ 53 …”

“…Accessibility in terms of an institution's ability to operate assessments virtually was also raised. Students thought the new platform was ‘an optimization of resources’ and ‘achieve[d] the same goals’ as face‐to‐face assessments 50 . Faculty members commented on ‘new resource requirements’, such as technology and additional time needed to organise this novel assessment, but found these to be an initial hurdle that once overcome was ‘relatively inexpensive’ 59,62 .…”

“…Furthermore, it will be important to continuously review and evaluate emerging technological solutions to help maximise assessment integrity through appropriate surveillance and sequestration. 46,47,59 Given that both students and examiners commented on the utility of vOSCE in preparing students for an anticipated greater use of telehealth in the future, 50,52,55,56,59,60 this may also be an area that can be prioritised in future assessment approaches.…”

“…47,62 Others struggled to develop rapport because of the reduction in non-verbal communication cues over the screen. One student commented it was 'difficult to maintain eye contact and generate rapport', 50 and another found it 'difficult to be empathetic' because of a 'disconnection'. 53 Faculty members were initially concerned about the standardisation and fairness of the virtual examination process but were more confident after examiner training and station calibration sessions.…”

“…Students thought the new platform was 'an optimization of resources' and 'achieve [d] the same goals' as face-to-face assessments. 50 Faculty members commented on 'new resource requirements', such as technology and additional time needed to organise this novel assessment, but found these to be an initial hurdle that once overcome was 'relatively inexpensive'. 59,62 One assessor was also not in favour of virtual assessment, commenting 'there is far more preparation for examiners … compared to face-to-face where the centres have prepped everything for the examiner to just turn up'.…”