Linking assessment to real life practice - comparing work based assessments and objective structured clinical examinations using mystery shopping

Abstract

Objective Structured Clinical Examinations (OSCEs) and Work Based Assessments (WBAs) are the mainstays of assessing clinical competency in health professions’ education. Underpinned by the extrapolation inference in Kane Validity’s Framework, the purpose of this study is to determine whether OSCEs translate to real life performance by comparing students’ OSCE performance to their performance in real-life (as a WBA) using the same clinical scenario, and to understand factors that affect students’ performance. A sequential explanatory mixed methods approach where a grade comparison between students’ performance in their OSCE and WBA (conducted as a mystery shopper visit at the workplace) was performed, students were then followed up with semi-structured interviews to discuss the students’ experiences in both settings. Students were third year pharmacy undergraduates in 2022. Overall, 92 mystery shopper (WBA) visits with students were conducted and 36 follow-up interviews were completed. The median WBA score was 39.2% lower compared to the OSCE score (p < 0.001). Interviews revealed students knew they did not perform as well in the WBA compared to their OSCE, but reflected that they still need OSCEs to prepare them to manage a real-life patient. Many students related their performance to how they perceived their role in OSCEs versus WBAs, and that OSCEs allowed them more autonomy to manage the patient as oppose to an unfamiliar workplace. As suggested by the activity theory, the performance of the student can be driven by their motivation which differed in the two contexts.